User Guide

What the tool does

This tool allows you to enter your confidence in probability judgements concerning an events / events of your choice. Here’s an example of such a confidence assessment from a 2013 IPCC report:

Estimates of the equilibrium climate sensitivity (ECS) … indicate that it is likely [i.e. with probability between 66% and 100%] in the range 1.5°C to 4.5°C with high confidence.

(IPCC AR5, WGII, TS, p81)

In the tool, for any event or variable (e.g. ECS in the range 1.5°C to 4.5°C) and probability interval concerning it (e.g. probability between 66% and 100%), you can enter your confidence in one of several formats (set out below and here).

You can also enter your confidence in an event or a range (without the probability interval), by setting the probability interval to be 100%-to-100% in the tool. In other words, an IPCC assessment such as:

In North America, extreme heat events currently result in increases in mortality and morbidity (very high confidence).

(IPCC AR5, WGII, TS, p42)

can be entered into the tool by assigning high confidence to the probability interval [100%, 100%] for the event “In North America, extreme heat events currently result in increases in mortality and morbidity”.

The tool also presents your confidence evaluations in the form of plots. In the process, it checks consistency of the evaluations, and flags any inconsistencies. (See below for details and here for background.)

User Session

The tool offers the possibility of returning to a previous session, and accessing—and subsequently modifying—all information entered and the resulting plots. All that is required is your user name and the code provided when your account was created. We recommend keeping you code in a safe place, for if lost your account cannot be accessed.

Structure

The tool comprises three functions. Use the tabs at the top to navigate them:

You can only access the Confidence and Plots tabs when an event has been specified and selected in the sidebar.

Events

This tab allows you to enter an event, and the probability intervals concerning it that are of interest.

Select Create New Event in the sidebar menu to enter the details of a new event. You may need to specify:

Event

Events can either be:

  • ranges of a variable (e.g. the average temperature in 2040 will be between 15° and 20°)
  • a single event (e.g. man will land on Mars before 2040)

In the case of a single event, specify the name of the event and the type. If you are interested in ranges of a variable, specify the variable name, the type, and the total range of possible variable values (e.g. the full range of possible average temperatures). You will be asked to fill in the variable ranges of interest (e.g. the ranges of average temperature about which you wish to elicit judgements) below.

You can also set the format you wish to use for confidence evaluation (see Confidence section).

You cannot change the total range of the variable after saving. However, you can change the confidence evaluation format at a later stage.

Range

Whenever the event is a variable, you must select ranges of the variable of interest that will be the objects of judgement.

You can remove a range or add a new one but clicking on the appropriate links.

Probability

For each yes/no event or range of the variable, you can enter a probability interval to be considered:

  • either from a pre-specified list
  • or add your own by typing it directly in. (Make sure to use the following format: [lower bound, upper bound])

The pre-specified list contains outright acceptance and rejection of the event or range (i.e. probability intervals [100%, 100%] or [0%, 0%]) as well as probability intervals drawn from the IPCC uncertainty guidance notes. (See their proposed verbal translation here.)

You may add / delete probability intervals from a range.

Ranges and probability intervals can all be changed when you wish, by selecting the appropriate event in the sidebar and the “Event” tab.

Don’t forget to click Save before leaving the tab.

You may access the events specified using the menu in the sidebar. Events may be deleted (and all information entered concerning them lost) by clicking on . When an event is selected, all ranges (if it is a variable), probability intervals and confidence evaluations are displayed.

Confidence

This tab allows you to enter your confidence in the probability judgements specified. Select the event of interest from the list in the sidebar.

If no confidence assessments have been entered, you can enter those for the relevant probability intervals in order. You can also “jump” to a given probability judgement or change a confidence assessment entered previously using the edit icon in the sidebar.

For a given probability judgement about an event / variable range, you can enter your confidence in it according one of the following format choices (specified in Event tab).

Confidence languages

Two confidence languages are available (see here for more details)

  • Numerical: confidence reported on a 0-100 scale
  • Verbal: verbal confidence terms are reported.
Evaluation formats

Four evaluation formats are available (see here for a detailed description).

  • Numerical: enter the confidence value directly
  • Numerical (aided): supported numerical confidence value reporting
  • Intervals: enter an interval of numerical confidence values
  • Verbal: enter the verbal confidence evaluation directly.

Don’t forget to click Save before leaving the tab.

Plots

This tab gives access to plots of your evaluations. Select the event of interest from the list in the sidebar. You can access plots of the probability assessments / distributions held at each confidence level; use the buttons to toggle the confidence levels displayed.

The plots show the range of probability assessments / distributions concerning the event / variable in question, consistent with the confidence evaluations entered. If the entered evaluations are inconsistent, this will be indicated.

The sidebar contains the list of all evaluations made concerning the event or variable of interest. Using the “ignore” checkboxes, you can select evaluations to ignore in the calculations of the plots. You can use these to explore the consequences of “suspending” different evaluations, for example to restore consistency.

See here for more theoretical background on consistency and the logic of confidence.

The available plots depend on whether they concern:

A single event

The entered evaluations imply, for each confidence level, a probability range concerning the event of interest, held to that confidence level. The probability ranges at the selected confidence ranges are plotted graphically.

If the evaluations imply no probability judgement for the event at a particular confidence level, then the whole 0%-to-100% range is plotted (as in the highest confidence level in the figure here). Note that the probability ranges tend to get smaller as the confidence level considered falls. This reflects the fact that, if you have high confidence in a probability judgement, you will adhere to it even when considering lower confidence levels (as discussed here and here).

For the numerical confidence language, the confidence levels are grouped into bins, with a plot available for each bin.

A variable

The entered evaluations imply, for each confidence level, bounds on the probability of various ranges of the variable. Graphical representation is provided for each selected confidence level, with a choice among the following formats:

  • Upper & lower CDFs: plots the upper and lower bounds on the cumulative probabilities, calculated directly from the evaluations entered, across the range of the variable. (These are the p-boxes derived from the evaluations entered, with no assumption on the form of the distributions.)
  • Parametric CDFs: plots representative parametric CDFs (cumulative distribution functions) respecting the bounds plotted in Upper / lower CDF. You may choose among several parametric forms (note that some may not be suitable for a given variable range and confidence report).
  • Both CDFs: plots the upper & lower CDFs and the parametric CDFs, from the selected family.
  • Parametric PDFs: plots the PDFs (corresponding to the parametric CDFs above), from the selected family.

If the evaluations imply no restriction on a particular cumulative probability at a particular confidence level, then the whole 0%-to-100% range is plotted. Of course, the interaction of different probability judgements concerning different ranges of the variable may have consequences for the admissible probability interval concerning a given range, and this is plotted in the graphs. Note that the upper and lower bounds get narrower as the confidence level considered falls, because, as noted, low confidence levels inherit probability judgements from higher ones (see here and here).

For the numerical confidence language, the confidence levels are grouped into bins, with plots accessible for each bin.

In the case of overlapping graphs for different confidence levels, the colour of the highest confidence level is usually adopted (as in the figure above).