Communicating Uncertainty

In the aftermath of the U.S election there’s been a lot written about why the polls were off, and the role of forecasting sites such as 538 and The Upshot. In particular I’ve been struck by the conversation about the problems that these sites, which use models to forecast the likelihood of each candidate winning, have in communicating the uncertainty and probabilities around their forecasts, and similarities with discussions in the climate world.

Fivethirtyeight's final election forecast

Now focussing in on 538 (disclaimer, I think these guys are great, and their data-driven reporting is sharp and insightful!), I see a lot of similarities with the problems faced in communicating uncertainty around climate projections, or climate impacts. To be clear, 538 wrote extensively about the assumptions their model was taking, and why uncertainty was much higher this year than previously. They also have a lot of different ways of trying to show the uncertainty in their forecasts. I think the problem, however, is with their default visualisation, which simply shows the changing average probability, without any range (see above). Now if you have a good understanding of probabilities that may be fine, but it’s clear from the ongoing criticism levelled at them, in print and on social media, that most people don’t, and either interpreted that figure as meaning Clinton would win 70% of the vote, or attached far higher certainty to it than a 1-in-3 chance merited. It looks like most people weren’t looking into the explanation, and alternative visualisations, but were taking the headline figure, shorn of context, and interpreting this in whatever way they wanted. The Upshot tried to contextualise their probability by comparing it to the chances of various things happening in an NFL game, which is an interesting approach, but I’d love to see how effective it was in influencing interpretation of the headline figures.

So what’s all this got to do with climate change? One issue that climate scientists have struggled with for years is how to avoid misinterpretation of their data and projections. There are three particular problems here; the first is that regardless of how you display the range of uncertainty, there is a strong tendency to take the average value, and assume this is the important information, rather than what the range is telling you about uncertainty and possible changes. The second is that no matter how well the written text explains the nuance and caveats of a projection, people in a rush will take what they need from the visuals, and ignore everything else. Third, and this is key; scientists assume a far greater ability to read and interpret graphs than actually exists outside of their community. This is fine if that’s your audience, but as soon as your work has the potential to be used in policy circles, or is public-facing, the assumption of a common understanding of what’s being displayed breaks down. This applies to anything from simple line graphs to clever and complex visualizations.

What do we do with all this? Well there’s a lot of very good work out there on how displaying data in different ways changes the messages different groups take away. I found this paper by Joe Daron, and this one by Elizabeth Stephens particularly useful, and there lots of resources on communication and climate change here. We’ve made a lot of progress in appreciating the difficulties in communicating climate change, but I’d suggest a guiding principle when thinking about presenting data should be ‘how could this data be mis-interpreted’!

Oh, and for 538? I think change the default view on the model to some kind of shaded envelope, without a central value, or vary the probabilities people see to reflect the range of possibilities? In any case a lot of testing among different groups to check how the data being displayed is interpreted. To some extent it’s an impossible task given confirmation bias and a natural tendency to filter and adjust information to fit world-views, but it’s also very important!