Category Archives: Forecasting

“They get it wrong 90% of the time”

I was on a long train journey a few days ago, and ended up in conversation with the person next to me. When I explained what I’m doing (a PhD project looking to improve sub-seasonal forecasting), I was greeted with the all-too-familiar response of “oh, that’s good because they get it wrong 90% of the time”. Doing my utmost to supress just how much that sentence annoyed me with its factual inaccuracy, I responded with a comment about how there is still so much to learn about the atmosphere-ocean system, and ‘things can only get better’ (perhaps that song by D:Ream should be the soundtrack to NWP – or is it still too tainted by Tony Blair?). I also recently saw a post on Reddit’s ‘showerthoughts’ saying “Forecasting the weather is the only job where you can get it wrong every day and still have a job” – I refrained from responding to that!

Of course, “they” don’t get it wrong 90% of the time (you could easily argue the reverse is true) and a forecaster won’t still be in the job if they’re getting it wrong every day.

In the last few days we have seen a spectacular example of what NWP is capable of with the forecast of Hurricane Florence. We knew days in advance that the storm would reach category 4 status. The track into the Carolinas is now without doubt (though the exact motion when the storm makes landfall remains uncertain). So how is it that a forecast of a huge, turbulent, dynamic vortex can be so incrediby accurate to an extent that even amazes meteorologists – and yet the general public can have such a different opinion about their experiences of weather forecasts? How is it that this perception is so ubiquitous?

I had noticed over the summer that there were noticeable issues with app forecasts – and noticeable failures – but there is often good reason. Showers are difficult to predict to postcode-level accuracy, and, sometimes, “shit happens”. A great example of the latter occurred earlier in 2018 in Reading, in perhaps the only time I’ve really experienced the forecast go completely wrong. The forecast: a dry, cloudy day. However, the inversion mixed out, resulting in clear, sunny skies. That in turn lead to unexpected solar heating and thus unexpected instability, which generated a heavy (but very isolated) shower over Reading (perhaps its localised nature was a response to additional urban heating – yet another complexity!). A wonderful non-linear response – and something which as a meteorologist made sense. However, to a member of the public, it was unexpected rain that may have left them irritatingly soaked – and perhaps fostered a resentment of weather forecasters.

Forecast accuracy has come on in leaps and bounds over the last 30 years, but it seems public’s trust has not increased accordingly. I think part of this is how quickly one adjusts to a ‘new normal’ – I can draw a parallel to Internet connection speeds (remember when 1 Mbps was fast, yet now seems painfully slow?). I think a large part comes down to a lack of understanding as to why a forecast may go wrong. Rather like a medical diagnosis, it may make logical sense as to why a Doctor misdiagnosed, but the patient’s response may be filled with anger and confusion. And part of that comes down to the human body seeming naiively simple (because we all have one!) in the same way forecasting the weather may seem simple (just some fancy graphics and looking at clouds, right?). Both are hugely complex, but its usually only the experts who truly comprehend that.

Thus, it’s my conclusion that the more meteorology we can get out there, the better the public will trust the forecasts we make.

Edit: after writing this post, I received a wonderful tweet which showed that, for some, the incredible accuracy of weather forecasts is understood.
 

Reading Half Cancellation: Do people believe forecasts?

This year’s Reading Half Marathon was cancelled at ~6:40 AM on the day of the race (Sunday March 18) due to substantial snow in Reading and the surrounding area (indeed, across most of England…). The cancellation of the event only ~4 hours before runners would be taking to the course was far from ideal, with many having travelled from afar and stayed the night in Reading. As expected, there was much consternation on Twitter and Facebook, with runners venting their frustration at the last-minute cancellation. As both a meteorologist and a runner, this clash of my two favourite things was a bizarre experience.

The question is – was this last-minute chaos avoidable?

The answer, in my opinion, is a firm YES. And I believe the problem lies with the trusting of forecasts.

On March 13 (5 day lead time!), it became very clear from forecast models that the “Beast from the East” would be returning in time for the race. Cold temperatures were certain, but the extent and intensity of snow was more difficult to predict. I tweeted the Reading Half, enquiring under what circumstances the race would be held off:

I received no reply. As I had seen the Bath Half cancelled with the first “Beast” event earlier in the year, I was very aware that the same fate could befall the Reading race.

By Friday March 16, the Met Office had issued yellow warnings for snow in Reading with a more severe amber warning nearby. The forecasts for Reading (models such as the GFS, ARPEGE, etc.) indicated 3-5 cm of lying snow on Sunday morning. With the model consistency, the severity of the alerts from the Met Office and the huge distances people travel for the Reading Half, I was convinced enough at this point that the race should be cancelled:

Still, the race organisers made no mention of snow and just kept mentioning cold weather. They were determined to ‘plough on’ (pun intended). The announcements were mainly that they were “gritting the route” – which is all fair enough, but hardly going to work very well with significant snowfall, and not going to help those travelling from afar.

Come Saturday, the Met Office forecasts had become worse for Reading, with heavy snow indicated all night. Forecast models still suggested 3-5 cm lying snow on Sunday morning across the south. Runners began to feel less at ease with the idea of it going ahead, and those travelling far wanted clarity.

By Saturday evening, with heavy snow already falling, the organisers cancelled the kids’ fun run, and left Half runners with the statement that a “final confirmation could not yet be made”. More bizarrely, they stated that “if conditions deteriorate further” then they would reconsider.

Prof. Mat Owens from the Reading Met department was just as confused as me, and expressed his feelings toward the race in a great tweet:

 

The forecast was for exactly that overnight deterioration to happen, and it was still the same message that had been said for days.

metoffice

Met Office forecast from 6 PM on Saturday indicating high likelihood of HEAVY SNOW.

I went to bed knowing the Half would be cancelled, yet having to “pretend” as though it were on…just in case it was…somehow. I woke up at 6:15 AM to the expected 3-6 cm of snow, and had to wait until 6:40 AM to find out it was indeed cancelled.

Now, before criticising too much, I want to say I love the Reading Half. I’ve run it twice, and both races being up there with the best experiences in my life. I’m aware it’s a tremendously big and important race.

However, why did it take until the snow was there, lying on the ground, covering the start line, the route and the finish line, with the M4 and M5 paralysed, for the organisers to believe the forecast? The situation that unfolded on Sunday morning was exactly as predicted. There must have been a moment on Sunday where the organisers said, “Oh…the forecasts were correct.”

The fact is, we can now forecast so accurately that a 1 day forecast is pretty much a certainty, unless it’s for a shower in a given location (which we might never be able to accurately predict). The reason we have made such advancements in forecasting power is precisely to avoid situations like this. It’s one of the greatest achievements of the human race. It’s also perhaps the most under-recognised.

What INFURIATES me, is that the announcement on the day claimed that conditions had deteriorated “more than forecasted”. That is, when it comes down to it, a lie. Either that, or whoever was providing the forecasts was not giving the best available forecast. They claimed to have been in contact with the Met Office…well, the Met Office forecasts I was looking at had expected exactly what happened.

 

Capture

The official race cancellation announcement stating that conditions “deteriorated more than forecasted overnight”.

In my opinion, the announcement was to try and make forecasters a scapegoat for poor event management, with too much concern for a false-alarm scenario: “what if we cancel it and then it doesn’t snow?”. I was lucky to have not been so badly affected by this as I live only 2-3 miles from the start line – for those who travelled, I feel deep sympathy.

I sincerely hope Reading Half have learnt their meteorological lesson. And I hope people begin to understand that, whilst of course sometimes the forecast goes wrong, we can now trust it in times like these – and that’s thanks to the pioneering work of many meteorologists, data assimilation scientists and computer programmers over the last few decades.

It really is quite amazing.

But if anyone knows what reasons they had to fully believe the race could safely go ahead until 6:40 AM on Sunday, then please let me know – I’m all ears.

Apply More Meteorology in Apps?

This summer (between the third and fourth years of my MMet degree), I’m working as a gardener in my hometown of Harrogate, North Yorkshire. Being outside and directly affected by the weather means it’s an obvious topic of conversation with customers, other tradesmen and my colleagues. I’m aware that British people always talk about the weather, but when you’re outside cutting a hedge in the rain, the topic is even more apt! As a meteorologist, this is interesting as I get to see the perception the general public have of current weather conditions and weather forecasts.

This blog post was inspired by a couple of quotes from people I met last week on the job:

“My weather app said it wasn’t going to rain today! And it said it was going to rain all day yesterday and it didn’t…I’ve just about given up. It’s obviously not a good app!”  [said by a nearby builder who had halted work during a morning downpour.]

“I don’t know if we’ll see thunder today, they keep pushing the storm back on my app…it was saying 5 PM now it’s 7 PM…I don’t think they know.” [said by a passer-by as the sky darkened.]

These quotes interested me because they showed the current relationship people have with weather forecast apps. Modern NWP allows apps to display highly localised forecasts with hourly forecast weather. This gives the end-user an appearance of forecast confidence, and the public seem to desperately crave a deterministic forecast. Hourly forecasts have much more chance of being perceived as “wrong” than a three hour forecast of dominant weather (in my opinion), since they suggest more accuracy and the public respond as such (rain at 10 AM that was forecast for 9 AM, for example, is wrong, but if 9-12 AM was forecast as rain, then it would be correct). But, crucially, a forecast has much more to it than what apps display.

The lack of this information in certain apps seems, from my experience, to suggest we can’t forecast the weather accurately, which means the public distrust meteorologists more.

A few of my colleagues are users of the UK Met Office app, which presents forecasts alongside a precipitation probability percentage. This is very well received and well applied. An example quote from a couple of weeks ago:

“The Met are saying rain till noon but it’s only 60% chance after 9 AM so we might be okay.”

Putting forecast uncertainty to the public has always been a contentious issue, but it seems (at least to some people) that it can be useful. If it didn’t rain after 9 AM, the Met Office’s forecast would not have been perceived as completely “wrong”, whereas without the probability, it might have been. The other useful aspect to the app is the provision of forecast radar maps – allowing users to see whether rain is widespread, or in the form of showers, and to make a judgement on the likelihood of encountering the apparent rain which is displayed in a mere icon on the forecast.

The second quote reminded me of the seriously difficult issue of forecasting convective showers, especially of the thundery kind, and then the even more difficult issue of conveying this to the public. Yes, it’s true, “they” didn’t know exactly when there was going to be thunder – in the end, there wasn’t any in Harrogate – but there was nearby to the east. Mere app icons can’t display this information. I would argue that the simple phrase, “Chance of thunderstorms” across several hours was better for the end-user than just trying to give a deterministic time-slot for a storm, as forecast updates and the shifting times added to the forecast confusion that the lady I spoke to was experiencing.

The idea of this post wasn’t just to sell the Met Office app, but it’s heading in the right direction with the probabilities, the easily-accessible video forecasts and the forecast maps. The public need more information, because when we hide this from them, then the average person isn’t going to realise what really goes on behind their forecast, and they will just criticise it out of ignorance.

Meteorologists think we can forecast the weather incredibly well and find it an amazing human achievement. Some of the public might agree, but plenty wouldn’t. This is no surprise given meteorologists understand what’s going on much more, but the more information we can give, the less likely it is people perceive the forecast as plain wrong. Uncertainty is a key aspect of meteorology and a key aspect of science – this needs to be in the forecast.

A Very British Obsession

Yesterday’s edition of The Times had an opinion piece by Matt Ridley (p.23) entitled ‘Trust experts on anything but the future’, apparently inspired by Michael Gove’s recent comments in the build-up to the EU referendum about the British public having “had enough of experts”. Naturally such an article also got rather bogged down in criticising long-range weather forecasting.

Now, I don’t appreciate the way the article was written with regards to this, and I did let Mr Ridley know of this in a rather protracted Twitter discussion (in hindsight, an unsavoury encounter whilst cycling earlier in the day had made me rather irritable).

I don’t think comments such as “[it] covers all possibilities, like an astrologer” in reference to a roughly equally-weighted long range Met Office probability forecast show a true understanding of its usefulness (and probably suggest that the British public aren’t ready for probabilities and some consider it a ‘cop-out’ of giving a deterministic forecast). The article also lists every major Met Office long-range failure, and then goes on to say “a blindfolded person throwing darts at a chart would have done better”. Hardly.

I see his point – that long term future predictions are so difficult to make accurately, that non-experts could be just as likely to get it correct. I always remember that I expected the winter of 2009-10 to be very cold, based on only a few patterns I’d noticed, and I was correct, whilst the Met Office’s prediction of a warm winter is famously wrong (and indeed, mentioned in the article).

It is true that the Met Office have had seasonal forecast failures. It was bold of them to issue the forecasts to the general public, because back in 2007 and even now, seasonal forecasting (especially in the UK) is inherently very difficult. The backlash they suffered after the 2009 summer (the forecast ‘odds on for BBQ summer’ ended with the wettest July on record) goes on. And articles like this one are probably fuelled by that. It’s not the only time I’ve seen a list of all the UKMO’s failures in an article, either.

However, I feel that an appreciation of the sheer nature of atmospheric chaos should come first before launching attacks on the Met Office (some of which, in the past, have claimed they are waste of money). Ridley notes this late on in his article, but says “in the cliché, a butterfly’s wing flap can lead to a hurricane” – the issue being that whilst the phrasing is cliché, it is just about true, especially for Britain, where ensemble plumes do often diverge markedly after 5-7 days, and in certain circumstances always will because the atmosphere has varying states of predictability.

My point through all this is the British public, famed for being ‘weather-obsessed’, actually have a surprisingly poor understanding of weather and climate (it being, quite literally, over their heads). Instead, the public obsession (and consequently, the media obsession) is proving the forecaster wrong – even when they themselves are wrong. “Are you going to go and work for the Met Office and then they’ll get it right?” is something I’ve been asked countless times. And yet, the Met Office do get it right, day after day. Incredibly so.

The UKV, for example, is oustanding, when compared with the older models that were used over a decade ago. Ridley’s article didn’t praise the Met Office at all, but I did learn his praise for the short term forecasts was cut for size (predictably). There is an innate desire amongst Brits to prove the Met Office wrong – hence the rise of the Daily Express headline-generators James Madden and Piers Corbyn. No matter how accurate the Met Office forecasts become, their Facebook posts are still covered in comments from people criticising them for being useless.

I feel people expect perfect weather forecasts. They expect us to be able to perfectly forecast the behaviour of a chaotic system weeks in advance. In my experience the layperson has little appreciation of (amongst other things) the sheer number of calculations, sources of error, and just quite the level of atmospheric understanding it takes to produce NWP models.

It’s an astonishing reflection of the development of computing technology over the last 20 years that we can forecast as well as we can nowadays – just compare NHC hurricane track cone sizes today compared with 10 years ago – and yet people continue to neglect it. If one day we made a perfect forecast, would the response be “wow, that’s an incredible human achievement” or “well, about time! I’ve been using seaweed all my life and it’s never let me down!”. I fear the latter. Perhaps it has something to do with an older generation. Perhaps it’s British negativity. Perhaps it’s poor scientific understanding by the public. Perhaps it’s about rebelling against what we’re told. Whatever it is, I don’t like it.

Now, let’s all sip our coffees in quiet appreciation as we watch the 12Z GFS roll out.