ECMWF, GFS, NMM And UKMO: How Have The Models Been Performing?

[summer blizzard]

how about overall marks out of 5 for the major synoptic pattern. Is the trough in the right place and depth, is the nearest upper ridge likewise?

Then 5 for the bit zoomed in over the UK area, direction of the isobars over the UK, their value and how close they are, what 500mb temperatures are predicted. That should give as good a guide to trying to use the model for predicting what the surface weather for the UK might be 10 days ahead?

For instance in the latest charts you show GEM has isobars from a diffferent direction to the other two as one difference.

Great idea, we will use this method then.

[J10]

how about overall marks out of 5 for the major synoptic pattern. Is the trough in the right place and depth, is the nearest upper ridge likewise?

Then 5 for the bit zoomed in over the UK area, direction of the isobars over the UK, their value and how close they are, what 500mb temperatures are predicted. That should give as good a guide to trying to use the model for predicting what the surface weather for the UK might be 10 days ahead?

For instance in the latest charts you show GEM has isobars from a diffferent direction to the other two as one difference.

I like that idea, as how much the significance to give to the UK weather was always an issue of mine.

[johnholmes]

Great idea, we will use this method then.

I look forward to seeing how this pans out through the rest of autumn and into winter SB.

If you want a hand if you are away or not able to do it pm me and if I'm around I'll step in.

[summer blizzard]

Next set of charts as well, pretty different solautions so there should be an easy winner/loser.

Just being UK centric here, but GFS looks like it may be most on the ball. Will have to wait for the 4th anyway.

[Isolated Frost]

it looks like ecm 1st, gfs 2nd and gem 3rd imo?

[johnholmes]

it looks like ecm 1st, gfs 2nd and gem 3rd imo?

2 days to go before a decision can be made

[J10]

To me the GEM looks very similar to what happened at the end of last week.

[summer blizzard]

it looks like ecm 1st, gfs 2nd and gem 3rd imo?

2 days to go before a decision can be made

To me the GEM looks very similar to what happened at the end of last week.

Will have to wait for the 4th (will score the two previous charts under the new system as well) but model ouptuts have changed in recent days likely due to the rapid intensification of Hurricane Ophelia which is entering the Jet Stream (it actually degenerated a few days ago but is now the strongest of the year). Certainly they all got it wrong again, but GEM most wrong.

[summer blizzard]

I have to go out but will mull over the result and post tonight.

[BlueHedgehog074]

Noticed the GFS seems to be quite consistant with the current West/south-Westerly wind breaking down to a North-Westerly wind as the high-pressure from the South-West seems to force the Low Pressure system(s) to retreat North-Eastwards around the weekend. I think it was early yestersday (or the day before), the GFS was still showing pretty much the same scenario, but with the winds having slightly more of a direct Northerly element around Friday and the early weekend. Other than that, it does seem to be rather confident with the outlook within the reliable timeframe.

I admit I haven't had a proper look at the ECMWF, except today, and seems to be in good agreement with the GFS about the weather pattern for the next few days, although at 120 hours out...

..ECMWF has the strong area of high-pressure positioned to the South-West while the GFS models this strong area of high pressue directly to the South - some slight disagreement going on here. Neverthless, the outlook for both models still remains pretty similar flicking through the pressue charts. It's really when we get to around 216 hours, things start looking different...

...but with this entering more of the Fantasy Island Zone, I suppose it is to be expected. It's true sometimes that FI can be closer, especially if the models have a hard time of trying to pick out short-waves and that, and, in turn, have an effect on the outcomes the models show.

In the 216 hours example, ECMWF seems keen for high-pressure to make an influence for the whole of the UK, while the GFS has Low-Pressure to the North, but I would imagine the South would stay quite dry with the pressure looking high their.

Although this is based on one run, and 2 models, I feel they are doing quite well, considering with the lack of disagreement between them (unless it's at 216 hours and beyond).

Edit: Be interesting to see how the results pan out.

Edited October 4, 2011 by Rainbow Snow

[johnholmes]

I've put the actual-well its the 18z so near enough and the 3 charts with comments below them. I have honestly no idea which is chart 1, 2 or 3 so anyone can make their own mind up which seems nearest and then say which model it is.

None are totally correct that is for sure. Depends on what anyone is looking for I suppose.

actual v T=240 charts.pdf

[summer blizzard]

They were indeed all very wrong.

Being UK centric we get...

Chart 1 (GFS)) - 1/5 (right ish pressure but wrong orientation, wrong wind direction and wrong idea)

Chart 2 (ECWMF)) - 2/5 (wrong wind direction but right idea about high presssure being near, wrong pressure)

Chart 3 (GEM)) - 3/5 (right idea about high pressure being near, just too far north east though right wind direction, wrong pressure)

Broadening the scope we get...

Chart 1 - 2/5 - Got the Azores Low but in wrong place and as a result had the Azores High way out west, also had the Jet Stream too strong exiting the USA

Chart 2 - 4/5 - Got the Azores Low a little west but in a good position and the Azores High just too far north west although did have the Jet Stream too strong exiting the USA

Chart 3 - 3/5 - Did not get the Azores Low but did get high pressure near the UK and had the Jet Stream weak exiting the USA

Overall then we get....

GFS - 3/10

ECWMF - 6/10

GEM - 6/10

Considering 3 different solutions i am suprised that none picked up on the actual.

[wimblettben]

When I wasn't a member to this forum I can remember times earlier in the year such as the Summer when I was just having quick looks to the models on a day to day basis and I can remember them chopping and changing very frequently, almost daily.

So I would say as a whole that all of the models didn't do particullarly well in the Summer but have seemed to have improved this Autumn in terms of how accurate they are.

[summer blizzard]

Next set of charts, broadly similar but with subtle differences.

[Radiating Dendrite]

When I wasn't a member to this forum I can remember times earlier in the year such as the Summer when I was just having quick looks to the models on a day to day basis and I can remember them chopping and changing very frequently, almost daily.

So I would say as a whole that all of the models didn't do particullarly well in the Summer but have seemed to have improved this Autumn in terms of how accurate they are.

The thing to look for is trends when it comes to the models.

The amount of times that they chuck something in at +288, for it to disappear and then reappear in the reliable is surprising!

Specifics - stay within +48, themes - +144, trends - beyond +144 as far as I am concerned.

[summer blizzard]

Next set of charts, broadly similar but with subtle differences.

We will of course have to wait, but at this stage it looks as though ECWMF has got a big F.

[J10]

http://hw.nwstatic.c.../00/h500slp.png

[summer blizzard]

Being UK centric the order was GFS followed by ECWMF and GEM who were equally bad for different reasons.

Looking at the broader pattern and i would again have to say that the GFS was best due to the Azores High positioning and heights over Greenland with both GEM and ECWMF being bad for different reasons.

Score - UK

GFS: 8/10 (right pressure and only model to show a cyclonic northerly element)

GEM: 6/10 (right pressure but southerly element to flow)

ECWMF: 4/10 (wrong pressure, wrong flow)

Scores - Wider

GFS: 8/10 (Greenland High, Scandi Low, Azores High in right place)

GEM: 5/10 (Azores High much further north west but a good Scandi Low)

ECWMF: 6/10 (Rightish over Scandi, right over Azores)

Overall...

GFS: 8/10

GEM: 5.5/10

ECWMF: 5/10

Running Score after round 2...

GEM: 5.75/10

GFS: 5.5/10

ECWMF: 5.5/10

[BlueHedgehog074]

Must admit, the GFS seems to have done fairly well with the prediction for 19th October - Low pressure positioning to the North of the U.K not that far off too. I agree with that 8/10 rating.

[BlueHedgehog074]

Next set of charts for the 16th November. Wonder who will win out of the trio?

GEM (from 00Z run)

GFS (from 12Z Run)

ECMWF (from 00Z run)

One thing I have noticed about the performance of the charts recently is the way that models such as ECMWF have been rather consistant with showing that big blocking to the East - even at 240 hours out. Seeing as blocks like this can be hard to push away, I suppose it makes sense.

[Snowmad79]

Im backing the GEM becuase it looks so much like tonights 12z UKMO/ ECM output

[summer blizzard]

Indeed it does although we have seen in previous comparisons that the models can look like falling into line with one solution in the 5-10 day range before backing one of the other solutions closer to the time.

I would love the GEM output for potential inversion reasons.

[BlueHedgehog074]

Almost completely forgot about this thread.

Regarding the performance, I do feel the models are struggling a little with the evolution for the week beyond next week. The GFS, for example consistently seems to churn out blocking solutions with either blocking out to the West, or blocking extending in from the East via a Scandi high. I do feel it hasn't quite got a grip on what it wants to do with it's possible blocking solutions (although to be fair, since most of the blocking it does show stays out in FI, I guess it's understandable why it keeps jumping from one solution to another).

The ECMWF, however, seems generally more sure of blocking coming in from the East, but one problem I have noticed is the Easterly potential with the possible undercutting lows/shortwaves keeps getting pushed back a bit. But, on the whole, I do feel the ECMWF is slightly more sure on how the pattern may develop.

Also, according to the latest verification results: http://www.emc.ncep.noaa.gov/gmb/STATS/html/acz6.html the ECMWF is still the top class performer with its red line broadly staying higher than the other lines from the other models. Plus, the fact the ECM is supposedly better at handling Easterly patterns makes me feel it could just be the model that leads the way with the possible pattern change (if a pattern change does occur).

The only aspect, aside from the fact the Scandinavia high blocking potential keeps getting put back, is that a few members such as John and Glacier Point do feel our blocking is likely to arise from the North-West instead. And should this be the case, it could be a big red cross for the ECMWF - and even for other models such as the GFS if it decided to go with ECM's incorrect solution. (As it happens the latest GFS 18Z GFS did go for a block to the North-East extending Westwards).

Edited November 17, 2012 by DiagonalRedLine

[Roger J Smith]

I don't know what the stats show, but the recent experience with Sandy tends to confirm this general impression:

The ECM continues to lead the pack and is the most reliable guidance more often than not. The GEM is now marginally better than the GFS which seems to be struggling a bit in the past two years. The UK model is probably ahead of the GEM for European forecasting and slightly behind for North American application. In North America, the NAM is the equivalent of the NMM, when it has the best solution people notice because it is rather unusual to see that.

The ECM did a superb job with Sandy giving us 8 days of reliable warning of what was coming, while the various other models were all over the place and generally too far east with their solutions until much closer to the event, even at three days they were continuing the trend of being too far east, but at times the GEM was almost as good as the ECM. The GFS at one point had Sandy moving out to sea and heading for the Azores. I suppose almost every other named storm this year did that, so ...