How Did The Models Perform?

[Rob K]

Which day was the "epic" ECM easterly run from last week?

[DTHFCJ]

Which day was the "epic" ECM easterly run from last week?

Pretty sure it was 5 december..........(wednesday)

Edited December 11, 2012 by BALE1

[Cloud 10]

A breakdown of the cold and frosty spell across much of the UK today with

some heavy rain and strong winds for many parts.

Today's chart.

The ecm,gem,gfs,jma and ukmo 12z runs from the 8th of December.

ecm.. gem..

gfs.. jma..

ukmo..

The GEM seems closest to the actual chart on this occasion followed by (shock horror!) the ukmo

and the ecm.The gfs and jma share the wooden spoon this time.

[Cloud 10]

Christmas day 2012 was cool and unsettled with some notably heavy showers,especially

near the South coast where thunder was reported.

Today's chart..

The ecm,gem,gfs,jma and ukmo 12z runs from 6 days ago.

ecm.. gem..

gfs.. jma..

ukmo..

The gfs and the jma were the closest this time with both modelling the troughing over the UK

very well.

[Zenarcher]

Been meaning to get round to doing this for the 29th of December the low caused the models to struggle greatly,

29th actual chart,

GFS

UKMO

NOGAPS

JMA

GEM

BOM

ECM

Close one between the ECM and UKMO and I remember the UKMO did perform the best while watching all this unfold. But overall the ECM wins this one.

Also I done a weather model experiment in 2012 and just got round to posting it in my blog http://forum.netweat...t-2012-results/

Edited January 4, 2013 by weathermaster

[Cloud 10]

Much colder weather across the UK this weekend,and with an ongoing Sudden Stratospheric Warming playing havoc with the model output,so lets see how they did for today.

Today's chart.

The ecm,gem,gfs,jma and ukmo 12z from 6 days ago.

ecm.. gem..

gfs.. jma..

ukmo..

An excellent effort by the gfs which pretty much got it spot on,followed by the ecm,jma and ukmo.

The gem takes the wooden spoon this time.

[bluearmy]

Much colder weather across the UK this weekend,and with an ongoing Sudden Stratospheric Warming playing havoc with the model output,so lets see how they did for today.

An excellent effort by the gfs which pretty much got it spot on,followed by the ecm,jma and ukmo.

The gem takes the wooden spoon this time.

i wonder if comparing the 00z would have yielded the same result ?

[Rob K]

Yes I am surprised GFS performed best - must be an artifact of the time chosen. As I recall, GFS was totally out on its own with not sending the weekend shortwave SE over the UK, and instead wanted to send it NE to Svalbard. It was consistent for several runs in a row before caving in.

[Cloud 10]

i wonder if comparing the 00z would have yielded the same result ?

The 00z gfs certainly didn't verify as well as the 12z...

gfs actual.. 00z predicted..

....mind you,the 06z and 18z had a good stab at it...

06z.. 18z..

Edited January 15, 2013 by Cloud 10

[Cloud 10]

Quite a day of weather across parts of the UK today with some Western areas

in particular having some disruptive snowfall.

Today's chart.

The ecm,gem,gfs,jma and ukmo 12z predictions from the 12th of January.

ecm.. gem..

gfs.. jma..

ukmo..

The GEM was the clear winner on this occasion,the ecm,jma and gfs all had to much energy

riding over the top of the cold block of high pressure,the ukmo had the high way to strong

and to far west.

[Cloud 10]

Model check for today as the cold spell comes to an end,although it did go out in style with

a widespread frontal snow event.

Today's chart.

The ecm,gem,gfs,jma and ukmo from January 20th.

ecm.. gem..

gfs.. jma..

ukmo..

All the models did well this time,the ecm and ukmo seem the closest.

[Zenarcher]

A very low deep pressure system was in the Atlantic during Saturday so lets see how the models handled this,

Actual chart from Saturday

GFS - Handled this very well by picking it up 6 days before, it did at first change it's path but consistently showed it well for the whole week without any slip ups.

What it said on Monday afternoon

ECM - It did wobble a bit on Sunday and Monday morning but by Monday afternoon on it's 12z run it had caught on to the idea and stuck with it as well so a little late compared to the GFS.

UKMO - Picked it up on Sunday morning that's 6 days off pretty amazing and on Monday kept with it as well.

But on Tuesday both the 00z and 12z runs had a wobble and made a few mistakes by pushing the low too far East by Wednesday morning it had corrected itself and went with a push more to the West.

GEM - Caught onto it by Monday afternoon.

Although during Tuesday and Wednesday had the low at times too far East but corrected itself on Thursday.

NOGAPS - Did a good job picking up on it by Monday afternoon.

Tuesday it made a few big mistakes making this low actually hit the UK but on Wednesday backed off the idea and pushed it back into the Atlantic.

BOM ACCESS - It did get the idea of the low but was clueless on where to place it and kept showing it hit the UK until Thursday when it placed it in the Atlantic.

CMA - Since it's new to us no one really knows how good it actually is well I will say I am not impressed by it at all, didn't show the low in the Atlantic until Thursday morning before then it was sending it all over the place.

Overall - I'm giving the GFS the win for this because of one reason it picked up on it and stuck with it the whole time no slip ups nothing next UKMO and ECM on the same scale well done to UKMO for picking up on it 12 hours before the GFS but the UKMO did have a few bad outputs and made mistakes at just 72 hours. ECM about 24 hours slower to pick it up but once it did it stuck to it. GEM comes next by doing okay then next NOGAPS. Lastly BOM and CMA on the same level here at the bottom.

Edited January 27, 2013 by weathermaster

[Cloud 10]

Well,today is "slider low" day,so how did the models get to grips with the pattern?

Today's chart.

The ecm,gem,gfs,jma and ukmo from the 4th of February.

ecm.. gem..

gfs.. jma..

ukmo..

The ecm takes the gold medal yet again followed by the ukmo.The jma was pretty good

but it had the pattern a touch to far east.

The gfs underestimated the blocking to the North-East,fancy that!

[Cloud 10]

A cold continental airmass starting to become established across the UK today,so lets

see what the models were predicting 6 days ago.

Going to include the NAVGEM from now on which is the replacement for the NOGAPS.

Today's chart.

The ecm,gem,gfs,jma,navgem and ukmo 12z run's from 6 days ago.

ecm.. gem..

gfs.. jma..

navgem.. ukmo..

A good effort from all the models with the GFS being the closest to the actual chart.

Good start for the NAVGEM as well.

[Cloud 10]

Well,the weather was once again in the headlines today as an extremely potent easterly blast

enveloped the country with upper air as low as -14 in some spots.

Truly exceptional for March.

Today's charts.

The ecm,gem,gfs,jma,navgem,nogaps,and ukmo from 6 days ago.

ecm.. gem..

gfs.. jma..

navgem.. nogaps..

ukmo..

The ECM and JMA seemed to do best this time.

I included the NOGAPS for sentimental reasons this time as this model is to be discontinued on the 13th of March,and fully replaced by the NAVGEM.

As it happens the NOGAPS did pretty well on this occasion,and certainly much better than

the NAVGEM.

As for the wooden spoon,the UKMO is the easy winner this time with a complete howler.

[Cloud 10]

An extremely negative AO (Arctic Oscillation) is driving the weather patterns across the

Northern hemisphere at the moment,with extensive high pressure in the Arctic

region displacing very cold air over Northern Europe and Russia.

The general pattern was well predicted by the ECM,UKMO and GFS.

today's chart..

ECM,GFS and UKMO runs from 6 days ago.

ecm.. gfs..

ukmo..

[Zenarcher]

Lets see how the models handled the recent storm in October,

 

21st October

 

GFS showed a 950mb low tracking over the UK

 

 

Both CMA and BOM showed a low further South,

 

 

 

None of the other models at this point picked it up.

 

22nd October

 

The models started doing silly things with the low most of them were still not keen on it but the GFS did still show it but a lot weaker and further South this time,

 

 

ECM sent it down more South missing the UK

 

 

But the UKMO showed it could still be really bad and still had the chance to hit the UK

 

 

23rd October

 

More models seemed to start showing a low but still none of them had a clue what to do with it.

 

The GFS seemed similar to the day before

 

 

ECM still sent it South although overall it had moved the low up North slightly more than the day before

 

 

But it was the UKMO and BOM charts that still showed stormy conditions keeping the interest and uncertain outcomes going. They showed a strong low over the UK.

 

 

 

24th October

 

The storm will arrive in 96 hours time and most models show it now and are starting to agree on the track but the CMA and NAVGEM seem uninterested in it and show nothing.

 

The ECM showed the worse chart for winds as a strong tight low passed by inland areas

 

 

Most agreed on the path but showed something less strong and the ECM only had support from BOM which still showed a tight low.

 

GFS stuck by what it had said for the past few days

 

 

25th October

 

Now just 72 hours away the models continued to disagree on things the confusion continues.

 

The GFS, ECM, JMA and NAVGEM showed a low crossing over central England like this

 

 

But the UKMO, GEM and BOM for the same time had the low over and done with by the afternoon for that day

 

 

The CMA was on its own with a strange theory that there would be no low at all.

 

26th October

 

48 hours away the storm has gained so much attention now and the models for the first time agreed on its track and timing something the UKMO and a few other models showed the day before

 

 

27th October

 

24 hours away and the models start to downgrade the low and make it arrive even earlier

 

 

28th October

 

This was the final outcome its strange to think something like that can cause the models to really struggle but with the jet stream in a tricky position it was really difficult for the models to get its track and deepness correct.

 

Edited November 9, 2013 by weathermaster

[garbagebags]

Whats the craziest model run anyone has ever seen. As in, compared to the previous run, completely different. Have they ever been broken with crazy readings?

[Zenarcher]

Whats the craziest model run anyone has ever seen. As in, compared to the previous run, completely different. Have they ever been broken with crazy readings?

 

December 11th 2011 for me I wrote it in my blog at the time,

 

For a few days every weather model had been showing a severe storm hitting mainly Ireland, England and Wales. It caused a lot of concern for the Met Office and the members on Net Weather. On the 11th of December the ECM done its evening run which was watched by 100's it made the storm nearly vanish, it caused a big surprised many thought it was a fluke, later the other models agreed with the ECM, I believe it's one of the ECM's finest moments.

 

Basically at the time all models showed a very deep low about 945mb hitting the UK giving average wind speeds over 60mph to parts of Ireland and most of England and Wales. To see some models show this just 96 hours away usually a time where they have pinned the track down then to change to nothing at all within the next 24 hours was amazing and shows anything can happen.

[garbagebags]

December 11th 2011 for me I wrote it in my blog at the time,

 

For a few days every weather model had been showing a severe storm hitting mainly Ireland, England and Wales. It caused a lot of concern for the Met Office and the members on Net Weather. On the 11th of December the ECM done its evening run which was watched by 100's it made the storm nearly vanish, it caused a big surprised many thought it was a fluke, later the other models agreed with the ECM, I believe it's one of the ECM's finest moments.

 

Basically at the time all models showed a very deep low about 945mb hitting the UK giving average wind speeds over 60mph to parts of Ireland and most of England and Wales. To see some models show this just 96 hours away usually a time where they have pinned the track down then to change to nothing at all within the next 24 hours was amazing and shows anything can happen.

 

Not to mention "The beast from the East" last December... On of the GFS's darkest moments. Thats was 72hrs out. Then nothing...

[Cloud 10]

Especially for GFS fans,a report on how it performed during 2013.

 

http://www.emc.ncep.noaa.gov/GFS/docs/GFS.performance.review.2013.pdf

 

 

Just on the off-chance you can't be bothered to go through the whole pdf :lol:  the main summary is below.

 

• There was no GFS upgrades in 2013. Instead, the system was moved from CCS 

to WCOSS suptercomputers. 

 

• In 2013, GFS continues to show forecast improvement of 500-hPa height AC. 

 

• GFS remains trailing behind ECMWF by ~0.3 days in the NH and by 0.7 days in 

the SH for useful forecast days (AC>=0.6). 

 

• Among the GFS daily four cycles, the 00Z cycle has the best forecast skill. It is 

not clear why the four cycles differ from each other. The difference cannot be 

solely explained by different observation data counts. 

 

• In the past ten years, GFS hurricane track and intensity forecast had been 

greatly improved in both the Atlantic and Pacific basins. However, in 2013 GFS 

track forecasts were slightly degraded in both basins. 

 

 

[knocker]

NCEP GFS 500 hPa NH {20 deg- 80 deg N] anomaly correlation :|: 120- hour Forecasts

 

Last 365 days 0.875

 

NCEP ECMWF 500 hPa NH {20 deg- 80 deg N] anomaly correlation :|: 120- hour Forecasts

 

Last 365 days 0.903

Edited April 27, 2014 by knocker

[Zenarcher]

Just posted the weather experiment of 2014 into my blog I will copy it over here for you to view https://forum.netweather.tv/blog/227/entry-4866-weather-models-experiment-2014-results/

 

I've done a yearly weather model experiment to find the best performing one.

The previous winners were,

2011 ECM
2012 UKMO
2013 GEM

 

This time the experiment took part during December 2014. We will be testing the nine weather models listed below,

European Centre for Medium Range Weather Forecasts = ECM
Japan Meteorological Agency = JMA
Met Office = UKMO
Canadian Weather Service = GEM
Global Forecast System = GFS
Global Forecast System Parallel = GFS(P)
Navy Global Environmental Model = NAVGEM
Global weather forecast model = GME
China Meteorological Administration = CMA

 

So how does it work? To make it fair all of the models are tested by using their 12z runs from the same day. All they have to do is forecast from 24 to 144 hours for a random location I've chosen the more accurate they are the more points are awarded and bonus points are awarded for how far they forecast out. For example the points awarded for getting something correct at 144 hours is much higher than at 24 hours.

 

24 Hours

 

As you would expect its pretty close between them with the ECM and JMA doing the best here.

 

 

48 Hours

 

The ECM and both the new and old GFS models all performed well here with the UKMO and NAVGEM not too far behind them.

 

This chart below shows the total scores so far and who has performed the best upto the current time scale. The ECM currently leads with both the GFS models behind.

 

 

72 Hours

 

The JMA performed the best here while the ECM and UKMO are not too far behind.

 

The latest total scores show the ECM still leading with the JMA and UKMO just behind. The GFS models are still close and last years winner the GEM seems to be performing below expectations.

 

96 Hours

Surprisingly its the old GFS model that tops the scores here and is comfortably ahead of second place. The CMA done a good job as well beating the UKMO and last years winner.

 

The latest total scores show the ECM still leading but the old GFS model has caught up with it and is only a few points behind.

 

 

120 Hours

 

The new GFS model done a great job here and performed much better than the older version. The ECM and NAVGEM also performed well at this time range.

 

The top scores up to this point show the ECM still leading with the JMA and the new GFS model behind it. Last years winner GEM is going to have to perform greatly at 144 hours if it wants the top spot again.

 

144 Hours

 

Once again the new GFS model performs much better here than not only the older version of the GFS model but than the rest as well it's so far ahead in points here. The CMA also done a good job while the ECM performed slightly poorer than expected.

 

The final results are shown below,

 

 

So it's the brand new GFS model that's the winner and it managed to beat the ECM at the last part of the experiment however the ECM still done a great job and was for most of the experiment the best performing model. The old GFS finished in third nearly 100 point behind the new version so this may be evidence than the new GFS model is indeed better. Next the JMA and UKMO finish still near the top and are close in points. NAVGEM finishes in the middle spot while last years winner GEM done terrible this time. CMA and GME finish in the final two spots at the bottom. 

Edited January 17, 2015 by weathermaster

[Dr. Astro]

Thats brilliant Weather Master.

 

Thumbs up.

[Recretos]

First I have to say good work to Weather Master. That was a very fun post.

 

But it is true that model verifications are made on a much larger scale, and not point wise, because that makes no real sense whatsoever. But I am sure you know that.