Meto Data Release

[VillagePlank]

Re: http://www.metoffice.gov.uk/corporate/pressoffice/2009/pr20091205.html

If there's been any good from this CRU-email fiasco, it is that the MetO are now going to publish data and methods in accordance with the scientific method. As far as I can see initially there will be over 1000 station data published, followed, after international agreement, up to 5000. Along with this how this data is conglomerated into a global mean will also be published. Hopefully, once and for all, this will show that the world has been warming and move the debate into a sensible arena based on qualification of warming and quantity of warming for each driver.

It's a first step.

However, some of the methods that the MetO are likely to use are almost certainly going to be tricky, so I thought I'd open this thread so awkward questions related to methods can be answered (by anyone who can), and, hopefully, it's a nice starting point for those wishing to know more about the technical details of how it has all been compiled and produced.

Given a clear mandate then, it should be easy for OT posts to be removed quickly

Edited December 6, 2009 by VillagePlank

[Thundery wintry showers]

Philip Eden mentioned in an article in Weather that New Zealand and the Netherlands have already moved in this direction.

Personally I am very pleased with the move. It will help to make the temperature records more reproducible and I have often thought that the cost to the community of station owners "protecting" these records outweighs the potential financial benefits to the owners of doing so.

[Methuselah]

I agree Ian...Because (IMO) at the end of the day, the 'deniers'' best friend has been the MetO's 'closed-shop' mentality...At least now (once we can all see the data) there'll need be more than just simply 'crying foul' required to back-up their claims...

[AtlanticFlamethrower]

I agree Ian...Because (IMO) at the end of the day, the 'deniers'' best friend has been the MetO's 'closed-shop' mentality...At least now (once we can all see the data) there'll need be more than just simply 'crying foul' required to back-up their claims...

Why do you continue to call them "deniers"? I'm sure the majority of your readers are offended by this term and curious as to why you are so dismissive of their view given the undeniable revelations of "Climategate". Given the sensible reactions to Climategate of people like George Monbiot, I'd suggest your dismissive position is more extreme than many individuals traditionally thought as radical and would encourage you to perhaps accept you might need to accommodate some of your critic's claims.

The data release will be useful. And it might just show where the AGW view has gone awry.

Edited December 6, 2009 by AtlanticFlamethrower

[dvdfjojo]

I must admit that I hate the term 'deniers' too - it's known for right-wingers who deny the holocaust ever existed despite undisputed evidence that it happened.

It's a hate term to describe irrational bigots. I don't know whether global warming is due to man or not and the scaremongering, non-release of data, politicisation makes me sceptical as to whether it is true or not. Until it is proven that increased CO2 is an undisputed driver of warming, I'll keep an open mind thank you very much. The constant belittling of people who doubt just alienates people who are genuinely interested as to whether it is man-made or not.

[Thundery wintry showers]

What we're seeing here is a classic case where "tarring everyone with one brush" can work both ways.

Some people do use the label "denier" to categorise anyone who happens to doubt the correctness of the prevailing view on AGW, a recent example being Gordon Brown. But to say that all those who use the term are categorising everyone in this way, is in itself tarring everyone with one brush.

There are people who deny that any evidence exists for AGW, people who deny that the world has warmed, people who insist that AGW is a scam intended to overthrow free market capitalism. They are the deniers, as distinguished from those who merely doubt the truth of the consensus, who are sceptics.

Actually every respectable scientist should be sceptical (I think a few others have raised this point before) because in climate research as new discoveries are found, newer ones can come up which cast doubt upon them- such is the progress of knowledge in such chaotic systems.

[Methuselah]

I must admit that I hate the term 'deniers' too - it's known for right-wingers who deny the holocaust ever existed despite undisputed evidence that it happened.

It's a hate term to describe irrational bigots. I don't know whether global warming is due to man or not and the scaremongering, non-release of data, politicisation makes me sceptical as to whether it is true or not. Until it is proven that increased CO2 is an undisputed driver of warming, I'll keep an open mind thank you very much. The constant belittling of people who doubt just alienates people who are genuinely interested as to whether it is man-made or not.

That is why I always try to differentiate between genuine sceptics with those who deny that AGW is even possible, let alone real...As TWS points out, all science is sceptical...The conflation of sceptics with deniers has never been my intention: IMO, a clear line should be drawn between the two!

Indeed, I'd rather leave conflation to those that wish to use it: cui bono?

[Solar Cycles]

That is why I always try to differentiate between genuine sceptics with those who deny that AGW is even possible, let alone real...As TWS points out, all science is sceptical...The conflation of sceptics with deniers has never been my intention: IMO, a clear line should be drawn between the two!

Indeed, I'd rather leave conflation to those that wish to uses it: cui bono?

Aye but you class me as a denier, even though I accept Co2 is responsible for some of the warming we endured!

[Methuselah]

Aye but you class me as a denier, even though I accept Co2 is responsible for some of the warming we endured!

Actually SC: no, I don't class you as a denier...

But, we have had some (sometimes alcohol-fuelled? ) debates in which we've both polarized our conceptions of each other's views, have we not??

[Solar Cycles]

Actually SC: no, I don't class you as a denier...

But, we have had some (sometimes alcohol-fuelled? :lol: ) debates in which we've both polarized our conceptions of each other's views, have we not?? :lol:

Aye indeed we have!

[VillagePlank]

I asked the MetO whether they would be releasing all of the data associated with the global weather stations, with particular relation to synoptic stuff like air-pressure. They replied that, no, only the temperature related stuff will be released.

Edited December 7, 2009 by VillagePlank

[Chris Knight]

Perhaps everyone was so fascinated by the happenings in Copenhagen they missed the release of the data here

There is a list of questions and answers on the page, some of which I have copied below:

Questions and answers about the data sets

Please select a question to open or close the answer.

1. Are the data that you are providing the “value-added” or the “underlying” data?

The data that we are providing is the database used to produce the global temperature series. Some of these data are the original underlying observations and some are observations adjusted to account for non climatic influences, for example changes in observations methods.

Back to top

2. What about the underlying data?

Underlying data are held by the national meterological services and other data providers and such data have in many cases been released for research purposes under specific licences that govern their usage and distribution.

Back to top

3. Why is there no comprehensive copy of the underlying data?

The data set of temperatures back to 1850 was largely compiled in the 1980s when it was technically difficult and expensive to keep multiple copies of the database.

Back to top

4. How can you be sure that the global temperature record is accurate?

The methodology is peer reviewed. There are three independent sets of global temperature that all clearly show the rise in global temperatures over the last 150 years. Also we can observe today that other aspects of climate are changing including reductions in Arctic seaice and glacier volume, and changes in phenological records, for example the dates on which leaves, flowers and migratory birds appear.

Back to top

5. Why have you not previously shared the HadCRUT data?

We have always provided the gridded HadCRUT product freely and without restriction for research usage.

Back to top

6. What about the underpinning observations on which the gridbox averages are based ?

The Met Office is not in a position to release that portion of the underpinning land station data for which we have yet to gain permission from the ultimate rights holders.

The data are owned by other countries and institutions and any such release would need to be agreed with these data providers. We are in the process of seeking this agreement from the owners of the underpinning data, so that we will be hopefully in a position to release more of the data in the future. The underpinning ocean data component of HadCRUT is available publicly at icoads.noaa.gov.

Back to top

7. Why are you releasing a subset of the data now?

We can only release data from NMSs when we have permission from them to do so. In the meantime we are releasing data from a network of stations designated by the World Meteorological Organization for climate monitoring together with any additional data for which we have permission to release.

We plan to release as much of the remaining data as possible in stages following issuance of letters to other NMSs requesting permission to publish.

Back to top

8. Why these stations?

The choice of network is designated by the World Meteorological Organization for monitoring global, hemispheric and regional climate and variability.

To compile the list of stations we have released we have taken the WMO list of GCOS Surface Network stations and Baseline Climate Reference Network stations, cross-matched it and released the unambiguous matches.

Back to top

9. Will releasing a subset skew the principal findings?

10. Does this subset constitute a new data set?

This is not a new data set. Data sets are only released when they have gone through the proper process of scientific review.

It is important that due scientific process is followed if we are to have confidence in our findings. If we were proposing this as a new data set then we would have submitted it for peer review and only released it once accepted. The three principal data sets have all undergone this process and therefore retain primacy.

Back to top

11. Why aren’t all the underpinning land station records available for free?

Making observations costs substantial amounts of money and requires a degree of technical expertise and training to meet internationally agreed standards prescribed by the World Meteorological Organization. Furthermore, these data, even at a monthly mean resolution, can have significant economic value to the rights holders. In many parts of the world, including the UK, national meteorological services (NMSs) are expected to act as commercial entities, returning at least to the taxpayer. Removing potential revenue streams could substantially harm many such organisations. We therefore cannot guarantee that all NMSs will permit release of station-level data.

Back to top

12. When will you release more?

As soon as we have all permissions in place we will release the remaining station records — around 5,000 in total — that make up the full land temperature record. We are dependent on international approvals to enable this final step and cannot guarantee that we will get permission from all data owners.

Back to top

13. How have you dealt with the FOI requests regarding releasing the underpinning global temperature data?

We take our responsibilities under the Freedom of Information Act very seriously and have, in all cases, handled and responded to requests in accordance with its obligations under the legislation.

We have been consistent in our responses in stating that the Met Office is not in a position to release the underpinning land station data as we do not have the authority to do so as the data are owned by other countries and any such release would need to be agreed with data providers. We are in the process of seeking this agreement from the owners of the underpinning data, so that we will be hopefully in a position to release the data in the future.

Back to top

14. What have you done to gain permissions?

We have facilitated a letter from the Climatic Research Unit to all rights holders requesting permission to publish the underlying station data. We are monitoring responses and actively pursuing the rights holders for a decision through our international relations team.

Back to top

15. Who is ultimately responsible for the land data record?

The University of East Anglia’s Climatic Research Unit has the responsibility for the land climate data portion of HadCRUT.

Back to top

16. Why is this responsibility with the UEA/CRU and not the Met Office Hadley Centre?

During the 1980s the UEA/CRU was funded, primarily by the United States ‘Department of Energy’, to collate a global land temperature record. Since then they have undertaken several major updates to the record increasing station density and time series completeness. This is why the UEA/CRU owns the primary IPR (Intellectual Property Rights) for the land climate records.

Back to top

17. So does the Met Office Hadley Centre have any involvement with the land climate data?

Since 2002 the Met Office Hadley Centre has formally assisted UEA/CRU by providing quality control and real-time updates for the land climate data set. The Met Office Hadley Centre is entirely responsible for the global sea-surface temperature component of the global mean temperature and also responsible for merging these series to create the HadCRUT product.

...

This clearly was not what some had hoped for - the raw station records - but a value-added, adjusted, and selected subset of the CRUTEM record.

It does beg the question, if it is the CRU's adjusted dataset, and not raw station data, what is the problem about releasing it in it's entirety?

Julia Slingo needs support: This email was circulating the UK universities earlier this week:

From: Gilbert, Pip On Behalf Of Slingo, Julia (Chief Scientist)

Sent: Monday, December 07, 2009 10:33 AM

To: Climate_Research; Met R&D all staff

Subject: URGENT: Supporting the science

Importance: High

Dear All,

As you are very aware, the science of climate change is under an

unprecedented attack and I know that many of you feel that we, as the

science community in the UK, should try to make our voice heard too. We

are therefore seeking a groundswell of support for a simple statement

that we, the UK science community, have the utmost confidence in the

science base that underpins the evidence for global warming. That

evidence has been arrived at through decades of painstaking and

meticulous research by many scientists across the world, who adhere to

the highest levels of integrity and honesty, the hallmarks of true

scientific endeavour. We come together now to defend our profession

against this unprecedented attack to discredit us and the science of

climate change.

I know this is very short notice but we would like to gather a list of

names from you and your scientific colleagues who support this move. We

would like to collect these names over the weekend and on Monday so that

a short letter, basically saying the above, can be released to the press

on your behalf on Tuesday, at the latest. If we can reach 100 signatures

or more from the UK academic community that would be a fantastic

response. Please can I request your help by asking you to not only

respond yourself, but also to send this on to scientific colleagues as a

matter of urgency.

The Met Office is able to provide help to pull these names together and

if you wish to support this statement then please send an email to:

julia.slingo@metoffice.gov.uk with ‘Yes’ in the Subject.

Many thanks,

Julia Slingo and John Hirst

Julia Slingo Chief Scientist

Met Office FitzRoy Road Exeter EX1 3PB United Kingdom

Why doesn't everyone here email her with ‘Yes’ in the Subject?

[Guest diessoli]

Hopefully, once and for all, this will show that the world has been warming and move the debate into a sensible arena based on qualification of warming and quantity of warming for each driver.

I believe you are overly optimistic here.

Given that the data and the methods have been available for quite a while now, I don't see how this will change things.

The debate you are referring to only exists because certain people are not prepared to accept the outcomes of a large amount of research.

The scientific debate has moved on to, for instance, what the impacts of the change will be.

D.

[VillagePlank]

I believe you are overly optimistic here.

Given that the data and the methods have been available for quite a while now, I don't see how this will change things.

The debate you are referring to only exists because certain people are not prepared to accept the outcomes of a large amount of research.

The scientific debate has moved on to, for instance, what the impacts of the change will be.

I'm sorry - but the source-code for climate models is not available for general release. Those are the methods that £trillions spend is based on. They haven't been published in any journal, they are not based on a reliable language such as SPARK (ever wondered why no plane in history has crashed due to a software error, nor any train? that's because the software is mathematically verifiable - the stuff that we know, so far, shows that the climate software is far from being verifiable) Indeed, Harry has spent a good chunk of his professional life trying to get to the bottom of a spaghetti codebase.

Oh, and btw, if methods are not reproducible (ie the source-code fully available) then that is anathema to the scientific method which requires that any assertion be reproducible. So, whilst I am certain you are sure that the science has moved on, I think the science hasn't even started, yet - since current AGW claims are simply non-reproducible assertions.

Edited December 12, 2009 by VillagePlank

[Guest diessoli]

I'm sorry - but the source-code for climate models is not available for general release. Those are the methods that £trillions spend is based on. They haven't been published in any journal, they are not based on a reliable language such as SPARK (ever wondered why no plane in history has crashed due to a software error, nor any train? that's because the software is mathematically verifiable - the stuff that we know, so far, shows that the climate software is far from being verifiable) Indeed, Harry has spent a good chunk of his professional life trying to get to the bottom of a spaghetti codebase.

I agree that not all climate model source code is available for download.

Some is, and has been for some time. I have worked with PUMA 7 or so years ago. This model had been developed to train students to use the much more complex ECHAM model.

For the latter the source code is indeed not available for free download, but if you are a researcher you can get access.

Can you give some specifics about what methods have not been published?

Oh, and btw, if methods are not reproducible (ie the source-code fully available) then that is anathema to the scientific method which requires that any assertion be reproducible. So, whilst I am certain you are sure that the science has moved on, I think the science hasn't even started, yet - since current AGW claims are simply non-reproducible assertions.

Is it, though? Then most scientists out there (in all fields) are by your definition unscientific.

For an conclusion to be reproducible you don't need the source code, you only need to know the method. People will take read these in the scientific literature and create their own implementation, which will lead to more robust results.

D.

[VillagePlank]

Can you give some specifics about what methods have not been published?

Obvious one is the piece of code that adds a 2nd degree polynomial to temperature data after 1930, and applies some other construct before 1930. Can you point me to the peer reviewed journal where that process has been described, please? It's in the source-code, and brief discussion of it is here.

Edited December 12, 2009 by VillagePlank

[Guest diessoli]

Obvious one is the piece of code that adds a 2nd degree polynomial to temperature data after 1930, and applies some other construct before 1930. Can you point me to the peer reviewed journal where that process has been described, please? It's in the source-code, and brief discussion of it is here.

Maybe there is nothing in the literature because the particular piece of code was not actually used for any publication.

Scientists play around with scenarios. The twiddle and speculate. I have no idea where this particular piece of code comes in (there is no context).

If you really want to cry foul over it you also have to show that this code was actually used to reach a particular conclusion _and_ why it is wrong to do it like this.

But just looking at just that little bit of code seems to be pretty pointless if you ask me.

But let's assume the author was naughty and used it to create a false warming post-1930 and published a paper based on this so that he can cash in on his wind park investments.

Other scientist would look at his paper take the data and try to replicate this stunning result. What do you think they would see?

Now assume other climate scientists would just download his code and the data. Now what do they do? Just run the program to verify the results? That would surely be pointless.

So all they can do is review the source code to say if it's legit. So scientific reproducibility is reduced to reviewing source code. I don't think you find many climate scientists that would be thrilled by that idea. Apart from that our good guys might not know IDL (they're used to python) and our mad scientist is an IDL guru who has hidden his manipulations very cunningly.

D.

[VillagePlank]

Maybe there is nothing in the literature because the particular piece of code was not actually used for any publication.

Scientists play around with scenarios. The twiddle and speculate. I have no idea where this particular piece of code comes in (there is no context).

If you really want to cry foul over it you also have to show that this code was actually used to reach a particular conclusion _and_ why it is wrong to do it like this.

But just looking at just that little bit of code seems to be pretty pointless if you ask me.

But let's assume the author was naughty and used it to create a false warming post-1930 and published a paper based on this so that he can cash in on his wind park investments.

Other scientist would look at his paper take the data and try to replicate this stunning result. What do you think they would see?

Now assume other climate scientists would just download his code and the data. Now what do they do? Just run the program to verify the results? That would surely be pointless.

So all they can do is review the source code to say if it's legit. So scientific reproducibility is reduced to reviewing source code. I don't think you find many climate scientists that would be thrilled by that idea. Apart from that our good guys might not know IDL (they're used to python) and our mad scientist is an IDL guru who has hidden his manipulations very cunningly.

D.

I trust you've read the link that I posted? I think you'll find that I've covered the caveats you mention - and pretty much agree with what you say.

However, we cannot tell that this is used or not used. The process simply isn't that transparent. And, to my mind, that is anathema to the scientific method.

[Guest diessoli]

I trust you've read the link that I posted? I think you'll find that I've covered the caveats you mention - and pretty much agree with what you say.

I started. But when I got to the point that somebody started to blather about how the station data is manipulated, I tuned out. Life's too short to wade through that sort of crap.

However, we cannot tell that this is used or not used. The process simply isn't that transparent. And, to my mind, that is anathema to the scientific method.

How do you expect to be able to? All you have is a piece of illegally published source code.

If you really want to know if the code was used and for what, why don't send the author an email and ask him or her?

D.

[VillagePlank]

I started. But when I got to the point that somebody started to blather about how the station data is manipulated, I tuned out. Life's too short to wade through that sort of crap.

The context is that of my discussion. It covers your caveats does it not?

How do you expect to be able to? All you have is a piece of illegally published source code.

If you really want to know if the code was used and for what, why don't send the author an email and ask him or her?

I have. No reply.

[Guest diessoli]

The context is that of my discussion. It covers your caveats does it not?

Yes it does.

I have. No reply.

That's a pity.

I've just checked the referenced blog again and seen that this is about Keith Briffa's code. So I guess you have send the email to him?

I hadn't realised that before or I would have said that it is unlikely that you will get a response. He's at CRU they will most likely not answer any queries whilst the investigation is going on. He might also still be ill.

Anyway I have taken this thread slightly off topic.

Let's wait for these published methods to come out.

D.

[Chris Knight]

The MetO have released part of the data as I mentioned here, and it is adjusted, homogenised data, and not the raw data that would meaningfully allow independent researchers to discover whether any adjustments or homogenizations have coloured the CRU temperature record that agrees with the other major long-term global temperature records (this is the reason given by CRU and UEA apologists that we should trust the data that has been generated by the CRU - that it agrees with NASA GISS and NOAA temperature records - so it should, they use essentially the same raw data, just slightly different adjustment and homogenisation algorithms, the American agencies are literally keeping up with the Joneses ).

Here is an interesting graph:

It shows how the unadjusted temperature rocketed in 1990 and beyond when loads of stations worldwide were discontinued from the network, many from the former Soviet bloc countries.

link to spreadsheet

The data sources and implications are discussed here, with other links

This whole "Global warming" thing is a matter of trust, and the agencies that control the data are not trustworthy IMO.

If the GHCN had up to 15,000 stations in 1970, why did the UK's premier climate research centre only hold data from about 5,000 historical stations. I see yet another smokescreen.

[The PIT]

The MetO have released part of the data as I mentioned here, and it is adjusted, homogenised data, and not the raw data that would meaningfully allow independent researchers to discover whether any adjustments or homogenizations have coloured the CRU temperature record that agrees with the other major long-term global temperature records (this is the reason given by CRU and UEA apologists that we should trust the data that has been generated by the CRU - that it agrees with NASA GISS and NOAA temperature records - so it should, they use essentially the same raw data, just slightly different adjustment and homogenisation algorithms, the American agencies are literally keeping up with the Joneses <_<).

Here is an interesting graph:

It shows how the unadjusted temperature rocketed in 1990 and beyond when loads of stations worldwide were discontinued from the network, many from the former Soviet bloc countries.

link to spreadsheet

The data sources and implications are discussed here, with other links

This whole "Global warming" thing is a matter of trust, and the agencies that control the data are not trustworthy IMO.

If the GHCN had up to 15,000 stations in 1970, why did the UK's premier climate research centre only hold data from about 5,000 historical stations. I see yet another smokescreen.

Interesting. How many of these stations were from soviet block???

Alos before anyone gets over excited the 100 years data is bound to show warming as the temps were recovering from the mini ice age anyway.

[jethro]

Here is the Russian side of the story as reported in The Telegraph:

http://blogs.telegraph.co.uk/news/jamesdelingpole/100020126/climategate-goes-serial-now-the-russians-confirm-that-uk-climate-scientists-manipulated-data-to-exaggerate-global-warming/

[The PIT]

Here is the Russian side of the story as reported in The Telegraph:

http://blogs.telegraph.co.uk/news/jamesdelingpole/100020126/climategate-goes-serial-now-the-russians-confirm-that-uk-climate-scientists-manipulated-data-to-exaggerate-global-warming/

Oops more possible corruption.