« PIOMAS January 2016 | Main | Global sea ice area record minimum? »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Kris

Could we believe that, this year's first century on NIPR_jaxa at the 30th of March: 13697343-13560921=136422 km2 minus !?

Diablobanquisa

Thank you, Rob.

Excellent, I agree that during 1935-1952 there are more data sources for August than for September: DMI and ACSYS direct observations in addition to AARI ones (except for 1940-1945, when the only available data source for both August and September is AARI). So, I'm looking forward to seeing your results for August.

And, yes, the 60s dip in September data is mostly due to the Atlantic and American sectors, that as you stated are missing before 1953.

Nevertheless, the 60s dip is still noticeable even if we start the time series from 1953, therefore relying only on direct observations for the whole Arctic (blue: Walsh; red: ours):

Cheers

Diablobanquisa

Just another comment to point out that, although there are more direct observations for August, your algorithm has a hard work ahead, because the missing areas are still huge:

- August vs. September direct observations, 1935 and 1952:


Anyway, I think your approach sounds really well, and I'm looking forward to seeing your August results. Take your time ;-) , and thanks again for your work.


Diablobanquisa

The most difficult years will be the 40s:

Neven

Diablo, I'm looking into something and was wondering if a map exists for September 1971. I've already downloaded the map on your blog, of course, but I was just wondering if there's something based on satellite data out there.

Jim Hunt

Have you seen this 2015 paper Neven?

The process of bringing dark data to light: The rescue of the early Nimbus satellite data

It suggests that satellite data for September 1971 is going to be hard to come by:

Nimbus IV launched April 8th, 1970 and collected data until April 14, 1971, when problems with the spacecraft’s ability to maintain Earth pointing began.

Nimbus V data runs from December 1972:

https://nsidc.org/data/docs/daac/nsidc0009_esmr_seaice.gd.html

Neven

I hadn't seen the paper, but I did peruse the NSIDC website to try and find something useful. Couldn't find much for 1971 though.

Diablobanquisa

Hi Neven,

As Jim has pointed out, ESMR passive microwave observations didn't start until December 1972.

So, if you are looking for a pan-Arctic chart for September 1971, our map or the 'new Walsh' one are your best options (although I should recommend ours ;-) ).

If you are interested in a particular region, you could use other and more detailed sources:

* The nice weekly regional ice charts from the Canadian Ice Service, for Chukchi-Beaufort-CAA-Huddson-Baffin: http://iceweb1.cis.ec.gc.ca/Archive/page1.xhtml?grp=Guest&mn=&lang=en

* AARI maps for Barents and the Siberian Sector: ftp://sidads.colorado.edu/pub/DATASETS/NOAA/G02176/pngs/

* ACSYS for Greenland Sea (August 31th): ftp://sidads.colorado.edu/pub/DATASETS/NOAA/G02169/ice_edge_positions/browse/aug/iep19710831.jpg

Diablobanquisa

I love the old Canadian charts:


Neven

That's great, Diablo, thanks. And, indeed, very cool maps!

Bill Fothergill

@ Diablo & Rob & Neven & Jim & anyone else

Are you guys aware of the archive material in Project Birdseye?

http://nsidc.org/data/docs/noaa/g02188-birdseye/

Bill Fothergill

I'm sure that somewhere on the NSIDC’s website, there's an article (or arcticle?) about Garrett Campbell and David Gallaher doing a digitization project of satellite photographic images from the 60's and early 70's.

Bill Fothergill

The grey matter grinds exceedingly slow these days.

The article about the data recovery/digitisation was called "Glimpses of sea ice past".
http://nsidc.org/monthlyhighlights/2013/04/glimpses-of-sea-ice-past

However, the link seems to be misdirecting, but there's a brief mention in the May 2013 Sea Ice News. (Just before the tribute to Katherine Giles - a lovely lady.)
http://nsidc.org/arcticseaicenews/2013/05/

The project is also mentioned here...
http://nsidc.org/the-drift/2013/07/new-life-for-old-data-nimbus-data-rescue-project-publishes-first-data-set/

Diablobanquisa

Thank you, Bill, I wasn't aware of the Birdseye Reports.

Regarding the recovery of visible satellite imagery from the 60s and 70s:

- New estimates of Arctic and Antarctic sea ice extent during September 1964 from recovered Nimbus I satellite imagery: http://www.the-cryosphere.net/7/699/2013/tc-7-699-2013.html

- http://cires.colorado.edu/news/press/nimbus-data-rescue/

- http://nsidc.org/data/nimbus/


Rob Dekker

Neven said

I'm looking into something and was wondering if a map exists for September 1971.

Here is the September 1971 pan-Arctic map from the latest Walsh series :

Click the image for a larger version.

Note that much of this map originates from the "Walsh and Johnson" source, which Diablo already mentioned may have a (high) "concentration bias". One example of that bias is the ice mapped in Baffin Bay, which appears to be larger than that from the Canadian Ice service (maps by Diablo above).

Even with this known "high" bias, this map may be useful.

Rob Dekker

Bill, Diablo,
That Nimbus I report about September 1964 is interesting. It states :

The Arctic 1964 extent is near the 1979–2000 average from the passive microwave record, suggesting relatively stable summer extents during the 1960s and 1970s

It is unfortunate that they don't give an actual number, but if we look at the Walsh, Diablo and Meier reconstructions :
https://diablobanquisa.files.wordpress.com/2016/02/new-walsh-ours-meier.png
1964 comes out quite a bit higher than "near the 1979–2000 average" in all three reconstructions...

Rob Dekker

I know we are some 100+ comments into this thread, but from the stuff I've learned, and with the NetCDF4 software working on the rich Walsh data sources, specifically the wonderful AARI data source running a continuous record from 1935 through 1978, I would like to present my best assessment of the September Arctic sea ice extent reconstruction :


This reconstruction uses observational data from AARI (half the Arctic) over the 1935-1978 period, while the rest of the Arctic relies on a climatology that uses the 1979-1997 satellite era average. For the record from 1979 onward, I use satellite observations.

The choice of climatology (1979-1997) for the 1935-1978 back-drop is that this climatology matches rather closely with the 1935-1978 AARI observations. For example, the 1935-1978 record for AARI that you see above averages at 7.36 M km^2, while the average of the backdrop 1979-1997 climatology averages at 7.35 M km^2.

Notably, the 'trend' in this reconstruction from 1935 - 1978 is 9 k km^2/year (some 1.3 %/decade) which is statistically insignificant, (since the standard deviation is 340 k km^2).

I am not sure exactly how this observational reconstruction compares to Meier et al and Diablo's work, but the argument is that this reconstruction is based on pure observational data, leaving the unobserved parts of the Arctic constant.

This is the best I have at this time.

Diablobanquisa

Rob wrote: "It is unfortunate that they don't give an actual number"

They give an actual number: 6.9 +/- 0.3 M km^2. And it's, indeed, much lower than Walsh's, Meier's and ours (ours is the lowest, with 7.98, Meier gives 8.28 and Walsh goes to 8.6).

However, I have many doubts about that estimate. It's a combination of Nimbus I visible imagery with AARI maps and Alaskan charts:


Fig. 6. Outline of September 1964 Arctic sea ice edge from Nimbus I (black dots), Alaskan ice charts (red), and Russian ice charts(blue). The pink line is the 1979–2000 median ice edge from the passive microwave-based NSIDC Sea Ice Index product.

- Those black dots north of Greenland and Ellesmere look suspicious. Is that an 'ice edge'? I guess that the whole area north or Greenland and Ellesmere should be ice covered (even though it could be some narrow coastal polynyas).

- According to the sea ice edge defined by the black dots at Greenland Sea, this Sea is almost completely ice free. This fact looks a bit strange. For instance, ACSYS gives this for August 18th:


So, I think that defining the sea ice edge just from Nimbus I visible imagery wasn't easy at all, and I'd say that their 6.9 estimate seems too low.

Diablobanquisa

Rob, regarding your latest graph:

- With the use of a climatology (1979-1997) for the whole pre-satellite period (1935-1978), you are supressing all the annual, decadal and multidecadal variability in the Atlantic and American sectors.

- With the use of that climatology during 1953-1978 you are excluding a lot of direct observations and reliable data for the Atlantic and American Sector (during 53-78 Walsh and our time series rely almost completely on direct observations).

- The 70s look underestimated. According to ESMR passive microwave satellite observations, the extent in the 70s was very similar to the 80s, whereas in your graph 70s are lower than 80s.

- Overall, I think that your graph underestimates the whole 53-78 period.

(We used a climatology before 1953 because there are huge areas without direct observations that must be infilled. But from 1953 onwards, when direct observations are available for the whole Arctic, I think that the use of a climatology makes no sense.
Of course, the uncertainty before 1953 always will be larger: our choice of the 35-52 climatology relying on SAT-SIE could be wrong, 'analog' infilling by Walsh could be wrong, and your 'analog' algorithm, when ready and working, could be wrong. But the fact is that before 1953 we all necessarily must rely on something different from direct observations, whereas from 1953 onwards we can rely on direct observations only).

Cheers

Neven

Thanks for the Walsh map, Rob.

Diablobanquisa

Maps for 1966 and 1969 derived from Nimbus II and Nimbus III visible imagery are also available:

- Late August 1966:


- Late September 1969:

Rob Dekker

Diablo said (at the core of criticism of my AARI-only reconstruction) :

With the use of that climatology during 1953-1978 you are excluding a lot of direct observations and reliable data for the Atlantic and American Sector

Yes. And I deliberately assumed the Atlantic and American sectors to be constant.

The reason is simple : The record of the Atlantic and American sectors over the 1935-1978 period is a mess.

September record from 1953-1978 is spotty at best, and filled in by the Walsh and Johnson source which in your own words has probably a "concentration bias" but we really don't have the original ice-edge observations any more. And the September record for the Atlantic and American sectors is completely absent for the 1935-1953 period.

The August record 1935-1978 is plagued by misplaced Kelly fields, which were (even if correctly placed) were doubtful "guesses" of where the ice-edge was.

And a "spatial-filling" method for 1935-1978 for either Sept or Aug would be difficult to validate and calibrate, since, for example, there are NO satellite-era August "analogs" for the sort of extended ice cover in the Greenland sea that we see in the August 1935 DMI map :
ftp://sidads.colorado.edu/DATASETS/NOAA/G02203/1935_08.jpg

So ANY spatial filling algorithm (and I'm looking even into EOF's and PC analysis) will have a VERY hard time producing reliable results, since the ice extent in the 1935-1978 period in August and September simple does not have enough reliable calibration data to work with.

That is why I relied on the only continuous source we have available, which is AARI, which at least covers half the Arctic, and why I kept the American and Atlantic side of the Arctic constant.
I'd be happy to change that (constant) climatology, but that will just move the whole graph up or down a bit and does not affect the (insignificant) trend over the 1935-1978 period.

Rob Dekker

What I'm kind of interested in now, is why did I find a nearly flat 1935-1978 record based on AARI observations, while Mahoney et al finds actually a DECLINE in sea ice extent over the same period using the same observations :
http://seaice.alaska.edu/gi/publications/mahoney/Mahoney_2008_JGR_20thC_RSI.pdf
See figure 6, Russian Arctic, summer graph.
It can't be that I restricted my analysis to September only, since the August reconstruction shows the same thing : nearly flat for the 1935-1978 period.
I guess I have some more work to do :o)

Rob Dekker

I noticed that Diablo previously commented on Mahoney et al, with the following applicable arguments :

- They left out western Kara sea (just in that area, there is more ice during 1946-1951 than during 1935-1945).
- Their methodology could lead to an extent overestimation when data are sparse (by assuming the same edge for a whole sea).

I'm not sure how much the western Kara has influence, but it's woth checking out using the Walsh data set. I'll do that.

The second argument is very interesting, especially since the AARI dataset is sparser in the early (30's) record. Not sure why that would cause an overestimation however.

Again, I have some more work to do, but until then, my AARI-based reconstruction over 1935-1978 stands.

Diablobanquisa

September record from 1953-1978 is spotty at best, and filled in by the Walsh and Johnson source which in your own words has probably a "concentration bias" but we really don't have the original ice-edge observations any more.

"Walsh and Johnson" (http://journals.ametsoc.org/doi/pdf/10.1175/1520-0485%281979%29009%3C0580%3AAAOASI%3E2.0.CO%3B2 ) is a reliable source for 1953-1978. The "concentration bias" doesn't mean that it's unreliable, but just that, if you want to compare it directly with passive microwave satellite observations, you should take into account that bias.
The bias can be quantified on the basis of the average difference between "Walsh and Johnson" and passive microwave satellite observations during the overlapping period (1972-1978).

So, I don't see any reason to exclude all the observations for the Atlantic and American Sector during 1953-1978.

In fact, we can check that your approach gives unrealistic results. For instance, on your September graph you show the 70s lower than 80s. That disagrees with satellite observations: http://climate.envsci.rutgers.edu/pdf/Cavalieri2003GL018031.pdf


And a "spatial-filling" method for 1935-1978 for either Sept or Aug would be difficult to validate and calibrate

Absolutely!
(But I think you only have to infill 1935-1952)

for example, there are NO satellite-era August "analogs" for the sort of extended ice cover in the Greenland sea that we see in the August 1935 DMI map

Well, surely you could find analogs during 1953-1978.

(Another question is that, probably, DMI direct observations themselves will have their own "concentration bias". That doesn't mean that those observations are wrong or unreliable. Just that, during the melt season, a satellite borne passive microwave sensor would give lower concentrations and extents than the operational and/or pre-satellite charts do. Taking this into account, even 1989 or 1998 don't look that bad analogs.)

Diablobanquisa

Regarding Mahoney: I think the earlier part of their record is not consistent.

Their data: ftp://sidads.colorado.edu/pub/DATASETS/NOAA/G02182/RArctic_ip_area.csv

For instance, for 1937 they only give a September number for Kara and Laptev, and they exclude Barents, ESS and Chukchi. How do they compare 1937 with later years when they give a number for the 5 Seas?

For instance, they are excluding this August 1937 map for ESS and Chukchi:


ftp://sidads.colorado.edu/pub/DATASETS/NOAA/G02176/pngs/1937/e19370915_tc.v0.png


Why? Because their methodology using longitudinal slices can't define a sea ice edge from this map, because there isn't a boundary between water and ice in ESS and Chukchi, but a boundary between water and 'no data'. But we do know that the blue area was open water! You, Walsh and us have assumed that the sea ice edge is at the boundary between water and 'no data'. Mahoney et al. assumed that this map wasn't useful at all for ESS and Chukchi.

Cheers


Diablobanquisa

they are excluding this August 1937 map for ESS and Chukchi

September

Diablobanquisa

Rob, I have calculated your bias during 1972-1978, and you should add 0.184 M km^2 to your numbers, in order to be consistent with passive microwave observations.

(https://diablobanquisa.files.wordpress.com/2016/04/robadj.png )

(you could choose a climatology 0.184 higher)

(although I disagree with using a climatology when reliable data are available, i.e. 1953-1978)

Diablobanquisa

I am not sure exactly how this observational reconstruction compares to Meier et al and Diablo's work


(click for larger versions)


For the satellite era, Meier and us use Sea Ice Index monthly means (https://nsidc.org/data/seaice_index/ ).

Rob and Walsh use mid month daily values from G02202 (http://nsidc.org/data/docs/noaa/g02202_ice_conc_cdr/index.html ).

Diablobanquisa

And vs Walsh


Diablobanquisa

I'd like to clarify that the suggested +0.184 M km^2 adjustment would be in order to match Walsh' satellite numbers. (you plotted your graph using Walsh' satellite numbers)

However, your 72-78 numbers almost directly match Sea Ice Index ones, so your time series using SII instead of Walsh would be like this (including also our time series):

(click for larger versions)


Both time series are reasonably similar. However, our main disagreements are:

- 1935-1952: you use a higher climatology than ours, so your values are somewhat higher.

- 1960s: you miss part of the 60s 'dip' because you use a climatology for the Atlantic and American Sector instead of direct observations (those described by Walsh and Johson 1978, Table 1).


Rob Dekker

Diablo, thank you for these comparisons. That is very nice of you.

Moving forward, I think we have a reasonable shot at reconstructing the 1935-1978 Arctic sea ice extent beyond what has been done already (by Walsh, Meier and you), but we would need to do a couple of things :

1) For the 1953-1978 period, we need to correct that "concentration bias" in the "Walsh and Johnson" source. For that we need to understand the nature and origin of the bias, and find a calibration period to homogenize it with other observational sources over the same period.
I read the "Walsh and Johnson" paper in more detail, and I think the nature of their concentration bias originates from their EOFs. It seems they created these 'smeared-out' ice concentrations we see in their source by possibly (implicitly) exchanging ice-concentration for ice-probability. If that is indeed the case, then we may interpret a 50% concentration in these Walsh and Johnson fields as "a 50% chance of finding the ice edge here".
I will run some tests over the next week to see if that is solving the issue.
At the same time, I can run some calibration tests to see which concentration in the "Walsh and Johnson" source gives the best correlation with the 1972-1978 overlapping period. For that, do you have the September (and/or August) extent numbers of passive microwave satellite observations over the period 1972-1978, as homogenized with the modern satellite era numbers ?

These tests and calibration and consequent adjustments to the 'cut-off' ice concentration for the "Walsh and Johnson" source will result in an 'adjusted Walsh' series (for September and August) for the 1953-1978 period that should be much closer to the truth.

2) For the 1935-1952 period : for August we really need to fix these Kelly fields that are misplaced by a month. I have some idea on how to do that, and I will run some experiments over the next week.

Once we have both this issues fixed we can provide a 1935-1978 record that is 'homogenized' to the best of our abilities (and observational sources). It would be interesting to analyse the trend over that August reconstruction.

However, for September 1935 - 1952, we will have to resort to some form of spatial and temporal fill-in, or resort to a 'climatology', which is much less trustworthy in my opinion.

Rob Dekker

About Mahoney et al, I agree with you that his early records are suspect.

But I don't think that the AARI charts that he omitted are the issue (of his find of declining sea ice extent). After all, he states in his paper that he did a lot of purging himself, which did not affect the result. The issue has to be grounded in the actual AARI charts that he DID use for his estimates of the 30's and 40's.

I first thought that possibly his early record was biased towards earlier (July and August) AARI charts, since there are few September records available in the 30's. That would bring the 'average' up a bit. I noticed some of that effect in his record of the Chukchi sea and the ESS, but to quantify it I need to run some numbers.

Either way I would like to get to the bottom of Mahoney's mystery declining sea ice extent in the early record of the Russian Arctic.

Diablobanquisa

do you have the September (and/or August) extent numbers of passive microwave satellite observations over the period 1972-1978, as homogenized with the modern satellite era numbers ?

Meier et al. (http://www.the-cryosphere.net/6/1359/2012/tc-6-1359-2012.pdf )and us used Cavalieri's numbers: ftp://sidads.colorado.edu/pub/DATASETS/nsidc0192_seaice_trends_climo/total-ice-area-extent/esmr-smmr-ssmi-merged/gsfc.nasateam.month.extent.1972-2002.n

Paper: http://climate.envsci.rutgers.edu/pdf/Cavalieri2003GL018031.pdf

We homogenized the 72-78 period in order to match Sea Ice Index: ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/Sep/N_09_area.txt

(The homogenized 72-78 values for September are 7.59, 7.42, 7.25, 7.62, 7.42, 7.26, 7.29)

(As far as I know, Cavalieri's gridded data aren't available. However, ESMR gridded data are available here: https://nsidc.org/data/docs/daac/nsidc0009_esmr_seaice.gd.html )

Diablobanquisa

Regarding Kelly fields, I am more skeptic. Even if we use the right ones, I don't think they are a reliable source, but a guess at best. (I say 'at best' because from 1939 onwards the white coloured areas on DMI charts they are derived from, don't look like an estimate at all, but just like an 'area without data')

Diablobanquisa

Regarding Mahoney, honestly I don't understand how they 'infilled' their many 'missing' seas and months during the 30s and early 40s in order to get their overall numbers for the whole domain.

And I think that they have missed reliable data that could be helpful to reduce the 'missing' seas and months. For instance, they don't give a number for ESS and Chukchi in September 1937, but I think that the map posted above allows an estimate.

In summary, I don't understand their results for the earlier part of the record.

Diablobanquisa

According to Mahoney's Fig. 6, 1937 is the highest summer value. For instance, it's higher than 1949.

I wonder how they reached that conclusion from AARI maps:

(September, 1937 vs. 1949; if we look at August, it's the same)


Rob Dekker

Diablo, indeed that 1937 - 1949 September difference is stunning, and puzzling how Mahoney could have concluded that 1949 was smaller than 1937.

I looked into the numbers for an explanation :
ftp://sidads.colorado.edu/pub/DATASETS/NOAA/G02182/RArctic_ip_area.csv

Here is the breakdown in months and seas for 1937 :

1937 :
               Jul  Aug  Sept  Summer
Barents       0.46    x   x    0.46
Kara          1.03 1.04 0.73   0.93
Laptev        1.11 0.93 1.06   1.05
ESS           1.36    x   x    1.36
W. Chukchi    0.48    x   x    0.48
TOTAL                          4.28

and for 1949 :

1949 :
               Jul  Aug  Sept  Summer
Barents       0.56 0.56   x    0.56
Kara          0.79 0.59 0.75   0.72
Laptev        1.19 1.06 1.00   1.10
ESS           1.40 1.39 1.32   1.38
W. Chukchi    0.42 0.31 0.28   0.35
TOTAL                          4.11

The 'total' number for each year (4.28 and 4.11) does seem to match figure 6 in Mahoney's paper. That's re-assuring :o)

Note that the difference between these two years is not that large. For example, year 1943 clocks in at around 3.4, which would be a starker contrast with 1937.

But all that aside, let us break it down by month, to find out where the difference comes from.

For July, 1937 clocks a total of 4.44 and 1949 comes in at 4.36. So Mahoney found that at least for July, 1949 is lower than 1937. Can we see that in the AARI maps ?

For August and September, for 1937 Mahoney reports only the Kara and the Laptev, and for September specifically, 1949 and 1937 are not that different in the Kara and the Laptev. Which kind of (with a bit of imagination) matches your animated gif.

So, it seems that the big difference between 1937 and 1949 was in the ESS and the Chukchi, and for these two seas, Mahoney does not report data in August and September.

Thus, I take my previous comment back.
It appears (at least for the 1937 to 1949 comparison) that the difference between Mahoney's report and what we see in the AARI charts comes from Mahoney's omission of charts that do not show an ice edge.

That happens more often for low-extent and late (September) charts, and thus his "uptick" in the 30's may be more an indication of low ice extent than of high extent as Mahoney suggests.

What do you think ?

Rob Dekker

I so, Mahoney's decline in summer sea ice in the Russian Arctic during the 30's and 40's is side-effect of his method, which discards AARI charts that show only open water at a particular latitude.

And the decline may be an indication that the Russian Arctic in the 30's had more open water than in the 50's.

With these caveats exposed, I put more trust in your (and my) method of using a climatology at the back-drop to AARI observations, and Walsh' method of spatial infilling. Mahoney's method is just confusing everything.

Diablobanquisa

I agree, Rob.

Rob Dekker

I had only 30 minutes today, but I ran a quick experiment to 'calibrate' the Walsh and Johnson source in the new Walsh series with the 1972-1978 satellite observations. So as to quantify the "concentration bias" we suspect in that source, and adjust the series accordingly.

Turns out the best fit is when the Walsh and Johnson ice concentration 'extent' concentration is set at 35 % instead of the normal 15%.

I'm a bit cautious to publish the resulting 'adjusted' Walsh series, since the fit is not very good, and I'd like to run some statistics on other methods of calibration (such as a simple scalar subtraction or replacing the Walsh and Johnson source with a 'fixed climatology' and calibrating that one.

Stay tuned.

Rob Dekker

Also, I wonder if we could use the Canadian Ice Service charts as a way to 'calibrate' the Walsh and Johnson source and quantify its "concentration bias". After all, the Canadian charts (some of which Diablo posted above, for 1971) were part of the Walsh and Johnson source in the first place AFAIK. And they cover some unique areas such as Baffin Bay and the Canadian Arctic, which are not covered by other sources.
http://journals.ametsoc.org/doi/pdf/10.1175/1520-0485%281979%29009%3C0580%3AAAOASI%3E2.0.CO%3B2
Table 1 does list the Canadian Meteorological Service.
Are these the same charts ?

Rob Dekker

Question for Diablo :
When you compiled this graph :
https://diablobanquisa.files.wordpress.com/2016/04/robadj.png
did you compile "Walsh adjusted to match ESMR" ?
Did you subtract a scalar that best matches the Walsh numbers with ESMR numbers over the training period (1972-1978) or did you do something 'gridded' ?

Diablobanquisa
Table 1 does list the Canadian Meteorological Service. Are these the same charts ?

I'm not sure whether they are the same charts I posted above: http://iceweb1.cis.ec.gc.ca/Archive/page1.xhtml?grp=Guest&mn=&lang=en (Weekly Regional Ice Charts - Black and White, from 1968 onwards).

Either way, they could be useful to 'calibrate' the Walsh and Johnson source. However, CIS Ice Charts could also have their own 'bias' when compared with passive microwave data: http://www.tandfonline.com/doi/abs/10.3137/ao.410405


Did you subtract a scalar that best matches the Walsh numbers with ESMR numbers over the training period (1972-1978) or did you do something 'gridded' ?

Nothing gridded, just a subtraction (but not directly to match ESMR, but to match 'adjusted' ESMR).

Firstly, I took Cavalieri's numbers (1972-2002) and Walsh's satellite numbers (1979-2013), and I calculated their mean difference during the overlapping period 1979-2002: Walsh's numbers are 0.35 M km^2 higher.

Secondly, I adjusted the 1972-1978 Cavalieri's numbers by adding 0.35, in order to match Walsh's satellite values post-1978, so that I obtained a consistent time series from 1972 to 2013 (CW7213).

Thirdly, I calculated the mean difference during 1972-78 between Walsh's original numbers and CW7213 ones (Walsh's numbers are 0.28 M km^2 higher).

Finally, I adjusted the 1935-1978 Walsh's original values by subtracting 0.28, so that the 1972–1978 period is consistent with Walsh's satellite numbers post-1978.

The same method could be used to obtain 72-78 values consistent with any satellite dataset. For instance, in order to match NSIDC's monthly means, 0.51 M km^2 must be subtracted from the Walsh's original 1972-1978 numbers.

Diablobanquisa
Finally, I adjusted the 1935-1978 Walsh's original values by subtracting 0.28, so that the 1972–1978 period is consistent with Walsh's satellite numbers post-1978.

1972

(Meier and us adjusted the 1953-1971 period by assuming the same bias than during 1972-1978)

Diablobanquisa

https://diablobanquisa.files.wordpress.com/2016/04/cccvs.png

Rob Dekker

Since the WalshJohnson is the dominant source in the Waslh series from 1953-1978 (determines some 75% of ice cover), it is important that we remove any 'bias' from this source.

To get the WalshJohnson source 'calibrated' to the adjusted EMSR observations from 1972-1978, I looked at two methods :

1) Subtract a fixed amount : over 1972-1978 original Walsh series is 450 k km^2 higher than the adjusted EMSR numbers Diablo mentioned above.
This works to get the 'mean' of the series alligned, but the resulting standard deviation of the remaining differences is 247 k km^2.

2) Adjust the WalshJohnson source ice concentration for 'extent' from 15% to 35% in the gridded product.
This also brings the mean in line with EMSR, but the standard deviation of the remaining differences is lower : 187 k km^2.

So adjusting WalshJohnson directly for 'ice-concentration' bias gives a more accurate match with EMSR numbers from 1972-1978 than simple scalar adjustment of the series.

That is a GOOD thing, because it means that the WalshJohnson source appears to be better matching regional and local ice cover than a plain pan-Arctic scalar adjustment.

Here is the result of the WalshJohnson adjustment for the entire Walsh series :

I'm not sure how the adjusted Walsh series compares to Meier et al and Diablo, and my (AARI-only) series, but at first glance it looks to me that all series are now better aligned with each other over the 1953-1978 period.
Which is a very encouraging find.

Diablobanquisa

Excellent Rob, you've done a great work adjusting directly for the 'ice-concentration' bias of the 'Walsh&Johnson' source.

However, a couple of comments:

- I think that in your graph you are using Walsh's satellite numbers from 1979 onwards. However, the homogenized ESMR values I posted above (7.59, 7.42, 7.25, 7.62, 7.42, 7.26, 7.29) are adjusted to match NSIDC's Sea Ice Index. So you should use Sea Ice Index from 1979 onwards (ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/Sep/N_09_area.txt) (Sea Ice Index values are 0.23 M km^2 lower than Walsh's ones).
(Alternatively, you could use the Walsh's satellite numbers, but adjusting the concentration before 1979 in order to match the 1972-1978 adjusted ESMR values that match Walsh's satellite numbers: 7.82, 7.65, 7.48, 7.85, 7.65, 7.49, 7.52)


- Now, 1935-1952 left. That's a different story.

Rob Dekker

Thanks Diablo.
To calibrate to the second ESMR series (7.82,...,7.52) I need to change the extent-cut-off concentration in the Walsh&Johnson source to 25%. And of course the corrected graph then moves a bit closer to the original Walsh series (which used 15% for all sources).
Also the standard deviation of the remaining differences goes up (to 217 k km^2).

But since we are now deep into detailed calibration issues, shouldn't be calibrate the AARI to the correct ESMR 72-78 series as well ?

Rob Dekker

Before we move on, let me summarize what we have found in the new Walsh series, so far :

1) The "Kelly fields" source seems to be misplaced one month. This makes the August fields look like July, and the July fields look like June, and it significantly affects these month's 'extent' reconstruction for the periods that the Kelly fields are used (1924-1956?).

2) The "Walsh&Johnson" source that dominates the 1953-1978 reconstruction suffers from a "concentration bias", which moves the Walsh reconstruction above reconstructions such as by Meier et al over the same period.

3) If the concentration bias in the "Walsh&Johnson" source is calibrated over the EMSR 72-78 data, it shows a better correlation if the EMSR data is used that is homogenized with the NSIDC Sea Ice Index satellite era than the satellite era that is currently used in the Walsh dataset.

It seems to me that it is time to contact Walsh and make him aware of our findings so far, for future updates of his dataset.

Rob Dekker

Regarding that EMSR data, do you have the extent numbers for 72-78 for August as well ? Homogenized on NSIDC sat era, and/or Walsh sat era. I'd like to calibrate Walsh&Johnson for August, and see if the concentration bias is the same or different than for September...

Diablobanquisa

Hi Rob,

- The homogenized 72-78 numbers for August (those that match NSIDC Sea Ice Index) are: 8.28, 8.2, 7.86, 7.65, 7.96, 7.32, 7.91.

(Right now, I can't calculate the numbers to match Walsh's satellite data, I'll post them later)


- Regarding AARI data, although I think that they are more consistent with passive microwave readings, I agree that we could try to adjust them as well, if neccesary. How do you suggest doing it?

(I'm thinking that maybe the best approach to adjust both Walsh&Johnson and AARI sources could be testing them against ESMR gridded data: ftp://sidads.colorado.edu/pub/DATASETS/nsidc0009_esmr_seaice/north/monthly/ Of course, it is a more difficult approach. If we do this, we should also check whether ESMR maps match Cavalieri numbers, and if that is the case we should use for the satellite era Cavalieri's numbers themselves.)


( - Moving to 1935-1952, I'm thinking that, when Walsh uses analogs from 1953-1978 to infill the missing areas during September 1935-1952, he is actually 'importing' the 'concentration bias' from the Walsh&Johnson source. I think it deserves an adjustment. However, we don't know which analogs were used for each year... pre-satellite, satellite, a mix...Anyway, when I have some time, I'll look at Walsh's September 35-52 maps looking for the 'concentration bias footprint', that should be a wider and 'blueish' Marginal Ice Zone. Any ideas about this issue? )

Diablobanquisa

August 1972-78 ESMR numbers adjusted to match Walsh's satellite data: 8.61, 8.53, 8.19, 7.98, 8.29, 7.65, 8.24.

Bill Fothergill

@ Diablo and Rob

I am currently engaged in a lengthening email exchange with a representative of the American Statistical Association. This was originally in relation to the following article ...

http://www.stats.org/climate-change-statistical-significance-and-science/

but has since extended to more general matters pertaining to the use of statistics in the climate change arena.

As long as I have permission from both of you, I would like to direct his attention towards this thread. The reason is so that he can see first-hand the difference between genuine debate and genuine scepticism, and what passes for scepticism elsewhere.

Rob Dekker

Hi Bill,
I would very much welcome a visit by a statistics expert.

If not only because the next thing we will look into is a statistically sound 'spatial filling' algorithm.

In its simplest form, the problem of spatial filling looks like (still informally) like this :

Given N number of 2D images on a grid of ice extent in the Arctic (the calibration images), and given M number of observations (ice or no ice) at M grid pixels, what is the probability of finding ice in the grid pixels that do not have observations. And subsequently, what is the probability density distribution of ice extent (mean and standard deviation).

Diablo, I get your point about 'importing' the 'ice concentration bias' from the Walsh&Johnson source into the infilling that Walsh did for 1935-1952. That effect may be real, but to know if it is, we would need to reverse-engineer Walsh' infilling algorithm (which may be needed any way to validate that part of his reconstruction) and even more important : we need to know if Walsh used the Walsh&Johnson source as an "ice observation" or not when he ran his 1935-1952 spatial infilling algorithm.

Rob Dekker

Regarding calibration issues, Diablo said :


(I'm thinking that maybe the best approach to adjust both Walsh&Johnson and AARI sources could be testing them against ESMR gridded data: ftp://sidads.colorado.edu/pub/DATASETS/nsidc0009_esmr_seaice/north/monthly/ Of course, it is a more difficult approach. If we do this, we should also check whether ESMR maps match Cavalieri numbers, and if that is the case we should use for the satellite era Cavalieri's numbers themselves.)

I agree that an approach using gridded data will be more accurate.
And yes, it is also more difficult.
Maybe we should keep this issue (calibration with ESMR) aside for the moment, since the uncertainties facing us in pre-1953 reconstruction (spatial and temporal infilling) likely will be much larger than the (minor) uncertainties we face if we want to accurately reconstruct post-1952 ice extent time series.

Rob Dekker

Diablo, in summary : If you would like to (maybe for a future publication) make a very accurate reconstruction of the 1953-1978 period, beyond what Meier et al did, then we should tease out the calibration issues of the various sources (AARI, Walsh&Johnson, and also NASA yearbook source which made 1963 the highest of them all).
But is you would like to focus on the 1935-1952 period, then we should attempt to construct a statistically sound spatial and temporal in-filling algorithm and validate it using the real ice observations over that period.
Your call which direction you want to take it.

Diablobanquisa

Bill, "Nihil obstat" from me.

Rob, I think that, after adjusting the 'Walsh&Johnson' source, the 1953-1978 period is rather reliable already.
(and I think it's very similar to the result presented on our paper, and already better than Meier et al. since they didn't integrate AARI data).

So, at this point, maybe I'd be more interested in the 1935-1952 period.

Bill Fothergill

"Nihil obstat"

Isn't that the name of the fast-acting, high-strength laxative that was recently reclassified as a biological weapon?
;-)

Now that I have the necessary two green lights, I'll get in touch with the ASA, and then it's over to them.

cheers guys

Rob Dekker

Before we move to the 1935-1952 period, here are the results of calibrating the Walsh&Johnson source to the 1972-1978 ESMR data, for the month of August.

For starters, the Walsh&Johnson source in August determines a significantly smaller portion of the Arctic than for September over the 1972-1978 period. Where in September Walsh&Johnson determines 75% of ice cover and 50% of ice edge (the rest being AARI), in August other sources (such as ACSYS and the Dehn collection) take over, and Walsh&Johnson is just used for the Canadian Arctic and Baffin Bay area.

Also, the mean of the complete Walsh series for August 1972-1978 is quite a bit higher than ESMR (486 k km^2 w.r.t. Walsh' satellite series, and a whopping 816 k km^2 w.r.t. Sea Ice Index satellite series).

To bring the 'mean' ice extent in line with ESMR, I had to reduce the ice-concentration-cut-off for extent to 32% (to match with Walsh sat era) and 53% (to match SII sat era). Also the standard deviation for the remaining differences with ESMR are not that good (224 k km^2 for Walsh-adjusted, and 287 k km^2 for SII-adjusted).

Interestingly enough, the best match with ESMR in August is obtained if we determine the ice-edge for extent to be at a concentration of 25% in the Walsh&Johnson source. Standard deviation from ESMR is then only 186 k km^2.

This tells me that probably the Walsh&Johnson source should be interpreted as having an ice-edge(for extent) of some 25%. That removes its high-bias for both September and August.
It also tells me that one or more of the other sources (Dehn collection, ACSYS, and/or AARI) still has a high-bias that is not yet accounted for,

But enough about these calibration issues.
Let us move on the the 1935-1952 period, and the 'spatial infilling' challenges that will bring.
I am working on implementing a first version of my 'matching' algorithm which should be able to find 'analogs' where observations are sparse. I intend to try that algorithm out first on August 1935, and see if we can do a better job there than the misplaced Kelly fields in the Walsh reconstruction.

Rob Dekker

Bill,
I just read that article you pointed to.
What a weird story by Michael Lavine.

As if he deliberately wants to misinterpret what Naome Oreskes is saying. Picking on minor definition differences and missing the big picture : If you KNOW that CO2 warms the planet, and you OBSERVE that the planet is warming, then do you really need more than 95 % confidence that the whole thing is happening by chance ?

And then that patronizing last sentence :

Yes, most scientists are skeptics. We do not accept claims lightly, we expect proof, and we try to understand our subject before we speak publicly and admonish others.

No, Lavine. Scientists do not expect "proof". Proof is for mathematicians and alcoholics. Scientists deal with "evidence".
And Naomi Oreskes explained the significance of the "evidence" a lot better than your article.

Sorry. Just had to get that off my chest.

I'm good now.

Jim Hunt

Rob - I feel I should point out that the "significance" of p-values is a hot topic in statistical circles just at the moment. See for example this Twitter conversation I had with John Kennedy from the UK Met Office recently:

https://twitter.com/jim_hunt/status/711953591678996482

Statisticians issue warning over misuse of p-values whilst muttering "like anyone cares"

Bill Fothergill

Rob,

In an email to Neven I offered the personal opinion that...

As regards the "problems" with what Naomi Oreskes wrote, I suspect it's a case of the statistical purists insisting that the establishment of a significant (or near-significant) p-value for a temperature trend doesn't actually say anything about the causal attribution.

Obviously I could be way off the mark, and there may be something entirely different behind the rather hostile view(s) expressed.

Bill Fothergill

Damn!

I clicked "post" instead of "preview"!

"Scientists do not expect "proof". Proof is for mathematicians and alcoholics."

On that very point, in one of my emails to ASA, I stated that...

By its very nature, science does not deal in permanence. To quote Professors Brian Cox and Jeff Forshaw in their book, "Why does E=MC squared?"...

"In science, there are no universal truths, just views of the world that are yet to be shown false."

However, the climate change debate is something different. We're not seeing subtle, esoteric points getting argued over in learned journals: instead it's ad hominem attacks in the mainstream media, and in the cottage industry blogs that have subsequently arisen.

Rob Dekker

Diablo,
Regarding the spacial filling algorithm that Walsh used, the documentation states :


If the surrounding months do not have data with which to fill the missing value for a given point or points, the field is compared to the ice concentration fields between 1900 and 2000 for the same month. How well a potential analog field matches the gap field is measured by spatial correlation for the sub-domain mapped out by the points that have data in the gap field.

And incidentally I am implementing an algorithm that is very similar.
Not done yet, so stay tuned, but I do expect to show some first results this coming week (time permitting).

Jim Hunt

Diablo and Bill may possibly be interested in the results of my own recent research into suspiciously selective reporting of the results of their research in certain quarters of the cryospheric blogosphere?

The Awful Terrible Horrible Arctic Sea Ice Crisis

As our regular reader(s) will be aware, Anthony Watts has been plagiarising our content and republishing it on his “Watts Up With That” blog. In a perplexing perversity he has also been refusing to publish content that we have happily contributed to the very same blog.

Rob Dekker

Finally had some time for a first implementation of a 'spatial' fill-in algorithm.
This first one is pan-Arctic. Specifically, for a particular month and year, find a 'analog' from the same month, in the 1953-2013 period, that best matches. Best match is defined simply as : the lowest area of grid points that differ in ice extent for the grid points that have an "observational source".

With that simple "pan-Arctic" algorithm, I ran some tests. I started with August 1935. We know that Walsh' newest reconstruction has a problem with the Kelly fields, and we can put that to the test.
Turns out that if we consider the Kelly fields as an "observational source", then August 1935 best matches with August 1961.
The match is not very good (1961 and 1935 still differ by 1.5 M km^2 (where one observed ice and the other did not, or the other way around) but interesting is that the extent of 1935 with Kelly fields (8.98) matches quite closely with 1961 extent of 9.03.

The same test of August 1935, but now disregarding the Kelly fields source, obtains a best fit with August 1953 (which ended up with 8.7 M km^2).

That result suggests that the Kelly fields over-estimate ice-extent in Aug 1935, which we know is the case, because Walsh placed the July Kelly fields in August. So that gives some confidence that we are on the right track.

Incidentally, I also tested Sept 1935, which is a notoriously difficult one to "spatial-fill", since only the Russian side (AARI) has ice-observations. If I ignore Walsh's spatial filling, and instead use this first simple pan-Arctic-match spatial filling algorithm, the algorithm finds August 2007 (of all years) to best match with September 1935's (AARI) ice observations.

That is a curious find, since obviously September 1935 did not remotely come close to 2007's 4.3 M km^2.
I think the problem is that this first, super-simple, algorithm considers ANY difference in ice-concentration (ice or no ice) as a 'difference'. So if in the year under test there is a big ice-flow at pixel X,Y, but in the reference year there is a similar ice flow at X+1,Y, it gets counted as TWICE the difference in extent.

So this simple algorithm is too sensitive to local ice dis-placement, and also it is "pan-Arctic", which means that it does not attach much value to localized ice observations (DMI or ACSYS) but instead attaches much more value to 'field' observations (such as AARI).

Next I have to find some way to 'balance' the different nature of these observations.

Diablobanquisa

Hi Rob,

I´m sorry, but currently I am too busy and I don't have much time.

Great work! Just a couple of comments/suggestions after a quick reading:

- Are you looking for 'analogs' on the 'unadjusted' values for 1953-1978? If that is the case, I think you should use the 'adjusted' ones (in order to avoid 'exporting' the 'concentration bias' and to keep the extent numbers consistent throughout the whole 1935-2013 period).(Alternatively, you could 'adjust' the 'concentration bias' on the 'analog' after finding it)

- Maybe you could test your algorithm using some years with a known result. For instance, you could take any year from the satellite era (or even from 1953-1978), remove everything except the Siberian Sector and look for analogs. And then, check whether the result matches the actual extent.
(That would test September. You could also remove everything except the Siberian sector and some patches at Greenland, Svalbard, Baffin... simulating those August months when DMI and ACSYS direct observations are available. Look for analogs and check the result.)

Cheers


Diablobanquisa

('adjusted values' to match Walsh satellite numbers, since you are using them)

Rob Dekker

OK. I'm sorry for the long delay.
I got struck-down by a nasty virus, which kept me in bed for much of the past 3 weeks.

But now I recovered, I finally implemented my spatial-fill-in algorithm, based on regional matches and pan-Arctic merge algorithm, which should set us free from Walsh' misplaced Kelly fields, and allow an objective algorithm to determine probability of ice in unobserved places.

I wanted any spatial fill-in algorithm to reflect the regional analogs of any observation (ice or not), rather than rely on a single pan-Arctic "climatology".

So this is what I did :
First, for every observation grid point, I determine an "analog" from the 1953-2013 period of the same month. That "analog" for each observation grid point represents the pan-Arctic ice cover that best matches regionally with that observation point.

That obtains thousands of "analog" pan-Arctic maps, one for each observation point of some month in some year before 1953.

Now, to determine the likelihood that ice was present at some unobserved point, for each analog, I check if it had ice present (larger than 15% concentration) and add probability if it does, and subtract probability if it does not, where the probability is weighted by the distance from that unobserved point to the observation point that the "analog" was matched to.

This is a computationally intensive algorithm (it takes some 15 minutes to calculate the ice distribution in some month of some year before 1953), but it obtains fair results : It weights matches of regional observation heavier than far-away observations, which should obtain better results than some single climatology.

And now we can exclude ice observations built-in to the Walsh projections (like the Kelly fields) at will, and play around with all variables.

I first ran this algorithm on August 1935, since that one has nice regional observations (from AARI on the Siberian side, and DMI on the Atlantic side). And excluded the Kelly fields.

So, result of the algorithm using AARI and DMI observations, is that the August 1935 ice extent comes in at 8.65 M km^2.

I believe this new result (8.65) is much more realistic than Walsh' result of 8.977 M km^2 (which we know is based on misplaced Kelly fields).

Next, I need to find a way to create a NetCDF file of this result, so I can show a pan-Arctic Panoply of my results.

Rob Dekker

Diablo, I will address your latest suggestions in a subsequent post.
For the moment, I just wanted to share my initial findings using a regionally based Match-and-Merge spatial fill-in algorithm to replace Kelly fields and other 'not-observed' sources.

Rob Dekker

Diablo, regarding
1) Adjustment for WalshJohnson concentration bias. Yes, I made that adjustment, and it does have some (not much) influence on the 1935-1952 reconstruction efforts.
2) Regarding testing sensitivity to eliminating known observations.
Initial experiments suggest that (as expected) by algorithm is quite good in filling in local gaps (created by artificially removing local observations). However, for large scale removal of observations (such as removing all of the AARI observations) it has a very hard to reconstruct such gaps across the Arctic. I believe that that has more to do with the 'independence' that the various corners of the Arctic show variability than with the integrity of my fill-in algorithm.

Moreover, one experiment I did to test sensitivity to removing observations (by removing ACSYS) described below, revealed an interesting issue with DMI observations as recorded in the Walsh data set. More in the next post.

Rob Dekker

I finally figured out how to write a NetCDF file, so I can view the results of my fill-in algorithm in Panoply (and share it here).

So, here is the results for August 1935, using the Walsh data set for the real ice observations (AARI, DMI and ACSYS), and filling in everything else with my regional Match-and-Merge algorithm (explicitly ignoring Walsh' Kelly fields, which we know are incorrect) :



Click on the image for a larger version.

Compare that image to the original Walsh August 1935 ice concentration map :
http://i1272.photobucket.com/albums/y396/RobDekker/seaice_Aug_1935_zpsh6r5qn9o.png

Note that my algorithm leaves real observations alone (reports the original ice concentration) but for all unobserved grid points, uses
a 'regional' analog from the 1953-2013 August data set. It then replaces the (unobserved) grid point with either 0% or 90% ice concentration, depending on if the merge of the nearby analogs expect to see (15%) ice at that point.

Now, here are a couple of observations :
1) For the Greenland sea and the Barents, August 1935 has ACSYS field observations, and thus (as expected) over these areas my reconstruction leaves the ice concentrations alone in these seas.
2) Over the Russian Arctic, my algorithm appears to determine much of the ice edge, and in fact at some points project more ice than the original Walsh reconstruction. That suggests that AARI observations in August 1935 were mostly of 'ice-free' type, so that the actual ice-edge ended up in 'unobserved' areas. I wonder if that is correct.
3) In Baffin Bay (where there were only a few DMI observations), the ice is sharply reduced w.r.t. the original Walsh Kelly fields. That was to be expected when we ignored the Kelly fields.
What is NOT expected is that this reconstruction still puts ice out beyond where DMI projected an ice edge in Baffin Bay.
ftp://sidads.colorado.edu/DATASETS/NOAA/G02203/1935_08.jpg
DMI clearly projects only a sliver of ice on the west coast of Baffin Bay, and my algorithm should not have projected ice further off the coast.
That suggests that the DMI ice observations in Walsh' data set include only "ice" observations, and NOT the fact that there is "water" right next to it.

My algorithm is very sensitive to things like that, as can be seen in the next experiment, where I ignore ACSYS altogether :
http://i1272.photobucket.com/albums/y396/RobDekker/result-1935-no_ACSYS_zpsoequpr4o.png

Here, you can clearly see that my algorithm projects some extra ice right next to the (DMI observed) ice in the Greenland sea. That would not happen unless the Walsh data set forgot to include at least a few pixels of water next to these DMI ice observations to suggest an ice edge. Instead, they project only the ice, and next to it is "unknown".

Note that due to this flaw in the DMI observations, the Greenland sea ice is a bit bigger in this (no-ACSYS) reconstruction, and that immediately reflects in more ice in Baffin Bay, and even the regional analogs suggest some extra ice as far as Hudson Bay.

This means that in order to do a correct reconstruction of the 1935-1952 time frame, I now need to manually go in an correct the DMI observations to reflect an "ice-edge" rather than just "ice" in some places.

It is this kind of issues that make reconstruction of past ice extent quite a hairy exercise....

Rob Dekker

Sorry. This : "Over the Russian Arctic, my algorithm appears to determine much of the ice edge, and in fact at some points project more ice than the original Walsh reconstruction. "
should read as :
"Over the Russian Arctic, my algorithm appears to determine much of the ice edge, and in fact at some points project LESS ice than the original Walsh reconstruction. "
In fact, these are quite a few areas where my method shows more water in the Russian Arctic in August 1935 than the Walsh reconstruction.

Diablobanquisa

Excellent, Rob, thank you for your work.
I'm sorry for my silence and long delays, but presently I'm too busy.
When I have more time for this (I don't know when...) I will read again and more carefully everything you have posted since April 19. I will have to think about it, and I guess I'll come with some comments, questions, and suggestions.
Cheers

PS: a couple of weeks ago I sent an email to Florence Fetterer in order to let her know the 'misplaced Kelly fields' issue. She answered me that they will look into it. I also mentioned the well known 'consistency issue' between pre-satellite and satellite data and she told me that she hopes they'll be able to address that with a new version over the coming year.

Rob Dekker

Hi Diablo,
Good to see you back, and don't worry.

I'm just kicking the tires of my new toy over here.

I think your note to Florence Fetterer is much more important, since that actually may make a difference in the official ice-extent history by Walsh et al.

After all, whatever I'm doing here now is just a different way of doing spatial/temporal filling of unobserved data points, which is by definition inaccurate.

Rob Dekker

I would like to present two different reconstructions that highlight the strength and weakness of my spatial fill-in algorithm.

The first one is August 1952.
This is a month where there is ample observation of the ice edge, and it results in a realistic reconstruction of the unobserved data points by my Match-and-Merge fill-in algorithm.

Here is the reconstruction that my algorithm came up with :


Note that this follows the ample AARI observations of the ice edge along the Russian Arctic, and follows ACSYS observations along the Barents, then nicely projects an ice edge along the Greenland Sea, and follows the DMI observations of ice along the South of Greenland.

The interesting projections are the ice in the center Baffin Bay, which looks realistic given the DMI ice observations there, and the ice that my algorithm projects in Fox Basin seems consistent with the general state of the ice observations from the area according to DMI's August 1952 chart :
ftp://sidads.colorado.edu/DATASETS/NOAA/G02203/1952_08.jpg

Also, the open water in the eastern Beaufort that my algorithm projects is consistent with the surrounding ice conditions in the Chukchi and Baffin Bay.

It all seems realistic, and at least an order of magnitude better than the vague ice projections in the original Walsh dataset :
http://i1272.photobucket.com/albums/y396/RobDekker/Screen%20Shot%202016-03-13%20at%2012.25.42%20AM_zpshfyb9zau.png

I believe that this is the most realistic August 1952 ice extent reconstruction to date. Better than Walsh' dataset, and better than a reconstruction with a "climatology" backdrop, like Diablo's.

Note that ice extent is 8.15 M km^2.

Rob Dekker

On the other hand, here is an example where my algorithm goes wrong.

Reconstructing September 1935 is a challenge by itself, since we ONLY have AARI ice observations. No observations from the other side of the Arctic.

And even with these AARI observations, there are some issues.
For example, AARI observations around the new Siberian islands consist of ONLY open water observations, quite close to the coast, but over a wide longitudinal area.

Diablo has a nice animated GIF where the September AARI observations show mostly open water :
https://diablobanquisa.files.wordpress.com/2016/03/output_3ax4kh.gif

All these "open water" data points along the Russian Arctic coast affect my algorithm, since they best match with .... year 2007.

The small patch of ice in the Chukchi does not prevent the overwhelming bias towards a "2007-like" Arctic state, which reflects in my algorithm projecting water across half the Arctic :
http://i1272.photobucket.com/albums/y396/RobDekker/result-9-1935_zpskwslzbyb.png

and resulting in a 2007-like 4.4 M km^2 estimate for Sept 1935.

That is obviously incorrect, especially since we just determined that (based on much more observations) Aug 1935 is in the 5.3-5.5 M km^2 area.

So this is a flaw in my algorithm.
It appears that my algorithm works best if there is a good mix of ice observations and water-observations in some area, and particularly good if there are a lot of ice-edge observations.

But if, over a large area, there are only "ice" observations, or only "water" observations, then my algorithm finds analogs that tend to amplify that.

Simply eliminating 2007 as a reference year alleviates much of the problem, but still leaves quite unrealistic swats of open water north of the open-water observations from AARI.

I expect that the problem would only be truely solved by adding more diverse (water and ice) observation points, possibly via temporal borrowing from neighboring months. Something that is not difficult to do in my algorithm.

So, more work ahead..

Rob Dekker

Correction: Aug 1935 is in the 8.3-8.5 M km^2 area.

Susan Anderson

Rob Dekker, that is some amazing work! Thank you.

Rob Dekker

Thanks Susan,
And I'm having lots of fun with this exercise.

Expect more to follow as I work out the sensitivities (such as that "open water" problem I describe above) in my algorithm, and work towards a "temporal" fill-in (using ice observations from prior or subsequent months to better estimate ice extent).
After that, I will publish my best estimate of ice-extent for August and September for all the years in the entire 1935-1952 period, and possibly before that too.

Diablo,
I discovered two additional issues with the Walsh data set :

1) DMI observations in the Walsh data set do not include information about the 'ice-edge' (the red lines next to the ice observations in the DMI graphs) :
ftp://sidads.colorado.edu/DATASETS/NOAA/G02203/1935_08.jpg
Such ice edge observations should be easy to include in a future Walsh data set by modeling at least a couple of "open water" pixels next to the ice.

2) I found a few "bad apples" in the post-1952 data set :
Here is August 1971 :

Obviously something went wrong there in the ice concentration estimate for the ACSYS source.

Similar problem in the August 1967 map, and there is a curious little "hole" in the central Arctic ice pack in the 1953 map.

Could you still send this info to Florence Fetterer for correction in a future version of the data set ?

Diablobanquisa
Could you still send this info to Florence Fetterer for correction in a future version of the data set ?

Hi Rob, yes, I will let her know.


In the meantime, Walsh et al. have published a paper presenting their dataset: http://onlinelibrary.wiley.com/doi/10.1111/j.1931-0846.2016.12195.x/abstract

As we already knew, the data and the documentation are available here: https://nsidc.org/data/g10010

Neven

I'll be doing a piece on the Walsh et al. paper around this year's minimum.

Jim Williams

"I'll be doing a piece on the Walsh et al. paper around this year's minimum." -- Neven

Sometime in October?

Rob Dekker

Neven said

I'll be doing a piece on the Walsh et al. paper around this year's minimum.

That's great, Neven, thank you !
Remember that in the discussion with Diablo, we found essentially three issues with the otherwise fine Walsh reconstruction:

1) The Kelly fields seemed to be misplaced by one month (August shows July Kelly fields, and July shows June Kelly fields). This results in a significant high-bias for the 1935-1952 reconstruction.
2) The Walsh & Johnson source has a high bias, which results in a high-bias of the 1953-1979 reconstruction.
3) The DMI source in Walsh reconstruction does not account for the "ice edge" that is depicted as a red lines in the DMI charts. This results in a small high-bias caused by ice fill-in of neighboring pixels that really should be open water.

Issue (3) was never resolved, but it is most likely not significant.

Issue (2) was quite easily resolved by adjusting for the ice concentration of the Walsh & Johnson source.

Issue (1) is much more difficult, since when we remove the Kelly fields, we need to create an alternative "fill-in" algorithm, which is what I worked on (results posted above) in May.

I left that work at the end of May, when I introduced temporal fill-in into my algorithm. That resulted into reconstructions for August and September, but unfortunately the difference between August and September for many of the years was unrealistically large (more than 1 million km^2 for many of the years in the 1935-1952 period).
I suspect that the "open water" observation sensitivity that I reported about earlier is playing a role here, but the 2016 melting season got in the way, but I did not investigate the issue further.

Now that you plan to post about the Walsh paper, it is time to pick up that work where I left it in May :o)

Rob Dekker

Well, it took a while, but here are the results of the Walsh series adjusted for the known issues :
- Kelly fields removed as a source, since they were clearly misplaced one month (Walsh had July Kelly fields in the August result).
This affects mostly the 1900-1953 period where Kelly fields are present.
- Walsh and Johnson source adjusted for high-bias (set a 25% concentration cut-off for extent instead of the usual 15%).
Issue discussed earlier in this thread.

I also replaced the Walsh spatial/temporal fill-in algorithm with my own regional match-and-merge spatial/temporal fill-in algorithm, using the 1953-2000 period for analogs. My algorithm uses data from August AND from July to do both spacial as well as temporal fill-in.

Here is the result for August, over the full 1850-2013 period of the series :

Couple of initial notes :
In general, my series shows less ice over the pre-1953 period than the original Walsh series. That was to be expected since the Kelly fields were misplaced by one month in the original Walsh series.

What is interesting is that the overall shape of the series is similar to Walsh's, with a 'dip' during the 40's, which matches the the temperature record over that period.

Also, to come full circle in this thread :
From 1935- early 70's the trend is positive, which is what Diablo also found in his reconstruction. Something I was arguing against hard, but now I see it clearly in my own data. Sorry I was fighting that for so long :o)

Finally, I am not quite ready to publish the September results. There is an 'open-water' bias in my algorithm which requires more validation work before I can publish it.

Rob Dekker

P.S. Click the image for a larger view.
Also, note that my algorithm is cpu intensive and thus it took 25 hours to produce this full series starting in 1850...

P-maker

Rob,
Amazing what you and others have achieved through an open and friendly dialogue in this thread.

To me it will be the "Dekker et al., 2016" series until further evidence appears.

Diablobanquisa

Great work, Rob! And I´m glad to see that finally your results are consistent with ours.

viddaloo
From 1935- early 70's the trend is positive, which is what Diablo also found in his reconstruction.

Splendid work, Rob! BTW, could the explanation for the positive pre-70s trend be increased snowfall in a slightly warmer Arctic, like we saw with Norwegian glaciers in the 1990s?

With only a little bit of global warming, the local glaciers in Hardanger and elsewhere grew and expanded in their outlets / arms, as temps were still below freezing but more moisture gave us more snow to add annually to the mountain ice caps.

Add more heat in the new century, and they're all in decline.

Rob Dekker

Thanks guys.
Viddaloo, the temperature record shows a decline over the Arctic from the 30's to the 70's which matches with the increase in ice over that period :
https://tamino.files.wordpress.com/2011/09/ghcnall.jpg

In fact, my main argument with Diablo's analysis was that he choose a climatology that implicitly assumes a direct relation between temperatures and ice extent (by choosing a climatology).

But my August reconstruction now now shows this relationship is there even based on pure observations of ice.

I'm not quite done yet though. Next, I'd like to investigate on which side of the Arctic the 'dip' (30's-70s) in ice extent originates, or if it was pan-Arctic.

Rob Dekker

In the graph below, I added the regional ice extent over the 1850-2013 period by sector :

- Russian sector from the Kara to Bering Strait
- American sector from the Bering Strait to Baffin Bay
- Atlantic sector from the Greenland sea and the Barents

Click the image for a larger version.

Noteworthy is that the 'dip' in ice extent from the 30's to the early 70's is apparent in all three sectors. So it seems that this dip (and the subsequent hill in the 70's) is pan-Arctic, although it seems most pronounced in the American sector.

It seems that Arctic sea ice follows the temperature over that 30's to 70's period, just like Diablo concluded from his work.
Although it is interesting that pre-1930, ice extent remains rather flat, although Arctic temperatures were certainly on the rise there.
https://tamino.files.wordpress.com/2011/09/ghcnall.jpg

Also note the irregularities around 1970, especially for the Atlantic sector. One reason for these spikes is that "bad apple" in August 1972 that I reported about a few comments back, but it is not the only spike.

Note that the pre-1953 reconstruction heavily relies on my Match-and-Merge algorithm, while the post-1953 reconstruction relies on the original Walsh sources, including the Walsh&Johnson source that we already had to adjust for concentration bias.

To be totally fair, the last reconstruction I will make will be one where I ignore the Walsh&Johnson source in the 1953-1979 period, and let my Match-and-Merge algorithm do the work. That would be a fair comparison between pre- and post-1953 reconstruction, and I'm curious how pronounced the 'dip' remains with that.

And after that I really will wait for Neven's post about the Walsh reconstruction, since this thread has become way too long already a long time ago :o)

Diablobanquisa

Great regional analysis, Rob!


Although it is interesting that pre-1930, ice extent remains rather flat, although Arctic temperatures were certainly on the rise there.
https://tamino.files.wordpress.com/2011/09/ghcnall.jpg

From 1880 to 1920 the temperature looks also rather flat. It seems to me that the significant warming started around 1920. The ice then should be older and thicker, and maybe the effects of that warming didn't become noticeable on late summer extent numbers until the 30s.

Anyway, I also think that the August extent numbers before the 30s are less reliable. Before mid 30s, AARI charts aren't available anymore, and the only direct observations are those from ACSYS and DMI charts, so the reconstruction relies more heavily on your match and merge algorithm.


To be totally fair, the last reconstruction I will make will be one where I ignore the Walsh&Johnson source in the 1953-1979 period, and let my Match-and-Merge algorithm do the work. That would be a fair comparison between pre- and post-1953 reconstruction, and I'm curious how pronounced the 'dip' remains with that.

I have some concerns about this, since I think that you have used the 1953-2000 period for analogs. So, your algorithm could have imported to 1935-1952 some data from 1953-1978 that now you are planning to exclude and replace with analogs from 1979-2000.
Maybe, to be totally fair, I think that you could use the 1979-2000 period for analogs for the whole 1850-1978 period. Although your cpu would need at least 25 hours more of work... ;-)

Either way, the results will be interesting, of course.

Cheers

Rob Dekker

Thanks Diablo,
The purpose of running without Walsh&Johnson source over the 1953-1978 period is to avoid any 'bias' in my match-and-merge algorithm when comparing to the period before 1953.
The reconstruction is running as we speak, and I will report on it tomorrow.

Yes, it is possible that there will be "cross-contamination" from that, but remember that my match-and-merge algorithm uses analogs only if they 'fit' locally. So it should be fairly insensitive to exactly which analogs we feed them as long as there is some analog that matches locally. My concern with reducing the analogs to the post 1979 period is that even the best analogs will underestimate the pan-Arctic ice extent because after 1979 ice extent was lower than in some prior periods. I will show you a pre-1900 reconstruction tomorrow to show what I mean. Maybe I should actually extend the 'analog' set by including July maps into the August assessment, just so we have some analogs that match with the higher extent in the pre-1930 reconstructions..

Yet, varying the analog period is good advice, if only to see how sensitive (or not) it is. I will run some experiments to see how that affects the reconstruction.

Which kind of brings us to the final frontier : How can we get some "uncertainty" margin into these reconstructions ? I wish I paid more attention during "statistics" class in college...

Rob Dekker

I ran the 1953 to 1978 period excluding the Walsh&Johnson source, instead letting my match-and-merge algorithm fill in the missing data from the remaining sources (Navo yearbooks etc) and the result is very encouraging :
The reconstruction still shows the 'hill' around the 70's that we have become accustomed to, but the sharp peaks and valleys in the 70's are less pronounced. Overall, for averages, not much of a change with the graph I showed above with the bias-adjusted Walsh&Johnson source.

That is really good news, since it tells my algorithm is doing realistic fill-ins even when taking out a dominant 'field' source like Walsh&Johnson.

Unfortunately, there are some known issues with Photobucket today
which means I can't show you the graph.
Maybe it is time to start a 'forum' topic on this work, or, Neven, will you still be posting a new ASIB post on the Walsh historic sea ice reconstruction ?

Neven
Maybe it is time to start a 'forum' topic on this work, or, Neven, will you still be posting a new ASIB post on the Walsh historic sea ice reconstruction ?

I will be publishing a blog post this weekend. Sorry for taking so long.

Rob Dekker

Photobucket is back up, so here is my latest 1850-2013 reconstruction, which omits the Walsh&Johnson source (over the 1953-1978 period) and replaces it by fill-in using my match-and-merge algorithm. This reconstruction was done to avoid any bias that my match-and-merge algorithm may have.

As you can see, the match-and-merge algorithm did not substantially change the reconstruction, and if anything, made it more plausible, since some of the extreme spikes during the 70's are now mellowed out and more consistent with Diablo's and Meier et al.

Diablobanquisa

Great work, Rob! It seems that your algorithm works fine.

In the meantime, I have noticed that the new HadISST2 Sea Ice Concentration dataset is available for download: http://www.metoffice.gov.uk/hadobs/hadisst2/

In 2014, Titchner and Rayner published a paper presenting the Sea Ice Concentration component of the new HadISST2 dataset:

Titchner, H. A., and N. A. Rayner (2014), The Met Office Hadley Centre sea ice and sea surface temperature data set, version 2: 1. Sea ice concentrations, J. Geophys. Res. Atmos., 119, 2864-2889, doi: 10.1002/2013JD020316.
http://www.metoffice.gov.uk/hadobs/hadisst2/Titchner_and_Rayner2014.pdf

However, until recently the gridded data haven't became publicly available.

Titchner and Rayner relied mostly on the old Walsh dataset, and they didn't incorporate AARI data.

Titchner and Rayner have performed a sort of 'reverse adjustment'. That is, they have adjusted the satellite data to 'match' with the pre-1979 values. However, their results look rather strange. The graph below compares HadISST2 extent numbers for September Arctic SIE (blue line) against those from Walsh et al. 2016 (black line) and our paper (red line) (click for a larger version):


Even if we adjust back the whole timeseries to match with the satellite data since 1979, HadISST2 still looks a bit strange (click for a larger version):


I'd dare to say that, after the publication of Walsh et al. 2016 (and our paper, and Rob's findings), the Arctic SIC data of HadISST2 is already obsolete and needs an update.


Rob Dekker

Thanks Diablo for posting the HadISST2 data.
Indeed that series seems already in need of an update.

At least they could update with the new Walsh data, but considering that we found some significant issues with that as well (most notably the Kelly fields issue and the Walsh&Johnson high bias) they may want to wait until Walsh updates his reconstruction.

Meanwhile, I'm running (as we speak) the September numbers through for the whole series. Then we can finally compare Meier et al, your numbers, Walsh reconstruction and my numbers and compare them.
That would be a nice finale to this very long thread :o)

Susan Anderson

That is great work, thanks. Picture = 1000 words.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)