Intercomparison-metrics working group
Ocean forecasting systems differ in many ways. Various intercomparison efforts of GODAE regional products have been undertaken or are in progress in order to quantitatively evaluate methodologies to judge the strengths and weaknesses of the forecasting systems. This requires collaboration between modelers to agree upon the definition of internal metrics which will be systematically used for assessing the quality and consistency of system outputs and the sharing of experience through critical analysis of inter-comparison exercises. The establishment of close interactions with the user communities will also be critical to define the most relevant ocean products to be evaluated.
The European Mersea Project in collaboration with other international GODAE projects have built a common methodology for real-time intercomparison of forecast systems. Internal metrics i.e. a mathematical definition of chosen diagnostics are defined and aim at testing the consistency, quality and performance of each system. The MERSEA Project has developed metrics and organised them into four classes (Class 1 to 4).
- the definition of metrics (Mersea Strand 1)
- the coherency and standardisation of products and model output
The metrics defined for Mersea Strand 1 fit the North Atlantic Ocean and Mediterranean basin for the existing basin-wide European operational system. They can be easily extrapolated and used as an example to develop further metrics for any other operational system oriented towards bio-geochemical, coastal, polar, global, specific basin issues within the MERSEA-IP and GODAE frame. Metrics are now being defined for polar areas as well as for the North Pacific Ocean.
(Credit to http://strand1.mersea.eu.org/html/strand1/intercomparison.html ; MERSEA Strand 1 is the predecessor of the Integrated Project (MERSEA IP, 2004-2008) that will use MERSEA Strand 1 results to build the ocean component of GMES (Global Monitoring for Environment and Security) by 2008.)
From these inter-comparisons, the performances (quality and timeliness) of the integrated systems (observing systems, data and product services, modeling and assimilation and product distribution) will be assessed and recommendations for improvements will be proposed. Inter-comparison activities:
- GODAE Intercomparison Workshop, 25-26 February 2004
- CLIVAR/GODAE Global Synthesis Evaluation Framework
- GODAE Intercomparison/metrics meeting, Hobart, Australia, 15/16 March 2007 (zip-file)
During IGST-XII it has been agreed to perform a demonstration of intercomparison among GODAE partners:
The main objectives are
i) to demonstrate operational ocean system in action;
ii) to assess the quality of hindcast ocean products;
iii) to perform the intercomparison among the different operational centers;
iv) to promote validation and intercomparison as part of future "ensemble multi model forecast" applications.
It has been decided to schedule the intercomparison exercise, in real time, during the Feb-April 2008, 3-months period. The assessment and intercomparisons rely on Class 1 and Class 2 metrics that are made available by each partner.
For the coordination of this exercise a intercomparison working webpages has been set up and can be accessed when logged into the page.
(Last Updated: 22-07-2008)