REOS Metrics
- Recommendations from CLIVAR basin panels
- Integrated Mass Transports - Observational Estimates
- Other activities related to defining ocean model metrics
Defining a common set of metrics to evaluate ocean model simulations depends on the purpose of the ocean model and the scientific questions it is being used to address. The challenge is to try to establish a consistent evaluation process for ocean models. It is hoped that the discussion that this webpage will foster will ultimatly lead to the development of a set of community-wide approved metrics, endorsed by WGOMD, that will provide a quality standard for ocean model performance that the community should endeavour to meet.
Recommendations from CLIVAR basin panels
The CLIVAR basin panels have been asked to provide WGOMD with input on ocean metrics to evaluate model performance in terms of ocean processes occurring in the different ocean basins.
 |
 |
 |
 |
Integrated Mass Transports - Observational Estimates
There are a number of climatologically important straits, throughflows and current systems whose integrated mass transport is measured observationally (though some have wide uncertainties). These mass transports provide a useful means of characterizing an ocean model simulation. The following recommendations are made regarding the diagnosis of model integrated mass transports:
- The following table notes the approximate geographical location of the straits and currents. Given considerations for model grid resolution and grid orientation, precise values for the coordinates may differ between models. In general, it is recommended that the simulated transport be computed where the strait is narrowest and shallowest in the model configuration, and where the model grid is closely aligned with the section.
- For most ocean model grids, the requested transports can be diagnosed by aligning the section along a model grid axis. In this case, it is straightforward to assign a positive sign to transports going in a pseudo-north or pseudo-east direction, and negative signs for the opposite direction. We use the term pseudo here as it refers to an orientation according to the model grid lines, which in general may not agree with geographical longitude and latitude lines.
- Some models may have a strait artificially closed, due to inadequate grid resolution. In this case, a zero transport should be recorded.
- References to observational estimates of the mass transports are provided in the table below. These references can be updated as new estimates become available. See References page for full citations and links to the articles where possible.
Table: Georgaphical locations of key sections and currents and observational estimates of integrated mass transport
|
Barents Opening |
16.8°E,76.5°N - 19.3°E, 70.2°N |
Randi et al. (2004): (a) Atlantic (saline) water. Annual mean: 1.5 Sv, seasonal variation from 1.3 to 1.7 Sv, (b) Coastal (fresh) water. 0.5-1.0 Sv. northward All volume transports are for poleward flow of Atlantic Water Smedsrud et al. (2010): 2.0 Sv eastwards (1 Sv=1°—106 m3/s). This net flow is derived from four sources; the inflowing Norwegian Coastal Current (NCC) at the southern coast, the Atlantic inflow in the central area, and two westward flows in the northern areas |
(see also Smedsrud et al. (2010)) |
|
Bering Strait |
171°W, 66.2°N - 166°W,65°N |
0.8 Sv northward |
|
|
Canadian Archipelago |
128.2°W, 70.6°N - 59.3°W, 82.1°N |
0.7-2.0 Sv southward |
Sadler (1976), Fissel et al. (1998), Melling (2000) |
|
Denmark Strait |
37°W,66.1°N - 22.5°W, 66°N |
0.8 Sv net northward, 3 Sv net southward |
|
|
Drake Passage |
68°W, 54°S - 60°W, 64.7°S |
135 Sv eastward |
|
|
English Channel |
1.5°E, 51.1°N - 1.7°E, 51°N |
0.1-0.2 Sv northward |
|
|
Pacific Equatorial Undercurrent |
155°W, 3°S - 155°W, 3°N (0-350m) |
24-36 Sv eastward |
|
|
Faroe-Scotland Channel |
6.9°W, 62°N - 5°W, 58.7°N |
3.8 Sv net northward, 2.1 Sv net southward |
|
|
Florida-Bahamas Strait |
78.5°W, 26°N - 80.5°W, 27°N |
29-35 Sv northward |
|
|
Fram Strait |
11.5°W, 81.3°N - 10.5°E, 79.6°N |
4 ±2 Sv southward |
|
|
Iceland-Faroe Channel |
13.6°W, 64.9°N - 7.4°W, 62.2°N |
3.8 Sv net northward, 1 Sv net southward |
|
|
Indonesian Throughflow |
100°E, 6°S - 140°E, 6°S |
15 Sv westward |
|
|
Mozambique Channel |
39°E, 16°S - 45°E, 18°S |
Riddernickhof et al., 2010: 16.7 Sv poleward (Seasonal variation: 4.1 Sv, Interannual variation: 8.9 Sv DiMarco et al., 2002: Mean southward transport with considerable seasonal variability. Previous estimates range from 5Sv northward to 26Sv southward. Stramma et al., 1997: 5Sv |
|
|
Luzon Strait |
Inflow to South China Sea |
Qu et al., 2009: The total transport through the Luzon Strait is essentially westward. Prior studies have arrived at a broad range of this transport estimate, varying from 0.5 to 10 Sv, while recent observations favor a value near the middle of the range. Tian et al., 2006: 6+-3 Sv (Tian et al, 2006) Qu, 2000: Annual Mean 3.0 Sv (westward) Chu an Li (2000): Annual Mean 6.5 Sv (westward) |
Chu an Li (2000) See also Yaremchuk et al. (2009) |
|
Taiwan Strait |
Inflow to South China Sea |
1.6 Sv (during the period 1999-2001) |
|
|
Windward Passage |
75°W, 20.2°N - 72.6°W, 19.7°N |
Smith et al, 2007: The transport entering Windward Passage is highly variable, including reversals to net outflow. Transports measured during the cruises ranged from -0.3 Sv (outflow) to 9.4 Sv (inflow), with an average inflow of 3.8 Sv. Corresponding transports derived from the current meter array range from approximately -5 to 15 Sv, with an average inflow of 3.6 Sv. On average there is net inflow in the surface and thermocline layers (above ~600 m), net outflow in the intermediate layer (~700-1200 m), and a deep inflow just above the bottom. Bulgakov et al., 2003: A compilation of earlier results Johns et al., 1999: Measurements made just to the northeast in the passage between Great Inagua Island, the Bahamas and Haiti, assuming that flow through this passage was a good approximation to flow through the Windward Passage. The estimate was made using directly observed currents in the upper 200 m, the mean transport from 9 crossings of the section: 2 Sv, stdev 1.5 Sv (westward) Kinder et al., 1985: 10 Sv (westward) Roemmich, 1981: 7Sv (westward) |
Smith et al. (2007, see also 2008) |
Other activities related to defining ocean model metrics
- Workshop on Ocean Model Metrics (25 February 2006, Hawaii, USA)
- CLIVAR GSOP Synthesis Evaluation Metrics
- MERSEA Integrated Project Metrics
Workshop on Ocean Model Metrics (25 February 2006, Hawaii, USA)
This one day workshop was held to discuss the current state of ocean model metrics and how the ocean modelling community, working together with the observations community can improve how ocean models are evaluated. Ocean models are being run at higher resolutions that should be capable of resolving not only the climatology but also high frequency and mesoscale variability. This motivates the development of metrics that help evaluate and understand the variability at the space and timescales resolved by the models, as well as the representation of physical processes.
Since the definition of metrics depends on the purpose for which the model is being used, the workshop divided the ocean modelling community into three groups: climate modelling (see here for the recommendations), operational modelling, and modelling for process orientated studies.
A key recommendation is that the design of process orientated experiments must be designed from the outset with model evaluation and comparison as a goal. Models and observations must both be components for the study to be successful. The workshop participants also recommended that a website be developped for the discussion of ocean model metrics, to facilitate access to observational datasets, and to feature a comparison of metrics derived from ocean models and from observations.
- Workshop recommendations for the evaluation of ocean models used in climate studies.
- Summary report from the workshop
- Webpage
CLIVAR GSOP Synthesis Evaluation Metrics
Intercomparison quantities proposed by the CLIVAR Global Synthesis and Observations Panel (GSOP). These are part of the CLIVAR/GODAE Global Synthesis Evaluation Framework. A draft version can be downloaded here (Version 3.1, 2006).
1. RMS model-data comparison
- Difference from WOA01 climatological (monthly, Jan.-Dec.) potential T & S
- RMS misfit from Reynolds SST
- RMS misfit from in-situ T & S profiles (including XBT, CTD, Argo, moorings)
- RMS misfit from altimeter-derived SSH
- RMS misfit from tide-gauge SSH
2. Meridional transports (discussion lead: A. Koehl)
- Timeseries of meridional MOC of the global ocean, Atlantic and Indo-Pacific as a function of latitude and depth and for the global ocean as a function of latitude and potential density.
- Timeseries of meridional heat and freshwater transports of the global ocean, Atlantic, and Indo-Pacific as a function of latitude.
- Time series of maximum MOC strength and heat transport at 25N, 48N in North Atlantic
3. Change in sea level, heat and salt contents (discussion leads: M. Balmaseda and A. Weaver)
- Monthly means of averaged temperature (proxy to heat content) and salinty over the upper 300m/750m and 3000m.
- Time series for spatial averages within a list of 30 pre-defined boxes in various parts of the ocean.
- Monthly means of sea level, and optionally steric height and/or bottom pressure.
- Time series for spatial averages within a list of 30 pre-defined boxes in various parts of the ocean.
4. Transport through key region (dsicussion lead: T. Lee)
- Indonesian throughflow volume transport
- ACC volume transport through the Drake passage.
- Florida Strait volume transport, temperature flux, and salinity flux.
5. Water mass characteristics (discussion leads: K. Haines and T. Lee)
- 18-C water volume in the N Atlantic Ocean, volumne-weighted average salinity of the 18C water as a function of month.
- Annual Maximum mixed layer depth within the Labrador sea and the T,S properties of that mixed layer.
- Warm-water volume in the equatorial Pacific (5S-5N, 120E-80W) AND tropical Pacific (20S-20N, 120E-80W),
- Depth of 20 degree isotherm in Pacific Ocean as a function of longitude, latitude, and month.
6. Indices (discussion lead: A. Fischer)
Sea surface temperature anomaly indices averaged over lat-lon boxes in the ocean:
- Pacific: Nino1+2; Nino3; Nino3.4; Nino4
- Indian: SETIO; WTIO
- N. Atlantic: Curry and McCartney transport index.
7. Surface Fluxes (discussion lead: L. Yu)
- Monthly means of net surface heat and freshwater flux as function of geographic location.
- Time mean of net surface heat flux and freshwater flux over entire model domain.
- Zonal averages of annual mean net surface heat flux and freshwater flux over the model domain.
MERSEA Integrated Project Metrics
A European system for operational monitoring and forecasting of the ocean physics, biogeochemistry, and ecosystems, on global and regional scales is being developed by the Marine Environment and Security for the European Area (MERSEA 2004-2008) Integrated Project. The system will be a key component of the Ocean and Marine services element of GMES (Global Monitoring for Environment and Security).
MERSEA Integrated Project: WP5 - Integrated System Design and Assessment Report, 2006 (Ref: MERSEA-WP05-MERCA-STR-0015-01.doc)
This document focuses on the diagnostic definitions of Class 1, 2, 3 and 4 metrics for validating and intercomparing ocean forecasting systems and their products in real time. The report contains the complete list of variables, including their netcdf attributes, as well as coordinates for the Class 2 sections following the tracks of the main WOCE and CLIVAR repeat sections and XBT sections and the Class 3 volume transport sections.
|
Class 1 |
2D and 3D daily averaged fields interpolated onto a common set of GODAE horizontal and vertical grids. Used as 'instantaneous' esitmates of the ocean mesoscale circulation for direct comparison with observed quantities. |
|
Class 2 |
Daily mean values at chosen section tracks or mooring locations. |
|
Class 3 |
Integrated quantities derived from Class 1 and 2 variables, (daily volume transports through chosen sections, overturning stream function and meridional heat transport per ocean basin). |
|
Class 4 |
Metrics to measure the performance of the forecasting system, limited to 'observational space' rather than 'model space'. Observations, preferably not assimilated by the forecasting system, are compared to the forecast and hindcast for a given day. |
Diagnostics Table
Download here a table compiled from the MERSEA report that summarises the list of diagnostics planned for an intercomparison between GODAE partners to verify 1) the consistency of the ocean dynamics and variables using the "best estimate" or "hindcast products, and 2) the quality, or precision, of the operational daily estimates of the ocean and sea ice circulation and dynamics.
