Calvalus

Portal for Earth Observation Cal/Val and User Services

Enter

The ESA Calvalus Study

Objective

Product development cycle ESA's Earth Observation (EO) missions provide a unique dataset of observational data of our environment. Calibration of the measured signal and validation of the derived products is an extremely important task for efficient exploitation of EO data and the basis for reliable scientific conclusions. In spite of this importance, the cal/val work is often hindered by insufficient means to access data, time consuming work to identify suitable in-situ data matching the EO data, incompatible software and limited possibilities for rapid prototyping and testing of ideas. In view of the future fleet of satellites and the fast-growing amount of data produced, a very efficient technological backbone is required to maintain the ability of ensuring data quality and algorithm performance.

The announcement of opportunities EO Cal/Val and User Services is a technology study of the ESA LET-SME 2009 call, investigating into an existing leading edge technology (LET) for their applicability in the EO domain. Specifically, LET-SME is a spin-in instrument encouraging the participation of SMEs to ESA technology. The LET-SME focuses on early stage development of Leading Edge Technologies, i.e. the ones likely to become the reference technologies for the near future, and have good chances of being infused into ESA projects and missions.

In accordance with the study's statement of work, Calvalus is a system that has been proposed to fully support the idea of LET-SME, thus with a strong focus on a selected LET which is described in this report.

Approach

The cluster Brockmann Consult GmbH proposed to develop a demonstration processing system based the MapReduce programming model (MR) combined with a Distributed File System (DSF). The basic approach was first published in 2004 by the two Google computer scientists J. Dean and S. Ghemawat (MapReduce: Simplified Data Processing on Large Clusters). The technology has been designed for processing of ultra large amounts of data and is based on massive parallelisation of tasks combined with a distributed file system, both running on large, extensible clusters solely comprising commodity hardware. All nodes in the cluster are equally configured and provide both disk storage and CPU power. Well known online services provided by Google, Yahoo, Amazon and Facebook rely on this technology. Its spin-in application to space born, spatial data is feasible and pertinent. Map reduce
The results of this study demonstrate that the processing of large amounts of EO data using MR and a DSF is efficient and advantageous. The demonstration system, Calvalus, basically comprises a cluster of 20 commodity computers with a total disk capacity of 112 TB at a total cost of 30 k€. The processing system software is based on Apache Hadoop, an open-source implementation of MR and DSF in Java and SNAP, an ESA Earth Observation Toolbox and Development Platform.

Calvalus gains its performance from massive parallelisation of tasks and the data-local execution of code. Usual processing clusters or grids first copy input data from storage nodes to compute nodes, thereby introducing I/O overheads and network transfer bottlenecks. In Calvalus, processing code is executed on cluster nodes where the input data are stored. Executable code can be easily deployed; the code distribution and installation on all cluster nodes is done automatically. Multiple versions of processing code can be used in parallel. All these properties of the Calvalus system allow users to efficiently perform cal/val and EO data processing functions on whole mission datasets, thus allowing an agile EO data product development and fast improvement cycles.

The different production scenarios and analyses implemented in Calvalus are inspired by the needs of the current ESA projects, such as CoastColour and Climate Change Initiative (CCI) for Land Cover and Ocean Colour, both of strong interest to an international user community. The implementation is focused on the Ocean Colour data processing and validation:

L2-Production: Processing of Level-1b radiance products to Level-2 ocean reflectances and inherent optical property (IOP) products.
L3-Production: Processing of Level-2 products to spatially and temporally aggregated Level-3 products.
Match-up analysis: Generation of match-ups of Level-2 products with in-situ data.
Trend analysis: Generation of time-series of Level-3 products generated from Level-2 data.

The Level-2 products in production scenarios 2 to 3 are generated on-the-fly from Level-1b using selected Level-2 processors and their required versions, processing parameters and LUTs. The Calvalus demonstration system currently holds the full mission Envisat MERIS Level-1b RR data as well as all MERIS Level-1b FR product subsets required by the CoastColour project.

Calvalus provides an easily operated web interface that allows users to order and monitor productions according the to four production scenarios, and to finally download the results. It also offers a Java production API, allowing developers to write new production scenarios.

Project

The Calvalus study has been performed in the time from September 2009 to October 2011. The Calvalus system is now being actively further developed.

The Calvalus team is

Dr Martin Böttcher, Brockmann Consult GmbH - Developer
Olga Faber, Brockmann Consult GmbH - Tester
Norman Fomferra, Brockmann Consult GmbH - Project manager / Developer
Dr Ewa Kwiatkowska, ESA - Project initiator / Technical ESA representative
Marco Zühlke, Brockmann Consult GmbH - Developer

Information

More information about the study and its results can be found in the Final Report. Other public documents you might be interested in are the Baseline Requirements and the Acceptance Test Plan. The Technical Specification is available for registered users only.

Brockmann Consult GmbH
Max-Planck-Str 2, 21502 Geesthacht, Germany
www.brockmann-consult.de
info (a) brockmann-consult (d) de
Tel +49 4152 889300
Fax +49 4152 889333