AI and machine learning at ECMWF

ECMWF is currently making a significant effort to support applications of artificial intelligence and machine learning and to identify how such applications may improve numerical weather prediction at the Centre. Many standard methods used by ECMWF scientists on a daily basis can be regarded as examples of machine learning. However, there has recently been a surge in new methods which have the potential to revolutionise the work of operational weather prediction centres. Such methods include the use of deep neural networks, which can learn the dynamics of very complex non-linear systems from data.

State of machine learning research at ECMWF

In January 2020, an internal workshop took place at ECMWF in which scientists and analysts presented their current machine-learning-related projects. The meeting was intended to create synergies and enhance communication between individual scientists who use or work on machine learning methods. This has put ECMWF in a position to efficiently spread information regarding upcoming scientific meetings, training opportunities, application needs, and the hardware and software infrastructure that is available for machine learning applications at ECMWF. The workshop revealed that there are some 25 projects at ECMWF that are using (or are going to use) machine learning in one way or another. As illustrated in the figure, applications are spread over the entire numerical weather prediction workflow.

Examples include:

1. bias correction of satellite observations
2. the learning of model error within data assimilation
3. the emulation of model components to increase computational efficiency in the forecast model
4. local downscaling of model output to improve predictions
5. the monitoring of the IT infrastructure.

The state of readiness of the applications varied greatly. It ranged from research projects in the planning phase, for example regarding the monitoring and assessment of reanalysis production, all the way to products that are already available in an operational context. The latter include the ecPoint tool for global probabilistic rainfall predictions and soil moisture retrieval from SMOS satellite observations for data assimilation. In addition, there are several active collaborations with external partners that involve machine learning research.

Next steps

The European Weather Cloud, which is being developed jointly by ECMWF and EUMETSAT, will likely play a very significant role for the development of machine learning tools in the future. This is true for work carried out at ECMWF and in our Member and Co-operating States. Researchers who have access to Cloud computing resources will be able to easily load training data from the data archive and to use standard machine learning software tools such as TensorFlow and Jupyter notebooks. These tools are rather different from the tools that are typically used in ECMWF’s supercomputing environments and they are in general more tailored to Cloud environments. There are currently plans for a hardware upgrade of the European Weather Cloud at ECMWF which would add enough GPU resources to the existing infrastructure to support the training of machine learning applications.

Furthermore, ECMWF and the European Space Agency (ESA) are organising a joint workshop on ‘Machine Learning for Earth System Observation and Prediction’ that will take place at ECMWF from 5 to 8 October 2020. ECMWF is also organising a new seminar series on machine learning starting in April 2020 and will deliver the first (virtual) training course on machine learning for ECMWF staff this spring.