White Paper One Contributor: Hans-Joachim Koppert

Contribution

Q3 Major technological breakthroughs and new applications (e.g. Artificial Intelligence, machine learning, new data, cloud computing) – where are they expected and what will be their impact? 

Cloud Technology

Data produced on a daily basis by large centers will be in the order of a few PB by the end of this decade. Even today only parts of the data are being made available to users (At DWD about one third of the daily production) and an even smaller  fraction is downloaded and actually used. The main reasons are the limited bandwidth of the Internet and the complexity of using the data provided.

Cloud technologies have developed into a mature technology in recent years, are available everywhere and offer a wide variety of functions. Cloud technologies support the so-called "bringing the data to the user" paradigm.  The bandwidth issue can be solved by providing the output of NWP models and big observational data in a cloud data storage. This will enable users from the weather enterprise to run their own NWP models and sophisticated applications in the cloud and so allow for new business opportunities. 

A prerequisite for a broad use is the provision of tools that allow an easy integration of the data into the processes of the users. The implementation of advanced data interactions such as subsetting or transformations is relatively easy to implement with the available cloud resources.  Cloud infrastructures provide a development and operating environment that improves effectiveness and efficiency on any scale, accelerating the development process and daily operations.

Implementing  NWP-Systems  with post processing, production, and visualization in the cloud could offer a unique advantage for developing countries. This approach will provide a more sustainable and cost effective solution, also allowing for an improved cooperation and burden sharing between partners. To implement this, certain competencies and financial resources must be available.

Artificial Intelligence (AI) and Machine Learning (ML) 

The amount and diversity of environmental data and its possible applications has grown and will continue to grow in the future. We see an abundance of unexploited observations and model results.  Machine Learning reverse engineers functions from data in a high dimensional space allowing complex relationships to be detected in disparate data sets. Although the operational application of Machine Learning (ML) is not yet widespread, many projects and use cases show promising results (Boukabara, 2019). 

ML can lead to significant progress over the whole range of 'classical' modeling, data assimilation and post-processing algorithms in the next decade. ML can be applied at each step of the weather forecasting process. Some examples are listed below: 

• ML can improve our utilization of observing systems, allowing forward planning of maintenance activities, improved quality control and intelligent filling of data gaps.
• ML can improve our ability to introduce information from complex observing systems into models through data assimilation  by providing better compression rates and thus allowing significantly increased precision and speed of forward operators 
• ML can help us model and estimate the observation error for data assimilation. 
• ML allows us to incorporating further complexity physical parametrizations or to train parameterizations on data with higher resolution than that of the model they are intended for.
• ML can be used in post processing of numerical variables allowing us to incorporate non-linear correlations
• ML allows us to estimate impact based on model output and further observations or statistically relevant data

A wide range of open source machine learning frameworks (e.g. TensorFlow, PyTorch) together with well-known programming languages like Python or C++ support the development of ML-applications. With the advent of dedicated accelerators, it became possible to tackle large-scale problems. A cloud infrastructure allows easy access to accelerator hardware.  If the meteorological data treasure is readily available, SMEs can develop successful applications, especially if they can bring together expertise from different disciplines.

The performance of a trained ML based application can provide forecasts within minutes thus allowing these applications to be used even for demanding real time applications like the automated detection of severe weather. We expect that in certain areas ML will lead to a significant boost in forecasting performance, especially in areas where new data sources can be exploited and combined with traditional data or where there is no clear understanding of the physics. It is still unclear whether ML applications will be able to outperform traditional methods. It is highly likely that the combination of both approaches will be the best way forward. 

Challenges remain for the realization of these opportunities described, The statistical nature of ML leaves open questions with regards to its conservation properties.  Lack of understanding of how KI algorithms function can introduce uncertainty into the forecast process, so that rigorous testing is necessary before operational implementation.  And large training datasets are needed to ensure that stable forecasts can be provided even when a rare event occurs.  

New Data and the Internet of Things

The Internet of Things is a system of interconnected computing devices. It delivers a huge volume of data and will grow significantly in the years to come. All sort of devices will provide environmental data: smart phones, low cost weather stations, cars, and many more.

There are many fields of applications where the meteorological community can benefit from data provided by the IoT.

A few examples show the potential and a possible usage:

• Crowed sourced data from low cost commercial weather stations. These stations can help to densify the observational network.
• Data from modern automobiles.  Acting as rolling measurement stations, they can provide a wealth of meteorological data. These data have the potential to be very useful to enable high spatial resolution analysis and to develop better nowcasting and warning systems, especially to support autonomous driving.
• Data from industrial installations like wind turbines. Wind data measured in wind farms could be used to enhance the wind forecast and in turn contribute to a more economical, and stable operation of the power grid.
• Crowd sourced data collected with apps offer a unique possibility to receive impact data of severe weather events

Although there is a lot of potential in these new data sets, there are also legitimate concerns with their usage. There are legal problems with data ownership and privacy protection. The data quality is not comparable to standard WMO weather stations. Quality control could become difficult due to data protection regulations, which could even make bias correction difficult. The global availability and the reliability of he the data provision cannot be guaranteed.

Hier kommt noch Text zur bodengebundenen Fernerkundung…vielleicht auch zu MTG-IRS (Aeolus?) 

Q4: “Technical improvement – accuracy & reliability vis-à-vis value & impact".

Environmental forecasts

On top of the steady improvements in NWP accuracy by one-day per decade,  the public and businesses enjoy additional increases in accuracy and potential value of meteorological data due to an explosion of the availability of weather and climate information on the internet. E.g. 25 years ago, most businesses and the public could only receive forecasts and warnings: via traditional media, on a few weather variables, for regional scales or a few cities, mostly in daily resolution, without uncertainty estimates and in a non-digital format. Now, the persistent challenges and new opportunities in the provision of fit-for-purpose forecasts and warnings are:

• to predict all weather and climate variables needed 
• on the true, often very local scale of the user,
• to deliver forecasts seamless across time, i.e. without the current breaks in the data due to the division into now/short/medium/seasonal/decadal-casting,
• to attach uncertainty estimates for all data based on reliable ensembles,
• to provide the data in an easy to use format, web service, app, etc. . 

To increase the usability of warnings and products further down the value chain, a significantly larger proportion of the data must become available to users outside of the meteorological community. In addition, metadata need to be locatable and provided in modern data formats (e.g. geoJSON instead of GRIB). Data provision must be configurable based on user requirements and individual user profiles. This lowers the barrier for the private sector to use the data.

External users and the private sector are particularly competent in the user- friendly post-processing of numerical analyses and forecasts. However, a major obstacle to the usage is the availability of consistent training data sets. Therefore, the provision of sufficiently long data sets of reforecasts and reanalysis of the actual modeling system is essential.

Warnings are understood when they are related to user`s lives. Using smartphone apps, personalized warning and forecast information could bridge the last mile to trigger the right response.

Success can be measured automatically, e.g. by the exponential increase in access to the data, or by verification of forecasts, which reveals for instance, that the use of statistical post-processing can improve local forecasts by an equivalent of 5-10 years progress in NWP.

Impact forecasts

 “Progressing from weather forecasts and warnings to multi-hazard impact-based forecast and warning services represents a paradigm shift in service delivery.” (WMO, 2015). The aim is to deliver more direct support for decisions and ultimately enable better preparedness and response to hydro-meteorological events. It requires:

• the digitization of and access to impact observations,
• easy access to high resolution indicators of vulnerability and exposition of people, infrastructure, etc.,
• a knowledgeable use of the exploding opportunities of AI for impact modeling,
• a substantial increase in cooperation with users impacted by weather and climate. 

Yet numerous challenges accompany this development, e.g. there is a need to: 

• negotiate the access to the impact data in light of the diverse ownership and commercial interests, 
• develop quality control for non-conventional data and to ensure provision of metadata,
• provide impact data also for real time assimilation and verification and not just for model development on past data, 
• establish the best dissemination platforms, protocols and formats, 
• ensure co-design between developers and users of the services,
• balance the differing business models of NHMS, private companies and academia in the development and 24/7 operational service delivery,
• provide training and education for understanding and usage of these new type of forecasts . 

Ultimately, the same scientific rigour used to develop NWP and climate models has to be applied for the development and testing of products, services and their use, including the qualitative and quantitative assessment of their socioeconomic benefits for user decisions. Harnessing social scientific knowledge and methods to ensure optimal usability and use of impact based forecasts and services has to become an integral part for the whole process.