Article by Dr. Mark Miesch, JCSDA JEDI Core Team
Graduates of the 4th JEDI Academy hold up their customary lightsabers in Monterey, California on February 28, 2020. It is typical for graduates of in-person JEDI academies to receive a lightsaber upon academy completion.
The 4th JEDI Academy was held last week (Feb 24-28, 2020) in Monterey, California. The JEDI Academy is a training session for new users of the Joint Effort for Data assimilation Integration (JEDI) data assimilation system for Earth System prediction. JEDI is used by JCSDA and affiliated partners for improving the accuracy of weather forecasts. It is innovative both in its technical implementation, leveraging the C++ language to establish generic high-level interfaces between observations and models, and in its governance by not only inviting input from the scientific community, but also encouraging community users to become developers and contribute directly to the code base. High-level interfaces such as the unified observation (forward) operator (UFO) allow JEDI to be used for an unlimited range of applications, from pedagogical toy models to research to operation workflows (R2O/O2R), to operational atmospheric, oceanic, and coupled forecast systems.
As in previous academies, the curriculum included both lectures and hands-on activities to familiarize employees, affiliates, and collaborators with the JEDI system, its use, and its management, including how to build and run JEDI, how to contribute code changes, how to engage in code reviews, and how to participate in discussions on future code developments. About 40 participants attended from affiliated institutions, including many Monterey locals from the Naval Research Laboratory.
Each day began with one or more lectures, followed by an interactive activity allowing participants to apply the principles covered in the lectures. Each afternoon then saw another round of lectures and activities. In order to carry out the activities, participants were divided into 20 groups of approximately two individuals each. Each group was given their own cloud computing instance to work on, provided through Amazon Web Services (AWS). This was the first Academy to make use of a new JupyterLab framework that allowed groups to access their Amazon compute instances through a web browser.
Lectures and activities on the first day gave participants an overview of JEDI, object-oriented programming, and the collaborative tools that are used by users and developers to manage, design, distribute, build, document, review, and test the JEDI code base. The second day was devoted to how observations are processed by JEDI through the Interface for Observational Data Access (IODA) and the Unified Forward Operator (UFO). Participants were given hands-on experience working with configuration files, adding observations to be assimilated by JEDI applications, and adding quality control (QC) filters. The third day covered how models are integrated into JEDI, along with related topics such as the B-Matrix Unstructured Mesh Package (BUMP) that can be used to generate static and ensemble background error covariance matrices for essentially any model. The final day began with a primer on C++ and how and why it is used by JEDI. Then participants heard a lecture on how to add a unit test in JEDI and were given the opportunity to add a test themselves in the ensuing activity. Another lecture covered how JEDI can be ported to run on laptops, workstations, clusters, commercial cloud platforms, and high-performance computing (HPC) systems using technologies such as software containers, machine images, and environment modules. Then, during the activity, participants were given the opportunity to install JEDI on their own systems with the help of the JEDI team.
Ultimately, the goal for JCSDA and the JEDI project is to create usable, open-source JEDI code that can be utilized by forecasters, academics, engineers, and anyone else with an interest in Earth system prediction. With each JEDI Academy, our format becomes more refined and we strive to establish these training sessions as a model that can be emulated by the community to provide students, researchers, and forecasters, to develop their practical coding skills and forecasting experience. It’s safe to say that JCSDA is leading the charge for innovative Earth system prediction, and the JEDI academy is a key component.