The Interface for Observation Data Access (IODA) is a component of the Joint Effort for Data assimilation Integration (JEDI) software system. From a user’s perspective, IODA handles the input of observation data and output of diagnostic data. Efforts to transform IODA from a prototype system into an operational system that can handle big data is under way, and the task of collecting requirements from the JCSDA partner organizations was initiated by the 2019 IODA Workshop. The workshop took place from February 11 through February 13, 2019 in Monterey, California, and was attended by representatives from the U.S. Navy, NOAA, NASA, ECMWF, UK Met Office, NCAR, UCAR and JCSDA. In total 41 people attended from 8 partner organizations.
The workshop opened on Monday (February 11th) with introductory remarks from two of the organizers Nancy Baker (NRL) and Yannick Trémolet (JCSDA). These talks were followed by speakers from the Navy, NOAA, NASA, ECMWF and the UK Met Office describing requirements for observation handling from their various perspectives. The opening presentations were very informative, and provided useful requirements information for IODA along with an effective foundation for the focused discussions that were to take place through the remainder of the workshop.
Three focused discussion sessions were conducted over the remainder of the workshop. Tuesday’s topics were “in-memory observation handling” and “File and I/O requirements for observation handling”. The third discussion session took place on Wednesday with the subject “pre- and post-processing diagnostics requirements”. Each discussion session was preceded by a set of talks given by representatives from the various partner organizations which served to prepare everyone for the associated discussion. Each set of talks were designed to allow the audience to hear from multiple perspectives including users of DA systems, scientist contributors to JEDI, and JEDI software developers. Material from the presentations was condensed, on the fly, into a list of prompts that were used to seed the subsequent discussion. All workshop participants attended all of the discussion sessions resulting in an effective means for communicating requirements and perspectives among the different partner organizations in the JEDI project.
Work is under way to compile the requirements gathered at the workshop and transform these into the necessary actions to satisfy them. The compilation step has yielded categories that represent different aspects of the IODA design. These categories include themes such as security (the need to handle classified and private data), reliability (an operational system needs to be running 24/7), ease of use, and portability (runs with a variety of hardware platforms, software compilers, and programming languages). We will be using agile software development practices, such as ZenHub tracking and code sprints, to implement solutions for the IODA requirements. The results of the compilation of requirements along with the tracking of the work will be posted where they can be viewed by the partner organizations.
Overall, the workshop was conducted in a highly positive and enthusiastic manner that solidly reflected the collaborative spirit of the JEDI project. People from different organizations showed up with the intention of shaping IODA into a world-class observation handling system. Many important details relevant to the IODA design and architecture surfaced during the time we spent together. We mention here only a few highlights. Although the Human Genome Project is famous for its dealing with big data, the amount of observation data that we are dealing with is roughly an order of magnitude larger. When you consider the model data that are also involved in DA flows, that difference becomes three times greater. The manner in which observation data is collected, organized and stored in files is incredibly diverse, and yet there appears to be real promise in creating a common method, or a very small set of methods, for organizing and storing those data within IODA. Attaining a common method of organizing and storing the vast amounts of observation data will go a long way in fostering the collaboration, exchange, and sharing of scientific results.