This blog was originally published on Feb 17, 2022.

Tracking climate change depends on data. The better the data available, the more the climate can be understood. Shelley McNeill, a federal contractor who works as a project manager and database architect with Riverside Technology at the National Centers for Environmental Information (NCEI), wanted to make it easier for users to track detailed weather station data so they can more accurately assess historical climate observing conditions at the local, regional, national, and global levels.

NCEI, part of the National Oceanic and Atmospheric Administration (NOAA), is the nation’s leading authority for environmental data. NCEI manages one of the most extensive archives of atmospheric, coastal, geophysical, and oceanic research in the world. NCEI’s land surface data comes from tens of thousands of weather stations across the globe, each with its own coordinates, identifiers, and instrument types, which can change over time. To understand how and where the data were collected, users access the Historical Observing Metadata Repository (HOMR). HOMR includes weather station details—with records stretching back to the 1700s—which are used to better understand the trajectory of climate change. NCEI produces global climate monitoring reports based on that data.

In 2011, NCEI’s station history team lost a significant portion of its funding due to changes in federal government contracts. As a result, it went from a double-digit team to just one part-time and two full-time employees. The staff downsizing included several developers and quality control staff. To continue its important work in global climate datasets, NCEI needed to find a way to do more with less.

“We needed to automate everything possible,” says McNeill.

As an experienced database administrator, programmer, architect, and developer, McNeill quickly envisioned how technology could play a role in addressing the problem.

Reinventing internal data management

McNeill developed the HOMR Stations Dashboard app to replace a legacy web application written in ColdFusion. Built on Oracle APEX, the HOMR Stations Dashboard automatically manages the station metadata ingest process written in Java/Groovy. It also handles quality control of station metadata coming from NOAA partners. Automating these processes rather than requiring employees to take them means scientists can more easily track critical climate change data from weather stations, including identifiers, names, locations, observation times, reporting methods, photos, equipment modifications, and siting. McNeill says the HOMR Stations Dashboard app provides cornerstone support to many of NCEI’s core land surface datasets and products.

It’s kind of crazy when you look at the resources we have been able to save by organizing and building this data management tool with APEX.
—Shelley McNeill, Contractor, Riverside Technology, National Centers for Environmental Information

The HOMR Stations Dashboard app also supports NCEI’s HOMR public-facing web application and JSON API, which allows the public to access data from various weather stations and view changes in stations over time. The public-facing web application has approximately 2,000 unique users and 9,000 page views per month. The HOMR database also feeds other NCEI access systems, including systems at regional climate centers.

McNeil says APEX allowed the team to leverage the business logic built into the database, but split functions into the most efficient design where code modules related to integration and access can be upgraded independently from one another and the database. This capability allowed the small NCEI team to upgrade its internal data management tool to APEX and build a web application separately with visualizations that truly serve the needs of external users. For example, HOMR offers a D3 JavaScript visualization tool with its web application for quickly viewing changepoints in station histories at its Asheville NC station.

Replacing the legacy infrastructure with the low-code HOMR Stations Dashboard allowed NCEI to realize significant resource savings. The APEX dashboard was built by a half-time developer resource—the previous application was created by five full-time developers.

“Losing staff is never a good thing, but it gave us a rare opportunity to reinvent our workflow and redesign our architecture, and APEX was the perfect solution,” McNeill says.

Lower costs, improved workflow

The HOMR Stations Dashboard app has also been a game-changer for workflow improvement at NCEI’s data center. For example, the team automated station integration streams monitoring and quality control, as well as all changes provided to a weather station.

Shelley McNeill, a federal contractor at the National Centers for Environmental Information, used Oracle APEX to help scientists track critical climate change data.

“Our quality control manager reviews almost 4,000 station configuration changes from the National Weather Service alone per year. That is only possible with automated quality control and an efficient management tool like this APEX app,” says McNeill. “APEX also allows services, visualizations, and web applications to be upgraded as technology progresses without having to re-architect and duplicate business rules, which will result in reduced cost of ownership long-term.”

Perhaps most critically, using APEX allows NCEI to support its mission of maintaining accurate weather station histories in the most efficient way possible with a much smaller staff.

“We went from having five quality control employees managing HOMR to one,” McNeill says. “It’s kind of crazy when you look at the resources we have been able to save by organizing and building this data management tool with APEX.”

APEX has also won fans in other parts of the NCEI organization. McNeill used APEX to build a similar infrastructure for NCEI’s World Data Service for Paleoclimatology, providing the organization immediate returns on its refactoring investment. Other areas within NCEI have also developed their own APEX apps, along with other NOAA line offices, such as Fisheries.

“APEX has really found a foothold here for internal data management,” says McNeill. “These are all nuanced use cases, so you’re not going to find existing software to manage these types of tasks. That’s the beauty of APEX—you can build what you need, and you can do it quickly.”

Dig deeper

Other articles in this series

Illustration: Wes Rowell
Photography: Liz Nemeth/Getty Images