Despite the trend towards widespread adoption of cloud-native formats and standards, much of the climate data publicly disseminated by NOAA has traditionally remained out of reach of people who could benefit most from it. Often, these potential users lack the necessary GIS and software engineering expertise to fully take advantage of the data. This includes everyone from scientists, to engineers, to urban planners, and even interested laypersons. But things have started to change. It is becoming increasingly obvious that pairing large language models (LLMs) with the right data and tools has the potential to bridge this gap, but how to do this well remains an open question.

It is exactly this question that we explored as part of our work on the NOAA-funded “Study to Determine Natural Language Processing Capabilities with the NCCF Open Knowledge Mesh (KM/NLP)” with a goal to build a conversational interface for asking arbitrary questions about NOAA’s climate data. We also wrote about this work earlier this year; with the project now concluded, we present a final highlight reel and share some takeaways.

Full workflow for the user query “For the Chesapeake Bay, show the Aug 2026 SST on the map. Also plot a histogram of the values and compute the mean.” Plus, a tour of the UI showing detailed results and provenance information.
Quick tour of the outputs for multiple other example user queries demonstrating the wide variety of questions the app supports. Examples include, in order: computing the trend in SST in the Gulf of Mexico over the past few decades, visualizing a timelapse of hurricane Milton, analyzing the 2026 El Niño, analyzing solar cycles, creating grid plots to compare SST trends in multiple seas, and visualizing temperature forecasts using ensemble weather forecast datasets.

The Foundations: Cloud-native Data Formats

If one wants to build an application that allows performing arbitrary analyses on real data, it is crucial to have that data stored in the most analysis-ready format possible. We started off this project experimenting with various ways to use NetCDF files directly, but found that to be too slow. We then transitioned Kerchunk-based virtual Zarr stores, and eventually to IceChunk stores. This required us to build a data ingestion pipeline that takes in NetCDF files from NOAA’s S3 buckets and builds virtual Zarr/IceChunk stores that reference those files. In the end, we ended up with 176 data variables across 36 distinct climate and weather datasets, which included some 3rd party Zarr stores published by dynamical.org.

An Agentic Harness Engineered for Trust

Our overarching goal was to produce reliable, consistent answers grounded in real data, and we designed each component of the workflow with that in mind.

Data Search Powered by a Knowledge Graph

To find the datasets most relevant to a given question, we use an agent to query a knowledge graph populated from metadata extracted from the original NetCDF files. This not only ensures that the search results are based on authoritative metadata, but it also speeds up searches. Because the results are presented to the agent in an easily digestible format, the need to access files at runtime or perform extensive parsing is precluded.

Graphic that says "Data Search Agent" at the top. There is a robot icon with arrows pointing to it from bubbles that say "SPARQL tool", "Fetch tool", and "Embedding search tool". Those bubbles also point down  to a block with many other lines and circles and arrows
Visualization of how the Data Search Agent works.

Data Analysis that is Scientifically Sound

For data analysis, on top of producing answers from real data, we wanted to ensure that the steps performed to arrive at any results were scientifically valid. A major design goal was to use, as much as possible, predefined “recipes” that had been written, reviewed, and validated by humans. This meant that letting the agent write and execute raw code was not the ideal approach.

The solution we arrived at was a compromise between the flexibility of the code-generation approach and the reliability and safety of the predefined-recipes approach. We call it Constrained Compositional Synthesis. Under this scheme, the agent can only use predefined recipes but is allowed to compose them together to represent complex multi-step algorithms. A rich enough function library (the collection of recipes), allows the agent to fulfill a very wide variety of data analysis requests.

Graphic labeled "Algorithm Agent" with a function library and a flow chart.
Visualization of how the Data Analysis Agent (AKA Algorithm Agent) works.

A Multi-Agent Setup

We distribute the responsibilities across three agents: a Data Search Agent and a Data Analysis Agent (AKA the Algorithm Agent) as discussed above, plus an Orchestration Agent that interacts with the user and delegates tasks to the other two. This separation of concerns allows us to keep each agent’s context clear of irrelevant data. It also allows us to evaluate and optimize each agent in isolation.

"Orchestration Agent" graphic with a box of tools including planning, geocoding, data search, and data analytics.
The Orchestration Agent and its tools.

Context Engineering, Structured Outputs, and the Probabilistic-Deterministic Interface

One of the design principles we stuck to was to move the fixed and predictable parts of the functionality to deterministic code as much as possible and to only leave the truly “magical” bits (such as understanding natural language and performing complex reasoning) to LLMs. This informed implementation choices such as:

  • A pass-by-reference mechanism whereby large data (such as raw GeoJSON) is automatically uploaded to cloud storage and the agents receive a reference to it (along with summarized metadata) that they can then pass around to other tools without getting their context overwhelmed. This also sidesteps the common pitfall, seen in naive implementations, of LLMs having to redundantly transcribe large amounts of data from one tool to the next, which is slow, error prone, and wasteful in terms of tokens.
  • Structured inputs and outputs with strict schemas and validation logic via Pydantic. All agents take in structured outputs as inputs and (with the exception of the Orchestration Agent) produce structured objects or tool calls as outputs. Even agent-to-agent communication–i.e., when the Orchestration Agent delegates tasks to other agents–is mediated through tool calls with strictly defined inputs and outputs. All of this ensures that we know the shape of the data moving through the system at any given time and provides a predictable interface between its probabilistic and deterministic parts.

AI Evals

When integrating LLMs into any system, it is important to thoroughly evaluate them against an evaluation dataset. This provides an objective measure of their performance that one can use to set expectations for the real-world performance and also to choose the right model for the job.

But while evaluations are important, they are also hard to do well. Challenges include: the open-ended nature of the inputs and outputs (even if structured), the inherent non-determinism in the outputs (even with temperature = 0), and the need to evaluate the LLM and the harness around it as a system rather than just the raw model.

We put a lot of effort into designing extensive evaluation datasets and used them to benchmark different configurations for the Data Analysis and Data Search agents and choose the optimal ones.

Data Analysis Agent (AKA Algorithm Agent) accuracy benchmarked by model variant.
Data Search Agent retrieval performance benchmarked by model variant.

Looking Forward

This application was intended as a prototype and we do not intend to open it to the public. Moving forward, however, we intend to subsume this work into our new Limner offering. Limner is a similar but even more ambitious undertaking that provides a conversational, agentic interface for not just climate data but all kinds of data–both geospatial and beyond.

If you’re interested in discussing this work, or Limner, or learning more about querying and analyzing geospatial data more generally, we’d love to connect! Reach out to our team on our contact page any time.

Acknowledgements

This work is a collaboration between Element 84 and TechTraverse as part of the NOAA-funded “Study to Determine Natural Language Processing Capabilities with the NCCF Open Knowledge Mesh” BAA. Project contributors include: Kendall Aubertot, Sean Malone, Jason Gilman, Evan McQuinn, and Stephanie Wall.