AI weather modeling: Spire and NVIDIA’s partnership presents true evolution in weather prediction

In an interview with Spire Meteorologist James Van Fleet, Gowan explores AI and the future of weather prediction, which he describes as the biggest technology jump in the field in decades, and what it means for Spire and businesses in general.

In the ever-evolving landscape of weather prediction, harnessing the power of artificial intelligence marks a significant leap forward. Dr. Tom Gowan, Spire Global’s Machine Learning, Modeling, and Data Assimilation lead, sat down with Spire Meteorologist James Van Fleet to discuss the groundbreaking partnership between Spire and NVIDIA and its implications for the future of weather forecasting. Drawing on his extensive background in meteorology and machine learning, Dr. Gowan sheds light on how this collaboration represents a seismic shift in the industry, offering insights into Spire’s unique approach to weather and climate solutions for businesses. Join us as we delve into the intersection of AI, weather modeling, and business innovation with Dr. Gowan’s enlightening perspectives.

Could you share a bit about your educational background and how it led you to a career in weather modeling, and now AI?

I’ve been fascinated by weather for as long as I can remember. Growing up in the Boston area of Massachusetts, I was constantly tracking nor’easters, particularly intrigued by snow and blizzards. I earned a Bachelor of Science in Meteorology from Penn State, renowned for its top-notch Meteorology program. During my studies, I developed a keen interest in modeling, Numerical Weather Prediction (NWP), and programming. This passion led me to pursue graduate studies at the University of Utah, where I obtained both my Master’s and PhD. My Master’s thesis focused on high-resolution ensemble modeling and verification, while my doctoral research delved into very high-resolution simulations of orographic and lake-effect precipitation. Additionally, I implemented machine learning techniques to downscale high-resolution precipitation forecasts using even higher-resolution precipitation analysis.

Can you share what drew you to work for Spire? Can you elaborate on the company’s vast offerings in the weather & climate space?

After defending my dissertation, Spire approached me for a job opportunity. As I learned more about the company’s mission, I was drawn in. Spire aims to help businesses adapt and thrive in a changing climate, offering unique solutions and technical expertise. With our proprietary data and comprehensive approach to the entire Numerical Weather Prediction process, from acquisition to actionable insights, Spire is well-positioned in the industry. Our advanced satellite technology allows us to gather precise atmospheric data globally, filling crucial gaps in observations. By integrating this data with cutting-edge models and machine learning techniques, we aim to revolutionize weather prediction.

How did the collaboration with NVIDIA come about, and what are the specific goals or objectives are you aiming to achieve through this collaboration?

We’re thrilled about our NVIDIA partnership! We had already been planning on investing in AI weather modeling, which has really gained a lot of steam over the past couple of years, and we really believe it’s going to be a huge paradigm shift in the weather community. NVIDIA recognized Spire’s potential as a leader in the industry and approached us. NVIDIA’s general business model is to find customers that they believe can leverage AI and specifically GPUs, which all these AI models run on, and really create products and build an entire large business around these. NVIDIA’s belief in us is that these AI models are out there, everyone has them. They’re relatively cheap and efficient to run. But Spire is uniquely positioned because we have our own proprietary data, and these AI models are all about data. If you have better and differentiated data, you can produce a more accurate and differentiated forecast. This partnership marks an exciting milestone in our journey.

Could you provide insights into the current challenges or limitations in traditional numerical weather prediction methods that you expect this partnership to address?

NWP are hard-coded models that have been around essentially since the invention of computers. They are a cornerstone of weather prediction science and have evolved over decades, driven by scientists’ better understanding of physics, better computer models, and better input data. The unfortunate thing about these models is that they are extremely compute intensive. At Spire, we run our own NWP models because these are still the gold standard for high-resolution short-term forecasts, albeit costly and complex. The massive code bases of these models make them tough to innovate on. Enter AI models, a game-changer. By training on vast NWP data, they replace hard-coded language with machine learning, offering faster and more accurate forecasts. AI models have efficiently learned how to encode physics much better than NWP systems. And because of that, they can be run extremely fast, less than one minute on one GPU, which opens up huge opportunities to run massive ensembles, which can really sample the entire distribution of possible weather outcomes and rapidly updating forecasts that extend much further into the future. What’s most exciting to the weather community is studies over the last year have shown that these models are starting to eclipse some of our NWP models in accuracy in addition to these massive computational performance benefits.

Spire is thrilled at the democratization of AI in weather modeling and recognizes the many contributions of the weather industry. Can you explain how Spire is uniquely positioned in AI-driven weather modeling? What value does Spire’s proprietary data bring to the table?

These AI models, primarily developed by tech giants and ECMWF, are increasingly open source, accessible even on a laptop with a single GPU. What sets Spire apart is our proprietary data and unique analysis. Most industry players as well as governments are using either NCEP’s initial conditions for their GFS or ECMWF’s initial conditions from the ECMWF medium-range forecast. But at Spire, we have our own that integrates data from both NOAA, ECMWF, and then adds Spire’s proprietary data on top. Then, additionally, we plan on training our own models from scratch that will integrate Spire’s core data products such as our GNSS RO data or GNSS R data, which includes soil moisture, ocean winds, and sea ice. We have a wealth of data that we can use to train these models and create our own proprietary Spire models. In essence, while AI models and techniques become ubiquitous, it’s the quality of data that truly differentiates. That’s our approach.

How will the integration of AI-driven weather prediction be transformative for various sectors, such as agriculture, transportation, shipping, energy, or disaster management, and what potential benefits can they expect?

In the short term, AI models excel in two areas compared to NWP: rapid updates for long-range forecasts and generating large ensembles for better outcome distribution. Because they’re so cheap, we can run AI models often potentially all the way up to 45 days approaching that kind of sub-seasonal time frame. The biggest benefit is that we can run these in massive ensembles. An ensemble is essentially when you run a model with varying initial conditions and some noise in the actual model itself to create a distribution of possible outcomes. With AI models, we can actually run 1,000 to 10,000 ensembles. Recent studies have shown that 1,000 ensemble members are needed to really capture the range of a distribution and capture those extreme events.

So, who will benefit most from those? We believe it’s going to be the energy commodity industry and the people that trade those assets. What’s really important to them is catching an event, an extreme event that other forecast models aren’t catching with an ensemble. You’ll have those 99th percentile events included in there. This benefits energy and commodity traders, utility companies, and disaster management. AI’s strength lies in capturing extreme events crucial for preparedness and any businesses that need intel on long-term forecasts.

However, for short-range, high-resolution forecasting, AI’s full potential may take time to develop. Spire continues to invest in high-resolution NWP models, ideal for customers requiring detailed high-resolution forecasts up to six days ahead.

Data quality and quantity are essential for accurate predictions. How does Spire plan to leverage NVIDIA’s technology to handle the vast amounts of data required for effective AI-driven weather forecasting?

What’s NVIDIA known for? Its GPUs and AI expertise. They are the world leader in this area at the moment. And, in order to train and run these AI models, a method, something we call inference, it requires data running through the GPUs to generate the weights of whatever kind of neural network architecture you’re running. And this requires GPUs that are very good at processing and can handle very high data throughput. And by partnering with NVIDIA, we have access to the world’s best GPUs that are the most reliable, fastest and most efficient ones for both training and inference. This means NVIDIA will handle the infrastructure side for us, and, now Spire can do what it’s best at, which is developing on the engineering, AI and science side. So, we believe this is really a perfect partnership to allow us to build a world class set of AI-driven models and really lead the industry and the world into this new paradigm of AI weather model forecasting.

How will the products you’re developing be consumed by customers? Are there specific platforms or interfaces they will use, or will they build their own using the data?

Some of our sophisticated customers prefer raw gridded forecast data, including temperature, pressure and humidity, etcetera, for analysis, which we readily provide. They will want to derive their own statistics and they have their data scientists that can work with the model output. We always offer that for those customers. Additionally, our Deep Vision™ display system will showcase AI model outputs in a unique way, focusing on key analytics rather than individual ensemble members. We’re going to derive analytics such as the probability of exceeding some extreme temperature thresholds. We’re enhancing our product lineup by integrating machine learning with AI weather models, enabling tailored solutions like energy forecasts for renewables and predictive models for energy demand. Collaborating with our maritime units, we optimize routes using AI-driven weather forecasts. Future plans include developing actionable solutions such as power outage forecasting and weather risk assessment, empowering businesses with actionable insights derived directly from our data.

Looking ahead, what are some exciting developments or advancements in AI weather modeling that you’re working on or anticipate in the near future?

AI weather modeling is rapidly evolving with daily advancements. NVIDIA’s CEO Jensen Huang introduced a new model called CorrDiff, a generative regional downscaling model, enhancing coarse data with detailed high-resolution output. It’s a promising innovation, which is even able to create new convective cells, new areas of wind gusts, and weather scenarios like that. I think that’s really exciting. That’s where NVIDIA is investing in the future. We plan to leverage such models in our global AI forecasts for richer insights. With AI at the forefront, future forecasts may bypass traditional methods, including the need to run data assimilation. Our partnership with NVIDIA signals exciting possibilities, ushering in a new era of meteorological innovation.

Learn more about Spire’s AI Weather Forecasting and how businesses can utilize it

Explore AI Weather Forecasting