Top 4 reasons weather predictions beyond a 10 day forecast can be unreliable
Since times immemorial, mankind has looked to the skies in search for clues of what the weather may be in the coming days.
Whether you’re planning a barbeque with your friends or are a shipping company with your vessel enroute to its destination, you will refer to the weather forecast available in your area to ensure that conditions lie in your favor. Whilst doing this, you’ll realise that you may have access to a 10 day forecast only.
In 2017, it was estimated by an article in the Harvard Business Review that weather disrupts the economic and operational performance of approximately 70% of companies around the globe.
Earlier this year, Ever Green, a 400 meter long container ship ran aground upon the Suez Canal as heavy gusts of wind blew it off-course.
As you may know, the Suez Canal is one of the busiest trade waterways for grains, oil, refined fuels and several other export goods. It is a strategic shipping route that links the East to the West. The Suez Canal blockage cost the world $9 billion in trade per day. This is just one instance of countless weather-related events that can impact your business. It is estimated that when aggregated, weather variability costs the USA $630 billion per year.
Whether you’re a port manager, the captain or a 2nd officer on a ship, or a maritime meteorologist, you will turn to weather forecasts with a strong hope for accuracy. However, you may know from personal experience that this may not always be the case.
SciJinks, a joint NOAA/NASA educational weather website, assesses that a 10 day forecast is correct 50% of the time. On the other hand, a 5 day forecast can accurately anticipate what the weather patterns will be around 90% of the time whilst a 7 day forecast holds an 80% preciseness level.
What is the reason for this? Why are weather predictions beyond a 10 day forecast deemed unreliable? Is it possible to reduce the margin of error in weather forecasting?
That’s the story we’re here to tell.
How is a weather forecast created?
Let’s first look at how weather forecasts are created. Creating a weather forecast, much less a 10 day forecast, is a complex process.
The first step to predict how the weather will be tomorrow is to comprehend what’s happening now, i.e. in the present moment.
Weather is monitored 24 hours a day through a global network of devices that includes ground sensors, weather balloons, different satellites, airplanes, and others. Usually, governments manage this network through their meteorological centers.
Before producing a weather forecast, we need to bring all these observations together to form a three-dimensional model of the current state of the Earth’s atmosphere. This is part of a highly intricate, specialised process termed as data assimilation.
The result of the data assimilation becomes the input to several numerical weather prediction models. These tap into the power of supercomputers which work their magic and predict how that initial state (the planet weather today) will develop in the following few days – and voila, your weather forecast is ready!
Why do forecast errors happen?
Our planet’s land mass is more than 510 million square kilometers, out of which a massive chunk is unobserved. This creates gaps and uncertainties in the initial conditions that constitute observation data.
Given the complexity of the nature of weather, our society doesn’t possess sufficient computing power to simulate it entirely. We also don’t know enough about the enigmatic forces that make up our universe – so our weather models have to conduct estimations.
On top of that, our world’s weather is a chaotic, highly interconnected system.
Any minor mistakes at the beginning of the forecast can transform into major errors a few days later, hence creating a ripple effect. For weather forecasts, having nearly correct input does not generate a nearly accurate forecast.
“A weather computer model is only as effective as its data inputs, it is evident then that the more reliable observation data it is fed, the higher the degree of accuracy can be hoped to be achieved in its weather forecast. This is just like other machine learning / predictive modelling in any other industries. Garbage data in = garbage data out. In other words, improving the quality and quantity of the data into a model will improve the outcomes (predictability) of the model.”
Share on Facebook Share on Twitter Share on LinkedIn
However, there are limitations to the quality of observation data collected. Let’s take a more detailed look at this:
Gaps in observation data
By now, it should be abundantly clear that a weather forecast is only as accurate as the observations upon which it was founded. When you look at the number of sensors collecting weather data across the globe, you may wonder where or why such observations gaps lie. Note that only the weather balloons and aircrafts collect weather data from the upper echelons of the atmosphere. The rest can be classified as surface weather observations.
- 10,000 weather balloon stations
- 3,000 commercial aircraft
- 10,000 surface stations
- 7,000 ships
- 100 moored buoys
- 1,000 drifting buoys
The above numbers are sourced from the World Meteorological Organisation and collect key insights about the world’s land, surface and ocean. They sound impressive – but our Earth’s atmospheric and geospatial distribution is diversely vast. It is also important to know that regional satellite and radar systems support the collection of weather observation data and we will explore why gaps still exist.
Inconsistent quality of data
Whilst certain corners of the globe can generate a consistent, authentic supply of weather observation data, other regions may not be able to match the same level of high quality and coverage.
Think about nations such as Africa which has only one-eighth the minimum density of weather stations termed as desirable by the World Meteorological Organisation.
Despite 70% of the continent’s population depending on agriculture for its food production and employment, local agencies and governments are not able to allocate resources to invest in building an adequate weather observation infrastructure. Doing so would benefit the farmers immensely to mitigate risk and optimise their farming practices.
Tapping into a pool of reliable ocean weather observations is also a complicated process. The surface area of all oceans is 140 million square miles and the majority of it is too remote for wide-band communications. Other elements also interfere in the transmission process such as storms, saltwater and breaking waves. As we’re not able to access each and every point of the high seas, it becomes hard to accumulate a sufficient level of data to create accurate weather forecasts.
It’s not just challenging to aggregate accurate weather observation data at the ground or ocean level – the atmosphere itself creates obstacles. To predict weather beyond the next few hours or to create a 10 day weather forecast, you require data points that flow above the surface – as that is where the actual weather patterns are in play.
So, what can you do?
You rise above the clouds and venture into the unknown.
Developed countries tend to own the resources that can be allocated to researching, producing and launching the biggest satellites into space. It’s only natural that they will choose to direct the coverage focus within their own topographical districts and not incorporate other parts of the globe, hence leading to inconsistent weather monitoring.
For longer ranges (beyond 5 or even 7 days), it’s critical to have observation points from higher altitudes in order to detect weather events that may take up to days to affect the weather we observe. The stratosphere has long waves (called Kelvin waves) that are known to directly impact the troposphere, where 75-80% of our atmosphere exists. These waves can affect the weather a few days ahead, also contributing to larger forecast errors beyond the 5th day.
In a utopian realm, everyone would have access to the observations from these different atmospheric layers. However, the reality is that not all nations are able to allocate the resources needed to build the technology that can record and scrutinise data points at this level. In other words, satellites capable of such services are available only to those who have the means.
Constant fluctuations in weather
The atmosphere is a complex ecosystem and defined by forces of nature that are beyond the power of human comprehension.
Since the atmosphere is in a constant state of flux and full of tumultuous flows, infrequent movement of air that creates storms, clouds and such. These build on each other and create layers of interconnectivity. The tiniest change such as a butterfly flapping its wings can exert a domino effect, impacting the other layers and magnifying into extreme weather events.
If the changes in observation data are not recorded and input correctly into the weather modelling systems, even the most minute error within what is called the starting conditions of the weather model, will lead to major errors.
Limitations in weather modelling systems
Now, it is clear that errors in weather forecasting occur as we don’t know what every small molecule in the atmosphere is up to. The truth is, even if we did, we don’t have the capacity to craft a complete digital twin of our entire planet. As a society that is more technologically advanced than ever before, we’re still not at that stage as to where we’ve generated computing power that’s capable of replicating the Earth within a weather model.
To run our models on our existing computers, we have approximated sizable areas that are treated as single entities in the simulations. This is referred to as the model resolution. This simplification is what causes deviations as all the weather processes are not completely simulated. For example, storms are weather events that occur on smaller scales and are difficult to forecast.
Some weather modelling methodologies are specialised in a niche area and may require the aid of other forecast service providers to fill the gap. This collaboration amidst the global weather forecasting system is the secret sauce behind the success of weather forecasts being more accurate than ever before in recorded history.
The existing weather models are effective in their performance – but may not be able to achieve 100% accuracy in predicting the weather. However, this only means that, as a society, we must keep pushing these boundaries and moving forward towards advancement.
What we can also do is improve our systems and the quality of the initial conditions provided. That itself is a major hurdle to cross as countries manage their own national centers that run global weather models. Private firms base their own forecasts from these data centers. Each unit has its own process of collecting multiple sets of observations, imposing different data assimilation techniques and then has varying weather models with different simulations or parameters set for processing. This high level of differentiation is what grants each different weather model its own set of strengths.
How can we help improve 10 day weather forecasts?
When you think about what’s been accomplished in the realm of weather forecasting and where we are not as compared to a century ago, it’s nothing short of awe-inspiring. Now that you’ve understood the complexity of the Earth’s weather and how a minute trigger can unbalance the whole universal system, the fact that we’ve reached this modern degree of weather prediction accuracy is mind-blowing as well.
Humanity began to dabble in weather forecasting over a century ago. Since then, governments and other stakeholders have injected massive investments into the research and development of this field. They have employed the brightest minds in the world to try and solve this vast, mystifying puzzle that constitutes weather and figure out how all its infinite pieces fit together in our worldly picture.
But like any human endeavour, we keep pushing forward and seeking ways to improve. This is what we are doing at Spire.
Satellites are resolving observational gaps
To address the issue of observational data, we have a constellation of over 100 nano satellites orbiting in close proximity to the Earth. These beautiful devices utilise a technique called radio occultation to collect over 10,000 atmospheric profiles a day.
Not only that, we launch new satellites with improved payload power as often as every six months/quarter. Even when COVID-19 sent the world into a lockdown and grounded aircrafts, there was a dearth of weather data. Our satellites continued to encircle in space and perform their job.
Our vantage point from the skies gives us a definite functional edge, not only in terms of accuracy of observation data collected but provides a continuous feed loop.
As we’re constantly collecting data from very high altitudes (from the Stratosphere to the measurements of the Kelvin Waves explained earlier) near the surface, Spire has eyes that monitor even the most under-observed regions. This includes but is not limited to appraising the depths of the oceans to the expanse of the Southern Hemisphere plus other remote territories. This way, we’re able to fill many of the gaps in the accumulation of weather observation data.
The United Kingdom Met Office evaluated Spire’s data (in smaller quantities) through a series of experiments from September 8 to December 8, 2019, studying a range of weather variables and forecast lengths of up to six days. Its results were resoundingly positive. “There is a substantial forecast benefit from assimilating Spire data,” the report found. “These benefits are seen for almost all forecast variables and lead times.
To take advantage of these observations that are available to Spire in unmatched volumes, we run our own data assimilation and global weather model.
Machine learning is resolving model & simulation problems
As far as where the limitations in weather model and simulation techniques exist, we’re excited to introduce machine learning principles in our new forecast system, fondly known internally as JUNO.
How does JUNO work? First, it analyses colossal quantities of past forecasts from different weather models (Spire’s proprietary global forecast and different public ones). Essentially, it compares what those several forecasts predicted versus what was later observed by our satellites plus other observation data sources. Then, it understands and learn’s each specific model’s strengths and weaknesses. Finally, armed with these new learnings, when new forecasts are being produced, JUNO can on-the-fly, analyse millions of such data points and generate optimised forecasts for specific locations. It can further customise forecasts for specific climates, regions, different types of weather events and more.
How are accurate weather forecasts changing the world?
This powerful combination of space satellites constantly scanning our planet and our machine learning platform optimising weather forecasts, together, is creating meaningful opportunities for companies across diverse industry verticals.
Our partner, PredictWind provides wind forecasts to the maritime and leisure sporting community and is always seeking ways to improve their services. PredictWind created an objective, in-market study to assess the accuracy of several different weather forecast models. PredictWind’s comparison ranked Spire as #1 for wind speed and wind direction prediction accuracy. For more detail, read our customer case study here.
In the renewable energy sector, we’re able to tailor our hub-height wind forecast to help our customers safely predict their power production capabilities and avoid financial penalties. For Insurance, our probabilistic forecast can accurately assess risks of adverse weather and help wind farm managers make safer decisions. With collaboration with entities such as Quiron, we’re helping to protect regions that are subject to high risks of wildfire through our accurate weather forecasts.
“Machine learning is allowing us to work with customers to create the solutions they need,” said Matthew Lennie, Machine Learning Engineer at Spire.
If you’d like to learn more, we’re excited to share that we will be publishing a short series of articles highlighting how machine learning is revolutionizing the field of weather data and improving 10 day forecasts across diverse industries. Stay tuned!