Climate, models and apps: but what about the data?

WSR-88D Radar
The danger of a lack of data is here. What can be done to prevent imminent, serious issues?

It’s 2021, and here in America, the meteorological profession is seeing a lot of software development and climate research going on. What could we expect 20, 50, 100, 500 years down the road in terms of climate? And this ongoing research doesn’t just look at temperatures, either. Dr. Victor Gensini from Northern Illinois University has done research on long term tornado forecasting and trends. One of his efforts is here:

https://atlas.niu.edu/ertaf/

And then there are the models. From seasonal to multi-decadal, much research is going on to see how the temperatures and weather are generally going to change in the years to come. How much climate change is going to occur, and how quickly?

Also going on the field of meteorology are software/app improvements. AllisonHouse, who make the Maps website; DTN, who make RadarScope, and many other software makers and weather companies provide apps for your phone, desktop or tablet; some are Web-based. We can now see whether we are in a tornado warning down to within a city block in seconds, thanks to plotting of warning polygons on GIS-heavy software like the Gibson Ridge (GR) products, and AllisonHouse Maps. Programmers are writing phenomenal software that would have been unimaginable 10 years ago…and the innovations keep coming hot and heavy!

But there’s one thing that supports them all…and without it, all of this becomes absolutely useless. And that is:

Data.

Without raw data to feed models, apps, and research, everything stops. Everything. And not just those listed above. If surface weather observations cannot be obtained from a major airport in a large city, you can have a partial or total ground stop. You can’t get tornado warnings to phones or apps, or to the media. That alone can cost many lives, or financial ruin by trucks and trains driving into tornadoes or bad weather. Some carry hazardous materials or fuel, and let’s just say an unwarned tornado hitting these while in a city can cause major, life-threatening problems that affect a number of people.

The word “crisis” is thrown around a lot these days, sometimes way too loosely. But I would like to put forth a concern that we are approaching a data crisis, in terms of reliability.

No, I’m not saying that AllisonHouse is unreliable. Just the opposite: we have achieved a nearly 100% uptime across all of our data feeds. But, that depends on airports, individuals, and the National Weather Service and the FAA to provide that data across modern, reliable acquisition and distribution systems. And all of these, to some extent or another, are beginning to, or are, breaking down.

But it’s not just a server/computer/fiber/Internet issue. Farmers who have been taking climate observations for decades are dying, and their farms are being sold to a company who is not interested in sharing their data with anyone, or the next generation of farmer decides that they don’t want to do that anymore. Or, the farm gets sold to developers. Airports close. Chicago no longer has a manned weather station downtown anymore after the airport was closed.

Then, you have smaller airports. The FAA uses temperature and humidity sensors that feed data to the National Weather Service. And then you have the Automated Surface Observation System (ASOS) weather stations that still require human intervention 24/7 to make sure the data is accurate and going out. The FAA wants to eliminate all observers at the large airports, but political pressure has kept meteorologists in jobs there…for now.

And then there’s the National Weather Service distribution system. This system is antiquated, and as more and more data gets shoved through the system, complete breakdowns are occurring on a semi-regular basis now. The latest one, as of this blog post, was embarrassing and scary:

https://www.washingtonpost.com/weather/2021/03/30/nws-internet-infrastructure-outages/

And just before that, their main data center in Silver Spring, Maryland was literally flooded after a water pipe break above their primary data center (don’t get me started on how a water pipe got placed above a very critical data center). This knocked out (ironically) buoy data for a week; and if their system goes down, NOBODY (including us) gets data. That was down for a WEEK. Shipping interests were not amused. The NOAA Weather Radio system is stale in its technology; it is still the best, most reliable way to receive alerts, but it warns via county…even if you are outside the warning polygon. (Still, that is absolutely NO reason not to get one. This is slowly changing as well.) Weather radio data is used by emergency managers, individuals, radio and TV stations to get the warnings out.

There are other blogs that have dealt with this, in part. But one thing that has been neglected is the data itself. AllisonHouse IS data; without it, we are nothing. We provide the data to whoever wants it, in whatever way we can process it to make your app work, and our individual customers use it for weather spotting, monitoring, forecasting, or assisting in making weather-based decisions. And we use as many redundant paths for that data as reasonably possible to keep the data going 24/7/365. During that National Weather Service (NWS) outage mentioned above, it kept the NWS forecasters in the dark for 7 hours. Thanks to redundancy I have in our system here, we lost anywhere from 20 minutes to 2 hours of data, depending on the feed. After doing this for 23 years, I know how to get the data and to be as reliable into our systems than any other weather company. We don’t have “sun outages” every spring and fall that others do with data that is acquired via satellite. We have connections via fiber to primary and backup NWS connections to keep the data flowing.

But when the system breaks down, my best efforts will fail. And at that point, nobody has data. Not us, not the NWS offices, not your app on your phone, tablet or computer; not the NWS websites, not us. It happened for 7 hours on a relatively quiet night in March of 2021. It will happen again.

And that’s the near-crisis we are in. Unless we understand how much VALUE raw weather data provides to us as a nation, we are headed for a data outage…and that next one has a better chance of occuring during a tornado outbreak. A blizzard. An ice storm. It will hamper or destroy research, and our nation’s climatology. It will hurt and/or kill precious people. It WILL happen, unless we route some of these resources back to the government data distribution system, and data sources, NOW.

And the loss of data in more rural areas needs to be addressed as well. A lack of promotion of “co-op” weather stations within the National Weather Service in recent years means fewer climate stations will be around in generations to come, as well as working with companies to make basic data available to researchers. That will make climate forecasting even more of a huge challenge that it already is.

When a sports team is having a bad year, a good coach always goes back to the basics to get back to a team that produces wins. It’s time the United States, its citizens, and the National Weather Service management fully realize how valuable raw data is, and quickly. Otherwise, the system will fail…and 23 years of experience on my end won’t be able to do a darn thing about it.

How do we fix the Weather Service part of the issue?

Read this:

https://www.allisonhouse.com/blog/its-beyond-time-for-another-national-weather-service-modernization/


Author’s note: We at AllisonHouse have data alarms that text and email me and our other employees when something goes down on our end, or at the National Weather Service, or private data vendor end. Until we intervene, we get alerts every 2 minutes until the issue is resolved one way or another. The amount of those I have received over the last 6 months has been staggering…and completely unacceptable.

No Comments Yet

Leave a Reply

Your email address will not be published. Required fields are marked *