Behind the data: AllisonHouse stays on through an ice storm

The rreception of the primary National Weather Service data feed nosedives as the ice piles up on our dish. How did we prevent any delay or loss of data? Go behind the scenes with AllisonHouse.
The signal-to-noise ratio and signal strength craters as a record ice storm covers our satellite dish in ice in Oklahoma City, OK on October 26, 2020. This is how we stayed “on the air” with no downtime for our customers, even with a loss of the dish in parts of the event, and a total loss of commercial power.

In our last blog post, I highlighted our refurbished satellite dish. Well, as it turns out, it wasn’t what we had hoped for. We needed two more upgrades and a fix to take care of other issues we discovered after I published the article. In typical AllisonHouse fashion, we got that done. Now, would these upgrades work and keep the dish stable in extreme weather?

A week later, we had 40-50 MPH winds for 24 hours that the dish took in stride. We knew the issues were solved, our dish was substantially better than even in *new* condition in handling reception during bad weather. I was like…nature, BRING IT!

Ummmmm…be VERY careful what you wish for.

Around October 18, the GFS model began showing a significant ice storm in and around Oklahoma City a week out, where our dish is located. I laughed. No way…it’s only October! But days later, the European and other models sounded the alarm.

Nobody, including myself, was laughing now.

On October 25, I got a text from Ryan Hickman, our COO, saying that there was a “99% chance” that we would have to switch to our backup satellite dishes in the Midwest for our data feeds. 30 minutes later, I made the call: it was a 100% chance. 12 hours before the first freezing raindrops hit our dish, I flipped AllisonHouse’s National Weather Service (NWS) primary data feed over seamlessly to the backup satellite dishes, with absolutely no downtime, and completely transparent to all of our users.

Then the ice storm hit, with almost wrath of God intensity. As you can see from the graphic above, there’s no doubt what happened to our carrier to noise (aka “signal to noise”) ratio, and our signal strength after the storm hit. You can tell when the ice significantly and very rapidly accumulated on our dish, and when it was still pouring freezing rain. Reception quality of our NWS feeds from the dish went straight into the dumpster. And the forecasts based on data and the models were even more alarming for the upcoming 48 hours!

When the GFS was cranking out 2″ of freezing rain west of Oklahoma City as the event was about to start, again, some people were laughing. But after the first wave of freezing rain “outperformed”, or were much higher than the European, NAM and other models had forecast…once again, the laughter stopped. On Tuesday morning, October 27, at 9:40 AM, Ryan texted me again. The data center where our dish is located had lost power, and was on a backup generator to keep things working. Then people started walking around inbetween rain bands in Oklahoma City, and we all realized how bad this was, is, and likely would be. Would our essentially brand new dish still even work after this?

As the event continued, I watched our signal go down, then up, and then down after each passing freezing rain band. We only lost data for about 4 hours over the 3 day event, but we took no chances. We stayed on our backups through October 28, until the ice melted and I was confident we were good to go, even though the backup generator was still going. And through all this time, there was no delay or loss of any of our data feeds, or imagery. Our dish suffered no damage, and reception went back to normal after it stopped raining hard on October 28th.

Diversity is a term that is widely used, and sometimes misused, these days. But in terms of data, one of the things we are proud of is depth of data diversity. We don’t host servers and have data inject points at one data center; we host servers at multiple centers. If one goes down, or a satellite dish has issues, other ones kick in, most of the time automatically. In fact, we have been up more than our host this year, and they have been up 99.999% of the time! And where we can get redundancy, we have it. By running the latest proven data ingest and processing software, we get data reliably and VERY quickly…many times measured in a few seconds, or less…not minutes. That doesn’t mean we’re perfect; nobody is. But you can rest assured with us that when outages are likely or are occurring, and things are hitting the fan, we have the experience, staff and resources to get out of a jam quickly, keeping you informed and the data flowing reliably, when bad weather strikes.

Footnote: my prayers go out to all those affected by the Texas/Oklahoma ice storm of October 25-28, 2020…and category 2 hurricane Zeta in Louisiana, Mississippi and Alabama, which occurred at the same time. There is an awful lot of damage with the ice storm, and with Zeta in southeastern Louisiana, and significantly northeast all the way into central Alabama.

2 Comments

  1. November 6, 2020 at 6:29 pm | Permalink

    Very interesting. Thanks!

  2. November 7, 2020 at 12:29 pm | Permalink

    As a long time radio/TV engineer, I know the satisfaction of being able to continue to serve the public even when weather condtions make the technical situation challenging. I’ve setup makeshift studios at tower sites and jury-rigged equipment to keep on the air during ice storms and tornados. You guys do a great job. Keep it up!!!

Leave a Reply

Your email address will not be published. Required fields are marked *