Behind the data: AllisonHouse stays on through an ice storm

The rreception of the primary National Weather Service data feed nosedives as the ice piles up on our dish. How did we prevent any delay or loss of data? Go behind the scenes with AllisonHouse.
The signal-to-noise ratio and signal strength craters as a record ice storm covers our satellite dish in ice in Oklahoma City, OK on October 26, 2020. This is how we stayed “on the air” with no downtime for our customers, even with a loss of the dish in parts of the event, and a total loss of commercial power.

In our last blog post, I highlighted our refurbished satellite dish. Well, as it turns out, it wasn’t what we had hoped for. We needed two more upgrades and a fix to take care of other issues we discovered after I published the article. In typical AllisonHouse fashion, we got that done. Now, would these upgrades work and keep the dish stable in extreme weather?

A week later, we had 40-50 MPH winds for 24 hours that the dish took in stride. We knew the issues were solved, our dish was substantially better than even in *new* condition in handling reception during bad weather. I was like…nature, BRING IT!

Ummmmm…be VERY careful what you wish for.

Around October 18, the GFS model began showing a significant ice storm in and around Oklahoma City a week out, where our dish is located. I laughed. No way…it’s only October! But days later, the European and other models sounded the alarm.

Nobody, including myself, was laughing now.

On October 25, I got a text from Ryan Hickman, our COO, saying that there was a “99% chance” that we would have to switch to our backup satellite dishes in the Midwest for our data feeds. 30 minutes later, I made the call: it was a 100% chance. 12 hours before the first freezing raindrops hit our dish, I flipped AllisonHouse’s National Weather Service (NWS) primary data feed over seamlessly to the backup satellite dishes, with absolutely no downtime, and completely transparent to all of our users.

Then the ice storm hit, with almost wrath of God intensity. As you can see from the graphic above, there’s no doubt what happened to our carrier to noise (aka “signal to noise”) ratio, and our signal strength after the storm hit. You can tell when the ice significantly and very rapidly accumulated on our dish, and when it was still pouring freezing rain. Reception quality of our NWS feeds from the dish went straight into the dumpster. And the forecasts based on data and the models were even more alarming for the upcoming 48 hours!

When the GFS was cranking out 2″ of freezing rain west of Oklahoma City as the event was about to start, again, some people were laughing. But after the first wave of freezing rain “outperformed”, or were much higher than the European, NAM and other models had forecast…once again, the laughter stopped. On Tuesday morning, October 27, at 9:40 AM, Ryan texted me again. The data center where our dish is located had lost power, and was on a backup generator to keep things working. Then people started walking around inbetween rain bands in Oklahoma City, and we all realized how bad this was, is, and likely would be. Would our essentially brand new dish still even work after this?

As the event continued, I watched our signal go down, then up, and then down after each passing freezing rain band. We only lost data for about 4 hours over the 3 day event, but we took no chances. We stayed on our backups through October 28, until the ice melted and I was confident we were good to go, even though the backup generator was still going. And through all this time, there was no delay or loss of any of our data feeds, or imagery. Our dish suffered no damage, and reception went back to normal after it stopped raining hard on October 28th.

Diversity is a term that is widely used, and sometimes misused, these days. But in terms of data, one of the things we are proud of is depth of data diversity. We don’t host servers and have data inject points at one data center; we host servers at multiple centers. If one goes down, or a satellite dish has issues, other ones kick in, most of the time automatically. In fact, we have been up more than our host this year, and they have been up 99.999% of the time! And where we can get redundancy, we have it. By running the latest proven data ingest and processing software, we get data reliably and VERY quickly…many times measured in a few seconds, or less…not minutes. That doesn’t mean we’re perfect; nobody is. But you can rest assured with us that when outages are likely or are occurring, and things are hitting the fan, we have the experience, staff and resources to get out of a jam quickly, keeping you informed and the data flowing reliably, when bad weather strikes.

Footnote: my prayers go out to all those affected by the Texas/Oklahoma ice storm of October 25-28, 2020…and category 2 hurricane Zeta in Louisiana, Mississippi and Alabama, which occurred at the same time. There is an awful lot of damage with the ice storm, and with Zeta in southeastern Louisiana, and significantly northeast all the way into central Alabama.

AllisonHouse dishes up repairs, and data from space!

AllisonHouse engineer Kevin Pirkle after repairing our dish. As you can see, it looks great! (Courtesy photo)

Some time ago, our dish got hit by lightning, and it was time for repairs. While we have been on a backup dish since that time, we needed to get our main dish back on the air. The backups are intended for that: backups. But after years of use, it was also time for an equipment refresh on our primary and main backup dish after being out in the elements for over a decade.

We replaced the cables, taped them up and had waterproof connectors put on. We peaked the dish for the best quality signal possible, and replaced the ground system as well, which saved us from getting lightning damage to our receiver and server. Just as importantly, our antenna LNB, which picks up the signal, got an upgrade as well. It uses the highest broadcast industry standards that broadcasters use for transmitting and receiving signals! It’s a real workhorse, and will hopefully serve us for a long time to come. This is important because we had been noticing more and more 5G signal interference creeping in, which has reduced our signal to noise ratio some (but still well above acceptable levels). This LNB filters out much of the unwanted 5G, and high-powered TV and FM signals that are getting stronger as new towers and upgraded transmitting sites are built nearby.

Up next: our “primary” backup dish. about 30′ from our main dish, will get the same refresh. This will ensure your data will be flawless for watches, warnings, and standard resolution Level 3 data.

And thanks to our new team member, Kevin, for doing the repairs! Lighning damage is no fun, but our goal is uptime, all of the time (except for brief maintenance), and with our system, we pretty much are there. It’s our commitment to you: very high reliable data and imagery without sacrificing quality, all the while keeping costs affordable to our users. And thanks to all of the engineers who keep everything working well every day!

July radar update

Level 2 TDWR data is expanding, and 6 replacement doppler radars will be installed in Canada this year.

Hello AllisonHouse users,

The rollout of Level 2 TDWR data, and the doppler radar replacement in Canada continues. Here’s an update on both:

1. The upgrades for bandwidth to handle level 2 TDWR radar data continues. At this time, sites are being upgraded as soon as they are able. Originally, TDWR TICH (Wichita), TOKC (Oklahoma City, and the two TDWR’s in Dallas/Forth Worth were the only sites to get Level 2 TDWR data. Now, a number of sites have it; check yours to see if your local area has them. St. Louis, New York and Chicago area TDWRs do not have super-res yet, but Milwaukee, Indianapolis, Washington DC, Boston, Pittsburgh and others now do. Be patient, they should all be online this winter!

2. The radar replacement in Canada continues. The United States replaced all our old radars of varying ages and genres, and added many more (more than double!) in the early and middle 1990s. Canada is now doing the same. 6 more old radars will be replaced by the end of 2020; they are being installed next to the old ones, or close by. When the installation is occurring, the old radar must be shut down so as not to harm the workers. Canada will replace all of the existing ones by summer 2023, while adding one radar in southern Alberta. The 6 being replaced this year include:

1. Shuler, Alberta
2. Dryden, Ontario
3. Holyrood, Newfoundland
4. Sainte-Francoise (Villeroy), Quebec
5. Woodlands, Manitoba
6. Val d’Irene, Quebec

There will be a training radar in Egbert, ON, coming online as well.

I must make one more note, however, of the TDWR data. The National Weather Service already sends out the data in super-resolution from the TDWRs in Level 3 format. This upgrade allows the full range of products in super-resolution on software that only displays Level 2 radar data, such as GRAnalyst.

Enjoy the upgrades, and I hope you are having a reasonably good summer, in spite of the pandemic!

Edit: Update: TMDW and TORD came online with level 2 data after a software update on July 29, 2020. Enjoy, Chicago area subscribers!