Data diversity and resiliency at AllisonHouse: making a great customer experience

The AllisonHouse pimary and backup satellite dishes for receiving weather data. Thanks to data resiliency, we are able to stay up, even when these dish antennas and ground system got hit by lightning last year. Since then, the antennas, coax and ground system were replaced and upgraded. Photo credit: Gilbert Sebenste/AllisonHouse

In my previous blogpost, I talked very briefly about data diversity and data resiliency. My boss wanted me to expound on that some, and how it affects you as a customer, improving your experience with us.

But first, a little background: the National Weather Service text products, satellite images from GOES-16 and GOES-17, many models, and level 3 radar products are sent via the Satellite Broadcast Network (SBN) from the National Weather Service. If you receive the feed outside of the National Weather Service, like we do, the same feed is given a different name: NOAAport (like how we call hurricanes in the eastern Pacific and Gulf/Atlantic Ocean, and typhoons in the western Pacific. It’s the exact same storm type, it just has a different name!). Anyone can pick up NOAAport with a large (think WAY bigger than a pay TV ) satellite dish, a receiver, cables, antenna (LNB for you tech geeks), support bracket and mast…and then a server in a data center to send it out to be processed. That’s expensive, but that’s one reason why we have to charge for the data. And, we have all of that equipment, top notch at that, to pick up the broadcast reliably. But why is having only one dish a bad idea?

Let me ask you a question: what would happen if lightning hit the antenna on the satellite dish, fried it along with the coaxial cable, and burned up the entire grounding system? That would make for a very bad day for us, right? Well, thanks to data diversity, you didn’t even know that it DID happen to us last year. Other dishes we have access to across the country kicked in and kept the data flowing without even a moment of data drop or loss (since then, we replaced everything but the receiver, server and dish, which were not damaged). This means that because of our data diversity, we are resilient to disastrous or catastrophic failures. We have redundancy to keep going with frequently no loss of data to the customer.

Data diversity means that if our satellite dish is affected by severe/bad weather, or a twice a year “sun outage”, the other backup dishes we have access to across the country kick in and automatically take over sending us data instantly. The end result is no downtime for us and you, the end user. In fact, when a feed goes down these days, typically the issue is outside of our control. It also means the same for our feeds coming in via fiber. We are one of a few data vendors have that have access to BOTH level 2 radar inject points from the National Weather Service. If one is down or delayed and the other is fine, you don’t notice a thing. Everything works just fine on your end. We get the data that is coming in fastest, process it, and then get it to you almost instantly.

Nearly 20 years ago, when AllisonHouse was new, outages would occur with software ingest crashes, network issues, power outages….our server was literally in Tyler Allison’s basement! In 2020, we have multiple servers all over the country in data centers that have copious amounts of backup power, with very high speed and redundant Internet connections. Blazing fast new servers, software ingest programs that are state-of-the-art and reliable, and multiple data ingest points along with excellent system administration mean very high reliability for our customers today. How high?

If you want to pay thousands of dollars every month for 24/7/365 service where an engineer is monitoring everything, we understand (and if you want that, just drop us an email!). Even so, our servers alert us 24/7/365 when one has a very rare issue, and we do stagger our work times during periods of severe weather in case something breaks. But, as a result of excellent equipment and system administration from Ryan Hickman, our CTO, we now consistently achieve 99.999% uptime on all of our services, and in some cases, that goes out 5 decimal points! And we’re constantly improving and frequently upgrading to make our equipment and our services the very best they can be.

Such redundancy and resulting reliability and resiliency should be priced around $300 a month for Level 2 data alone, according to the research we have done. But we want to bring reliable service at an affordable price to everyone! But because we like to eat and sleep in apartments and houses, we do have to charge for our data. So, having said that, we believe we have the best combination of value and service of any weather company, anywhere in the world. Give us a try…and for those who are already our customers, THANK YOU for placing your hard-earned money with us. We couldn’t do what we do without you, and we are most grateful and humbled for that.

And if you want to see our reliability in action, as an example, check out:

https://www.allisonhouse.com/pages/status/uptime

(Note: the National Weather Service had an 18 minute outage that affected EVERYBODY on November 12, and they were unable to send warnings and radar data to anyone, including WITHIN the NWS. We can’t work around that, as you might imagine).

So, if you are interested in extremely reliable data, AllisonHouse Maps, or helping you with making your weather software the best it can be, drop us a line at our support email address. We would love to help you out!

Behind the data: AllisonHouse stays on through an ice storm

The rreception of the primary National Weather Service data feed nosedives as the ice piles up on our dish. How did we prevent any delay or loss of data? Go behind the scenes with AllisonHouse.
The signal-to-noise ratio and signal strength craters as a record ice storm covers our satellite dish in ice in Oklahoma City, OK on October 26, 2020. This is how we stayed “on the air” with no downtime for our customers, even with a loss of the dish in parts of the event, and a total loss of commercial power.

In our last blog post, I highlighted our refurbished satellite dish. Well, as it turns out, it wasn’t what we had hoped for. We needed two more upgrades and a fix to take care of other issues we discovered after I published the article. In typical AllisonHouse fashion, we got that done. Now, would these upgrades work and keep the dish stable in extreme weather?

A week later, we had 40-50 MPH winds for 24 hours that the dish took in stride. We knew the issues were solved, our dish was substantially better than even in *new* condition in handling reception during bad weather. I was like…nature, BRING IT!

Ummmmm…be VERY careful what you wish for.

Around October 18, the GFS model began showing a significant ice storm in and around Oklahoma City a week out, where our dish is located. I laughed. No way…it’s only October! But days later, the European and other models sounded the alarm.

Nobody, including myself, was laughing now.

On October 25, I got a text from Ryan Hickman, our COO, saying that there was a “99% chance” that we would have to switch to our backup satellite dishes in the Midwest for our data feeds. 30 minutes later, I made the call: it was a 100% chance. 12 hours before the first freezing raindrops hit our dish, I flipped AllisonHouse’s National Weather Service (NWS) primary data feed over seamlessly to the backup satellite dishes, with absolutely no downtime, and completely transparent to all of our users.

Then the ice storm hit, with almost wrath of God intensity. As you can see from the graphic above, there’s no doubt what happened to our carrier to noise (aka “signal to noise”) ratio, and our signal strength after the storm hit. You can tell when the ice significantly and very rapidly accumulated on our dish, and when it was still pouring freezing rain. Reception quality of our NWS feeds from the dish went straight into the dumpster. And the forecasts based on data and the models were even more alarming for the upcoming 48 hours!

When the GFS was cranking out 2″ of freezing rain west of Oklahoma City as the event was about to start, again, some people were laughing. But after the first wave of freezing rain “outperformed”, or were much higher than the European, NAM and other models had forecast…once again, the laughter stopped. On Tuesday morning, October 27, at 9:40 AM, Ryan texted me again. The data center where our dish is located had lost power, and was on a backup generator to keep things working. Then people started walking around inbetween rain bands in Oklahoma City, and we all realized how bad this was, is, and likely would be. Would our essentially brand new dish still even work after this?

As the event continued, I watched our signal go down, then up, and then down after each passing freezing rain band. We only lost data for about 4 hours over the 3 day event, but we took no chances. We stayed on our backups through October 28, until the ice melted and I was confident we were good to go, even though the backup generator was still going. And through all this time, there was no delay or loss of any of our data feeds, or imagery. Our dish suffered no damage, and reception went back to normal after it stopped raining hard on October 28th.

Diversity is a term that is widely used, and sometimes misused, these days. But in terms of data, one of the things we are proud of is depth of data diversity. We don’t host servers and have data inject points at one data center; we host servers at multiple centers. If one goes down, or a satellite dish has issues, other ones kick in, most of the time automatically. In fact, we have been up more than our host this year, and they have been up 99.999% of the time! And where we can get redundancy, we have it. By running the latest proven data ingest and processing software, we get data reliably and VERY quickly…many times measured in a few seconds, or less…not minutes. That doesn’t mean we’re perfect; nobody is. But you can rest assured with us that when outages are likely or are occurring, and things are hitting the fan, we have the experience, staff and resources to get out of a jam quickly, keeping you informed and the data flowing reliably, when bad weather strikes.

Footnote: my prayers go out to all those affected by the Texas/Oklahoma ice storm of October 25-28, 2020…and category 2 hurricane Zeta in Louisiana, Mississippi and Alabama, which occurred at the same time. There is an awful lot of damage with the ice storm, and with Zeta in southeastern Louisiana, and significantly northeast all the way into central Alabama.

AllisonHouse dishes up repairs, and data from space!

AllisonHouse engineer Kevin Pirkle after repairing our dish. As you can see, it looks great! (Courtesy photo)

Some time ago, our dish got hit by lightning, and it was time for repairs. While we have been on a backup dish since that time, we needed to get our main dish back on the air. The backups are intended for that: backups. But after years of use, it was also time for an equipment refresh on our primary and main backup dish after being out in the elements for over a decade.

We replaced the cables, taped them up and had waterproof connectors put on. We peaked the dish for the best quality signal possible, and replaced the ground system as well, which saved us from getting lightning damage to our receiver and server. Just as importantly, our antenna LNB, which picks up the signal, got an upgrade as well. It uses the highest broadcast industry standards that broadcasters use for transmitting and receiving signals! It’s a real workhorse, and will hopefully serve us for a long time to come. This is important because we had been noticing more and more 5G signal interference creeping in, which has reduced our signal to noise ratio some (but still well above acceptable levels). This LNB filters out much of the unwanted 5G, and high-powered TV and FM signals that are getting stronger as new towers and upgraded transmitting sites are built nearby.

Up next: our “primary” backup dish. about 30′ from our main dish, will get the same refresh. This will ensure your data will be flawless for watches, warnings, and standard resolution Level 3 data.

And thanks to our new team member, Kevin, for doing the repairs! Lighning damage is no fun, but our goal is uptime, all of the time (except for brief maintenance), and with our system, we pretty much are there. It’s our commitment to you: very high reliable data and imagery without sacrificing quality, all the while keeping costs affordable to our users. And thanks to all of the engineers who keep everything working well every day!