Heads down: Level 2 radar data outage…but not for us!

WSR-88D Radar
Will your screen go blank in the third week of January? Photo courtesy: NOAA/NSSL

Over the next few days, if you are a weather weenie, you’ll hear the drumbeats that the entire National Weather Service data center in Boulder, Colorado, including their web farm there, will be without power for nearly 24 hours as they do emergency repairs and upgrades on January 21 and January 22, 2021. The NOMADS server, which hosts free level 2 radar data and model guidance (and a lot more than that!), will be down for two days. If history is any guide, it could be significantly longer than that. This could affect any other free sites, which could be overwhelmed by the additional traffic.

So, does that affect AllisonHouse? The answer is…

No, except for model guidance.

We get the Level 2 data via fiber.  If you have someone you know who needs level 2 radar data on those two days, there are other free servers…but we have the fastest and newest ones for you to use. Our data feed gets us our Level 2 data within 2 seconds of when it leaves the National Weather Service forecast office, and in many cases, the delay is under one second. If you don’t want to take a chance of your radar screen going blank, get a subscription to AllisonHouse. No contracts…pay for a month and see if you like us. We think you will!

Otherwise, you might see a blank screen from your local radar site for at least a few days…

It’s (beyond) time for another National Weather Service modernization

GOES-17 satellite artist image
The National Weather Service must be modernized again soon, or it risks catastrophic data and service outages to the public.

Back in the 1990s, there was big buzz in the weather world about the “National Weather Service (NWS) Modernization”. I used to have a map of that which explained what was going to happen at every site across the country. But it basically broke down to this:

1. National Weather Service offices (WSO’s)…small ones which were there to augment the main Weather Forecast Offices (WFO’s) for every state were to be closed, and most offices were either moved, or started as new. Example: the Peoria, IL and Springfield WSOs were closed. But, WFO Lincoln, IL spun up to serve both areas. The Rockford, IL WSO was closed, and the Chicago WFO took over for their warning area of responsibility.

2. ASOS. Back in my college days, I was unnecessarily snarky about these things going in, saying that there would be problems and they wouldn’t work well (I was halfway right). Paid weather observers would all be canned, and these automated stations would do better than a manned observation (at major airports, as we now know, that’s a bad idea, but every year, we’re getting closer to that reality).

3. WSR-88D’s. Out with the WSR-57’s and WSR-74’s. We complain about coverage holes today, but if you are under 30, you would croak at the radar coverage, or lack thereof if you saw a coverage map of what we once had!

4. Modern buildings or facilities, weather balloon sites, and more. It was great!

Fast forward to December, 2020. The National Weather Service is pumping out more models than ever, more data than ever, more forecasts than ever…but they haven’t gotten bigger pipes and more and better servers to handle it. Two days before I wrote this article, all NWS servers didn’t have updated forecasts on their website for most of the day. A few weeks prior to writing this article, a server went down which made the NWS forecast offices unable to send warnings to their main center in Washington, DC. Several severe thunderstorm warnings never got out on time, except via manual override on NOAA Weather Radio. And, NOAAport, the flagship National Weather Service data feed, which we receive via satellite at AllisonHouse, is nearly saturated and cannot carry most of the products to its offices at full/high resolution today.

Furthermore, if NOAAport dies, so does NOAA Weather Wire Service (NWWS), an all-text product feed that was designed to never go down and is subscribed to by the media and those who really needed it for urgent alerts. EMWIN was designed for emergency managers, but it is slow, and if NOAAport dies, the text warnings and other products on that feed go down as well. Even worse, the way the system is set up now, if NOAAport dies, *NO* NWS website will update, as the feed is received via satellite…even at the same complex where it is sent out, and at their web farms! And, the NWS public websites are within the NWS computer network, meaning forecast offices share bandwidth with you and me to get surface observations, watches and warnings, upper air and satellite data, models and super-resolution radar.

A lack of vision, and simple planning and resource allocation, has gotten the National Weather Service to this point at the end of 2020, requiring them to do some potentially disastrous data throttling to the public (see my previous blog). And while I’m doing some finger pointing here, and it’s well deserved…wise people have told me if you can’t offer a solution, in most cases…don’t complain. Well, I’m complaining, I’m not backing down, and I DO have many answers to fix the very serious and major problems in the National Weather Service, and here they are for your perusal.

I’ll try to put the most critical issues and how to handle them in the list below, not necessarily in any order, except for #1.

1. A scalable, redundant, content delivery network (CDN).

https://en.wikipedia.org/wiki/Content_delivery_network

This is just standard practice these days. If you have a website or data and people log in to get it, servers automagically ramp up to handle the server load in diverse data centers across the country, AND it handles the increased bandwidth needs automatically as well. This would all be OFF the National Weather Service internal network. No outside traffic (save for VPN’s) are allowed on their private network which has its own internal data feed and servers! This is how most companies do it these days. One time, we had a brief, very partial website outage, but all of the major components stayed up, and none of our data went down, and that part of the website which did go down was restored quickly. We were and are able to easily—and automatically—route practically everything instantly to another server.

During a “clear day” across the country, we have fewer data servers running for our customers; this saves money. On outbreak days, we can spin up large numbers of servers automatically, on the fly, up and running in seconds, based on our criteria, and nobody notices a thing, except the customer sees our connections and data as FAST and RELIABLE…always! Again, most businesses do that these days.

1a: This means NOAAport gets off of satellite, and goes to a fiber-only feed. Data from it goes on the CDN’s for public use, and is accessible by educational institutions, and weather companies who demonstrate need. Sorry, Boy Scout project, this is off the table; use the public servers which can deal with it. NWS, Weather Enterprise companies (regardless of size), and serious weather weenies only get this raw data feed.

1b: That includes radar data, including the new radar interface.

2. Move rural WFO offices to major cities or suburbs that can support 1 and preferably 10 gigabit network connections. OK: I live in a rural area. With technology, this is no longer a slap in the face to me except for one thing (and read on for that). And every office gets 1 gigabit portioned such that more than adequate amounts are used for data transmission and reception, and the rest for office use. This means 1 gb up AND down…fully diplexed fiber. No cable modems, no 1 gigabit down/20 megabits up. Dual parity, end of story. Radars would be connected by fiber from their current locations (or could be moved if it would prove to be helpful).

3, In response to #2, someone will be watching storm chaser, spotter live streams, and webcams. And if you are in a Critical Weather Day (CWD) designation, instead of being told you cannot do this due to bandwidth limitations, you will be told you MUST do this, as appropriate, and given the bandwidth and staffing or volunteers to do so. Not being in a rural area has handicapped WFO’s with major population centers who don’t understand what is happening in rural areas.

4. The concept of “Virtual WFO”. This is 2021 (in a few weeks, as I type this). NO WFO should be allowed to continue operations when the life/property threat is high, and everyone else has evacuated. This is uncalled for with technology we have today. I can run AWIPS 2 on my 7 year old laptop after I put in a solid state drive and 16 GB of RAM. If you are at NHC, WFO Miami, New Orleans, Key West, Houston, Charleston, etc etc…and the entire metro is being evacuated including your office due to a cat 4/5 hurricane, it’s time to leave. A hurricane watch means the WFO spins down in hours, and staff are temporarily moved/housed at a new WFO after their families are taken care of, or they become a virtual member to a cooperating WFO. Work from a different office; or via VPN…working remotely IF the forecaster is able to do so. The 2020 pandemic is proof of concept that this can work! This goes for major earthquakes which could take out one or more WFO’s, even more so if a major tsunami is generated. To be blunt: what good is a dead meteorologist, or a meteorologist with a dead family because they couldn’t be taken care of by the mother or father properly? Putting forecasters in harms way unnecessarily needs to stop! This is completely unnecessary and absurd in 2021.

5. NOAA Weather Radio with GPS. Polygon data is transmitted in the digital bursts. You should be able to know, down to roughly 200′, whether or not you are in a watch or warning, or a disaster or emergency declaration.

6. Automated upper air soundings twice daily from EVERY WFO to better capture mesoscale environments. Or, maybe not every WFO, but more in areas that need more, if we can’t afford every launch from every office.

7. Every WFO should have full access to the CBS, NBC, ABC, and FOX TV networks, and from every TV market they serve. For example, in Chicago/Romeoville, their county warning area (CWA) includes every county in the Chicago DMA (Designated Market Area, or TV market based on counties the stations in the market serve most frequently), most counties in the Rockford, IL DMA, and one county in the Springfield/Decatur/Champaign market. In the Davenport, IA, forecast office, they would get a feed from the Quad Cities, and also the Cedar Rapids, IA, Waterloo, IA, Rockford, Illinois, Quincy-Hannibal, MO, and Kirksville, MO television markets. To that end, the feds should work with DirecTV or Dish Network or YouTube or Locast.org if they cannot get them all with an antenna. And, at least one TV antenna would be mandatory at all offices where they can get signals from at least one market, so that they can receive the broadcasts if IPTV, satellite or cable TV is down. And, a monitor “wall” to be able to monitor them at all each WFO.

8. Skycams with full pan-tilt-zoom (PTZ) remote capability should be set up, particularly in rural areas where spotters can be few. Have them be hosted by public schools. Or, if a TV station or private entity has a bunch of them, work with them to gain access. This includes state Department of Transportation (DOT) cameras.

9. New dual-polarization radars set up near Kirksville, MO; one in southeast Oklahoma, Charlotte, NC, one in northern Minnesota and a few other critical gaps.

Getting these would modernize the Weather Service in 2021, would handle future needs, thanks to scalability…and continue world-class service for every American. The core function of every government that ever has, is, or will exist from a country is to minimize loss of life to the extent reasonably possible. By doing these things, the National Weather Service will be in a great position and will be enabled to carry out its sworn duty to protect life and property in the United States of America with the necessary tools of our age to do so. My sincere hope is that these, or other reasonable proposals, can be established to get out of the crisis that the National Weather Service finds itself in today, and will find itself even moreso in the very near future,

Your constructive feedback is welcome! What would you like to see that I missed?

The National Weather Service wants to seriously throttle data usage on their websites

This slide from a National Weather Service conference call in early December of 2020 was the shot across the bow to National Weather Service website users, the private sector, app users, the media, and more. 25% of users could be left in the cold, including those seeking model guidance, and app users relying on National Weather Service websites as they run out of bandwidth. (Graphic courtesy National Weather Service)

Back in November of 2020, the National Weather Service (NWS) sent out a stunning administrative message on its primary data feed: it was running out of bandwidth, and would have to start throttling usage of all NWS and NWS-related *websites* for heavy users, or else they could all become unreliable. They set a conference call about it on December 6, and it was a wild one. The gist of it can be found here:

https://www.washingtonpost.com/weather/2020/12/09/nws-data-limits-internet-bandwidth/?fbclid=IwAR0GZZWPlChwE8WOtxbNIM5xxLfQN7b8vMxSnp2llO11ITahRGUw2z5mjik

Starting in the first quarter of 2021, they would start throttling heavy users. This meant, in the words of the National Weather Service, that 25% (1/4, ie, one quarter) of its users would be left out in the cold.

This would target two groups of people: app users and developers, and those downloading a lot of model guidance (HRRR, NAM, GFS, etc). Some fit into both of those categories. And it would target the private sector, who uses a lot of that data to help others make life-saving decisions (including AllisonHouse). Needless to say, this was met with a lot of concern, and scorn.

I won’t get into the politics of it, but as an AllisonHouse subscriber, you deserve to know if your data will be missing, or delayed. The rest of this article is to transparently tell you how we won’t be affected.

1. We receive the primary National Weather Service data feed via a satellite dish in Oklahoma City. On that feed come surface and upper air data, GOES (and other) satellite imagery, level 3 radar, a considerable number of MRMS products (all of the heavily used ones), some high-resolution model data. This includes all watches/warnings/advisories from the National Weather Service. Thus, this data will be unaffected by any website issues.

2. We receive Level 2 (and derive from it super-res level 3) radar data, FAA METARs/surface data, MRMS radar, MADIS, mesonets, earthquake, fire, MPING and Spotter Network data via fiber that does not go through any of their websites. Thus, this data will be unaffected.

3. Models. That’s really the only concern. Or is it? I asked our CTO, Ryan Hickman, about it….and completely understanding the issue, he simply said “There’s nothing to be concerned about”. We have the tech, the tools, the data, great people, and the data channels to avoid any outages or delays from National Weather Service data throttling on websites. We simply bypass them.

So, current customer, you have no worries; we have you taken care of. If you are an app developer, and are worried you’ll be left with a blank radar display, or without reliable or timely watches and warnings, talk to us. We provide or make available extremely reliable radar data and warnings to popular apps like RadarScope, GRlevel3, RadarOmega, and more! And if you are concerned, Emergency Manager, or storm enthusiast, that your data will be delayed or missing when it really matters, talk to us. We don’t have server limitations or bandwidth issues! Head to our support pages and send us an email. We love to hear from current and also potential customers!

Until next time, stay 6′ apart from those outside your home, electrified fences, and those who put ketchup on hot dogs in Chicago. And have a Merry Christmas, Happy New Year, and whatever holiday you celebrate!