Saturday, September 17, 2011

SeaStorm connectivity issues

I've received a few reports of SeaStorm refusing to download some (or all) storm data, instead giving the dreaded "Unable to download" message.  Last night I finally started seeing it as well.

I did some digging and there is a DNS issue afoot.  There are reports of sporadic access to the NHC site based on what DNS servers people are using.  At home I use Wifi on my Android devices, and when I switched my LAN's DNS resolver to openDNS (208.67.222.222) or Google DNS (8.8.8.8) instead of Comcast (75.75.75.75), the problem went away.  Here are the results using nslookup:

Google DNS:

$ nslookup www.nhc.noaa.gov 8.8.8.8
Non-authoritative answer:
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Name:    edge-rev.lb.noaa.gov
Addresses:  140.172.17.23
          129.15.96.23
          140.90.33.23
          140.90.200.23
Aliases:  www.nhc.noaa.gov
          edge-nws.woc.noaa.gov

Comcast DNS:

$ nslookup www.nhc.noaa.gov 75.75.75.75
*** cdns01.comcast.net can't find www.nhc.noaa.gov: Server failed
Server:  cdns01.comcast.net
Address:  75.75.75.75

IMPORTANT: While from these results it looks like Comcast is the culprit, it could also be a configuration problem at NOAA, which would explain why some other non-Comcast DNS servers are affected as well.

In the end, all web browsers and apps are at the mercy of working DNS in order to operate, so until either Comcast or NOAA fixes this problem, changing your DNS is the only workaround.  (Please note that something as simple as switching from Wifi to mobile data works too, depending on your carrier's DNS.  T-Mobile's DNS seems to be working fine, for example.)

UPDATE 1:  The problem appears to have been fixed:

$ nslookup www.nhc.noaa.gov 75.75.75.75
Non-authoritative answer:
Server:  cdns01.comcast.net
Address:  75.75.75.75

Name:    edge-rev.lb.noaa.gov
Addresses:  140.90.200.23
          140.172.17.23
          129.15.96.23
          140.90.33.23
Aliases:  www.nhc.noaa.gov
          edge-nws.woc.noaa.gov

UPDATE 2:  Aaaaand it's back.

UPDATE 3:  For those curious, the problem seems to be caused by NOAA changing DNSSEC keys faster than Comcast's DNS server cache was expiring.  They've manually flushed the zone for now, but hopefully they'll have a solution in place before the next cycle.

UPDATE 4:  Apparently this problem is never going to be fixed.  I've added an "Alternate DNS" option to settings to use Google's DNS servers for name resolution as a workaround.  It seems to have fixed the issue for most users, but not all.  This issue is a thorn in my side and I'll continue looking for a better solution.