Posted: 2007-01-26 18:00:00
Today, when I got out of bed, and started up my system, zero of our hosted domains at Linode were resolving.
Tracking down the problem, I realized that while I could resolve the Linode.com nameservers [ns1.linode.com and ns2.linode.com] and ping their IPs — I could not get an answer to any record queries… Everything was dead in the water… Same effect as pulling the plug on the server.
I filled out a support ticket, and went to the Linode IRC channel. I showed that ‘dig @ns1.linode.com devside.net’ was not returning anything, while the NS server IP’s were reachable. Pretty quickly someone was working on it, and got it resolved.
I left the house for a few hours, only to come back to the same exact issue. Again, went to the IRC channel, notifying whoever was listening that the Linode nameservers were not responding to queries…
A very odd experience was about to follow that moment, a defining moment, a moment that separates your life into before, and after that moment…
No one seemed to know that the nameservers were freaking out, not answering queries for hours, maybe days at a time — for god only knows how long — maybe for the last two month? Hosted domains were not being resolved, nor did the users seem to care about that fact.
I asked something about the DNS issue, I don’t remember what. Suddenly, the users on the channel turned on me like a pack of hungry wolves. To them I was crazy, that I was demanding a solid DNS Service, that no one provides authoritative DNS Servers, and I must have been born with a silver spoon in my mouth for demanding some type of a $1200/month rackspace managed account service. They informed me that I should just go get a ZoneEdit account and run DeveloperSide.NET off it. That the DNS Service from Linode.com is beta and I should be grateful it even exists [never did I notice any beta labels]. I thought to myself whats next, when will I wake up from this dream? Could the provided Server IPs also be dynamic? Have I just moved DeveloperSide.NET to someone’s garage?
At that point, it really hit the fan, they started to come out of the woodwork from every direction. I was informed that there is no difference between an ISP/Host’s NS Servers and something you run yourself locally. I told them that many differences exist, the simplest one is the ping time to the Server’s IP. Here is an example…
ping ns1.linode.com Average = 34ms
ping ns1.dynside.net Average = 247ms
[ns1.dynside.net is at 90ms this time around — still 3 times the wait with all the uncertainty]
I was told that I have a serious misunderstanding of how DNS works. You can’t ping a nameserver, it does not answer to ICMP resource record queries.
Battling the mob, I managed to get through to someone at Linode who went working on the problem, again. It was fixed [something to do with Nagios taking up too many resources and the ‘oom killer’], but not before another problem occurred. This time, the devside.net zone would not get loaded to the nameservers. Some type of timing issue. Resolved after some time.
Not being able to rely on a host’s nameservers (with that notion being ludicrous IMHO), I’ve updated the list of nameservers with my domain registrar and have added ns1/ns2.dynside.net to the list. I have no idea how long DeveloperSide.NET has been down.
On top of all that, I’m still getting old cached dns data from some nameservers that still have it at ev1server.net. What a mess.
All in all, I cannot recommend Linode.com as a VPS provider at this time. As it turns out, their nameserver service _is_ beta and has only been implemented for a few months. To me, that just make no sense whatsoever, for a VPS provider — that has been in business for a while. I’m going to stick around, see what happens. But any more problems like this, and I’m gone.
It has been 4 months since the original post. Linode has been good ever since. A few hiccups here and there, but overall I have not had any major issues. Not bad for $20/month. And I was pleasantly surprised, when checking my account a few days ago, that it has been upgraded from 128MB to 256MB… Though I will need to backup and reboot before I get to allocate the extra ram.