Re: MIS Outage 6/17/2011
At 11:00 am est time, MIS had an outage on our primary Internet connection. This
outage affected all websites operated by MIS. It was caused by an Internet Service
Provider (ISP) experiencing a hardware failure. Once it was determined that the
ISP would not be able to correct the issue immediately, MIS followed procedure and
began switching to another connection.
As a backup, MIS maintains two additional Internet connections that are from other
vendors. Unfortunately, a second, unrelated failure occurred at the vendor which
maintains our DNS information. This is the primary means by which our clients are
able to connect to MIS. We were delayed in updating this information for a period
of one and half hours. This delay magnified the problem for our customers. Once
we were able to update the vendor information with the new Internet connections
all websites were again available. However, it takes anywhere from 10 minutes to
24 hours for a client’s DNS server to be updated. It has been our experience that
this process generally happens for most in the US within one to two hours, but at
that point on Friday, it put the recovery time at roughly three hours from the original
event.
We apologize for the inconvenience this problem caused. We do understand your frustration
in trying to submit and access your work. A similar frustration existed here as
we attempted to reestablish connectivity. Our current up-time for 2011, including
this incident is 99.9%, and to ensure that we are doing all that we can to serve
you, an analysis has be done of this crisis, as well as how best to deal with it.
Our number one priority is sustained availability by means of a a faster fail-over.
Within the next few days, we will be instituting a change that should reduce the
downtime, even in an extreme incident such as this one, to less than 5 minutes.
We thank you for your paitience and your patronage.
MIS IT Department