On Friday and Saturday, March 30 and March 31, 2001, the TrainWeb servers experienced the longest outage that we have ever experienced in our 5 years of existence. The outage started late Friday afternoon and restoration wasn't until late Saturday afternoon. All websites and e-mail services and e-mail forwarding hosted on the TrainWeb servers were down during this periord. If TrainWeb is providing any e-mail services for you, any e-mail sent to you during that period was most likely returned to the sender as undeliverable. Some of the RAILcams continued to work during the outage as we have moved them to an off-site server to reduce the load on our own in-house internet feed.

The problem was not with our servers, but was rather with the internet feed into our computer room. In the past 5 years, the internet feed to the TrainWeb servers has been down very rarely. On those rare occassions, the internet feed has only been down for a few minutes, and even more rarely for an hour or two. We have been very pleased with the reliablity and quality of our internet feed, which actually predates TrainWeb and has been in service for almost 10 years.

Up until the last couple of years, our internet feed was provided by CerfNet, one of the original and largest internet backbone providers. Within the last two years, CerfNet was acquired by AT&T. Thus, our internet feed is now provided by AT&T. The local-loop from our local switching office to our computer room is provided by PacBell. Usually, an internet outage can be traced to a problem on the backbone or at a major switch and can be resolved without dispatching repairs crews to the field. In this case, the problem was in the field and a crew had to be dispatched to trace the actual line between the local switch and our computer room. Due to the problem occurring on a Friday evening, delays were created because crews on-call had to be brought in by the phone company and other people had to be located that could provide access for the phone company to various telephone utility rooms. The internet feed was finally restored by AT&T and PacBell late Saturday afternoon.

We apologize for any inconvenience this may have caused. We are in the process of doing what we can do to reduce the probability of further outages and to reduce the duration of any future outages that are unavoidable. Currently, all our servers are in the computer room that is in the location of a sister company to TrainWeb. We have installed a new T1 feed directly into the TrainWeb office. This will provide the TrainWeb staff with easy and convenient 24 hour access to all our equipment. AT&T and PacBell will also have easier access to intermediate connection points should a problem ever occur in the field again. We are looking forward to this transition taking place sometime within the next 60 days.