| To view this page in chronological
order click on these clock symbols. |
|
Friday March 23rd-Wednesday March 28th,
2001-Classification : Moderate
Incident Report #8 - E-mail Service Experiencing Intermittent
Problems
9:00 AM - E-mail service stopped processing mail. Upon
checking bandwidth logs we found that our network was receiving
a large amount of data which was using all available bandwidth.
All non-critical systems and services were shut down with no
affect. This data transfer lasted until 9:45 AM at which time
the transfer stopped and our e-mail server began processing
mail again. Further researched revealed that the cause of this
was an extremely large e-mail sent to one of our customers by
someone with a large amount of bandwidth, choking out all other
traffic. The customer was contacted and some settings were changed
on our server to prevent this type of incident from being repeated.
4:20 PM - E-mail service stopped processing mail. An
application on our mail server responsible for our browser e-mail
client interface began using 100% of both CPU's resources. This
service was stopped and restarted, restoring mail service. This
continued to happen intermittently for approximately 4 days
with periods between incidents varying from 2 hours to 12 hours.
Each incident resulted in loss of e-mail service for less than
15 minutes. This was not immediately reported to this page because
one of the possible causes for this type of issue could have
been a Denial Of Service(DOS) attack. This type of attack is
common on the Internet and consists of a single user flooding
a website with multiple requests until the system cannot keep
up. Once this possibility was ruled out by adding some additional
software to our network designed to prevent these attacks, we
began changing settings on the client software that the vendor
recommended until the program remained stable. We have had one
incident in the last 96 hours and we will continue to monitor
this application.
This Incident
Total downtime e-mail service - approx. 3 hours (6 incidents).
Totals Since 1/11/01 (Date this logging began)
Greenville Dial-up
Total downtime for incoming calls-5 hrs 17min (2 incidents).(99.7%Uptime)
Total disconnection incidents - 3.
Wilson Dial-up
Total downtime for incoming calls- 2 hrs 14 min. (99.9% Uptime)
Total disconnection incidents - 0.
High Speed Access
Downtime - 0 mins
Web Hosting
Total downtime for WWW1 - 0 minutes.
Total downtime for WWW2 - 0 minutes.
Total downtime for WWW3 - 5 hours 34 min (2 incidents).(99.7%
Uptime)
E-mail Service
Total downtime - 3 hours (99.8% Uptime)
Friday - March 2nd, 2001 - Classification
: Minor
Incident Report #7 - Newsgroups Not Responding
9:00 AM - We received several e-mails listing problems
with our Newsgroup service. We purchase this service from another
company for our customers. We contacted this company and found
that they have moved their servers. We then updated our server
to reflect the new information, but this change may take several
days to process due to the nature of the server. If you would
like to be able to access these groups immediately you can make
the following changes:
If you were using news.geeksnet.com, open the properties
of that Newsgroup, go to the servers tab and replace news.geeksnet.com
with 209.25.157.130.
If you were using anonymous.geeksnet.com, open the
properties of that Newsgroup, go to the servers tab and replace
anonymous.geeksnet.com with 209.189.89.243.
If you were using binaries.geeksnet.com, open the properties
of that Newsgroup, go to the servers tab and replace binaries.geeksnet.com
with 204.181.81.3.
If you were using spamkillers.geeksnet.com, open the
properties of that Newsgroup, go to the servers tab and replace
spamkillers.geeksnet.com with 204.0.78.21.
If you do not make these changes the service will begin to
work sometime within the next few days. If you have any questions
concerning Newsgroups please check http://www.geeksnet.com/support/usenet.html.
We have updated our contact info with this company, and should
receive notices in the future of any changes they are making.
Wednesday - February 28th, 2001
- Classification : Notice
Scheduled Maintenance
We have been notified by Sprint that they will be conducting
scheduled maintenance on their Greenville switch on Monday March
12th, 2001 between 2 am and 6 am. This maintenance will intermittently
affect Internet access in the Greenville and Wilson areas during
that time.
Wednesday - February 28th, 2001
- Classification : Severe
Incident Report #6 - Wilson Dial-up Customers Unable to
Authenticate
6:45 AM - Our primary authentication server failed to
authenticate dial-up users due to software errors. Our Greenville
dial-up racks transferred authentication to our secondary server
and service was not interrupted for those customers. The Wilson
racks did not transfer authentication and rejected all access
requests.
8:56 AM - We were notified by a customer that he could
not log onto the Internet through the Wilson dial-up. It took
approximately 5 minutes to determine the problem and adjust
the settings on the Wilson racks to transfer authentication
to the secondary server. Users were immediately able to log
on. Additionally it was found that our Geeksnet voicemail/paging
system was unplugged from the phone line and was not answering
calls. This resulted from a technician using the line during
the day on Tuesday for troubleshooting purposes - a new company
policy now restricts the use of this line for anything other
than the Geeksnet voicemail system. By studying our usage logs
for the Wilson dial-up it seems that the authentication was
functioning last night and early this morning, although there
may have been intermittent problems. We apologize for the lapse
in service and especially for the absence of the voicemail/paging
service.
This Incident
Total downtime for incoming calls- 2 hours 14 minutes.
Total disconnection incidents - 0.
Totals Since 1/11/01 (Date this logging began)
Greenville Dial-up
Total downtime for incoming calls- 5 hours 17 minutes (2 incidents).
Total disconnection incidents - 3.
Wilson Dial-up
Total downtime for incoming calls- 2 hours 14 minutes.
Total disconnection incidents - 0.
High Speed Access
Downtime - 0 mins
Web Hosting
Total downtime for WWW1 server- 0 minutes.
Total downtime for WWW2 server- 0 minutes.
Total downtime for WWW3 server- 5 hours 34 minutes (2 incidents).
Monday - February 26th, 2001 -
Classification : Minor
Incident Report #5 - Web interface for E-mail goes down
6:42 AM - An application error by our mail server web
interface software caused our online mail interface to go down.
This did not affect any other mail services - mail was still
being processed and was available through client programs.
9:37 AM - We were notified by a customer that he could
not access his mail via the web interface. The page was checked,
found to be down, and the service was restarted. This is the
first incident involving this software. We have been using it
since December 1999. No changes or upgrades are planned - we
will monitor this package for future issues.
Thursday - February 22nd, 2001
- Classification : Moderate
Incident Report #4 - Web Server goes down
4:42 PM - An application error by our remote access
software caused one of our webservers (WWW3) to go down. This
server hosts approx. 40 websites. These sites were not available
until the server was restored.
5:16 PM - We were notified by a customer that her webpage
was down. The server was checked, found to be down, and was
rebooted. This server will be added to our immediate notification
system in the near future. Additionally the remote access software
will be replaced with a different package.
This Incident
Total downtime for www3 server - 34 minutes
Totals Since 1/11/01 (Date this logging began)
Greenville Dial-up
Total downtime for incoming calls- 5 hours 17 minutes.
Total disconnection incidents - 3.
Wilson Dial-up
Total downtime for incoming calls- 0 minutes.
Total disconnection incidents - 0.
High Speed Access
Downtime - 0 mins
Web Hosting
Total downtime for WWW1 server- 0 minutes.
Total downtime for WWW2 server- 0 minutes.
Total downtime for WWW3 server- 5 hours 34 minutes (2 incidents).
Monday - February 19th, 2001 -
Classification : Notice
Webtrends Server Goes Operational
We have added a new server that will run Webtrends software
to analyze traffic to the web sites which we host. The Webtrends
software provides our hosting customers with information about
their websites such as number of hits, most popular pages, number
of return visitors, regions where visitors are from, busiest
time of day, as well as over 100 additional items. This service
can be setup for any of the sites we host. Please contact our
office for information regarding this service.
Monday - February 12th, 2001 - Classification
: Moderate
New Line Issues Solved
6:00 PM - We reached our dial-up capacity at our Greenville
location previous to the recent upgrade at 6:00 PM and immediately
began having busy signals.
6:15 PM - An additional check of all of the recently
installed components showed all of our equipment was functioning
properly. We placed a call to Sprint's Network Operations Center
to open a trouble ticket.
7:43 PM - Sprint corrects their configuration problem
and the new modems and phone lines start accepting calls.
Friday - February 9th, 2001 - Classification
: Notice
Upgrade Complete
6:00 PM - The new modems and phone lines have been installed
at our Greenville dial-up and have tested o.k. Because of how
this equipment works with Sprint's telephone lines we cannot
run an actual test until all of the previous modems are being
used. Therefore we will not have a final confirmation that these
new lines are working until the next time our previous capacity
is reached. This is anticipated to happen on Monday evening
at approx. 8PM.
Thursday - February 8th, 2001 -
Classification : Notice
Repairs Completed to WWW3 Webserver
2:15 PM - The failed hard drive was replaced and the
mirror was reconstructed, restoring fault tolerance. See the
entry made on January 30th for more information regarding this
incident.
Monday - February 5th, 2001 - Classification
: Notice
Scheduled Maintenance Completed
9:15 AM - The scheduled server maintenance for the server
room at our Winterville Location was completed. Our dial-up
service, mail server, and WWW1 webserver were restored to service
at 6:10 AM. All other servers were brought back online by 7:15
AM.
Wednesday - January 31st, 2001 - Classification
: Notice
Scheduled Server Maintenance
There will be a scheduled maintenance on Monday morning, February
5th, 2001, from 3:00 AM to 6:00 AM EST. All services will be
temporarily unavailable during this period.
This time will be used for routine preventative maintenance
and reorganization of equipment. This maintenance is important
for us to continue providing the best quality of service to
our customers.
Tuesday - January 30th, 2001 - Classification
: Moderate
Incident Report #3 - Hard Drive Failure on Web Server
5:30 PM - The primary hard drive of our www3 webserver
failed causing the system to go down. The mirrored backup drive
was promoted to the primary and on automatic reboot the system
paused for a user input. This system is not on our critical
alert notification system.
9:30 PM - We were notified by a customer that his webpage
was down. The system would not respond remotely so a technician
was sent to the server.
9:38 PM - The technician arrived at the server and brought
the server back up. This system hosts approximately 40 websites.
Due to this failure these sites were not available for 4 hours.
Additionally this server will have to be brought down for repairs
at least twice in the next couple days. We hope that each repair
session will last less than 30 minutes.
This Incident
Total downtime for www3 server - 5 hours (including repairs)
Totals Since 1/11/01 (Date this logging began)
Greenville Dial-up
Total downtime for incoming calls- 5 hours 17 minutes.
Total disconnection incidents - 3.
Wilson Dial-up
Total downtime for incoming calls- 0 minutes.
Total disconnection incidents - 0.
High Speed Access
Downtime - 0 mins
Web Hosting
Total downtime for www3 server - 5 hours
Friday - January 19th, 2001 - Classification
: Notice
Upgrade
We have ordered a new modem card and phone lines for our Greenville
dial-up. This upgrade will increase our capacity at that dial-up
by 25%. Sprint takes about 10-15 business days and we will post
on this page as soon as those upgrades are complete. We apologize
for any busy signals you may get during this period - usage has
increased faster than expected.
Friday - January 12th, 2001
- Classification : Severe
Incident Report #2- Loss of dial-up authentication and
total disconnection
6:45 PM - The total control rack in Greenville stopped
responding. This equipment is responsible for answering dial-up
calls and authenticating account info. This did not immediately
affect users already connected. No new connections were allowed.
7:15 PM - The rack was not responding to any remote
administration so a technician was dispatched from our Winterville
location to the Greenville Central Office (C.O.)
7:30 PM - Technician arrives at the C.O. and restarts
Total Control Rack. This disconnects all users.
7:40 PM - Rack completes reboot and accepts incoming
dial-up calls for 5 minutes. Troubleshooting begins.
11:45 PM - We determined that an I.P. violation by one
of our static I.P. customers was causing the ethernet port of
the total control rack to stop responding. The user was removed
from our authentication and all users were able to connect.
This error was responsible for the issue at 9:30 am as well
as intermittent problems throughout the day. A software incomparability
between 3com's HyperArc manager and the Total Control Rack also
contributed to the downtime.
This incident
Total downtime for incoming calls at Greenville dial-up- 5 hours.
Total dial-up customers disconnected during reboot - 18 users.
Totals Since 1/11/01 (Date this logging began)
Greenville Dial-up
Total downtime for incoming calls- 5 hours 17 minutes.
Total disconnection incidents - 3.
Wilson Dial-up
Total downtime for incoming calls- 0 minutes.
Total disconnection incidents - 0.
High Speed Access
Downtime - 0 mins
Friday - January 12th, 2001 - Classification
: Severe
Incident Report #1- Loss of dial-up authentication and
total disconnection
9:32 am - The total control rack in Greenville stopped
responding. This equipment is responsible for answering dial-up
calls and authenticating account info. This did not immediately
affect users already connected. No new connections were allowed.
9:34 am - The rack was not responding to any remote
administration so a technician was dispatched from our Winterville
location to the Greenville Central Office (C.O.)
9:45 am - Technician arrives at the C.O. and restarts
Total Control Rack. This disconnects all users.
9:49 am - Rack completes reboot and accepts incoming
dial-up calls.
This incident
Total downtime for incoming calls at Greenville dial-up- 17 minutes.
Total dial-up customers disconnected during reboot - 34 users.
Totals Since 1/11/01 (Date this logging began)
Greenville Dial-up
Total downtime for incoming calls- 17 minutes.
Total disconnection incidents - 1.
Wilson Dial-up
Total downtime for incoming calls- 0 minutes.
Total disconnection incidents - 0.
High Speed Access
Downtime - 0 mins
Thursday - January 11th, 2001 - Classification
: Notice
State of the Network
Our Greenville Dial-up is operating near its capacity. Our user
to modem ratio is still under 6 to 1 - typical ISP models have
a ratio as high as 7 to 1 and many operate higher than that for
short periods. We have experienced some busy signals for a 15
minute period on several evenings over the last 2 Weeks that are
unusual for our ratio level. We will be ordering more modems and
phone lines in the very near future. We appreciate your patience
as we experience these growing pains.
Our Wilson Dial-up was recently upgraded and should not have
any busy signals for some time to come!
Our overall network bandwidth utilization is under 50%. This
means that our customers should consistently have the fast data
transfers our service is known for.
We have recently discovered a programming error in our billing
software that affected approximately 60 Geeksnet customers. We
have worked with the software vendor to correct the issue and
all affected customers have been notified.
Recent Issues
On Monday January 1st at approximately 10 PM we had a network
hub fail in our server room which affected user authentication
and mail services. Any users on the network before the failure
were not affected other than the mail service, however any new
connections were unable to authenticate and were disconnected.
The hub was replaced at the following morning at 8:20 am. This
is the longest network outage in over 6 months. This incident
also showed us a flaw in our technician notification system. We
have modified a setting on this system to correct this problem.
On Wednesday January 3rd the system which acts as the phone
mail system had a hard drive failure. This disabled our answering/paging
system for 2 days. We apologize for any missed calls.
Gee-News Archive List:
home 
