Jul 30, 2018
To our
valued clients,
On Saturday,
July 28th, ASD experienced an issue with our database that caused a disruption
of service. The issue began sporadically on Saturday morning around 10:30 am
and ASD’s Technical Team attempted to diagnose and correct the problem. Between
10:40 am and 12:55 pm our system was restarted in different configurations four
times. Each time it came back and appeared to be working fine, but after
15-20 minutes would begin to fail again. During this time calls were still being
received and handled by ASD.
At 1:15 pm
the outage became more widespread and ASD began to initiate the company’s
Crisis Notification System to alert our clients that their phone lines may be
unprotected. This is the first time ASD’s Crisis Notification has been
activated since we experienced an issue with one of our carriers back in 2010.
(We now realize the wording should have been more precise on the Crisis
Notification to make it clear to our clients the issue was on ASD’s end and not
due to a problem with the funeral home phone lines).
At this
point our Technical Team discovered a particular database routine was running
and doing a full table scan which was clogging our system and causing the
failures. This routine was turned off. By 2:10 pm agents began to
take calls again. By 2:27 all agents were in taking calls. After 45
minutes of running without problems and ensuring all systems were fully
restored, the “System Restored” message was sent to all clients.
We are
deeply sorry for any inconvenience or problems this issue may have caused our
clients and their callers. Our answering service is committed to providing the
most reliable answering service for the funeral profession and we will take
measures to ensure this unfortunate event does not recur. Our company has
invested in
many redundancies to prevent a disruption of service.
In response to
this outage, ASD has discovered the full table scan was initiated because an
SQL database index faulted. We have never seen a faulted index before in
the 10+ years of using SQL. We are investigating this fault and trying to
prevent any future faults by utilizing new software management tools, which
focus on preemptive error detection.
ASD also plans to update our Crisis System with a more precise message. In the future, the message will state, “This is a Real Crisis Alert. ASD is unable to answer phone calls because of technical difficulties. Please make sure your phone lines are not call forwarded to ASD until further notice.” Our Crisis Notification was first created in 2010 and before this past Saturday, we never needed to initiate the system.
While we are hopeful the safeguards we have in place will prevent us from ever needing to utilize this system again, if another outage were to occur we want to ensure our clients are notified without any delay. One way to verify this is to take a moment to review the Crisis Contact Information listed for your funeral home. To view this, just click here and enter your login information.
You can also visit myASD.com, login and click on the ‘Crisis Notify’ link at the bottom of the options menu. In the event of a system outage, every phone line, cell phone and email listed on the account’s Crisis Contact sheet will be notified.
In order to receive timely mobile notifications during an ASD crisis, you must opt-in to ASD’s Crisis Notification Text Alerts. This is a requirement of the FCC.
- Your cell phone number must be listed on your funeral home account’s Crisis Contact information.
- You must text the word “Alert” to 22300. You will receive a reply from our Emergency Notification Provider.
Once you opt-in to ASD’s Crisis Notification Text Messages, you will receive an automated confirmation reply that looks like this
Following these steps will ensure that in the event of a total ASD system failure, we will be able to quickly communicate with our clients the scope of the emergency.
The outage on Saturday was very unfortunate and we know that it caused a lot of headaches for our clients, but we are very committed to learning from this experience and using it to make our answering service even more reliable in the future. Thank you to everyone for your understanding and patience. We have never been more grateful for our incredible clients!
Sincerely,
The Czachor
Family and ASD Staff
About The Author
Jess Farren (Fowler)
Jess Farren (Fowler) is a Public Relations Specialist and Staff Writer who has been a part of the ASD team since 2003. Jess manages ASD’s company blog and has been published in several funeral trade magazines. She has written articles on a variety of subjects including communication, business planning, technology, marketing and funeral trends. You can contact Jess directly at Jess@myASD.com
