Site Slowness
Incident Report for DrChrono
Postmortem

Site Slowness

Summary

On June 1st, 2021, the EHR platform experienced intermittent slowness throughout the day. There were multiple but related causes for the slowness during the day.

Timeline (EST, 24-hour clock)

Date/Time Activity
2021-06-01 12:10 Support team started receiving tickets for site slowness.
2021-06-01 12:17 Ops team notified the Support Team about a spike in queue times.
2021-06-01 12:20 An uptick in errors to an API endpoint was identified.
2021-06-01 12:26 Identified an increase in latency and response times with a third-party API which DrChrono relies on for services to our clients. This latency also caused the DrChrono platform to have an increase in latency and response times.
2021-06-01 12:30 Ops team continued searching for additional causes.
2021-06-01 12:49 The Support Team indicated the issue was still present; additional tickets for general slowness received.
2021-06-01 12:52 Ops team investigated new tickets.
2021-06-01 12:58 Ops and Support teams continued investigating the issue. 
2021-06-01 13:19 Status page updated for incident.
2021-06-01 15:23 Members of the Engineering Team were added to the investigation efforts since the previous investigation started ruling out the underlying infrastructure as a root cause.
2021-06-01 16:29 A portion of the integration between DrChrono and OnPatient was disabled as a mitigation step.
2021-06-02 09:32 Disabled integration appeared to reduce the number of tickets.
2021-06-02 11:40 The Engineering Team began developing a hotfix to improve performance related to the disabled component in the DrChrono and OnPatient integration.
2021-06-02 14:00 Status Page updated and incident resolved.

Contributing Factors

Several factors were affected the site performance during this time:

  1. Increased traffic on the day after a holiday.
  2. The decrease in performance of a third-party API causing an increase in latency for DrChrono customers.
  3. The increase of appointment notifications between DrChrono and OnPatient causing decreased performance for DrChrono customers.

These factors contributed to each other and caused a cascade of performance degradation that eventually highlighted the issue of the appointment notifications from DrChrono to OnPatient as an underlying cause.

Impact

Customers with a large number of appointments scheduled in the EHR app were likely to experience general slowness issues in the EHR app.

Corrective Actions

The release of the hotfix developed by the Engineering Team should prevent this exact situation from occurring in the future.

Posted Jul 06, 2021 - 10:46 PDT

Resolved
This incident has been resolved. We will create a postmortem with details about today's incident within the next week. If you're still experiencing any system performance issues, please let our Support team know, and they will assist you.
Posted Jun 02, 2021 - 14:13 PDT
Monitoring
Our engineering team has pushed a temporary fix to address the reports of sitewide slowness and trouble saving data. We are seeing normal response times and will continue to monitor the system. Please reach out to our support team if you continue to run into any issues.
Posted Jun 02, 2021 - 12:02 PDT
Update
Our engineering team continues to investigate the cause of sitewide slowness and trouble saving data. We do not have additional updates at this time but will provide more information once it is available.
Posted Jun 01, 2021 - 14:05 PDT
Update
Our engineering team is continuing to investigate the cause of sitewide slowness and trouble saving data. Once we have additional information on the issue, we will provide an update here.
Posted Jun 01, 2021 - 12:53 PDT
Update
Our engineers are continuing to investigate the cause of sitewide slowness and trouble saving data. We will continue to provide updates here as we receive them.
Posted Jun 01, 2021 - 11:30 PDT
Investigating
We are currently investigating reports of sitewide slowness and trouble saving data that appear to have begun at approximately 9:00 am PST. We will provide an update with additional information as soon as possible.
Posted Jun 01, 2021 - 10:18 PDT
This incident affected: drchrono.com, drchrono iPad EHR, and drchrono iPad Check-In Kiosk Application.