On June 1st, 2021, the EHR platform experienced intermittent slowness throughout the day. There were multiple but related causes for the slowness during the day.
Date/Time | Activity |
---|---|
2021-06-01 12:10 | Support team started receiving tickets for site slowness. |
2021-06-01 12:17 | Ops team notified the Support Team about a spike in queue times. |
2021-06-01 12:20 | An uptick in errors to an API endpoint was identified. |
2021-06-01 12:26 | Identified an increase in latency and response times with a third-party API which DrChrono relies on for services to our clients. This latency also caused the DrChrono platform to have an increase in latency and response times. |
2021-06-01 12:30 | Ops team continued searching for additional causes. |
2021-06-01 12:49 | The Support Team indicated the issue was still present; additional tickets for general slowness received. |
2021-06-01 12:52 | Ops team investigated new tickets. |
2021-06-01 12:58 | Ops and Support teams continued investigating the issue. |
2021-06-01 13:19 | Status page updated for incident. |
2021-06-01 15:23 | Members of the Engineering Team were added to the investigation efforts since the previous investigation started ruling out the underlying infrastructure as a root cause. |
2021-06-01 16:29 | A portion of the integration between DrChrono and OnPatient was disabled as a mitigation step. |
2021-06-02 09:32 | Disabled integration appeared to reduce the number of tickets. |
2021-06-02 11:40 | The Engineering Team began developing a hotfix to improve performance related to the disabled component in the DrChrono and OnPatient integration. |
2021-06-02 14:00 | Status Page updated and incident resolved. |
Several factors were affected the site performance during this time:
These factors contributed to each other and caused a cascade of performance degradation that eventually highlighted the issue of the appointment notifications from DrChrono to OnPatient as an underlying cause.
Customers with a large number of appointments scheduled in the EHR app were likely to experience general slowness issues in the EHR app.
The release of the hotfix developed by the Engineering Team should prevent this exact situation from occurring in the future.