Scheduled Operation - Potential Issues with saving
Incident Report for DrChrono

Root Cause Analysis: 05/07/2022 Scheduled Operation - Issues with saving


During a scheduled database operation, customers were unable to save appointment, billing, and chart information.


Date/Time (All PST) Activity
2022-05-07 10:20 Scheduled index creation operation started.
2022-05-07 10:20 Operations team monitored index creation activity.
2022-05-07 10:54 Support team started to receive tickets about inability to save appointment, billing, and chart information.
2022-05-07 11:32 Operation is completed on the primary database server and is initiated on the read replicas. This blocks all other updates on the replicas and causes replication lag to grow.
2022-05-07 11:38 Status page created to reflect identified status.
2022-05-07 11:54 Status page updated to reflect progression of index creation.
2022-05-07 12:21 Status page updated to reflect the near completion of the index creation.
2022-05-07 13:03 Status page updated to monitoring.
2022-05-07 13:50 Status page updated to resolved.

Contributing Factor(s)

The scheduled index creation that was ran caused our databases to block (and queue) any other transactions from occurring until the index was fully created.


Customers were unable to save appointment, billing, and chart information.

Corrective Actions

Moving forward, all database scheduled operations will use a different schema change tool and this will ensure tables can be altered without blocking read or write operations. Additionally, scheduled operations will have maintenance pages in place during the scheduled operations windows as well as be scheduled outside business hours or when traffic volume is typically low.

Posted May 13, 2022 - 06:52 PDT

This incident has been resolved. We apologize for the inconvenience caused today. A post-mortem will be available via this status page incident within the next week.
Posted May 07, 2022 - 13:50 PDT
Thank you for your patience. All work has been completed and the platform should be fully operational now. We will continue to monitor so please let us know if you still experience any issues from this point on.
Posted May 07, 2022 - 13:03 PDT
We are almost through the scheduled operation and we expect the platform to be fully functioning within the hour. We will provide an update sooner if possible.
Posted May 07, 2022 - 12:21 PDT
We are continuing to make progress on the operation.
Posted May 07, 2022 - 11:54 PDT
The DrChrono team is currently running a scheduled operation, for which we had not anticipated any impact to the platform. We apologize for any inconvenience but we expect this operation to be completed soon. We will update with progress here.
Posted May 07, 2022 - 11:37 PDT
This incident affected:, drchrono iPad EHR, drchrono iPad Check-In Kiosk Application, DrChrono Telehealth Platform,, onpatient iPhone PHR, and Public Website.