On 4/12/2021, DrChrono experienced issues related to our task systems that prevented new tasks from being created in the task system and receiving refill requests from Surescripts in the DrChrono platform.
The issue has been identified and corrected. We do not expect to experience recurring issues of this nature in the future.
| Date/Time | Activity |
|---|---|
| 2021-04-10 04:12 | The source of the issue manifested. |
| 2021-04-10 11:37 | Initial reports that customers were experiencing an issue creating tasks and that the Open task status was missing. |
| 2021-04-10 16:19 | Suspected cause of the issue has been identified but not verified on the production database, so a Jira ticket was created. |
| 2021-04-12 09:50 | Suspected cause of the issue has been verified on the production database. |
| 2021-04-12 09:57 | First stabilization step was applied. Customers not using task templates are able to create tasks again. |
| 2021-04-12 10:23 | Source of the issue was identified in the database log. |
| 2021-04-12 11:45 | Second stabilization step was applied. All customers are able to create tasks again. |
| 2021-04-12 12:47 | Task Creation Issue has been identified as resolved. |
| 2021-04-12 15:35 | Internal RCA posted to Engineering team; identified receiving refill requests from Surescripts was an issue and started investigation. |
| 2021-04-15 | Corrective actions have been applied. |
In the task system, there are global default statuses shared with all practices; Open, In Progress, On Hold, and Complete. In the DrChrono API, there was an error in the validation logic that would allow users to modify any of these global default statuses which would result in the status(es) being re-assigned to the practice group that made the API request. This resulted in the status no longer appearing for all users.
This issue impacted all customers and was localized to the following areas for two days based on the common error messages seen.
| Description | % of Errors Related to Incident |
|---|---|
| Surescripts sending messages from pharmacy to prescriber via DrChrono | 49.97 % |
| Creating/updating a Task | 33.45 % |
| Patient sending message to provider via OnPatient | 15.43 % |
| Creating LabDocument via API | 1.15 % |
| TOTAL | 100.0 % |
Corrective Actions
The root cause, identified as an unintentional API request, was patched and tested. While developing the patch, remediation steps were performed on the database to make the task system operational again.