Issues occurred on February 18, 2022 where customers were unable to access the following: DrChrono website, iPad EHR, iPad Check-In app, and OnPatient.
All times are PST.
| Date/Time | Activity |
|---|---|
| 2022-02-18 10:19 | Support team started to receive tickets about the site being inaccessible. |
| 2022-02-18 10:19 | DevOps investigates issue. |
| 2022-02-18 10:30 | Status page is created to reflect investigating status. |
| 2022-02-18 10:31 | DevOps identifies issues with missing system file directories. |
| 2022-02-18 10:31 | DevOps redeploys servers to create missing system file directories. |
| 2022-02-18 10:53 | Status page updated to reflect identified status. |
| 2022-02-18 11:05 | All servers have recreated the missing system file directories. |
| 2022-02-18 11:11 | Update is posted on status page to reflect all systems are operational. |
| 2022-02-18 11:13 | Status page updated to resolved. |
The primary factor was an erroneous script that was executed in an attempt to free web server disk space. The erroneous script removed the current system file directories instead of older releases.
Redeploying the web servers with the latest code release re-created the missing/deleted system file directories.
All customers were unable to access the DrChrono website.
Aside from the stabilization steps mentioned, safeguards have been put in place to prevent further issues from arising. All potentially critical script and code executions on production will be vetted/executed through a front-running web server prior to production being accessed.