Customers were unable to save demographic information, and HelloFax/Updox users were unable to receive faxes. This was caused by a recent change around storing file ownership information on our AWS storage server.
All times are PST.
| Date/Time | Activity |
|---|---|
| 2021-08-19 01:00 | Error first appeared on Sentry: https://sentry.drchrono.com/drchrono/drchrono-web/issues/249249/events/10325449/ |
| 2021-08-20 10:29 | The Support Team received a notification about patients being unable to save demographic information and HelloFax/Updox users being unable to receive faxes. |
| 2021-08-20 10:35 | Engineering team amends a previous hotfix related to S3 file storage that will tackle these issues. |
| 2021-08-20 11:06 | Status Page updated to reflect investigating status. |
| 2021-08-20 11:37 | Engineering team completes hotfix. |
| 2021-08-20 12:00 | Status Page updated to include additional issues of patients being unable to complete onboarding after signing consent forms (through OnPatient). |
| 2021-08-20 13:03 | Hotfix solution is approved. |
| 2021-08-20 13:27 | DevOps implements hotfix solution. |
| 2021-08-20 13:30 | Status Page updated to identified. |
| 2021-08-20 14:00 | Production deployment is ongoing. |
| 2021-08-20 14:30 | Production day-time deployment is completed. Hotfix solution implemented. |
| 2021-08-20 14:48 | Status page updated to monitoring. |
| 2021-08-20 15:26 | Customers report continuing issues for fax services. |
| 2021-08-20 15:56 | Status page updated to reflect further investigation/another solution to be implemented. |
| 2021-08-20 16:35 | Database hotfix being implemented. Hotfix will update one of the S3 database table constraints. |
| 2021-08-20 16:37 | Database changes have been run on staging and production. |
| 2021-08-20 16:49 | Status page updated to monitoring. |
| 2021-08-20 17:22 | Support team received a ticket about error when opening up PDF faxes. Error on website required Adobe Acrobat to view PDF(s). |
| 2021-08-20 18:00 | Engineering team works on a hotfix to update the client signature version for the S3 client which will address the PDF error. |
| 2021-08-20 18:19 | Status page updated to identifying Adobe Plugin error message. |
| 2021-08-20 19:04 | DevOps implements hotfix solution. |
| 2021-08-20 19:40 | Hotfix solution has been implemented to Production. Hotfix solution resolves previous error messages for new faxes. Faxes prior to this hotfix are not able to be opened. |
| 2021-08-20 19:59 | Status page updated to monitoring. |
| 2021-08-23 09:43 | Status page updated to notify customers about engineering team looking into recovering faxes sent on Friday, August 20th. |
| 2021-08-23 09:21 | Support team receives reports about customers being unable to save patient demographics. |
| 2021-08-23 10:24 | Engineering team works on a hotfix to resolve this issue, the hotfixes on 2021-08-20 did not resolve the patient demographics issue. |
| 2021-08-23 10:57 | Status page updated to investigating. |
| 2021-08-23 11:53 | Hotfix solution created - changed S3 signature version from v4 to s3v4. |
| 2021-08-23 12:20 | Status page updated to identified. |
| 2021-08-23 12:53 | DevOps implements hotfix solution. |
| 2021-08-23 13:57 | Hotfix solution implemented on production. Customers are able to view faxes sent on Friday, August 20th. |
| 2021-08-23 14:14 | Status page updated to monitoring. |
| 2021-08-23 15:27 | Status page updated to resolved. |
The major contributing factor was recent changes made to the S3 file storage bucket. Adding additional information about file owners caused issues to spread in various DrChrono services.
A previously required component for auditing metadata was made optional - this requirement threw a high number of errors and was the root cause of the aforementioned issues. Applying the correct versioning requirements for S3 resolved any additional PDF issues that arose.
This issue affected customers' ability to use certain features of the website. Namely, this affected saving patient demographic information, ability to receive/view faxes and complete onboarding via OnPatient.
Production day time deployment allowed hotfix solution to be implemented with minimal, if any, service disruption. Requirements about file ownership information were changed. Ultimately updated S3 client version to s3v4 and updated database constraint related to one of the S3 tables. These solutions will prevent a similar situation from happening again.