Unable to save patient demographic changes, receive faxes, complete onboarding via OnPatient

Incident Report for DrChrono

Postmortem

OnPatient, HelloFax and Updox Service Disruption

Summary

Customers were unable to save demographic information, and HelloFax/Updox users were unable to receive faxes. This was caused by a recent change around storing file ownership information on our AWS storage server.

Timeline (PST, 24-hour clock)

All times are PST.

Date/Time	Activity
2021-08-19 01:00	Error first appeared on Sentry: https://sentry.drchrono.com/drchrono/drchrono-web/issues/249249/events/10325449/
2021-08-20 10:29	The Support Team received a notification about patients being unable to save demographic information and HelloFax/Updox users being unable to receive faxes.
2021-08-20 10:35	Engineering team amends a previous hotfix related to S3 file storage that will tackle these issues.
2021-08-20 11:06	Status Page updated to reflect investigating status.
2021-08-20 11:37	Engineering team completes hotfix.
2021-08-20 12:00	Status Page updated to include additional issues of patients being unable to complete onboarding after signing consent forms (through OnPatient).
2021-08-20 13:03	Hotfix solution is approved.
2021-08-20 13:27	DevOps implements hotfix solution.
2021-08-20 13:30	Status Page updated to identified.
2021-08-20 14:00	Production deployment is ongoing.
2021-08-20 14:30	Production day-time deployment is completed. Hotfix solution implemented.
2021-08-20 14:48	Status page updated to monitoring.
2021-08-20 15:26	Customers report continuing issues for fax services.
2021-08-20 15:56	Status page updated to reflect further investigation/another solution to be implemented.
2021-08-20 16:35	Database hotfix being implemented. Hotfix will update one of the S3 database table constraints.
2021-08-20 16:37	Database changes have been run on staging and production.
2021-08-20 16:49	Status page updated to monitoring.
2021-08-20 17:22	Support team received a ticket about error when opening up PDF faxes. Error on website required Adobe Acrobat to view PDF(s).
2021-08-20 18:00	Engineering team works on a hotfix to update the client signature version for the S3 client which will address the PDF error.
2021-08-20 18:19	Status page updated to identifying Adobe Plugin error message.
2021-08-20 19:04	DevOps implements hotfix solution.
2021-08-20 19:40	Hotfix solution has been implemented to Production. Hotfix solution resolves previous error messages for new faxes. Faxes prior to this hotfix are not able to be opened.
2021-08-20 19:59	Status page updated to monitoring.
2021-08-23 09:43	Status page updated to notify customers about engineering team looking into recovering faxes sent on Friday, August 20th.
2021-08-23 09:21	Support team receives reports about customers being unable to save patient demographics.
2021-08-23 10:24	Engineering team works on a hotfix to resolve this issue, the hotfixes on 2021-08-20 did not resolve the patient demographics issue.
2021-08-23 10:57	Status page updated to investigating.
2021-08-23 11:53	Hotfix solution created - changed S3 signature version from v4 to s3v4.
2021-08-23 12:20	Status page updated to identified.
2021-08-23 12:53	DevOps implements hotfix solution.
2021-08-23 13:57	Hotfix solution implemented on production. Customers are able to view faxes sent on Friday, August 20th.
2021-08-23 14:14	Status page updated to monitoring.
2021-08-23 15:27	Status page updated to resolved.

Contributing Factor(s)

The major contributing factor was recent changes made to the S3 file storage bucket. Adding additional information about file owners caused issues to spread in various DrChrono services.

Stabilization Steps

A previously required component for auditing metadata was made optional - this requirement threw a high number of errors and was the root cause of the aforementioned issues. Applying the correct versioning requirements for S3 resolved any additional PDF issues that arose.

Impact

This issue affected customers' ability to use certain features of the website. Namely, this affected saving patient demographic information, ability to receive/view faxes and complete onboarding via OnPatient.

Corrective Actions

Production day time deployment allowed hotfix solution to be implemented with minimal, if any, service disruption. Requirements about file ownership information were changed. Ultimately updated S3 client version to s3v4 and updated database constraint related to one of the S3 tables. These solutions will prevent a similar situation from happening again.

Posted Aug 25, 2021 - 09:03 PDT

Resolved

We have deployed a change that will now allow customers to view faxes received between 8:15 pm EST - 10:45 pm EST on Friday, August 20th. If the fax page advises that a pdf plugin is still required, please hard refresh your web browser. A hard refresh clears your browser cache for a specific page, which forces it to load the most recent version of that page.
For Mac, on Chrome and Firefox press Shift+Command+R. Safari press Cmd+Option+R.
For Windows, all browsers Ctrl+F5
Faxes that you were expecting to receive Thursday evening, August 19th, through 8:15 pm EST on Friday, August 20th unfortunately are not recoverable. We greatly apologize for this inconvenience and thank you for your understanding during this incident.

Posted Aug 23, 2021 - 15:23 PDT

Update

Update:
Customers may not have received faxes between the evening of Thursday, August 19th, until 8:15 pm EST on Friday, August 20th. Our team released a hotfix to resolve this issue on Friday. We ask that you reach out to the sender for any faxes you may be missing.
After the hot fixed was released, faxes began appearing for our customers. However, we started hearing from customers that they were receiving instruction that a pdf plugin was required to view the fax upon opening between 8:15 pm EST - 10:45 pm EST on Friday, August 20th. Our team worked to resolve the issue so that any new faxes being sent should be available to view, but this fix does not apply to any faxes that came in during the time above. We ask that you please reach out to the sender for anything time-sensitive to request that they resend those faxes from Friday, August 20th, between 8:15 pm EST - 10:45 pm EST.
Our team is looking into the possibility of recovering any faxes that may be missing, and we will keep you apprised of this status. However, as mentioned above, please reach out to the sender if there is anything time-sensitive that you cannot read in your message center today or that you were expecting to receive on Friday, August 20th, and didn't.

Posted Aug 23, 2021 - 09:43 PDT

Monitoring

A fix has been deployed to resolve the remaining issue of inbound faxes not loading and advising the user that an adobe acrobat plugin is needed. Incoming faxes from this point onward should arrive and load in the message center without trouble. We will continue to investigate whether previously received faxes that refuse to load can be recovered but at this time do not have an answer. That part of the investigation will continue into Monday.

Posted Aug 20, 2021 - 19:59 PDT

Identified

It has been reported that although faxes are now being received, pdf documents will not load advising the user that an adobe acrobat plugin is needed. Our engineering team is working now on an additional fix to release for this particular problem tonight. We will update here when a timeframe can be estimated for the release.

Posted Aug 20, 2021 - 18:19 PDT

Monitoring

We have deployed another fix for inbound faxes. Again, please do note that any faxes sent to you today will need to be resent. We sincerely apologize for this inconvenience. Please let our support team know as you did earlier if you continue to have any further issues today. Thank you.

Posted Aug 20, 2021 - 16:49 PDT

Identified

We believe we have identified the cause for inbound faxes continuing to fail and will update again with an estimated timeframe for resolution. Please note that any faxes sent to you during today's fax outage will require the sender to resend them to you. These faxes are not recoverable in the DrChrono system.

Posted Aug 20, 2021 - 16:35 PDT

Update

Thank you for reaching out to our support team to let us know that inbound faxes are still not being received. We are continuing to review this issue now.

Posted Aug 20, 2021 - 15:56 PDT

Monitoring

A fix has been implemented for patient demographics not saving, unable to receive faxes, and patients unable to complete onboarding via OnPatient. Please let our support team know if you continue to experience any of these three issues today.

Posted Aug 20, 2021 - 14:48 PDT

Identified

The root cause has been identified for patient demographics not saving, unable to receive faxes, and patients unable to complete onboarding via OnPatient. We are coordinating the release to fix these issues shortly and will provide another update when the fix has been deployed.

Posted Aug 20, 2021 - 13:30 PDT

Update

We have also added to today's investigation reports of patients unable to complete onboarding after signing consent forms via OnPatient. We will continue to provide updates here via our status page. Thank you for your patience during this time.

Posted Aug 20, 2021 - 12:00 PDT

Investigating

Our team is currently investigating reports of the inability to save patient demographic changes or receive faxes. We apologize for these inconveniences and will follow up here as we have additional information.

Posted Aug 20, 2021 - 11:06 PDT

This incident affected: drchrono.com, drchrono iPad EHR, onpatient.com, onpatient iPhone PHR and 3rd party services (HelloFax, Updox).