Office filter on calendar, Approval and Denial eRX refill requests, Telehealth office selection

Incident Report for DrChrono

Postmortem

Description

On 05/13, customers were having issues on some screens where certain sections, such as office locations, would go missing or stop working.

Timeline

All times are in PST.

Date/Time	Activity
2022-05-12 09:13	A separate issue prompts the DevOps team to release a mid-day hotfix to all servers, rolled out at a slower pace to prevent significant customer impact. This causes some issues for customers accessing the site during this time.
2022-05-13 07:00	The errors are found to consist of pages not rendering correctly, which are cached pages so they do not fix automatically upon navigating to another page. Users are instructed to perform a hard refresh, which fixes the issue.
2022-05-15 07:51	The engineering team is alerted by reports of users having problems when they open the calendar screen. An incident is posted to the status page.
2022-05-18 06:25	The root cause is found to be in the previous deployment. Process changes are put in place so this does not cause a similar issue in the future.
2022-05-20 08:44	Status page is updated to monitoring.
2022-05-25 08:27	A member of the DevOps team puts mitigation measures in place that should prevent this issue going forward.

Contributing Factor(s)

Our deployment steps do not currently account for frontend files that might be changed during a deployment but still need to be served for a while. Those failed requests get cached and stick around for a long time.

Stabilization Steps

No system actions were performed immediately. Processes have been changed since then to make sure our deployments minimize any potential impact.

Impact

Customers were having issues loading offices in the calendar page and other UI problems throughout the application.

Corrective Actions

The following internal process improvements have been implemented:

A page rule to prevent failed requests from being cached for long periods of time.
Revised process to keep frontend files between deployments or store them to a static file server.

Posted Jun 02, 2022 - 07:09 PDT

Resolved

This incident has been resolved. An RCA will be available shortly.

Posted Jun 02, 2022 - 07:07 PDT

Monitoring

A fix has been implemented and we are monitoring the results. If you continue to experience this issue, please reach out to our support team through https://drchrono.com/help with the details so we can further troubleshoot with you.

Posted May 20, 2022 - 08:44 PDT

Update

We are continuing to investigate the issue to determine its root cause. If performing a hard refresh does not resolve this issue, please reach out to our support team so we can further troubleshoot this with you.

Thank you for your patience.

Posted May 13, 2022 - 14:56 PDT

Investigating

Some users are experiencing an issue with offices not appearing on the calendar page, not able to approve or deny an eRx refill request, and telehealth office selection missing on the Offices page. While we investigate this problem, in the meantime performing a hard refresh of the webpage will resolve these problems. A hard refresh clears your browser cache for a specific page, which forces it to load the most recent version of that page. The following instructions specify how to perform a hard refresh based on web browser and operating system:

For Mac, on Chrome and Firefox press Cmd+Shift+R. Safari press Cmd+Option+R.

For Windows, all browsers Ctrl+F5

Posted May 13, 2022 - 07:52 PDT

This incident affected: drchrono.com.