An update to the Quest data format prevented the system from processing the compendium update.
All times are EST.
| Date/Time | Activity |
|---|---|
| 2021-07-27 09:25:32 | Started receiving initial reports of issues with Quest test orders. Investigation started. |
| 2021-07-27 09:36:08 | Updated the support team that there were no obvious causes for the issue. |
| 2021-07-27 09:39:57 | Support team reached out to Quest to see if there were any issues on the Quest side. |
| 2021-07-27 10:07:23 | Informed by Quest that they did not have any issues on the Quest side. |
| 2021-07-27 10:12:05 | Status Page created. |
| 2021-07-27 10:28:57 | Support team pulled a ticket from a previous occurrence from March 2019. |
| 2021-07-27 10:32:36 | Problematic scheduled job identified. |
| 2021-07-27 10:34:36 | Specific error from scheduled job identified. |
| 2021-07-27 10:39:22 | Restarted scheduled job. |
| 2021-07-27 10:55:25 | Restarted scheduled job finished and testing started. |
| 2021-07-27 10:57:16 | Support team was still getting errors during test submissions. |
| 2021-07-27 11:03:22 | Additional warnings during parsing of orders from Quest. |
| 2021-07-27 11:07:28 | Investigation of warnings by the DevOps and engineering teams started. |
| 2021-07-27 11:19:35 | Engineering team started developing a fix based on potential changes to the Quest data format. |
| 2021-07-27 2:34:17 | Staging database tables copied to production to provide a temporary fix. |
| 2021-08-02 2:50:00 | Jira Ticket accepted into the Provider team’s 08-02 sprint. |
A lost or unsent notification from Quest about the changes in the data format prevented us from processing the updated file.
Impact was widespread to all customers utilizing Quest for test orders.
The Quest compendium data format has changed and requires development effort from the engineering team.
The automated update process has been halted while the underlying cause is addressed.
We have also discussed the notification issues with Quest are are working towards improving those channels.
The engineering team will be updating the existing process to account for the data format changes.