RTE outage impacting Trizetto payers
Incident Report for DrChrono


Issues occurred on November 12, 2021 which made RTE inaccessible for the Trizetto clearinghouse.

Timeline (PST, 24-hour clock)

All times are PST.

Date/Time Activity
2021-11-12 08:17 Started receiving initial reports of the issue.
2021-11-12 08:40 Engineering team responds to issue with a potential cause.
2021-11-12 08:40 Engineering team reaches out to clearinghouse for additional info.
2021-11-12 08:40 Engineering team responds to issue with a potential cause.
2021-11-12 09:22 Clearinghouse responds with issue origin.
2021-11-12 09:22 Engineering team works on a hotfix.
2021-11-12 09:29 Status Page updated to investigating.
2021-11-12 10:03 Engineering team is investigating recent changes to SFTP password change as potential cause.
2021-11-12 10:28 Status Page updated to identified.
2021-11-12 11:27 DevOps deploys hotfix to staging.
2021-11-12 11:46 RTE is verified as resolved in staging.
2021-11-12 12:01 Devops starts deploying hotfix to production.
2021-11-12 13:17 Hotfix is deployed to production.
2021-11-12 14:01 Testing team verifies hotfix resolved RTE issue.
2021-11-12 14:14 Status Page updated to resolved.

Contributing Factor(s)

Scope of routine SFTP password rotation was not properly analyzed. SFTP password was found to be used by a variety of third-party services not originally known.

Stabilization Steps

To stabilize the platform a code issue was identified and a hotfix was created, deployed, and validated.


The issue impacted RTE only for the Trizetto clearinghouse.

Corrective Actions

Engineering fixed the identified code issue and the fix was deployed and verified. Code change enabled RTE to use the previous SFTP password. Additional documentation on the procedure + safety measures for secrets rotation will be created and shared with all engineering teams.

Posted Dec 21, 2021 - 12:37 PST

This incident has been resolved.
Posted Nov 12, 2021 - 14:14 PST
A fix has been implemented and we are monitoring the results.
Posted Nov 12, 2021 - 13:40 PST
The issue has been identified and a fix is being implemented.
Posted Nov 12, 2021 - 10:28 PST
We are currently experiencing issues with RTE connectivity with a few of our payers that are supported through Trizetto. We are working on a solution to resolve the issue and reestablish connection as soon as possible.
Posted Nov 12, 2021 - 09:29 PST
This incident affected: 3rd party services (Trizetto).