Issues occurred on November 12, 2021 which made RTE inaccessible for the Trizetto clearinghouse.
All times are PST.
| Date/Time | Activity |
|---|---|
| 2021-11-12 08:17 | Started receiving initial reports of the issue. |
| 2021-11-12 08:40 | Engineering team responds to issue with a potential cause. |
| 2021-11-12 08:40 | Engineering team reaches out to clearinghouse for additional info. |
| 2021-11-12 08:40 | Engineering team responds to issue with a potential cause. |
| 2021-11-12 09:22 | Clearinghouse responds with issue origin. |
| 2021-11-12 09:22 | Engineering team works on a hotfix. |
| 2021-11-12 09:29 | Status Page updated to investigating. |
| 2021-11-12 10:03 | Engineering team is investigating recent changes to SFTP password change as potential cause. |
| 2021-11-12 10:28 | Status Page updated to identified. |
| 2021-11-12 11:27 | DevOps deploys hotfix to staging. |
| 2021-11-12 11:46 | RTE is verified as resolved in staging. |
| 2021-11-12 12:01 | Devops starts deploying hotfix to production. |
| 2021-11-12 13:17 | Hotfix is deployed to production. |
| 2021-11-12 14:01 | Testing team verifies hotfix resolved RTE issue. |
| 2021-11-12 14:14 | Status Page updated to resolved. |
Scope of routine SFTP password rotation was not properly analyzed. SFTP password was found to be used by a variety of third-party services not originally known.
To stabilize the platform a code issue was identified and a hotfix was created, deployed, and validated.
The issue impacted RTE only for the Trizetto clearinghouse.
Engineering fixed the identified code issue and the fix was deployed and verified. Code change enabled RTE to use the previous SFTP password. Additional documentation on the procedure + safety measures for secrets rotation will be created and shared with all engineering teams.