Processing Delays for ERAs and Claims Status Updates
Incident Report for DrChrono
Postmortem

Root Cause Analysis: 3/23/2021, Electronic Claims Processing Slowdown

Description

Summary: Claims Payments processing started processing slower and got behind after 3/17/21.

Timeline

All times are in PST

Date/Time Activity
2021-03-23 9:23 AM ERAs & 277s from Trizetto not getting processed and tracked under the Quick View > EDI Received screen was reported via the RCM Team & Customer Support. Found files processed prior to 3/18 processed much faster. For example, a 4 KB file was taking 5 min when it used to take ~12s.
2021-03-23 10:45 AM Removed changes to Redis cache and the cloudwatch logging that were rolled out on 3/18 as we were concerned these were adding to the slow processing times.
2021-03-23 1:42 PM Setup NewRelic tracing for the cron job commands. Then found max_statement_time was spinning up db connections for each claim query in each file which was unnecessary. This took up 94% of the processing time. We put a patch in place to not set the max_statement_time=500 for Trizetto/Gateway.
2021-03-29 5:45 PM Put in hot fix to bypass and speed up a failure on initial matches to an appointment and provider, which were taking 1 min per each claim that failed and the time to try to write to the audit_log which would timeout on large files.
2021-03-30 5:48 PM Profiled the claims processing against staging. Set up parallel processing for Trizetto/Gateway with 100 workers. Got MONyog logs for MariaDB configuration changes and did not identify configuration changes that would have impacted query times and timeouts.
2021-03-31 8:15 AM Emdeon and Waystar were still processing too slowly or not at all.
2021-03-31 9:23 AM Moved Emdeon/Waystar exceptions back to inbox and restarted the parsing jobs.
2021-03-31 2:45 PM Monitored Trizetto’s backlog progress, and moved new files from the SFTP folder and exceptions back in for processing.
2021-03-31 9:48 PM Parallelized the Waystar and Emdeon processes as well.
2021-03-31 9:48 PM Patched max_statement_timeout for Waystar and Emdeon jobs as well.
2021-04-01 2:33 PM Split large Trizetto 837 files for 2021-03-24 and 2021-03-26.
2021-04-05 11:23 AM Developed a script & bot to monitor the status of the claims processing for internal use.
2021-04-05 4:15 PM Profiled the claims processing against the prod db/replicas and identified 99.5% of the time was spent on DB connections to a ‘Warehouse’ DB that had been removed from Rackspace and the Celery workers Django settings had CONN_MAX_AGE = 0 which creates and disconnects to the DB for every transaction. Removed the connection to the removed DB.
2021-04-10 10:01 AM Set the Celery worker max timeout to 90Ks (25 hours) to process large files (e.g. 900 KB).
2021-04-12 5:52 PM Put in hot fix for 277 parsing logic issues (temporal order issue).
2021-04-12 5:52 PM Put in fix for the RCM EDI Received page to show received and processed status dates.
2021-04-13 1:38 PM Reviewed and processed 277 and ERA exceptions.
2021-04-19 5:56 PM Put in hot fix for parsing Emdeon’s mrf files (version of the 277s) to check if ERA was already processed and if not, process 277.
2021-04-22 1:38 PM Moved older mrfs for Emdeon out. Continued processing the backlog of Emdoen mrf 277 files through 2021-04-22.
2021-04-22 2:22 PM Created a separate queue for files that timeout in parsing to manually call the parser for specific files to ensure we process them starting with the oldest files first through 2021-04-23.
2021-04-22 2:22 PM Research into duplicate 277/ERA files. Identified dups were sent by Trizetto/Gateway. For Emdeon, the files have updated statuses for the claims.
2021-04-23 10:23 AM Split large Trizetto 837 files for 2021-04-22 date over 7K KB to re-send to Trizetto/Gateway.
2021-04-24 9:23 AM Received MemoryError exceptions starting on 2021-04-23 at 3:12 AM due to too many claims being submitted at once. Added extra memory for the cron job to resolve this issue.
2021-04-28 1:23 PM Fixed Trizetto/Gateway 837 that ended part way through a claim for 03-31-2021 and re-sent to Trizetto/Gateway.
2021-04-29 5:18 PM Split large Trizetto/Gateway 837 files for 2021-03-29 and 2021-03-31 dates over 7K KB to re-send to Trizetto/Gateway.
2021-04-30 10:21 AM Developed & ran a script to re-bill claims that were stuck “in process at clearinghouse” claim status with a last EDI date greater than a week.
2021-05-04 8:42 AM Developed a script to handle duplicate records, to process the initial record first, then wait for that record to complete before processing the other record.
2021-05-04 10:23 AM Adding change to handle IntegrityError that occurs when there are duplicate claims sent by the clearinghouses for ERAs.
2021-05-04 11:00 AM Continuing development work to split claim files (ERAs, 277s) by claim to distribute across 100 workers for processing.
2021-05-04 3:42 PM Breaking out logging for each clearinghouse claims processing for better monitoring and troubleshooting of claims processing issues.

Contributing Factors

  • A ‘Warehouse’ DB that had been removed from our environment and the Celery workers Django settings had CONN_MAX_AGE = 0 which creates and disconnects to the DB for every transaction. We were still trying to make a connection to the DB which no longer existed for every claim query.
  • Had a max_statement_time of 500s (~8 min) which was hit often when the process was unable to get a return on the query in a timely manner.
  • The previous process was single threaded. We were unable to process multiple files and claims at once, and only one file at a time.
  • There were some issues with the filters on the Quick View > EDI Received page for the RCM team to use to investigate claim file statuses. Engineering needed to look into statuses of claims and add new monitoring for everyone to see and review the status of the files and claims processing.
  • In order to catch up on the backlog of claims due to the time spent on DB connection calls and max_statement_time outs of 500s (~8 min), we chose to process the ERAs first, and then had to develop checks to see if we processed a claim with an ERA file before processing the older backlog of 277s.
  • We didn’t have a limit on the size of the 837 files sent back to the clearinghouses which caused some issues for Trizetto/Gateway unable to process larger files (>7K MB).
  • Some customers were re-submitting claims and duplicates caused further slowdown in processing from 2021-04-22 to 2021-04-24 due to IntegrityErrors we were hitting by processing the duplicate claims. We also hit a MemoryError on 2021-04-23 at 3:12 AM due to too many claims being submitted at once and needed to add extra memory for the cron job to resolve this issue on 2021-04-25.

Impact

  • There were several delays in processing 277 and ERA files for Trizetto/Gateway, Emdeon, and Waystar.

Corrective Actions

  • Parallelized the Trizetto/Gateway, Emdeon, and Waystar claims file processing and setup multiple workers.
  • Put in hot fix to bypass and speed up a failure on initial matches to an appointment and provider, which were taking 1 min per each claim that failed and the time to try to write to the audit_log which would timeout on large files.
  • Removed the connection in our configuration to the retired ‘Warehouse’ DB which was causing unnecessary connection tries for each claim query.
  • Monitored Trizetto’s backlog progress, and moved new files from the SFTP folder and exceptions back in for processing.
  • Set the Celery worker max timeout to 90K sec (25 hours) to process large files (e.g. 900 KB).
  • Added extra memory for the cron job to resolve MemoryError exceptions.
  • Split large Trizetto 837 files and resent files to Trizetto/Gateway for processing.
  • Developed & ran a script to re-bill claims that were stuck “in process at clearinghouse” claim status with a last EDI date greater than a week.
  • Developed a script to handle duplicate records, to process the initial record first, then wait for that record to complete before processing the other record.
  • Adding change to handle IntegrityError that occurs when there are duplicate claims sent by the clearinghouses for ERAs.
  • Continuing development work to split claim files (ERAs, 277s) by claim to distribute across 100 workers for processing.
  • Breaking out logging for each clearinghouse claims processing for better monitoring and troubleshooting of claims processing issues.
Posted May 27, 2021 - 11:59 PDT
Resolved
DrChrono has resolved the parsing issue with 277s and ERAs from Waystar, Trizetto and Emdeon/Change Healthcare, and statuses and receipts should now be reflecting correctly in DrChrono. Please note that some normal fluctuations in the percentages of files parsed could occur as we receive the constant but variable influx of daily claims from the clearinghouses. If you notice anything incorrect with claims from March and April, please chat with our Support team or submit at ticket at drchrono.com/help so we can investigate further.
Posted May 04, 2021 - 14:20 PDT
Monitoring
DrChrono has resolved the parsing issue with 277s and ERAs from Waystar, Trizetto and Emdeon/Change Healthcare, and we’ll be closely monitoring the percentage of claims complete to make sure everything is working as expected. Please note that some normal fluctuations in the percentages of files parsed could occur as we receive the constant but variable influx of daily claims from the clearinghouses. If you notice anything incorrect with claims from March and April, please reach out to our Support team so we can investigate further.
Posted Apr 29, 2021 - 13:19 PDT
Update
We have seen an overall improvement in percentages of claims complete, despite a slight dip in the processing of 277 files from Trizetto. We are now also caught up with the 277 files from Emdeon and the ERA files from Trizetto. We continue to release processing improvements daily, and we expect to continue to see a positive percentage of progress overall. Please note that some normal fluctuations in the percentages of files parsed could occur as we receive the constant but variable influx of daily claims from the clearinghouses.

Emdeon/Change Healthcare is still experiencing continuing disruptions in their Vision portal but has made progress restoring some of its functionality:
* The Reporting & Analytics functionality of the Vision portal and access to the Change Healthcare payer list was restored on April 26, at noon CT.
* Claim statuses and workers compensation claim attachment statuses for the month of April are currently still unavailable as Change Healthcare teams work to restore the April data.
* Any claim information received from April 1 thru April 21 will be available by 6 a.m. CT on April 27.
* The Change Healthcare team continues to work on an ETA for the remainder of April's data, and will continue to provide updates as this information becomes available.

The claims file from April 22 has now been processed successfully by Trizetto.

277:
Trizetto: 93%  (DOWN FROM PREVIOUS DAY)
Emdeon: 100%  (CAUGHT UP)

ERA:
Trizetto: 100%  (CAUGHT UP)
Emdeon: 97%   (UP FROM PREVIOUS DAY)
Posted Apr 29, 2021 - 05:57 PDT
Update
We continue to see improvements to our percentage of claims complete, and we expect to see additional improvements over the next few days. Please note that some normal fluctuations in the percentages of files parsed could occur as we receive the constant but variable influx of daily claims from the clearinghouses.

Emdeon/Change Healthcare is still experiencing continuing disruptions in their Vision portal but has made progress restoring some of its functionality:
* The Reporting & Analytics functionality of the Vision portal and access to the Change Healthcare payer list was restored on April 26, at noon CT.
* Claim statuses and workers compensation claim attachment statuses for the month of April are currently still unavailable as Change Healthcare teams work to restore the April data.
* Any claim information received from April 1 thru April 21 will be available by 6 a.m. CT on April 27.
* The Change Healthcare team continues to work on an ETA for the remainder of April's data, and will continue to provide updates as this information becomes available.

We have worked with Trizetto to submit the claims file from April 22. We are now waiting for an acknowledgment of successful processing within the next 24 hours.

277:
Trizetto: 94% (UP FROM PREVIOUS DAY)
Emdeon: 100% (CAUGHT UP)

ERA:
Trizetto: 99% (UP FROM PREVIOUS DAY)
Emdeon: 95% (UP FROM PREVIOUS DAY)
Posted Apr 28, 2021 - 08:36 PDT
Update
The processing improvements we released over the weekend have improved our percentage of claims completed, and we expect to see additional improvements over the next few days. Please note that some normal fluctuations in the percentages of files parsed could occur as we receive the constant but variable influx of daily claims from the clearinghouses.

Emdeon/Change Healthcare is still experiencing continuing disruptions in their Vision portal that could affect your ability to see the latest updates on claims there. However, they have stated that this does not impact claims processing, claim delivery to payers, and reporting. 

We are working closely with the team at Trizetto to make sure the claims from April 22 are correctly submitted.

277:
Trizetto: 86%  (UP FROM PREVIOUS DAY)
Emdeon: 100% (UP FROM PREVIOUS DAY)

ERA:
Trizetto: 99%  (UP FROM PREVIOUS DAY)
Emdeon: 93%  (UP FROM PREVIOUS DAY)
Posted Apr 27, 2021 - 10:27 PDT
Update
We continue to make steady positive progress processing the backlog of 277s and ERAs, and over the weekend we made changes that allowed us to more quickly process these claims. Please note there was a slight decrease in the progress for Emdeon ERAs, since we are still receiving duplicate claim submissions. We are putting controls in place to minimize the impact duplicates have in the claims parsing process. 

DrChrono continues to submit daily claims files to all of our clearinghouses. However, Emdeon is having issues with the last daily batches from April 21 - 23rd. They cannot locate these files, even though DrChrono received acknowledgment from Emdeon confirming the reception of the files. We are actively working with Emdeon to resolve the issue. Please note that claim processing is not affected. DrChrono is correctly submitting the daily batches, providers will continue to receive payments and reports, but clearinghouse status details are delayed in displaying in the Vision portal.

277:
Trizetto: 68.33%  (UP FROM PREVIOUS DAY)
Emdeon: 61.28%  (UP FROM PREVIOUS DAY)

ERA:
Trizetto: 98.62%  (UP FROM PREVIOUS DAY)
Emdeon: 85.80% (DOWN FROM PREVIOUS DAY)
Posted Apr 26, 2021 - 07:41 PDT
Update
We released a new hotfix yesterday that should increase CPU performance and decrease its usage, thereby speeding up claim processing tasks. We expect this recovery to take a few days before seeing significant changes in the claim counts, as we are also ramping up the number of available threads to process multiple claims at the same time.

Positive progress has been made on the 277 files of Trizetto and Emdeon. We’ve also seen positive progress on the ERA files of Emdeon, but we are still receiving a high number of ERAs from Trizetto that seem to due to multiple claim submissions. This has affected the percentages of overall progress. We continue to work on identifying this issue so that we focus solely on processing the correct files. Waystar has continued to process normally and without delays.

Please continue to contact support here if you have any questions about a claim that you feel should be resolved, rather than re-submitting.

277:
Trizetto: 64.08%
Emdeon: 97.4%

835:
Trizetto: 60.55%
Emdeon: 89.04%
Posted Apr 23, 2021 - 10:01 PDT
Update
We saw a reduction in claim processing speed overnight and have identified two potential options to address this, one of which entails a fix that we can potentially release today with the goal of speeding up the process.

We've also received an unusually high number of files from the clearinghouses, particularly from Trizetto. This increased our total files and decreased progress percentage-wise. We continue to investigate this issue, but preliminary findings suggest that there are duplicates and/or triplicates of the same claim. For this reason, if you need clarification for claims with a first or last submission date of April 19-23, please contact support, rather than re-submitting claims.

Percentage of claim updates completed:
277:
Trizetto: 58.48%
Emdeon: 97.32%

835:
Trizetto: 78.3%
Emdeon: 84.18%
Posted Apr 22, 2021 - 10:04 PDT
Update
We are continuing to parse new ERA files coming in each day for all clearinghouses. Waystar is up to date, while Trizetto and Emdeon continue to make progress. Database timeouts and non-timeout errors remain the two types of errors that are preventing a full recovery for the two remaining clearinghouses.

We’ve increased the amount of time that a query can last, allowing ERAs to be accurately parsed. We are still investigating non-timeout errors. The remaining error types are less recent and considered non-incident related bugs that will be addressed accordingly.

277s:
We continue parsing all new 277 files from Waystar and Trizetto. A fix was pushed to production that will parse the 277 backlogs for Trizetto and Emdeon. This will also enable new 277 files coming in from Emdeon to be parsed. For this reason, we ask customers not re-submit 277s as re-submitting will make processing take even longer.

If you are still not receiving information on a claim that should be updated, please contact support at drchrono.com/help with the claim number available so that we can assist further. We will post additional updates here each day until the issue is fully resolved. You may also find general claim information in our support article: https://support.drchrono.com/hc/en-us/articles/360059952791-Lifecycle-of-a-Claim
Posted Apr 21, 2021 - 17:23 PDT
Update
Thank you for your patience as we continue to work through the backlog of 277/ERA files that still need to be parsed so that we can update statuses and receipts in DrChrono. Please note that this does not affect any claims being submitted to your clearinghouse. If you are with Waystar, your information should be current. If you are with Trizetto/Gateway, we have processed 97.48% of the 277 files and 99.6% of the 835 files. If you are with Change Healthcare/Emdeon, we have processed 99.95% of the 277 files and 88.56% of the 835 files. We expect to be current by early next week. For additional context, please review the attached Knowledgebase article that provides information on a claim's lifecycle. If you think your claims are getting stuck somewhere in this process, please reach out to our support team at drchrono.com/help to assist you. We will post additional updates here as we get them. https://support.drchrono.com/hc/en-us/articles/360059952791-Lifecycle-of-a-Claim
Posted Apr 16, 2021 - 07:59 PDT
Update
Thank you for your patience as we continue to work through the backlog of 277/ERA files that still need to be parsed, so that we can update statuses and receipts in DrChrono. Please note that this does not affect any claims being submitted to your clearinghouse. If you are with Waystar, your information should be current. If you are with Trizetto/Gateway or Change Healthcare/Emdeon, we have processed all March ERA files and most 277 files, and are currently processing files from April and finishing the 277 files from late March that failed. We expect to be current by early next week. We will post additional updates here as we get them.
Posted Apr 12, 2021 - 10:23 PDT
Identified
We are still monitoring the ERA processing issues that have been occurring with the clearinghouses. Our team has identified the processing issue and is working towards a fix. We will provide another update as we have more information. Please note that this only impacts statuses and receipts updating in DrChrono - this issue does not impact claims processing with your clearinghouse.
Posted Apr 07, 2021 - 06:21 PDT
Update
We are still monitoring the ERA processing issues that have been occurring with the clearinghouses. There are currently processing delays with Emdeon/Change Healthcare that have arisen again. Our team continues to investigate this and we will provide another update as we have more information. Please note that this only impacts statuses and receipts updating in DrChrono - this issue does not impact claims processing with your clearinghouse.
Posted Apr 05, 2021 - 14:47 PDT
Monitoring
We appreciate your patience. We have resolved the ERA processing issues with TriZetto, Waystar, and Emdeon/Change Healthcare, and statuses and receipts are now updating in DrChrono. You may still experience slight delays to information as we work through the backlog, but please reach out to our support team if you have questions about specific claims.
Posted Apr 01, 2021 - 18:23 PDT
Update
We are continuing to investigate the root cause of the issue. Our team has put in place an alternative workflow to speed up the process of claims from TriZetto and Waystar and update statuses and receipts in DrChrono. This does not affect claims being submitted to your clearinghouse. We will continue to update you as we have more information.
Posted Apr 01, 2021 - 09:43 PDT
Update
We are continuing to investigate the root cause of the issue. However, our team has put in place an alternative workflow to speed up the process of claims from TriZetto and update statuses and receipts in DrChrono. This does not affect claims being submitted to your clearinghouse. We will continue to update you as we have more information.
Posted Mar 31, 2021 - 07:32 PDT
Investigating
We are aware of significant processing delays with ERAs and Claims status updates in DrChrono. We are actively working on identifying the root cause. Please note this does not affect the payments posting or bank account transfers but impacts the receipts not being added to DrChrono. This does not affect claims being submitted to your clearinghouse. We will continue to keep you updated as we have more information.
Posted Mar 30, 2021 - 11:44 PDT
This incident affected: 3rd party services (Emdeon/Change Healthcare, Trizetto, Waystar).
← Current Status Powered by Atlassian Statuspage