Lexi - Significant outage

Incident Report for Ai-Media

Postmortem

At 2:22am EDT / 6:22am UTC on August 9, an AI-Media staff member performing a deployment to a staging environment inadvertently reconfigured a routing table used by the production LEXI Live system. This resulted in all active, cloud-based LEXI Live jobs- as well as any jobs started thereafter- being unable to generate captions until the condition was rectified at 8:10am EDT / 12:10pm UTC. Additionally, the outage did not correctly trigger failover to LEXI DR systems, except where users had jobs scheduled in EEG Cloud; scheduled jobs failed over correctly. Further work is planned to prevent similar disruptions going forward.

Posted Aug 19, 2025 - 15:12 UTC

Resolved

Our team believes this to be resolved as of 8:10AM ET. We are still closely monitoring and an investigation into the root cause is still ongoing.
Posted Aug 09, 2025 - 13:09 UTC

Monitoring

Our team has implemented a fix and we are monitoring.
Posted Aug 09, 2025 - 12:23 UTC

Investigating

Starting at approximately 2:25AM ET on 8/9/25, the Lexi system began experiencing difficulties allowing new jobs to be created. This is a critical issue that is actively being investigated by our team. We will update as soon as we have more information.
Posted Aug 09, 2025 - 06:25 UTC
This incident affected: Lexi.