Lexi scheduling issues - 11/21/24
Incident Report for Ai-Media
Postmortem

What happened?

Today around 12:05pm ET / 17:05 UTC, we deployed an update to our production Lexi environment, specifically to address the scheduling system. Unfortunately this deployment contained a configuration error which made it unable to communicate with the database that stores Lexi scheduled events. This prevented scheduled Lexi events from starting, and also prevented users from accessing the Lexi Scheduling page on eegcloud.tv. Once we discovered this issue, we rolled back the change, which restored functionality. Lexi scheduling was fully operational again by 1:16pm ET / 18:16 UTC.

What are we doing about it?

Going forward we will:

  • Review & update our testing procedures in order to identify problems like this before they're deployed
  • Implement additional alerts so we can be notified more quickly about any production-impacting issues

‌Ai-Media understands the importance of a reliable cloud service for closed captioning and we apologize for the disruption this incident caused to Lexi users. If there are any follow-up questions on this incident, please submit a ticket to eeg.support@ai-media.tv.

Posted Nov 21, 2024 - 23:33 UTC

Resolved
Lexi's scheduling functionality has been fully restored and continues to operate as expected. Marking this incident as resolved, post-mortem to follow.
Posted Nov 21, 2024 - 19:59 UTC
Monitoring
Our engineering team has implemented a hotfix to restore access to the Lexi scheduler. We are monitoring to ensure everything is operating normally.
Posted Nov 21, 2024 - 18:19 UTC
Investigating
We have received reports that the Lexi scheduler is inaccessible and/or failing to start scheduled events. We are investigating and will report back here once we have more information.
Posted Nov 21, 2024 - 18:11 UTC
This incident affected: Lexi.