iCap and LEXI Incident

Post Incident Review Lexi/iCap Outage (8 December 2025)

Overview:

Incident Title: LEXI/iCap Outage – 8 December 2025

Date & Time (AEDT): 18:49 - 20:25

Date & Time (EST): 2:49 - 4:25

Date & Time (UTC): 7:49 - 9:25

Affected Service: iCap, Lexi

Impact Scope: Global

Duration: 96 minutes

Severity: Critical

‌

Summary of Incident:

On Monday 8 December 2025 at 07:49 UTC (02:49 EST / 6:49pm AEDT), iCap/LEXI services experienced a global outage, resulting in an inability to provide automated captioning and related services.

Our teams implemented human captioning where possible to minimise client impact. Services were restored at 9:25 UTC, with full verification completed before confirming resolution.

Timeline of Events (UTC):

• 7:49 – Outage detected by service delivery coordinators.
• 7:49 – 8:20 – Initial troubleshooting and escalation; human captioning initiated in APAC where possible to mitigate disruption.
• 9:05 – Technical investigation identified root cause.
• 9:10 – 9:15 - Fix implemented and verified by technical teams.
• 9:25 – Service restoration confirmed; impacted clients notified.

Impact:

• Duration: 96 minutes
• Services Affected: LEXI/iCap Live automated captioning, translation, and voice services
• Clients Impacted: Multiple broadcast and enterprise clients globally.

Root Cause

The iCap primary server became unresponsive for a short period which prompted an auto failover of load to the secondary server, in accordance with standard failover procedures. The secondary server became overloaded by the surge of connections.

The primary server auto recovered partially however it could not effectively communicate its health status to the secondary backup server to allow it to resume load.

Resolution:

A manual restart resulted in full health restoration of the primary server which triggered the load being effectively transferred from the secondary backup server to the primary. At full health the primary server resumed services load effectively.

Corrective Actions & Improvements:

1. Capacity review and uplift of both the primary and secondary server  
2. Upgrades to the communication process between the primary production server and the secondary backup server
3. Optimization of auto alerting

Commitment to Clients:

We recognize the impact this incident had on our clients’ operations and deeply regret the disruption. AI-Media is committed to:

• Maintaining transparent communication during and after incidents.
• Strengthening our technical safeguards to reduce the likelihood of similar outages.
• Continuously improving our disaster recovery capabilities to ensure service reliability.