Real-Time Analytics are temporarily paused

Incident Report for Envoy

Postmortem

On Tuesday morning, we noticed delays in analytics job completion, which resulted in stale data appearing on customer dashboards. Some infrastructure components also experienced intermittent restarts, leading to brief service disruptions. Our engineering team investigated the issue in collaboration with AWS and identified a degraded task within the underlying cloud infrastructure as a contributing factor. As a precaution, we paused data processing, reverted recent changes, and initiated a controlled restart process around 4 PM PT, completing full recovery by approximately 10 PM PT.

AWS later indicated that the behavior was due to a rare condition in their storage subsystem, which can affect performance when certain transactions run concurrently. Upon the internal corrective actions, all systems have since returned to normal. 

If you have any concerns or require additional information, please don't hesitate to contact our Customer Support team via hi@envoy.com or contact your dedicated account manager.

Posted Apr 03, 2025 - 10:26 PDT

Resolved

The underlying task has been completed, and analytics data is now up to date. We appreciate your patience while the functionality was being restored.
Posted Apr 01, 2025 - 21:44 PDT

Investigating

Due to an internal process temporarily impacting system performance, we've paused the tasks that update analytics in real time. These processes will resume after office hours, and we expect analytics functionality to be fully restored by 9 PM PST.

Until then, dashboards may show stale data. We appreciate your understanding.

For any questions/concerns, please reach out to Envoy Support via hi@envoy.com
Posted Apr 01, 2025 - 16:21 PDT
This incident affected: Analytics.