On Tuesday morning, we noticed delays in analytics job completion, which resulted in stale data appearing on customer dashboards. Some infrastructure components also experienced intermittent restarts, leading to brief service disruptions. Our engineering team investigated the issue in collaboration with AWS and identified a degraded task within the underlying cloud infrastructure as a contributing factor. As a precaution, we paused data processing, reverted recent changes, and initiated a controlled restart process around 4 PM PT, completing full recovery by approximately 10 PM PT.
AWS later indicated that the behavior was due to a rare condition in their storage subsystem, which can affect performance when certain transactions run concurrently. Upon the internal corrective actions, all systems have since returned to normal.
If you have any concerns or require additional information, please don't hesitate to contact our Customer Support team via hi@envoy.com or contact your dedicated account manager.