What: Forecast API endpoints experienced missing data & increased errors
Start: 2025-09-23 09:35 UTC
End: 2025-09-23 14:56 UTC
Time To Recovery: 321 minutes
Outage Duration: 46 minutes
Endpoints: /forecast, /forecast/historical, /signal-index
User impact: API requests to /forecast, /forecast/historical, and /signal-index were taking longer than usual to query for data due to missing or delayed data at the time of request. In most cases requests were successful, though recent data were missing. For some of this time, intermittently, these endpoints also had an increased error rate (partial outage totaling roughly 46 minutes; this extent varied by user).
What went wrong: A low-level cluster planning tool inadvertently filled a critical data store, which interrupted the forecast data generation. Missing data slowed the API response time, and the API did not scale to meet more requests in parallel.
How we fixed it: Several adjustments were made to recover data generation. We identified a number of improvements to our recovery process to limit the impact of similar events going forward.
We apologize for any disruption the data delays and errors caused. We are committed to maintaining a secure and reliable API, and though we're proud of how well we perform, we still look for ways to improve.