tag:status.watttime.org,2005:/historyWattTime API Status - Incident History2024-03-28T08:47:42-07:00WattTime APItag:status.watttime.org,2005:Incident/180424722023-08-03T00:30:00-07:002023-08-03T09:09:11-07:00Stale data in some regions<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>00:30</var> PDT</small><br><strong>Resolved</strong> - We observed an unusual delay in the publishing of CO2 MOER data on the /data and /index endpoints in some US grid regions.<br />Data in those regions was stale starting at 2023-08-03T07:25 UTC.<br />This issue was resolved and any missing data was backfilled by 2023-08-03T15:40 UTC.</p>tag:status.watttime.org,2005:Incident/178563482023-07-30T23:16:27-07:002023-07-30T23:16:27-07:00Scheduled Maintenance: 30 July 2023 - Impact: none to very minor<p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>23:16</var> PDT</small><br><strong>Completed</strong> - All systems are now back to operational condition.</p><p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>23:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>10:45</var> PDT</small><br><strong>Scheduled</strong> - What: Scheduled API maintenance for security and database upgrades<br /><br />Expected Impact: None to Minor<br />Downtime - less than 5 minutes <br />Data recency - there may be a period where data is stale for up to 10 minutes<br /><br />When: During a 3 hour window on July 30-31<br />2300-0200 PDT</p>tag:status.watttime.org,2005:Incident/177500502023-07-03T01:13:43-07:002023-07-03T03:22:17-07:00Increased Error Rates<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>01:13</var> PDT</small><br><strong>Resolved</strong> - An automated database migration event caused an increase in error rates on the API. This has been identified and resolved.</p>tag:status.watttime.org,2005:Incident/157319752023-01-22T23:19:23-08:002023-01-22T23:19:23-08:00Scheduled Maintenance: 23 Jan 2023 - Impact: none to very minor<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>23:19</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>23:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>11:11</var> PST</small><br><strong>Scheduled</strong> - What: Scheduled API maintenance for security and database upgrades<br /><br />Expected Impact: None to Minor<br /> Downtime - none expected<br /> Data recency - there may be a period where data is stale for up to 10 minutes<br /><br />Duration: 3 hr maintenance window<br />2300 Sunday, Jan 22 - 0200 Monday, Jan 23</p>tag:status.watttime.org,2005:Incident/157319692023-01-15T23:50:19-08:002023-01-15T23:50:19-08:00Scheduled Maintenance: 16 Jan 2023 - Impact: none to very minor<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>23:50</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>23:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>11:10</var> PST</small><br><strong>Scheduled</strong> - What: Scheduled API maintenance for security and database upgrades<br /><br />Expected Impact: None to Minor<br /> Downtime - none expected<br /> Data recency - there may be a period where data is stale for up to 10 minutes<br /><br />Duration: 3 hr maintenance window<br />2300 Sunday, Jan 15 - 0200 Monday, Jan 16</p>tag:status.watttime.org,2005:Incident/136613712022-12-19T01:33:51-08:002022-12-19T01:33:51-08:00Database and security upgrades (#2)<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>01:33</var> PST</small><br><strong>Completed</strong> - Maintenance has been completed successfully. All systems now operational.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>23:55</var> PST</small><br><strong>Update</strong> - Phase 2 is progressing, there will be a possibility of transient errors as workloads are moved around in response to underlying instance changes.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>23:05</var> PST</small><br><strong>Update</strong> - Phase 1 is now completed. No downtime was observed. Phase 2 will begin and is likely to take the rest of the maintenance period.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>23:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>07:34</var> PST</small><br><strong>Scheduled</strong> - Scheduled API maintenance for security and database upgrades<br />Expected Impact on API: None to Minor<br />- Downtime - unlikely, but if it occurs, expected to be less than 10 minutes<br />- Data recency - no stale data expected<br /><br />Maintenance will be in two phases<br />- Phase 1 includes a possibility of downtime and is as a result of a security reconfiguration of load balancers.<br />- Phase 2 may result in an increased number of 502 or 504 errors, requiring retries. We will update status as this progresses.</p>tag:status.watttime.org,2005:Incident/136613502022-12-12T00:00:25-08:002022-12-12T00:00:25-08:00Database and security upgrades (#1)<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>00:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>23:53</var> PST</small><br><strong>Update</strong> - Second phase of maintenance is now in progress - there may be some data recency issues for a short period</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>23:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>07:31</var> PST</small><br><strong>Scheduled</strong> - Scheduled API maintenance for security and database upgrades<br />Expected Impact on API: None to Minor<br />- Downtime - none expected<br />- Data recency - there may be a period where data is stale for up to 15 minutes</p>tag:status.watttime.org,2005:Incident/100397432022-05-20T13:26:48-07:002022-05-20T13:26:48-07:00/data intermittent outage<p><small>May <var data-var='date'>20</var>, <var data-var='time'>13:26</var> PDT</small><br><strong>Resolved</strong> - We have deployed a fix and we have not seen any other failed requests since.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>12:45</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>12:44</var> PDT</small><br><strong>Monitoring</strong> - WattTime observed some failed requests (504) to the /data endpoint.<br />There were a total of 18 minutes within the following window where we observed failed requests.<br />Start: 05/20/2022 10:45:39 AM (Pacific Time, US)<br />End: 05/20/2022 11:26:39 AM (Pacific Time, US)<br />We have put a temporary fix in place and that has stabilized the endpoint.</p>tag:status.watttime.org,2005:Incident/98735012022-04-25T15:23:00-07:002022-04-28T10:32:22-07:00API security update resulted in a brief outage<p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>15:23</var> PDT</small><br><strong>Resolved</strong> - A regular security update to our production API resulted in a networking issue. <br />The API was not responsive from 22:23 UTC to 22:54 UTC on April 25, 2022.<br />This issue was resolved by upgrading the version of the affected component.</p>tag:status.watttime.org,2005:Incident/97852342022-04-12T13:30:00-07:002022-04-14T14:28:53-07:00Repeating values on /data<p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>13:30</var> PDT</small><br><strong>Resolved</strong> - WattTime made a change to how our cache works for the /data endpoint recently; that change contained a bug. The impact is that since 2022-04-12 20:58+00:00, if you queried /data with a starttime within the last 30 days you would have received erroneously repeating values. This affected all grid regions (BAs).<br /><br />We rolled back the bug and verified the issue was resolved, as of 2022-04-14 20:33+00:00. <br /><br />We recommend that if you regularly query /data to store MOER values in your own database (e.g. for analysis purposes), you should rerun your queries that you made between 2022-04-12 20:45+00:00 and 2022-04-14 20:45+00:00 to overwrite the values you may have saved in your own database.<br /><br />We're sorry if this has caused any inconvenience.</p>tag:status.watttime.org,2005:Incident/91053122022-01-29T17:15:35-08:002022-01-29T17:15:35-08:00Database update (AWS)<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>17:15</var> PST</small><br><strong>Completed</strong> - Database update has completed successfully without downtime.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>16:21</var> PST</small><br><strong>Update</strong> - Database upgrade is in progress - we don't expect any downtime although there may be stale data for a short time towards the top of the hour.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>16:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>11:17</var> PST</small><br><strong>Scheduled</strong> - This outage will allow us to perform mandatory updates required by AWS.<br />Outage Duration: Up to 30 minutes<br />Outage Window: 6 hours (The outage will occur within this window)</p>tag:status.watttime.org,2005:Incident/91520412022-01-24T06:14:47-08:002022-01-24T06:14:47-08:00Timeouts on /ba-from-loc and /data<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>06:14</var> PST</small><br><strong>Resolved</strong> - Internal and external testing shows that endpoint behavior is back to normal. We apologize for any impact this caused.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>05:28</var> PST</small><br><strong>Monitoring</strong> - We identified the issue and deployed a fix.<br />Responses from /ba-from-loc and /data are consistent and fast again.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>05:04</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting certain endpoints, including /ba-from-loc and /data. We will update this incident as we learn more.</p>tag:status.watttime.org,2005:Incident/88581522021-12-15T07:15:00-08:002021-12-15T11:18:02-08:00API outage<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>07:15</var> PST</small><br><strong>Resolved</strong> - An AWS outage caused the WattTime API to go down for a total of 28 minutes between 12/15/2021 07:24:21 AM PST and 12/15/2021 07:57:21 AM PST. The API has been operational since then.</p>tag:status.watttime.org,2005:Incident/87398542021-12-01T21:00:00-08:002021-12-02T00:50:39-08:00Increased error rate identified on /data endpoint<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>21:00</var> PST</small><br><strong>Resolved</strong> - Requests on the API data endpoint during the period 20:00 to 21:00 PST showed increased error rates.</p>tag:status.watttime.org,2005:Incident/83995622021-11-04T17:57:16-07:002021-11-04T17:57:16-07:00Increased error rate<p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:57</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:47</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:28</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.watttime.org,2005:Incident/81787792021-10-14T14:06:45-07:002021-10-14T14:06:45-07:00Minor API maintenance<p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>14:06</var> PDT</small><br><strong>Completed</strong> - It looks like we made it through the period without any outages. Woo!</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>13:42</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>11:21</var> PDT</small><br><strong>Update</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>11:08</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>12:53</var> PDT</small><br><strong>Scheduled</strong> - Within this window, we will be deploying some code updates to production. There is a very small chance of API down time that will be very brief. Please contact us at support@watttime.org if you have any questions.</p>tag:status.watttime.org,2005:Incident/78872072021-08-31T15:54:35-07:002021-08-31T15:54:35-07:00Intermittent connection issues<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>15:54</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>15:41</var> PDT</small><br><strong>Monitoring</strong> - Intermittent connection issues have been identified on the WattTime corporate website. A fix has been implemented and we are monitoring the situation.</p>tag:status.watttime.org,2005:Incident/78553192021-08-29T15:56:11-07:002021-08-29T15:56:11-07:00API Updates<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>15:56</var> PDT</small><br><strong>Completed</strong> - Maintenance has been completed.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>15:50</var> PDT</small><br><strong>Verifying</strong> - Works have been completed, we are now verifying.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>23:40</var> PDT</small><br><strong>Scheduled</strong> - We will be updating the endpoints that service our API. There may be some instability in the API. If your client uses DNS caching, it may be affected. You can either disable this or reduce the cache time to less than 5 minutes to compensate.</p>tag:status.watttime.org,2005:Incident/64454692021-03-02T08:26:29-08:002021-08-26T23:27:36-07:00This is an example incident<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>08:26</var> PST</small><br><strong>Resolved</strong> - Empathize with those affected and let them know everything is operating as normal.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>08:16</var> PST</small><br><strong>Monitoring</strong> - Let your users know once a fix is in place, and keep communication clear and precise.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>08:22</var> PST</small><br><strong>Identified</strong> - As you continue to work through the incident, update your customers frequently.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>08:52</var> PST</small><br><strong>Investigating</strong> - When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.</p>