Apr 25, 11:39 UTC
Resolved - Service has returned to normal. We've identified a small number of malfunctioning workers in the API worker pool that were unable to handle requests. This caused some requests to time out. We have cycled the worker pool to ensure any affected workers are removed.
Apr 25, 10:57 UTC
Investigating - We're experiencing higher than usual request latency. We're currently investigating the issue, and will provide an update soon.
Apr 25, 10:57 UTC
Investigating - We're experiencing higher than usual request latency. We're currently investigating the issue, and will provide an update soon.
Apr 16, 21:30 UTC
Resolved - Database performance has returned to normal.
Apr 16, 20:10 UTC
Monitoring - Database performance is returning to normal levels. Response team are monitoring.
Apr 16, 20:03 UTC
Update - We're continuing to investigate the database performance issue. Impact is isolated to a subset of customers.
Apr 16, 19:00 UTC
Investigating - We're monitoring a performance degradation in one of our pipelines database shards causing some increase in agent API latency
Apr 16, 20:10 UTC
Monitoring - Database performance is returning to normal levels. Response team are monitoring.
Apr 16, 20:03 UTC
Update - We're continuing to investigate the database performance issue. Impact is isolated to a subset of customers.
Apr 16, 19:00 UTC
Investigating - We're monitoring a performance degradation in one of our pipelines database shards causing some increase in agent API latency
Apr 16, 20:03 UTC
Update - We're continuing to investigate the database performance issue. Impact is isolated to a subset of customers.
Apr 16, 19:00 UTC
Investigating - We're monitoring a performance degradation in one of our pipelines database shards causing some increase in agent API latency
Apr 16, 19:00 UTC
Investigating - We're monitoring a performance degradation in one of our pipelines database shards causing some increase in agent API latency
Mar 31, 01:14 UTC
Completed - The scheduled maintenance is now complete. Resuming normal service.
Mar 31, 00:27 UTC
Verifying - Upgrades to in-memory caches for Web UI and APIs are complete.
We are not expecting any further impact to service. However, we are actively monitoring all components for any unexpected disruption.
Mar 30, 23:52 UTC
Update - Test Engine has returned to service.
Upgrades to the Web UI and APIs are continuing, and those components continue to show slightly elevated error rates.
Mar 30, 23:31 UTC
Update - Routine updates are now in progress.
- Test Engine is has been taken offline for upgrades. Test Engine elements will not be visible when viewing builds. Expected return to service within 30 minutes.
- We are observing slightly elevated error rates on the Web UI and other APIs.
Mar 30, 23:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 28, 00:21 UTC
Scheduled - From 30 March 23:00 UTC to 31 March 03:00 UTC, we will perform routine software updates to our in-memory caching infrastructure to improve performance, security and reliability. While we have taken steps to minimize disruption, we anticipate brief periods of increased error rates and/or elevated latency during this 4 hour window.
The Buildkite Dashboard and APIs may be intermittently slow or return errors. Assignment of work to your Agents may be delayed, and Agents may receive errors when submitting logs or status updates, for periods of up to 10 minutes.
A full service outage is not expected, but in that highly unlikely event we will update promptly on https://buildkitestatus.com.
Mar 31, 00:27 UTC
Verifying - Upgrades to in-memory caches for Web UI and APIs are complete.
We are not expecting any further impact to service. However, we are actively monitoring all components for any unexpected disruption.
Mar 30, 23:52 UTC
Update - Test Engine has returned to service.
Upgrades to the Web UI and APIs are continuing, and those components continue to show slightly elevated error rates.
Mar 30, 23:31 UTC
Update - Routine updates are now in progress.
- Test Engine is has been taken offline for upgrades. Test Engine elements will not be visible when viewing builds. Expected return to service within 30 minutes.
- We are observing slightly elevated error rates on the Web UI and other APIs.
Mar 30, 23:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 28, 00:21 UTC
Scheduled - From 30 March 23:00 UTC to 31 March 03:00 UTC, we will perform routine software updates to our in-memory caching infrastructure to improve performance, security and reliability. While we have taken steps to minimize disruption, we anticipate brief periods of increased error rates and/or elevated latency during this 4 hour window.
The Buildkite Dashboard and APIs may be intermittently slow or return errors. Assignment of work to your Agents may be delayed, and Agents may receive errors when submitting logs or status updates, for periods of up to 10 minutes.
A full service outage is not expected, but in that highly unlikely event we will update promptly on https://buildkitestatus.com.
Mar 30, 23:52 UTC
Update - Test Engine has returned to service.
Upgrades to the Web UI and APIs are continuing, and those components continue to show slightly elevated error rates.
Mar 30, 23:31 UTC
Update - Routine updates are now in progress.
- Test Engine is has been taken offline for upgrades. Test Engine elements will not be visible when viewing builds. Expected return to service within 30 minutes.
- We are observing slightly elevated error rates on the Web UI and other APIs.
Mar 30, 23:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 28, 00:21 UTC
Scheduled - From 30 March 23:00 UTC to 31 March 03:00 UTC, we will perform routine software updates to our in-memory caching infrastructure to improve performance, security and reliability. While we have taken steps to minimize disruption, we anticipate brief periods of increased error rates and/or elevated latency during this 4 hour window.
The Buildkite Dashboard and APIs may be intermittently slow or return errors. Assignment of work to your Agents may be delayed, and Agents may receive errors when submitting logs or status updates, for periods of up to 10 minutes.
A full service outage is not expected, but in that highly unlikely event we will update promptly on https://buildkitestatus.com.
Mar 30, 23:31 UTC
Update - Routine updates are now in progress.
- Test Engine is has been taken offline for upgrades. Test Engine elements will not be visible when viewing builds. Expected return to service within 30 minutes.
- We are observing slightly elevated error rates on the Web UI and other APIs.
Mar 30, 23:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 28, 00:21 UTC
Scheduled - From 30 March 23:00 UTC to 31 March 03:00 UTC, we will perform routine software updates to our in-memory caching infrastructure to improve performance, security and reliability. While we have taken steps to minimize disruption, we anticipate brief periods of increased error rates and/or elevated latency during this 4 hour window.
The Buildkite Dashboard and APIs may be intermittently slow or return errors. Assignment of work to your Agents may be delayed, and Agents may receive errors when submitting logs or status updates, for periods of up to 10 minutes.
A full service outage is not expected, but in that highly unlikely event we will update promptly on https://buildkitestatus.com.