Test Runner Network Issues
Incident Report for Ghost Inspector
Resolved
Amazon's US-East-1 S3 has been marked as resolved by their team. We've been caught up and operating normally for the past hour now, so we're marking this incident as resolved.
Posted Feb 28, 2017 - 14:46 PST
Monitoring
Amazon's US-East-1 S3 service appears to be recovering and our test running queue is catching up. We're still receiving some timeouts and 500 errors from S3, but the rates are decreasing. We're continuing to monitor the situation.
Posted Feb 28, 2017 - 12:40 PST
Identified
We have identified the issue as large scale outage of Amazon's US-East-1 S3 service, which is used by Ghost Inspector. Ghost Inspector is still operational during this outage, but:

- Capacity is reduced
- File uploads during tests are unavailable
- Result screenshots and video are not being capture

We are working to remedy these issues as much as we can on our end.
Posted Feb 28, 2017 - 10:48 PST
Investigating
We're currently investigating a higher rate of failing tests due networking issues on our test running instances.
Posted Feb 28, 2017 - 10:22 PST