At approximately 11:30am PDT a code change was deployed to our test running service that updated the way we handle JavaScript processing in our step execution. This code had been fully tested in our staging environment and had the green light for production.
Shortly after the deployment around noon PDT internal alarms regarding the queue backlog and some other test runner alerts were received internally that indicated diminished test runner capacity. Immediate steps were taken to restore capacity however it was not identified that the code change was causing an ongoing problem until approximately 10pm PDT and test runner capacity gradually diminished over time throughout the day as browser processes ran beyond the 10-minute test run limit due to high resource consumption.
Once the team realized capacity was diminishing again we identified the recent code change as the culprit and immediately rolled back the code.
Service was restore to normal operating capacity around approximately 11pm PDT. Any customers having received a warning of "10 minute timeout" are encouraged to re-execute the test and contact support if any issues remain.