Delivery rate of email notifications has been restored. We are closing this incident ticket.
Posted Aug 27, 2021 - 09:22 PDT
Update
We'll be leaving this status open for the next 12 hours as we monitor delivery rates and sender reputation recovery.
Posted Aug 26, 2021 - 17:58 PDT
Monitoring
All fixes are complete and email notifications are flowing normally. However, you may find that some Ghost Inspector emails are being flagged as spam in your mail service at this time. This is due to a decline in our sending reputation which we are working to repair. Our expectation is that our sending reputation will return to normal within 24 hours and our emails will no longer be flagged as spam.
The emailing sending reputation decline and slow downs we experienced resulted from a configuration error in our (completely isolated) test environment that allowed for an EC2 instance to relay public email through SendGrid over port 25 for a short period of time. This allowed spam email to be relayed during that window of time which interrupted delivery of legitimate email and hurt the sending reputation of our SendGrid account. We have confirmed that this relaying of email was not facilitated through an intrusion and did not result in any access to Ghost Inspector data. We are outlining and implementing additional measures to ensure that this type of incident cannot occur again in the future. We apologize to our customers for the interruption in email notifications from our service.
Posted Aug 26, 2021 - 16:17 PDT
Update
We have identified and resolved the root cause of the issue which is leading to delayed email notifications, emails going to spam folders, and in some cases, emails being dropped entirely. We are working with SendGrid (our email provider) to resolve the impacts as quickly as possible and will follow up with a detailed explanation of issue once the effects have been resolved.
Posted Aug 26, 2021 - 15:00 PDT
Identified
We are investigating a slow down in the delivery of test notification emails.