Thursday @ 11:30 EST we were doing a support call with a client, during this call we were working out issues with our notifications for this customer.
We then placed some test notifications in what we believed to be a single teams queue. These messages got added to several teams queues on accident.
On Friday May 15, 2020 at 11:26 the notification service sent updates to approximately 10 bots that included valid Jira issue information but invalid updates attributed to users unknown to the customer.
These updates may have occurred 1 or more times during the effected period which has been identified as May 15, 2020.
There is no indication of a breach of our systems and customer Jira instances are not believed to be incorrectly updated. The issue appears to be due to user error and not due a security incident.
What we believe occurred A Nextup admin was testing a client issue and incorrectly placed test messages in multiple teams queues. An update was later received from Jira for a valid change. The application logic did not correctly pass the team element between services. The orphaned update was incorrectly attached to an additional valid notification. Additional validations have been added to the system along with additional logging to capture future potential occurrences along with additional debugging procedures to prevent future issues.
The system processes millions of notifications daily and additional teams have not reported any issues after the known incident. We are continuing to monitor the application for future occurrences.