Degraded Platform Performance

Incident Report for Salsify

Postmortem

At 11 AM EST Friday January 17, 2020, Salsify received an influx of events that caused significant delays to key Salsify services.Although most of the Salsify application continued to function as expected, customers experienced:

Out-of-date product and asset search results
Delayed workflow triggers and completion
Delayed email notifications
Long-running imports and publications
Discrepancies on product list pages

Our engineering team worked quickly to mitigate the issue. At 3 AM EST Saturday January 18, 2020, Salsify processed the remaining backlog of events, bringing our system back to a stable state. There was no data loss during the incident.

In the coming months, our team is dedicating significant resources to ensure platform stability, optimize event processing, and reduce the risk of a recurrence. If you have any questions or comments, please reach out to our support team by submitting a request at https://support.salsify.com/hc/en-us/requests.

Posted Jan 21, 2020 - 16:42 EST

Resolved

The issue impacting the Salsify application has now been resolved and the application is running as expected. If you have any questions or concerns, please reach out to our support team at https://support.salsify.com.

Posted Jan 18, 2020 - 11:58 EST

Monitoring

We've mitigated the underlying issue and are currently processing through our backlogs.

Posted Jan 17, 2020 - 18:44 EST

Identified

We've identified the root cause and are working towards a solution.

Posted Jan 17, 2020 - 18:05 EST

Update

We're currently experiencing platform degradation with updates enqueued for up to 6 hours. Our team continues to investigate the issue with urgency. We apologize for the inconvenience and appreciate your patience. More updates to follow.

Posted Jan 17, 2020 - 15:41 EST

Update

We are currently investigating an issue that is causing delayed processing of product audits, workflow trigger, and subscriptions.

Posted Jan 17, 2020 - 14:52 EST

Investigating

We are currently investigating an issue where digital assets are not displaying on the index page after a successful upload.

Posted Jan 17, 2020 - 14:31 EST

This incident affected: Salsify Application (Imports and Indexing, Salsify Application).