On 2023-12-15 at 18:49 UTC we began observing high latency and timeouts for a random small fraction of queries to Salesforce for a very specific group of customers. For these impacted customers, this resulted in the inability to load the forecast page in Clari. We mitigated this problem for the affected customers by changing the Clari settings to operate in a cached mode starting around 2023-12-18 at 17:00 UTC while we investigated the issue with Salesforce to address the underlying timeout problem. On 2024-01-05, Salesforce confirmed the source of the timeouts was a misconfigured node that was causing some requests to fail. They removed the faulty node from service at 2024-01-05 17:33 UTC and performance has returned to normal for the impacted customers. No customer data was at risk during this incident.
Posted Jan 08, 2024 - 18:57 UTC
Resolved
We have confirmed the issue has been resolved for the impacted users. A summary of the incident will be posted within the next three US business days.
Posted Jan 05, 2024 - 19:36 UTC
Update
Our engineering teams are continuing to work with Salesforce and some of the most heavily impacted customers to solve the underlying network issues that are creating the problem.
The forecast page timeouts have been solved for most impacted customers by adjusting the caching settings that Clari uses.
The next status update will be when the state of this incident changes.
Posted Jan 04, 2024 - 01:21 UTC
Monitoring
Our engineering teams have resolved the issue and we are monitoring the fix. Any customers who are still seeing problems loading their forecast pages should report the problem by opening a support ticket via the support portal.
Posted Dec 18, 2023 - 16:58 UTC
Identified
Our engineering teams have identified the issue and are implementing a fix.
The fix at this time is to increase the caching window in order to reduce the back-end call volume to Salesforce. We are implementing this adjustment to the most heavily affected customers.
The next status update will be after the fix has been applied.
Posted Dec 18, 2023 - 16:19 UTC
Investigating
Our engineering team is actively investigating reports from some customers that their forecast grid is showing up blank (back-end timeouts).
Posted Dec 15, 2023 - 23:00 UTC
This incident affected: Clari Revenue Platform (Forecast).