Elevated error levels on survey and dashboard

Incident Report for Peakon

Resolved

All systems are now back to operating as normal.

Posted Mar 17, 2020 - 08:23 CET

Identified

The web applications are now operating normally. Background jobs are still being processed at a reduced rate.

Posted Mar 17, 2020 - 08:14 CET

Update

We are continuing to investigate this issue.

Posted Mar 17, 2020 - 08:12 CET

Update

We have identified the root cause to be a slow database query, and will be looking at ways to improve performance. Meanwhile, we are processing background jobs, including email sending, at a reduced rate. Error levels have now returned to normal and we are monitoring the situation closely.

Posted Mar 17, 2020 - 07:51 CET

Investigating

We are seeing an increased error rate for the Peakon dashboard and survey applications due to high database load. We are investigating to find the root cause of the issue.

Posted Mar 17, 2020 - 07:29 CET

This incident affected: Dashboard, Survey, and API.