This issue is now resolved. Performance is back to normal. While we have fixed the cause of the issue, we will be working thorugh a post mortem process. This will help us prevent similar incidents in the future, allow us to quicker diagnose and fix similar and related issues as well as improve processes.
Posted Sep 02, 2019 - 12:37 CEST
Monitoring
We believe we have solved the underlying issue and have reenabled search. It might be slower than usual while things stabilize.
Posted Sep 02, 2019 - 12:13 CEST
Update
We have disabled all search for now, as we recover the search cluster. This is in progress and we expect that to be completed soon.
Posted Sep 02, 2019 - 11:57 CEST
Identified
We have identified the issue as being a heavily loaded search cluster. We are working to determine the root cause and add additional capacity.
Posted Sep 02, 2019 - 11:39 CEST
Investigating
We are seeing errors performing searches in Peakon. We are investigating the issue.