This issue is now resolved. Performance is back to normal. While we have fixed the cause of the issue, we will be working thorugh a post mortem process. This will help us prevent similar incidents in the future, allow us to quicker diagnose and fix similar and related issues as well as improve processes.
Posted Sep 02, 2019 - 12:37 CEST
We believe we have solved the underlying issue and have reenabled search. It might be slower than usual while things stabilize.
Posted Sep 02, 2019 - 12:13 CEST
We have disabled all search for now, as we recover the search cluster. This is in progress and we expect that to be completed soon.
Posted Sep 02, 2019 - 11:57 CEST
We have identified the issue as being a heavily loaded search cluster. We are working to determine the root cause and add additional capacity.
Posted Sep 02, 2019 - 11:39 CEST
We are seeing errors performing searches in Peakon. We are investigating the issue.