Last update: 8th October, 2021
This page will be updated with details of any major incidents or scheduled maintenance affecting the Pando services.
Details about past incidents can be found below, along with uptime and API performance information.
Data published from February 2021 onwards.
|Month||Average API response time||Average messaging system response time|
Data published from September 2020 onwards.
Past incidents (all times are Europe/London)
31 August 2021 – API Performance Degradation & Short Outage
A database server issue caused a period of degraded API performance between 10:18:00 and 10:21:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards. There was a short outage between 10:31:11 and 10:31:41 to apply mitigation measures to prevent this from re-occurring.
31 August 2021, 10:35:00
24 August 2021 – API Performance Degradation
A database server issue caused a period of degraded API performance between 08:55:56 and 09:08:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards.
24 August 2021, 11:00:00
7 July 2021 – Pando Messaging System Outage
The Pando Clinical Messaging System was unavailable from 16:04:48 to 16:12:46. Users were able to log in, but were unable to send or receive messages during this time.
8 July 2021, 08:00:00
10 May 2021 – Pando Service Outage
The Pando Service was unavailable from 12:36:00 to 13:14:00. A software error caused a database server to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
10 May 2021, 17:08:30
7 April 2021 – Short Unplanned Outage
The Pando Service was disrupted from 16:14:18 to 16:15:34. A configuration error prevented API operations from working correctly. Some message deliveries were delayed. Full normal service was quickly restored. Mitigation measures have been applied to prevent this issue from reoccurring.
7 April 2021, 16:55:00
8 March 2021 – Severe Performance Degradation
The Pando Service was disrupted from 09:53:30 to 09:57:00. A scheduled daily job failed to run correctly causing database performance problems. The job should have taken one or two seconds to run, but instead took six minutes. The problem with the daily job was quickly identified and fixed, and will not reoccur.
9 March 2021, 11:00:00
2 March 2021 – Short Messaging System Outage
All Messaging Services were unavailable from 11:47:10 to 11:47:50, during which time users were unable to send or receive messages. This was caused by a configuration change being pushed out to all messaging servers simultaneously rather than the servers being updated one-by-one. The problem with the deployment system was quickly identified and fixed, and will not reoccur.
2 March 2021, 16:00:00
1 February 2021 – Pando API Outage
EC2 and RDS failures in the AWS euw-az3 availability zone caused a failure in the Pando API at 11:08:00. Messaging services were unaffected for logged in users. Although the API has been engineered (and demonstrated) to be tolerant of availability zone failure, this particular set of circumstances was unusual and unforeseen. Full service was restored at 11:19:55, and we have subsequently extended our ability to cope with AZ failure.
9 February 2021, 09:35:00
Incident history published from February 2021 onwards.