Service Information

Last update: 8th October, 2021

This page will be updated with details of any major incidents or scheduled maintenance affecting the Pando services.

Details about past incidents can be found below, along with uptime and API performance information.

Uptime

MonthUptime
October 2021100%
September 2021100%
August 202199.988%
July 202199.982%
June 2021100%
May 202199.913%
April 202199.997%
March 202199.990%
February 202199.973%
Uptime/SLA as defined by standard Pando commercial contract.
Data published from February 2021 onwards.

Performance   

MonthAverage API response timeAverage messaging system response time
October 202196ms3.9ms
September 2021110ms3.9ms
August 2021140ms6.2ms
July 202184ms5.9ms
June 202180ms5.0ms
May 2021111ms4.0ms
April 202172ms3.7ms
March 202181ms3.5ms
February 2021190ms4.2ms
January 2021153ms3.3ms
December 2020121ms2.8ms
November 2020111ms2.7ms
October 202090ms2.6ms
September 202072ms2.6ms
Target (non-contractual) maximum time 200ms for API response, 50ms for messaging system response.
Data published from September 2020 onwards.

Past incidents (all times are Europe/London)

31 August 2021 – API Performance Degradation & Short Outage

A database server issue caused a period of degraded API performance between 10:18:00 and 10:21:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards. There was a short outage between 10:31:11 and 10:31:41 to apply mitigation measures to prevent this from re-occurring.
31 August 2021, 10:35:00

24 August 2021 – API Performance Degradation

A database server issue caused a period of degraded API performance between 08:55:56 and 09:08:00. Message delivery was unaffected during this time, but users would have experienced delays in logging in, searching and patient cards.
24 August 2021, 11:00:00

7 July 2021 – Pando Messaging System Outage

The Pando Clinical Messaging System was unavailable from 16:04:48 to 16:12:46. Users were able to log in, but were unable to send or receive messages during this time.
8 July 2021, 08:00:00

10 May 2021 – Pando Service Outage

The Pando Service was unavailable from 12:36:00 to 13:14:00. A software error caused a database server to become unresponsive. Mitigation measures have been applied to prevent this issue from reoccurring.
10 May 2021, 17:08:30

7 April 2021 – Short Unplanned Outage

The Pando Service was disrupted from 16:14:18 to 16:15:34. A configuration error prevented API operations from working correctly. Some message deliveries were delayed. Full normal service was quickly restored. Mitigation measures have been applied to prevent this issue from reoccurring.
7 April 2021, 16:55:00

8 March 2021 – Severe Performance Degradation

The Pando Service was disrupted from 09:53:30 to 09:57:00. A scheduled daily job failed to run correctly causing database performance problems. The job should have taken one or two seconds to run, but instead took six minutes. The problem with the daily job was quickly identified and fixed, and will not reoccur.
9 March 2021, 11:00:00

2 March 2021 – Short Messaging System Outage

All Messaging Services were unavailable from 11:47:10 to 11:47:50, during which time users were unable to send or receive messages. This was caused by a configuration change being pushed out to all messaging servers simultaneously rather than the servers being updated one-by-one. The problem with the deployment system was quickly identified and fixed, and will not reoccur.
2 March 2021, 16:00:00

1 February 2021 – Pando API Outage

EC2 and RDS failures in the AWS euw-az3 availability zone caused a failure in the Pando API at 11:08:00. Messaging services were unaffected for logged in users. Although the API has been engineered (and demonstrated) to be tolerant of availability zone failure, this particular set of circumstances was unusual and unforeseen. Full service was restored at 11:19:55, and we have subsequently extended our ability to cope with AZ failure.
9 February 2021, 09:35:00

Incident history published from February 2021 onwards.