How Many Hottest Days of the Year (So Far)?

Summary

• The average number of days in a year which can be called ‘hottest day of the year so far’ is 20.0
• If we want to restrict the number of ‘noteworthy hottest days of the year so far’ to 3/year, we can restrict noteworthiness to days which are at least 20˚C and at least 1.6˚C hotter than the previous hottest day
• The places which set the most ‘hottest day of the year so far’ records are in the south east of England

Introduction

I often see news articles along the lines of ‘beach-goers out in force as record temperatures reached in Cornwall’. By ‘record temperatures’ these articles often mean ‘hottest day of the year so far’ (which I will abbreviate to HDSF). This often left me wondering: are ‘hottest day of the year so far’ stories really noteworthy? There must be several days in any year upon which you could write this type of story. In the UK, what is the average number of days in a given calendar year which could be described as the ‘hottest day of the year so far’?

After some unfruitful internet searches, I was left wondering. I considered writing to the team on the fantastic radio programme More or Less (I'm an incredibly loyal listener), but eventually decided to dig into the numbers myself. This post shares some of the findings.

And who knows, maybe one day I'll get my two minutes of fame on More or Less due to this post.

How Many Hottest-Days-So-Far?

I started directly with the simple question I wanted to answer: what is the average number of days in each year which could be described as the ‘hottest day so far’ (HDSF) in that year. For a day to be an HDSF, the highest recorded temperature at any station on that day simply needs to exceed the previous highest recorded station at any temperature on any day earlier in that year. So straight to the headline statistic:

The yearly average number of HDSFs in the UK between 1920 and 2020 is 20.0.

It's more or less what I expected — in fact, it's remarkably close to my rough-but-informed initial estimate¹.

We can also just look at the distribution:

The mean number of yearly HDSFs is 20.05. The median is 20. The standard deviation is 4.49. The minimum number of HDSFs is 11, in 1964, and the maximum is 34 in 1995.

At this point, you might be interested as to what a typical year looks like in terms of maximum daily temperature. 1978 has 20 HDSFs, so I plotted the daily max temperature over that year below.

What about the year with the most/least HDSFs, 1995/1964?:

We can also see whether there is a trend to the number of yearly HDSFs

It almost looks as if there's a discontinuity in the late 1960s, after which the average increases from 18.0 to 21.9. I'm not sure what causes this: perhaps an increased number of active weather stations or more reliable recording equipment leads to fewer sudden jumps in daily max temperature, and thus more yearly HDSFs.

Noteworthiness

The original motivation for this investigation was to count how many days newspapers could publish a ‘hottest day so far’ story each year. You might object that it's not noteworthy if a day is only 0.1˚C warmer than the previous HDSF, and therefore we shouldn't count such days.

To investigate this, I propose the concept of ‘ΔT-noteworthiness’: A day is considered ΔT-noteworthy (abbreviated ΔT-HDSF) if the maximum recorded temperature on that day is at least ΔT˚C more than the previous maximum recorded daily temperature². According to this definition, in the previous section we considered 0˚C-HDSFs. We can see a concrete example of the difference between 0˚C-HDSFs and 1˚C-HDSFs in the plot below:

Now we can answer questions such as ‘How does the noteworthiness threshold ΔT affect the number of HDSFs?’

We can also investigate which years have the most noteworthy temperature gaps. Initial investigation suggests 1963 is the year with the largest gap. On 12th Jan 1963, a temperature of 16.7˚C was recorded in Exeter Southam station, exceeding the previous max. recorded temperature of 8.1˚C by 8.6˚C. Apparently this was a particularly cold winter — one of the coldest on record in the UK.

I think this measurement is probably a mistake: while Exeter Southam station reported a high of 16.7˚C, the mean daily high temperature in the rest of Devon on this day was -1.7˚C, and the maximum daily high in Devon-excluding-Exeter-Southam was just 0.6˚C. It seems implausible that the temperature at the Exeter Southam station was actually 16˚C warmer than anywhere else in Devon.

The most noteworthy plausible gap was on 4th May 2006, when a gap of 6.3˚C was recorded. This looks less likely to be a mistake, due to a prolonged period of higher temperatures following the spike. The high was 27.7°C (compared to a nationwide average of 18.1°C), measured in Northolt, London. I can't find anything in the news about the date (other than a council election taking place in Northolt on this day!). There was a heatwave later in the summer though, with highs of 36.5°C recorded

(ΔT, Tmin)-Noteworthiness

It would seem strange to write a ‘hottest day’ news story about a 12˚C day, even if it's several degrees hotter than the previous hottest day so far. Also, it seems strange that we should always count 1st Jan as an HDSF. We can define a new criterion: (ΔT, Tmin)-noteworthiness. A day is (ΔT, Tmin)-noteworthy if it is ΔT-noteworthy and the temperature is above Tmin.

I think it's reasonable to set Tmin = 20˚C. I think it's also reasonable to write the ‘hottest day so far’ news story about 3 times per year. We can use this to make a recommendation to the press standards organisation: a day must be (1.6˚C, 20˚C)-noteworthy in order to write a ‘hottest day so far’ story.

It would be very interesting to investigate the circumstances under which newspapers publish ‘hottest day so far’ stories. Is (ΔT,Tmin)-noteworthiness a good model for empirical newsworthiness? My guess is that it depends partially on something like (ΔT,Tmin)-noteworthiness, but with additional factors: mainly whether or not the date falls on a weekend/bank holiday, but to a lesser extent on general news cycle events and time since the last article. If anyone takes a look at this, I'd be interested to hear your findings!

Where are the Hottest-Days-So-Far?

Since we have per-station data, we can investigate at which weather stations the (0,0)-HDSFs are recorded.

No surprise, these tend to be concentrated in the South East of England. We can plot these on a map

You'd guess that stations at lower latitudes, and perhaps more eastern longitudes. This is borne out in the data.

The above results are based on the count of HDSFs between 1920 and 2020. However, not all weather stations have been around for the same amount of time, so simply considering the raw HDSF count is biased toward larger stations. If we divide by the number of days the station has been operational, we can adjust for this bias, at the risk of introducing more variance into the ranking.

When are the Hottest-Days-So-Far?

When in the year do the hottest days so far occur? We'd probably expect these to be concentrated at the start of the year, and in the summer months, with autumn and later having basically zero chance of being hottest so far. We can plot this on a calendar:

As expected, the first few days are very likely to be HDSFs, and the latter half of the year has very few. The peak time for HDSFs is late April/early May³. The latest recorded HDSF was 13th September, in 2016. It even made the news! Unsurprisingly this record was measured at Gravesend.

Modelling

In future, I might revisit the question of whether we can reconstruct the observed data via simple modelling.

It appears that the daily max temperatures over a year could be modelled as a sinusoid with heteroscedastic noise (unsure of noise distribution, but Normal would probably be a good start). There's probably also significant autocorrelation so assuming daily noise is independently distributed might not be good.

Data

(I was going to include this section at the start, for the chronological narrative, but my guess is that most people won't be interested, so I moved it to the end, like an appendix)

I decided to jump straight in to computing statistics from temperature data. I initially questioned whether it was possible to skip data processing by looking at average temperatures and measures of spread. In the end, I decided against this because it would rest on assumptions I'd struggle to validate without access to the data anyway. I also expect the best I could do with easily accessible data is to build a model where daily high temperatures vary sinusoidally with independent daily Gaussian noise — but this is misleading, as I'd expect significant autocorrelation. Besides, playing with the data can be fun, and allow for more questions to be answered.

I needed a source of daily temperature information. I'm English, so naturally mostly consume British media, and therefore restricted the data to temperatures in the UK. The obvious place to go for UK weather data is the Met Office.

It turns out that the Met Office make available daily temperature information from each of their weather stations, stretching right back to 1853! The MIDAS dataset⁴ is available from the CEDA archives. Unfortunately, accessing this data is not so straightforward. The data is split up into different files, each containing a year's worth of temperature data for one weather station. I looked for a single file I could download which had all this data to no avail. The data is accessible via the CEDA FTP sever. I'd have to crawl over the relevant directory.

A note on the tools I used here: I was going to make this into a notebook-style document, with code interleaved with text. However, this is quite low priority for me, though I might revisit this post in the future if I have lots of free time. So I'll just vaguely describe what I did to process the data. Throughout, I used Julia, which is by far my favourite programming language for ‘scientific computing’ tasks.

To download the data, I used FTPClient.jl, which ended up being quite straightforward. Once I had each file, I had to parse it. CEDA use a format called BADC-CSV, which as far as I'm aware is a CEDA ‘in house’ format. I couldn't find a ready-made BADC-CSV parser for Julia, but fortunately it took only small modifications to the functionality provided by CSV.jl to read the files.

Once I parsed the data, I saved it to a new file to avoid having to scrape the CEDA FTP server every time. I chose to save it to an HDF5 file using HDF5.jl, preferring the HDF5 format over CSV since the resulting data was 1591×63,000. With over 100M entries, many of which were empty (due to weather stations not operating in certain periods), making use of HDF5's compression and incremental read/write.

Following this, I obtained an HDF5 file containing the maximum temperature recorded at each UK weather station on each day between 1853 and 2021. If you'd like this file (50MB) to do your own investigations, send me an email.

In my calculations I only used data from between 1920 and 2020, out of an unvalidated concern that data reliability and availability would be lower before 1920. Besides, 101 years of data should be enough!

Working with Julia for this was smooth. A common criticism I see of Julia is that the package ecosystem is undeveloped relative to more popular languages like Python or R. While perhaps true in general⁵, I think the issue is often exaggerated, and for the purposes of this project, the Julia packages worked well. The only real issue I had was in trying to set fillvalues for using the HDF5 package, which is something I didn't struggle with in the python package.