This meeting on 6 December 2022 was inspired by the recent publication of a book entitled simply
Noise. Interpreting ‘noise’ to mean ‘random variation’, I predicted that this book would be of interest to statisticians. The author, Daniel Kahneman, had previously written
Thinking fast and slow that focused on how the human brain functioned while making decisions.
In this second book he focuses on what he calls ‘noise audits’. These are designed to reveal the differences and the inconsistences that are found when two or more ‘judges’ assess the same situation. The judges that Kahneman has in mind include:
- Financial and logistics professionals who predict the cost and duration of projects;
- Loss adjustors who predict settlement figures for insurance claims;
- Criminal court judges who specify the length of prison sentences, etc.
These are judges who may have had little or no contact with statistics and would likely need the services of a statistician to analyse the results of a noise audit, especially as Kahneman offers little advice on analysis and, perhaps remarkably, makes no mention of the progress made by scientists, engineers and business managers in the reduction of measurement errors.
Three members of the RSS Quality Improvement Section designed a two-hour workshop that might appeal to statisticians as well as potential judges. It would need to be interactive and to contain some content that was free of statistical jargon, and be delivered remotely.
We built the workshop around a simulated noise audit that required all participants to assess, or judge, the lengths of several lines shown one by one on the screen. These numerical judgements were then passed to a data analyst who carried out a ‘measurement systems analysis’ to obtain the repeatability and reproducibility of the judgement process.
The meeting went very smoothly and was followed by a very lively discussion amongst the participants. Their familiarity with the assessment of measurement error was clearly extended in many cases, with several agreeing that they now felt ready to assist a noise audit. One attendee confided that she was glad that she had become aware of this workshop before her company embraced noise audits.
This workshop is scheduled to be repeated at the RSS Conference on Thursday 7 September 2023.
Written by Roland Caulcutt