Propensity Matching for Robust Service Delivery

Imagine you've commissioned a significant policy change, a new service or a targeted intervention. You have enough data to measure outcomes but you're not necessarily confident that the causal link between the outcome and the programme of work is strong. Wouldn't it be great to have more confidence that the programme of work is not only effective, but serving those it was created for?

Introducing propensity matching!

What is propensity matching?

Propensity matching is a statistical technique that can give more confidence to service providers, policy makers, or anyone responsible for ensuring that evaluations of large programmes of work are robust. It is a useful technique for drawing causal links between an intervention and an outcome, minimising the possibility that the true association is being obscured by other characteristics. Propensity matching can be used in combination with randomised control trials, or in situations where a randomised control trial has not been done / is not possible.

How does propensity matching work?

Rather than walking through the specific technicalities of propensity matching (you can leave that up to us!), we think the easiest way to understand it is through a hypothetical example.

Let's say that you are commissioning an intervention to support students to achieve NCEA level 3. The intervention provides students with access to a tutor and aims to address learning barriers by presenting information in a more accessible way. In this case, it would be important to compare different groups of students to see if the intervention is working as hoped. Once exam results are in, you end up with one dataset that contains the information of students who accessed the intervention and another that contains the information of people who did not. However it is not as simple as comparing the two datasets given that other factors can affect NCEA results, such as the amount of labour (paid or unpaid) students do outside of school and whether any received 'merit' or 'excellence' grades at NCEA levels 1 and 2. These factors are called 'confounders'.

If the two groups have different confounders it is difficult to draw conclusions about the causal effect of the intervention. In the example used, it would difficult to measure whether giving students access to a tutur or not is having a direct effect on NCEA level 3 results unless important confounders are controlled for.

Propensity matching does just as what the name suggests: It matches the intervention and control datasets on those potentially confounding characteristics to get a pool of representative control cases that look just like the group that experienced the intervention. Following this, you would contrast the NCEA results of the two post-matched groups. This comparison is more likely to estimate the causal effect of the intervention than simply comparing the two datasets, because the two groups are now more similar in terms of other potentially associated characteristics. The results of this comparison give you and anyone else involved in the intervention process (eg, commissioners, designers, facilitators) more confidence that these efforts are really helping students to achieve NCEA level 3.

Another great thing about propensity matching is that it can be used for assessing the unmet need in a population. The representative control groups created in the matching process can be used to identify whether there are individuals who are not currently being serviced, but who have the characteristics of someone who should be. In this way, propensity matching can be a fantastic tool for driving equitable access and outcomes.

Who should think about propensity matching?

Anyone who funds, leads, or is responsible for delivering / ensuring the effectiveness of large programmes of work should consider propensity matching as an integral part of the project. If you find yourself asking questions that sound similar to:

• Does this (service, policy, programme, intervention) actually work?

• Does this (service, policy, programme, intervention) serve the people it was intended to or needs to?

• How can we measure the size of the impact that this (service, policy, programme, intervention) had?

• Is there anyone in Aotearoa who should have access to this (service, policy, programme, intervention) but currently does not?

Then propensity matching could be an especially useful and cost-effective technique for getting answers.

Anything else I should know?

Like all work with data, it is important that any information used for propensity matching is treated ethically and with the people behind the data in mind. Nicholson Consulting have a team of experienced, friendly, and passionate data scientists ready to provide support. Get in touch if you would like to chat more!