Estimating the Impact of Extension Activities

Extension teams are made up of awesome people trying to do amazing things, but how do we measure the impact of the work these teams are doing?

hero image

Written by

photo of author Vaughan Wilson
Vaughan Wilson

Research manager

027 543 2124 vaughan.wilson@scarlatti.co.nz

Superpower: Throwing a frisbee

Fixations: Excel

Vaughan leads our impact modelling work area. His work also spans workforce research, extension programme design, and oragnisational change. He’s frequently involved in projects involving both quantitative and qualitative analyses.

His experience involves working with clients across a range of primary industries including horticulture, dairy, and sheep and beef, where he enjoys developing tools and resources that are interactive and intuitive for a range of uses.

Vaughan holds a Master of Agribusiness (with Distinction) and a Bachelor of AgriCommerce in Agricultural Economics, both from Massey University. During his Masters, he worked with Zespri to conduct a situational analysis of the economic impact of sooty mould in New Zealand’s kiwifruit industry.

Share

The challenge

Extension teams are made up of awesome people trying to do amazing things, but it can be difficult to measure the impact of the work these teams are doing on the intended financial, environmental, and/or social outcomes.

This matters when extension teams are discussing the return on investment (ROI) with investors at the start or end of the project. The lack of good and accessible information to estimate the ROI could mean that projects which should have been funded were not, or projects that should not have been funded were.

There are three challenges associated with collecting the information useful for these discussions:

  1. Time - the delayed time taken for an extension programme to have an impact on outcomes,
  2. Attribution - the attribution of changes in outcomes to the extension activities (while considering other factors), and
  3. Cost - the cost associated with undertaking large scale data collection to measure practice changes.


Modelling can be seen as complement measurement in an evaluation of extension activities to address these challenges.

Logic_model_400x530

Scarlatti's QUICK response

The ‘QUICK’ – Quantifying and Understanding the Impact of Capability and Knowledge – modelling framework has been developed to quantitatively characterise the causal chain depicted in intervention logic models (see image above); training leads to changes in capability and motivation, which leads to practice changes, which leads to better outcomes.

The key logic for underpinning the QUICK framework include:

  1. Each extension activity reaches a certain proportion of the population,
  2. As individuals participate, they will improve their capability and motivation,
  3. Individual’s who have participated directly will share their learnings within their social networks, improving the capability and motivation of individual’s indirectly,
  4. As an individual’s capability and motivation increases, so does their ability to change their practices,
  5. As individual’s change their practices, they will achieve improved financial, environmental, or social outcomes.

Dealing with complexity

This is an immensely complex system; however, a lot of the complexity disappears as soon as the modelling scope is reduced from the whole system, down to an individual level.

The QUICK framework has been constructed using an agent-based approach where the above logic of participation is repeated for each individual in a virtual population where participation is randomly initiated depending on several factors. Effectively, this approach creates a virtual database of all the practices and the changes that are occurring for different segments of the virtual population at different points in time.

Handling uncertainty

As is the case with most forecasting activities, there is a certain level of uncertainty which must be addressed by the QUICK framework.

The approach to handling uncertainty in the QUICK framework suggests that the approach would be better described as semi-quantitative, where ‘semi’ refers to the preference to have inputs and outputs as ranges rather than a single number.

The framework is more interested in being robust in the approach and confident that the outcomes fall within a range rather reporting outcomes as a precise number.

Applications of the QUICK framework

The QUICK framework has been applied to several different contexts for Scarlatti’s clients in New Zealand’s primary industries (e.g., Beef + Lamb NZ, Extension E350, Ministry for Primary Industries) including:

  • evaluating a programme retrospectively to estimate the impacts the activities undertaken have and will have in the future
  • designing a programme to determine what combination and scale of extension activities could be the most effective at achieving the desired outcomes given the available funding
  • monitoring the estimated impact of a programme’s activities overtime as activities are implemented.


Despite the historic application in New Zealand’s primary industries, the QUICK framework is robust enough to be applied in any extension context which targets a change in behaviour.

Watch a presentation on the QUICK framework

Scarlatti’s Director, Adam Barker, presented the QUICK framework at the 2022 APEN (Australia-Pacific Extension Network) virtual conference.

Check out Adam's presentation, which covers the QUICK framework in more detail and discusses its application, to the Ministry for Primary Industries’ Extension Services programme.

Interested in estimating the impact of your extension activities?

Contact the impact modelling team

You may also like

Three Tips to Creating an Engaging Survey - that you might not have heard before

A great survey does more than collect valuable insights. But how do we keep people engaged?

ArrowRight

How Conjoint Analysis Helped Santa and his Elves

A conjoint analysis survey uses trade-offs as a way for people to indicate their preferences. Find out what value Santa's elves place on their wellbeing and the environment.

ArrowRight

How Effective is Your Wellbeing Programme, Really?

An effective wellbeing programme involves a lot more than encouraging people to move and eat their veggies.

ArrowRight