Using the IDI to test big ideas and make better decisions

The right tools for big questions

Some questions are just too big – or too expensive – for traditional research methods like surveys and interviews. Especially when you're trying to understand complex systems across an entire sector or population. That’s where the Stats NZ Integrated Data Infrastructure (IDI) comes in. The IDI lets approved researchers securely link data about people’s employment, education, visa status, and more, giving a rich, population-level view that’s hard to match. In 2024, the Food and Fibre Centre of Vocational Excellence (Food and Fibre CoVE) came to Scarlatti with a tricky question:

Could coordinating seasonal work and offering training opportunities help ease long-standing workforce shortages in the food and fibre sector?

Workforce shortages affect many sectors. Efforts to attract and retain workers often have limited or short-lived impact, so other avenues are being explored such as this question.

Answering a question like this would normally require a substantial budget if conventional research methods were used. But by turning to the IDI, we were able to test this idea in a robust, quick, and cost-effective way.

Not sure what the IDI is? Check out our explainer article The secret weapon to understanding the people of Aotearoa.

Turning data into answers

Access to the IDI is just the beginning. Knowing how to use it – how to frame the right questions, navigate the datasets, link the right variables, and interpret the outputs – takes expertise.

That’s where our team comes in. With extensive experience in working within the IDI, we know how to uncover answers to complex questions. In this case, we helped Food and Fibre CoVE explore whether seasonal workers could be supported into longer-term employment.

We started by breaking down the big idea into specific, data-driven questions:

Who are seasonal workers?
Where and when do they work?
How many transition into permanent work?
What barriers or enablers of these transitions can be seen in the data?

Using the IDI, we cross-referenced employment data with information like visa type, region of residence, and education history. This let us follow workforce pathways, explore seasonal, retention, and return patterns.

From there, we used a mix of statistical analysis and machine learning to test the feasibility of attracting seasonal workers into long-term roles, including:

Estimating the likelihood of a seasonal worker entering permanent work, based on attributes like age, region, and visa type.
Analysing employer proximity to population centres and potential impacts on recruitment.
Measuring return rates of seasonal workers over time.

It’s not just about pulling numbers – it’s about asking the right questions, knowing where to look, and understanding what the data can (and can’t) tell you.

What we found

Once we had a clear picture of what to look for, we extracted the data, analysed the results, and developed an interactive dashboard to share the findings with Food and Fibre CoVE – and other interested stakeholders.

Two insights stood out:

Seasonal work doesn’t line up across sectors. Most food and fibre industries are busy at the same time – typically summer – making it hard to offer year-round employment by linking seasonal roles across sectors

Many seasonal workers already move into permanent work, but the shortages persist. Between 15-20% make that shift each year. But even with this transition, food and fibre industries continue to face workforce shortages.

In short, better coordination of seasonal labour alone isn’t likely to fix the problem. But this finding was still valuable as it helped focus attention on where a more realistic opportunity lies – retaining people beyond their first year, when dropout rates are highest.

Clarity to move forward

Our research didn’t confirm the original hypothesis – but it gave Food and Fibre CoVE something just as valuable: clarity.

By exploring ideas with real-world data, they were able to rule out a solution that looked promising on paper but didn’t stack up in practice. And in doing so, they identified where effort is more likely to pay off – supporting and retaining workers who’ve already entered the sector.

This project proves how complex, sector-wide questions can be tackled without relying on assumptions or guesswork. With the right expertise, the IDI can test bold ideas, highlight what’s working, and help decision-makers focus their resources where it matters most.

“Food and Fibre CoVE has worked with Scarlatti's IDI team across a range of projects to deeply understand data and insights that can enable vocational excellence, drive innovation and focus on attracting and retaining a talented workforce […] Scarlatti's IDI team have delivered an excellent service to ensure the research is underpinned by robust methodology, and can continually enhance how vocational education and training can be delivered for the benefit for learners, providers and industry.”

― Fiona Windle, Portfolio Manager, Food and Fibre Centre of Vocational Excellence

Learn more about this project

Food and Fibre CoVE's official report
Interactive dashboard

Talk to us about your big question

Scarlatti has a team of approved researchers who can help you assess whether the IDI is right for your needs. Whether you're exploring workforce dynamics or tracking the impact of social programmes, we can help you unlock insights and achieve better outcomes, faster.

If you’ve ever thought “there’s no way I could get the data to answer this”, you might be surprised what we can achieve for you.

IDI articles series

The secret weapon to understanding the people of Aotearoa
Stay tuned for our upcoming articles