Purpose of impact measurement: the bigger picture

Before you know it, impact measurement plays a role in your project. Every time that you reflect on your project’s progress and its effects, you make intuitive assessments based on available knowledge. Having an impact measurement system transparently structures your and others’ reflections and observations so you can communicate reliably with stakeholders.

Measuring impact helps a project and its stakeholders to understand the progress of their efforts and their impact. This can help to:

1. improve the impact of struggling projects
2. encourage others to replicate successful projects,
3. inspire financers to continue funding where they can see past progress.

This chapter introduces you to what impact measurement entails. We show you how to develop your own impact measurement system in five steps and provide practical tips along the way.

You don’t need prior experience to read this chapter. However, one vital tip is to hire a specialist to ensure your impact measurement is credible and robust. This helps with both communication and investment and funding. In other words, it’s a role that more than pays for itself.

When does impact measurement become important?

Integrating impact measurement from the beginning of your project makes it more efficient and useful. This means you have better data on hand when impact measurement is formalised at the last step of the 4 Returns five process elements.

Good examples of early integration are:

• Baseline research that informs the priorities as you draft your landscape plan. A baseline includes using existing evidence and literature on effective interventions, saving you a lot of time in doing research yourself.
• Make an impact measurement routine as you prototype new projects and programmes. This allows you to draw lessons from your past impact, which allows you to continuously improve your work.
• Think about impact measurement in your budget and planning. Allocate enough money to hire skilled people and create space in your planning and agreements with financers for adaptive management.

Impact measurement step-by-step

While we recommend hiring a specialist for impact measurement, most people involved in a 4 Returns project will be involved in it in some way. These steps will help you understand what happens in impact measurement, why each step makes sense and how they fit together.

Step 1: Determine your goals for impact measurement

Think about why you’re measuring your impact. Find out what goals are most important for you, because that determines partly the design of your impact measurement system.

Three goals are often prominent:

• Accountability your most important stakeholders, such as financers, staff, and communities in your landscape that want to see evidence that what you do works.
• Communication: telling your story to a broader audience to spread your work and involve new stakeholders. Your impact measurement system and storytelling efforts (see chapter ‘Storytelling for the 4 Returns’) must be well integrated.
• Learning about the effectiveness of your work. By researching the effects and inner workings of your project, you can learn what works and adjust your strategy accordingly.

Once you know your priorities for impact measurement, you are less likely to collect data you won’t use, and this will save you money and time. Moreover, prioritising helps you to determine what you will not do, because these three goals can be conflicting. For example, data for communication is very different from data used for learning, and you can only spend your time and money once.

When Wij.land started in 2017, they immediately implemented a straightforward measurement system for the purpose of accountability towards funders. In this way, they transparently showed what the organisation spent time and money on. Since then, all activities, adoption of practices and relationships that are built are kept in their data management system. Now that Wij.land is growing, they are thinking about how to make use of the data for the purpose of organisational learning.

Step 2: Dedicate budget and time

Research is expensive. Incorporating it into your budget and hiring ensures your ambitions can be reached. Your budget is the limiting condition for the feasibility of the following steps.

Budget

As a rule of thumb, 5-10% of your total project budget will go into impact measurement. If you are innovating and need data for rapid improvement, this may be higher. The budget includes people, data management systems and technical tools.

Tip: Hire the right people

An important choice that can break or make your impact measurement is to hire the right people. Some necessary skills are:

• a solid research background or typical impact measurement
• research skills include quantitative and qualitative analysis skills, such as setting up surveys and interview formats and being able to implement and analyse these.
• working with basic computer tools (such as word, excel, or GIS)
  effective communication, e.g. speaking the local language, being seen as neutral by the target population, and as someone they trust.
• Experience with the themes you are working on – e.g. working with farmers or in land rehabilitation.

Timeframe

Getting concrete results takes time. Setting up a well-tested measurement system generally takes one year. The five steps in this chapter will take some months. Then, collecting, analysing and evaluating the first batch of data – including baseline data – will take at least half a year. Meaningful trend analysis requires at least three years of data. So, give yourself some time before expecting concrete results.

Fortunately, just gathering first insights does not have to take that long. Moreover, not all organisations measure the impact of their activities from the start. Undertaking retrospective research to try to reconstruct what has happened, takes less time.

Tip: count on revising your measurement system every 4-5 years. Most organisations change their strategies and activities once every 4-5 years, focusing on new priorities, target groups and partners. Changing your strategy also means adapting your measurement system. This does not mean that after five years all gathered information is useless, but it does mean that some trend analyses may not be continued.

Step 3: Identify learning questions

Learning questions formulate the impact you want to measure, so you can work out the right methods for data collection. Examples of learning questions are:

• do the changes we expect from our activities really happen? For example, with the 4 Returns: do we create inspiration? Or do we create social return? Etcetera.
• why, or why not do we create this change? What are success factors or what hinders our impact?
• what happens when a return, such as the returns of inspiration, goes down?
• can we save money by doing things differently?

Tip: Prioritisation is key

Once you start asking people their learning questions, they often have a lot of suggestions. Our advice is to start small because you easily create a lot of unnecessary work. You can always do more once some of your learning questions have been answered.

Tip: Involve stakeholders

Involve your most important stakeholders to identify their learning questions. For example, if the goal you determined in step 1 is accountability towards financers, ask your (potential) financers about their most important learning needs.

The first step in identifying learning questions is scoping: setting boundaries on the interventions you want to know the impact of. This is especially important if you work in an organisation with many different projects: prioritise what projects are the focus of the impact measurement system.

There are five categories of learning questions, according to the OECD-DAC. Knowing these may help you prioritise:

• Is our approach relevant: do people need this change?
• Is our approach effective: does it work the way we thought it would and what assumptions should we test?
• Is our approach impactful: does it create the changes we want? In this you can also look a bit further: is your approach creating negative or unintended effects?
• Is our approach sustainable: are the changes it creates long-lasting?
• Is our approach efficient: is it using your resources in the best way possible?

Using a Theory of Change for prioritising learning questions

For prioritisation, it is common practice to use a Theory of Change. A Theory of Change shows how you think that all your activities lead to change and impact, visualised in impact pathways. Generally, impact pathways consist of 4-8 steps showing the logic of how your activities create change. The total of all impact pathways and steps equals the total of learning questions you can formulate around effectiveness (question category 2) and impact (question category 3).

Here’s an example of the Theory of Change of one of our partners. Wij.land created a Theory of Change for their 2022-2026 strategy.

The green area shows their 4 main activities and more concrete activities in the white blocks. The blue part shows all lower- and higher-order changes Wij.land wants to create with these activities, and how Wij.land thinks the impact pathways (blue lines) work. For example, the impact pathway on the left:

• by developing and giving access to financial instruments, Wij.land hopes to create adoption of instruments among farmers,
• through which farmers get access to better financial services from the financial sector, for example, payment for ecosystem services schemes,
• which allows farmers to contribute to improvement of ecological functions on their land.

Every step in the Theory of Change represents a learning question: does one change lead to another? For example: in the case of Wij.land you can research whether indeed access to financial instruments lead to adoption and what is necessary for this to happen. As there are many links in the Theory of Change of Wij.land, they will need to prioritise what learning questions are most important for them to gain insight in.

Therefore, besides serving many other purposes (see for example chapter 7 strategising”, where the Theory of Change is used as a tool for strategy design). A Theory of Change is a good way to show learning questions and helps to understand how much work it is to research everything. Often it is only feasible to prioritise two or three impact pathways. Here is a template for developing your own Theory of Change, and here is an elaborate guide on how to develop a Theory of Change.

Tip: Do not only prioritise ‘ultimate’ changes, for example, ‘more biodiversity’ or ‘better policies’ as you cannot link them to your own activities. You need to measure your activities (output) and lower order changes (e.g. ‘more awareness among policy makers’) that lead to higher orders ones (e.g. ‘a change in farming practices’).

Alternative for the Theory of Change: explorative research

In less predictable contexts, we use explorative research instead of a Theory of Change to determine learning questions. Explorative research means we suspend expectations about what changes we will find. For example, working in war zones, areas of natural disaster or social upheaval, we approach the field with an open mind and harvest the changes we see. We see where these changes are connected to our own activities. A well-tested and useful methodology in explorative research is outcome harvesting.

Step 4: Design measurement methods

Once you know the learning questions and what answers you need from impact measurement, you can translate this to designing the measurement system. At this point you can easily get lost: there are many different research methodologies that you can use, and many choices to be made. For this reason, we advise you to hire someone with a solid and diverse research background, who can guide you through this technical process. For example, in the case of Wij.land one person in a team of seventeen is responsible for impact measurement, managing the research agenda, instructing colleagues and manage organisational learning.

We include this technical (and maybe difficult) chapter in the hope it gives you more clarity on the choices that need to be made, the products and tools that need to be designed, and therefore, what capacity you need. Based on this chapter you can ask yourself these kinds of questions:

• Are we able to carry the design and implementation of an impact measurement system ourselves, in our current capacity, or do we need to hire someone?
• What does impact measurement require from our stakeholders and partners, and are they able and willing to contribute to this?
• What steps do we need to start with, and how fast or slow do we want to develop our impact measurement system?

Start with exploring existing literature and databases

Existing literature and databases are a great first step in answering your learning questions. Start by typing certain key words into scholar.google.com. Even without a license, the abstracts often provide the important information you need. Or you can investigate credible open-source publications. The WOCAT database shows the effect of many different sustainable land management practices. A good database on protected area management is the World Database of Protected Areas, and an overview of threatened species you can find on the IUCN Red List. Another useful app is consensus.app, an AI-based tool to get the most important conclusions from articles.

Tip 6: Often literature is more suitable for answering questions related to higher impacts. For example: suppose that a learning question is whether training farmers in regenerative agriculture helps them adopt new practices, leading to better soil quality, water regulation and biodiversity. Existing literature will not tell you whether your training affects the farmer. But it will tell you more about the links with higher impacts, i.e. of practices of regenerative agriculture on soil quality or biodiversity. Moreover, it gives you knowledge on best practices regarding training of farmers, and what are other important factors enabling or disabling farmers in their transition. This also informs the design of your approach.

Register your activities and outputs

Next, register your activities and outputs. For example, the number of trees planted, number of events organised, number of participants in these events, and number of hectares reached. Your outputs form the basis of your impact measurement system. You then relate these outputs to your ‘higher’ effects in your Theory of Change. Registering your outputs is already a lot of work. Starting early allows the organisation to get used to collecting data and gives the organisation a better estimate of the feasibility of measuring higher order effects. It also allows you to communicate about your work on a short term.

Design indicators

For each learning question identified in step 3, develop one or more indicators that together provide an indication of the answer to the learning questions.

In our farmer training example, an indication of an effective training to farmers is to test their knowledge at the beginning and at the end, to see a difference in the score. You can add a satisfaction survey in which farmers state whether they learned something new, and how they evaluate the training. The indicators in these cases would be: 1) the difference in scores on the test, and 2) the evaluation of participants of the novelty of the material in the training.

In 2018 Commonland together with its partners designed the first set of indicators to measure the 4 Returns. These can be used for both inspiration and guidance in developing your own indicators related to your own Theory of Change.

Choose measurement methods

You need to choose a measurement method and develop qualitative or quantitative measurement tools for each indicator. Quantitative tools result in numbers and visuals, answering the ‘what’ question. Qualitative methods result in stories, text and quotes, giving meaning to numbers and telling more about the ‘why’ and ‘how’ question. Our advice is use a ‘mixed method’ approach that uses both quantitative and qualitative methods. Any method you use requires you to be well trained in it, or to have an experienced supervisor.

Some frequently used methods are:

Betterevaluation.org provides examples of many more research methods. If you get lost we suggest returning to the ones above.

Other choices

A few choices remain to finalise the measurement system.

1. Frequency: how often are you going to measure each indicator? The most common frequencies are yearly smaller assessments (‘monitoring’), or larger assessments every 2-3 years (‘evaluations’).
2. Research group: where and from whom do you gather the data. Often this is the target population of your intervention. However, a control-group comparison will research people outside your project sphere. Moreover, a research group needs to able to reflect on changes, which is not always the case with the target population. Then you have to find alternative people that can reflect on the changes that they see.
3. Researcher: who performs the research. This should be someone with the necessary skills, and the right position towards the research group, unbiased and with no strained relationship.

In a master excel file, show one indicator per row. The columns show the learning questions they relate to and your choices for the factors above.

Review

Now go back through your indicators and review 1) how well they relate to the goals of your impact measurement system (step 1), and 2) whether their measurement is feasible for your time and budget. Adjust your measurement system until you can answer positively to both questions.

For researching higher-order effects in their Theory of Change, Wij.land cooperates with knowledge institutes and universities including several studies on how different practices impact the soil quality. Wij.land is now developing new measurement methods to research middle-order effects in the Theory of Change, e.g. the usefulness of new tools to farmers. This research requires more capacity from the team, both in time and skills.

Tip: Data triangulation is the gathering of the same indicator among different research groups and different methods. Try to apply it as much as possible within your available means. In our working example, we previously mentioned combining a satisfaction survey among farmers with a knowledge test that gives a score. This is an example of data triangulation – both are tests of the learning question – are farmers learning from the training?

Step 5: Develop tools, install routines and data management structures

The last step is to develop the apps and tools necessary for:

• Data collection: the surveys, interview formats etc.
• Data warehousing: apps and tools used for data collection and storage, such as survey monkey, google forms, etc.
• Analysis: apps and tools like statistical analysis apps such as STATA or SPSS, and apps for visualisation, like Excel or PowerBI).

In your data analysis, make sure that you log all changes that you make to the data, so the analysis can be replicated.

Tip: Data management: your research angel.

If designed well, a good data management system saves a lot of time and frustration and is worth an upfront investment. Many research errors stem from data that is not stored and analysed well – and these are costly to correct. Be aware that personal information such as names and contact details falls under privacy legislation. They must be safely stored and deleted after a certain amount of time. One way around this is to anonymise data as much as possible.

Develop routines for collecting, analysing and interpretating impact data

Besides developing all apps and tools, you also need to create internal routines for collecting, analysing and interpreting the data and ensure that team members are well instructed. Part of this is already determined in step 4, when you designed the frequency of the measurement of the indicators. However, how often data is collected and analysed does not automatically translate in how and how often the findings are discussed, used and reflected upon. We advise you to have clear routines, so that everyone in the partnership knows when to deliver what data, when this is analysed and discussed and when this is communicated, depending on the goals that it is used for. Many organisations have yearly cycles, often culminating up to the time the annual report is released, or towards strategic reorientations.

In the case of Wij.land, procedures and routines they worked out really well. A few examples of the tools Wij.land developed, are:
– A flowchart depicting when and how everyone in the team should collect data and via what ways they should put this in their data management system. The team regularly discusses and updates this flowchart, so that everybody is using it in the most effective way.
– A manual for the team members explaining why impact measurement is important and what is expected from each person.
– The data management system is made as simple as possible and has an accessible interface, allowing everybody to use it in their own way.
– Every year, the team updates their knowledge and learning agenda, prioritising the most important learning questions, and determining how these should be answered.

What are your organisation’s key practices and systems for measuring impact? Did you gain any new insights to apply from this chapter? Let us know in the comments!

This chapter was written by Joanne Pieterse.