Vizzuality playbook in progress
This project is maintained by Vizzuality
Because we are in the early stages of impact measurement, this document is less of a helpful prescriptive guide and more a plan to make us better at measuring the impact of our work.
Our purpose is of social change, to influence the behaviour of people through design and technology. We’d like to know how we’re doing on that. Our clients come to us because we can demonstrate the impacts our projects have. We can point to how many hectares of forest people have helped protect because they used GFW. We can point to the CSOs and NGOS that used the Goal 16 and GDA Dashboards to hold their governments to account.
People also come to us because we put in the time and effort to research, experiment and innovate. Because we take impact monitoring so seriously.
At the moment we don’t really measure our impact. For an organisation that aims to make the world a better place, we don’t actually know if we are helping people live better lives or making a difference to the environment.
All we have at the moment are analytics: XX users went to GFW last month, YY% are using this feature that we think will help them make decisions. We get some feedback every now and then, but that’s it. And this is obviously really useful for our internal processes, because it helps us work out if the technology we build actually works. But that’s just in the browser.
We need a better way of identifying if the technology we build is generating the impacts we intend it to have. Then we will know if the features we build are actually useful, not just if they’re used/ usable.
So how do we measure that? This document outlines a couple of methodologies for doing this and the financial/temporal implications of doing this.
There’s a number of different tactics for evaluating the impact of a sustainable development project. The general measure of success is some kind of positive change in an indicator - more children survive to age 5, more girls get educated etc - against a baseline state, before the intervention. You might get randomised trials as well, with comparisons made to a baseline state and against a control group of some kind, but these are often costly and time-consuming to assess. The data can come from objective measures as well as end-of-project surveys.
Within the UN/ NGO space the idea of SMART (NOTE: Specific, Measurable, Achievable, Realistic, Time-bound - https://en.wikipedia.org/wiki/SMART_criteria ) objectives within a broad ‘theory of change’ are commonplace. An intervention is deemed effective if it reaches its intermediate goals, which in theory should lead to long-term impacts.
Have we increased knowledge?
Have people taken actions?
How effective were these actions?
Using a sustainable development lens - economy, environment and society
Problem - hard to attribute causality to our thing, especially as so many things change. SO need to look at individual level, marginal changes. “Could you live without it?”
There are four groups of activities to bring about this strategy.
Set goals in Discovery. During our research we will be talking about product definition, who our users are, and how we want to achieve impact. As an additional output of our research, we should write out some target metrics that measure the success of the site itself (time on site etc) and whether it achieves the impacts we expect. Example metrics could include:
% of time on site on key pages (map/ dashboard etc)
Number of times data is downloaded
Number of citations
Retirement strategies. Every site has an objective: to achieve some goal like ending deforestation, making governments more transparent, etc. As well as writing success metrics in discovery, we should discuss how we would know we’ve solved the problem, so the site would no longer be needed. We call this a retirement strategy.
Surveys. This is a relatively low cost, but potentially high reward activity. The easiest way to do this is to install a Survey Monkey or Hotjar widget into the site, prompting users to complete a survey. We could also ask out clients to send out longer surveys, where they have a long list of potential users.
Interviews/ tests. Doing as we do already throughout our projects, but trying to see how people have already achieved impact with the tool. This could also double up as training/ onboarding for key stakeholders.
Observations and ethnographies. In this activity we would go and visit our users where they are. We would watch people, learn from them, and see how we could improve their lives. We could also get them involved in the design process (if done at the beginning of a project) so we can co-create impactful solutions.
Retrospective after each project with client. Around a month after a project has been launched, we should hold a retrospective with the client to analyse the feedback they’ve received so far and how it’s going. This would take the form of a classic retrospective, arranging feedback received into “what went well” and “what can be improved”.
**Six month retrospective (big projects only). **We would also consider repeating this exercise six months after a project ends. This retrospective would take findings from analytics as well as comments received by the client, to assess who is using the tool, how, and if it’s been effective.
Extend the ‘embed phase’ with more tests and interviews. Where we think it’s possible, we should extend the definition of the embed phase of our project proposals to include many more tests and interviews in the months after a product is launched, so we can see how people are using it in real life.
Funding M&E as a separate contract. As part of the project follow-up tasks we could ask our clients if we could conduct monitoring and evaluation projects with them. During these M&E contracts we would aim to conduct some of the activities under ‘reaching out to users’, but with the specific objective of creating feature ideas that could be implemented under another implementation contract.
Annual Impact reports. The idea here is to summarise our work in the context of the Sustainable Development Goals - how much good are we achieving across the world? The report would be simple (no more than 8 pages?) and cover a mix of analytics findings, user testimonials and client feedback. We could share the report internally and externally.
Invite clients to talk about our projects. This activity is more internally focussed. As Toby Gardner did with Trase at the Christmas All Hands meeting last year, we would invite our client to make a quick presentation to the whole company about the project we’ve developed and the impact it’s having.
Lightning talks. We should aim to hold one lightning talk every quarter describing our projects’ impacts.
Have an annual OKR about impact. Another way to formalise this, and communicate our impact, is to have an OKR specifically mentioning the impacts we want to achieve in a year. We could select 5-6 high priority projects and what we want to achieve with each one.
Tech
Recursion - we use the tech we build to monitor the impact. GFW shows that things are going down.
Emphasis on the developing world - test our assumptions about democratisation etc