Klik her for at læse artiklen på dansk
With each passing day, they move closer: the impact standards that are designed to make it possible to measure and compare the impact of everything we do to benefit the climate, the environment, the vulnerable and the sick.
With fully developed impact standards in hand, social organizations can know if they are making a bigger social difference than the neighbouring organization. Foundations can assess whether they get the most out of a grant to Save the Children or Children’s Conditions. And investors can read which impact startup will give them the most value for their money – both socially and financially.
To some this sounds like Paradise on earth and the very prerequisite for us to solve the problems facing people, animals and plants. Others warn that there is not just one, but several snakes lurking in the thicket.
With the help of Pietro Micheli, we take a look at the potential and challenges of the standards. And we try to give you a sense of how to deal with them. And let it be said right away: Balance is a central concept.
There must be a common playing field
Let’s begin with a declaration of our guide through this landscape: Pietro Micheli is a professor at Warwick Business School in Coventry. He has worked with performance measuring for 20 years. And he is one of the three authors of the book “Measurement Madness”.
Organizations cannot simply measure on their own. There needs to be a common playing field.
Pietro Micheli, Professor, Warwick Business School
The title of the book might lead some of you to believe that Pietro Micheli belongs to the fiercest critics of impact standards. And truth be told, he has seen his fair share of craziness in the measurement industry.
Still, it basically makes good sense to have common standards for impact measurement, he believes.
“Organizations cannot simply measure on their own. There needs to be a common playing field that ensures that you measure and report in a uniform way. It creates pressure for increased impact, and it keeps the actors responsible,” says Pietro Micheli.
The challenge is to get the standards designed correctly, he points out.
“The standards must be sufficiently general to cover a wide range of organizations. If they become too specific, they are not standards anymore. Conversely, they must not be so general that they become irrelevant for the organizations. Because then they will start fiddling with them. It’s a difficult balance to strike,” says Pietro Micheli.
We will come back to the professor a little later. Because first we need to add a bit of context.
DIY measurements benefit no one
There is a whole range of impact standards on the way. And they are carried forward by the capital-rich segment of the impact universe.
For example, the Global Impact Investing Network (GIIN) has developed the Iris+ standard. Correspondingly, a band of investors has developed Impact Management Norms.
And if that wasn’t enough to deal with for social entrepreneurs, there’s also the Impact Evaluation Standard – not to mention the comprehensive register of social impact data that’s under way, The Impact Genome.
Why all this gains ground, Jason Saul, the creator of The Impact Genome, has expressed quite precisely.
“Do-it-yourself measurements are neither beneficial for cash-strapped non-profit organizations nor for those donors who lack meaningful knowledge and therefore cannot determine the impact their donations make.”
In other words, there is a need to be able to compare the impact of different efforts.
Impact standards are a huge step
That logic has its prominent proponents. Management professor Michael Porter states in an article about measurement of health care that rapid improvements in any area require measurement.
“Establishing measures of value (outcomes and costs) is the single most important step to improving the health system,” he writes.
It will revolutionize the market’s ability to deliver solutions to our major societal challenges.
Sir Ronald Cohen, businessman and investor
The British businessman and advocate for impact investing, Sir Ronald Cohen, is another proponent for impact standards.
“Today we calculate one thing in tonnes, another in liters and diversity in percentages. But when you start having impact standards, you can incorporate the numbers into your calculations of a company’s value, put impact in relation to the company’s growth and profitability and compare which companies deliver the most impact for the money. It’s a huge step,” he told Impact Insider earlier this year.
And in a recent comment on a topic on ESG in The Economist, he writes that the way out of greenwashing is to make impact standards an integrated part of ordinary investment practice.
“It will revolutionize the market’s ability to deliver solutions to our major societal challenges alongside increasing prosperity,” writes Sir Ronald Cohen.
Simplification leads to distortion
So much for the advocates, because as mentioned, there are also opponents of impact standards.
One of them is Professor Jerry Z. Muller, author of the book “The Tyranny of Metrics”.
In the book, he states that standards are seductive because they organize and simplify knowledge. And that makes it easy to compare people and institutions.
“But that simplification can lead to distortion, since making things comparable often means that they are taken out of their context, history and meaning,” he writes and continues:
“The result is that the information appears more secure and authoritative than is actually the case. Reservations, ambiguity and uncertainty have been peeled away.”
The difficult thing is to know what to actually do and what to do with the data you get, states Jerry Z. Muller.
“To the extent that we try to develop performance measures for an environment or organization that is complex, unique or different from other environments and organizations, standardized performance measures will be inaccurate and misleading,” writes the professor.
Beware of outside pressure
And now we come back to Pietro Micheli. He believes, as previously mentioned, that impact standards are useful in holding social organizations accountable for delivering the necessary results.
“Many social organizations report on inputs and activites and how many people they have reached. But based on my knowledge of the sector, they report very little on what difference they actually make to their target group. Because it is difficult,” says Pietro Micheli.
“And precisely because it is difficult, standards can push organizations to deliver on real outcomes instead of focusing on activities and processes. There has to be some pressure from outside,” says Pietro Micheli.
The pressure comes from donors, investors and public authorities, who increasingly expect the organizations to be able to account for the difference they make for the funds they receive.
When you are put under pressure […] you will try to circumvent the system.
Pietro Micheli, Professor, Warwick Business School
And this is where it becomes difficult.
Because as soon as punishment and reward enter the picture, the urge to change the rules of the game arises.
“When you are put under pressure because someone is evaluating whether you are doing what you are expected to do, you will try to circumvent the system. That mechanism is inevitable,” says Pietro Micheli.
Creaming, gaming and cheating
And once the velvet gloves are off, ingenuity flourishes. Three classic game spoilers are creaming, gaming and cheating.
Creaming occurs in connection with interventions where, for example, an organization is paid to help vulnerable unemployed people into work. Paying for success creates an incentive to skim the cream by offering the effort to the most resourceful of the vulnerable population.
This mechanism is well known and found in many fields. In “The Tyranny of Metrics”, Jerry Muller tells, for example, about surgeons whose pay level depended on the success rate of their operations. Consequently, they would only operate on those patients with the greatest chance of survival. The rest were on their own.
Gaming is another classic, where you formally live up to the rules of the game, but not to the spirit of the rules.
And then there is cheating. Regular cheating, where one does not abstain from fabricating data in one’s own favor.
“We see quite a few cases of that,” says Pietro Micheli.
He tells how baggage porters at British airports handled the demand to get suitcases out quickly to waiting passengers. The first suitcase had to be delivered on the conveyor belt after a certain number of minutes. And it was. But then a quarter of an hour could pass before the rest of the suitcases arrived. This is gaming.
In certain places, the baggage handlers had extra suitcases standing, which were thrown onto the belt as the first ones. And voila: all formal requirements met. This is cheating.
Too many measuring points make the picture blurry
Fiddling is hard to avoid completely. But the more relevant the metrics are, and the more the volunteers, employees and organizations to be measured are involved in the design, the lower the risk, says Pietro Micheli.
Physical presence also sharpens the value of data.
“There is more gaming in cases where you rely on self-reporting. If you go into the field and see what the organizations are doing, the amount of gaming becomes less,” says Pietro Micheli.
You also have to keep in mind that extensive measuring tools do not necessarily provide better data. Because if the requirements are too extensive, the organizations tend to become selective and report on what shows them in a better light, says Pietro Micheli.
“You think that the more metrics you use, the more you hold the organizations accountable and make them transparent. But it can backfire, because too many measuring points can make the picture blurry instead of transparent,” he says.
This brings us back to balance. Although standards must have an overarching character, it is necessary to adapt them to the reality in which they are to be used.
“You have to tailor the measurements so that they are relevant in relation to the organization whose work you are measuring. Otherwise, it will be perceived as an external inspection. And this rarely provides meaningful data,” says Pietro Micheli.
We have to learn together
Measuring meaningfully is a difficult discipline. And it does not come naturally to everyone, notes Pietro Micheli.
“At the beginning of my career, I was often surprised by how bad organizations are at measuring their performance. But it is a difficult exercise, and many do not have the skills for it,” he says and continues:
“People confuse goals and means and don’t quite know what data they want and what to do with it. Some of this stuff is just pretty obscure.”
At the same time, there is a need to further develop the measurement methods and our understanding of what we are measuring, points out Pietro Micheli. He reminds us that a thermometer – which we take for granted today – was around 300 years in the making, because it took a long time to understand what temperature actually was.
That’s why Pietro Micheli calls for patience – and for us to develop together the standards (and their use) that will give us knowledge about the impact we create with our efforts.
“When there is a forum where you can see what others have done and perhaps ask questions about your own practice, there is real room for improvement. But expecting people to just understand this is naive. Measuring meaningfully is more sophisticated than we usually think,” says Pietro Micheli.
