How Might We Use the Right Tools and Methods for the Task? How Might We Use the Right Tools and Methods for the Task?
How Might We Use the Right Tools and Methods for the Task?
The Daily GOOD
Get our daily dose of information and inspiration. Sign up Now ›
We recently met an organization who complained that it seemed as though evaluation methods are often simply an exercise in "checking the box" to appease their board and funders. For this organization, and many others like them, evaluation can be a time-consuming, costly process that does not lead to better decision-making or better work processes.
This is also at the heart of why so many large and expensive evaluation reports sit on the bookshelf-because before the evaluation was undertaken, no one answered the question:
"What are we trying to provide with our evaluation?"
Answering this question first can lead you to choosing the right methods to use in the evaluation, as well as ensure that the information will be used productively for decision-making and outcomes.
The answer to this question broadly falls into the following two categories:
- Intuitive comfort that we understand the complexity of the problem and the way forward to the right solutions
- Analytical confidence that our solutions are having the intended effects and we can move forward to scale with assurance
Because different sets of tools are better designed to meet either one goal or the other, we first have to address if our organization or funders currently need intuitive comfort or analytic confidence to move forward with a project. At IDEO, we've found that both are essential for decision making. ??Some funders that focus on early idea development need smart empirical approaches to feel comfort in pursuing new or risky investments. In these cases, the appropriate tools and methods skew towards the qualitative. Some methods to employ in order to cultivate an understanding of complex systems include ethnographic research, storytelling, and sounding boards. These methods are typically more open-ended, focused on discovery, and inspire the human capacity to synthesize complex data in order to make decisions.
On the other hand, funders that focus on the later phases of optimizing and scaling solutions need confidence in making investment decisions in ideas that have been proven to work. The appropriate tools and methods need to provide analytical robustness and often must be grounded in quantitative data. Some methods to employ in these cases include controlled studies, direct outcome measures, and regression analysis to determine statistically significant causal effects.
There are also some methods that we have found allow the mixture of both richness of understanding and analytical robustness. These hybrid approaches can generate insights that deliver on both the comfort and the confidence level:
- Choice exercises-at IDEO we have used statistical methods that can be applied to learn about how people trade-off value amongst different options. This can be done with very small samples and as part of an interview. The starting point is a comprehensive list of the value drivers underlying a solution. As a result, we understand what drivers or options we need to prioritize in our designs.
- Embedded stories-whenever we build a database, we have to be able to connect to real world stories to validate what we do. Being able to switch back and forth between data and stories is invaluable to not just understand the What but also the Why of observed change.
- System view with data support-linked to the system view approach, we can start building a holistic picture of the world we are operating in. Using stories, we can understand the multivariate system and its causal relationships. After we have achieved this we can extract, abstract, and implement experiments or models to help us measure effectiveness and optimize our offer.
Questions for discussion:
- What methods do you employ to create informed intuition and analytical confidence in decision making?
- Does your organization need complex understanding to move forward or rigorous measures to make decisions?
Tatyana Mamut is an economic anthropologist and content guide for IDEO's Design for Human Systems work. Johannes Seemann is a business designer at IDEO focusing on bridging qualitative and quantitative methods in insight generation. Read their last entry in this series here.
Turning Rubbernecking in Bangladesh into a Lifesaving Moment Without 9-1-1 or a reliable ambulance system, one med student and 100 volunteers launch a mobile-based emergency response system
Female Monks Challenge Buddhism’s Misogynistic Tendencies Long relegated to being the handmaidens of the more revered male monks, devout Thai women are now establishing their own religious order
If You Really Love Nature, Don’t Live Anywhere Near It Almost universally, people living in urban locations have a much smaller environmental footprint.
For Ernesto Yerena, Los Angeles is the City of Hustle and Hope Artist Ernesto Yerena’s visual love letter to the City of Hustle and Hope.
Books Stop Bullets at Tragic FSU Shooting A tragic shooting, a confusing profile of a would-be-killer, and a student saved by his library books
These Grandmas Smoke Pot For The First Time. And They Absolutely Love it. They take a few epic bong rips before waxing poetic on the merits of ironing, mistake a vaporizer for a sex toy, and stonily lose track of whatever thoughts they were briefly attempting to articulate.
If You See One Iranian Vampire Western Movie This Year, Make it This One The chador-wearing, skateboarding, vampire protagonist of A Girl Walks Home Alone at Night would fit right in to a John Hughes movie
There’s No Reason for Any Nation Not to Vaccinate its Feral Dogs Targeting the semi-wild dogs that roam city streets and rural hamlets all over the world can break the chain of rabies transmission and eliminate cases in humans
The Secret Origin of Neil deGrasse Tyson It took perseverance, intense training, and a willingness to defy expectations to turn a curious kid into the sharp, affable scientist we know today.
VITAMINS 101: Know What You Need Get the dish on your nutrition
Games Theory: 6 Views of a Mockingjay Just how socially relevant is The Hunger Games? Let us count the ways.
Here’s to You A toast to local hotspots around the world The best of the world’s neighborhood nooks.