The biweekly ‘So What?’ guide highlights advice, events, and tips — mostly from the advocacy and evaluation worlds, selected by the Aspen Planning and Evaluation Program.
Measuring Success in the Marketplace of Ideas
With a hat tip to Sarah Alvarez of the Aspen Institute’s Latinos and Society program, check out this smart if slightly self-serving analysis by Arthur Brooks in Harvard Business Review of his efforts to measure impact at the American Enterprise Institute (AEI). Brooks, AEI’s president, notes two typical mistaken responses to questions about impact in the ideas business: claiming that think tanks are “sui generis,” so their impact can’t be measured and the “lamppost” error that leads us to measure what we can, regardless of whether it is meaningful. Brooks offers a pretty candid assessment of what AEI has learned by trial and error in choosing various “proxies” for actual policy impact.
Julia Coffman notes that the Niskanen Center, a libertarian think tank, published this “conspectus” elegantly summarizing its theory of policy influence and metrics for success. It offers a pithy and well-researched argument in favor of focusing on elite policy insiders:
Remember, public opinion is a creation of elite opinion, so targeting elites and influencing them has more impact than directing efforts towards non-elite audiences.
Brooks at AEI and Niskanen’s team arrive at similar metrics: publication of their ideas in designated “elite” outlets, invitations to testify to Congress or state legislatures, face-to-face meetings with key officials, requests from staff or elected officials to provide briefs or legislative drafts, etc. Nothing stunning here from the perspective of advocacy evaluation; APEP has used similar metrics in working with clients. But we appreciate these think-tank efforts to quantify their contribution to policy and social change based on clearly defined theories — especially when, like Niskanen, they wisely acknowledge that they cannot directly attribute policy change to their efforts. We salute their candor.
Here at the Aspen Institute, impact assessment is a challenge that we sometimes ignore until a pesky donor asks for evidence that we are making a difference. We duck the challenge for good reason: it’s real hard, dang it, especially when the product is dialogue, leadership development, and, um, a good society. But we’re on it.
Catching Up on Contribution Analysis (Instead of Sleep)
You overslept and missed APEP’s 8:15 am breakfast on February 23rd featuring Robin Kane and Carlisle Levine discussing their experience using contribution analysis in policy evaluation. Or you weren’t in Our Nation’s Capital. No worries! Here’s the link to our fancy webcast featuring some terrific questions from fellow evaluators, advocates, and funders. You can download the meeting handout here and the presentation slides here. And no, you cannot download actual pastries here.
