The Challenges of Measuring the Impact of Narrative Practices

The assessment of social change through storytelling requires what methods? How do we measure impact? Too often it is done through donor dollars. But a growing number of artists and organizers and strategists are more intentionally coming to work together to use narrative power for changing public attitudes and policy development. The process of narrative change operates through multiple channels including entertainment and strategic communication, art, and activism, each intentionally aimed at shifting perception and behavioral patterns..

A four-part framework developed by Yewande O. Addie, David Hanson, Emily Melnick, Melody Mohebi and Annie Neimand hopes to enable organizations to link their storytelling initiatives to quantifiable outcomes. The framework promotes experimental approaches because social transformation processes follow intricate and unpredictable patterns. A mural has the ability to transform how people discuss certain topics. A documentary has the power to influence which policies receive funding priority.

They propose a versatile theory of change framework which states that performing [x] actions will produce [y] effects which will lead to [z] results. For instance, if a film humanizes incarcerated individuals, then viewers may empathize more deeply, so ultimately support for criminal justice reform may grow. But the film may fundamentally be glorifying career criminals or those mentally unfit to live in the general population.

In narrative medicine we believe stories matter, but we struggle to measure the emergence, or increase in empathy, and how it improves patient provider relationships, corporate team cohesion, or a community cause. Narrative change similarly when applied to broader social mores attempts shifts. Once again, what measurement might be used are critical to determining immediate and longer term impacts.

Organizations and qualitative studies need to contemplate shifting from the use of less than robust surveys, to creating more universally agreed upon and purposeful assessment methods. The evaluation process might require identification of which beliefs underwent re-consideration or transformation. 

The process of assessing narrative change proves to be extremely difficult to accomplish. The effects of change take time to develop and follow unpredictable patterns which become visible through extended observation periods. Standard metrics are lacking. The process of determining cause and effect becomes complicated when numerous organizations operate within the same environment with biased agendas. Data is extremely malable depending on how you chose to collect it. Funding for evaluations often comes from partisian groups, and the distribution of power determines which studies will receive recognition. These challenges coupled with e ffects that are often nonlinear, along with the lack of standardardized metrics end up amplifing reliance on too few measurenments.

Despite these challenges, organizations can learn from their efforts by embracing complexity and asking: What changed, for whom, and why does it matter?

So, in a world shaped by stories, how do we ensure measurements of the impact of narrative practices reflect their true power?

-Lauren Manning

Organizations committed to social change through storytelling can use this four-part framework to design and evaluate their narrative strategies.

By Yewande O. Addie, David Hanson, Emily Melnick, Melody Mohebi & Annie Neimand

Many organizations working toward social change understand the power of stories to transform how people think about and act on issues. It follows that a common question among movement leaders, community-based organizations, philanthropy, government, and entertainment is: How do you measure narrative change? These leaders want to know if their work is making a difference. Particularly now, when so many organizations are experiencing seismic changes in their work toward a fairer and more just world; they are rightly looking to ensure that the stories they tell are seeding the conditions for change.

Social change work is complex, non-linear, and long-term, and it isn’t always possible to fully trace the effects of a single campaign, media piece, or storytelling effort. However, organizations can take steps to connect the dots between the stories they tell and measurable shifts in beliefs, behaviors, culture, and institutions. This article presents a flexible framework for measuring narrative change efforts that organizations can creatively adapt to their unique work. Developed by a group of social scientists turned evaluation- and systems-change practitioners who work on issues ranging from pop culture to public health, this framework offers a way (versus the way) to document impact and learnings related to stories that aim to drive change. We encourage organizations to experiment as they incorporate these ideas into their work.

What Is Narrative Change?

Narrative change uses the power of stories to shape how people understand issues and the world. It can bring different groups of people together, shed light on issues that are important to everyone, and help people see new possibilities for positively changing the world. It’s an important (though underfunded) aspect of social change interventions, and philanthropists, artists, nonprofits, strategic communication firms, and social movement organizers all use it to achieve particular social change goals.

But narrative change isn’t just one thing. People and organizations across the field approach it in different ways. We see narrative change work predominantly falling into three broad buckets we call “narrative orientations,” a concept originating from narrative strategist Samantha Wright.

There are likely more orientations than these, and in most cases, practitioners use a combination of approaches. However you define it, incorporating narrative orientation into your theory of change—what you hope to achieve through your work—sets the stage for an appropriate measurement strategy.

A theory of change follows an if-then-ultimately logic. In this model, “x” is the activity, “y” is the immediate outcome, and “z” is the desired, longer-term change. You can simply fill in the blanks, and what you plug in for your theory of change becomes a roadmap for what to measure.

If we do [x], then [y], so ultimately [z].

Here are some examples in practice, starting with a theory of change for an arts and entertainment narrative orientation: If we create a film that highlights the power of rehabilitation and community within the US prison system, then audiences will gain a deeper understanding of the humanity and resilience of incarcerated individuals, so ultimately public perception toward people who are incarcerated will shift and support for reentry initiatives, prison education programs, and criminal justice reform will grow. This reflects the theory of change behind the film Sing Sing, which follows a group of incarcerated men staging a play as part of a prison theater program.

For an example of a theory of change related to a strategic communications narrative orientation: If we create a documentary to strategically translate research on the efficacy of community crisis response interventions, then ideas about what is possible for public safety will expand, so ultimately society will reduce reliance on policing and invest more in mental health-centered solutions. HEART: Serving Our Neighbors in Crisis is a 30-minute documentary about Holistic Empathetic Assistance Response Team (HEART), an alternative crisis response program in Durham, North Carolina. HEART works with police, firefighters, and emergency medical response teams as part of the city’s Community Safety Department. It consists of four units that cooperate to de-escalate crises and help individuals get care: Mental health clinicians at 911 call centers divert calls where appropriate; unarmed, three-person teams respond to nonviolent behavioral health and quality-of-life situations; care specialists connect people to ongoing services; and teams that pair clinicians with police take on higher-risk situations. The film was shaped by regional research findings from RTI International’s Carolina Cohort of Cities project, which analyzed more than 3 million emergency call records across 7 jurisdictions and revealed that fewer than 2 percent of emergency calls involved violent crime. The documentary showcases the lived experiences of Durham-based community members who benefited from these alternative response services and offers insights for communities exploring alternatives to policing that can inform policy discussions and shift narratives about resource allocation.

Finally, a theory of change example related to an organizing narrative orientation: If young people of color who are navigating the immigration system share their experiences and political education as part of a larger initiative to engage young people in politics, then this community will build meaningful connections and increase their political engagement while amplifying a new, diverse narrative. So ultimately support for fair policies and stronger, safer communities will grow. This is the model for the US-based, immigrant youth-led network U nited We Dream’s Unafraid to Lead! Campaign.

Some initiatives might use a mix of these. Take the Fight Forever Chemicals impact campaign, which accompanied the release of Dark Waters, a film that features an attorney seeking to expose a toxic waste-dumping scheme. The project had both strategic communications and an arts and entertainment orientation. The film garnered an audience in its own right and raised awareness of highly toxic, persistent chemicals, while the campaign, led by Participant Media, engaged with advocates to address US and European legal and corporate policies demanding stronger environmental health protections.

As noted above, an organization’s theory of change and narrative orientation will inform what it measures. A social impact entertainment company, for example, would want to know the reach of their film or television series (such as how many people saw it and who they are), as well as how viewers perceived the issue before and after the film, and their intentions moving forward (such as whether they plan to learn more, get involved, or vote a particular way on an issue). An organization running strategic communications for a campaign would measure the effectiveness of a message, who it reached, and the number of people who followed the call to action. A social movement organization measuring narrative organizing would measure things like increased collective identity, the number of community members who shared their stories, the community's increased influence, and the community’s perception of the issue before and after the organizing effort.

At its core, measurement is the systematic tracking of change indicators. You collect these indicators using different methods, depending on what you want to know and your resources, to track your theory of change. And, as a result, you learn where you need to update your theory of change.

It’s best to start with a baseline of data that tells you where the group of people you want to engage are starting from and the existing narrative landscape. By establishing baselines and tracking progress, you can strategically adapt activities, ensuring that you align your efforts with desired outcomes.

It’s also essential to document context—the underlying conditions influencing how and why social change happens. Specifically, this means establishing the social, political, historical, and economic conditions in which you’re working and the current narratives in the communities you serve. For example, at the start of the COVID-19 pandemic, vaccine hesitancy was largely shaped by historical mistrust in medical institutions, particularly among communities of color. Public health campaigns initially focused on combating misinformation and promoting scientific trust. However, as the pandemic evolved, new narratives emerged concerning personal freedoms, employer mandates, and vaccine access disparities. Tracking these shifts allowed organizations to refine their messaging to address specific fears and emphasize community-based, trust-building strategies that increased vaccine uptake.

What you measure matters, and you can’t measure everything. It’s best to focus on quantifiable and qualitative indicators of change to track your progress over time. Below is a simple framework that breaks these down into four interconnected categories that work across all narrative orientations, and help identify what to measure: message and messenger, reach (activities), people (outcomes), and culture and structures (impact).

Step one: Identify your message and messenger indicators. What is your narrative orientation and intervention? What are you putting into the world to shift the narrative conditions? The guide “Measuring Narrative Change” from social sector consulting firm ORS Impact, offers a helpful list of ways to consider how your narrative functions, including whether you are “creating,” “countering,” “amplifying,” or “reframing” messages. Indicators you would track here might include how your messages address power structures, which voices they center, and whether the stories you share are emotionally engaging and include characters audiences identify with. These indicators tell you whether a story transports the audience into the world of the character, an important aspect of storytelling for social change. Tracking them will allow you to tell the story of how your organization engaged in narrative change.

Step two: Include indicators that will tell you if you have reached your intended community. Who have you reached? For entertainment media, use the data available based on your distribution platform alongside press coverage. For in-person interventions like film screenings for corporate leaders or policy makers, or events where people engage in collective storytelling activities like a protest or social movement gathering, you want to know who attended and whether they were the folks you hoped to engage. You might assess this through analysis of invitation response forms and sign-in sheets.

For digital campaigns, such as campaigns that rely on sharing messages via social influencers or the distribution of short-form content, track the reach of relevant posts (through things like impressions and mentions) and engagement (for example, reshares, likes, and comments). You can do this through standard social media listening, before-and-after public opinion polling, and even research methods typically used to evaluate brand awareness and consumer behavior.

Narrative change measurement efforts often stop after steps one and two. However, these indicators don’t tell the full story; the question of whether your interventions have helped move beliefs or behaviors remains. Of course, it’s nearly impossible to prove that a single narrative intervention caused a specific behavioral effect because the world is too complex, fast-changing, and full of complementary narrative interventions. Nevertheless, taking two additional steps allows you to describe the work you did and the shifts you observed to determine your strategy going forward.

Step three: Assess the immediate outcomes for the people you seek to engage. This is your opportunity to identify shifts in knowledge, perceptions, attitudes, beliefs, and behaviors. Assess what has changed for the people you reached based on your narrative orientation. Narrative organizers might ask: Have the power and influence of the people most affected by the issue increased? For arts and entertainment: Do viewers feel called to action? To collect this information, you can use methods like surveys and polling, focus groups with representative samples, and even creative community-tailored approaches to demonstrate impact.

For example, the American Factory documentary film, which follows the reopening of a factory in Ohio, explores the nature of work and workers in the United States. The impact campaign that accompanied its release, also run by Participant Media, brought together workers, labor groups, corporations, and other organizations through film screenings, panels, and discussions. Campaign organizers surveyed participants on what they heard or discussed at the events through printed survey cards. (Having assessed dozens of campaigns, we’ve found that providing something tangible like pen and paper increases participation in evaluation at in-person events.) Organizers then used the results, alongside interviews and participant observation, to understand how the gatherings affected participants’ attitudes and intended actions—for example, whether their experience motivated them to host follow-up meetings with other participants, join a new network, or implement learnings at their own organization.

Step four: Connect the dots between your indicators to tell the story of the intervention’s contribution to cultural and structural change. Measurement at this scale is challenging due to the complexity and the long-term nature of change. Seeing the true effects of narrative change takes decades, not the length of a grant or campaign cycle. The task, then, is to track the contribution of your work within a particular period of time, drawing connections between it and observable shifts in cultural discourse, social norms, and how structures operate.

You can track public discourse through media analysis, social listening, policy documentation, and organizational change. You can also use techniques such as ethnographies, ripple effect mapping, and outcomes harvesting to assess the broader social impact of your work.

Working with a set of core principles as you carry out these steps can help ensure that your interventions support larger social change efforts in the field and build power for communities facing the greatest oppression.

1. Start from the center: Your narrative change strategy and measurement plan should come from the people or communities whose stories you are telling (i.e., the interest-holders). For example, a narrative change strategy focusing on hunger should involve people who access food pantries to help design the evaluation. A narrative change strategy focused on community safety should involve neighbors, social workers, and law enforcement to design the evaluation. Doing so allows you to identify metrics that will be valid for the community you’re working with, and develop rigorous and relevant evaluation questions. It also gives you the best opportunity to understand the effectiveness of your narrative strategy. This principle views collaboration, co-creation, and relationship-building as essential to narrative evaluation. The Movement Strategy Center’s “Spectrum of Engagement to Ownership” offers models for collaborating with community, and can help you assess whether interventions are merely informing people about an intervention or measurement strategy or involving them in the solutions process.

2. Let go of prediction and embrace surprise. It’s important to move beyond prediction. So often, organizations have concerns about controlling the outcomes of an intervention, but when you’re dealing with people, you often have little control. Embrace the unexpected. It's useful to view surprises, shifts, and unanticipated outcomes not as failures but as opportunities to learn and adapt. Flexibility and openness to unexpected changes can reveal new insights, strengthen coalitions, and guide strategies toward meaningful and equitable change.

For example, imagine a campaign aiming to reshape a narrative about refugees from “burdens” to “contributors.” Initially, the organizers share stories of refugees starting new businesses. Unexpectedly, they find significant engagement from local artists who want to collaborate with refugee artisans. This surprise leads the campaign to highlight joint art exhibitions and cultural exchanges, expanding the narrative to include the vital cultural contributions refugees bring. Letting go of prediction allows for innovation, and helps ensure that narrative change efforts remain rooted in community needs and responsive to evolving contexts.

3. Focus on contribution. Given that directly attributing social change to a single narrative intervention is nearly impossible, it’s helpful to focus on describing the specific activities that comprised your narrative intervention, and the output and outcome data you’ve collected along the way. Look at the hyperlocal impacts your intervention had, as well as other, related narrative change efforts. Each data point is a piece of a puzzle and helps show the contributions of an intervention within a larger context.

4. Engage in constant learning. Consider the higher purpose of your work. At its core, learning is about understanding what you should stop doing, what you should do differently, and what you should do more of. Evaluation is best when it’s designed to build learning. Too many evaluations sit untouched in Google Drive or deep in an organization's online archives. Instead, share what you learned during projects and in before- and after-action reviews with your team and the field. Establishing practices and making space for learning fuels creative problem-solving.

If this seems overwhelming, remember you don’t have to do everything at once. Do what you can with what you have available to you. Resources like Kate Davies and Joseph Walton’s book Our Future, Our Voice can help. Ultimately, anyone looking to measure and improve their theory of change can take steps, however small, to understand the contribution of their work to the bigger picture. Social change is complex, and there’s no single formula or number to measure it, but leading with values and grounding evaluation efforts in a simple framework can do a lot to tell the story of narrative change.