Conversations about impact in documentary: beyond fear & loathing.

Aufderheide, Patricia

[ILLUSTRATION OMITTED]

The conversation about impact is an old one, as we know, thanks to the careful analysis of Brian Winston, particularly about the rhetorical games that John Grierson could play with funders and officials. (1) As soon as someone said, well, what good is art to this important policy issue? he argued that film was the most persuasive possible medium. But if someone said, Prove it, he said, This is art, its power is immeasurable. For many years, documentarians' funder pitches lived on that two-sided argument.

With the affordances of an always-connected, user-centric media universe, the conversation about is also a new one. We are all generating data all the time, after all, and it's all feedback for somebody. That data generation is both the fuel for businesses ranging from Facebook to Uber, and it is also a new tool of business management.

This data-rich media environment has had a big effect on social-issue documentary, especially because of its dependence on subsidy, whether from public or private patrons. Funders are far more demanding of better evidence of effectiveness, or, in the current parlance, impact, than in the analog era, when impact could be claimed without being shown. "Impact" has become a buzzword in documentary with the push to entrepreneurialize the documentarian. Practitioners funders as well as makers--chafe under the weight of quantitative metrics.

Impact conversations are currently mostly practitioner-driven. They usually occur outside academia, and often show a lack of awareness of research approaches in social science. They also lean heavily on knowledge of management practices in the nonprofit environment. Why that should be is worth thinking about. I propose that the shift toward a focus on extracting more information about the information funders fund is to some extent related to changes in management practices with the rise of Silicon Valley.

Some of the largest funders of media are also nerds, or otherwise leaders of the digital businesses that run on information. Thus, the Bill and Melinda Gates Foundation both funds media and impact assessment; the Skoll Foundation (fed by profits from eBay) funds media and establishes benchmarks for measuring impact. Crowdfunders as well want feedback and evidence. More traditional funders are being given mandates by leaders and boards to provide evidence of impact for media investments.

An "ecosystem of change."

The evolution of impact conversations is also related to the changes in the basic business models of social-issue documentary film. Documentary film has historically depended on relatively few sources to fund work: private patrons (Joris Ivens' rich Chinese Communist backer for Indonesia Calling; the family trust fund); public broadcasting (the Canadians' National Film Board is a gold standard); sweat equity (almost no social-issue documentaries make money, Michael Moore always excepted). With the neoliberal-fed decline in public broadcasting, the expansion of production capacity, and the proliferation of screens, filmmakers more than ever need to sell themselves and each project to a variety of investors, each of which has to be approached in terms of their own interests and investment criteria. Filmmakers as entrepreneurs need to have management skills and tools.

An "ecosystem of change," as the strategic consultancy Active Voice terms it, has evolved, especially in the U.S., over the last 15 years, building upon venerable "outreach" organizations such as Active Voice. Strategic consultants include such organizations as Aggregate, Borderline Media, Harmony Media, and Picture Motion. Social-change organizers at places like Art & Democracy and the Citizen Engagement Lab include media design in their strategic thinking. Commercial production funders of social-issue film include the Impact Producers Group and Cactus Three. Associations include the National Association of Media Arts and Culture (NAMAC) and the Indie Caucus. Events include The Good Pitch (uniting issue funders with media makers), the Center for Media & Social Impact's Media That Matters conference, and strands at film festivals such as SXSW and Tribeca. The Ford Foundation has made permanent an experiment in dedicated funding for social-issue films, its Just Film fund, and the Fledgling Fund's resources are dedicated to impact strategies. Other funders, such as the John D. and Catherine T. MacArthur Foundation and Chicken & Egg Pictures, also fund impact strategies. Tools include Harvis, an app that lets audience members register their emotional reactions; OVEE, which allow for group viewing with feedback; the Harmony Institute's StoryPilot; and of course the welter of off-the-shelf analytics available from Google and YouTube. Internationally this effort is met in the United Kingdom with the multifaceted work of BritDocs--a funder, information resource, convenor, and more--and of the Scottish Film Institute, associated in part with Edinburgh University, which both coproduces work and produces analyses of impact.

The conversation about impact, and the growing ecology of organizations, is taking place within a much larger context that is relentlessly neoliberal in both its results-oriented language and expectations. That neoliberal trend can make it easy to discount the importance and relevance of asking questions relevant to anyone who believes they are investing their mortal time in communicating something because it can change something. This impact emphasis is also generating a lot of nervousness among people working to some degree outside the ruthless, ratings-driven world of commercial film and television, who have not previously been asked how exactly they are making any kind of a difference, and who in the past have been confident that if they could not answer that question precisely, neither could anyone else. It has also been easy for makers to naturalize the very real ratings concerns of any broadcast environment, even among the PSBs, rather than to see them as crude metrics of their own.

The level of anxiety is neatly measured in what happened after the public release of The Participant Index, which combines survey data with analytics from social media and raw numbers for viewership to create rankings for impact on two scales: emotional and level of engagement (or action). This drew a firestorm of scorn, outrage and indignation on one of the most popular listservs about documentary impact, particularly from strategic consultants. The group focused on the Index's reduction of a broad set of social processes put in motion by documentary interventions into one number, itself a synthesis of two reductive numbers. Several blog posts resulted. In one, Patricia Finneran, summarized the concern with the question: "Will outcome measurement trump nuanced understanding of the human experience? " In another, the director of Active Voice summarized the concerns of filmmakers as, "How can you assign a numeric value to art? "The director of Aggregate warned that looking for short-term goals could lead to funding of "more didactic narratives about issues that lend themselves to relatively straightforward solutions." (2)

Some, in short, argued that some art was irreduceable to measurement; some that more sophisticated tools were needed; and some that a deeper understanding of the circulation of information was needed. This latter point has been argued repeatedly by social scientists, strategic consultants and funders. One funder, Diana Barrett of the Fledgling Fund, expressed this succinctly on the impact listserv (June 9, 2015): "in many cases, the possible changes are long tailed and more difficult to specify and measure, yet the changes in both awareness and action may be significant."

In a discussion that developed around the draft of this paper in fall 2015 on academia.edu (https://www. academia.edu/s/984961463a), filmmakers, distributors and critics repeated some of the prevalent anxieties. Commenters variously expressed concern that measurement clashed with the reality that film was an art form; that immediate post-facto measurement could not take account of serendipity; that highly targeted films could reach small audiences with enormous real-world effects; that all measurement was an example of neoliberal creep; that measurement subsumes a filmmaker's agenda to the funder's; and that measurement turned the filmmaker into a tool of activists. In general, metrics seem to have been imagined as immediate post-facto, numerical results reflecting the same values as, say, ratings.

Easy targets, more complex realities.

The Participant Index (PI) posed almost too easy a target, because it features a synthetic number. (Pi's actual practice of impact measurement is more sophisticated than some of the easy outrage around it suggests, even though this challenge is no easier for documentary than for any other media phenomenon, and possibly harder because of the one-time nature of most documentary viewing.) The hostility directed at it was partly fueled by the frustrations of negotiating a very large and real problem with inadequate conceptual or logistical tools--as was demonstrated again in the academia.edu discussions.

So this is not an easy problem, but most people who are approaching this seriously know that. Strategic communications experts and funders have done thoughtful work worth looking at, in the vast space between the binary of "assign a number" and "art cannot be measured." Most of it is in the tactical space of articulating how to identify and find proxies to measure one's ambitions.

One of the central texts for the field is a 2008 paper from the Fledgling Fund, which grounds all of its work in an aspiration to sound management practice. The paper methodically demonstrates how to set goals, and match them with appropriate expectations, and thus metrics. It stresses the importance of close analysis of what social-change goals are for a project, and what the state of play in the society and media ecology is--which will then guide an understanding of what kinds of interventions are useful or even possible:

   it is critically important to understand the state
   of the movement and where an issue is in the
   public consciousness in order to set realistic
   expectations for impact. It is not reasonable to
   expect broad social change if there is little public
   awareness that a problem exists. In some cases,
   just getting audiences to see the film, connect
   with the story and better understand an issue is
   enough. This awareness is the first step to social
   change. (3)

The Fledgling Fund's framing has added nuance and grounding to an arena that badly needed it. The field also benefits from increasingly nuanced and accessible discussions of aligning metrics with aspirations, using a variety of metrics, and capturing nuance with ethnographic detail and case study in related non-profit literature. (4) To restate some obvious points from these discussions: metrics can be qualitative and quantitative, they need to reflect goals, they need to be appropriate to the questions asked, they cannot cover all possible outcomes. The enormous value of being involved with questions of metrics from the start is being able to set some of the terms of measurement, appropriately to the project.

At the same time, the field has evolved without articulating deeper expectations for how communications functions to create--reinforce or change--reality in a society. Specifically, articulating the theoretical basis for how communications effects change is often missing. This of course is not merely a failing of the field, but a source of great theoretical controversy within communications, where the mere articulation of a position associated with cultural studies can be seen as provocative by some social scientists. But it is an important, and even bedrock, conceptual issue for those trying to map the social change they try to achieve. It is not something that practitioners are likely to do on their own.

Most people in practitioner listserv conversations, one-to-one conversations and festival/conference panels are neither blind believers in empiricism or woolly-headed do-gooders. They are trying to answer a reasonable question--does what I do matter? And if so for whom? But the conceptual tools are still evolving, and the conversation have, for the most part, taken place in rooms closed to those equipped to discuss these more nuanced questions.

Meta-analyses emerge, the academics join.

As the impact conversation has developed, discussion is beginning to become deeper, with academics more engaged. Consider first a meta-analysis of approaches to impact. In an overview that includes both the state of the field and a map of methodological approaches to measuring impact, social scientist and filmmaker Caty Borum Chattoo, Executive in residence at American University, strongly urges people in the documentary world to learn from past social science research and methodologies, to avoid reinventing wheels. (5)

She provides a breakdown of five basic methodologies, with their limitations: audience survey, content analysis, experiment, in-depth interview/focus groups; ethnography. She then provides a breakdown of metrics and available tools to get at them, in four categories: Digital & Media Coverage Metrics; Audience Impact; Content & Cultural Impact; and Institutional Impact.

In these categories, one approach of great and deserved current interest, is networking analysis. This demonstrates the connections people are making, as well as showing when they are making them and when they make other ones. It tells a study about use--both how much and what kind--and about framing (within what context those who connect with the story understand it). Network analysis typically draws from social media use, and so has the limitations of its data--social media sites filter their data, not all data is available (Twitter charges for anything that is not very recent), and not everyone's connections are being made on social media. It can also be done at the textual level, to show relationships between texts, and at the level of published media, to show relationships between issues and sites.

Results can still be very interesting and provocative. Take, for instance, a study of SOPA/PIPA coverage between 2010 and 2012: http://cyber.law.harvard.edu/ research/mediacloud/2013/mapping_sopa_pipa/#. Note, however, the crucially important analysis done side by side with the visualization; the analysis often highlights elements not necessarily central to the visualization. Thus, generating the data was crucial to the analysis, but the analysis was still an interpretation of information with a remarkable amount of ambiguity.

Scholar Jana Diesner's work on social computing, out of U of Illinois-Champaign-Urbana, analyzes press and social media around documentaries for evidence of impact.' Her work, which combines expertise on networking research with access to proprietary materials from filmmakers, shows how computing can do some of the work of analysis when academics and practitioners work together. She shows, for instance, the difference in type of press coverage for two different films about women and violence, "Pray the Devil Back to Hell," about Liberia, and "I Came to Testify," about Serbia. These show the difference between a press focus on the film as a film (the first) and about the issue (the second). Knowing that difference, however, is just the beginning of interpretation.

Chattoo strongly recommends a diversity of methods and a matching of them to diverse goals. These need not be social science approaches, either. This is evident in BritDocs' evaluation of impact of The Invisible War. It includes metrics ranging from Nielsen ratings (2.1M viewers) to DVD sales (17K) to website views (475K) to social media mentions over time to changes in military policy and named legislative proposals.' None of this is analysis of changed attitudes or behaviors, which would have required some kind of analysis that collected data, either directly or indirectly, from viewers. It would also require a theory of circulation and effects.

Another demonstration of the evolution of theoretical discussion around impact is a recent report from the field of social-issue games," executed both by social science consultants and academics, which draws heavily on impact assessment in documentary work. The report identifies a conversation so confused that the first step has to be identifying the different threads:

   Not just beginners, but our best journals and
   public awards can inadvertently overlook full
   categories of impact, and disagree on what
   evidence looks like. Creativity is too easily and
   unhealthily pitted against impact design. Even
   the language of "double-blind trials" can ironically
   blind our field to certain types of impact.

   Success may require new umbrella language
   to enable meaningful comparison and improve
   coherence and efficacy--especially across stakeholders.
   Power may need to be shared, rather
   than giving preference to either researchers or
   designers.' (9)

The researchers found five big problems with the current conversations on impact, creating ways that people speak past each other. These five problems have strong echoes in the social-issue documentary world as well:

* Defining impact too narrowly. (Did the Foldlt games teach people about how proteins work?

No. But it did get many people to suggest innovative ways that proteins might fold, which helped scientists.)

* Key terms are politicized. Having strategic goals and an assessment strategy become ends in themselves for some. Impact assessment becomes a rigid standard used for exclusion.

* Evaluation methods are inflexible. (Typically ethnography and interviews are looked down upon, in comparison with something quantitative.)

* Awards and promotions prematurely crystallize what is meant by social impact.

* Typologies are siloed, e.g. by subject matter, by type of gameplay, by popular moniker.) (10)

The prospect of "power sharing," as the Stokes report calls it, between practitioners and academics is a heartening concept. It is a space ready for occupancy, where synergies could be found between people who need subject matter and data, and people who need analysis and a conceptual framework within which to exit the squirrel cage fed by buzzwords.

[ILLUSTRATION OMITTED]

Even more promising is the possibility that academics from different regions of academia could both learn from each other and inform their fields successfully. For instance, the field of education has a long scholarly history of heavily-researched evaluation, both formative and summative, (11) with direct relationships with the field of educators. Research has also been done on the effectiveness of documentary film for educators. (12) Assessment requirements in higher education have triggered discussions of metrics in related areas, and at least one film scholar has found that quantitative metrics for evaluating the quality of student films led to useful feedback. (13) Scholars and practitioners working on documentary impact could build both knowledge and relationships across these silos as well.

Lurking behind the impact conversation.

There is plenty of work to be done. This impact conversation is happening against a murky backdrop within the field of documentary. Lack of clarity about the social role of both such films and filmmakers is endemic and historical. ("Is it good for the sheep?" one Navajo famously asked the anthropological filmmakers visiting him.) (14) Impact questions slop over into ethics questions, about the functions and obligations of filmmakers.

Here is a demonstration. In May 2015, controversy erupted over a showing of the 1991 film Paris Is Burning, about transgender voguing in New York. Organizers of a Brooklyn event hosted an all-white panel, which triggered a local controversy. Activist Heather Dockray indicted social-issue filmmakers in general, including Steve James, Joshua Oppenheimer and Eugene Jarecki:

   The controversy behind Paris is Burning raises
   important questions about race, class, and even
   storytelling. At what point does a director/
   storyteller's job end? Once the movie is released
   in theaters, and streaming on Netflix? Or far
   later, after substantive change has been made?
   Documentary journalists almost always pick
   the former, but that doesn't lessen the hurt felt
   by the subjects left behind. It's difficult for a
   storyteller to know when to end a story when its
   plot, and its pain, feels endless. (15)

NAMAC's Wendy Levy articulates a justification that portrays documentarians as "art-ivists," part of a social change movement:

   They spend time crafting story-driven engagement
   strategies, community screenings, interactive
   digital tools & apps, performances, panels,
   petitions, workshops--all to insure that when
   the lights come up after their film has screened,
   the story belongs to the community, connections
   have been made, relationships forged, and a very
   big space has been created for their voices and
   the next steps in the struggle.

   Collaboration is, and must be, the new innovation
   --and we must take the time to cultivate
   partnerships where those people and organizations
   doing relevant work on the ground get
   invited into the filmmaking process early and
   are prepared to carry the stories into the world
   with a voice even more nuanced, inclusive and
   effective than the filmmaker's alone... Can we
   join together to upend a Hollywood system
   that perpetuates primitive and outdated power
   structures, marginalizes the voices of women
   and people of color, and turns art and entertainment
   into cultural appropriation? (16)

This vision of intrepid filmmakers inspiring and possibly leading a movement to overthrow society's power structures is a very far cry from the experience and expectations of filmmakers, most of whom at least until very recently saw themselves primarily as media makers, not primarily as organizers. The distance is partly measurable in terms of the confusions in the impact conversations themselves.

Academic and professional challenges.

Academics have major contributions to make to this discussion, not least by clarifying expectations about what, how and when communications affect actions. Too often, half-developed tools are thrown at undefined problems, scorn substitutes for analysis, and opinions float free of empirical research.

From a professional point of view, there may be a deeper question still, one that floats along the fast-moving river of digital change. For makers, what ultimately may be at issue is not whether a film or even a kind of film makes a measurable difference, or even whether measuring difference is a worthy exercise in dealing with a cultural intervention. What may be at the base of the discussion is what a documentary filmmaker is and does.

As filmmaking itself becomes available to non-experts; as filmmaking professionals start making apps, games, infographics and interactives; as every NGO, political party and think tank comes to have its own media strategist/producer, the impact discussion starts to look like the beginning of a much bigger one. Lurking at the back of the room in the impact discussion is the question of exactly what role makers of these arguments see themselves playing in the ecology of communication that forms the fabric of reality, as their own century-old place in the economic ecology is shifting.

Filmmakers have historically understood themselves to be artists. Documentary film has sometimes been slighted in that discussion, as photography has, because of its representative claims. The impact challenge threatens to move it further into an instrumental realm. The impact challenge calls the bluff of the Griersonian shell game--is it art? Is it propaganda? Is it a social welfare tool? It forces documentarians to articulate a goal and demonstrate whether it was reached. It pushes them--if Wendy Levy is right--in the direction of being more like organizers and convenors than artists.

This unsettled question of role is happening in other places too, although it is different depending on the professional mission. Journalists are having to rethink their sacred value of objectivity. Educators are having to step sagely off the stage. Ad makers can no longer burnish reputations on the cleverness of their taglines, and are employing small armies of wonks. A participatory culture, as envisioned by Henry Jenkins and colleagues, is struggling to emerge in spite of constraints of inequality, strictures of traditional business models, professional expectations and creaky pre-digital laws and policies. (17)

This is a deeply uncomfortable moment, when professional identity itself is in play. I submit that the manifest anxiety and tensions around impact metrics within the documentary ecology have, among other things, the function of deflecting a much-needed conversation in a direction many in our innumerate society can agree on: You cannot reduce the effect of a cultural intervention to a number. In the process, it is easy to ignore the huge power afforded to makers by the ability to connect so intimately with their users, to get feedback so rich and detailed on their uses, to learn so much about communities of use even while in the process of making the film.

Yet bigger conversations--about what expectations makers themselves have of their craft and practice, about how their professional identity is morphing, about which of the traditions of documentary carry forward into this set of expectations--can go begging while makers rally around the easy conclusion that you can't reduce art's effects to a number and academics produce work for each other.

A cross-cultural and ongoing conversation about principles and theories behind and methods of measuring impact is worth having, across boundaries of professions and across academic ecologies as well. It might be an essential part of the evolution of documentary in a participatory culture.

Notes

(1) Brian Winston, Claiming the Real: The Griersonian Documentary and Its Legitimations (London: British Film Institute, 1995).

(2) Jessica Clark to Media Impact Funders, July 29, 2014, http://mediaimpactfunders.org/nyt-piece-kicks-dialogue-on-documentary-impact-into-high-gear/.

(3) Diana Barrett and Sheila Leddy, "Assessing Creative Media's Social Impact," (New York: The Fledgling Fund, 2009), 7.

(4) Jessica Clark and Tracy Van Slyke, Beyond the Echo Chamber: Reshaping Poiitics through Networked Progressive Media (New York: The New Press: Distributed by W.W. Norton & Co., 2010); Beth Kanter, Katie Delahaye Paine, and William T. Paarlberg, Measuring the Networked Nonprofit: Using Data to Change the World, First edition, ed. (San Francisco, California: Jossey-Bass, a Wiley Imprint, 2012); Allison H. Fine, Momentum: Igniting Social Change in the Connected Age (San Francisco, CA: Jossey-Bass, 2006).

(5) Caty Borum Chattoo, "Assessing the Social Impact of Issues-Focused Documentaries: Research Methods & Future Considerations," (Center for Media & Social Impact: American University, 2014).

(6) Jana Diesner, Jinseok Kim, and Susie Pak, "Computational Impact Assessment of Social Justice Documentaries," The Journal of Electronic Publishing 17, no. 3 (2014); Jana Diesner and Rezvaneh Rezapour, "Social Computing for Impact Assessment of Social Change Projects," in International Social Computing, Behavioral Modeling and Prediction Conference (Washington, DC2015).

(7) Britdoc, "The Invisible War," (London, UK: Britdocs, n.d. [2013]).

(8) Benjamin Stokes et al., "Impact with Games: A Fragmented Field.," (Pittsburgh, PA: Games for Change and the Michael Cohen Group, 2015).

(9) Ibid., 4.

(10) Ibid., 12-33.

(11) Anthony E. Kelly, Richard A. Lesh, and John Y. Baek, Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching (New York London: Routledge, 2008).

(12) Raul Zaritsky, "Design Methods for Educational Media to Communicate," in Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching, ed. A. E. Kelly, R. A. Lesh, and J. Y. Baek (New York: Routledge, 2008); Ricki Goldman, Video Research in the Learning Sciences (Mahwah, N.J.: Lawrence Erlbaum Associates, Inc., 2007).

(13) Frank Tomasulo, ""You Can Play Mathematical Equations on the Violin!": Quantifying Artistic Learning Outcomes for Assessment Purposes," Journal of Film and Video 60, no. 3/4 (2008).

(14) Sol Worth and John Adair, Through Navajo Eyes; an Exploration in Film Communication and Anthropology (Bloomington,: Indiana University Press, 1972), 4.

(15) Heather Dockray to Brooklyn Based, May 13, 2015, http:// brooklynbased.com/blog/2015/05/13/celebrate-brooklyns-paris-burning-screening-sparked-fire-facebook/.

(16) Wendy Levy to Documentary, May 21, 2015, www.documentary. org/blog/furor-over-paris-burning-raises-burning-questions.

(17) Henry Jenkins, Convergence Culture: Where Old and New Media Collide (New York: New York University Press, 2006); Henry Jenkins, Sam Ford, and Joshua Green, Spreadable Media : Creating Value and Meaning in a Networked Culture, Postmillennial Pop (New York ; London: New York University Press, 2013).