Policy legitimacy, rhetorical politics, and the evaluation of city-street video surveillance monitoring programs in Canada.

Lett, Dan ; Hier, Sean ; Walby, Kevin 等

INTRODUCTION

IN AN IMPORTANT discussion of evaluation research pertaining to contemporary surveillance technologies, Haggerty (2009) argues that managerialism as a governing mentality has contributed to the rise of an evidence-rather than ethics-based paradigm for evaluating and justifying a range of surveillance systems. He allegorizes evaluation research as an "unregulated knife fight" where "rules do not apply" and claims about objectivity function as an ideological gloss to mask political interests. For Haggerty, debates about the extent to which surveillance systems "work" hinge more on the ways in which adversaries debate one another on methodological grounds than they do on objective scientific evidence or the ethics of conducting public surveillance.

Haggerty's arguments about the rhetorical politics of evaluation research are a welcome addition to the literature on surveillance, security practices, and crime control. His primarily theoretical arguments, however, can be supplemented and refined through empirical investigation. To this end, we use Haggerty's arguments as an entry point to examine the role of rhetorical politics (1) in achieving and maintaining policy legitimacy in one surveillance domain: city-street video surveillance in the province of Ontario, Canada. We explain how video surveillance systems in Ontario are partially legitimized through a set of rhetorical policy gestures that appear to comply with principles institutionalized in the Ontario Information and Privacy Commissioner (IPC)'s privacy protection policy framework on public-area video surveillance. Examining the role of rhetorical politics in Ontario's video surveillance security culture is important because the existing literature on camera evaluation--a literature whose aim is to improve techniques of evaluation by refining scientific measures (see Burns-Howell and Pascoe 2004; Ratcliffe and Taniguchi 2008; Ratcliffe et al. 2009; Welsh and Farrington 2004)--has not sufficiently explained how rhetorical framing is used to legitimate evaluation findings.

This article has three sections. In the first section, we examine the literature on evidence-based policy evaluation. We also describe Ontario's city-street monitoring evaluation practices in the context of the Ontario IPC's privacy protection policy framework on public-area video surveillance. Following a brief note regarding method, we report on video surveillance program evaluations in four cities, granting special attention to the rhetorical use of evaluation findings in the context of municipal politics and debates. We demonstrate how the rhetorical politics of video surveillance evaluation research strives for legitimacy by appearing to adhere to the IPC's best practices guidelines--guidelines that are formulated to minimize rhetoric in system design and deployment. (2) We also discuss the issue of evaluation atrophy, whereby stakeholders either reduce their efforts to conduct evaluations or cease altogether when external and internal pressures to maintain legitimacy are absent. In the final section, we identify four scenarios for academics, system administrators, and privacy protection advocates to improve the design of evaluation protocols and monitoring programs.

POLICY LEGITIMACY AND CITY-STREET VIDEO SURVEILLANCE

A sizable body of scientific evidence on the crime-reduction potential of public video surveillance indicates that monitoring systems are able to realize their stated objectives in only a small number of physical locations (e.g., parking areas) and under specific conditions (e.g., camera positioning, effective communication channels among law enforcement agencies). There is evidence that other measures such as street lighting and security guards are more effective than video surveillance in preventing property and violent crime (Welsh and Farrington 2009). Another set of studies finds that city-street video surveillance is not very effective in reducing crime in city centers (Gill 2003; Gill and Spriggs 2005; Welsh and Farrington 2004, 2005, 2009). Yet, city centers are the most common location where city-street video surveillance cameras appear in Canada.

The absence of strong empirical support for the crime-reduction potential of city-street video surveillance is problematic for stakeholders who strive for policy legitimacy when promoting video surveillance. Policy legitimacy refers to a confidence among stakeholders and members of the public that policy options are justified, appropriate, and fair. Policy options need to be framed in a manner that appears to address putative problems by generating appropriate policy solutions (cf. Schon and Rein 1994).

Policy legitimacy has two main components. The first component is substantive legitimacy: the ways in which the substantive content of a policy aligns with the dominant attitudes of stakeholders and members of the public (i.e., the constituents find policy options reasonable). Achieving substantive legitimacy in the video surveillance policy process can be relatively straightforward in Canada, especially when policy options are framed in terms of crime reduction. Available evidence from public opinion research shows moderate-to-high levels of support for video surveillance (Gazso and Haggerty 2009; The Surveillance Project 2008), although this support varies depending on the proposed location of surveillance cameras and how questions are posed to survey and focus group participants (Lett et al. 2010). Variation in the strength of public support for video surveillance (especially across regions) is one reason why the process of building substantive legitimacy is often accompanied by emotive appeals to bolster support (cf. Wallner 2008). (3)

Policy legitimacy also entails procedural legitimacy: the ways in which policy advocates persuade stakeholders and members of local communities that formal standards of policy making have been addressed (Wallner 2008). Procedural legitimacy is achieved when policy formation appears to be based on scientific evidence (Ekos Research Associates 2004) and when it appears to involve some form of public participation in decision making (e.g., a public survey). The realization of procedural legitimacy requires policy makers to adhere to and respect established procedures for introducing and justifying public policy. The process involved in achieving procedural legitimacy can sometimes involve the kind of deliberative methodological antagonisms described by Haggerty (2009). In the case of policy formation on Ontario's city-street video surveillance systems, procedural legitimacy is achieved primarily through demonstrated efforts to adhere to the Ontario privacy commissioner's privacy protection policy framework (Hier and Walby 2011).

The IPC of Ontario publishes Guidelines for the Use of Video Surveillance Cameras in Public Places (Cavoukian 2007/2001). Guidelines are designed to assist provincial and municipal institutions and law enforcement agencies in deciding whether the collection of personal information by means of video surveillance is justifiable and, if so, how to build privacy protection into monitoring programs. The principles informing Guidelines include provisions for determining the necessity for video surveillance as a last resort and the importance of conducting privacy impact assessments prior to commencing with monitoring activities; public consultations and the legality of monitoring practices; collection of personal information, the formation of surveillance policies, and design and implementation concerns; and matters pertaining to the access, use, disclosure, auditing, and disposal of surveillance records.

All surveillance stakeholders in our study strive to achieve policy legitimacy by addressing Guidelines stipulations. We are specifically interested in how stakeholders address Guideline #4:

The use of each video surveillance camera should be justified on the basis of verifiable, specific reports of incidents of crime or significant safety concerns.

[...]

Consultations should be conducted with relevant stakeholders as to the necessity of the proposed video surveillance program and its acceptability to the public. Extensive public consultation should take place (Cavoukian 2007:4).

On the surface, the stipulations covered in Guideline #4 speak to a straightforward process of rational procedural legitimacy (e.g., adopting evidence-based policies, conducting broad and meaningful public consultations). A problem with the stipulations, however, is that Guidelines offer no details on what constitutes adequate public consultations or how to operationalize terms like "safety concerns." Also lacking are indications about how to identify stakeholders who represent the interests of the community. Owing to such ambiguity, the task of policy interpretation defaults to surveillance stakeholders, many of who have a vested interest in promoting systems.

Irrespective of the techniques used to build support for policy initiatives, the realization of substantive and procedural legitimacy is influenced by policy framing. Policy framing occurs at two levels: rhetorical frames and action frames. "Rhetorical' is not used here in the pejorative sense of "duplicitous," but in the sense of a persuasive enunciation "on the record" of how policies will work. Rhetorical frames describe the ways in which "groups ... portray issues deliberately in certain ways so as to win the allegiance of large numbers of people who agree (tacitly) to let the portrait speak for them" (Stone 1988:171) through the "persuasive use of story and argument" (Schon and Rein 1994:32). Rhetorical frames offer a rationale for what a policy will achieve. By contrast, action frames provide the technical reasoning that dictates how a policy will achieve its objectives.

The likelihood of achieving policy legitimacy is high when rhetorical and action frames align. However, as policy framing solidifies into practice, unexpected outcomes can arise in the form of "back talk" (Schon and Rein 1994:123). Back talk refers to "messages sent back to policy designers that surprise them by violating their taken-for-granted assumptions and tacit action frames" (Schon and Rein 1994:123). Examples of back talk in the process of developing video surveillance systems include negative evaluation results and equipment failure/malfunction. Critical cases of back talk occur when problems with action frames (i.e., policy is not working as expected) undermine rhetorical frames (i.e., objectives are not being met). When back talk threatens the policy legitimization process, policy designers can respond by either redesigning or reframing policy initiatives. Regardless of how policy designers deal with back talk, however, all program designs succumb to evaluation atrophy: the declining importance granted to evaluation protocols once a certain level of legitimacy is realized.

The study

Drawing from data collected during a seven-year investigation of city-street video surveillance in Canada, we analyze program evaluations in four Ontario cities: Sudbury, London, Hamilton, and Thunder Bay. (4) We selected these cities for three reasons. First, programs in each city fall under common legal jurisdiction: municipal and provincial privacy laws and the office of the IPC of Ontario. Although program administrators are not obligated to follow the stipulations set out in the IPC's Guidelines, (5) each city in the sample addressed Guidelines, often in consultation with privacy advocates, in an effort to achieve policy legitimacy. Second, the sample is composed of programs that have generated evaluation data under a common policy framework that are amenable to comparison and examination. Third, each city has a history of policy discussion about cameras extending at least as far back as 2000, which has resulted in plenty of long-term data to study. We do not claim that our sample represents all systems in Ontario. Instead, the sample is used to provide data to support theoretical and applied arguments about video surveillance policy making.

Our analysis is based on documents and interview transcripts pertaining to the evaluation process in each city. We are particularly interested in the relationship between program objectives and evaluation techniques. Each city's policy design locates specific program objectives within a corresponding action frame because surveillance programs are "invested with such starkly different hopes [that] the cameras' successes should be evaluated using criteria relevant to each purpose" (Haggerty 2009:282). We show how stakeholders in each city strive for policy legitimacy in the context of stated program objectives, and how the interpretation and dissemination of each cities' evaluation findings rely on rhetorical framing techniques and the negotiation of back talk.

PROGRAM EVALUATION, RHETORICAL POLITICS, AND POLICY LEGITIMACY

Evaluations are structured according to one of three models: (1) pilot evaluations, (2) rolling evaluations, and (3) one-shot evaluations. Despite important variation among evaluation strategies and the techniques used to gauge system effectiveness (explained below), evaluations are often based on crime statistics, public opinion research (e.g., surveys or public consultations), and qualitative/anecdotal, data (e.g., telephone calls, emails). Evaluations commonly mix multiple data categories.

Pilot evaluations operate for a specified period of time prior to the formal launch of programs. The purpose of a pilot project is to determine if systems are realizing stated objectives prior to investing long-term organizational and financial resources. Pilot evaluations vary in terms of the techniques used to gauge effectiveness, but they most often focus on statistical and public opinion data (as well as physical operations).

Rolling evaluations tail the continual collection of data that is periodically reviewed (usually annually). Sometimes rolling evaluations take place at the end of each monitoring season in cases where cities only conduct monitoring during summer months. Rolling evaluations are often incorporated into an overall assessment of crime rates and police statistics for certain city areas. Both pilot and rolling evaluations include an evaluative component whose findings are intended to inform decisions about a program's future direction.

In contrast to pilot and rolling evaluations, one-shot evaluations are not usually part of initial program design but become key components in the planning and revision stages. One-shot evaluations entail a one-time supplementary or emergency appraisal. Supplementary evaluation efforts are opportunistic attempts to provide extra data as a component of program evaluation design to inform the direction of monitoring systems. Emergency one-shot evaluation efforts are responses to legitimacy crises. Legitimacy crises are moments when system validity is called into question.

A central component of realizing policy legitimacy is rhetorical framing. In what follows, we show how evaluation protocols are informed by stated program objectives. Programs objectives are developed to address the putative concerns of surveillance stakeholders, yet evaluation findings often pose problems for stakeholders in one of two ways: first, they fall short of addressing all stated program objectives; second, they do not always yield empirical data that lend themselves to the realization of substantive and procedural legitimacy. When stakeholders encounter actual or potential problems with evaluation results, they rely on rhetorical framing techniques to make authoritative claims despite somewhat ambiguous or contrary results. In more critical cases of back talk, stakeholders are forced to adjust program objectives.

The routine rhetorical dimensions of program evaluation raise two key issues with respect to Guidelines generally and the IPC's role particularly. The first issue is whether Guidelines assist stakeholders in formulating evaluation techniques capable of measuring effectiveness. As we show, the Guidelines (and the IPC's interpretation of the Guidelines) encourage stakeholders to identify objectives and evaluate programs but they do not provide any guidance on how evaluations of objectives should be carried out. The second issue is whether the Guidelines provide a framework that enables the IPC to follow up on evaluation recommendations. As we show, stakeholders interpret and transmit evaluation results using rhetorical framing techniques with little accountability to the IPC. We use these findings to reflect on the status of the IPC as a privacy advocate in the final section of the article.

Sudbury's Eye in the Sky

The first evaluation policy we examine pertains to Sudbury's Eye in the Sky monitoring program. Sudbury's program was formally launched in 1997. Prior to the formal launch of the system, a four-month pilot project was initiated with the primary objective to reduce antisocial behavior. In actuality, the one-camera pilot project was operational for just one month. Systematic data on monitoring patterns, police interventions, or incidents of antisocial behavior were not collected, yet stakeholders reported a 10% reduction in antisocial behavior during the pilot phase (Greater Sudbury Police Services Board 2005). An absence of effective evaluation techniques capable of empirically measuring the pilot's objectives led stakeholders to rely on rhetorical framing strategies to bolster support for the camera.

When a three-camera program was formally launched in 1997 (and expanded in subsequent years), the list of program objectives also expanded: crime reduction, forensics, increasing actual and perceived public safety, and enhanced commercial activity. However, the program design did not entail any evaluation component until 1998, when the IPC received a complaint about the downtown cameras. Responding to the complaint, the IPC recommended an independent audit of the system to legitimate its continued operation. The recommendation was based on privacy commissioner Ann Cavoukian's (1998) best practices discussion document that eventually informed the content of the Guidelines (published in 2001). The Sudbury Police Service contracted the consulting firm KPMG to conduct a one-shot emergency evaluation to measure how program objectives were being met.

The evaluation aspired to assess all of Sudbury's program objectives. Drawing on police statistics and findings from interviews with police officers, as well as findings from surveys distributed to 58 people in the downtown area and interviews with business people, several conclusions were presented in the final report (Polani 2000). For example, the report claimed that 79% of the population of Sudbury and 98% of business owners support the cameras. The report also claimed that between 1996 and 1999, assaults and robberies declined by 38% and property crime by 44%. The report even presented comparative data from other Canadian cities to conclude that Sudbury's crime rates declined at a rate greater than communities of comparable size--findings that were reproduced in cities across the country for the next five years (Hier 2010).

Few surveillance stakeholders in Sudbury received the KPMG report as a validity measure attesting to the effectiveness of the cameras. Stakeholders realized that the data analyses were unreliable and that the findings were questionable. Still, when program objectives were not clearly met by the evaluation, stakeholders used the findings to rhetorically promote the system within and beyond Sudbury. Rhetorical framing happened in two main ways. First, stakeholders were keen to promote speculative claims presented in the report that 300 to 500 criminal offences were deterred by the cameras, that no displacement effect had occurred, that violent and property related crimes declined after the cameras were installed, and that the system was used to assist in locating missing children (see, e.g., Lions Club 2002). This set of rhetorical claims addressed program objectives pertaining to the crime-reduction benefits of the system. Second, stakeholders were keen to promote the claim that downtown Sudbury saved $600,000.00 to $800,000.00 direct and indirect monetary losses. This rhetorical claim addressed program objectives pertaining to the economic benefits of the system.

The KPMG report was published before the release of the IPC's Guidelines. Following the release of Guidelines in September 2001, the IPC approached stakeholders to request an explanation for how the system was justified by continuous or periodic review. The IPC specifically recommended a more focused external audit in light of the poor quality of the KPMG report. Stakeholders responded in two ways. First, they created an audit template that briefly responded to 38 questions posed by the IPC. The audit simply described system features. Second, stakeholders created a public survey. In 2004, 2,000 surveys were distributed to community members including downtown businesses, seniors, and secondary school students.

The results of the survey were not favorable. Of the 557 surveys that were returned, 387 (69%) came from students. Stakeholders therefore decided to analyze the data by age group to account for an anticipated skewing of the data. Among the findings were that 34% of respondents in age group of 14 to 25 did not know about the cameras and that nearly 50% did not feel safer with cameras--during daytime or nighttime. Despite survey back talk, stakeholders emphasized findings that respondents of 55+ years (17% of respondents) of age were supportive of the cameras. Stakeholders concluded that students' responses are unreliable and that some of them need to be educated about the purpose of the cameras.

The initial objective of Sudbury's monitoring program was to reduce antisocial behavior. Even before the formal launch of the program, the system relied on rhetorical framing to legitimize program objectives. When the objectives were expanded, it became even more difficult to legitimize the system in the absence of rhetorical framing. Stakeholders therefore sought policy legitimacy by adhering to recommendations forwarded by the IPC (and that were eventually institutionalized in Guidelines). In both instances, they were forced to negotiate back talk. In fact, the significance of rhetorical framing in Sudbury was only fully realized after the Sudbury Police Service posted the KPMG report on their website in 2000. As Anglo-Canada's first system, the Eye in the Sky became an important rhetorical resource for the promotion and evaluation of subsequent programs across Ontario.

London's Downtown Camera Project

The rhetorical influence of the KPMG report (and claims made by stakeholders in Sudbury) was strongly observed in the promotion, justification, and evaluation of Ontario's second monitoring program: London's Downtown Camera Project. The rhetorical influence on the Downtown Camera Project is interesting not least because of London's different evaluation structure. The 16-camera system was launched in November 2001 with the objectives of reducing crime; intervening in crimes in progress, forensics, enhancing commercial activity downtown; and enhancing public perception of downtown safety (crime reduction was the chief objective). To initially legitimate the focus on crime reduction, the Coordinating Committee for Community Safety--an ad hoc volunteer group that came together in the aftermath of a violent murder in downtown London--published a report citing claims made by stakeholders in Sudbury (prior to 2001) that crime rates in monitored areas of downtown Sudbury dropped 20.4% after the introduction of surveillance cameras, in turn enabling economic rejuvenation (Coordinating Committee for Community Safety 1999).

In contrast to Sudbury's initial reliance on a pilot project, however, stakeholders in London evaluated effectiveness on the basis of annual rolling evaluations that were presented to the City's Community and Protective Services Committee (CPSC)--a municipal oversight community composed of councilors and city staff. The annual reporting system was put in place after the IPC met with program stakeholders in July 1999, encouraging the city to take ownership of the program and to introduce regular evaluations.

The annual reports presented findings relevant to evaluation on impact on crime reduction, yet comparative data concerning reported crime statistics both inside and outside the area surveyed by cameras, both pre- and postcameras, provided little evidence that they were reducing or displacing crime. For instance, the program steering committee's annual evaluation of the Downtown Camera Project was submitted to the CPSC in February 2003; it revealed an increase of 25% in reported crime in the camera area between 2001 (precamera) and 2002 (postcamera). To explain the increase, the report invoked rhetorical claims present in the KPMG report: "Sudbury showed minor increases in offences in the first year. It was not until the second and third years that offences decreased considerably" (City of London 2003). (6) The report concluded with a recommendation (repeated in the 2004 report) that three full years of evaluation data are collected before decisions are made about crime reduction.

Subsequent annual evaluations contained a set of data collected by an officer assigned to "determine whether the cameras have had an effect in reducing number of occurrences in the area, if the cameras have simply pushed the incidents away from the intersections, or if they have had any effect at all" (City of London 2005). The 2005 report, for example, analyzed data specific to each individual intersection under surveillance and found that crime statistics remained consistent across pre- and postmonitoring periods. The report concluded that there was no demonstrable reduction in crime.

Based on data presented in the 2005 report (and subsequent reports), stakeholders changed program objectives from emphasizing crime prevention and reduction to forensics and investigation. Subsequently, program objectives were changed following a second round of IPC consultations. After the IPC published the Guidelines in 2001, they wrote to the city to present a full report of the program (structured around the Guidelines) and to specifically request an explanation for how the system was being regularly evaluated. Reframing of the objectives was necessary to strive for substantive legitimacy in lieu of supportive evaluation findings.

Hence, despite stakeholder's openness about the failure to demonstrate a statistical impact on crime, each annual report summary contained virtually the same sentence, "based on the results of this evaluation, it has been demonstrated that the Downtown Monitored Surveillance Camera Program continues to assist in enhancing community safety, crime prevention, and the desirability of the downtown in London." To bolster support for the latter claim in an effort to deal with evaluation back talk, three other forms of data are invoked. First, statistics pertaining to number of incidents reported (range: 100-196 per annum) and requests for footage (range: 25-109) are offered as evidence that cameras are effective. Second, each report contained anecdotal references to particular uses of cameras during or following incidents of crime or other issues. (7) Third, public perception of safety is assessed in some annual reports, once through a question distributed by London Neighborhood Watch in which 50% of respondents reported feeling safer since the cameras were installed (sample number not specified), and twice through reference to results of the Public Needs Survey (2002 and 2005) pertaining to perceptions of safety in the downtown. In other words, if statistics fail to show that a program meets an objective (e.g., reducing crime), alternative evidence may be substituted to substantiate the claim.

Hamilton's Monitoring Program

In contrast to Sudbury's pilot evaluation and London's rolling evaluation, the initial program design structure in Hamilton--the third city we analyze--contained no evaluation measure at all. Yet following interactions with the IPC that set off a series of attempts to strive for procedural legitimacy, the program incorporated several evaluation techniques. What is significant about Hamilton's evaluation strategies is that they were encouraged by the IPC but the latter did not assess and act on the evaluation results in a manner seen in Sudbury and London. Hamilton's one-time consultation with the IPC not only posed certain implications for the course of the program but also more importantly signified the beginning of the IPC's withdrawal from active and direct involvement with program stakeholders across the province (around 2004).

Early planning for Hamilton's six-camera system was initiated in 1998 when members of the business community learned about Sudbury's system. The program was designed by police with the core objectives of reducing crime and increasing economic revitalization. Evidence for the crime reduction and revitalization capabilities of video surveillance was based on rhetorical claims presented in the KPMG report.

As the Hamilton Police Services planned to activate the system in 2002, the IPC became involved. A meeting was held with stakeholders; the IPC stressed the importance of policy design. Chief among their recommendations was broad public consultation prior to launching the system. The public consultation (the first of two) represented a one-shot evaluation aimed at building procedural legitimacy.

The public consultation process entailed soliciting feedback through a telephone survey, media monitoring, and various communications directed to the police. It also entailed community consultation forums held at 12 locations across the city. At every consultation forum, a police officer described program goals and a panel of committee members that was appointed to evaluate the forums (and report back to the chief of police) spoke about the importance of video surveillance. Each forum concluded with a show of hands to indicate support for the program.

Despite the fact that the public forums were attended by only 259 people (with attendance ranging from a single person to 120 people; two forums alone accounted for 83% of all attendees), the evaluation committee reported that 77% of attendees supported the program. The committee did not report on back talk that emerged during consultations. For instance, at a forum held at the central library in downtown Hamilton (where nearly 100 people were in attendance), audience members began to challenge the ethics of city-street video surveillance. Frustrated by the lack of support for the camera initiative, a member of the evaluation committee walked out of the forum in protest (Morse 2002). Committee members subsequently dismissed the critical comments on the grounds that they came from an organized group of political science students affiliated with the local university. Combined with reported findings from a telephone survey that 84.3% of Hamiltonians believe that video surveillance is an effective technology and that 82.1% support the program, a six-camera pilot project was launched in 2004.

A main component of the pilot project entailed a Citizens Review Committee that was charged with reviewing statistical evidence pertaining to crime rates (robber, theft from auto, break and enter, assault, stolen autos, mischief) for the period prior to and during monitoring, as well as details of incident reports involving cameras. The Citizens Review Committee--composed of business people, the police, and the city--was put in place to continue striving for procedural legitimacy. Findings from the pilot program were presented in a 2005 mid-pilot report (Hamilton Police Services Board 2005) and a final report (Hamilton Police Services Board 2006).

The final report endorses the monitoring program based on three main pieces of evidence. First, findings from the pilot project suggested a large reduction in all measured crime rates, ranging from 15 (mischief) to 57% (stolen autos). Second, a further phase of community consultation forums held in 2006 suggested that 89% of attendees support the cameras. Third, examples of uses of cameras in policing-related incidents are given. However, abandoned in the final report are warnings presented in the midterm report that attribute positive effects of the cameras to the fact that the pilot project coincided with a "special emphasis placed on policing the core [that] contributed to the reductions" (Hamilton Police Services Board 2005). Specifically, the pilot project coincided with increased auxiliary foot and bike officers in and around the downtown, the deployment in the core of a new staff sergeant with 13 officers at his disposal, and a dedicated operation targeting the use of crack cocaine. Added to which, the midterm report indicated that similar reductions in crime rates occurred in areas adjacent to but not covered by the cameras. With these concerns in mind, the Citizens Review Committee emphasized findings from the consultations and concluded that broad public support provides adequate grounds to declare the pilot project successful. In other words, the initial objectives were crime reduction and economic revitalization, yet the program was rationalized based on the perceptions of a relatively small group of consultation attendees.

Thunder Bay's Eye on the Street

The final city that we analyze is Thunder Bay. Policy formation in Thunder Bay is significant not because of the kinds of direct IPC intervention into evaluation processes that we observed in other cities but rather the ways that stakeholders modeled their program based on the three other cities' program designs. In other words, the evaluation component--and thereby procedural legitimacy--of Thunder Bay's Eye on the Street was modeled on rhetorical politics in other cities.

In fall of 2000, business and police representatives visited Sudbury to observe the Eye in the Sky. The following year, members of the business community proposed a camera system to City Council. Council's endorsement was based on statistical data from Sudbury that appeared to attest to the efficacy of surveillance cameras in reducing crime and enhancing economic activity (Management Team 2001). The proposal identified four objectives: to increase perceptions of safety; to increase pedestrian traffic in the downtown cores; to reduce the potential for crime; and to revitalize commercial areas.

To bolster procedural legitimacy, an Implementation Committee was appointed to design an 11-camera system. The Implementation Committee presented a program design to Council in April 2001. The design was loosely based on promotional and operational materials pertaining to Sudbury's Eye in the Sky but did not entail clear operational protocol. As the proposal was being considered, the federal privacy commissioner, George Radwanski, publically challenged video surveillance in the city of Kelowna, British Columbia. It was only at this time that the Implementation Committee became aware of the IPC's Guidelines.

The federal privacy commissioner's challenge did not halt planning efforts in Thunder Bay, but it did encourage the Implementation Committee to structure the program around the Guidelines. In accordance with guideline #4's stipulation to hold broad public consultations, members of the Implementation Committee conducted public presentations at 13 locations around the city. This first attempt at public outreach (there is a second attempt that we comment on below) was based on the Hamilton Police Service's public forums. The results appeared promising to surveillance stakeholders. A reported 80 to 90% of residents of seniors' homes and a downtown condominium complex attended and estimated attendance for a Town Hall meeting ranged from 175 and 360 people. The Implementation Committee reported on two negative comments and approximately 300 positive ones (Management Team 2003). Equipped with data suggesting community support, the Implementation Committee returned to Council with a revised implementation proposal as well as a Code of Practice.

The Code of Practice included an official "Program Evaluation Model" calling for multiple forms of evaluation, including monthly audits, public surveys, annual reviews, police feedback, and anecdotal evidence sourced from operations staff. Between 2006 and 2007, stakeholders generated monthly statistics--coded interpretations of camera operator observations and requests for footage from police, together with which camera was involved--to demonstrate the use of cameras. The first set of evaluation findings were published in 2008, covering all data since inception of the program.

In the first published evaluation, stakeholders compared post- and precamera crime activity statistics, finding that crime activity increased in the south core whereas it decreased in the north core. These findings are rhetorically framed, evidenced by the contextual proviso provided in the evaluation analysis section: "since there are many factors within the community that impact these statistics and outcomes, it is difficult to draw conclusions about the program" (Management Team 2008). Only some of the cameras are used on a regular basis, and this raises questions about guideline recommendations to justify the use of cameras. (8) There are also claims about the deterrence capability of the cameras that are not substantiated by data: "although cameras in some locations appear to have not been required for camera evidence or used by operators to highlight events, the fact that a camera is located in a site is a deterrent" (Management Team 2008). The results of the statistical evaluation are thus less than favorable and not easily interpreted. Yet the conclusion includes a statement by the City of Thunder Bay endorsing the success of the program, stating that "... there are no recommendations by the Steering Committee of the Thunder Bay police service to change these locations at this time" (Management Team 2008).

To complement these statistical data, a supplemental evaluation aimed at gathering public feedback was conducted from 2005 to 2006. Stakeholders received 234 surveys from members of the public in camera areas (67.5% believed cameras increased convictions; 63.6% felt it improved feelings of safety). Surveys distributed to business owners by the Business Improvement Association received such a low response rate that the data were not included in the evaluation study. These survey results were interpreted as a legitimization of the program. The report concluded "it is recommended that as a result of the positive reaction to the program and the results to date, that the evaluation phase of the program be concluded and the Eye in the Street program continue" (Management Team 2008). Subsequent reports continue to detail police uses of cameras; reports from 2009 to 2010 conclude that no program changes are required. The conclusion of the report to Council cast the Eye on the Street in a favorable light, although the report notes that 13 percent of respondents believed that the cameras had no effects whatsoever.

Thunder Bay's program is important because it demonstrates the rhetorical efforts involved in establishing video surveillance systems and in interpreting evaluation results. Importantly, Thunder Bay pointed to the legitimacy of cameras in Sudbury, London, and Hamilton as a way of legitimizing the initiative. Stakeholders' efforts at gauging substantive legitimacy were inspired by Hamilton's experience in the context of the IPC's request for public consultations. Both Hamilton and Thunder Bay proceed from incomplete evaluation studies, apparently unimpeded by their inability to substantiate the core system objectives.

Rhetorical Frames and Evaluation Atrophy

Our analysis suggests that cities are often able to claim legitimacy in spite of problematic evaluation results through four main rhetorical frames. First, if statistics fail to show that a program meets an objective (e.g., reducing crime), an alternative measure is substituted to substantiate the claim. This can take one of two forms: qualitative reports of police uses of cameras to suggest that would-be crimes have been averted and public opinion data suggesting that people believe cameras reduce crime. In Hamilton and Thunder Bay, for instance, the reports point to public consultation data as demonstrating program success in lieu of persuasive and comprehensive statistical data. In addition, rhetorical gesture to crimes that have been averted is common in all evaluation reports that we analyzed. This use of alternative, proxy measures to substantiate claims about meeting core objectives is common in the evaluation reports.

Second, whereas evaluation reports focus on one or two objectives as evidence of efficacy, the stated program objectives are often more all-encompassing. The problem is a mismatch between stated project objectives and what the evaluations attempt to measure. This might be the result of over-ambitious rhetorical framing at the stage of program design, as all cities tend to focus on a single program goal (most often crime reduction) instead of the broader array of program objectives. The mismatch between stated project objectives and what the evaluations measure also connects to what we call evaluation atrophy, whereby cities begin to conduct less robust and comprehensive evaluations over time.

Third, cities sometimes engage in a form of methodological gerrymandering, whereby the process through which evaluation results are achieved is black boxed. The "black box" refers to unacknowledged causal assumptions that guide data interpretation (e.g., if crime rates decrease while cameras are present, then cameras "caused" the decrease). In Hamilton, for instance, other police initiatives targeting downtown crime are not accounted for in the presentation of evaluation results. Analysts may use opportunistic conjecture when interpreting unexpected or contradictory findings (e.g., reporting crime reduction as a deterrent effect, but crime increased as a detection effect). A major source of methodological gerrymandering is variability in interpretation of the IPC's Guidelines. Because the IPC's Guidelines do not offer standardized measures and operationalized criteria, the methodological variability and the lack of benchmarks results in diverse evaluation structures.

Fourth, stakeholders may supplement their original evidence base with references to evidence beyond the local setting. This was observed in Thunder Bay when stakeholders cited Sudbury and Hamilton evaluation and consultation data; in London when stakeholders relied on rhetorical claims appearing in the KPMG report; and in Sudbury when stakeholders invoked rhetorical claims about the scope of cameras in the United Kingdom. This is a rhetorical strategy in which Sudbury's KPMG report has played a key role. Ironically, the KPMG report resulted from the IPC's recommendation for stakeholders in Sudbury to commission a comprehensive, independent evaluation, but the rhetorical "success" of the study has assuaged the concerns of other communities who might otherwise have felt compelled to carry out their own, potentially more exhaustive, evaluations.

Evaluation Atrophy

Although four rhetorical frames were used in each city to bolster procedural legitimacy, efforts to conduct evaluations decline or cease altogether when external and internal pressures to maintain legitimacy are absent. Alterations to evaluation measures are critical because they play a central role in achieving substantive and procedural legitimacy. Most cities revise aspects of monitoring programs over time, making reactive changes to monitoring hours, technology, and funding mechanisms. However, there are several conditions under which cities may succumb to evaluation atrophy. In cities where pilot projects are introduced, the completion of pilots may signal the end of structured evaluation efforts. In cities where rolling evaluations are used, structured evaluation efforts can decline when external and internal pressures to maintain legitimacy are absent. The withdrawal of the IPC from actively meeting with program administrators has lessened the external pressure on surveillance stakeholders to demonstrate substantive and procedural legitimacy.

The circumstances of evaluation atrophy vary, involving institutional fatigue, management transfer, or communication breakdowns. Three cities, Sudbury, Hamilton, and Thunder Bay, either reduced or ceased evaluative efforts following the completion of pilot projects. In Sudbury, for example, the Sudbury Police Service struggled to attract the caliber and quantity of volunteers that they had anticipated. The results of this are two-fold: first, the cameras are less often used and have been reduced to a passive recording system, and second they are not fully evaluated. As a result, the profile of the Eye in the Sky diminished and advisory committee activity declined. The program manager currently conducts a scaled-back evaluation consisting of a periodic review of relevant incident reports and program logbooks.

Evaluation atrophy is also evident in Hamilton. Following the completion of Hamilton's pilot in 2006, the project was expanded and full management of the system was passed to the Hamilton Police Service. The officer who currently manages Hamilton's program has planned annual evaluations but has not conducted evaluations since 2006. Hamilton's program is currently being utilized as part of the city's Ministry of Community Safety and Correctional Services funded Addressing Crime Trends In Our Neighborhoods (ACTION) taskforce. The ACTION taskforce directly addresses many of the issues that cameras were intended for, such as reducing violent crime, drug-related crimes, and enhancing public safety. (9) According to the police spokesperson, decisions about how to evaluate the cameras in light of the emergence of the ACTION taskforce are yet to take place.

Thunder Bay has also deviated from their rolling evaluation model. A 2007 City of Thunder Bay corporate report containing annual evaluation results retroactively converts their rolling model into a pilot scheme: "[i]t is recommended that as a result of the positive reaction to the program and the results to date, that the evaluation phase of the program be concluded and the Eye in the Street program continue" (Management Team 2008). All reference to annual evaluation is removed from Thunder Bay's code of practice from 2007 onward. Subsequent corporate reports continue to provide police statistics relevant to the program, but public opinion data collection ceased in 2007.

The examples above suggest that evaluation can become irrelevant to camera surveillance programs when external and internal pressure to continually demonstrate legitimacy is relieved. A significant finding is that the IPC no longer provided external pressure generating regular and rigorous evaluations in Ontario cities after 2004 and 2005. London holds the distinction of adhering to its annual evaluation techniques since the program's inception. As Hier (2010) argues, London developed the most advanced governance structure (based on IPC intervention) and this is one reason why stakeholders continue to maintain their evaluation structure and continue and strive for substantive and procedural even though the IPC is no longer involved.

DISCUSSION AND CONCLUSION

We have provided an analysis of how four city-street video surveillance systems are evaluated. Our data demonstrate several problems with the current paradigm of evaluating video surveillance programs in Ontario. One problem is that monitoring programs are promoted, designed, and supported based on tacit assumptions about their efficacy that may not be substantiated or even assessed via evaluation. It is possible that support for monitoring programs would remain high if they were promoted solely on what appear to be their demonstrable merits (such as their ability to serve as a general policing tool). But by the time evaluation results show that cameras do little to reduce crime, increase safety, or enhance commerce, the question of their substantive legitimacy no longer has political currency. A second problem is that the Ontario IPC's Guidelines play an important role in procedural legitimacy, but they are interpreted with great variation, selectively followed, and often ignored. The ambiguity of the Ontario IPC's Guidelines and the absence of oversight render them ineffective in ensuring meaningful evaluation is carried out for the duration of monitoring projects. A third problem is the tendency to defer to video surveillance policy "successes" in other contexts, thereby bypassing or overruling local deliberative processes. This mode of achieving policy legitimacy--known as "policy laundering" (Hosein 2004)--risks a situation of infinite regress, whereby cities that legitimize their own surveillance practices through reference to policy success in other cities subsequently become putative exemplars for other cities. Given the shortcomings of current evaluations, below we present four scenarios concerning how evaluations might be revised and be improved.

There are four scenarios that arise related to the legitimacy problems identified in this article. The first scenario aligns with Haggerty's (2009) idea of a knife fight: remove the IPC's evaluation Guidelines. In the absence of a standardized template for formal evaluation procedures, cities will have to negotiate legitimacy on local terms. Cities may again be compelled to make the ethical case for surveillance as a technology that may or may not help address the problems that we have raised. However, removing formal evaluation measures will not necessarily lead to a rise in public engagement or critical attention directed at cameras. Nor will it curtail use of rhetorical frames; each city that experienced back talk when their statistical studies failed deferred to the authority of police officers' use of cameras to legitimize continued monitoring. The provision for evaluation does, at least, present opportunities for motivated critical interventions.

A second scenario would provide greater enforcement powers to the Ontario IPC and their counterparts in other Canadian provinces so that they could regulate camera surveillance systems effectively. In privacy advocacy circles, this is called giving privacy watchdogs some teeth. However, there are several barriers. First, the Guidelines would have to be more precise and enforceable. Second, the IPC would need to create special units dedicated to camera surveillance headed by designated inspectors. The IPC would also need to advertise this new unit to policing agencies across the provinces. Related, third, they would need to recognize video surveillance as a pressing privacy concern. At present time, cameras are nowhere near the top priority. Fourth, the criteria for program success would need to become clearly operationalized; privacy commissioners are not necessarily the people who have the expertise to complete this task.

A third scenario would be to formalize the role of academic adversaries within the process of forming rhetorical and action frames. Schon and Rein (1994) advocate a triadic approach to policy research, whereby academics are invited to collaborate with designers and policy practitioners to encourage reflection at all stages of policy design and execution. The reasons for a third party interlocutor are substantiated in our study, when one reflects on the significant role played by critical attention emerging in the case of London. Academics could mitigate against procedural and ethical problems by: (1) consulting with designers to discourage promoting programs based on difficult-to-substantiate claims; (2) advising on evaluation methods likely to produce meaningful data; (3) mediating and ensuring the continuance of dialogue between designers and practitioners; (4) ensuring back talk is acknowledged and acted upon; and (5) providing feedback to privacy commissioners on the effectiveness of guidelines.

A final scenario (overlapping with the others) is to place more emphasis on prospective policy evaluation. This is evaluation that happens before an organization adopts and implements a policy (see Mossberger and Wolman 2003). Prospective policy evaluation would require cities interested in city-street video surveillance to take the issue of efficacy seriously by studying previous evaluations and assessing whether video surveillance can remain a viable policy option given the lack of evidence pertaining to its usefulness in crime control and the rhetorical framing required to sustain its legitimacy.

References

Burns-Howell, T. and T. Pascoe. 2004. "Crime Prevention Evaluation: A Realistic Framework Based on Experience and Reality." Criminology and Public Policy 3(3):527-34.

Cavoukian, A. 1998. Video Surveillance: The Privacy Implications. Information and Privacy Commissioner of Ontario, IPC Practices #10. Toronto: Information and Privacy Commissioner of Ontario.

Cavoukian, A. 2007/2001. Guidelines for the Use of Video Surveillance Cameras in Public Places. Toronto, Ontario: Office of the Information and Privacy Commissioner of Ontario.

City of London. 2003. Annual Evaluation Report of the Downtown Monitored Surveillance Camera Project. Community and Protective Services Committee, Meeting Minutes, London, February 23. Released by the City of London's Manager of Records and Informations Services.

City of London. 2005. Annual Evaluation Report of the Downtown Monitored Surveillance Camera Project. Committee." CSPC Meeting, March 21. Document released by City of London's Manager of Records and Information Services. Report #2005-03-21.

Coordinating Committee for Community Safety. 1999. Police/Public Interaction Subcommittee: Report and Recommendations. Ontario: City of London.

Ekos Research Associates. 2004. Canadian Attitudes toward Crime Prevention. Final Integrated Report. Ottawa: Ekos Research Associates.

Gazso, A. and K. Haggerty. 2009. "Public Opinion about Surveillance in Post-9/11 Alberta: Trading Privacy for Security?" Pp. 141-59 in Anti-Terrorism: Security and Insecurity After 9/11, edited by S. Magnusson. Halifax: Fernwood.

Gill, M. 2003. CCTV. Leicester: Perpetuity.

Gill, M. and A. Spriggs. 2005. Assessing the Impact of CCTV. Home Office Research Study Number 292. London: Home Office.

Greater Sudbury Police Services Board. 2005. "Lion's Eye in the Sky." Powerpoint Presentation, April 11, Sudbury, ON. Document Released by Greater Sudbury Police Service.

Haggerty, K. 2009. "Methodology as a Knife Fight: The Process, Politics, and Paradox of Evaluating Surveillance." Critical Criminology 17:277-91.

Hamilton Police Services Board. 2005. CCTV Program--1st Year in Review (PSB 02-033c). Hamilton, ON: Hamilton Wentworth Police Services Board.

Hamilton Police Services Board. 2006. CCTV Program (PSB 02-033e). Hamilton, ON: Hamilton Wentworth Police Services Board.

Hay, C. 1995. "Mobilization through Interpellation: James Bulger, Juvenile Crime and the Construction of a Moral Panic." Social and Legal Studies 4(2):197-223.

Hier, S. 2010. Panoptic Dreams: Streetscape Video Surveillance in Canada. Vancouver: University of British Columbia Press.

Hier, S., J. Greenberg, K. Walby and D. Lett. 2007. "Media, Communication, and the Establishment of Public Camera Surveillance Programs in Canada." Media, Culture, and Society 29(5):727-51.

Hier, S. and K. Walby. 2011. "Privacy Pragmatism and Streetscape Video Surveillance in Canada." International Sociology 26(6):844-61.

Hosein, I. 2004. "The Sources of Laws: Policy Dynamics in a Digital and Terrorized World." The Information Society 20(3):187-99.

Innes, M. 2004. "Signal Crimes and Signal Disorders: Notes on Deviance as Communicative Action." British Journal of Sociology 55(3):335-55.

Lett, D., S. Hier and K. Walby. 2010. "CCTV Surveillance and the Civic Conversation: A Study in Public Sociology." Canadian Journal of Sociology 35(3):437-62.

Lions Club. 2002. "Fact Sheet." Document released by Greater Sudbury Police Force and Sudbury Metro Centre. Sudbury, Ontario.

Management Team. 2001. Corporate Report 2001.126. City of Thunder Bay, ON.

Management Team. 2003. Corporate Report 2003.246. City of Thunder Bay, ON.

Management Team. 2008. Corporate Report 2008.004. City of Thunder Bay, ON.

McDermott, R. 2007. "Huge Sinkhole Cripples London's Core." London Topic, October 31. Retrieved May 25, 2011 (http://www.londontopic.ca/article.php?artid=5612).

Morse, P. 2002. "Video Camera Public Hearings Find Opposition to Proposal." Hamilton Spectator, April 6, p. A03.

Mossberger, K. and H. Wolman. 2003. "Policy Transfer as a Form of Prospective Policy Evaluation: Challenges and Recommendations." Public Administration Review 63(4): 428-40.

Polani, O. 2000. "Evaluation of the Lion's Eye in the Sky Video Monitoring Project." Retrieved September 2, 2012 (http://wayback.archive.org/web/*/http://http://www.police. sudbury.on.ca/publications/reports/KPMG.pdf).

Ratcliffe, J.H. and T. Taniguchi. 2008. CCTV Camera Evaluation: The Crime Reduction Effects of Public CCTV Cameras in the City of Philadelphia, PA Installed During 2006. Philadelphia, PA: Temple University.

Ratcliffe, J.H., T. Taniguchi and R.B. Taylor. 2009. "The Crime Reduction Effects of Public CCTV Cameras: A Multi-Method Spatial Approach." Justice Quarterly 26(4): 746-70.

Schon, D. and M. Rein. 1994. Frame Reflection. New York: Basic Books.

Stone, D. 1988. Policy, Paradox and Political Reason. Glenview Illinois: Scott, Foresmen & Co.

The Surveillance Project. 2008. The Globalization of Personal Data Project: An International Survey on Privacy and Surveillance. Kingston, ON: Queen's University.

Wallner, J. 2008. "Legitimacy and Public Policy: Seeing beyond Effectiveness, Efficiency, and Performance." Policy Studies 36(3):421-43.

Welsh, B. and D. Farrington. 2004. "Surveillance for Crime Prevention in Public Space: Results and Policy Choices in Britain and America." Criminology and Public Policy 3(3):497526.

Welsh, B. and D. Farrington. 2005. "Evidence-Based Crime Prevention: Conclusions and Directions for a Safer Society." Canadian Journal of Criminology and Criminal Justice 47(2):337-54.

Welsh, B. and D. Farrington. 2009. Making Public Places Safer: Surveillance and Crime Prevention. Oxford: Oxford University Press.

Sean Hier, Department of Sociology, University of Victoria, Victoria, British Columbia, V8W3P5. E-mail: shier@uvic.ca

(1) The term rhetorical politics refers to the logic and assumptions that structure the utterances, writings, and other forms of rhetoric that policy makers use to explain and make the case for specific policies (see Schon and Rein 1994:32-36). Policy makers commonly invoke rhetorical frames: the persuasive use of story and argument in policy debates that are used to construct a policy problem.

(2) Privacy commissioners' guidelines are composed of a ser of recommendations designed to ensure that surveillance practices conform to Canadian, provincial, and municipal privacy protection law. Guidelines have played an important historical role in shaping how cities deploy and evaluate video surveillance cameras in Canada (for a detailed explanation, see Hier 2010).

(3) Emotive appeals commonly draw their power from signal crimes: the ways in which some crimes signify the dangers associated with certain places, groups, or activities (Hay 1995; Innes 2004). The promotion of Canadian monitoring programs frequently entails local signal crimes but a common referent is also the 1993 murder of Jamie Bulger in the United Kingdom (Hier et al. 2007).

(4.) Since 2005, we have investigated the dynamics involved in designing and using city-street video surveillance systems. We have conducted over 635 interviews with police officers, city councilors, business people, and numerous other stakeholders.

(5.) Compliance with both the content and spirit of Guidelines is voluntary rather than compulsory. The stipulations in Guidelines are based on municipal and provincial privacy protection law, but the recommendations themselves are advisory in nature.

(6.) This report also echoes some of the reasoning used in the KPMG report, as it hypothesizes that the 25% increase in reported crime may be attributable to the caraeras' detection capabilities.

(7.) The reports contain accounts from police officers and city officials that cameras have been useful in identifying criminals, alerting authorities about medical emergencies, and--in a well-documented case that enhanced the profile of London's cameras--responding to the sudden emergence of a potentially catastrophic sink-hole, engulfing an entire downtown intersection within seconds (McDermott 2007).

(8) Two cameras alone account for 484 of the 586 incidents recorded for the entire 2006.

(9) See http://news.ontario.ca/mcscs/en/2011/01police-and-province-working-togethere-to-rid-ontariostreets-of-guns-and- gangs.html.