Reflections on the "policy-relevant turn" in research.

Garvin, Theresa ; Lee, Renee Gravois

AS RESEARCHERS, WE FACE INCREASING PRESSURE TO MAKE OUR WORK policy-relevant. Some of the impetus comes from within the academy as we try to make our work more usable and useful; other pressure comes from funding agencies, which demand that policy relevance be addressed in proposals and reports. In this article, we challenge the assumption that this "policy-relevant turn" in research has only positive implications and reflect critically on how the demands for policy relevance can influence our work. Throughout the discussion, we identify some of the potential subtle influences that the policy-relevant turn may have in shaping decisions and activities at each stage of the research process.

Over the past decade, researchers have been increasingly asked to make research "policy-relevant" (Wilson, 2002; Auriat, 1998; Shahidullah, 1998). This is true not only for those of us working around issues of social justice, but also for social science researchers across multiple disciplines. Certainly, the focus on policy relevance is not completely new (see, e.g., journals such as Social Policy, Health Policy, Policy Sciences, and Journal of Public Policy and Marketing). However, recent years have seen interest in policy relevance become more "mainstream" in academia. For instance, concern for addressing policy relevance is increasingly found in disciplines such as political science (Candler, 2000; Lijphart, 2000; Hart, 1998), sociology (McDonald et al., 2000; Rehg, 2000; Schrader-Frechette, 2000), and economics (Erard and Ho, 2001; Block, 1999; Cameron and Ndhlovu, 1999; Turner et al., 1998). Likewise, policy relevance emerges as an issue in cross-disciplinary work, including immigration and migration studies (McDonald et al., 2000; Portes, 1999; Duncan, 1998), poverty studies (Korpi and Palme, 1998; Marcoux, 1998), criminology (Braga, 2001 ; Bridges et al., 1997), religious studies (Carroll, 2000), rural studies (Hodge and McNally, 1998), and ecology (Vaughan et al., 2001; Urquhart et al., 1998; Parsons and Daniel, 1988).

The current impetus for policy relevance comes from many quarters, including researchers themselves as they strive to ensure their work makes a difference (see, e.g., Baldwin, 2000). Further, commissioned work requires action-oriented outcomes; government and agency contracts demand concrete recommendations; and research agencies increasingly request that investigators identify policy relevance in proposals and final reports. This "policy-relevant turn" has crept into our work in a way that impels researchers to think carefully about the potential outcomes and use of our research and, in many cases, to specifically outline plans for communicating findings to communities, decision-makers, and policymakers in a language and format that make results "accessible." Endeavors such as participatory processes (Nichols, 2002; Mathie and Greene, 1997) and empowerment evaluation (Fetterman, 2001) attempt to include alternate voices and more applied outcomes in many research venues. Much of this work recognizes the importance of values in the conduct of research (House and Howe, 1999) and of ensuring ethical approaches in policy-relevant work (Neale et al., 2003).

The policy-relevant turn generally assumes that making research findings accessible and applicable in the "real" world will bring only benefits. Policy-relevant research provides funders and society with "bang for their buck" through solving problems and using research to "change the world." This is especially important as researchers and funders seek concrete ways for research to make a real difference in society and in the lives of the people with whom we work. However, as policy relevance becomes more mainstream in research, we ask: What are some of the potential dangers of the policy-relevant turn and how may it subtly change how investigators approach their work?

In this article, we invite researchers to think critically and reflectively about the influence of the policy-relevant turn in their research. The challenges of many other aspects of socially relevant work have received attention in recent years, including reflection on issues such as entering the field (Sixsmith et al., 2003; Wood, 2001), working with participants in a responsible manner (Corbin and Morse, 2003; Goodwin et al., 2003; Liberman, 1999), engaging in grounded theory (Charmaz, 2003; Goulding, 2002; Strauss and Corbin, 1998), addressing issues of power and representation (Kincheloe, 2001; Bishop, 1998), and representing our work to lay and expert communities (Tedlock, 2003). However, relatively little attention has been given to the increasing emphasis on policy relevance and what it might mean for the conduct of research--and research on social justice in particular.

When investigators consider issues of policy relevance, it is often at the end-stage of the research. Even funding agencies seem to consider communication and relevance to be an afterthought or "tagged-on" portion of the application process. However, we see the policy-relevant turn as permeating all stages of the research process--from conceptualizing the problem to collecting and analyzing data, as well as communicating findings. We are increasingly asking ourselves: How does the quest for policy relevance affect actions and decisions at every stage of the research process? How might the goal of policy relevance influence understanding of the phenomena under study?

We begin with a very large caveat: we believe strongly in the importance of making one's work applicable in the real world and beneficial to real people. We do policy-relevant, applied, social justice work. As graduate students a decade ago, we urged our team leaders to consider social justice issues, to be more responsive to community needs, and to engage in long-term relationships to ensure that results were moved into community action. It is from this position inside the paradigm of applied social research that we feel comfortable raising questions, sharing tensions we have experienced, and challenging our research colleagues to think critically about the implications of the policy-relevant turn.

In our own experiences, doing policy-relevant work has raised a number of challenges, such as acknowledging the different ways that groups construct and utilize knowledge (Garvin, 2001), crossing differences in national policies and contexts (Garvin and Eyles, 2001), and including lay researchers and community members in research partnerships (Lee et al., 1999; Garvin, 1995). We have encountered tensions negotiating between overarching social theories and the application of those theories "on the ground" and in communities (Castleden and Garvin, 2004; Lee et al., 1999; Wilson et al., 1999). We have identified a number of problems in communication between expert and lay groups (Lee and Garvin, 2003). Moreover, we have faced challenges in how to most effectively convey our findings so that our research is not only actionable (or policy-relevant), but also communicates effectively to and with multiple audiences, particularly those groups taking part in action research and using that work for social and political change (Lee et al., 2004; Wilson et al., 1999; Bristor et al., 1995; Garvin, 1995). In this context of doing policy-relevant research, we have begun to reflect critically on the assumption that making research more accessible and policy-relevant creates only benefits. Our article is written in this spirit of critical reflexivity, and we hope to engage other researchers as we identify and examine some of the tensions presented by the policy-relevant turn.

From the outset, we recognize that many influences discussed here are not new and, in fact, reflect ethical and methodological tensions that have existed for many years. Sociologists of science recognize that the very act of asking questions about the world is inherently social and political (Barnes et al., 1996; Salter, 1988; Latour, 1987; Latour and Woolgar, 1979). The questions researchers ask are products of who we are, where we live, and what we believe (Barnes et al., 1996; Ravetz, 1996). Investigators have long struggled over the role that values play in research and, more recently, have begun to question the traditional enlightenment vision of "truth." As a result, researchers now question whether the subject-object distinction can and should be applied to the world around us (Tesh, 1988; DeMartini, 1982).

At the same time, traditional models of policymaking continue to see science as the truth-seeker and creator of objective knowledge (Yanow, 1996; Heineman et al. 1990) and policymakers as political actors whose driving motivation is the manipulation of power (Stone, 1997; Majone, 1989). Yet these distinctions are becoming increasingly blurred (Albaek, 1995; Tesh, 1988; DeMartini, 1982). Calls for broadening our thinking about the nature of research--such as calls for new frameworks that go beyond the traditional dichotomies of subject/object,fact/value, basic/applied, and science/policy--are on the rise (see, e.g., Stokes, 1999).

We therefore believe that the policy-relevant turn provides a timely opportunity for researchers to stop and consider the subtle challenges of policy relevance and our responsibility to ensure that information is responsibly translated into policy. For example, research has provided strong evidence about the link between sun exposure (through tanning) and increased risk of developing skin cancer (see, e.g., Mukhtar and Bickers, 1993). However, applying this knowledge in the policy arena is not clear and straightforward and may even raise social problems (Ravetz, 1996). The apparent policy conclusion would be to reduce the

incidence of tanning in the general population, thus reducing skin cancer incidence. But how could we achieve this? Tanned skin remains a status symbol in many Western societies and therefore is a social construction (Wilson et al., 1999). Some have recommended changing tanning behavior by warning of the "dangers" of dark skin. However, in a multicultural society, implying that dark skin is less desirable is problematic since most countries of our globalized world include peoples of many shades and colors (Ibid.). Thus, we must approach the development of "policy-relevant" findings cautiously, with recommendations that take into account social values and social context. Generic meta-theories and decontextualized policy recommendations will not likely provide workable solutions in this era of policy relevance.

In the following discussion, we offer critical reflections on how the increased emphasis on policy relevance may (or may not) influence how researchers work. From our own experience, we examine the key stages of the research process and identify some important ways that the demands for policy relevance can affect decisions and activities at each stage. Admittedly, we ask many more questions than we are able to answer. Likewise, our discussion is not exhaustive, nor is it meant to be. Rather, we present these thoughts as a springboard for discussion and challenge researchers to engage in their own critical reflection on the assumptions, tensions, and implications of the policy-relevant turn in research.

Formulating the Research Problem with a Policy-Relevant "Gaze"

Therborn (1980) argues that ideology operates in society through helping to define what exists, what is good, and what is possible. A key insight of this work is that boundaries can be set through ideology. In the case of research, bound aries are drawn around what topics are deemed important to study, and what theories, methods, and epistemological foundations are acceptable and valued (Barnes et al., 1996): the "lens" of the research approach. This concept is not novel. Feminists have long discussed the problem of the male "gaze" in research that delineates boundaries by placing women's experiences outside the scope of academic work (Rose, 1993). Thus, how we approach the research--the "gaze" we bring--shapes how we frame research problems, how we ask questions, and what answers we permit ourselves to find.

We wonder, therefore, whether the policy-relevant turn has influenced our gaze. And we question whether that policy-relevant gaze might restrict the boundaries of our investigations and how we view the world. If our gaze leads us to seek only policy-relevant data and findings, what might we overlook? What questions might we not ask? Similarly, how might the way in which desired policy-relevant outcomes are defined influence the questions asked and the findings deemed important? For instance, if the goal of a particular welfare reform policy is defined solely as decreasing the number of people on welfare, other important research questions, such as the education, job, and life opportunities facing individuals post-welfare, may be overlooked. In other words, might the quest for policy relevance hinder our attempts to "think outside the box"?

Moreover, if our gaze leads us to search only for solutions and repairs, might we position ourselves to miss the unexpected? We may miss some of the most interesting parts of the work, as well as subtle yet critical power relations, in our hurry to solve problems. More dangerously, we may simply pass over the more challenging research questions because they may not seem policy-relevant at a particular place or time.

Developing the Research Project Within Institutional Constraints

Researchers have multiple and overlapping identities, and we are responsible for answering to multiple constituencies within specific sets of institutional demands. Although we may feel compelled to do socially and policy-relevant work, we also have to please chairs, deans, tenure and review committees, funding agencies, communities, and, increasingly, corporate funding sources. In developing research projects (at the design and operationalization stages), how does the desire for policy relevance influence our choice of phenomena to study? Our choice of study sites, participants, and communities? Further, how do institutional constraints affect our location as the researcher?

An example of such tensions is that of short funding cycles--a persistent though not new tension. Most funding agencies have cycles of two to four years. Although this timeframe may facilitate the generation of up-to-date research, it also limits the types of issues we examine. With policy relevance in mind, might funding time constraints encourage us to explore only questions that can be answered in a two- to four-year period? Might this state of affairs further threaten the acceptance of longer-term research agendas? Issues such as relations over the lifespan, institutional power relations, community dynamics, generational changes, and other long-term social and behavioral investigations have long struggled to find a place in this world of short timelines and compressed funding cycles. We fear that, in the move to increased policy relevance, this work may once more be left by the wayside because it may not be amenable to the action-oriented results expected within the confines of short-term projects. How does the funding cycle, coupled with the need to prove policy relevance, influence how we undertake research? How does it influence who we engage? In addition, how might funding cycles, and similar institutional constraints, work for or against the researcher's desire for contributing to real change and social justice?

A second example of institutional restraint involves how a researcher's location in the project can be determined, a priori, based on policy relevance. Academics generally walk a fine line between maintaining credibility in an academic milieu (via "distance" and "scientific dissociation") and advocating for social or political change in the name of policy-relevant work (being an "activist"). When we become politically invested in research, we risk undermining the very academic credibility that grants us the expertise to do the work in the first place. Expanding the researcher's role to one of political activist and policy influencer shifts us into a vast gray area rife with contradictions--as the identities of researcher and activist do not map onto one another without much difficulty and tension within academic institutional life (Fine and Weis, 1998). It seems that researchers--not only during fieldwork, but also at every stage of a project's life--must choose between the opposing stances of scientific dissociation and political investment. With the call for policy relevance, we are increasingly drawn into taking such a stand, yet the academic reward system remains based on the assumption of clinical dissociation. How neutral are we when making recommendations? Brandon (1984) argues that policymaking is, by definition, a moral endeavor requiring value statements, and such a stand conflicts directly with the presumption of dissociation and neutrality on which the academy is structured.

Constraints of funding cycles and conflicts inherent in a researcher's "position" between academia and policymaking are just two examples of how institutional structures can and do influence research. As the demands for policy relevance increase, we wonder, will the institutional structures in which we operate shift as well? For example, will tenure and review committees recognize and validate our membership on advisory committees and policy councils? Such work can take as much, if not more, time than the research project itself. We speculate that such acknowledgement is unlikely in the short term as institutional change occurs slowly and incrementally (Pal, 1997; Sabatier and Jenkins-Smith, 1993; Lindblom, 1959). So we are left with the question: How will researchers negotiate the tension between institutional constraints and the demands of the policy-relevant turn?

Doing the Research with Policy Relevance in Mind

The tensions of policy relevance extend beyond the early stages of project conceptualization and design, and into the conduct of the research itself. How does the goal of policy relevance influence researchers' daily decisions on how to do research in the field? How does it change the contradictions and complexities of fieldwork? Day-to-day decisions on how to engage the research community, who has power to set research agendas, and which knowledge is deemed important to the research project can all be influenced by the goal of developing workable, policy-relevant recommendations at the end.

The policy-relevant turn can also affect our research relationships in the field. For example, when engaging in grounded, community-based work, how does the goal of policy relevance affect boundary-setting between researcher, participant, and policymaker? This issue is especially important in terms of knowledge production. Traditional models of research assume a simple and linear transfer of knowledge from expert (researcher) to layperson (especially in health, see Lee and Garvin, 2003). Yet we suggest that reversing that pattern and transferring lay knowledge to researchers and policymakers is just as important in the goal of constructing workable policy responses. How does the quest for policy relevance affect researchers as we navigate such tensions inherent in the process of co-constructing knowledge? Perhaps one advantage of the policy-relevant turn is to move us away from the position of expert and toward sharing expertise, experience, and knowledge. In other words, perhaps policy relevance makes us more open to the idea that researchers are not the only experts--our participants are also experts.

As we are all aware, the very process of undertaking fieldwork changes the "field." Therefore, what do researchers do when the policy recommendations offered at the close of a project are already underway, facilitated by the process of the research itself? For example, in one of our projects, we explored women's health experiences in an underserved area. The process of bringing groups of women together provided the venue for them to mobilize, and they began to work together for greater health care access. The project therefore resulted in gains for the women and the community. In the process, however, a primary policy-relevant recommendation--the need for action groups in the community--was negated, as action groups had already begun work. In such cases, researchers face the tension of how far to go to ensure that recommendations are novel (often considered desirable) versus simply confirmation of actions already in progress (often considered less desirable). Even more difficult is knowing where to draw the line in terms of judging the effect of the research. This challenge is particularly important to those involved in evaluation work because where we draw the boundaries of "effects" helps to define the success of the project. Challenges such as these are not unique to the conduct of research on social justice issues, but are applicable to most social research. Therefore, as we conduct research with policy relevance in mind, we must continuously consider how the policy-relevant turn influences our activities and relationships in the field.

Interpreting Results for Policy Relevance

Policy relevance not only influences how and what we study; it also affects our interpretation of data. When sifting through thousands of pages of transcripts, or hundreds of social and economic variables, how do we make decisions on dividing and classifying findings? Do our evaluations of which findings are policy-relevant (or not) render our interpretation any more or less thorough? Any more or less credible? Further, what voices gain credibility in the interplay of power and knowledge (Foucault, 1980) when expert opinions (those of the researcher) decide what is or is not policy-relevant for decision-makers and the general public? What voices might we silence in the process of such judgments? To what extent can we overcome these distinctions through granting greater power to participants to have a voice in interpreting results? Moreover, we must constantly consider: Who does the interpretation benefit?

One key danger in data analysis and interpretation is the temptation to search only for what is lacking, with the purpose of "filling the gap" with required policy recommendations. The danger of this interpretive lens is that it encourages a focus on deficiencies. By centering on what is not, rather than what is, we run the risk of painting as victims the very people we wish to assist and empower. Might looking for deficits during the analytical process lead us to overlook some of the strengths, resources, and capabilities that people and communities possess (Lee et al., 1999)? Perhaps we even close off our thinking about solutions, or the kernel of a solution, that may already exist within the people and places we explore. By implicitly adopting an analytical strategy focused on deficits, what other findings do we potentially overlook? While "fixing" those problems amenable to policy solutions is an important and worthy goal, we must be careful not to narrow our interpretive approach to searching out only problems, lest we deny or discount other findings, nuances in the data, and/or contextual influences such as culture (Morse, 2001). We challenge our research colleagues to ask: How might our interpretive analyses be shaped by the imperatives of policy relevance?

Communicating Policy-Relevant Findings

A key feature of the policy-relevant turn is the communication of findings to decision-making and lay audiences. No more is it sufficient to communicate results simply through presentations and papers geared to academic audiences. Yet what happens in the knowledge transfer process when researchers parcel out only those ideas deemed "relevant" for public or policy consumption? What assumptions do researchers make in determining relevancy to particular audiences? Might investigators omit important findings in the attempt to communicate in an accessible manner? Do we risk oversimplification? Further, we may even present only those findings that appear actionable, and relegate the more difficult or uncomfortable findings to the background.

When faced with several quotations about a given theme, how do we select the quote to present to policymakers to communicate the situation? On what criteria do we choose? Which chart or graph best represents our statistical data? In qualitative research, the impetus of policy relevance might encourage us to select the most emotionally compelling quotations. But such examples might not be the most articulate. What is the result of this choice in terms of how we represent the community we study to the outside world? If we present original voices and texts with grammatical and language differences, do we run the risk that policymakers may view our participants as less than credible or deserving? By focusing on statistical relations identifying those with "less" or "more," do we risk decontextualizing individuals and communities? Perhaps we turn them into "case studies" in order to provide examples that audiences can relate to and understand. Perhaps we employ other representational strategies (such as ideal types) to communicate our findings. What are the implications of these choices? Do some of our representations deny the agency of individuals and/or the social context? Although issues of representation have received much attention, the policy-relevant turn provides subtle new implications to consider. So again we ask: What might be lost in the process of determining how to communicate and which findings to communicate (or not)?

Finally, by focusing only on issues deemed actionable (or amenable to intervention), do we sometimes present only those informants' needs that we think can be solved within the confines of current institutional and policy structures? Perhaps we even cherry-pick those needs perceived as easy, quick, or inexpensive to address? Such decisions may shortchange the wide-ranging breadth of participants' needs, the more complex needs that might be more difficult to solve, and a number of worthy initiatives that fall outside the scope of current policy imperatives. As a result, might limiting the scope of what we decide to communicate in turn limit the potential for positive change?

A Conclusion, but Not an End ...

The research we do is challenging and demands constant reflection and reevaluation. Our discussion here presents some, though clearly not all, of the possible tensions that persist in research, and questions the added demands of the policy-relevant turn. We pose many questions about the influence of policy relevance and ask whether and how it influences our work. We would like to answer "no" to these questions; to deny that policy relevance has crept into our work in a way that influences how and why we do what we do. In reality, however, we think that we would answer "yes" to at least some of our own questions.

The goals of the policy-relevant turn are laudable. Research should help our society, our communities, and individuals. Our work should be applicable, useful, and helpful to advancing policy in the public interest. And money spent on research (especially public monies) should show tangible, policy-relevant results that can be applied in the real world. In an era of globalization and rationalization of economic interests, funders are increasingly requiring evidence of "money well spent." In this article, we have identified some of the tensions and challenges we have faced in our own research, and have raised many questions about how the usual tensions of research might be further influenced by the quest for policy relevance. At each stage of our projects, we now question the influence of the policy-relevant turn, and we challenge our research colleagues to do the same. This article, we hope, will spur a discussion and critical evaluation of the advantages, disadvantages, and subtle implications of the increasing policy-relevant demands in research. Only through critical self-reflection and open discussion can we address the challenges and tensions researchers face in this newly developed research milieu.

REFERENCES

Albaek, E. 1995 "Between Knowledge and Power: Utilization of Social Science in Public Policymaking." Policy Sciences 28: 79-100.

Auriat, N. 1998 "Social Policy and Social Enquiry: Reopening Debate." International Social Science Journal 50,2: 275-287.

Baldwin, S. 2000 "Interactive Social Science in Practice: New Approaches to the Production of Knowledge and Their Implications." Science and Public Policy 27,3: 183-192.

Barnes, B., D. Bloor, and J. Henry 1996 Scientific Knowledge: A Sociological Analysis. Chicago: University of Chicago Press.

Bishop, R. 1998 "Freeing Ourselves from Neo-colonial Domination in Research: A Maori Approach to Creating Knowledge." Qualitative Studies in Education 2: 199-219.

Block, S.A. 1999 "Agriculture and Economic Growth in Ethiopia: Growth Multipliers from a Four-Sector Simulation Model." Agricultural Economics 20: 241-252.

Braga, A.A. 2001 "The Effects of Hot Spots Policing on Crime." AAAPSS 578:104-125.

Brandon, W.P. 1984 "Public Policy as the Continuation of Moral Philosophy by Other Means." Policy Studies Review 4: 60-70.

Bridges, L., J. Hodgson, M. McConville, and A. Pavlovic 1997 "Can Critical Research Influence Policy? A Response to Max Travers." British Journal of Criminology 37,3: 378-382.

Bristor, J.M., R.G. Lee, and M.R. Hunt 1995 "Race and Ideology: African American Images in Television Advertising." Journal of Public Policy and Marketing 14: 48-59.

Cameron, J. and T.P. Ndhlovu 1999 "Keynes and the Distribution of Uncertainty: Lessons from the Lancashire Cotton Spinning Industry and the General Theory." Review of Social Economy 57,1.

Candler, G.G. 2000 "The Professions and Public Policy: Expanding the Third Sector." International Political Science Review 21,1: 43-58.

Carroll, J.W. 2000 "Reflexive Ecclesiology: A Challenge to Applied Research in Religious Organizations.'" Journal for the Scientific Study of Religion 39,4: 545-557.

Castleden, H. and T. Garvin 2004 "Therapeutic Landscape as Sacred Land: Wolf in Sheep's Clothing?" R. Riewe (ed.), Borders and Boundaries. Winnipeg, Canada: Aboriginal Issues Press, University of Manitoba. In press.

Charmaz, K. 2003 "Grounded Theory: Objectivist and Constructivist Methods." N.K. Denzin and Y.S. Lincoln (eds.), Strategies of Qualitative Enquiry. Thousand Oaks: Sage.

Corbin, J. and J.M. Morse 2003 "The Unstructured Interactive Interview: Issues of Reciprocity and Risks When Dealing with Sensitive Topics." Qualitative Inquiry 9,3: 335-354.

DeMartini, J.R. 1982 "Basic and Applied Sociological Work: Divergence, Convergence, or Peaceful Co-existence? The Journal of Applied Behavioral Science 18,2: 203-216.

Duncan, H.W. 1998 Policy Research: The Metropolis Experiment. Ottawa, ON: Citizenship and Immigration Canada.

Erard, B. and C.C. Ho 2001 "Searching for Ghosts: Who Are the Nonfilers and How Much "fax Do They Owe?" Journal of Public Economics 81: 25-50.

Fetterman, D.M. 2001 Foundations of Empowerment Evaluation. Thousand Oaks: Sage.

Fine, M. and L. Weis 1998 The Unknown City: Lives of Poor and Working-Class Young Adults. Boston: Beacon.

Foucault, M. 1980 Power/Knowledge: Selected Interviews and other Writings, 1972-1977. Brighton: Harvester Press.

Garvin, T. 2001 "Analytical Paradigms: The Epistemological Distances Between Scientists, Policymakers and the Public." Risk Analysis 21,3: 443-455.

1995 "We're Strong Women: Building a Community-University Research Partnership." Geoforum 26,3: 273-286.

Garvin, T. and J. Eyles 2001 "Public Health Responses for Skin Cancer Prevention: The Policy Framing of Sun Safety in Australia, Canada and England." Social Science and Medicine 53: 1175-1189.

Goodwin, D., C. Pope, M. Mort, and A. Smith 2003 "Ethics and Ethnography: An Experiential Account." Qualitative Health Research 13,4: 567-577.

Goulding, C. 2002 Grounded Theory: A Practical Guide for Management, Business and Market Researchers. Thousand Oaks: Sage.

Hart, P. 1998 "Preventing Groupthink Revisited: Evaluating and Reforming Groups in Government." Organizational Behavior and Human Decision Processes 73,2-3: 306-326.

Heineman, R.A., W.T. Bluhm, S.A. Peterson, and E.N. Kearny 1990 The World of the Policy Analyst: Rationality, Values and Politics. Chathan, NJ: Chatham Publishers.

Hodge, I. and S. McNally 1998 "Evaluating the Environmentally Sensitive Areas: The Value of Rural Environments and Policy Relevance." Journal of Rural Studies 14,3: 357-367.

House, E.R. and K.R. Howe 1999 Values in Evaluation and Social Research. Thousand Oaks: Sage.

Kincheloe, J.L. 2001 "Describing the Bricolalge: Conceptualizing a New Rigor in Qualitative Research." Qualitative Inquiry 7,6: 679-692.

Korpi, W. and J. Palme 1998 "The Paradox of Redistribution and Strategies of Equality: Welfare State Institutions, Inequality and Poverty in Western Countries." American Sociological Review 63,5: 661-587.

Latour, B. 1987 Science in Action. Cambridge, MA: Harvard University Press.

Latour, B. and S. Woolgar 1979 Laboratory Life: The Construction of Scientific Facts. Thousand Oaks: Sage.

Lee, R.G. and T. Garvin 2003 "Moving from Information Transfer to Information Exchange in Health and Health Care." Social Science and Medicine 56,3: 449-464.

Lee, R.G., J.L. Ozanne, and R.P. Hill 1999 "Improving Service Encounters Through Resource Sensitivity: The Case of Health Care Delivery in an Appalachian Community." Journal of Public Policy and Marketing 18,2: 230-248.

Lee, R.G., V.A. Taylor, and R. McGetrick 2004 "Toward Reducing Youth Exposure to Tobacco Messages: Examining the Breadth of Brand and Non-Brand Communications." Journal of Health Communication. Forthcoming.

Liberman, K. 1999 "From Walkabout to Medication: Craft and Ethics in Field Inquiry." Qualitative Inquiry 5,1 : 47-63.

Lijphart, A. 2000 "Definitions, Evidence and Policy." Journal of Theoretical Politics 12,4: 425-431.

Lindblom, C.E. 1959 "The Science of 'Muddling Through.'" Public Administration Review 19: 79-88.

Majone, G. 1989 Evidence. Argument and Persuasion in the Policy Process. New Haven: Yale University Press.

Marcoux, A. 1998 "The Feminization of Poverty: Claims, Facts, and Data Needs." Population and Development Review 24,1 : 131-139.

Mathie, A. and J.C. Greene 1997 "Stakeholder Participation in Evaluation: How Important Is Diversity?" Evaluation and Program Planning 17: 299-304.

McDonald, D.A., L. Zinyama, J. Gay, E de Vletter, and R. Mattes 2000 "Guess Who's Coming to Dinner: Migration from Lesotho, Mozambique and Zimbabwe to South Africa." IMR 34,3: 813-841.

Morse, J.M. 2001 "The Cultural Sensitivity of Grounded Theory." Qualitative Health Research 11,6: 721-722.

Mukhtar, H. and D.R. Bickers 1993 "Environmental Skin Cancer: Mechanisms, Models and Human Relevance." Cancer Research 53: 3439-3442.

Neale, J., J. Owen, and D. Small 2003 "Encouraging the Use of Codes of Behaviour in Evaluation Practice." Evaluation and Program Planning 26,1: 29-39. At http://dx.doi.org/10.1016/S 0149-7189(02) 00083-6. Accessed: June 12, 2003.

Nichols, L. 2002 "Participatory Program Planning: Including Program Participants and Evaluators." Evaluation and Program Planning 25,1: 1-14. At http://dx.doi. org/10.1016/S 0149-7189(01) 000,14-1. Accessed June 10, 2003.

Pal, L.A. 1997 Beyond Policy Analysis: Public Issue Management in Turbulent Times. Scarborough: ITP Nelson.

Parsons, R. and T.C. Daniel 1998 "Assessing Visibility Impairment in Class I Parks and Wilderness Areas: A Comparison of Policy-Relevant Methods." Society and Natural Resources 1,3: 227-240.

Portes, A. 1999 "Models and Realities: The Consequences of Immigration." Contemporary Sociology 28,4: 387-390.

Ravetz, J.R. 1996 Scientific Knowledge and Its Social Problems. London: Transaction Publishers.

Rehg, W. 2000 "Critical Science Studies as Argumentation Theory: Who's Afraid of SSK?" Philosophy of the Social Sciences 30,1 : 33-48.

Rose, G. 1993 Feminism and Geography. Minneapolis: University of Minnesota Press.

Sabatier, PA. and H.C. Jenkins-Smith 1993 Policy Change and Learning: An Advocacy Coalition Approach. San Francisco: Westview Press.

Salter, L. 1988 Mandated Science: Science and Scientists in the Making of Standards. Boston: Kluwer.

Schrader-Frechette, K. 2000 "Radiobiological Hormesis, Methodological Value Judgments and Metascience." Perspectives on Science 8,4: 367-379.

Shahidullah, S.M. 1998 "Useful Sociology: Can Sociological Knowledge Be Valuable for Policymaking?" International Journal of Sociology and Social Policy 18,1:86-112.

Sixsmith, J., M. Boneham, and J.E. Goldring 2003 "Accessing the Community: Gaining Insider Perspectives from the Outside." Qualitative Health Research 13,4: 578-589.

Stokes, D. 1999 Pasteur's Quadrant: Basic Science and Technological Innovation. Washington, D.C.: Brookings Institute.

Stone, D. 1997 Policy Paradox: The Art of Political Decision Making (Second Edition). New York: Kluwer.

Strauss, A. and J.M. Corbin 1998 Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory'. Thousand Oaks: Sage.

Tedlock, B. 2003 "Ethnography and Ethnographic Representation." N.K. Denzin and Y.S. Lincoln (eds.), Strategies of Qualitative Enquiry. Thousand Oaks: Sage.

Tesh, S.N. 1988 Hidden Arguments: Political Ideology and Disease Prevention Policy. New Brunswick, NJ: Rutgers University Press.

Therborn, G. 1980 The Ideology of Power and the Power of Ideology. London: NLB.

Turner, R.K., W.N. Adger, and R. Brouwer 1998 "Ecosystem Services Value, Research Needs and Policy Relevance: A Commentary." Ecological Economics 25: 61-65.

Urquhart, N.S., S.G. Paulsen, and D.P. Larson 1998 "Monitoring for Policy-Relevant Regional Trends over Time." Ecological Applications 8,2: 246-257.

Vaughan, H., T. Brydges, A. Fenech, and A. Lumb 2001 "Monitoring Long-Term Ecological Changes Through the Ecological Monitoring and Assessment Network: Science-Based and Policy Relevant." Environmental Monitoring and Assessment 67,1-2: 3-28.

Wilson, K., T. Garvin, and C. McMullan 1999 "The Bronzed Aesthetic: The Social Construction of Women's Tanning." M. Denton, M. Hadjukowski-Ahmed, M. O'Connor, and I.U. Zeytinoglu (eds.), Women's' Voices in Health Promotion. Toronto: Canadian Scholar's Press: 204-217.

Wilson, W.J. 2002 "Expanding the Domain of Policy-Relevant Scholarship in the Social Sciences." Political Science and Politics 35, 1: 14.

Wood, W.W. 2001 "Rapport Is Overrated: Southwestern Ethnic Art Dealers and Ethnographers in the 'Field.'" Qualitative Inquiry 7,4: 484-503.

Yanow, D. 1996 How Does a Policy Mean? Interpreting Policy and Organizational Actions. Washington, D.C.: Georgetown University Press.

THERESA GARVIN is Director of the Community, Health and Environment Research Centre at the University of Alberta (Department of Earth and Atmospheric Sciences, ESB 1-26, University of Alberta, Edmonton, Alberta, Canada, T6G 2E3; e-mail: Theresa.Garvin@ualberta.ca). Her research examines the role of scientists, policymakers, and the public ira negotiating contentious and complex environment and health issues. As a social scientist situated in the Faculty of Science, Dr. Garvin works closely with other social scientists, as well as with researchers in environmental and biological sciences, government officials and policymakers, and community groups and nongovernmental organizations. RENEE GRAVOIS LEE is Associate Professor of Marketing and Advertising at Quinnipiac University (Department of Marketing and Advertising, School of Business, Quinnipiac University, 275 Mount Carmel Avenue, Hamden, Connecticut, 06518; e-mail: Renee.Gravois.Lee@quinnipiac. edu). Her research interests include public policy issues in marketing and health care, health care delivery for at-risk populations, social marketing, and qualitative research methods.