Problematising aspects of evidence-based policy: an analysis illustrated by an Australian homelessness policy 1985-2008.

Fopp, Rodney

Introduction

The past few decades have witnessed growing international demand for policies to be based on evidence (Banks 2009a; 2009b: Head 2010; Pawson 2006). While there has been some modification of this objective over time, at least in the academic literature (Nutley et al. 2013; Marston & Watts 2003), in the public domain policies continue to rely on this justification because allegedly they avoid ideology and normative values. Instead, policies are purported to be firmly grounded in fact and the relevant science. Thus, in Australia 'evidence-based policy', or EBP (O'Dwyer 2004: 2) is advocated by politicians, bureaucrats, and policy developers, including parts of the non-government sector, which also invokes it in order to urge policy responses to issues of concern. As used by its staunch defenders, the need for EBP is axiomatic, and the adduced evidence is regarded as neutral and therefore uncontestable. While agreeing that policies are ideally informed by sound data (such as about the extent and characteristics of the target group, and the personal or social 'need'), the following article focuses on some aspects of the contested nature of social scientific research, and critically evaluates specific features of the popular notion of evidence-based policy.

The demand for evidence-based policy extends almost universally to policies in public health, and increasingly in education, employment, urban affairs, community development, and the vast array of policies drafted under the rubric of 'welfare', including aspects of housing policy and homelessness (Australian Government 2009; Jones & Seelig 2005; O'Dwyer 2004). In Australia the term EBP remains popular in discourse amongst the commentariat, the media, and politicians. As attested by the evaluation requirement built into the Supported Accommodation Assistance Program, policies for homeless people in Australia have purportedly been based on evidence. Further, the demand for evidence-based policy regarding homelessness is epitomised by three more recent documents published by the Australian Federal Labor Government between 2008-09. In each of these documents the use of, and requirement for, evidence-based policies was fundamental.

The first example comes from the Green paper Which Way Home? A New Approach to Homelessness (Commonwealth of Australia 2008). It argued that the 'Data and research on homelessness in Australia is limited, especially on the cost-benefit and cost-effectiveness of homelessness strategies and programs' (Commonwealth of Australia 2008: 11). The Government's subsequent White Paper, The Road Home: A National Approach to Reducing Homelessness, claimed that: 'There is an urgent need to improve the evidence base to inform the delivery of high-quality services to people vulnerable to homelessness' (Homelessness Taskforce 2008: 58, emphasis added). The Taskforce also noted the imperative to measure the efficacy and cost-effectiveness of services and programs (Homelessness Taskforce 2008: 58).

A third official Federal Government document, which illustrates the meaning and use of EBP among politicians, was specifically about research. The National Homelessness Research Agenda 2009-2013 (Australian Government 2009), to which $11.4 million was allocated, outlined 'the national priorities for research that will contribute to the whole-of-government response to homelessness', avowing that the 'Agenda provides a guiding framework for building a cohesive evidence base' (2, emphasis added). It also included a National Homelessness Research Framework, the 'Aim' of which was 'To improve the evidence base for preventing and responding to homelessness' (4; Fopp 2011). The above three examples highlight the demand for EBP in policies for people who are homeless.

At the most basic level, policy as a response to a social issue is of necessity reliant upon supporting evidence of some kind: the nature and purpose of policy as a set of procedural strategies intended to alter existing circumstances must in the first instance ascertain what those circumstances are. In this sense the intuitive appeal of EBP is understandable (Marston & Watts 2003: 144-145). Policies are a response to a perceived social need or a socially or politically defined 'problem'; in policy formulation and development, policy makers use data or information as a rationale for addressing the social need. Yet, as has been noted by others (Nutley et al. 2013; Marston & Watts 2003), while such claims may seem axiomatic and unproblematic to their protagonists, they pose a range of questions.

For example, what are the defining hallmarks of information that is socially regarded as valid evidence? How is such evidence acquired? On what grounds is such evidence regarded as socially valid and approved? Who or what organisation approves it? On what theory of knowledge or methods is such evidence based, and why? If the evidence is disputed or perhaps even contradictory, or there is evidence for two policy options, on what basis is a determination made? Another set of questions relates to the process of transition from evidence to policy. Evidence may be used to suggest that particular action is required to address a social need or problem, but, since the evidence by itself does not instrinsically constitute or suggest a specific policy action, on what basis is a particular policy selected?

In order to exemplify and highlight certain dilemmas with EBP, the following article proffers as an example an Australian policy for people who are homeless. The Supported Accommodation Assistance Program (hereafter SAAP or program) went through at least four iterations (SAAP Mark I-IV) between 1985 and 2008, when it was officially terminated. Subsequently, the services provided under the erstwhile program were renamed Specialist Homeless Services, and incorporated into the National Affordable Housing Agreement in 2009 (AIHW 2013: 1). In the following, findings from the official National Evaluations of SAAP under the Agreement between the Commonwealth of Australia, on the one hand, and the states and territories, on the other, will be used to investigate the evidence-base on which policy trends were recommended and subsequently formulated and developed (Chesterman 1988; Lindsay 1993; SERC & AHURI 1999; Erebus Consulting Partners 2005).

The government-commissioned reports of the National Evaluations of SAAP are significant because they were intended to provide the evidential basis for the next iteration of the program. The salient point here is that the evidence used to adduce the conclusions in this paper are found in the official government commissioned National Evaluations of SAAR The latter were regarded as the 'evidence-base' which was socially accepted at the time.

This paper explores neither the influence of political positions or values in the transition from evidence to policy, nor the inner machinations of bureaucratic and government decision-making. The function of the particular example of SAAP is not to claim that the conclusions drawn are generic and applicable to all policies. Rather, the use of the SAAP example is intended to identify implications of EBP that often remain unacknowledged, particularly when such evidence is considered to be neutral, and when the move from evidence to policy is regarded as axiomatic. Thus, the paper has two main aims: to submit to critical scrutiny some aspects of what is commonly regarded as the self-evident nature of evidence-based policy; and to consider the implications of this analysis by means of an illustration, which--as will be briefly outlined in the Conclusion--will retain its relevance because of the continuing lack of affordable accommodation for people who are homeless. While the paper challenges aspects of EBP, some of which is found in the academic literature, its primary focus is on the social and political popularity of the term and its understanding in Australia as attested by the three government documents referred to previously (Commonwealth of Australia 2008; Homelessness Taskforce 2008; Australian Government 2009).

After a brief examination of the relevant literature, the paper proceeds to analyse aspects of the contested nature of evidence in the policy-relevant academic disciplines, and the 'evidence-policy nexus'. Evidence from the National Evaluations of SAAP is then adduced in order to explore the extent to which the principal source of socially accredited evidence from the National Evaluations was adopted in policy formulation. Several implications of the evidence-policy nexus regarding SAAP development are then examined. The paper concludes by examining two pertinent issues relating to the post-SAAP situation. The first concerns recent research in which the nexus between evidence and policy for homelessness was problematic (Parsell et al. 2014); the second considers empirical evidence for the continuing post-SAAP need for exit points, that is, affordable accommodation.

The paper argues that there are certain widely regarded criteria (so as to be socially approved) of what constitutes evidence, and explores the scenario of seemingly rival sets of evidence in the transition from 'evidence' to policy. The argument is that the example of SAAP policy over two decades belies any straightforward connection between evidence and policy, and highlights how the selection of one set of evidence over its rival challenges the social perception that EBP is axiomatic and neutral. Consequently, in this example, policy is underdetermined by narrow socially approved definitions of 'evidence'.

Analysing aspects of the 'evidence' in evidence-based policy--origins and use

EBP emerged in the Australian setting after evidence-based policy making was adopted in the United Kingdom during the Blair Labour government. EBP was considered to be particularly successful in the area of health policy, and also in the context of 'severe public expenditure cuts and the need to ensure that scarce funds are allocated in ever more cost-effective ways' (Nutley et al. 2009: 5; Marston & Watts 2003: 146-150). In a critical review of the literature, Lisel O'Dwyer (2004: i) observed of EBP that it is

   based on research that has undergone some form of quality assurance
   and scrutiny.... This distinguishes it from public policy based on
   more conventional policy developmental processes where intuitive
   appeal, tradition, politics, or the extension of existing practice
   may set the policy agenda.

In the academic literature, any initial optimism about the term and what EBP offered has been tempered by increasingly critical analysis of the issues surrounding the term and its implementation (O'Dwyer 2004: 2). Of particular interest has been a cluster of issues related to the contested nature of social science research and evidence and, in particular, its relation to policy formulation and development. Some commentators have noted that evidence-based policy has been qualified to 'evidence-informed policy' (Nutley 2009: 3, 21-23) or 'evidence-influenced' or 'evidence-aware' policy (Marston & Watts 2003: 145; Jones & Seelig 2005: ii, 1).

One of the reasons for these caveats in the relevant academic literature has been that in the policy context evidence is often presented as hierarchical: some evidence is privileged, other evidence is regarded as inferior. Nutley and colleagues (2009: 11-14) comment on the tendency in the literature to argue that 'hierarchies based on study design' are 'too narrow', 'underrate good observational studies', that they exclude 'useful information' and 'provide an insufficient basis for recommendations.' Yet despite this problematisation, in Australia, at least, the clamour for EBP appears undiminished. This is attested by the three Australian Labor Government publications mentioned previously.

This paper analyses and illustrates the current popular assumption in Australia that evidence-based policy is neutral, independent and bipartisan and that, because policy is based on 'evidence', it inevitably transcends the cut and thrust of political debate, neutralises conflicting values, and functions as an antidote to ideology. In the following examination 'evidence' in evidence-based policy is analysed by distinguishing two referents of the term. The first (1) refers to the general and prior manner in which the 'evidence' in 'evidence-based policy' is acquired. This point largely concerns the theory of knowledge--epistemology --and methodology which produces research findings and results which count socially as 'evidence'. The emphasis here is on how any socially approved evidence per se is generated, and what are the origins of such 'evidence' which make it socially legitimate so that it warrants the appellation 'evidence' and is socially accepted as such.

The second (2) referent draws attention to the specific findings or results of research on different topics on which policy is allegedly or should be based. Pertinent here are the actual research findings or results about a particular topic, which ensue when the epistemology and methodology adopted in (1) are followed. In research about people who are homeless the epistemology or method deployed might be the standard epidemiological and statistical approaches which, inter alia, provide results and evidence about (2) above, for example, in this case the number of people who are homeless, and the links between homelessness and employment or physical and mental health.

What causes research findings to count socially as 'evidence'?

This section concentrates on analysing the first referent above, namely, the socially ordained approach by which research results and findings become regarded as 'evidence'--what Parsell and colleagues (2014: 81) describe as 'what "counts as evidence'"--or become socially valorised as 'evidence'. In question form, the salient issue is: What particular epistemology and methodology needs to be adopted for research findings to be recognised socially as 'evidence'? What are the social processes of knowledge production and validation that cause some claims to be regarded as socially regnant?

The brief analysis in the following is based on the argument that knowledge claims are socially and hierarchically organised so that some have a monopoly of social approval and others are subjugated (Fuller 1988; Fuller 1987; Nutley et al. 2009). Generally speaking, the method which counts socially as providing the evidence for policy is the 'scientific method' as it is practiced in the natural, health and medical sciences. With few exceptions, the epistemology and methodology on which it is based has, at least in the highest political and social echelons, so colonised all rivals that those disciplines that do not subscribe to their direct application--such as ethics, social and political theory, and some aspects of policy formulation and development--are regarded as inferior forms of knowledge. According to the socially ascendant view, this 'scientism' (Voegelin 1948; 1952) allegedly extricates us from ideology and values by its rigour and testing, its replicable and large-scale trials, its generalisability, maintenance of robust standards in the peer-review process, neutrality and independence.

In the public realm to even question the predominance of the scientific method may be considered a serious mistake. The scientific method is generally regarded as the best means of achieving the public health, medical and technological breakthroughs that have facilitated the betterment of the conditions in which some people live. Yet pointing to what makes socially sanctioned knowledge 'evidence', and thereby scrutinising the dominance of the scientific method, does identify what is often taken for granted in the social and political popularity of evidence-based policy, namely, that the evidence which is reckoned as suitable for policy is of a certain ilk, is privileged and allegedly devoid of presuppositions.

In this paper I am not offering an epistemological or philosophic analysis of such theories of knowledge let alone a critique, even though the 'scientific' method is based on some epistemological and other presuppositions which cannot be validated by the scientific knowledge which is so privileged (see, e.g., Polanyi 1973; Kuhn 1970). Further, in the policy illustration used in the following analysis to elucidate the argument (SAAP), the policy-relevant information comes from those parts of the social sciences which emulate the natural, health and medical sciences. At this point it is sufficient to note that the acceptance of what has been called 'scientism' is the basis of the privileging of 'evidence' in evidence-based policy.

Analysing aspects of the contested nature of 'evidence' in evidence-based policy

In problematising evidence-based policy, there is another issue which is even more relevant to the following analysis regarding SAAP, namely, the implications for policy confronted by rival evidence, conflicting research results, contested findings. The previous section highlighted that evidence used in policy is socially certified because it has been obtained using a socially sanctioned epistemology and methodology. But in many instances in social policy (such as about education, family, housing, urban affairs, welfare) there is often contesting evidence or research findings. In other words, there are competing or rival sets of research findings about specific issues. This is not necessarily a criticism. In fact, it may be that the tension in existing research becomes the focus of future research.

But the issue of rival research findings does raise a number of dilemmas for research-based policy. For example, in the situation of conflicting or rival findings, which socially eligible 'evidence' is selected for policy formulation and development, why and on what basis? In such a scenario, who or what organisation (including politicians and public servants) selects one set of research findings, including some evidence which they regard as valid, and excludes others --and why?

For example, consider the following scenario which is relevant to the example explored later in this article. Let us assume that in the literature and research about best-practice policy for people who are homeless, there is evidence that highlights the complex individual problems and high needs with which clients present to agencies, and that this has been documented in a number of studies over a period of time. The policy implication which is deemed to follow from this is that most resources should be deployed to provide services that address such individual issues which, in turn, are also regarded by policy makers as prerequisites to exiting post-SAAP-agency accommodation.

However, let us further assume that the evidence also emphasises the necessity for more affordable and appropriate exit points for clients in agencies. Without it, the evidence suggests, there is a bottleneck in SAAP agencies, and a backlog of potential clients who cannot obtain the agency-services to meet their stated needs. In addition, in this scenario, there is also robust evidence that the provision of independent, affordable and appropriate housing for people who are homeless is best practice in reducing homelessness (Tsemberis 2010; Johnson et al. 2012).

Here we have two sets of, what their protagonists would call, compelling evidence which have the potential to yield different policy options. For the sake of the argument let us assume here that limited resources prevent deriving a policy which would be a compromise of the two sets of competing evidence. At this juncture the salient question then becomes: on what basis could one set of evidence be chosen against another? On what grounds would one set of evidence be selected?

Let us now assume that our hypothetical, non-compromise policy choice was to allocate resources to agencies in order to address the personal issues of their homeless clients as a prerequisite to moving on to external (non-SAAP) independent accommodation. In other words, the resultant policy for people who are homeless emphasises diagnosing and addressing personal issues as a prerequisite to moving on. This, in turn, poses the following questions: on what basis was the research evidence about complexity of the presenting needs of clients regarded as more significant for policy change than the need for affordable and independent exit points? In the context of rival evidence, on what grounds was one set of evidence excluded as the evidence-base for policy and another set included? Was the exclusion of this evidence based on the 'best' evidence apropos what works to reduce homelessness, or was it based on other criteria?

The nexus between evidence and policy

The context of the above scenario of rival evidence concentrated on the selection of one set of evidence, and why. However, this in turn raises the issue of the relationship between the selected evidence and the policy it is purportedly consistent with. Is the policy selection based exclusively on the evidence or are there other external or extrinsic factors--such as political priorities, perhaps concerning funding, issues of feasibility, adverse or unintended consequences of or for related polices, or values--that influence evidence choice? Salient here is the transition from evidence to policy, and the ensuing links between them. As conceded by some advocates of evidence-based policy, extrinsic, non-evidential arguments for a particular policy can be considered an appropriate element of policy formulation (Banks 2009a; 2009b: Head 2010). Exploring the links or steps in the progression between evidence and policy is here referred to as investigating 'the evidence-policy nexus'.

Thus, in addition to what counts socially as valid knowledge in regard to its function and capacity as knowledge--issues of epistemology and methodology --and of rival evidence, evidence-based policy raises the issue of the transition from evidence to policy. The evidence or research findings informing social policy--which includes education, employment, family, housing and welfare policy generally--is rarely comprised of simple inductive steps. The link between evidence and policy is a highly complex and politicised process entailing decisions that involve, among other things: the inclusion and scope of the target group, including, for example, means-testing; limitations imposed on the policy by budgetary constraints; and considerations of the interface between policies and the implications of other policies on them. Put differently, a reason for complexity in policy formulation and development is that the evidence does not speak for itself, and that research findings do not speak about policy and policy implications. Specific and selected evidence is given voice in the policy formulation process, and by the decision-making process, which selects some evidence as relevant to policy while excluding others.

Even in public health policy, for example, a proven vaccine for disease relief or prevention may be scientifically incontrovertible but, in a context where funding issues and balancing budgets are highly contentious, the question of who should be targeted and who should pay are complex political and economic issues in which a variety of trade-offs may be necessary. Thus, resultant policies are not based exclusively on the known evidence which demonstrates the proven efficacy of a vaccine. In this way, rather than evidence determining policy, policy is underdetermined by the evidence.

Debates concerning the most effective goals (realising objectives) and efficient objectives (lowest unit cost) are an essential part of policy development. As socially produced, credible evidence and facts cannot speak for themselves, the transition from evidence to policy is, of necessity, political in the sense that it centres on those institutions which influence, generate and legislate policies. It is inevitable that in the transition from evidence to policy there will be debates about any conflicting or competing evidence, about norms (regarding what should be done), about how the policy should be implemented, who should be targeted and why, about how a suggested policy might have unintended consequences or adversely influence another policy in a related or unrelated area, and so on.

By itself this political engagement is not problematic. It is inevitable; it is valued in a liberal democracy. However, the involvement of the legislative and executive arms of government and other private competing interests may be adverse if certain policy voices are drowned out or not heard, if certain relevant evidence is discounted or excluded, or if policies are inequitable in their target scope or in their delivery.

Put differently, by itself the politicisation of policies is the norm in the liberal democratic west. Issues of what is valued, and concomitant social objectives, have a critical part to play in the way evidence considered relevant for policy is assessed and incorporated into policy. What has the potential to undermine the democratic policy-making process, however, is the dominant discourse that suggests that it can be depoliticised--that matters are settled, and thus neutral, and that the evidence-policy nexus is clear-cut and unilateral.

At this stage in the paper three main points regarding evidence-based policy have been outlined. First, the 'evidence' in evidence-based policy originates in the socially ordained and credentialled scientific production and social validation of knowledge which is, in turn, based on a set of epistemologies and methodologies. It is because the objects of research have been submitted to, and have arisen from, this form of socially sanctioned knowledge production that the subsequent 'evidence' as research findings has clout, and is afforded social credit sufficient to warrant policy development. In the following example, the National Evaluations of SAAP had such status.

Secondly, the socially qualified 'evidence' in evidence-based policy is sometimes conflicting and contested such that one set of evidence is, of necessity, selected over others. Why some knowledge produced in a socially validating way is included, and other dissenting evidence produced the same way is excluded, is not always clear. But, in such circumstances selection and, therefore, rejection of some evidence has occurred.

Thirdly, in much of the dominant discourse on the topic, the notion of evidence-based policy assumes an unproblematic nexus between evidence and policy, something akin to a one-to-one correspondence between the two. In practice, however, there are other factors, including extra-scientific factors which are, of necessity, invoked when the link between selected evidence and a particular policy is forged.

Illustrating the argument

In the following analysis, the characteristics of evidence-based policy outlined above are exemplified by exploring the history of a particular national Australian policy for people who are homeless. There are several reasons for choosing this policy. First, the Supported Accommodation Assistance Program (SAAP) was a national policy that involved agreements enshrined in legislation between federal, state and territory governments. The antecedents of the program were the plethora of agencies and programs with different funding sources that were streamlined when SAAP began in 1985. A second reason for choosing SAAP to highlight the issues germane to the ensuing analysis was its longevity (19852008), involving four iterations (Marks I-IV: 1985, 1989; 1994; 2000; SERC & AHURI 1999: 2; Erebus Consulting 2004: 1-4), which permits analysis of policy development over the four Agreements.

Thirdly, and most significant, is that all versions of SAAP were subject to rigorous and systematic National Evaluations, which included all states and territories. The evaluations were enshrined in SAAP legislation and were long-running, comprehensive and systematic, involving independent commissioned research that was used by the evaluators, and leading to amendments in the next version or 'Mark' of SAAP. As a consequence, the National Evaluations can be regarded as the accepted official 'evidence-base' for the subsequent modified version of SAAP and were used to appeal to, and defend, modified directions in the program. In policy formulation and development, the status of the National Evaluations was authoritative--and it is because of that status that they are used in the following example. Thus, for the above reasons SAAP provides a unique opportunity to inquire (1) about the National Evaluations and particularly their evidence and findings, and (2) if, and to what extent, the policy changes in the subsequent iterations of SAAP were based on such evidence or the selection of one set.

In the following it is not possible to examine at length the debates within SAAP. In one sense, there is no need to do so because the program was continually modified in accordance with one set of findings from which a more interventionist approach to clients was defended. Such interventions were regarded by policy makers as a precondition to exiting SAAP services despite the evidence from all National Evaluations that there was one pertinent issue that undermined the efficacy of the program. This issue was the lack of external, affordable and appropriate accommodation for SAAP-users when they were ready to leave agencies.

The lack of exit points

While there was funding for capital projects within SAAP (called the Crisis Assistance Program: Lindsay 1989: 9-10), one of the main findings of all National Evaluations of SAAP was the 'the lack of exit points'. The term 'exit points' refers to affordable and appropriate independent tenancies to which persons or households could move and reside when ready to leave temporary accommodation in SAAP (Lindsay 1993: 60; Neil & Fopp 1994). The exit points are usually a rental property in the private or public sector, or in what is known as community housing such as that run by community housing cooperatives (Jamrozik 2005: 293-300). The lack of exit points meant that clients stayed in SAAP longer than necessary and were potentially subject to more interventions in their personal lives. The net result of the lack of exit points was a problem for agencies because clients ready to exit SAAP agencies experienced difficulty in locating affordable, appropriate accommodation, thus contributing to a bottleneck in SAAP agencies and a backlog of potential clients waiting to receive SAAP services.

In light of the above, it is noteworthy that all four official National Evaluations of SAAP emphasised the lack of exit points. For example, the first National Evaluation mentioned that 'improvements in the SAAP systems are dependent on major changes in related health, welfare and housing systems' and the need for more public housing (Chesterman 1988: 36, 87). The second Evaluation (1993) also drew attention to the fact that SAAP workers were 'constrained' by the 'lack of access to suitable housing for many of their users' (Lindsay 1993: 46). Further, SAAP stakeholders to this Evaluation recognised that the 'lack of exit points ... was a serious barrier to SAAP's fulfilment of its role ...' (Lindsay 1993: 60). Significantly, it also stated that 'the most intractable problem facing SAAP is the lack of suitable, affordable housing for SAAP users when they leave SAAP services' (Lindsay 1993: 107, emphasis added).

The third National Evaluation also reported that agencies had enormous difficulty assisting clients to become independent in 'the absence of adequate exit points or when clients have high and complex needs' (SERC & AHURI 1999: 105, emphasis added). Reiterating previous findings, the fourth National Evaluation reported that the

SAAP aim to make its clients independent is impeded by the lack of affordable and appropriate housing as exit points from SAAP, the impact of high rates of unemployment and poverty, and the diversity of problems faced by clients including mental health and domestic violence (Erebus Consulting 2004: 5, emphasis added).

The report also stated:

It must be acknowledged that the potential for interaction among structural, institutional and individual causes [of homelessness] is strong. It would appear that the single most influential structural cause relates to the lack of affordable, sustainable long term housing. Compared to other countries Australia has a very low proportion of public and social housing ... and people who are homeless have a limited number of options. The work of SAAP agencies to achieve significant and lasting outcomes for their clients is substantially constrained by the lack of long term affordable housing. Indeed until this issue is resolved, it is difficult to address the issue of appropriate exit points from supported accommodation in many instances (Erebus Consulting 2004: 139, emphasis added).

There are a number of reasons for the paucity of affordable accommodation experienced by clients ready to leave SAAP accommodation. These include the downsizing of public housing stock in Australia (Fopp 2012), the subsequent government support of private home ownership, and government support for the private construction sector to increase supply by incentives such as negative gearing. Another reason for the lack of exit points is the market failure to deliver at the less profitable but more affordable end of the market suitable for SAAP clients trying to move to an independent tenancy (Carson et al. 2003: 3; Fopp et al. 2004).

With the evidence that was compiled consistently by the four National Evaluations of the program about the lack of exit points--'the most intractable problem', the 'most influential structural cause', outcomes 'substantially constrained' by--it may be thought that there was sufficient warrant for the development of policies, programs and services within SAAP that addressed the entrenched problem clients experienced as they attempted to find affordable, independent and appropriate accommodation. However, in the years of the program this repeated imperative from the official research did not receive the policy attention it would appear to merit to increase the number of independent housing and other forms of independent accommodation at the affordable end of the market. Instead, what unfolded in SAAP eroded the independent accommodation objective found in SAAP Mark I to make way for an increasing non-independent transitional accommodation objective within SAAP agencies. As the following will illustrate, this resulted in greater intervention and management of clients even when clients were considered to be ready to move. The policy implications of the evidence selected is explored in the next section.

Changes in SAAP to address the bottleneck caused by lack of exit points

The moving-on objective was displaced by non-independent transitional accommodation in SAAP

The original objective of SAAP Mark I was to provide funds to non-government agencies so they, in turn, could provide (1) supported accommodation and support services in order that (2) SAAP clients would be able to move to independent living (Chesterman 1988: 10-11). However, in SAAP Mark II the objective of the program was changed to provide what was known as transitional accommodation and supporting services (Lindsay 1993: 23). Whereas in SAAP Mark I the supported accommodation was intended to be crisis or emergency accommodation for 2-4 weeks, in SAAP II 'transitional supported accommodation' and 'long term supported accommodation' within SAAP was for 3-6 months and even longer in future iterations of the program. As a consequence of the lack of exit points, the objective of SAAP Mark II was modified to allow for alternatives other than affordable independent accommodation, and to permit longer stays in SAAP services (Lindsay 1993: 23). The transition from short-term crisis to medium-long term transitional accommodation in SAAP was retained in future versions of the program.

From independent accommodation to residualist problematisation of clients

SAAP Mark I was intended to provide support in order for homeless people to move to their own rental tenancy in either the private, public or community housing sectors. While the first National Evaluation (Chesterman 1988: 46-47) made a distinction between those who were houseless, on the one hand, and those who were in crisis, on the other, the second National Evaluation stated that the 'people who seek help from SAAP have a very diverse range of problems, only one of which is homelessness' (Lindsay 1993: 29, emphasis added).

This was the beginning of a different focus of the program, namely, the increasing and ascribed complexity of needs and the problematisation of persons who presented to services for people who were homeless. Thus, the second National Evaluation claimed that 'SAAP users confront a range of complex and sometimes interrelated problems, among which homelessness is but the most obvious' (Lindsay 1993: 38). The third National Evaluation continued the theme (SERC & AHURI 1999: 106): 'The experience of SAAP services is that the presenting problem for the homeless--the need for accommodation--is "the tip of the iceberg".' By that time it was 'estimated that up to 18 percent of all SAAP clients' were categorised as having 'high needs'. This included clients who had 'intensive needs', thus 'compromising functioning and [the] ability to meet basic needs which often manifest in difficult behaviours [which] are likely to be ongoing' (SERC & AHURI 1999: 42).

The fourth National Evaluation (Erebus Consulting 2004: 103) suggested 'shifting the focus away from homelessness per se (for which the logical outcome is a home) to a focus on resolving and assisting those in crisis for whom homelessness, or the risk of homelessness, is a consequence.' This undermined the original independent living objective of the program. It transformed a program designed to provide homeless people with supported accommodation in the very short term, and until they could acquire their own, into one that would allow 'SAAP services to move away from being seen as accommodation services providers ... to providing a crisis support service which includes [transitional agency] accommodation' (Erebus Consulting 2004: 103).

Furthermore, the National Evaluation (Erebus Consulting 2004: 104) noted that the 'impact' of proposed changes to SAAP would also involve a 'shift in the outcomes required of SAAP services to reflect their role in assisting those in crisis and achieving outcomes other than those related to their housing' (Erebus Consulting 2004: 104). In so doing, the fourth National Evaluation also appeared to agree with other evidence from the National Evaluations outlined in this section. It noted 'an increasing international trend to begin to recognise some of the structural causes of homelessness', while 'in the Australian context there has also been a growing emphasis by the Australian Government on those individual or personal factors that can increase the risk of homelessness' (Erebus 2004: 138).

Analysis

It should be emphasised that for the duration of the program the National Evaluations were comprehensive, detailed, and lengthy and, while managed within the federal, state and territory bureaucracy, were based on wide-ranging commissioned research and consultation with key stakeholders--although minimally with people who were homeless. It is clear from the National Evaluations that their respective authors used and provided the latest evidence about the program and homelessness to assess the previous version of SAAP and make recommendations for the next. Undoubtedly, the National Evaluations comprised the officially recognised 'evidence' on which future SAAP policy would depend. Regarding evidence-based research, several conclusions can be drawn.

One way to interpret the evidence from the National Evaluations is to see two competing research findings or results. One set entailed the paucity of exit points, which created a backlog of potential clients attempting to gain SAAP services and a bottleneck of clients in SAAP who, while ready to move on, were impeded from so doing due to the lack of external, affordable and appropriate housing. The other set of evidence was the high needs and complexity of 'problems' attributed to clients. As the evidence from the National Evaluations shows, those who designed the subsequent iterations of SAAP selected for policy formulation and development what they regarded as the credible evidence about the complexity of needs and the problematisation of clients. In so doing the cogent evidence about lack of exit points, and doing something about that issue, was minimised in policy formulation and development.

From the analysis of the evidence from the National Evaluations it is possible to discern a point made previously: that in evidence-based policy there is sometimes competing or conflicting evidence about one object or, as in this case, findings about different aspects of the topic under investigation. In the history of SAAP, one set of findings was selected--the ascribed needs of clients--and other findings were discounted to the point of being proscribed for policy purposes--in this case, the independent evidence about the lack of exit points.

One could speculate about why the presenting attributes of clients was chosen as the basis for policy change in SAAP, although detailed substantiation is beyond the scope of this paper. Suffice to say that the architects of the change in direction may have believed that trying to obtain more affordable accommodation was not feasible, or that increased rent assistance to allow ex-SAAP clients to compete in the private housing market would be unsuccessful and, therefore, more transitional--as distinct from affordable, independent--accommodation in SAAP was required. Perhaps they were convinced of the need for greater intervention to address imputed personal issues, such as considerations of health or unemployment, intending that the prolonged time spent in SAAP accommodation would allow that intervention, even if the agency agreed that clients were ready to leave. But there is one aspect of the policy changes which has been demonstrated, namely, that one set of evidence was preferred as the basis for policy change and another was excluded.

Moreover, within the confines of the analysis provided, it is possible to take this one step further. As the evidence required a choice between results about different topics--one regarding the presenting traits of SAAP clients and the other the lack of exit points--the selection does not seem to be based on the better science or the superior evidence; there was evidence for both themes in the research findings. Arguably then, the process of selection and exclusion was not based on the superior evidence. Perhaps the selection was based on what was regarded as feasible and the most likely to contribute to the continuation of SAAP. In which case, as has been pointed out before (Marmot 2004), the selection reflected not evidence-based policy so much as policy-based evidence--evidence selected on policy grounds, including beliefs about the direction of the policy--or the selection was based on pragmatic grounds concerning what was feasible in the budget or the private and public housing settings.

In the scenario outlined, and those like it, it is possible that other non-evidential yet normatively and value-based choices constrained, or decided evidence selection. This is in stark contrast to what is suggested by the motto which is the focus of this article, namely 'evidence-based policy'. In the public domain policy is ideally chosen and based on evidence. In the SAAP example, the justification for policy changes was under-determined by the evidence. Such justifications may have included political values and beliefs, or the most pragmatic choices in the circumstances.

Conclusion

At this point several points are worth emphasising. First, this article is not arguing against the use of evidence in policy formulation and development. Clearly, policy requires the information which a variety of evidence provides. However, the argument and corroborating evidence do show that, in the SAAP example, scientific and research 'evidence' was not the only element in policy formulation and development.

Secondly, this paper does not deny that some SAAP clients exhibit complex and high needs. Indeed, regarding SAAP, it might be argued that, on the basis of the National Evaluations, it was preferable in policy terms to realign the policy to cater for the evidence of the complex and high needs of clients rather than fund affordable housing within SAAP or outside it. After all, in one National Evaluation, it was reported that complex personal need affected nearly a fifth of all presenting clients (SERC & AHURI 1999: 42). As argued, this is the direction the program took. To adapt or modify the policy on the basis of such evidence may have been understandable, although, according to the one National Evaluation cited, four-fifths were not in this category of SAAP clients.

However, there was another set of evidence in the consecutive National Evaluations. The evidence adduced in the SAAP example attests to a change in policy that apparently chose some evidence over and against another set. In the history of SAAP, the evidence on the need to address ascribed client needs was selected for policy change despite the evidence regarding the lack of exit points. As documented, this distorted the original SAAP objective and altered the policy so that it did not address the evidential imperative for more exit points. There was also additional evidence during later stages of SAAP showing that provision of support services in independent housing for people who had been homeless or in SAAP agencies had been successfully tried (Tsemberis 2010; Johnson et al. 2012). Furthermore, the policy direction did not address one of the main causes of homelessness--the lack of independent affordable accommodation--so that a proportion of those who did leave SAAP later returned.

Thirdly, it might be argued that longer-term transitional accommodation in SAAP was preferable because it allowed clients to remain in SAAP when there were no exit points for them. While probably true, the funds expended on many clients in the form of greater management and intervention were unnecessary in instances in which clients were assessed as ready to live independently. Their temporary residence in longer-term transitional accommodation did not address the issues of the backlog of clients attempting to enter SAAP services, although it did entail SAAP fulfilling an expanded monitoring and surveillance social function (Fopp 2002).

Significantly, the issues examined in this paper are not confined to the duration of SAAP. First, a recent publication highlights a post-SAAP example in which one of the preferred policy initiatives was not based on evidence. Cameron Parsell and colleagues (2014) examined the popularity of an international model addressing homelessness that originated in the United States. Known as 'Common Ground', the policy provides 'supportive housing offering a range of onsite social services for residents in large congregate settings' (2014: 69). The authors make the point that despite there being 'very little (research) evidence to support the efficacy of the common ground model' (70), and 'critical voices pointing to the risk and disadvantages' (71), the policy attracted wide-ranging support from some senior State political leaders, bureaucrats, private sector leaders and philanthropic sponsors. It seems that in the post-SAAP scenario, the decision to adopt the policy was, like the issue of exit points in SAAP, under-determined by evidence.

On the basis of their interviews with its key proponents, the authors argue that Common Ground was adopted (76) because it was regarded as 'new, radical and exciting': its benefactors were willing to provide large building for the projects, the institutional nature of which had been criticised (71); it would add to social housing stock--585 units have been built in Australia with 'approximately half allocated to people exiting homelessness' (76); and, finally, because the policy was consistent with 'several broader policy agendas' including defined targets for reducing homelessness and facilitating social inclusion among vulnerable groups (76).

The paper by Parsell and colleagues (2014) provides a pertinent post-SAAP postscript to my argument in this paper. In addition to the finding that in the post-SAAP scenario Common Ground was underdetermined by empirical research evidence, their paper resonates in some respects with mine. First,

Parsell and colleagues corroborate the evidence about the lack of exit points in the National Evaluations, referring to 'the very tight rental housing market in Australian cities' and citing one respondent (76) as endorsing Common Ground because it added extra housing stock. The point about the lack of exit points in the post-SAAP Australia will be augmented below but it is, at least, central to the underlying housing policy context for the implementation of Common Ground.

Secondly, the lack of evidence in policy development which Parsell and colleagues (2014) outline was not only under-determined by evidence; its development also meant the exclusion of other policy alternatives including, in this instance, a policy known as 'Housing First'. As the authors make clear (71), Housing First was a less-institutional and multi-site option that had undergone evaluation and for which evidence was available (Tsemberis 2012). Unlike Common Ground, Housing First, which had been evaluated, avoids the expansive institutional model for transitional accommodation--in-house support in large buildings, with security and supervision arrangements--in favour of independent and mainstream housing options in which support is offered from an external source (71). Indeed, one of Parsell and colleagues' concerns is the 'regressive reinforcement of institutionalisation of homeless people' (84). Just as in the SAAP evidence from the National Evaluations outlined in my paper, Common Ground was selected in the post-SAAP situation with little regard for the evidence from a rival policy option, namely Housing First.

Thirdly, the type of evidence that Parsell and co-authors advocate is similar to that used as the evidence base in this paper. The authors concede that they have 'not attempted to assess the substantive merits of Common Ground in Australia' because that would have required 'robust evaluative evidence' (84) which, while underway, was currently unavailable. Understandably, until such evidence was available the authors would not offer 'a firm substantive judgement' (84). What is particularly relevant to the argument in my paper is the nature and type of evidence valorised as significant in such an evaluation. According to the authors, the 'robust evaluative evidence' required was described as a 'number of formal state-sponsored evaluations' (84). As emphasised, this is precisely the type of evidence provided by the National Evaluations of SAAP. They were commissioned by governments, mandated in the SAAP legislation, and the evidence was gathered during the evaluations of the entire program. The accruing evidence of all four major National Evaluations would constitute the type of robust research results required for EBP. However, this evidence was subordinate to the competing evidence emanating from the National Evaluations of SAAP for different policy settings.

The findings from the SAAP National Evaluations have contemporary urgency because the lack of exit points remains problematic. An Inquiry into Youth Homelessness (National Youth Commission 2008) found that with respect to young people the 'Lack of exit options into stable, affordable housing was raised in every public hearing in the country and in many of the submissions received' (National Youth Commission 2008: 236). The report of the same inquiry also noted that: 'Without adequate exit points, the SAAP system struggles against stagnation and against disappointing young people's hopes of escaping homelessness' (National Youth Commission 2008: 237).

The official Government Green Paper mentioned earlier recognised the high turn away rates in SAAP noting that 'Each night 620 new people seek SAAP accommodation and almost 360 of them are turned away' (Commonwealth of Australia 2008: 33). This was attributed to a 'a lack of exit points, a mismatch between [the] number of crisis beds and [the] demand for crisis accommodation and no new crisis beds' and repeated use by those who return (33, emphasis added). In addition to lower allocation rates for public housing (Fopp 2012), a recent respected official Government report has shown that a 19.2 per cent increase in the number of Australians living in temporary accommodation between 2006 and 2011 was mainly due to more people staying in the program subsequent to SAAP, namely, the Specialist Homeless Services (COAG 2013: 6, 13).

Recent reports (Anglicare Australia 2013; Wesley Mission 2013) have also demonstrated the continuing difficulties faced by people who are homeless or on low incomes--as the homeless usually are--acquiring affordable, independent accommodation. Anglicare Australia's (2013) national data revealed that only one per cent of listed rental properties in Australia were affordable for single-person households, two per cent for couples, and only 0.9 per cent for couples with children. A recent survey of community sector agencies by ACCOSS found that '61 % of all respondents said improving housing availability and affordability is the top policy priority' (ACOSS 2013: 10). Thus, the difficulties exiting SAAP and its predecessor, Specialist Homeless Services, continue despite a raft of post-SAAP measures designed to increase the availability of affordable housing, including the National Affordable Housing Agreement and the National Partnership Agreement on Homelessness--which became operational in 2009 and the National Rental Affordability Scheme, which was launched in July 2008 (RNPHWG 2011; CO AG 2013).

The history of SAAP in Australia (1985-2008) demonstrates that policy basis and direction is more complex than the motto 'evidence-based policy' suggests. As the SAAP example shows, certain official evidence was selected as socially credible and other evidence was discounted or ignored, particularly that relating to the paucity of exit points and, it should be noted, the preferences of people who are homeless. The history of the Program also reveals that, in the context of conflicting or competing evidence, a specific set of evidence was selected on grounds which were not associated with the criteria which would make it 'better' evidence. It was merely different evidence about a different topic, and the selection of this policy for homeless people was extra-evidentiary, all of which, despite the social assumptions regarding EBP, exemplifies the complex and political nature of 'evidence-based policy' and the 'evidence-policy nexus'.

References

AIHW (Australian Institute of Health and Welfare) (2013) Specialist Homeless Services 2012-2013, Canberra, Australian Government.

ACOSS (Australian Council of Social Services) (2013) Australian Community Sector Survey 2013, National Report, ACCOSS Paper 202, Strawberry Hill, New South Wales, ACOSS.

Anglicare Australia (2013) Anglicare Australia Rental Affordability Snapshot, Canberra, Anglicare.

Australian Government (2009) National Homelessness Research Agenda 20092013, Canberra, Department of Families, Housing, Community Services and Indigenous Affairs.

Banks, Gary (2009a) Challenges of Evidence-Based Policy-Making, Canberra, Commonwealth of Australia.

--(2009b) Evidence-based policy making: What is it? How do we get it?, Canberra, Australian Government.

Carson, E., Fopp, R., & Parker, S. (2003) Vulnerable in the City: A study of accommodation and other services for vulnerable adults in inner Adelaide, Social Policy Research Group, University of South Australia.

Chesterman, Colleen (1988) Homes Away from Home, Supported Accommodation Assistance Program Review, Canberra, AGPS.

Commonwealth of Australia (2008) Which Way Home? A New Approach to Homelessnes: Green Paper, Canberra, Commonwealth of Australia, May. Council of Australian Governments, CO AG (2013) Homelessness 2011-2012:

Comparing performance across Australia, Canberra, COAG Reform Council. Erebus Consulting Partners (2005) National Evaluation of the Supported

Accommodation Assistance Program (SAAP IV) Final Report, Canberra, Commonwealth of Australia, May.

Fopp R. (2012) 'Public Housing applications and allocations: what official statistics can tell us about allocation and ratios', University of Adelaide, Refereed Conference Proceedings, Sixth Australasian Housing Researchers' Conference, Adelaide, 8-10 February.

--(2011) 'The National Homelessness Research Agenda: Questioning State Sponsored Homelessness Research', Parity, 24 (9), 1-7.

--(2002) 'Increasing the potential for gaze, surveillance and normalisation: the transformation of an Australian policy for people who are homeless', Surveillance and Society 1 (1), 48-65, http://www.surveillance-and-society (accessed 22 July 2015).

Fopp, R., Parker, S. with Hume, A., Talbot, C, Carson, E. & Quinn, C. (2004) Getting Out and Staying Out: Pathways to Independent Living: Best Practice, Critical Interventions and Strategies for Addressing Homelessness in South Australia: Pinal Report. DHS/AHURI Project (Adelaide: University of South Australia and Uniting Care Wesley), pp. 1-115.

Fopp, R., Carson, E. & Parker, S. (2003) 'Paying off the Landlord's House': Entering and Sustaining Home Ownership Among South Australians on Low Incomes, Adelaide, University of South Australia, Social Policy Research Group.

Fuller, Steve (1988) Social Epistemology, Indianapolis, Indiana University Press.

--(1987) 'On regulating what is known: a way to social epistemology', Synthese, 73, 145-183.

Head, Brian (2010) 'Evidence-based policy: principles and requirements', in Productivity Commission, Strengthening Evidence Based Policy in the Australian Federation, Volume 1 Proceedings, Roundtable Proceedings, Canberra, Productivity Commission, pp. 13-26.

Homelessness Taskforce (2008) The Road Home: A National Approach to Reducing Homelessness, White Paper, Canberra, Department of Families. Housing, Community Services and Indigenous Affairs.

Jamrozik, Adam (2005) Social Policy in the Post-Welfare State; Australian society in the 21st century, Frenchs Forest, NSW, Pearson Education Australia.

Johnson, G., Parkinson, S. & Parsell, C. (2012) Policy Shift or program drift? Implementing Housing First in Australia, Melbourne, AHURI.

Jones, Andrew & Seelig, Tim (2005) Enhancing research-policy linkages in Australian housing: an options paper, Melbourne, AHURI.

Kuhn, Thomas S. (1970) The Structure of Scientific Revolutions, 2nd ed., Chicago, the University of Chicago Press.

Lindsay, Mary (1993) Moving Forward, Report of the National Evaluation Accommodation Assistance Program, Canberra, AGPS.

Marmot, M.G. (2004) 'Evidence based policy or policy based evidence?', British Medical Journal, 328/7445, 906-907.

Marston, G. & Watts, R. (2003) 'Tampering With the Evidence: A Critical Appraisal of Evidence-Based Police-Making', The Drawing Board: An Australian Review of Public Affairs, 3 (2), 143-163.

National Partnership on Homelessness Working Group (RNPHWG) (2011)

Review of the National Partnership Agreement on Homelessness, Canberra, November.

National Youth Commission (2008) Australia's Homeless Youth, A Report of the Inquiry into Youth Homelessness, Brunswick, Victoria, National Youth Commission.

Neil, Cecily & Fopp, Rodney (1994) Homeless in Australia, Causes and Consequences, Melbourne, Australian Housing and Urban Research Institute.

Nutley, S., Powell, A. & Davies, H. (2013) 'What counts for Good Evidence: Provocation paper for the Alliance of Useful Evidence', Research Unit for Research Utilisation (RURU), School of Management, University of St Andrews, February.

Nutley, S., Walter, I. & Davies, H. (2009) 'Past, present and possible futures evidence based policy', in G. Argyrous (ed.), Evidence for Policy and Decision-Making: A Practical Guide, 1-23.

O'Dwyer, Louise (2004) A critical review of evidence-based policy making, Canberra, Australian Housing and Urban Research Institute.

Parsed, C., Fitzpatrick, S. & Busch-Geertsema, V. (2014) 'Common Ground in Australia: An Object Lesson in Evidence Hierarchies and Policy Transfer', Housing Studies, 29, (1), 69-87.

Pawson, R. (2006) Evidence-based Policy: A Realist Perspective, London, Sage.

Polanyi, Michael (1973) Personal Knowledge: Towards a Post-Critical Philosophy, London, Routledge 6c Kegan Paul.

Social and Economic Research Centre, Australian Housing and Urban Research Institute SERC & AHURI (1999) National Evaluation of the Supported Accommodation Program (SAAP III), Canberra, Department of Family Services.

Tsemberis, S. (2010) 'Housing First: Ending homelessness, promoting recovery and reducing cost', in I. Ellen & B. O'Flaherty (eds) How to House the Homeless, New York, Russell Sage Foundation.

Voegelin, E. (1952) The New Science of Politics: an Introduction, Chicago, The University of Chicago Press.

--(1948) 'The Origins of Scientism', Social Research, 15 (4), 462-494.

Wesley Mission (2013) The Wesley Report: Homelessness and the next generation, Sydney, Wesley Mission.