Comprehending the Value Structures Influencing Significance and Power Behind Experiential Education Research

Opening Address SEER 2004

The purpose of my presentation today is to draw attention to several of the dynamics connecting research to the current value and livelihood of the experiential education field. These dynamics ultimately have a large effect on our ability to serve the clients with whom we work, as well as the advancement of the field.

While my opening statement sounds innocuous enough, the rest of my talk will probably be much more contentious. At the heart of my beliefs lies an unsettling thought that some of the assumptions our field operates under mislead us in our efforts to validate and advance the field. It also has led to us conduct research in ways that aren't really that beneficial for the present or future of the experiential field. I guess before I get started, I'd like to do a little "begging for your collegial forgiveness," since I may upset some you with a number of the things Fm going to say.

I've always liked the interpretation of research as a field whose roots come from the Latin term "rescisco" (meaning "to find out facts or understand and know through inquiry") and "servo" (meaning "to watch or observe"). Along with these valuable processes, when designing and conducting research, there are equally valuable considerations researchers need to be aware of that I don't think we pay enough attention to:

1. Who is affected by our research?

2. Who evaluates our research?

3. How will the research be used once it is completed?

Many of these dynamics are interrelated in a systemic manner. For example, several stakeholders involved in the system surrounding one research study on youth may include a single adolescent, peer group, family, school, neighborhood, local community, state/provincial government, or profession, as well as the supporting institution of the researcher, federal government, and global community. Each of these constituents make judgments on what our research tells them, whether our research is of value to them, and how they can use elements of our research to improve their lives.

It is important to acknowledge and understand the interconnectedness between these stakeholders and their value systems. The systemic influence from one group of stakeholders to another is like the various influencing substructures displayed in Figure 1. Each has its own individual characteristics, but previous structures are contained within the larger structures of a greater system. When stakeholder systems complement one another, research tends to achieve its intended results; when these stakeholder systems are conflicted, problems tend to arise.

In additional ways, these interrelated structures are like the stainedglass lenses of a kaleidoscope. Each has its own pattern and characteristics, but the depth and complementary "colors" of the entire kaleidoscope system change depending on the orientation and juxtaposition of these layers. What seems most important in achieving a valued "end result" is to orient the various structures in an appropriate alignment that will achieve a high degree of value. Note that, one stained-glass layer rarely achieves the desired results; this typically occurs only when multiple layers or lenses complement one another in order to achieve the intended and greater objective.

As with kaleidoscopes and research agendas, certainly some alignments are valued more that others. Striving to achieve such a structural alignment certainly should be a goal in research systems. Unfortunately, as a field I think we need to be more aware of this and proactively seek to structure and align the layers of our field's kaleidoscope of research to create a more effective path in advancing the field. I don't think we do a very good job of this; and because of this, a lot of times I don't think we are really that relevant to anyone other than ourselves (we become "legends in our own minds"). I believe research in experiential education sometimes holds this role in the organization. Colleagues in AEE present adages of "Well, isn't that nice that you've found something you're interested in and you like to do it." I fear the underlying subtext in our own professional organization is that research has become one of those many things in AEE that people don't need to understand and isn't really that relevant [to them]. And in a nice but patronizing way, most AEE members steer away from research since they don't see it as involving them, and that it really doesn't make any kind of difference in their lives.

On the contrary, I believe we can make an incredible difference in the field and the lives of clients in experiential education programs, but we are currently failing to do so. Let me provide a personal example of how the misalignment of substructures can have dramatic effects at my own institution. One of the things I have spent some time researching over the last 20 years is the effect of adventure programming with student orientation programs (e.g., see Gass, 1984, 1987, 1990, 1991, 1999; Gass, Garvey, & Sugerman, 2002). A number of elements of my research in this area centered on examining program benefits to clients, peer groups, some members of the profession, and other substructure recipients of the program. When beginning my research agenda in 1980, many of the other surrounding substructures were aligned and congruent with the intended purposes of my research to examine and produce certain intended results. For example, the value system of the funding source (i.e., the Dean of Student Affairs Office) was aligned with the same values and intended outcomes of research as the [student orientation] programmers. These values were a focus on higher academic achievement, retention, and a vision of the purpose of student affairs in order to promote the personal and social development of undergraduate students.

Due to a number of factors (e.g., turnover in Student Affairs leadership, increases in university admissions and less concern about attrition, less focus on student development and greater focus on appropriate punitive sanctioning methods), the alignment of the value in the funding substructure changed. Funding was cut from this office in the final rounds of 2004-2005 budget negotiations because priorities had shifted. Retention was not as large an issue as it once was due to high enrollment patterns, and more attention was centered on better methods to identify problematic students (e.g., rioting Red Sox fans; underage drinking) and greater efficiency in removing them from the University. Despite all of these pieces of valid research, the failure of the research agenda to align itself with shifting priorities and changes in the intended focus of the program failed to maintain an "oriented alignment" throughout the entire systemic value structure (e.g., adaptive programming; research effectiveness of programming to reduce "antisocial" student behavior).

Researcher's Substructure

Funding sources aren't the only substructures with particular values to consider. Note that researchers themselves are nested within their own particular value structures. Examples of five common value structures of researchers that can influence research alignment include:

1. Conducting research that can show a sustained and programmatic focus rather than research on a series of seemingly unrelated activities. Researchers are rewarded for establishing a professional identity where their work is valued and judged significant by colleagues in their discipline, area, and related areas of study.

2. Producing a research profile where "staging" occurs (i.e., research agendas where researchers develop a regiment of several published articles along with articles in press, articles in review, works in progress, and work they plan to initiate over the next few years). Work is valued when it demonstrates a pattern of continued growth in the discipline and increasing sophistication as a researcher.

3. Having greater value is placed on researchers who demonstrate sustained and programmatic research. Researchers, whose work is conducted successfully (e.g., publications in refereed journals and presentations at professional conferences) over several years, are usually judged more favorably than researchers who appear generally inactive or unsuccessful for several years and then demonstrate a spurt of activity.

4. Sustaining adequate productivity. While no "numeric standard" exists, quantity of publications is certainly valued. Some researchers set a goal of one highly regarded and refereed publication each year. However, each researcher resides within an organization where such a goal may be "incredible" given other duties; in another institution, this level of "production" would lead to a termination of employment.

5. Producing quality products for research. The quality of one's scholarship certainly receives some form of value around relevancy, timeliness, innovation, rigorous conceptualization, and sound procedures. The greater these connected values, the higher regard of researchers and their work.

Government and External Agencies Substructure

One final example is the existing governmental "stained-glass lens" substructure surrounding researchers and the clients they serve. Of great contention right now is the advancement in the fields of education and mental health with certain research paradigms and lists of documented "effective programs." Three operational examples of this government phenomenon at the U.S. federal level include: (a) Funding Support, (b) The "List" Phenomenon, and (c) Marketing and Advocacy of Valued Programs.

Funding Support

More than ever, certain research paradigms are valued more than others, and research paradigms with higher values are actively being promoted by government funding agencies. For example, with the driving force of the Leave No Child Behind Act of 2001 and the Education Sciences Reform Act of 2002, the U.S. Department of Education will spend $3.3 billion this year to support educational rehabilitation, research and development, statistics, and assessment (U.S. Department of Education-http://www.ed.gov/index.jhtml). To aid scholars in the type of research this government agency is interested in funding, the U.S. Department of Education has produced a "user friendly guide" for "identifying and implementing educational practices supported by rigorous evidence" (see U.S. Department of Education, cover page). This guide reflects the current position of the largest educational funding source in the U.S. to support quantitative, randomized, and controlled experiments similar to research conducted in medicine and welfare reform.

Evidence regarding how this value system is currently translated into funding opportunities can be found in several U.S. Department of Education "Requests for Applications" (RFAs). One current application certainly related to experiential education is the Field-Initiated Evaluations of Education Innovations (CFDA # 84.305, application receipt date: December 16, 2004). The purpose of this funding opportunity is to examine and ultimately promote "promising education interventions designed to improve academic outcomes and other student behaviors that have a direct impact on academic outcomes" (U.S. Department of Education, p. 1). In this RFA (not unlike other current funding proposals), randomized control trials are recognized as providing the "strongest evidence of the impact of a particular intervention" (U.S. Department of Education, p. 3). Any research protocol not possessing a randomized design must be "high quality quasi-experiential designs" (U.S. Department of Education, p. 4). The RFA guidelines even proceed to identify the type of high quality quasi-experiential designs it values (i.e., regression-discontinuity design).

The "List" Phenomenon

Regarding the basis of its value systems and research conducted on programs, the U.S. Government and other agencies are creating "lists" of certain programs that are promising, effective, and model programs in areas such as education and mental health. Many of these lists also identify programs not "documented" as valid interventions for achieving intended outcomes. While this "list" phenomenon is growing, three recent but well developed examples of this include the following:

Children, youth, and family lists. While certainly not the only list for this area, the Promising Practices Network website "highlights programs and practices that credible research indicates are effective in improving outcomes for children, youth, and families. The information pertains to children from the prenatal period to age 18, as well as the families and communities in which they live. This site provides useful information to decision-makers, practitioners, and program hinders who must choose among many possibilities for improving results for children, youth, and families" (Promising Practices Network, 2004, p. 1).

Mental health lists. The U.S. SAMHSA (Substance Abuse and Mental Health Services Agency) (see http://www.modelprograms.samhsa.gov) has developed a three tiered system ranking programs of their effectiveness in substance abuse and mental health areas (see Figure 2).

The three levels of the system (see Figure 2) are programs that are:

1. Promising-"Promising" programs have been implemented and have demonstrated scientifically evidenced positive outcomes with SAMSHA issues. While potentially researchable for validation, promising programs have not yet been shown to have sufficient rigor and/or sustained positive outcomes required of "Effective" programs. Promising programs may be elevated to Effective program status following successful demonstration and documentation of program effectiveness.

2. Effective-Effective programs are well-implemented and have been evaluated to the point of producing a consistently positive pattern of results across SAMSHA issues and/or replications. Effective programs typically have been disseminated to more than one program.

3. Model-Model programs are well-implemented, well-evaluated programs proven to be exceptionally effective according to rigorous standards of research. Programs at this tier have been carefully implemented, maximizing the probability for repeated effectiveness. Model programs are often replicated to provide quality materials, training, and technical assistance to other schools and/or programs across the State. Model programs exemplify the best, most effective SAMSHA practices.

Youth violence lists. Regarding both primary, secondary, and tertiary prevention and intervention programs, this website identifies model and promising programs that work in areas associated with youth violence. Published by the U.S. Surgeon General's Office, this website also identifies the cost effectiveness of the program (see http://www.surgeongeneral.gov/library/ youthviolence/chapter5/sec3.html#Primary Prevention).

Associated with all of these listing processes is a small but emerging process denying the use of federal monies to be used to train professionals in programs not on the "list." For example, if an adventure therapy program for at-risk youth is not on a list of model intervention programs, program staff may not use federal training dollars to pay for an adventure training program focused on this issue.

Marketing and Advocacy of Valued Programs

Government agencies and their representative programs are becoming more active in promoting and informing the public (as well as professions) as to what educational efforts and interventions are "valued" and should receive support and implementation. Conversely, programs not valued are also marketed as "failing" and not to be used. Two web-based examples of this include:

The "What Works Clearinghouse" (WWC) of the U.S. Department of Education (http://whatworks.ed.gov/), established to "provide educators, policymakers, researchers, and the public with a central and trusted source of scientific evidence of what works in education" (http://whatworks.ed. gov/whatwedo/overview.html). The site identifies and promotes "valued" programs for certain populations and interventions, and creates accessible databases and reports providing educational administrators, teachers, policy makers with information on the effectiveness of educational interventions (programs, products, practices, and policies) that intend to improve student outcomes. Programs are divided into categories of:

* Studies meeting evidence standards

* Studies meeting evidence standards with reservations

* Studies not meeting evidence standards

* Studies not in the screening process

* Studies currently in review

For example, on the topic of Peer-Assisted Learning (i.e., educational practices designed to improve academic outcomes by using students to teach one another in pairs or small groups), the WWC has gathered 300 studies with:

* 15 studies meeting evidence standards

* 0 studies meeting evidence standards with reservations

* 0 studies not meeting evidence standards

* 176 studies not passing screening process

* 109 studies currently in review

Data are currently being collected on several educational programs. Three notable areas related to experiential learning include:

1. Character Education-the use of character education programming in K-12 schools;

2. Delinquent, Disorderly, and Violent Behavior-interventions aimed at preventing or reducing disruptive, illegal, or violent behavior among middle and high school students; and

3. Dropout Prevention - interventions in middle school, junior high school, or high school designed to increase high school completion.

The second website example is "Press Releases." In their marketing and advocacy efforts, government agencies are issuing press releases on their findings in terms of what is currently "valued" in particular programming areas. One recent example is a National Institutes of Health (NIH) press release issued on October 15, 2004, on violence prevention programs (see http://www.nih.gov/news/pr/oct2004/od-15.htm). In this press release, the NIH panel states that "scare tactics...group detention centers, boot camps, and other 'get tough' programs often exacerbate problems by grouping young people with delinquent tendencies, where the more sophisticated instruct the more naïve" (NIH, 2004). Similarly, "the practice of transferring juveniles to the adult judicial system can be counterproductive, resulting in greater violence among incarcerated youth" (NIH, 2004, p. 1).

Conclusion

Please don't leave today's presentation with the thought that since I may not have represented the research paradigm or area you're using, I am "against" your work or don't think you conduct good research. Nor am I saying to perform research just because a certain methodology is "popular" or funded. However, what I am trying to say is we need to be aware of the various substructures existing within the experiential education research paradigm and how the values associated with each substructure influence one another. We also need to consider why certain approaches are favored and promoted by these substructures with more than just a passing cynicism toward prominent traditions.

Two recommendations I have for aligning the various research substructures for the type of vision we would like to achieve with research agendas in the field are as follows:

Promote, advocate, and seek out partnerships between industry providers and researchers. One way to achieve the alignment I'm suggesting is to implement research programs possessing an appropriately structured systemic perspective. By this I mean to align and orient substructures to complement one another, and begin to proactively devise researchable programs with this intention. In such a process, researchers seek out meaningful relationships with industry leaders and devise agendas advancing the goals of service providers as well as the field. In turn, industry leaders ally and support researchers by: (a) being open to adapting programming to create more valid and powerful research processes; and (b) finance research efforts over short- and long-term periods through internal dollars or co-authoring external grants. Both parties must align their efforts to produce more meaningful foci on the long-term benefit of interacting with research agendas of three to10 years in length or greater, to achieve objective acceptance based on traditional standards (e.g., making the "list" of effective and model programs). Organizations such as the American Youth Foundation (AYF), Project Adventure, Inc. (PA), and OBHIC (Outdoor Behavioral Healthcare Industry Council) have recently increased their already well-established commitment to the field by agreeing to strive toward such critical goals. Note that such efforts are done for both the benefits to their own programs, as well as a continuing position of servant leadership to the experiential field.

Appropriately train students and other professionals to produce valued research. In my opinion, we currently are not training emerging professionals with the rigor we need to produce research in the field that makes a critical difference outside of our own field. Whether situated in higher education or outside these institutions, programs involved in the training of future experiential educators need to redouble efforts to support the production of quality researchers who are connected to programming but well prepared to produce scholarship. The focus of this training should be on advancing the field both internally as well as externally in allied professions such as education, mental health, adolescent work, resource land management issues, etc.

Hopefully, these thoughts spur each of us on to the betterment of our work and our efforts to better the lives of our clients, as well as one another. My sincere best to all of you in your efforts to find out facts or understand and know through inquiry, as well as your efforts to understand who is affected by our research, who evaluates our research, and how our research should be used once it is completed.

References

Gass, M. A. (1984). The value of wilderness orientation programs at colleges and universities in the United States. Durham, NH: University of New Hampshire. (ERIC Document Reproduction Service No. ED 242 471)

Gass, M. A. (1987). The effects of a wilderness orientation program on college students. Journal of Experiential Education, 10(2) 30-33.

Gass, M. A. (1990). The longitudinal effects of an adventure orientation program. Journal of College Student Development, 31(1) 33-38.

Gass, M. A. (1991). Review of longitudinal studies of adventure orientation programs. Journal of Experiential Education, 14(1), 47-48.

Gass, M. A. (1999). Programs in higher education. In J. C. Miles & S. Priest (Eds.), Adventure education. State College, PA: Venture.

Gass, M. A., Garvey, D. E., & Sugerman, D. (2003). The long-term effects of a first-year student wilderness orientation program. Journal of Experiential Education, 26(1), 34-40.

National Institutes of Health (2004). Panel finds that scare tactics for violence prevention are harmful: Good news is that positive approaches show promise. Retrieved October 27, 2004, from http://www.nih.gov/news/pr/oct2004/od15.htm

Promising Practices Network (2004). Proven and promising programs. Retrieved October 27, 2004, from http://www.promisingpractices.net/programlist.asp U.S. Department of Education (2003). Identifying and Implementing Educational Practices Supported By Rigorous Evidence: A User Friendly Guide. Retrieved October 27, 2004, from http://www.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid.pdf

U.S. Department of Education (2004). Field-initiated evaluations of educational innovations. Retrieved 27 October, 2004, from http://www.ed.gov/programs/edresearch/2005-305f.pdf

U.S. Department of Education (2004). Mission Statement. Retrieved October 27, 2004, from http://www.ed.gov/about/overview/mission/mission.html?src=ln

Michael A. Gass, Ph.D., University of New Hampshire, Durham.

E-mail: mgass@unh.edu

Copyright Association for Experiential Education 2005
Provided by ProQuest Information and Learning Company. All rights Reserved