Assessment in wilderness orientation programs: Efforts to improve college student retention

This study examined the use of assessment in wilderness orientation programs in the United States. Fifty-seven programs responded to a survey that sought information on assessment and descriptive variables. Assessment variables included program goals and assessment method. Descriptive variables included program size, length, and time spent in wilderness or outdoor setting. Programs were also asked about administrative location, program leader academic degree level, time dedicated to program, and administrative contracts with outside organizations. These data provide insights into the assessment struggles and successes of wilderness orientation programs. Recommendations for enhancing assessment are provided.

Wilderness orientation programs offer incoming college and university students the opportunity for an adventurous or challenging experience, as well as important guidance into the academic world they are newly entering. Orientation programs, in general, have become increasingly important to institutions of higher education because these institutions face increasing costs of operation, including higher tuition, increased fees, limited service and course offerings, and reductions in faculty and staff positions (Mullendore & Abraham, 1993). Wilderness orientation programs offer a strategy for colleges and universities to increase retention rates and offer their students exciting programs. Some research suggests that orientation in a wilderness or outdoor setting does enhance self-concept and affects the formulation of academic and social goals (Devlin, 1996; Ewert, 1983; Gass, 1986, 1987, 1990). However, little is known regarding assessment techniques used by these programs (Berman & Berman, 1996).

An orientation program unable to demonstrate its relative value to an institution through periodic assessment will be open to criticism and question. Mullendore and Biller (1993) suggested that one question will be: "Is this program really necessary and can the institution afford it?" (p. 179). Mullendore and Abraham (1993) stated that programs that receive funding are those that demonstrate assessment strategies that successfully link program planning not only to program effectiveness, but also to program outcomes, and especially retention.

Assessment may be utilized by wilderness orientation programs to identify outcomes such as: increased grade point average, improved self-concept, completion of a degree program, or other variables associated with retention. Wilderness orientation is defined in the current study as a program designed to assist individuals in adapting to a new environment by using wilderness settings and experiences to facilitate or enhance changes or adaptations to university life. Upcraft and Schuh (1996) defined assessment as any effort to gather, analyze, and interpret evidence that describes institutional, departmental, divisional, or agency effectiveness. The purpose of the study was to describe the use of assessment in wilderness orientation and the environment in which that assessment occurs. This information will assist wilderness orientation professionals by clarifying the use of assessment, identifying commonalities among programs, and indicating suggestions for improvement.

Assessment and Evaluation in Wilderness Orientation

Gass (1986) described three phases of development within wilderness orientation programs. The first phase involves the use of wilderness experiences solely to develop interpersonal relationships through the use of outing clubs. This phase has been traced to 1935 and the Dartmouth College Outing Club. The second phase of development occurs with the introduction of Outward Bound principles and practices. These principles and practices gave form to the wilderness orientation program. The third phase involves the application of current research on orientation to design programs to meet the needs of incoming students. Research on traditional orientation programs has generated a wealth of information regarding the meeting of student needs (Mullendore & Biller, 1993) and, thus, this third and current phase involves the integration of that information into the design and practice of wilderness orientation. Davis-Berman and Berman (1996) found that 43% of the programs they sampled reported no inclusion of any evaluation effort. Thirty-eight percent of that sample indicated no follow-up activities and 49% reported occasional follow-up with participants (Davis-Berman & Berman, 1996).

Methods

This descriptive study utilized a mail questionnaire following the survey methods described by Dillman (1978). The questionnaire solicited responses on selected variables relating to program description, program goals, and assessment methodology during the spring of 1999 for programs occurring during the summer of 1998. Four research questions were posed by this study: What are the goals of wilderness orientation programs? What is a description of the wilderness or outdoor activities used by these programs? What social, academic, and behavioral domains are assessed by wilderness orientation programs? What methods of assessment are utilized by these programs to determine program success with regard to variables associated with retention (i.e., social, academic, and behavioral domains)?

The subjects of the study were identified through professional associations (e.g., Association for Experiential Education, National Orientation Directors Association) and previously published research on wilderness orientation (Davis-Berman & Berman, 1995). Subjects were also identified through informal electronic inquiry (i.e., internet search, queries to professional list serves, etc.). Eighty-nine programs were identified, and 57 usable responses form the data for this study, resulting in a response rate of 64%. Each subject was identified as director or supervisor of an orientation program for incoming college freshmen with the goal or aim of improving the students' adjustment to university life by utilizing a wilderness or outdoor experience as a significant portion of the program content.

Assessment Variables

For the purpose of this study, the assessment methodologies of the wilderness orientation programs served as variables. Summative assessments considered for this study are the social, academic, and behavioral domains related to increased retention in students. Each social, academic, and behavioral domain identified for inclusion in this assessment study is identified in a review of previous research (i.e., Gass, 1990).

The social and academic domains include positive peer group development, development of career plans, positive interaction with faculty, increased interest in academics, preparation for college academics, setting student expectations with actual university offerings, improved decision-making skills, improved small-- group skills, reduced stereotyping, increased cultural sensitivity, increased student satisfaction, adjustment and maturity, having fun, enhanced student leadership skills, and epiphanies, or flashes of insight, among participants. Behavioral domains are: grade point average, self-concept, motivation, locus of control, sense of community, and activism. Participant responses were also sought on indications of pre-assessment, debriefing, feedback during a program, use of evaluation by program leaders and participants, and longitudinal tracking of participant performance (Gass, 1986; O'Keefe, 1989).

Descriptive Variables

Data were obtained on the following descriptive variables: program activity, program length, number of participants, program goals, location, leader training, leader academic level, title, percent of time dedicated to program, employment status (e.g., full- or part-time), fees paid by students, requirement of attendance for students and faculty, and use or nonuse of outside contractors to achieve program objectives.

Instrumentation

The survey instrument was developed via review by an expert panel of 10 professionals and academics with knowledge of wilderness orientation programs (Fowler, 1988, 1995). A pilot study was administered to a purposive sample (N = 10) of the population of known programs in order to enhance face and content validity. Revisions were made to the instrument per expert review and comment.

A Description of Wilderness Orientation Programs

Wilderness orientation programs vary across several operational factors. These variables include program length, time spent in wilderness or outdoor setting, activity, number of participants, and requirement of participation. Activities utilized by wilderness orientation programs (Table 2) most frequently include group process, fireside discussion, backpacking, camping, camp cooking, journaling, low-ropes challenge, day hike, rock climbing, map and compass, canoeing, and community service. Means and standard deviations were calculated using dummy variables to estimate the population parameters.

Seventy percent of the programs (n = 40) indicated spending from 80% to 100% of their time in a wilderness or outdoor setting. Nine programs indicated spending less than 60% of their time in a wilderness or outdoor setting.

In this study, the program size (in terms of number of participants) ranged from 6 to 975; however, 84% of the programs have participant levels less than 250. The total student participation for respondent programs is 8,378 for this sample. The programs reported a range of length in days (See Figure 1). Programs with a length of 2-3.5 days and 4-5.5 days and 6-8 days encompass most programs. Longer orientation programs, lasting from 9 to 26 days, accounted for only 12% of responding programs. Programs lasting 3, 5, and 6 days occurred most frequently. Davis-Berman and Berman (1996) found that the average length of wilderness trips was 7.8 days with a range of 1 to 30 days, and each trip had an average of 10 students and two leaders.

Administrative Demographics

Several questions were posed regarding program administrative structure, with an emphasis on the administrative leader of the wilderness orientation program. Analysis of program administrative location indicates a division between multiple administrative locations as well as a range in the variety of administrative locations. As shown in Table 3, two-thirds of programs can be found in single administrative locations, such as student activities, residence life, an academic department, a retention management unit, or in an outdoor program. The remaining one-third of the programs indicate joint operation with an outdoor program.

Davis-Berman and Berman (1996) found that 27% of the responding programs in their study were free-standing entities in the university, and 38% reported sponsorship through the school's student life organization.

Universities do not require students or faculty to participate in wilderness orientation programs, as a whole. In the present study, schools reporting that all or some students were required to participate comprised 21.1% of all responding programs. Fewer schools require that faculty participate in wilderness orientation programs. Furthermore, schools reporting that all or some faculty were required to participate comprised 8.8% of all responding programs.

The position title of the administrative leaders of responding programs, for most programs (56.2%), is director or assistant director whereas coordinator (n = 11) accounts for 19.3% of all programs. The education level of program leaders indicates that most program leaders possess an advanced degree with master's (n = 28) and doctoral (n = 4) degrees held by 56.1% of all program leaders. More than one-third of all program leaders held an undergraduate degree (n = 21). Twothirds of respondents indicated that less than 30% of their position was dedicated to the wilderness orientation program (Figure 2). Respondents whose position is 70% to 100% dedicated to the program accounted for 14.1% of all programs. In terms of administrative location, three-fourths (n = 43) of responses indicate that the program was operated solely by the university. Twelve programs indicated that an organization outside the university was contracted for operation of all or part of the wilderness orientation program.

The Use of Assessment by Wilderness Orientation Programs

Program Goals

Davis-Berman and Berman (1996) developed the description of wilderness orientation in a survey of 38 programs. Thirty-two percent of the private schools and 42% of the public schools reported a social rather than an academic philosophy.

Respondents in the present study were provided a list of program goals. There were asked which of the goals coincided with those of the program; they were also asked for a ranking (1 = most important; 5 = least important). Space was provided for respondents to write in goals not listed on the instrument. Table 4 illustrates the frequency and percentage of the sample for all program goals selected. Program goals shared by most programs are positive peer group development, improved decision-making skills, improved small-- group skills, increased student satisfaction, adjustment and maturity, having fun, enhanced student leadership skills, and enhanced self-confidence/self-esteem.

A second grouping occurs with the program goals: reduced stereotyping, increased cultural sensitivity, positive interaction with faculty, preparation for college academics, setting student expectations, epiphanies among participants, and increased interest in academics. Although leave-no-trace/wilderness living skills were not included as a program goal on the survey instrument, 19.3% indicated them as a goal.

In order to gain an understanding of the overall importance of each goal, the ranked program goals (1-5) were weighted according to their selection. Goals selected in Gl were multiplied by 5 (G2x4, G3x3, G4x2, G5x1). The resulting scores were summed to provide a relative scale for program goals across all ranked goals (Table 5). The goals of positive peer group development, enhanced self-confidence/self-esteem, and having fun exhibited the greatest relative frequency followed by adjustment and maturity. Enhanced student leadership skills, improved small-group skills, and increased student satisfaction form a second cluster of weighted goals. Program goals directly related to academics hold much lower relative standing in ranked goals.

Davis-Berman and Berman (1996) noted apparent

changes in distribution of the goals of wilderness orientation programs toward a more social orientation. Their results indicate greater emphasis on facilitating social interaction and development than was found by O'Keefe (1989). This change could reflect increased competition for students among private schools and use of such programs to market the school and attract students (Davis-Berman & Berman, 1996).

Assessment Methods Used in Wilderness Orientation

Assessment of program performance was measured across goals ranked by respondents as the top five goals in order of importance. For each ranked goal, respondents indicated the method by which performance on that goal was measured by selecting from the list made available to them or writing in another method. Assessment methods indicated were: pre-/postprogram questionnaire, postprogram written evaluation, informal discussion with leaders, formal discussion with leaders, exit interviews with participants, group discussion with leaders and participants, and others. Figure 3 illustrates the assessment methodology utilized for each ranked goal.

The assessment methodologies most frequently used are group discussion with leaders and participants, formal discussion with leaders, informal discussion with leaders, and postprogram written surveys. Methodologies volunteered by respondents include use of focus groups, use of journals, written papers, and retention tracking. Use of similar assessment methodology appears to be common across goals, suggesting that goal characteristics play a minimal role in the selection of assessment strategy.

The sample was queried as to the type of assessment utilized before, during, and after the wilderness orientation program (Table 6). The most frequent methods used were the provision of program leaders with the specific goals of the program and the use of formal staff debriefings following the program. Secondly, programs sought assessment most frequently from leaders and participants following the program via informal discussion. Assessment was sought least during the program and student performance following the program was tracked by only 30% of programs in the current study.

Table 7 illustrates the distribution of assessment action across all wilderness orientation programs. Each assessment action was given equal weight and combined into the following scale: 1 = minimal assessment action, to 6 = maximum assessment. Those programs conducting assessment at each opportunity will yield higher scores on this scale than those that do less assessment. Leader instruction regarding the specific goals of the program is included as a gauge for the establishment of program goals prior to the beginning of the program. Programs indicating assessment levels of 4 and 5 (70.2% of all programs) indicates that a significant amount of assessment action occurs. Twelve programs indicate assessment levels less than 4 (21.1%). Means and standard deviations were calculated using dummy variables to estimate population parameters.

Summary

The findings of this study indicate wide variance among wilderness orientation programs. This spread occurs in program goals, assessment methodology, and operational and administrative factors. Most programs support pro-social goals, such as positive peer group development, enhanced self-confidence and self-esteem, and having fun. Academic goals such as easing transition to college and increased interest in academics were held by fewer programs.

Assessment in wilderness orientation programs occurs mainly as group discussion with leaders and participants and informal discussion with leaders. Less frequently occurring are formal discussions with leaders and use of postprogram written evaluations. Programs typically assess all of their goals using the same methodology for every goal. Most programs do not access assessment information throughout the wilderness orientation experience.

A wide range of activities was reported. Group process, fireside discussion, backpacking, camping, camp cooking, journaling, low-ropes challenge, day hiking, rock climbing, map and compass, canoeing, and community service are activities utilized by most wilderness orientation programs. Seventy percent of programs spend most of their time in a wilderness or outdoor setting, and the duration of most programs is between three and six days. There is a range of administrative location, leader title, education level, percent of position dedicated to the program, and contractual agreements with outside organizations for the operation of the program.

Program Goals

The goals of wilderness orientation programs, when weighted according to ranked importance, reveal an emphasis on social goals with a minor focus on academic objectives. The goal structure reflected by most programs in the current study includes pro-social elements that may assist a student's social functioning and indirectly aid in student retention. The top three most prevalent goals selected are positive peer group development, enhanced self-confidence/self-esteem, and having fun. The goals of most wilderness orientation programs retain the social focus identified by Davis-- Berman and Berman (1996).

Academic goals relevant to retention, such as positive interaction with faculty, preparation for college academics, increased interest in academics, and setting student expectations with actual university offerings were selected by less than half of wilderness orientation programs. However, the academic goals, when considered as adjusted weighted ranks, fall into the bottom third of ranked goals selected by wilderness orientation programs. Both Davis-Berman and Berman (1996) and O'Keefe (1989) identified this social emphasis among programs.

Assessment in Wilderness Orientation

Wilderness orientation programs primarily use informal assessment to determine performance, although the formal staff debriefing was commonly used in the process. The emerging model seems to portray assessment in wilderness orientation as informal and after the fact. Very few institutions access all of the assessment opportunities available to them. There was no comparison of changes in student attitudes or behavior via assessment before and after the program (such as Gass, 1990), although a few colleges/universities did assess students during the program. Preassessment of participants, a practice with many practical applications, was performed by less than one-quarter of all programs. The postprogram assessment utilized by most programs was sought informally from leaders and participants. The assessment efforts of wilderness orientation programs do not match those of traditional academic orientation programs.

Program Operations

From an operational standpoint, wilderness orientation programs adopt a standard set of activities for wilderness travel. The model that emerges from this study is an outing lasting from 3-6 days wherein 80% to 100% of the program time is spent in the wilderness or an outdoor setting. Group activity usually involves rock climbing, low ropes challenge, backpacking, map and compass work, camping, and camp cooking. Elements designed to enhance group development involve group process, fireside discussion, and the use of journals. The wide range of possible wilderness or adventure activities likely reflects the location of the particular program, the available geography, and the resources of the outdoor program. The number of participants in wilderness orientation programs responding to this study ranges widely from six to 975 students.

Administrative Demographics

Results of this study show that three administrative structures support wilderness orientation programs. Nearly 60% of programs reside in an outdoor program; however, half of these programs are co-located within student activities, residence life, academic departments, or as part of a retention management operation. The remaining third of the programs are located solely in student activities, residence life, academic departments, or retention management. Despite the presence of retention management as a supporting administration, the targeting of student populations "at risk" for non-- persistence was not indicated by the programs, nor was there a faculty requirement to attend these programs.

Leaders of wilderness orientation programs typically hold advanced degrees and, in a few cases, the leader holds an academic position within the respective institution. Seventy-seven percent of program leaders report that less than 20% of their time is dedicated to the wilderness orientation program, and only 10% indicated their position is 90%-100% dedicated to the program. Seventy-five percent of program leaders indicated that there was no use of outside organizations (i.e., contract agreements) for the operation of the wilderness orientation program.

Recommendations for Programs

Five primary factors emerge upon review of the data. First, social goals remain predominant over academic goals, confirming a trend toward more social goals identified by Davis-Berman and Berman (1996). Second, an assessment of the programs, as well as of the students, is an informal process for most schools. Third, a common set of outdoor activities was portrayed in this sample, and these activities follow a generic outdoor program model. Fourth, wilderness orientation programs exist in three general administrative structures: outdoor programs, student life, and a combination of outdoor programs and student life locations. Last, the education level of program leaders is not a factor that inhibits assessment. The combination of these factors suggests a general model for wilderness orientation programs.

If the intent of the orientation program is to develop a social network among students in order to limit the stress of beginning a college career, then this social goal emphasis is justified (Mullendore & Abraham, 1993). However, if the goal of the particular program is to enhance a student's orientation to academic endeavor, construct intellectual ties, and adjust student expectations, this social emphasis may need to be reviewed or altered (Mullendore & Biller, 1993). Wilderness orientation programs may not lend themselves to an easy integration of academic goals and thus, they appear to retain an emphasis on having fun and making friends.

Furthermore, orientation professionals in the field currently inform program activity leaders of the goals of the program prior to its beginning and solicit assessment from leaders and participants following the program in an informal fashion or use formal staff debriefings. The use of preassessment measures for program participants may enhance assessment of program success, as well as serve as an indicator for the students themselves with regard to their preparedness for college. The longitudinal tracking of student performance may also yield more objective indications of program success.

Several operational factors confound assessment in wilderness orientation. The administrative location of programs may be reflected in the program goals. For example, two-thirds of programs are located or co-located within outdoor programs, which have historically had personal and social development goals and a lack of formal assessment procedure. Assessment is utilized much more extensively in student life administrations. Furthermore, more than half of program leaders indicated that less than 20% of their time is dedicated to the wilderness orientation program. Program leader educational levels seem to indicate that most leaders have had exposure to research methodology (e.g., 56.1% indicated have in a Master's or Ph.D. program) usually associated with advanced academic degrees. In addition, the lack of inclusion of faculty in programs limits student association with the persona of academia, which may be as important as the construction of positive peer networks.

Wilderness orientation professionals may enhance their assessment efforts by developing assessment techniques that yield pre-/postmeasures of student development. These techniques should be widely shared among programs. Wilderness orientation programs may also benefit from an examination of stated goals. If retention is the ultimate goal of the program, then the academic goals may need to be revisited; however, a purely social orientation would have merit as an orientation tool as well. The administrative location of these programs may contribute to the lack of formal assessment and academic goals - outdoor programs have a tradition of social goals and lack an assessment tradition. Since Leave No Trace emerged as a goal of some wilderness orientation programs, the reliance of these programs on outdoor activity, such as camping, hiking, and rock climbing to accomplish their goals, raises the question of their impact on the environment.

Suggestions for Additional Research

Correlational research on wilderness orientation for colleges and universities will enhance knowledge of program performance. The development of practical assessment techniques that are resistant to time pressure would focus the outcome information available to a program. Investigation of the effect of a program's administrative location would enhance understanding of appropriateness of the philosophical direction of these programs. Investigation of the effect on participants of programs with pro-social goal structures as compared with programs with pro-academic goal structures would also benefit orientation efforts.

References

Davis-German, J., & German, D. (1996). Using wilderness orientation to facilitate adjustment to college: An updated description of wilderness orientation programs. Journal of Experiential Education, 19(1), 22-28.

Davis-German, J., & German, D. (1995). Wilderness new student orientation programs: American colleges and universities. Association of Experiential Education [on-line]. Available: http://www.princeton.edu/-oa/berman.html.

Dillman, D. A. (1978). Mail and telephone surveys: The total design method. New York: John Wiley & Sons.

Devlin, A. S. (1996). Survival skills training during freshman orientation: Its role in college adjustment. Journal of College Student Development, 37(3), 324-334.

Ewert, A.W. (1983). Outdoor adventure and self-concept: A research analysis. College of Human Development and Performance, Department of Recreation and Park Management, Center for Leisure Studies, University of Oregon.

Fowler, F. J. (1988). Survey research methods. Applied social research methods series. Vol. 1. Thousand Oaks, CA: Sage.

Fowler, F. J. (1995). Improving survey questions: Design and evaluation. Applied social research methods series. Vol. 38. Thousand Oaks, CA: Sage.

Gass, M. A. (1990). The longitudinal effects of an adventure orientation program on the retention of students. Journal of College Student Development, 31, 33-38.

Gass, M. A. (1987). The effects of a wilderness orientation program on college students. Journal of Experiential Education, 10(2), 30-33.

Gass, M. A., Kerr, P. J., & Garvey, D. (1986). Student orientation in wilderness settings. In R. J. Kraft & M. Sakofs (Eds.), Experiential education in schools (p. 320-330). Boulder, CO: Association for Experiential Education.

Mullendore, R. H., & Abraham, J. (1993). Organization and administration of orientation programs. In M. L. Upcraft, R. H. Mullendore, B. 0. Barefoot, & D. S. Fidler, (Eds.), Designing successful transitions: A guide for orienting students for college. (Monograph series no. 13) (pp. 61-78). National Orientation Directors Association.

Mullendore, R. H., & Biller, G.H. (1993). Orientation standards, evaluation and assessment. In M. L. Upcraft, R. H. Mullendore, B. 0. Barefoot, & D. S. Fidler, (Eds.), (pp. 169-182). Designing successful transitions: A guide for orienting students for college. (Monograph series no. 13). National Orientation Directors Association.

O'Keefe, M. (1989). Freshman wilderness orientation programs: Model programs across the country. Life beyond walls: Proceedings of the 1988 national conference on outdoor recreation. Fort Collins, CO.

Upcraft, M.L., & Schuh, J.H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco, CA: Jossey-Bass.

Shayne Galloway is an assistant instructor and research assistant at Indiana University's School of Health, Physical Education, and Recreation, where he is pursuing a doctorate in human performance. He may be reached at sallowa@indiana.edu.

Copyright Association for Experiential Education Fall 2000
Provided by ProQuest Information and Learning Company. All rights Reserved