Follow-Up Evaluation of Career-Counseling Programs:

OVERVIEW

In this digest, "career counseling" refers to activities intended to improve individuals' ability to make career decisions (Spokane, 1991). This includes individual and group career and employment counseling, job-search training, career education, career-planning courses, etc. "Follow-up evaluation" refers to the assessment of program outcomes (effects) on one or more occasions after completion of a program.

Meta-analyses (Lipsey & Wilson, 1993; Oliver & Spokane, 1988) have shown that career counseling produces gains as large as those generated by well-developed psychological, educational, or behavioral interventions in general. Additional follow-up evaluations are needed, however, to improve our understanding of why career counseling is effective, with whom, on which outcomes, for how long, and under what conditions. This seems especially true of Canada, where a mere 15-30% of career-counseling programs include follow-up assessments; 35-45% are evaluated only through counselor-client contact during counseling, and 25-40% are not evaluated at all (Conger, Hiebert, & Hong-Farrell, 1994). FOLLOW-UP EVALUATION OF CAREER COUNSELING:

PURPOSES, OUTCOMES, AND PROCEDURESPurposes.

The fundamental reason for conducting follow-up evaluations of career counseling is the impossibility of judging the true value of such programs without the extended time perspective afforded by follow-up (Morell, 1979). The more time that elapses after clients finish a program, the more likely that the program effects will have either decreased to pre-intervention levels, maintained themselves, increased, and/or emerged as unanticipated consequences. Other purposes for follow-up include establishing realistic expectations of what a program can and cannot accomplish, learning how to improve a program, helping decision-makers change the structure or funding of a program, or gathering political information for defending (or attacking) a program (Morell, 1979).

Outcomes to Assess.

Researchers should use various instruments to assess career information:

Multiple measures of the same outcome (e.g., questionnaires, interviews, and counseling-center records). Different types of measures (e.g., behavioral, archival, and attitudinal) (Oliver, 1979).

specific measures (e.g., the attainment of clients' individual career-counseling goals) and global measures (e.g., job satisfaction) should be used (Oliver, 1979).

The longer the interval between program completion and follow-up, the harder it becomes to relate clearly participants' actions to program activities (Morell, 1979). Thus, short-term outcome measures should often receive priority (Oliver, 1979).

Instruments of known reliability and validity from previous research are recommended, and objective, non-reactive measures (e.g., archival data, cost data) should be employed along with subjective measures (e.g., ratings, self-reports) (Oliver, 1979). Furthermore, both intermediate and ultimate outcomes of career counseling should be assessed. Intermediate outcomes (e.g., job-seeking skills) lead to final outcomes (e.g., employment status), and show why a program succeeds or fails, and allow improvements to be made (Morell, 1979)."Learning outcomes" (e.g., self-awareness, opportunity awareness, decision-making skills, and transition skills) are important intermediate outcomes of career counseling and precursors of subsequent socio-economic outcomes, such as earnings (Conger et al., 1994).

Major outcome domains to consider (Oliver & Spokane, 1988) include career decision-making (e.g., accuracy of self-knowledge, appropriateness and realism of choice, career information-seeking, decidedness, satisfaction); effective role functioning (e.g., academic performance, job-interview skills, career maturity, self-esteem, anxiety, need for achievement); and evaluation of counseling (e.g., ratings of satisfaction or effectiveness).

A multidimensional set of rating scales has recently been proposed for supplementary evaluation outcome measures (Spokane, 1991, pp. 219-224). These scales cover the domains of persistent search and exploratory behavior, information, realism, barriers, hope and morale, activity level, congruence, cognitive framework, commitment and predicament appreciation, goals and options, decisional process, anxiety, and performance.

Procedures.

The key purpose(s) for conducting a follow-up evaluation (see above) needs to be specified in advance, to guide choices about the most appropriate follow-up time-frame, outcome measures, and procedures.

Follow-up evaluations are typically "post hoc." implemented only after a program has begun, and are thus unable to benefit from random assignment of participants to programs or adequate control groups. Nevertheless, post hoc evaluations are well worth doing as long as the evaluator considers plausible rival hypotheses and recognizes that the evaluation will inevitably be less informative than if it had been pre-planned and well controlled (Morell, 1979).

Carrying out more than one follow-up assessment after program completion allows a profile of program effects over time to be determined. Confidence in the results of post hoc evaluations increases when there is convergence among several "naturally occurring" comparison groups: groups similar to those being studied but not receiving career counseling, the past performance of the study group itself prior to receiving career counseling, or successive program cohorts (Morell, 1979). Although post hoc evaluations do not allow the establishment of causal relations, they do permit reasonable judgments about possible or even probable relationships between program activities and client changes during the follow-up period (Morell, 1979).Qualitative data based on program participants' opinions should be used as a check on quantitative data, and vice versa. Also testing a priori hypotheses about expected relationships will enhance the interpretability of findings in post hoc, correlational evaluations (Morell, 1979).

Numerous techniques can increase response rates in follow-up surveys, including personalized letters, repeated telephone or mail reminders, registered mail, and payment for participation. A surprisingly high proportion of former program participants can often be located through the mail, telephone directories, public records, personal visits, specialized newspapers, alumni associations, etc., and programs can maximize successful follow-up rates by obtaining information during counseling that is relevant to maintaining contact (Morell, 1979). Some understanding of the direction and magnitude of attrition bias can be gained by comparing early and late responders, and responders and nonresponders.

Two Examples of Follow-Up Evaluation.

In a follow-up study conducted 3-6 months after career counseling had ended, Nevo (1990) found that clients rated discussions with their counselor as the single most useful component of career counseling, followed by ability tests, career-related reading, and interest inventories; felt that career counseling helped them more in promoting self-understanding than in fostering a specific career decision; were more satisfied with counseling if they had been helped in both the personal and career spheres, rather than in one sphere only; rated their counselors' assistance in helping them organize their thinking and become more aware of their interests and abilities as the most important factor in their satisfaction with counseling.

In a case study, Kirschner, Hoffman, and Hill (1994) found, at an 18-month follow-up, that a former career-counseling client had maintained her original counseling gains and crystallized her career goals. At a 5-year follow-up, the client identified her career-counseling experience as very influential in helping her achieve several important outcomes: a positive job change; a high degree of job satisfaction; greater awareness of the need to be more active in her career decisions and interpersonal relationships; and increased self-understanding, self-acceptance, and self-esteem. CONCLUSIONWe have identified some of the key purposes, outcomes, and procedures that career-counseling practitioners and administrators need to consider in planning useful follow-up evaluations of their services. Follow-up assessment provides advantages that no other evaluation strategy offers--information on program staying power, a profile of program effects over time, and the identification of unintended consequences (Morell, 1979). Thus, follow-up evaluation merits systematic application. REFERENCESConger, D. S., Hiebert, B., & Hong-Farrell, E. (1994). Career and employment counselling in Canada. Ottawa: Canadian Labour Force Development Board.

Kirschner, T., Hoffman, M. A., & Hill, C. E. (1994). Case study of the process and outcome of career counseling. Journal of Counseling Psychology, 41, 216-226.

Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.

Morell, J. (1979). Follow-up research as an evaluation strategy: Theory and methodologies. In T. Abramson, C. K. Tittle, & L. Cohen (Eds.), Handbook of vocational education evaluation (pp. 217-248). Beverly Hills, CA: Sage.

Nevo, O. (1990). Career counseling from the counselee perspective: Analysis of feedback questionnaires. Career Development Quarterly, 38, 314-324.

Oliver, L. W. (1979). Outcome measurement in career counseling research. Journal of Counseling Psychology, 26, 217-226.

Oliver, L. W., & Spokane, A. R. (1988). Career-intervention outcome: What contributes to client gain? Journal of Counseling Psychology, 35, 447-462.

Spokane, A. R. (1991). Career intervention. Englewood Cliffs, NJ: Prentice-Hall.

Robert J. Flynn, Ph.D., is an associate professor of psychology and clinical supervisor at the Centre for Psychological Services, School of Psychology, University of Ottawa.

ERIC Digests are in the public domain and may be freely reproduced and disseminated. This publication was funded by the U.S. Department of Education, Office of Educational Research and Improvement, Contract No. RR93002004. Opinions expressed in this report do not necessarily reflect the positions of the U.S. Department of Education, OERI, or ERIC/CASS. Title: Follow-Up Evaluation of Career-Counseling Programs: ERIC Digest. Document Type: Information Analyses---ERIC Information Analysis Products (IAPs) (071); Information Analyses---ERIC Digests (Selected) in Full Text (073); Descriptors: Career Counseling, Career Education, Career Guidance, Careers, Counseling Services, Counseling Techniques, Counselor Performance, Evaluation Criteria, Evaluation Methods, Followup Studies, Foreign Countries, Measurement Objectives, Program Evaluation Identifiers: Canada, ERIC Digests

###

[Return to ERIC Digest Search Page]