A two-stage evaluation of a sport psychology internship.

Weigand, Daniel A. ; Richardson, Peggy A. ; Weinberg, Robert S. 等

Professional accountability and program evaluation have become necessary ingredients for a thorough examination of the impact of applied sport psychology on its clientele (Dishman, 1983; Gould, 1990; Silva, 1989; Smith, 1989; Vealey, 1988; Weinberg, 1989). In essence, the onus is on sport psychologists to protect the public's welfare by demonstrating the effectiveness and competency of the individuals delivering psychological techniques. If we are to learn from our experiences and improve our interactions as mental skills consultants, we must empirically assess the behavioral consequences of the educational services, consultations, and interventions we provide for athletes, coaches, parents, and officials.

Partington and Orlick (1987a) recently suggested some important ethical, educational, and professional reasons for evaluating sport psychology consultations. Ethically, it is important to understand for whom and under what conditions, mental skills techniques benefit or hinder the psyche development of an athlete or coach. Educationally, it is necessary to determine which combination of mental skills creates the "best" mentally prepared athletes. And professionally, it is vital that professional practice be assessed for reasons of credentialing and investigation of complaints.

Credentialing identifies the competencies and training required for professionals to adequately provide services. That is, the credentialing process establishes criteria for defining who sport psychologist are, or should be (Smith, 1989), and what they can, or cannot, do. Usually included among these criteria are the minimum educational and practical experiences necessary for competent delivery of services, often including some form of internship. For example, the Association for the Advancement of Applied Sport Psychology's certification guidelines (AAASP, 1990) states that certified sport psychology consultants, in addition to meeting other criteria, must receive "supervised experience with a qualified person, during which the individual receives training in the use of sport psychology principles and techniques" (p. 3). This should include a assessment of the client's needs as well as an evaluation of the intern's competency to adequately meet those needs. Unfortunately, at present, no agreed upon method of evaluation exists for sport psychology consultants or interns.

Several reports utilizing elite athletes and teams, however, have included subjective evaluations of the effectiveness of sport psychology consultations. For example, Gould, Tammen, Murphy, and May (1989) had sport psychology consultants who were affiliated with the U. S. Olympic Committee rate their own effectiveness. In addition, two special issues of The Sports Psychologist (Roberts, 1989; Roberts & Halliwell, 1990) included several articles evaluating consultant effectiveness in the delivery of sport psychology services to professional athletes and teams. While these reports have provided rich information concerning apparently successful techniques, they are primarily self-reports by the sport psychology professionals and not the athletes' perceptions of the effectiveness of the psychological skills received. Assessing athletes' perceptions of what worked and what did not work would seem to be a more appropriate method for evaluating the effectiveness of mental skills training than asking providers how well they thought the information was received.

Along these lines, researchers have recently begun to evaluate consultants' effectiveness and overall performance from the recipient's perspective (e.g., Gould, Murphy, Tammen, & May, 1991; Orlick & Partington, 1987; Partington & Orlick, 1987b). Orlick and Partington (1987), through semi-structured interviews, found that Canadian Olympic athletes viewed effective consultants as likeable, very applied, flexible, knowledgeable, accessible, and caring. When Canadian Olympic coaches who had worked with sport psychology consultants were interviewed, the most valued consultants were found to provide clear, practical, concrete strategies, related well with athletes, were energetic and hard working, flexible, open, available, and demonstrated useful and relevant sport psychology skills (Partington & Orlick, 1987b).

The results of the above interviews led to the development of the Sport Psychology Consultant Evaluation Form (CEF) (Partington & Orlick, 1987a). The CEF's consultant characteristics items were found by Partington and Orlick to be discriminating and useful for understanding athletes' judgments of consultant effectiveness. The CEF's temporal items (amount/type of contact) were also found to be useful for identifying important components of consultant effectiveness. In addition, data generated by the CEF paralleled the descriptions of effective consultants gathered in the interviews of Orlick and Partington's earlier work.

Gould et al. (1991), using the CEF to evaluate U.S. Olympic sport psychology consultants found similar ratings of effectiveness to that of Partington and Orlick's, providing support for the items on the CEF. Unfortunately, Gould et al. also found high levels of multicolinearity between the consultant characteristics, suggesting that many of the consultant characteristics were tapping highly similar constructs. In addition, the CEF is limited in one additional way. Specifically, the athletes' and coaches' perceptions of consultant effectiveness are limited by the primarily quantitative format of the CEE With the exception of a "catch-all" item ("Do you have any recommendations to improve the quality or effectiveness of the sport psychology consultation service being offered?"), the CEF has no provision for open-ended responses. By providing a framework which limits data collection to certain predetermined responses, the CEF fails to capture the potential depth and detail that could be obtained via qualitative methods (e.g., Locke, 1989; Martens, 1979, 1987; Sage, 1989; Strean & Roberts, 1992; Patton, 1980). Incorporating more open-ended questions or space for comments accompanying each item would permit respondents to use whatever words they want to describe consultant effectiveness.

Qualitative methodologists (e.g., Patton, 1980) also suggest that program evaluations should be dynamic. That is, rather than assessing consultant effectiveness at one point in time, as previous research has done, it would be beneficial to evaluate athletes' and coaches' perceptions at several times subsequent to the consultation. This would facilitate the identification of the mental skill techniques which are easily utilized over time, which require further instruction to implement, and which produce the greatest satisfaction. In addition, this would allow the clientele time to think about what the consultant did well and what areas of presentation need improvement.

Therefore, the purpose of the present study was to conduct a dynamic two-stage consultant evaluation which utilized: (a) forced-choice and open-ended questionnaire responses of athletes and their coach participating in American NCAA Division I women's basketball who had a sport psychology intern as part of the coaching staff, and (b) follow-up (two months postseason), standardized interviews of the head coach and selected athletes (one with a receptive and one with a less receptive reaction to the role of the intern). Four substantive areas were addressed including consultant's effectiveness, the types and use of mental skills, rapport with the consultant, and an overall evaluation of the internship program. General concerns regarding the internship experience included the following: (a) Was the intern's training in applied sport psychology adequate to meet the needs of the coach and team?, (b) Was the intern competent in meeting those needs?, and (c) What lessons were learned which may improve future internships?

Method

Subjects

The sample consisted of 12 collegiate NCAA Division I female basketball players and their female head coach (in the Southwestern United States) who worked with a sport psychology intern for the course of an entire season. The athletes ranged in age from 18 to 22 years, with a mean of 20.25 (SD = 1.14); the coach's age was 36 years. The athletes' years of participation in organized basketball ranged from 7 to 13 years, with a mean of 10.67 (SD = 1.88); the coach played organized basketball for 7 years and has coached organized basketball for 12 years.

Protocols

Two separate protocols were utilized in the present investigation. A questionnaire and a standardized interview were developed addressing four substantive areas of a sport psychology internship: (a) consultant's effectiveness, (b) the types and use of mental skills (c) rapport with the consultant, and (d) an overall evaluation of the internship program. Both protocols are discussed in detail below.

Internship evaluation questionnaire. A questionnaire was specifically developed for this study to evaluate a sport psychology consultant internship (see Table 1 for the questionnaire items and the appropriate scaling for each item). This was accomplished by (a) utilizing the forced-choice and open-ended format of a teacher evaluation questionnaire (McKeachie, 1986), (b) incorporating the sport-specific constructs of Partington and Orlick's (1987a) Sport Psychology Consultant Evaluation Form (CEF), and (c) including a qualitative assessment to enhance and support the quantitative data.

First, the questionnaire incorporated the combined forced-choice and open-ended format, some of the general categories, and selected items of The Student Perceptions of Learning and Teaching questionnaire (SPLT) (McKeachie, 1986). The SPLT's categories and items for effectiveness, rapport, and overall evaluation of a teacher were changed to refer to a basketball-specific sport psychology consultant. For example, the category "Impact on Students" was changed to "Consultant's Impact" and the item, "The instructor is friendly" was changed to "The consultant was friendly to me."

Second, the questionnaire items tapped similar constructs to Partington and Orlick's (1987a) CEF (a measure of the amount and type of athlete-consultant contact, as well as consultant's characteristics) but were less ambiguous and more inclusive. To achieve this goal, several changes were made from the CEF's format to the present questionnaire including: (a) CEF items which combine characteristics (e.g., "Seemed open, flexible, and ready to collaborate/cooperate with me") were made into separate items (e.g., "The consultant was approachable"; "I felt free to disagree with the consultant"; and "The consultant seemed willing to develop a mental training program specifically to meet my needs"); (b) unidimensional CEF items (e.g., "How effective was this consultant?") were expanded to multidimensional items (e.g., "The consultant gave good examples of the concepts" and "The consultant was helpful whenever I had a problem"); and (c) accompanying each item on the questionnaire was a space for "Comments" (which the CEF omits), permitting qualitative clarification and/or amplification of the quantitative ratings.

Finally, the questionnaire also incorporated five additional open-ended questions designed to evaluate the relationship between mental skills and the athletes' approach to (a) basketball and life outside of basketball, (b) the best thing(s) the consultant did for individuals and the team, and (c) suggested areas of improvement. The five open-ended questions are listed in Table 2.

Internship evaluation interview. The same four substantive areas were also addressed via a standardized interview. A standardized open-ended interview consists of a set of questions [TABULAR DATA FOR TABLE 1 OMITTED] carefully worded and arranged with the intention of taking each respondent through the same sequence and asking each respondent the same questions with essentially the same words (Patton, 1980; see Table 3 for specific questions).

Procedures

The questionnaires were administered while the team was on the bus for their final road games of the season and return of the anonymous instrument was to be an indication of consent. All 13 subjects (i.e., 100%) returned completed assessments.

Three semi-structured interviews took place approximately two months postseason. The first involved the head coach, the others involved two selected athletes. One athlete had a receptive reaction and one athlete had a less receptive reaction to the role of the intern (receptive and less receptive reactions were determined by higher versus lower effectiveness scores on the quantitative portion of the survey and by subjective evaluations from the head coach). It was emphasized that all data would be kept strictly confidential. The interviews were conducted by one of the intern's supervisors to facilitate an open and relaxed atmosphere where the respondents could talk freely about the intern. The interviews lasted approximately 45 rain and were tape-recorded in their entirety.

Qualitative Data Analysis

Initial content analyses were conducted by the first author who had a background in qualitative research methodology. Subsequently, all phases of the analyses were thoroughly examined by the second and third authors who are qualified experts in applied sport psychology. All raw-data themes as well as second-order themes were discussed until a triangular consensus was obtained. Therefore, agreement between all three researchers was reached at each level oft he analysis. The qualitative data analysis consisted of two phases: (a) identify themes from the comments associated with each questionnaire item (Table 1) and the five additional open-ended questions (Table 2), and (b) identify themes from the interview responses. The following seven-step process was followed:

1. Every comment and open-ended response from the questionnaire was recorded and categorized according to the four dimensions under investigation: (a) consultant's effectiveness, (b) the types and use of mental skills, (c) rapport with the consultant, and (d) overall evaluation.

2. Separate raw themes (responses that captured the major ideas conveyed) as well as similar themes (multiple responses which seemed to refer to one similar theme) were identified from the comments and open-ended responses.

3. Following the general procedures outlined by Scanlan, Stein, and Ravizza, (1989) and Gould, Eklund, and Jackson (1992), an inductive content analysis was conducted to identify common themes of greater generality from the lists of subsection raw-data themes generated in Step 2. Second-level themes were labeled "higher-order themes." The highest-level themes (those of the greatest abstraction) were the four general dimensions stated in Step 1.

4. All three tape-recorded interviews were transcribed verbatim, resulting in 22 single-spaced pages of interview data. The coach's interview produced 476 lines of data, the interview with the player with a positive reaction to the function/role of the intern produced 210 lines of data, and the interview with the player with a negative reaction to the role of the intern produced 179 lines of data.

5. The quotes (or paraphrased quotes) from all three interviews were categorized into the four dimensions under investigation (see Step I) and according to the separate interview questions (Table 3).

6. Separate raw themes (responses that captured the major ideas conveyed) as well as similar themes (multiple responses which seemed to refer to one similar theme) were identified from the interview responses.

7. Similar to Step 3, an inductive content analysis was conducted to identify common themes of greater generality from the lists of subsection raw-data themes generated in Step 6. Second-level themes again were labeled "higher-order themes." The highest-level themes (those of the greatest abstraction) again were the four dimensions stated in Step 1.

Results

Quantitative Analysis

The means and standard deviations for each forced-choice question are contained in Table 1. Means for each category indicate the athletes rated the consultant as having a moderate impact (M = 3.10), very effective (M = 4.14), and having an excellent rapport (M = 4.68). Overall evaluations indicate 75 percent of these athletes would like the team to continue its relationship with the consultant in the future, all of them (i.e., 100%) would recommend the consultant to other athletes and coaches, and the information provided by the consultant was rated as valuable (M = 3.83). In addition, the consultant's general effectiveness was rated slightly higher for the individual (M = 3.75) than for the team (M = 3.17)

Qualitative Analysis

Results of the content analyses are presented in two sections: (a) the themes generated by the questionnaire's comments and open-ended responses, and (b) the themes generated by the interviews. Higher-order (H-O) themes are discussed along with representative quotations and paraphrasing interspersed to illustrate and clarify the themes. The dimensions and associated themes are hierarchically represented in Figures 1 and 2.

Analysis of Comments and Open-Ended Responses

As can be observed in Figure 1, a total of 108 raw responses were extracted from the questionnaire's comments and open-ended responses. Subsequently, these raw responses were organized into 55 raw-data themes and further abstracted into 15 H-O themes which were classified according to the following four dimensions.

Consultant's effectiveness. The first H-O theme, titled helpful, included references which indicated the consultant was helpful, always "gave a good answer," and that "every athlete benefited in some way." Within the knowledgeable theme, comments suggested the consultant "used good basketball applications," "knew the game well," and was "knowledgeable about mental skills."

Types and use of mental skills. The first H-O theme, titled practice required, indicated that mental skills in sport require practice just like strategy or physical skills require practice. The two comments were: "I'd never practiced mental skills before he taught us how" and "I'm now learning how to build confidence in myself and others."

Not all of the reactions to the use of mental skills in sport were positive. The second HO theme, titled negative reactions, described their negative reactions to mental skills this way: "If I think, I screw up" and "It's hard to focus on mental skills during the season - better to do it during the off-season."

The third and fourth H-O themes, titled skills which improved basketball and life outside of basketball were responses to two specific open-ended items on the questionnaire (Table 2). The mental skills which seemed to enhance the way these athletes and their coach approached basketball included goal setting, relaxation training, positive self-talk, confidence building, mental imagery, and competition mental plans (pre-competition, competition, and post-competition routines). Of that same set of psychological skills, confidence building, goal setting, relaxation training, and positive self-talk also seemed to improve the way these subjects approached life away from basketball. One athlete stated, "I [now] set goals in the classroom as well as [on] the court."

Rapport with the consultant. The first H-O theme, communication, reflected the importance of good communications skills including references to the consultant encouraging or "building the athletes up," keeping what was discussed confidential, being "easy to talk to," and being "a good listener."

The second rapport H-O theme was identified as task focused. This theme was revealed in these two comments: "He's very serious about his task" and "I felt confident in his ability to make correct judgements for the betterment of the team."

The third H-O theme, concern for the relationship, suggested that when a consultant is available, caring, and concerned for the athletes, a "good working relationship" will develop. One player said, "He was always available and even if he had something going on, he would drop what he was doing and listen to me."

Overall evaluation. The first H-O theme, presentations, indicated two individuals thought the presentations were too detailed or "boring." While it may be argued that these raw-data themes might fit within the effectiveness dimension, they may or may not have had an impact on the internalization and subsequent utilization of the presented information. Thus, they were categorized within the overall evaluation dimension.

The raw-data themes identifying the best things the consultant did for individuals and for the leant were responses to two specific open-ended items on the questionnaire (Table 2). Subsequent content analysis of these raw-data themes revealed three H-O themes associated with each question: (a) taught mental skills, (b) provided encouragement and support, and (c) provided a new perspective (see [ILLUSTRATION FOR FIGURE 1 OMITTED] for specific raw-data

themes).

The last H-O theme within the overall evaluation dimension consisted of several suggested areas for improvement. Suggestions included treating college athletes as adults (not as high school athletes), not pushing athletes to "open up" (i.e., let them come to you), "staying a mental coach" not a strategy or physical skills coach (i.e., maintaining your boundaries), "make presentations more fun," and getting involved with the team during preseason (the internship began approximately I month into the season).

Analysis of Interview Responses

As can be observed in Figure 2, a total of 79 raw responses were extracted from the three transcribed interviews (37 from the coach, 19 from the "positive" player, and 23 from the "negative" player). Subsequently, these raw responses were organized into 40 raw-data themes and further abstracted into 14 H-O themes which were classified according to the following four dimensions.

Consultant's effectiveness. The H-O theme titled helpful included such quotes as "[The consultant] helped some individuals," "he added to practice," and "the Freshmen really benefitted." The coach also revealed that "He helped me personally in developing a positive orientation through the use of self-talk." The positive player said, "He helped me evaluate [myself]," and the negative player added, "I'm sure some of the things he did helped you think about your own game."

However, the consultant's effectiveness was not seen to impact whether the team won or lost. The second H-O theme, not helpful, represents the following quote from the positive player: "I don't think he had an effect on whether we won or lost."

The third H-O theme titled knowledgeable was generated from comments concerning the consultant's understanding of the use of mental skills in basketball. The coach said that one of the consultant's strengths was "[his] knowledge of the game. [The athletes] would have known if he didn't know what he was talking about." Both the positive and negative player also indicated that "He knew his stuff." The positive player added, "[he] gave good examples."

Types and use of mental skills. The first H-O theme, specific skills which improved basketball included goal setting, confidence building, self-talk and imagery. For example, the coach remarked that "Goal setting [improved my players' approach to basketball] because I don't think kids know how to set goals early in life and why you would set them." She felt she personally benefitted "[by] developing a positive orientation through the use of self-talk." The negative player believed that "Mental imagery was beneficial because it helps me see myself doing things the right way, over and over again, so I won't have to concentrate on doing the skill when I actually have to perform."

Unfortunately, according to the second H-O theme, not all of the mental skills information was approvingly received. The coach thought that "Imagery wasn't effective. It might have made a difference if we had started using it during preseason. Trying to incorporate it when you're trying to perform is harder if you don't already have the [ability to use imagery]." The negative player added that "Self-talk was not very much help because when 1 do something wrong, I just get madder and I have to let it go for a second and refocus - I don't want to think about it." (It is interesting to note that refocusing was not perceived as "thinking.")

Rapport with the consultant. The first H-O theme, part of the team, was generated because all three interviewees believed the consultant to be an integral member of the team, although from slightly different perspectives. The coach said, "Yes, I feel like he was [an integral part of the coaching staff]" and the positive player agreed, "Yes, he added to the bench." However, the negative player saw the consultant as "part of the staff, but not the coaching staff" (maybe more like the physical trainer?).

The second H-O theme, communication, was due to the consultant being considered trustworthy, enthusiastic, "easy to talk to," "a good listener," and "honest." The coach indicated the consultant was trustworthy stating, "If I didn't want him to say the things I told him, I don't think he did." The positive played added, "I knew that whatever I said, he would keep confidential."

Being task focused, the third H-O theme, refers to the consultant maintaining his boundaries, doing what he was asked to do, and performing those duties in a professional manner. None of the three interviewees thought the consultant overstepped his boundaries. The coach, in response to a question concerning writing a letter of recommendation for the consultant (Table 3), said, "I would because I think that he does a great job of knowing his boundaries."

The fourth H-O theme, titled concern with the relationship, was generated from raw responses which reflected the consultant's availability and sensitivity to the athletes' needs. For example, the coach believed that the "biggest thing that he did purposely !to establish rapport] was to be available. He didn't have to come, but he was there all the time."

Overall evaluation. The first H-O theme, acceptance, was reported to be due, at first, to the consultant being introduced to the team by the coach, then after the athletes began to interact with the consultant their acceptance was dependent upon his availability and competence. The negative player indicated that, "We accepted him at first because the coach brought him in. After we got to know him, we accepted him for what he knew. The positive player added that "[We] began to accept him as [we] began to understand what was going on." The coach felt that "At first, until they trusted him, they probably didn't think of him as being a very valuable tool, until they had a chance to get to know him and see that he was going to be ... available."

The second and third H-O themes were titled positive and negative reactions to the presentations. On the positive side, the information was perceived by the coach to be "slow and methodical" and "more individual than group." In contrast, the players' perceptions indicated that "He gave us too much, more than we needed to know and could use" as well as "We only listened to some of the [information presented]."

The fourth and fifth H-O themes within the overall evaluation dimension refer to several suggestions concerning procedural and presentation improvements. The first procedural recommendation was to get involved with the team during the preseason. One recommendation made by the positive player was that "it would have been better if he had been with us from preseason [on]." The second procedural recommendation was to build in some successful outcomes which should reinforce the positive influence of practicing mental skills. The coach said,

Sport psychology needs success. He won individuals over because he'd talk to them (e.g., about confidence building) and they'd have a good game because of what they'd talked about and that would convince them. He didn't have the opportunity to win over the entire team because we needed success [The team had a season record of 8 wins and 18 losses], we needed him to tell us this is what we need to do and then we needed to have success.

The recommendations concerning improving the presentations included making them detailed during the preseason and short review sessions throughout the season, making them at the athletes' level of understanding (i.e., at an adults level for college-aged athletes), and delivering them at home, not on road trips away from school (most presentations were made on road trips, rather than at home, because the consultant believed the athletes would be more willing to listen if there was nothing else to do. Unfortunately, these athletes wanted to spend their free time on road trips either relaxing or studying).

Discussion

The purpose of the present investigation was to conduct a dynamic two-stage evaluation of a sport psychology consultant internship. Several aspects of this evaluation are particularly relevant for sport psychology practitioners and internship supervisors including: (a) valued consultant characteristics and procedures, (b) utilization of mental skills, and (c) methodological considerations.

Valued Consultant Characteristics and Procedures

Similar to Orlick and Partington's (1987; Partington & Orlick, 1987b) interviews of Canadian athletes and coaches, the present investigation revealed that the characteristics most valued from a sport psychology consultant included being helpful, knowledgeable (about mental skills as well as the particular sport at hand), caring, understanding, available, trustworthy, enthusiastic, communicative, task focused, very applied, and part of the team. Sport psychology interns should therefore be well educated in applied, sport-specific, mental skills as well as available for the entire season.

These athletes also indicated that it is important for consultants (a) to make sure presentation of mental skills information is directed at the appropriate age level (e.g., towards adults for collegiate athletes) yet also fun; (b) to present detailed information during preseason talks with follow-up review sessions throughout the season; (c) to present information at home, not on road trips where student-athletes like to relax or study; and (d) to not push athletes to "open up," instead, let them come to you when they are ready and willing to learn. Thus, sport psychology interns need to be able to present information which is age-appropriate, presented more than once (detailed the first time with subsequent shorter reviews), and not forced upon individuals who do not desire it.

Utilization of Mental Skills

The utilization of mental skills among these athletes revealed that such skills as goal setting, relaxation training, positive self-talk, confidence building, imagery, attentional focus, and competition mental plans were perceived to be effective in and out of basketball. In particular, confidence building and goal setting were seen as the most important mental skills for use in basketball as well as outside of sport. Goal setting's importance may have been due, as the coach suggested, to the fact that goal setting is something relatively new to these athletes. Or, it may have been due to the relatively poor season the team was experiencing (8 wins and 18 losses). The method of goal setting that was presented focused on setting short-term, task-related goals, rather than on specific outcomes (i.e., winning). Thus, it seems probable that the reason for goal setting's acceptance was that they could set attainable goals which provided some form of positive reinforcement while still experiencing a losing record.

These results reveal the importance of a broad-based mental skills education. In addition, it seems clear that no single mental skill was most effective for all individuals. Clearly, sport psychology interns must therefore have an extensive array of techniques available and be competent enough to evaluate each athlete's situation to determine which might work best.

Methodological Considerations

The combined utilization of forced-choice and open-ended questionnaire items permitted a more comprehensive assessment of the internship than would be possible using solely quantitative or qualitative instruments. One of the strengths of quantitative results is that they generate a normative reference that supervisors can utilize to evaluate every internship. Each successive intern's performance ratings can be compared to previous intern ratings to evaluate the overall effectiveness of the internship program. In addition, this should also generate feedback on the quality of education sport psychology students are receiving prior to their internships. Thus, quantitative data is useful for certification evaluations and assessing potential complaints from clients. The development of normative standards for competent consultant characteristics seems necessary for the evaluation of adequate delivery of services. Unfortunately, quantitative data lacks the unlimited response potential of qualitative data.

The present utilization of comments, open-ended questions, and standardized interviews permitted these athletes to express their opinions without being restricted to predetermined categories. The above information concerning valued consultant characteristics and procedures probably would not have come to light utilizing solely quantitative measures. In addition, the mental skills which these athletes believed to be most valuable inside and outside of sport would likely have been missed if only quantitative measures were employed. Therefore, if the goal of effectiveness evaluation is to determine our successes and failures, it seems vital that we expand our assessment techniques as much as possible so that we obtain as complete information as we can. There is a need for valid and reliable quantitative instruments (e.g., for norm-referencing), but there is also a need for open-ended responses and interviews.

A final strength of the qualitative nature of the present investigation which deserves comment was the three perspectives utilized in the interview stage of the evaluation. Obtaining information from the head coach, a player with a positive reaction to the function/role of the consultant, and one with a negative reaction permitted an assessment of differing perspectives. We anticipated that the coach and her players might have perceived the intern's influence differently, and the results seem to support this expectation. In addition, we anticipated that players who viewed the intern's effectiveness in a more or less positive manner would also view their experiences differently. Again, the results seem to support this hypothesis. Therefore, multistage evaluations should attempt to obtain a cross-section of evaluations, and if possible, they should attempt to gather information from some clients who viewed the experience favorably and other who reacted unfavorably.

The two-stage evaluation, one stage at the end of the season, the second two months later also was an important improvement from existing research on consultant effectiveness. Program evaluations need to be dynamic (Patton, 1980). We practitioners of sport psychology need to evaluate the long-term effect of the mental skills presentations we make. It is vital for us to determine which of our techniques can be learned in a single presentation, which require refresher courses, and which ones are never internalized because of their difficult nature. In addition, the interviews, although based on similar categories to the questionnaire, generated a slightly different set of responses and themes than did the comments and open-ended items on the questionnaire. Thus, new information was gleaned after these individuals had a chance to contemplate and assess their interaction with the intern.

Conclusion

In conclusion, the present evaluation seems to suggest that this internship was a success. Overall, the responses from these athletes indicated the consultant was adequately trained and competent to meet the needs of the team. Some of the vital lessons learned include beginning internship programs during the preseason, be readily available, assess the client's expectations prior to beginning the consultation, and always assess the effectiveness of the process once it is oven Future internship assessments should include both quantitative and qualitative data collection so that normative information can be generated along with open-ended responses which clarify and amplify the quantitative results.

Table 2

Open-ended Questions

* Which mental skills technique(s), if any, have improved the way you now approach basketball practice and competition?

* Which mental skills technique(s), if any, have improved the way you behave outside of sport?

* What was the best thing(s) the consultant did for you?

* What was the best thing(s) the consultant did for your team?

* What area(s) of improvement (i.e., more or less of) would you suggest for the consultant?

Table 3

Internship Evaluation Interview Questions

CONSULTANT EFFECTIVENESS

1. In general, did the consultant add to or detract from your practices and games?

2. Was the consultant an integral part of your coaching staff? If yes, how?

3. What are your feelings about the consultant's work in regard to performance outcomes?

4. In general, how important was the information presented or discussed by the consultant for you'? for your team?

5. What would you say if you had to write a letter of recommendation for the consultant (or would you be willing to write one at all)?

MENTAL SKILLS

6. Which of the six mental skills content areas (mental plans, confidence, goal setting, relaxation, imagery, and self-talk), presented by the consultant, did you think was most important and why?

7. What sort of comments did your team have about mental skills?

8. How were the group discussions (or presentation) concerning mental skills?

9. How much did you personally contribute to any discussions concerning mental skills'?

10. In what way(s) did the consultant remind you about your need for using mental skills?

11. How was the consultant at letting you know how successful you were using mental skills'?

12. What were the consultant's strengths and weaknesses when discussing mental skills?

RAPPORT

13. What strategies did the consultant use to develop rapport with the (a) coaching staff? and (b) athletes?

14. Was the consultant friendly, approachable?

15. In what way(s) did the consultant seem to (or not) "fit in" with the team?

16. Did you have confidence that the consultant was trustworthy?

17. Did he go beyond mental coaching to basketball coaching?

GENERAL (use underlined words for coach)

18. Do you think (your athletes really believed) the consultant helped the team or did they accept him because (you), the head coach, made him a part of the team?

19. (For coach only) With your responsibilities as head coach, did you ever feel thai you had to devote too much time to the consultant?

20. In retrospect, are there any suggestions or concerns you would like to share with us about our intern program or the benefits/liabilities of sport psychology consultants working with intercollegiate teams?

We would like to thank the players and coaches of the University of North Texas Women's Basketball Team for their cooperation in this investigation.

References

AAASP (1990, Winter). AAASP passes certification criteria. Newsletter, Association for the Advancement of Applied Sport Psychology, p. 3-8.

Dishman, R. K. (1983). Identity crises in North American sport psychology: Academics in professional issues. Journal of Sport Psychology, 5, 123-134.

Gould, D. (1990). AAASP: A vision for the 1990s. Journal of Applied Sport Psychology, 2, 99-116.

Gould, D., Eklund, R., & Jackson, S. A. (1992). 1988 U.S. Olympic wrestling excellence: I. Mental preparation, precompetitive cognition, and affect. The Sport Psychologist, 6,358-382.

Gould, D., Tammen, V., Murphy, S., & May, J. (1991). An evaluation of U. S. Olympic sport psychology consultant effectiveness. The Sport Psychologist, 5, 111-127.

Gould, D., Tammen, V., Murphy, S., & May, R. (1989). An examination of US Olympic sport psychology consultants and the services they provide. The Sport Psychologist, 3, 300-312.

Locke, L. F. (1989). Qualitative research as a form of scientific inquiry in sport and physical education. Research Quarterly for Exercise and Sport, 60, 1-20.

Martens, R. (1979). From smocks to jocks. In P. Klavora & J. V. Daniel (Eds.), Coach, athlete and sport psychologist (pp. 57-62). Champaign, Il: Human Kinetics.

Martens, R. (1987). Science, knowledge, and sport psychology. The Sport Psychologist, 1, 29-55.

McKeachie, W. J. (1986). Teaching tips: A guidebook for the beginning college teacher (8th ed). Lexington, MA: D.C. Heath and Company.

Orlick, T., & Partington, J. (1987). The sport psychology consultant: Analysis of critical components as viewed by Canadian Olympic athletes. The Sport Psychologist, 1, 4-17.

Patton, M. Q. (1980). Qualitative evaluation methods. Beverly Hills, CA: Sage.

Partington, J., & Orlick, T. (1987a). The sport psychology consultant evaluation form. The Sport Psychologist, 1, 309-317.

Partington, J., & Orlick, T. (1987b). The sport psychology consultant: Olympic coaches' views. The Sport Psychologist, 1, 95-102.

Roberts, G. C. (Ed.). (1989). Special theme issue: Delivering sport psychology services to the 1988 Olympic athletes. The Sport Psychologist, 3, 299-385.

Roberts, G., & Halliwell, W. (Eds.). (1990). Special theme issue: Working with professional athletes. The Sport Psychologist, 4, 329-430.

Sage, G. H. (1989). A commentary on qualitative research as a form of scientific inquiry in sport and physical education. Research Quarterly for Exercise and Sport, 60, 25-29.

Scanlan, T. K., Stein, G. L., & Ravizza, K. (1989). An in-depth study of former elite figure skaters: II. Sources of enjoyment. Journal of Sport & Exercise Psychology, 11, 65-83.

Silva, J. M. (1989). Establishing professional standards and advancing applied sport psychology research. Journal of Applied Sport Psychology, 1, 160-165.

Smith, R. E. (1989). Applied sport psychology in an age of accountability. Journal of Applied Sport Psychology, 1, 166-180.

Strean, W. B., & Roberts, G. C. (1992). Future directions in applied sport psychology research. The Sport Psychologist, 6, 55-65.

Vealey, R. S. (1988). Future directions in psychological skills training. The Sport Psychologist, 2, 318-336.

Weigand, D. A., & Weinberg, R. S. (1992, October). Evaluating a season-long sport psychology consultation with a collegiate women's basketball team. Paper presented at the meeting of the Association for the Advancement of Applied Sport Psychology, Colorado Springs, CO.

Weinberg, R. S. (1989). Applied sport psychology: Issues and challenges. Journal of Applied Sport Psychology, 1, 181-195.

Daniel A. Weigand, Ph.D. is a Principal Lecturer of Psychology in the School of Physical Education, Sport, & Leisure at De Montfort University, Bedford, MK40 2BZ UK; Peggy A. Richardson, Ph.D. is with the Department of Kinesiology, Health Promotion, and Recreation at the University of North Texas, Denton, TX 76203-3587 USA; Robert S. Weinberg, Ph.D. is with the Department of Physical Education, Health, and Sport Studies at Miami University, Oxford, OH 45056 USA.