An Instrument for Recording Coaches' Comments and Instructions During Time-Outs.

Hastie, Peter A.

This paper presents an instrument for recording coaches' interactions with their players during time-outs in sports such as basketball, volleyball and ice hockey. The instrument consists of four major categories which have been deemed to account for the communicative statements made by coaches to their players during breaks in play These statements are: technical statements, those relating to skill performance; tactical statements, those relating to strategic game matters; psychological statements, those relating to the emotional/cognitive aspects of play; and other statements, which are those statements unrelated to the game, or which are not likely to provide any benefit to the players. Interobserver reliability of the instrument was measured at .955 using a weighted kappa technique. The instrument provides a useful tool for researchers wishing to compare coaches in different game situations such as winning or losing, or to compare coaches working with different age levels or with different win/loss records .

From its earliest significant publications, research on coaching has been characterized by studies concerned with the "glass box" rather than the "black box," that is, research concerned with the coaching process itself rather than research concerned with inputs and outputs. For example, the first attempts at studying coaches were descriptive accounts of what coaches do and what transpired during practice sessions (e.g., Tharpe & Gallimore, 1976). These studies incorporated techniques of systematic observation, defined by Croll (1986, p. 1) as "the process whereby an observer or group of observers devise a systematic set of rules for recording and classifying classroom events." Indeed, a major development during the 1980's was a proliferation of systematic observation instruments designed specifically for coaching (e.g., Crossman, 1985; Lacy & Darst, 1984).

The study of Tharpe and Gallimore (1976), for example, provided a detailed analysis of the coaching behaviors of John Wooden of UCLA basketball fame. Closely following Tharpe and Gallimore was the introduction of the Coaching Behavior Assessment System (Smith, Smoll & Hunt, 1977), which provided great impetus into the study of coaching. Several studies employed CBAS to describe coaching behaviors, (e.g., Perkins, 1990; Senne, 1989) while the instrument has also been used in correlational studies and as an independent variable (e.g., Smith, Zane, Smoll & Coppel, 1983).

Other notable developments include the Arizona State University Observation Instrument (Lacy & Darst, 1984), which has been used to compare the behavior of coaches in a number of contexts (e.g., Claxton, 1988). A further instrument, the Coaching Behaviors Observational Recording System (Tannehill & Burton, 1989) was developed as an evaluative tool for the American Coaching Effectiveness Program education courses.

These studies of coaches in action and the concurrent development of observation instruments have provided a great driving force into the study of coaching; and there is now a considerable database that has identified different coaching behaviors in a number of contexts. Indeed, Darst, Zakrajsek, and Mancini devoted a chapter to "Coach/Athlete Climate Analysis" in "Analyzing Physical Education and Sport Instruction" (Darst, Zakrajsek, & Mancini, 1989, pp. 329-396).

From studies using systematic observation, it has become clear that the context of the setting has a significant impact upon coaching behaviors. For example, research has demonstrated different coaching behaviors between winning and losing coaches (e.g., Claxton, 1988), between coaches at different age levels (Segrave & Ciancio, 1990), and for the same coaches in pre-season and in-season (Lacy & Goldston, 1990), as well as between practice sessions and game play (Chaumeton & Duda, 1988).

Differences have also been found between coaching behaviors in individual and team sports settings. Tennis coaches have been recorded as being less involved with practice sessions, providing less instruction and having higher ratios of positive to negative feedback than football coaches. From these and other data, Claxton and Lacy (1986) concluded that to search for a universal coaching model encompassing all sports was not an appropriate direction for future research. Nevertheless, the knowledge base of coaches in action is still small and in need of expansion. The extent of replication in coaching studies is minimal, and perhaps the major justification for continuing to engage in systematic observation of coaches and to continue to develop valid observation instruments is that coaching occurs in so many diverse settings.

One context of coaching that has seen relatively little empirical research is that of the time-out. Sports that incorporate time-outs include volleyball, basketball, ice hockey, and American football, although the number and duration of time-outs varies with the sport. To date, literature concerning the content of time-outs has been limited to anecdotal writings in popular coaching magazines (e.g., Demerchant, 1990; Dickson, 1990). These papers are directed more towards practicing coaches as helpful hints. Lacking is a quantitative examination of a segment of coaching behavior, that is, interactions with athletes during time-outs. Indeed, only three studies could be identified which provide quantitative examination of the time-out context (Boutmans & Swillen, 1991; Duke & Corlett, 1992; Kozar, Whitefield, Lord & Mechikoff, 1993). However, none of these studies examined coaches' interactions with players during the time-out, and hence did not provide insight into the glass-box of coaching study.

Boutmans and Swillen (1991) investigated the influence of the time-out on the score of the team that called it, and determined that it had a positive influence, and that the largest probability for a positive influence seemed to exist with score difference between minus and plus four points. Duke and Corlett (1992) examined the factors which influenced a coach's decision to call a time-out. From questionnaire data, they found that coaching experience, gender, and team success influenced time-out decision making. Kozar et al. (1993) determined that the calling of a time-out before an opponent took free throw late in matches had no significant effect in reducing free-throw percentage.

The purpose of this study then, was to develop an instrument that would provide a record of a coach's comments during time-outs. In this way, further research on in-game communication and instruction would be enhanced.

Method

The development of this instrument proceeded through four stages. The main concern was to develop an analysis system that could categorize all possible statements made to players during time-outs.

Phase 1: The development of the Instrument

In developing an observation instrument, it is important to consider content, construct, discriminate and predictive validity. That is, there needs to be a derivation of categories which adequately describe the coaching process during time-outs, and which encompasses all interactive comments made by coaches to their players.

Two strategies were used to develop categories for the instrument, while fulfilling the requirements for content and construct validity. First, data were sought concerning the main tasks of coaching during time-outs. Second, a review of the literature and examination of already existing categories in other observation instruments were conducted.

The main tasks of time-outs. Duke and Corlett (1992) reported that coaches have four main reasons for calling time-outs. These are to reorganize the team, give the players rest, motivate the players and to give strategic information. Coaches were deemed to call time-outs depending upon offensive game events, defensive games events, the attentional state of the players, the emotional state of the players, the physical state of the players, and the development of strategy. It was from this rationale that the major categories in the current instrument were derived.

Review of coaching observation instruments. In developing the categories, analysis was conducted of the categories in the four most commonly used coaching observation systems. These were CBAS (Smith, Smoll & Hunt, 1977), the Coaching Behaviors Recording Form (CBRF, Langsdorf, 1980), the ASUOI (Lacy & Darst, 1984), and CBORS (Tannehill & Burton, 1989).

A total of 30 items described in these instruments were deemed as being likely to apply to time-out situations, and were consistent with the major tasks identified by the surveyed coaches. Table 1 shows a reduction of those 30 items into 10 categories.

The derivation of the major categories. Four categories were developed to encompass the major functions of coaching statements. These were "technical statements," "tactical statements," "psychological statements," and "other statements." Within each category, a number of subcategories were developed to specifically differentiate the interactive comments belonging to that category.

Technical statements are those statements the coach makes to the players about the performance of skills in the match. These statements could involve correction, praise, or criticism. Tactical statements are those statements about past or future tactical actions or decisions made by players. These include questions made by the coach about future strategic plans. Psychological statements are those related to the psycho-emotional aspects involved with playing, such as concentration, arousal, self-esteem and confidence. The fourth category includes statements that do not provide players with any "plan of action" for the upcoming components of the match, and are essentially deemed as unhelpful or irrelevant in the context of the match. The instrument, then, consists of four major categories and fourteen subcategories. These are described in Tables 2-5 (See APPENDIX).

Phase 2: Determining the Discriminate Validity of the Instrument

To achieve discriminate validity, a situation must exist where a number of subjects correctly classify coaches' statements to the correct categories. If the degree of agreement among the raters is high, then there is a good indication that the ratings do in fact reflect the dimensions they are purported to reflect. High levels of agreement then, would satisfy discriminate validity requirements of the instrument.

Subjects. Ten varsity coaches from a North American university were the subjects in this phase. The coaches were active in eight different sports, representing both men's and women's teams, team and individual sports, thus increasing the potential generalizability of any positive findings. The sports were basketball, volleyball, tennis and track for men and women, football, ice hockey and soccer for men, and women's field hockey. The coaches provided informed consent.

Procedures. Each coach was given two sheets of paper. The first consisted of a list of 30 statements made by coaches during time-outs in a high school girls' volleyball tournament. The second consisted of the category definitions and examples of statements fitting those categories (as in Tables 2-5). The coaches were given no training in the system, and were simply asked to use the criterion sheet to place each of the 30 statements into one of the 14 subcategories. Since the trial statements were directly recorded from coaches, they represented the content that was a criteria for content validity (see Borg & Gall, 1983).

To statistically determine validity, a weighted kappa technique was used. The kappa statistic (k) is a test for square two-way tables, and is a form of correlation appropriate for the measurement of consistency of categorical variables. The square two-way tables in this instance were (a) the categories designated to statements by the coaches and (b) the "gold score" or correct category as selected by the researcher. Kappa then, provides a measure of interrater consistency by focusing on the diagonal to see if it contains more counts than are expected by chance. Fleiss (1981) considers scores greater than .75 to indicate strong agreement.

A weighted kappa allows for attribution/quantification of the relative seriousness of each possible disagreement. That is, in determining K for this instrument, weights were given to subcategories to indicate their belonging to one of the four major categories. In this way, the "extent of error" of the coach's allocation of example comments could be determined. For example, if the correct response was "technical corrective" and the coach selected "technical positive" then this was a closer match than "general encouragement" since it belonged in the same major category. The K weights for categories with three items were as follows; 1 = correct response, .892 = incorrect, but in the same category, and .429 = in the wrong category, while weights for the four item categories were 1, .857 and .392.

Results. The range of percentages of correct answers was from 93.3% (4 coaches) to 100% (2 coaches). The weighted K for the 10 coaches was .955, indicating excellent consistency. For the four categories, the correct percent was as follows: technical statements 95.8%, tactical statements 98.9%, psychological statements 93.0% and general statements 100%. The K for the four categories was .958. In reporting these scores, it must be noted that all ten coaches were from the one institution. However, given they represented different activities, the high levels of agreement across categories and subcategories suggested that they could discriminate coaches' statements into the appropriate categories of the instrument.

Phase 3: Determining the Predictive Validity of the Instrument

If this instrument were to have predictive validity, it could adequately cater for all statements made by coaches during real time-out situations. Coaches' comments to players during 30 time-outs at a high school girls' volleyball tournament and during 20 time-outs at a high school boys' basketball tournament were recorded by microcassette and later transcribed and content analyzed. Two researchers trained with the instrument independently allocated these statements to one of the 14 categories.

All statements made by coaches could be allocated to one of the 14 categories of the instrument. The level of agreement between two researchers reached 98 percent for all time-out statements.

Phase 4. Determining the Reliability of the Instrument

To assess reliability, a situation must exist where a number of subjects can reach high levels of agreement and these levels are consistent over time. Twenty volunteer university students participated in this phase of the study. All were enrolled in either an undergraduate measurement and evaluation class, or a physical education methods class. The author taught neither class.

The volunteer students were trained in the allocation of coaching statements to correct categories. During the training phase, discussion was made of the rules relating to each category, and the trainees were given brief tests in which they were to define categories and score coaches' comments. At the completion of the training session, the students were given a set of 25 statements to code. This test was readministered one week later with no feedback being given in the interim. The index of stability was calculated using the Wilcoxon matched pairs signed rank test.

No significant differences were found between the first and second tests (T= 6, p [less than] .025). Means scores were 24.5 on the first test and 23.8 on the re-test. This indicates the students were able to develop a reliable coding protocol with training.

Discussion

The instrument that has been developed in this study provides a reliable and valid method for coaches and/or researchers to gather data about comments and interactions made during time-outs. The value of such an instrument is twofold. First, it allows for the investigation of descriptive questions that would further strengthen the knowledge base on coaching behaviors. These may include, what sort of statements do coaches make during times, and are these statements of direct benefit to the players, or are the simply emotional tirades? Sport psychologists have encouraged a shift away from winning as a focus of athletic performance, towards the more proximal tasks of playing. These are factors over which the athlete has more direct control, and include concentration, arousal levels, and stress management (Martens, 1987). Maintaining a focus on a process (or the present) rather than an outcome (future) orientation has shown to be beneficial to the athlete in the short term (see Bond, 1985), as well as long term (e.g., Fenker & Lamboitte, 1987).

In a time-out situation, one coach might present process oriented statements, trying to keep the players "on track" and focused on the next actions of the game, while another coach under stress might focus on "the need to win this one to stay in the race for the play-offs."

The second use of the instrument developed in this study is as a tool to assist in correlating time-out information with post time-out action. That is, patterns of communication which result in positive post time-out play action could potentially be identified. Sample questions that may be addressed using the techniques of the present study include (a) the success rate of calling time-outs, and (b) which behaviors are related to improvements in performance following time-outs?

Finally, this instrument could be used as a tool in intervention studies. That is, data collected using the instrument might be used to help increase the number of desirable comments made during time-outs and decrease the number of nonproductive remarks.

Address Correspondence To: Peter A. Hastie, Dept. of Health & Human Performance, 2050 Memorial Coliseum, Auburn University, Auburn, AL 36849-5323.

References

Bond, J. (1985). Winning: Is it everything? Sports Science and Medicine Quarterly, I (4), 6-9.

Borg, W. R., & Gall, M.D. (1983). Educational research (4th ed.). New York: Longman.

Boutmans, J., & Swillen J. (1991). Time-out: Key to successful coaching. In G. Tenenbaum & D. Eiger (Eds.). Coach education: Proceedings of the Maccabiah-Wingate International Congress on Sport and Goaching. (pp. 170-175). Netanya: The Emmanuel Gill Publishing House.

Chaumeton, N.R., & Duda, J. L. (1988). Is it how you play the game of whether you win or lose?: The effect of competitive level and situation on coaching behaviors. Journal of Sport Behavior, 11), 157-174.

Claxton, D.B. (1988). A systematic observation of more and less successful high school tennis coaches. Journal of Teaching in Physical Education, 7, 302-310.

Claxton, D.B., & Lacy, A.C. (1986). A comparison of practice field behaviors between winning high school football and tennis coaches. Journal of Applied Research in Coaching and Athletics, 1, 188-200.

Croll, P. (1986). Systematic classroom observation. London: The Falmer Press.

Crossman, J.E. (1985). The objective and systematic categorization of athlete and coach behavior using two observation codes. Journal of Sport Behavior; 8, 195-207.

Darst, P.W., Zakrajsek, D.B., & Mancini, V.H. (Eds.), (1989). Analyzing physical education and sport instruction. Champaign, IL: Human Kinetics.

Demerchant, R. (1990, December). Maximizing the volleyball timeout. Scholastic Coach, pp. 60-61,82.

Dickson, B. (1990). The proper use of timeouts. Basketball Coach, 22(2), 24-26.

Duke, A., & Corlett, J. (1992). Factors affecting university women's basketball coaches timeout decisions. Canadian Journal of Sport Sciences, 17, 333-337.

Fenker, R. M., & Lamboitte, J. G. (1987). A performance enhancement program for a college football team: One incredible season. Sport Psychologist, 1, 224-236.

Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: Wiley.

Kozar, B., Whitefield, K. E., Lord, R.H., & Mechikoff, R. A. (1993). Timeouts before free-throws: Do the statistics support the strategy? Perceptual and Motor Skills, 76 (1), 47-50.

Lacy, A.C., & Darst, P.W. (1984). Evolution of a systematic observation system: The ASU coaching observation instrument. Journal of Teaching in Physical Education, 3 (3), 59-66.

Lacy, A.C., & Goldston, P.D. (1990). Behavior analysis of male and female coaches in high school girls' basketball. Journal of Sport Behavior, 13, 29-39.

Langsdorf, B.V. (1980). Systematic observation of football coaching behavior in a major university environment. (Doctoral dissertation, Arizona State University, 1979). Dissertations

Abstracts International, 40 (8), 4473A.

Martens, R. (1987). Coaches' guide to sport psychology. Champaign, IL: Human Kinetics.

Perkins, G. S. (1990). Comparisons of male high school basketball coaches of winning male and female programs through systematic observation. (Doctoral thesis, University of Southern Mississippi)

Segrave, J. O., & Ciancio, C. A. (1990). An observational study of a successful Pop Warner football coach. Journal of Teaching in Physical Education, 9, 294-306.

Senne, T. A. (1989). Relationship among selected coaching behaviors and selected attitudes and self-esteem of girls participating in youth softball. (Master's thesis, University of Illinois at Urbana-Champaign)

Smith, R. E., Smoll, F. L., & Hunt, E. (1977). A system for the behavioral assessment of athletic coaches. Research Quarterly, 48, 401-407.

Smith, R. E., Zane, N. W. S., Smoll, F. L., & Coppel, D. B. (1983). Behavioral assessment in you sports: Coaching behaviors and children's attitudes. Medicine and Science in Sport and Exercise, 25, 208-214.

Tannehill, D., & Burton, D. (1989). Coaching behaviors observational recording system (CBORS). In P. W. Darst, D. B. Zakrajsek, & V. H. Mancini (Eds.), Analyzing physical education and sport instruction (pp. 379-389). Champaign, IL: Human Kinetics.

Tharpe, R. G., & Gallimore, R. (1976). What a coach can teach a teacher. Psychology Today, 9(8), 75-78. Category Descriptions from Four Major Observation Instruments Category CBAS CBRF ASUOI CBORS Instruction about skills *** *** * Positive reinforcement of skill *** * *** Negative reinforcement of skill *** * *** Corrective feedback of skill *** * *** General encouragement/praise *** * *** *** General criticism/scold *** * *** *** Game irrelevants/other *** *** *** *** Hustle *** *** *** Questioning * *** Strategy * * * * Note. (***.)these categories had direct equivalents in the final instrument (*.)these categories were parts of a category in the final instrument

APPENDIX

Category Definitions and Examples for Technical Statements

Technical corrective (TC)

Definition: Coach gives the player corrective information about skill performance.

Examples: "You need to get bend your knees more in defense" "Try to toss the ball higher on the serve"

Rules: Statement must include information about skill correction or improvement, and be stated in a nonthreatening manner.

Technical positive (T+)

Definition: Coach makes a positive statement to players about skill performance.

Examples: "The defense out there is great" "Great blocking Tammy, they can't get past you"

Rules: The statement is positive, but does not include remediation.

Technical negative (T-)

Definition: Coach scolds a player for poor performance of skills.

Examples: "You're just locked into position. Move your feet to the ball" "That block was way too weak"

Rules: The statement must include reference to specific skill performance. Otherwise it is coded as criticism.

Category Definitions and Examples for Tactical Statements

Tactical proactive (TP)

Definition: Coach gives directions about future strategic options.

Examples: "We want to serve to the front line players"

"We're going to play the green defense"

Rules: Statements are made about tactics or strategies, rather than skill performance.

Tactical reactive (TR)

Definition: Coach makes a comment (often negative) about some decisions the players have just made.

Examples: "I told you not to commit block, stay down"

"We can't just dig the ball over, we have to keep attacking"

Rules: Statements must refer to actions that have been completed. Needs to be reference to players decisions, otherwise it is coded as criticism.

Consequence statements (CS)

Definition: Tactical statements made by the coach that relate to future events, based upon aspects the players can attend to in the present.

Examples: "If we can play tough defense for the next few rallies, they will probably start to make errors"

"It's important that we don't miss any serves for the next few rallies"

Rules: Places a value on certain actions. Nearly always "if-then" statements.

Asking questions (Q)

Definition: Coach asks questions to players related to game actions.

Examples: "Who are we going to serve to?"

"What defensive pattern are we following?"

Rules: Questions must relate to playing issues such as tactics or skill performance.

Category Definitions and Examples for Psychological Statements

General encouragement (E)

Definition: Coach makes positive reference to players with ihe purposes of rewarding, increasing confidence or self esteem.

Examples: "Good job Beth, way to go."

"That's it girls, top stuff'

Rules: Statements are general in nature. Do not refer to skill execution.

Criticism (C)

Definition: Coach makes negative reference to players which might reduce confidence.

Examples: "That was rubbish"

"What do you think you're doing? How can you play like that?"

Rules: Statements are general and do not include specific reference to specific skill performance or tactical decisions.

Attention focus statements (AF)

Definition: Coach uses techniques to refocus attention to the "now" tasks, or to help control levels of anxiety or concentration.

Examples: "Let's focus on our next job. Just play one rally at a time"

"Just take a few deep breaths and get our minds on the job"

Rules: Refer to specific factors which the players can control.

Reassurance statement (R)

Definition: Coach makes comments to reassure or relax players.

Examples: "Just go nice and easy, we'll come back"

"We're going well, everything's working now"

Rules: These statements are more general than "attention focus statements". They do not provide instruction for players.

Category Definitions and Examples for Other Statements

Truisms (TI)

Definition: Comments made by the coach that are stating the obvious.

Examples: "We've just missed the last four serves!"

"We're getting killed. We've got to get the momentum back"

Rules: These comments could be observed by any spectator, and do not provide remedial assistance.

Non-sequiturs (NS)

Definition: Coach makes a statement/question unrelated to the game.

Examples: "Is everyone OK?"

"How's your ankle Dale?"

Rules: Include questions not related to skill performance or tactics.

Hustles (H)

Definition: Brief statements made by the coach (usually at the end of timeouts) to bring the team together or to reorient the focus.

Examples: "Let's go"

"Stay tough"

Rules: Are more general than "attention focus statements", in that they do not prescribe specific actions.