Teachers' perceptions of the factors influencing their engagement with statistical reports on student achievement data.

Pierce, Robyn ; Chick, Helen ; Gordon, Ian 等

Abstract

In Australia, as in other countries, school students participate in national literacy and numeracy testing with the resulting reports being sent to teachers and school administrators. In this study, the Theory of Planned Behaviour provides a framework for examining teachers' perceptions of factors influencing their intention to engage with these data. Most teachers perceived the data to be useful, but there were some negatively held views. For both primary and secondary teachers, males were more positive and had weaker perceptions of barriers to their use of data from system reports compared to females. Teachers who had studied statistics at the post-secondary level and/or attended relevant professional learning generally felt more capable of using the data, and senior teachers and principals were more favourably disposed to using these kinds of statistical reports. Many teachers had concerns about the timeliness of the data's release and the effort required to interpret them.

Keywords

Attitudes, behavioural change, factor analysis, national competency tests, statistics, teachers

Introduction

Over the past decade, data-driven decision-making for informing classroom instruction and whole school planning has been strongly promoted by government education authorities in many countries, including Australia. Adoption of data-driven decision-making, especially with a focus on nationally collected student assessment data, may require behavioural change in school principals and teachers. Adapting to the imposition of externally mandated testing processes and the abundance of performance data arising from these tests requires different approaches and practices from those enacted in the past. Underlying factors affecting each individual's response to opportunities for behaviour change are not always obvious. For education professionals, their response to an expectation that they engage deeply with statistical reports on their students' achievements will, among other factors, be influenced by their beliefs and attitudes regarding statistical data. Gal, Ginsburg, and Schau (1997) emphasised the influence of underlying beliefs on data use, and they documented inhibitions due to mathematics/statistics anxiety. Timperley (2005), who worked with New Zealand school leaders, found that teachers 'did not believe that they could influence the low literacy achievement of their students and so analysing achievement data was irrelevant to their practice' (p.1). Yates (2008) reported that many in education neither trust nor value the large-scale statistics that provide objective evidence on which to base practice. Such reports, along with our pilot study (see Pierce & Chick, 2011), underline the need to assess affective factors, in addition to statistical knowledge, if effective support for principals and teachers to gain workplace statistical literacy is to be developed.

In this paper we report on a study, framed by the Theory of Planned Behaviour, that explores teachers' attitudes, subjective norms and perceived behavioural controls that may impact on teachers' intentions to engage with system reports of student assessment data in order to inform their planning. In the sections that follow, we first consider other research on factors affecting the use of statistical data in education contexts and then give brief details of the Theory of Planned Behaviour. This background is followed by details of the study and its associated results with some discussion, before finishing with a consideration of some implications and conclusions.

Statistical literacy: beliefs and attitudes

Based on her USA research findings, Jere Confrey (2008) claimed that teachers' effective use of data to inform issues of equity and instruction requires the development of a statistical mindset. This mindset consists not only of the concepts and procedures of statistics but also of a belief in statistical inquiry as a means to address complex problems. Indeed, the broader construct of numeracy involves 'more than knowing about numbers and number operations[; it] includes an ability and inclination to solve numerical problems [and] demands familiarity with the ways in which numerical information ... is presented in graphs, charts and tables' (British National Numeracy Project, Department for Education and Employment, UK, 1998; emphasis added). Alternatively, numeracy has also been described as 'the capacity to recognise and understand the role of mathematics in the world around [us] and the confidence, willingness and ability to apply mathematics' (Australian Curriculum and Assessment and Reporting Authority, 2010, emphasis added). More specifically, statistical literacy requires sufficient knowledge and understanding of numeracy, statistics and data presentation to make valuable use of quantitative data and summary reports in a personal or professional setting, along with a positive disposition towards the use of such data (Ben-Zvi & Garfield, 2004; Watson, 2006).

Concerns have been expressed about teachers' capacity and inclination to interpret student assessment data. For example, The Australian newspaper (Ferrari, 2011) published an article headed 'Teachers are "phobic" over test data', which stated that: One of the nation's most senior education bureaucrats says that teachers lack the expertise to analyse student results, leaving them bewildered, confused and excluded from the debate over national tests and the MySchool website.... [T]he profession had been "dragged reluctantly" to the debate on measuring student achievement and was largely ambivalent about the value of analysing student test results. (p.7)

Such claims suggest that it is appropriate to examine more closely the levels of knowledge that may limit the extent to which a teacher or principal can engage with data, together with the affective factors that come into play. Even disregarding the fact that teachers may have negative attitudes towards the testing process itself, widespread community negativity about statistics may also be reflected in teachers' willingness to use data. Attitudes, beliefs and perceptions have long been identified as influencing the extent of engagement with statistical information. Wallman (1993) identified misunderstanding, misperception, mistrust and misgivings as significant 'mis-es' for statistics, and, in a general statement, The Statistical Society of Australia (2005) noted that statistics has a poor public image. Such a negative regard for statistical information, combined with low confidence in making sense of such data, may restrict the engagement of teachers and principals with data about their students. As an example of one study into affective variables and actual engagement with data, Gal et al. (1997) showed that limited confidence with quantitative data can inhibit the learning and use of statistics. This suggests that looking at statistical knowledge alone may be insufficient to understand why (and whether) teachers engage with statistical reports about their students' performance. It seems unlikely that principals and teachers will engage fully with quantitative system data--and use them as a basis for decision-making and planning unless they perceive statistics as valuable and have a degree of self-confidence about being able to use them.

Framework: the theory of planned behaviour

Using statistical data on student achievement as key evidence for decisions regarding whole school programme and classroom teaching planning requires change in many teachers' decision-making processes (Boudett & Steele, 2007; Boudett, City, & Murnane, 2005). Until recently such extensive data sets have not been available, and there has not been the same level of expectation of accountability for outcomes. Engaging with these new sources of data in a meaningful way, and in the face of greater scrutiny, has demanded new actions of teachers to make sense of and act on the information contained within the statistical results they receive. Ajzen (1991) claimed that behavioural change is closely linked to intention to change. He put forward the Theory of Planned Behaviour (TPB) as a framework for studying intention to change. The TPB is framed to allow consideration of attitudes and perceptions that may either enable, or present barriers to, a person's intention to change. Ajzen (1991) summarised the theory as follows: The theory of planned behaviour postulates three conceptually independent determinants of intention. The first is the attitude toward the behaviour and refers to the degree to which a person has a favourable or unfavourable evaluation or appraisal of the behaviour in question. The second predictor is a social factor termed subjective norm; it refers to the perceived social pressure to perform or not to perform the behaviour. The third antecedent of intention is the degree of perceived behavioural control which ... refers to the perceived ease or difficulty of performing the behaviour and it is assumed to reflect past experience as well as anticipated impediments and obstacles. As a general rule, the more favourable the attitude and subjective norm with respect to a behaviour, and the greater the perceived behavioural control, the stronger should be an individual's intention to perform the behaviour under consideration. The relative importance of attitude, subjective norm, and perceived behavioural control in the prediction of intention is expected to vary across behaviours and situations. (p.188, emphases added)

Ajzen (1991) suggested that such factors are usually found to predict behavioural intentions with a high degree of accuracy, applicable in studies across the health, social and behavioural sciences (see e.g. Armitage & Conner, 2001). Research involving TPB has consistently shown that attitudes, subjective norms and perceived behavioural controls are strongly predictive of behavioural intent. Ajzen went on to assert that research tells us that a person's intentions, in combination with perceived behavioural controls (especially if aligned with actual behavioural controls), can account for a considerable proportion of variance in actual behaviour. In terms of researching and identifying these factors, Francis et al. (2004) suggested that a useful gauge of behavioural intentions may be formed from responses to even a small number of items targeting each of the three key areas (attitudes, subjective norms and perceived behavioural controls).

Today Australian schools are provided with statistical reports that summarise and illustrate data related to various aspects of schools and their students (see for example Figure 1). Many of these reports relate to student achievement and we have chosen to refer to these as System Reports on Student Achievement (SRSA) since different teachers will be familiar with different reports depending on the system that supplies them (e.g. state and national agencies). In Australia, the best-known example of SRSA is the annual report on the National Assessment Program--Literacy and Numeracy (NAPLAN), which is provided for students in Years 3, 5, 7 and 9. In Victoria, other examples of SRSA include the Department of Education and Early Childhood Development's (DEECD) 'School Level Reports' and the VCE Data Service reports on students' Victorian Certificate of Education results. Applying the TPB in the study reported in this paper, we chose to explore teachers' (i) attitudes toward using SRSA (including NAPLAN) as a basis for making decisions regarding school programmes and classroom teaching (for example, the extent to which data are believed to highlight areas of deficit); (ii) subjective norms with respect to using data from SRSA (for example, what they perceive other staff expect); and (iii) perceived behavioural controls over their use of SRSA data (for example, whether they perceive they have sufficient statistical knowledge to interpret the data). The TPB thus provides a framework for this research.

The study

Prior to the data collection reported in this paper, a pilot study and Stage 1 data collections were conducted. The pilot survey was administered to groups of mathematics and English teachers from schools with whom we were working for another purpose (Pierce & Chick, 2011). The results from that survey suggested that, while many teachers saw value in paying attention to NAPLAN data, there was considerable variation in teachers' attitudes and perceptions related to the use of such data and that in many cases these attitudes and perceptions may have presented barriers to the teachers' intentions to use such data to inform their planning. These results justified setting up a two-stage formal investigation. This study was conducted with teachers from government schools in the Australian state of Victoria. In Victoria at the time of the study, the DEECD operated through a structure of regions further divided into networks. Stage 1 involved collecting data through an extended paper-based survey administered face-to-face and followed by focus group discussions. The paper-based survey comprised both open and closed questions about affective factors and about actually interpreting typical examples of SRSA data. The 150 participants for Stage 1 came from a cluster sample of 20 schools: 10 primary and 10 secondary (two of each, randomly selected from one randomly-selected network within each of five regions). Teachers' responses to the paper-based survey items and their comments within the focus groups then informed the development of an online survey. This revised survey comprised previously successful items and others refined to reflect the words commonly used by the teachers who participated in the focus groups, and was designed to take less than 30 min to complete. The online administration of this substantive survey formed Stage 2 of our investigation and the results of one section, about the affective factors, are reported in this paper. From an initial list of government schools supplied by the DEECD, language schools, special schools and the 20 schools that had participated in a pilot study for this project were removed. This left a sample frame of 1412 schools. A simple random sample of 104 schools was chosen. The online survey was made available to all teaching staff at these selected primary and secondary schools in an attempt to obtain data from approximately 1000 teachers from across the Australian state of Victoria. The full survey included: (i) demographic items, including questions about last formal study of statistics and attendance at relevant professional development programmes; (ii) items probing the extent of teachers' access to data; (iii) the Data Use Perceptions instrument based on TPB, containing the affective items that are the focus of this report; and (iv) questions to assess teachers' professional statistical literacy (not reported here). For the items that form the focus of this paper (shown in Table 2), the teachers were asked to indicate, on a five-point Likert scale, their level of agreement or disagreement with 18 statements. The first seven items related to attitudes, the next four probed the teachers' subjective norms, and the last seven focused on teachers' perceived behavioural controls.

[FIGURE 1 OMITTED]

Data analysis

Two questions from the survey were analysed, using the full cluster sample structure of the data. It was found that there was little evidence of statistical clustering, with design effects (on the standard error scale) of less than 1.06, and we therefore have analysed the data assuming it is, in effect, a simple random sample.

First, frequency data were collated for the demographic variables. Second, Principal Components Analysis (PCA) was undertaken on the responses to the 18 items that make up the Data Use Perceptions instrument. The goal of this analysis was to reveal structure in the data. PCA involves extracting linear composites of observed variables. This approach was chosen because it is relatively simple both computationally and in terms of understanding. Once components had been confirmed, factor scores were evaluated for each respondent and descriptive statistics calculated for each factor. The effect, if any, of the various demographic variables was analysed by comparing mean factor scores using t-tests (two groups) or ANOVA (more than two groups). Multivariate analyses were explored, but combinations of variables did not add significance to the explained variability and so only the results from the simple analysis are reported. All statistical calculations were performed using SPSS PASW Statistics 18 (n.d.). Summary results are presented in the section that follows and further details are provided in the Appendix.

Results

Of the 104 schools randomly selected for the study, 63 agreed to participate and all but one forwarded the request to participate to staff. Four of the selected schools did not yield any respondents by the end of the survey period. In all, from the invited schools, 704 of a possible 1835 teachers (38%) provided responses. Some key attributes of the resulting sample are summarised in Table 1. The response rate for later items in the survey varied. For the 18 items framed on the TPB (the Data Use Perceptions instrument shown in Table 2), the number of respondents varied from 649 to 656. More than two-thirds of the respondents were females and the majority (55%) worked in secondary schools. Table 1 suggests this breakdown is similar to the corresponding demographic across all schools according to the DEECD statistics for 2011. Graduate teachers (generally those early in their careers whose focus is mainly on their own classrooms) may be under-represented and expert teachers (more senior teachers, with wider school responsibilities) over-represented in our random sample. The secondary teachers who responded to the survey had a diversity of teaching across all secondary level subject disciplines. The respondents varied in their 'years of teaching', with the most common (29%) category of length of service being three to 10 years.

In terms of their stated background in statistics as a basis for working with data, almost half of the respondents indicated that they had studied at least some statistics at the postsecondary level, although this may only have been as a topic within some other subject. However, it is pertinent to note that 21% indicated that they had never studied statistics and a further 10% had not studied statistics after the compulsory years of schooling. This experience was different for primary and secondary teachers. Of the primary teachers, 27% had never studied statistics compared with 16% of the secondary teachers. At the other end of the statistical experience scale, 35% of the primary but 58% of the secondary teachers had studied at least some statistics at the post-secondary level. Most of the survey respondents had attended at least one professional learning programme related to data (67% of primary and 82% of secondary teachers). In most cases, these professional learning programmes targeted understanding of SRSA data and were often prepared and presented by the data service providers.

The survey items probing attitudes, subjective norms and perceived behavioural controls are shown in Table 2, together with the percentage of respondents who chose agree (A) or strongly agree (SA) (combined) for each item (a 5-point Likert scale was used). For items 1-11, agreement or strong agreement with the statement would suggest that the teacher is likely to engage with the data, but for items 12-18 the reverse is the case.

The strongest positive responses were for items 3 and 6, indicating strong agreement that SRSA are useful for identifying topics in the curriculum that need attention and for whole school planning. Despite this positive outcome, some caution should be noted. Although for most attitude items those respondents who chose a neutral or negative response are in the minority, the numbers are still sufficient that this group could be very influential. TPB suggests that teachers who hold these attitudes will be less likely to engage with the data, and the data suggest that around 20% of teachers are in this category.

The majority of the teachers appear to feel confident about interpreting the SRSA. Despite the limited statistics background of nearly one-third of the respondents, only 16% and 15% respectively agreed or strongly agreed with statements 12 and 13 suggesting they 'don't have a good maths brain' and cannot 'adequately interpret the SRSA' they receive.

Confirming factors

The Data Use Perception items shown in Table 2 were subjected to principal components analysis (PCA) using SPSS-PASW Statistics 18 (n.d.). Prior to performing the PCA, the suitability of the data for this analysis was assessed. Inspection of the correlation matrix revealed the presence of many correlations of 0.3 and above, and the Kaiser-Meyer-Olkin value was 0.83. Both values support the factorability of the correlation matrix. A PCA indicated four components that had obvious interpretations. They also had eigenvalues greater than 1 and explained 24.8%, 15.3%, 9.2%, and 6.3% of the variance, respectively, and 55.6% in total. The resulting weightings on each of the factors are shown in Table 2, with weights with absolute values of 0.45 and above indicated in bold. The factors that emerged can be named in terms of the underlying TPB framework: Factor 1--Attitudes; Factor 2--Perceived Behavioural Controls; and Factor 3--Subjective Norms. The structure confirms the original survey design but Item 9 ('My students' parents take an interest in our school's SRSA') may be excluded as it did not contribute greatly to any component (see Table 2). A fourth component identified by the PCA has only one item, a statement targeting teachers' perception of the timing of the release of results--a behavioural control over which the teachers have no control. Since this item stands alone, we have not counted it as a factor for the purpose of further analysis.

Factor scores

Factor scores for each respondent were created using an additive model with values from the components matrix (Table 2) associated with each component score (indicated by bold) used as weights. Zero weights were effectively assigned to the non-bold values listed against items not included in the specified factor score. The descriptive statistics for each factor are shown in Table 3. For each factor, the minimum and maximum scores recorded coincide with the lowest and highest weighted scores available (rounded to one decimal place). On average, teachers had a positive response to both attitude and subjective norm items.

Links between factors and demographics

Links between each of the three main factors identified in Table 3, and the six demographic attributes reported in Table 1 were investigated by comparing group means. The items that made the strongest contributions to the Data Use Perceptions scale were those assigned in the factor analysis to attitude and perceived behavioural controls, but details for all three factors are given below. Independent t-tests (for the variables gender, school type and attended relevant professional learning) and ANOVAs (on years of teaching, last statistics formally studied and DEECD teaching classification) were conducted to explore the impact of demographic characteristics on each factor. The results of these tests are summarised in Table A1, with further details in Tables A2 to A10 found in the Appendix. For each of the statistically significant results below, the effect sizes were small (Cohen's d in the range 2.1-3.3). From a practical perspective, many of these differences are small.

Attitude. The construct Attitude is generated by aggregating responses using the weights from Component 1 in Table 2, so scores are determined predominantly by responses to items l through 7 and item 17. Persons with a high score on this construct tend to agree with statements 1 through 7 and to disagree with statement 17, indicating that they see SRSA as useful to them in a range of ways (learning more about their students, grouping students, identifying curriculum topics that need attention, planning lessons and so on). Persons with low scores on this construct value the information less and are less inclined to see it as useful.

On average, males held more positive attitudes than females ([[bar.X].sub.f] = 14.8, [[bar.X].sub.m] = 15.6), secondary school teachers held more positive attitudes than primary school teachers ([[bar.X].sub.p] = 14.6, [[bar.X].sub.s] = 15.4) (and those teachers who had attended a relevant professional learning programme were more likely to show more positive attitudes than those who had not attended ([[bar.X].sub.PL] = 15.2, [[bar.X].sub.NPL] = 14.5). In addition, there was a difference related to DEECD classification: principals and leading teachers were more positive than graduate, accomplished or expert teachers ([[bar.X].sub.G] = 14.7, [[bar.X].sub.A] = 14.8, [[bar.X].sub.E] = 14.6, [[bar.X].sub.L] = 16.4, [[bar.X].sub.P] = 16.6). However, there was no significant difference related to either the last time teachers had formally studied statistics or their number of years of teaching.

It was conjectured that the lower mean attitudes from primary teachers were associated with the high proportion of female primary teachers. There is a statistically significant association between gender and school type ([chi square] (df= 1)= 23.92, p < 0.001). While 51% of the female teachers and 31% of the male teachers worked in primary schools, 79% of the primary teachers and 61% of the secondary teachers were females. However, the interaction between gender and school type for attitude was not statistically significant (F(df= 1, 625) = 3.093, p = 0.079). As illustrated by Figure 2(a), there is not clear evidence regarding any interaction between gender and school type affecting attitudes. Similarly, it was conjectured that gender imbalance (more males that females) may explain the significantly higher mean attitude scores from principals. There is a statistically significant association between gender and DEECD classification ([chi square] (df= 4) = 11.05, p = 0.026). For Graduate, Accomplished and Expert classifications, 75%, 72%, and 70%, respectively, were females while for the Leading teacher and Principal classification, this dropped to 64% and 53%. However, the interaction between gender and DEECD classification for attitude was not statistically significant (F(df= 4, 619)= 0.390, p = 0.816). As illustrated in Figure 2(b), there is not clear evidence regarding any interaction between gender and DEECD classification affecting attitudes.

Subjective norms. The construct Subjective Norm is generated by aggregating responses using the weights from Component 3 in Table 2, so scores are determined predominantly by responses to items 8, 10 and 11. Persons with a high score on this construct tend to agree with these statements, indicating that they perceive that their colleagues, school leadership and 'the Department' expect them to engage with SRSA.

Perception of subjective norms did not vary significantly with gender, school type or last statistics studied. It appeared to be a stronger factor for those who had attended a relevant professional learning programme ([[bar.X].sub.PL] = 6.6, [[bar.X].sub.NPL] = 6.2), had been teaching for longer ([[bar.X].sub.<3] = 6.1, [[bar.X].sub.3-10] = 6.3, [[bar.X].sub.10-20] = 6.6, [[bar.X].sub.20-30] = 6.7, [[bar.X].sub.>30] = 6.9) or were in a higher DEECD classification ([[bar.X].sub.G] = 6.1, [[bar.X].sub.A] = 6.3, [[bar.X].sub.E] = 6.6, [[bar.X].sub.L] = 6.7, [[bar.X].sub.P] = 7.0).

[FIGURE 2 OMITTED]

Perceived behavioural controls. The construct Perceived Behavioural Control is generated by aggregating responses using the weights from Component 2 in Table 2, so scores are determined predominantly by responses to items 12 through 16. Persons with a high score on this construct tend to agree with these statements, indicating that they expect to have difficulty dealing with SRSA data due to barriers such as their own perceived ability, lack of confidence, workload and time constraints. These teachers are less likely to engage with such system data.

Perceived behavioural controls might militate against teachers engaging in the use of SRSA data (these are issues that are perceived to exert external influence over behaviour). This factor appeared to be stronger in females than males ([[bar.X].sub.f] = 8.4, [[bar.X].sub.m] = 7.7), and in those who had not ever attended a relevant professional learning programme ([[bar.X].sub.PL] = 8.1, [[bar.X].sub.NPL] = 8.6). There were no evident differences associated with school type and years of teaching. Those who had studied statistics at post-secondary level rated lower on this factor ([[bar.X].sub.n] = 8.9, [[bar.X].sub.<10] = 9.0, [[bar.X].sub.11/12], = 8.2, [[bar.X].sub.PostS] = 7.8), as did principals and leading teachers ([[bar.X].sub.G] = 8.6, [[bar.X].sub.A] = 8.2, [[bar.X].sub.E] = 8.5, [[bar.X].sub.L] = 7.6, [[bar.X].sub.P] = 7.4).

Implications and conclusions

The 18-item Data Use Perceptions instrument clearly identified three factors that are of practical importance with regard to teachers' potential engagement with system reports of student achievement, such as government literacy and numeracy testing results. These confirm the theoretical underpinnings of the research, as they clearly reflect the three 'determinants of intention' in the Theory of Planned Behaviour, namely attitudes, subjective norms and perceived behavioural controls. The teachers" attitudes are generally positive and reflect their perceptions of the usefulness of the data, but the teachers are sensitive to subjective norms in that they feel some social pressure to engage with data. Finally, there are both locally imposed and systemically imposed behavioural controls (self-perceived ability to handle data, and the external constraint of the timing of the release of results) that are also involved. All of these are of practical importance and shifts in teachers' perceptions of any of these factors may influence their engagement with statistical data. We had expected behavioural control and subjective norms to have a strong influence, based on some strongly-held opinions expressed in responses from some of the participants in the focus groups in Stage 1 of the study; however, the results from the survey suggest that these are influential but not as strong as we anticipated.

The results of the study indicate that teachers appear to be relatively confident about working with SRSA data, with 63% disagreeing or strongly disagreeing with the statement 'I don't feel I can adequately interpret data'. Two words of caution are necessary, however. First, it should be noted that analysis of some of the statistical literacy items in the later part of the online survey indicates that there are significant issues with the adequacy of teachers' data interpretation, with less than 20% of respondents able to accurately interpret key features of one of the common forms of data representation in the reports (Pierce & Chick, 2013). Second, there is still a large proportion of teachers lacking in confidence, with over one-third of teachers expressing a neutral response or agreement with statements like 'I don't have a good maths brain' and 'I don't feel I can adequately interpret the SRSA I receive at our school'. Indeed, only 38% of the teachers claimed they disagreed with the idea that the amount of data presented is overwhelming and only 41% suggested they could deal with data relatively efficiently. The data show, then, that although confidence dominates, for some it is misplaced, and there are still many teachers lacking confidence or fluency. With TPB suggesting that intentions to use data are influenced by these attitudes, norms and perceived behavioural controls, and with Confrey (2008) identifying the importance of a 'statistical mindset' and Gal et al. (1997) implying a relationship between confidence and use of statistics, our results suggest that there is still reason to be concerned about whether teachers will be willing and/or able to use SRSA effectively.

It is likely that the need of teachers to be able to engage with data about their students' performance will increase in coming years. If such data are to be put to effective educational use, then teachers need to have the willingness and capacity to engage with this statistical information, to see it as valuable, and to have the statistical literacy to interpret it. The results of this survey suggest that, while a majority of teachers intend to engage with the SRSA data, there is still a large number of teachers who are negatively disposed towards the data. These teachers lack confidence in dealing with the data or have concerns about the usefulness of the data (for example, because of its late arrival, which may be perceived to limit its applicability to the students to which it applies because they have moved on). Clearly, there is scope to increase teachers' confidence (and knowledge) in data handling and improve their perceptions of its professional value for them and so increase the likelihood that they will engage with the data successfully.

Conflicting interests

This research was partly funded by the Victorian Department of Education and Early Childhood Development and the Victorian Curriculum and Assessment Authority, which are involved in the production and dissemination of student achievement data.

Funding

This research has been funded by the Australian Research Council (LP100100388), the Victorian Department of Education and Early Childhood Development, and the Victorian Curriculum and Assessment Authority.

DOI: 10.1177/0004944113496176

Appendix

Summary of statistical comparison of mean factor scores

In Tables A2 to A8, the idea of homogeneous subsets of means indicates that any means marked by the same superscript as another mean are in a subset for which those means are not statistically significantly different from each other. Table A1. Results of comparison of means using t-tests. N Mean SD t p Gender Attitude Female 435 14.8 3.3 -2.58 0.010 Male 194 15.6 3.3 Subjective norm Female 443 6.5 1.2 0.46 0.648 Male 204 6.4 1.2 Behavioural control Female 446 8.4 2.4 3.81 <0.001 Male 200 7.7 2.3 Timing control Female 445 2.1 0.6 2.64 0.009 Male 204 2.0 0.6 School type Attitude Primary 273 14.6 3.5 -2.71 0.007 Secondary 356 15.4 3.1 Subjective norm Primary 279 6.5 1.2 1.33 0.184 Secondary 368 6.4 1.3 Behavioural control Primary 280 8.2 2.3 0.10 0.923 Secondary 366 8.2 2.5 Timing control Primary 280 2.2 0.6 4.70 <0.001 Secondary 369 2.0 0.6 Attended relevant professional learning? Attitude Yes 476 15.2 3.3 2.30 0.022 No 152 14.5 3.3 Subjective norm Yes 489 6.6 1.2 3.42 0.001 No 157 6.2 1.2 Behavioural control Yes 489 8.1 2.4 -2.60 0.010 No 156 8.6 2.4 Timing control Yes 492 2.1 0.6 0.92 0.361 No 156 2.1 0.6 Table A2. Years of teaching and attitude: results of comparison of means using ANOVA. ANOVA N = 629 F = 0.67 p = 0.616 Years n Mean SD < 3 122 14.8 (a) 2.99 3-<10 184 14.9 (a) 3.31 10-<20 106 15.4 (a) 3.28 20-<30 135 15.2 (a) 3.49 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A3. Years of teaching and subjective norms: results of comparison of means using ANOVA. ANOVA N = 647 F = 6.57 P < 0.001 Years n Mean SD <3 127 6.14 (a) 1.01 3-<10 186 6.32 (ab) 1.25 10-<20 107 6.57 (abc) 1.23 20-<30 140 6.63 (bc) 1.19 >30 87 6.83 (c) 1.41 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A4. Years of teaching and perceived behavioural controls: results of comparison of means using ANOVA. ANOVA N = 646 F = 2.08 p = 0.082 Years n Mean SD <3 127 8.7 (a) 2.35 3-<10 187 8.0 (a) 2.22 10-<20 107 8.3 (a) 2.53 20-<30 137 8.1 (a) 2.46 >30 88 7.9 (a) 2.64 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A5. Teacher classification and attitudes: results of comparison of means using ANOVA. ANOVA N = 629 F = 6.87 p < 0.001 Classification n Mean SD Graduate 110 14.7 (a) 3.03 Accomplished 194 14.9 (a) 3.23 Expert 209 14.6 (a) 3.59 Leading 66 16.4 (b) 2.92 Principal 50 16.6 (b) 3.17 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A6. Teacher classification and subjective norms: results of comparison of means using ANOVA ANOVA N = 647 F = 7.22 p < 0.001 Classification n Mean SD Graduate 114 6.1 (a) 1.14 Accomplished 196 6.3 (ab) 1.21 Expert 211 6.6 (b) 1.21 Leading 70 6.6 (abc) 1.38 Principal 56 7.1 (c) 1.10 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A7. Teacher classification and perceived behavioural controls: results of comparison of means using ANOVA. ANOVA N = 646 F = 4.94 p = 0.00 1 Classification n Mean SD Graduate 115 8.6 (a) 2.37 Accomplished 197 8.1 (ab) 2.27 Expert 208 8.5 (a) 2.44 Leading 69 7.7 (ab) 2.41 Principal 57 7.3 (b) 2.56 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A8. Last statistics studied and attitude: results of comparison of means using ANOVA. ANOVA N = 629 F = 2.26 p = 0.081 Last stats n Mean SD Never 130 14.7 (a) 3.64 [less than or equal to] Year 10 60 15.1 (a) 2.73 Year 11/12 137 14.7 (a) 3.20 Post secondary 302 15.4 (a) 3.37 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A9. Last statistics studied and subjective norms: results of comparison of means using ANOVA. ANOVA N = 647 F = 0.87 p = 0.458 Last stats n Mean SD Never 134 6.5 (a) 1.34 [less than or equal to] Year 10 62 6.5 (a) 1.10 Year 11/12 137 6.3 (a) 1.15 Post secondary 314 6.5 (a) 1.25 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript. Table A10. Last statistics studied and perceived behavioural controls: results of comparison of means using ANOVA. ANOVA N = 646 F = 9.0 1 p < 0.001 Last stats n Mean SD Never 133 8.8 (a) 2.70 [less than or equal to] Year 10 64 9.0 (a) 1.94 Year 1 1/12 137 8.2 (ab) 2.46 Post secondary 312 7.8 (b) 2.26 Homogeneous subsets of means (based on Bonferroni post hoc tests, [alpha] = 0.05) are indicated by the same letter in the superscript

Acknowledgements

We wish to acknowledge the contribution of other members of the research team: Roger Wander, Sue Helme (University of Melbourne), Jane Watson (University of Tasmania), Michael Dalton, Magdalena Les (VCAA) and Sue Buckley (DEECD).

References

Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 197-211.

Armitage, C., & Conner, M. (2001). Efficacy of the theory of planned behaviour: a meta-analytic review. British Journal of Social Psychology, 40, 471-499.

Australian Curriculum and Assessment and Reporting Authority. (2010). The Australian National curriculum. numeracy. Retrieved from http://www.australiancurriculum.edu.au/ GeneralCapabilities/Numeracy.

Ben-Zvi, D., & Garfield, J. (Eds.). (2004). The challenge of developing statistical literacy, reasoning and thinking. Dordrecht, The Netherlands: Kluwer Academic Publishers.

Boudett, K., City, E. A., & Murnane, R. J. (2005). Data wise: a step-by-step guide to using assessment results to improve teaching and learning. Cambridge, MA: Harvard Education Press.

Boudett, K., & Steele, J. L. (2007). Data wise in action: stories of schools using data to improve teaching and learning. Cambridge, MA: Harvard Education Press.

Confrey, J. (2008). Teaching teachers to use data to inform issues of equity and instruction. In P. Ernest (Ed.), Philosophy of Mathematics Education Journal (Special issue on justice, part 1), Vol. 20. Retrieved from http://people.exeter.ac.uk/PErnest/pome20/.

Department for Education and Employment (DfEE). (1998). The implementation of the National Numeracy Strategy: the final report of the numeracy task force. London: Author.

Ferrari, J. (2011). Teachers are 'phobic' over student test data. The Australian, Jul 15, p. 7.

Francis, J., Eccles, M., Johnston, M., Walker, A., Grimshaw, J., Foy, R., ... Bonetti, D. (2004). Constructing questionnaires on the theory of planned behaviour: a manual for health services research. Newcastle-upon Tyne, UK: University of Newcastle.

Gal, I., Ginsburg, L., & Schau, C. (1997). Monitoring attitudes and beliefs in statistics education. In I. Gal, & J. B. Garfield (Eds.), The assessment challenge in statistics education (pp. 37-51). Amsterdam: IOS Press and the International Statistics Institute.

Pierce, R., & Chick, H. (2011). Teachers' intentions to use national literacy and numeracy assessment data: a pilot study. The Australian Educational Researcher, 38, 433-447.

Pierce, R., & Chick, H. (2013). Workplace statistical literacy for teachers: Interpreting box plots. Mathematics Education Research Journal, 25, 189-205.

SPSS (n.d.). SPSS Prediction analysis software. Retrieved from http://www-01.ibm.com/software/ analytics/spss/.

Statistical Society of Australia Inc. (2005). Statistics at Australian universities: an SSAI sponsored review. ACT: Author.

Timperley, H. S. (2005). Instructional leadership challenges: the case of using student achievement information for instructional improvement. Leadership and Policy in Schools, 4, 3-22.

Wallman, K. K. (1993). Enhancing statistical literacy: enriching our society. Journal of the American Statistical Association, 88, 1-8.

Watson, J. M. (2006). Statistical literacy at school. Mahwah, NJ: Lawrence Erlbaum Associates.

Yates, G. (2008). Roadblocks to scientific thinking in educational decision making. Australasian Journal of Special Education, 32, 125-137.

Robyn Pierce

Melbourne Graduate School of Education, University of Melbourne, Melbourne, Australia

Helen Chick

Faculty of Education, University of Tasmania, Tasmania, Australia

Ian Gordon

Statistical Consulting Centre, University of Melbourne, Melbourne, Australia

Corresponding author:

Robyn Pierce, Mathematics Education Academic Group, Melbourne Graduate School of Education, University of Melbourne, Melbourne, VIC 3010, Australia. Email: r.pierce@unimelb.edu.au Table 1. Summary information about respondents (rounded percentages) and approximate overall DEECD percentages. Study DEECD Item Response sample (%) (%) Gender Female 69 70 Male 31 30 School type Primary 45 45 Secondary 55 55 DEECD teacher Graduate teacher 13 20 classification (by Accomplished teacher 31 33 increasing seniority) Expert teacher 47 35 Leading teacher 9 12 Principal 9 (a) Study sample (%) Years of teaching <3 20 [greater than or equal to] 3 but <10 29 [greater than or equal to] 10 but <20 17 [greater than or equal to] 20 but <30 21 [greater than or equal to] 30 13 Last formal study of Never 21 statistics as a Year 10 or earlier 10 subject or a topic Year 11/12 21 within a subject Post-secondary 48 Attended professional Yes 75 learning programme(s) No 25 related to the use of SRSA (a) Sample reflected an average of 1 per school. Table 2. Item statements for the data use perceptions instrument and percentage of respondents across all schools component matrix for the data use perceptions instrument. Determinants of intention Item (as planned) no. Items % Agree Attitudes 1 SRSA tell me things about 55 my students that I had not realised. 2 SRSA are helpful for 67 grouping students according to ability. 3 SRSA are useful for 82 identifying topics in the curriculum that need attention in our school. 4 SRSA are useful for 64 identifying an individual student's knowledge. 5 SRSA are helpful for 58 planning my lessons. 6 SRSA are useful to inform 79 whole school planning. 7 NAPLAN tests are well- 28 designed to assess our students' achievement. Subjective 8 Student achievement data 84 norms are something that my school's leadership team expect me to pay close attention to. 9 My students' parents take 29 an interest in our school's SRSA. 10 At our school if you don't 30 make use of SRSA then you are not seen as a good team player. 11 At our school we are 51 under pressure from the Department to respond to SRSA. Perceived 12 I don't have a good maths 16 behavioural brain. controls 13 I don't feel I can 15 adequately interpret the SRSA I receive at our school. 14 SRSA take too long to 29 interpret. 15 The amount of data 34 presented in the SRSA I see is overwhelming. 16 Practical constraints mean 15 that it is not possible to change teaching in my area in response to SRSA. 17 I find that most SRSA are 12 not relevant to my teaching. 18 The timing of the release 54 of NAPLAN reports reduces their usefulness. Determinants Components of intention Item (as planned) no. 1 2 3 4 Attitudes 1 0.62# 0.28 -0.21 0.09 2 0.68# 0.32 -0.13 0.18 3 0.69# 0.19 0.07 0.20 4 0.65# 0.30 -0.18 0.09 5 0.75# 0.24 -0.17 -0.02 6 0.70# 0.17 0.07 0.14 7 0.58# 0.12 -0.20 -0.29 Subjective 8 0.42 0.20 0.53# 0.03 norms 9 0.35 0.16 0.20 -0.27 10 0.10 0.21 0.63# -0.45 11 0.02 0.35 0.69# -0.11 Perceived 12 -0.22 0.52# -0.20 -0.41 behavioural controls 13 -0.32 0.70# -0.23 -0.21 14 -0.39 0.71# -0.06 0.13 15 -0.35 0.74# -0.11 0.10 16 -0.42 0.48# -0.02 0.27 17 -0.64# 0.24 0.05 0.07 18 -0.12 0.15 0.45 0.58# Items are listed in order of the determinants of intention that they were intended to address. In constructing scales, all items except 9 and 18 were included as intended. % Agree records the aggregated percent of 'Agree' and 'Strongly Agree' responses to each item. Percentages have been rounded to the nearest whole number. Weightings with absolute values greater than 0.45 are recorded in bold. Note: Weightings with absolute values greater than 0.45 are indicated with #. Table 3. Descriptive statistics for three factor scores. No. N items Minimum Maximum Mean SD Attitude 629 8 1.5 22.7 15.1 3.3 Subjective norm 647 3 1.8 9.2 6.5 1.2 Perceived behavioural control 646 5 3.2 15.7 8.2 2.4