Testing spelling: how does a dictation method measure up to a proofreading and editing format?

Daffern, Tessa ; Mackenzie, Noella Maree ; Hemmings, Brian 等

Introduction

School teachers are held accountable for the development of students' language and literacy skills by the education community, policy administrators, parent bodies and the media (Snyder, 2009). Moreover, the emphasis on accountability and data-based decision making in schools has led to an increased need for reliable and sensitive measures of academic performance (Al Otaiba & Hosp, 2010). Assessment comes in different forms and can serve several purposes: (a) identify strengths and needs in student academic achievement; (b) measure improvements in performance over time; and (c) determine the efficacy of instructional approaches (Kohnen, Nickels & Castles, 2009; Westwood, 2005).

This paper is concerned with the assessment of Standard English spelling achievements. Planning and implementing optimal and targeted instruction in spelling is largely contingent upon adequate spelling assessment (Kohnen et al., 2009). However, it has been argued that existing measures of spelling performance are 'not sufficiently structured or standardised to provide the reliable, sensitive data that teachers need to plan instruction' (Al Otaiba & Hosp, 2010, p. 4).

Specifically, the study reported here aimed to establish if there is a relationship between performance as measured by a proofreading and editing format, and performance based on a dictation format.

Issues with assessing competency in spelling

In Australia, popular spelling assessment tools have broadly been characterised by dictation formats (see, for example, Westwood, 2005), as well as proof reading and editing formats (see, for example, Australian Curriculum, Assessment & Reporting Authority (ACARA), 2016). However, researchers such as Critten, Pine and Messer (2013) and Willett and Gardiner (2009) contend that these approaches measure different aspects of spelling. From their analysis of the proofreading and editing format adopted in the annual Australian National Assessment Program-Literacy and Numeracy (NAPLAN) to measure spelling performance of students in Years 3, 5, 7 and 9 since 2008 (ACARA, 2016), Willett and Gardiner (2009) assert that the format does not accurately measure student knowledge of the spelling system. They claim that assessments based on production from dictation have 'fewer confounding variables' because students can 'focus all their cognitive resources on the activity of spelling a single word at a time' (Willett & Gardiner, 2009, p. 15). They also suggest that 'students have higher facility rates' when completing dictation tasks, while proofreading tasks are challenging because of 'readability' issues, including difficulty with the identification of 'misspelling cues' (spelling errors) even before any correction is made (Willett & Gardiner, 2009, p. 15). In contrast, Critten et al. (2013) argue that the cognitive demands are greater in dictation production tasks. Specifically, they found that the mean number of words correctly recognised by the children in their study (n=101 aged 4 to 6 years) was higher than the mean number of words correctly produced (Critten et al., 2013, p. 206). Consequently, they assert that 'representations may be more advanced for recognition' tasks than production tasks 'even though the type of knowledge required for both tasks is arguably the same' (p. 202).

Qualitative methods of spelling assessment have also been adopted. For example, Sharp et al. (2008, p. 213) gathered responses from students' retrospective self-reports, by asking them questions such as 'How did you spell--?' and 'What did you do to decide on those letters?'. Coding and analysis of their data resulted in the identification of behaviours and strategies used when spelling, providing valuable insights into the cognitive processes that underpin spelling. In addition, spelling assessment has involved analyses of spelling errors observed in free compositional writing (see, for example, Bahr, Silliman, Berninger & Dow, 2012). Although observations of compositional writing can yield important insights regarding the extent to which spelling knowledge may be applied when composing such texts, an analysis of spelling errors is limited to the words a student chooses to include. As Kohnen et al. (2009) point out, some students may avoid words that are problematic for them to spell, or their word choice may be restricted on the basis of topic, genre and targeted audience. Moreover, students' level of receptive and expressive vocabulary knowledge may influence word selection (Bahr, Silliman & Berninger, 2009). Kohnen et al. (2009) encourage the use of diagnostic tests to identify strengths and weaknesses in students' spelling. In particular, they caution against the sole use of tools that are based on observations of free writing samples or purely on real word spelling tests, with the assumption that such measures may lead to an underestimation of spelling difficulties because some students may have come to remember how to spell certain words that are tested. Kohnen et al. (2009) instead promote the application of a variety of spelling assessment tools, and in particular, those that are based on nonwords (pseudo words, or nonsense words). The use of non-word measures allows the assessment of various linguistic skills involved in spelling to be measured and remove the possibility of whole-word knowledge.

Theoretical considerations

Spelling assessment regimes have traditionally been based on stage or phase theories of spelling development or have tended to provide a summary of words that are correct and those that are not (see, for example, Bear, Invernizzi, Templeton & Johnston, 2012; Ehri, 2005; Gentry, 2012). Given that the types of information gathered by assessment measures can influence instructional approaches in spelling, it is imperative that they reflect contemporary perspectives of learning to spell. According to Bahr et al. (2012, p. 1587), 'it takes a long time to develop a robust ... lexicon that coordinates phonology, orthography and morphology and supports word-specific, conventional spelling'. With this in mind, methods of assessment should seek to capture such linguistic nuances in children's spelling. Triple Word Form Theory (TWFT) offers a non-linear stance of learning to spell, contending that students are capable of drawing concurrently on phonological, orthographic and morphological skills from the early years of learning to write (Berninger, Abbott, Nagy & Carlisle, 2010; Richards et al., 2006). TWFT has been validated in a series of brain imaging studies (see for example, Berninger et al., 2010) and behavioural studies (Garcia, Abbott & Berninger, 2010; Nagy, Berninger & Abbott, 2006), offering a potentially innovative and well-grounded framework from which to assess proficiency in spelling. According to TWFT, spelling is an essential word formation process and product of writing, and requires the coordination of three linguistic codes. These codes have been succinctly defined by Bahr, Silliman, Danzak and Wilkinson (2015):

1. The phonological code . functions as an analyser of phonemes in spoken words;

2. The orthographic code . serves to analyse letters, letter groups, and larger letter patterns in written words; [and]

3. The morphological code . analyses root words, prefixes, and inflectional and derivational suffixes in both spoken and written words. (p. 74).

Words with phonologically regular constituents may require encoding of individual phonemes into their corresponding grapheme units (Garcia et al., 2010). For example, the individual letters in the word rob can be written by encoding each phoneme using a corresponding letter (r-o-b). While many disyllabic and polysyllabic words may be morphologically complex, they may still contain constituents that are phonologically regular. For example, when encoding the phonologically regular medial graphemes in the word recognition (-gni-), accurate blending of those three medial phonemes needs to occur at a syllable juncture. The polysyllabic nature of this word may pose difficulties with phonological encoding. Simultaneously, application of the rules that govern how affixes attach to base words or roots (e.g., the derivational suffix, ion, in recognition) may be required when spelling polysyllabic words. Indeed, as Apel (2014) points out, the determination of whether or not affixes are correctly applied (e.g., recognition/recognishun) when spelling is one plausible method of measuring morphological awareness.

Not all words in the English language contain one-to-one phonological correspondence, but they do follow orthographic conventions. Specifically, 'positional constraints' is a term often used to explain how the positioning of a particular phoneme within a word determines how the phoneme is likely to be orthographically represented (Bahr, 2015; Holmes & Ng, 1993; Treiman & Kessler, 2006). For example, it is possible to use vowel doublets in initial, medial or final positions of words, as in eerie, need and tree. Consonant doublets can plausibly be used in medial positions of words (as in the word, bottle) and final positions (as in the word, fall) but they are very rarely used in the initial position of words (as in the word, llama) (Read & Treiman, 2013). Potential orthographic confusions can also occur when representing the ou versus ow diphthong (as in out or how). In this instance, the correct spelling choice for the diphthong is determined by specific positional constraints. That is, the graphemes ow are required if the diphthong is present in the final position of a word (as in how) or if followed by the grapheme l (as in growl), or a single n (as in town). For the majority of other words constituting the same diphthong, the graphemes ou are usually needed instead (in words such as ground, stout and couch). The region of the brain responsible for processing the orthographic word form is thought to be sensitive to such letter sequences rather than the visual shapes of individual letters (Richards et al., 2006). Developing heightened orthographic sensitivity requires knowledge of the 'legal' (conventional) letter patterns within words (Conrad, Harris & Williams, 2013).

Critically, while systematic linguistic error analysis of encoded words has the potential to identify specific breakdowns in spelling (Silliman, Bahr & Peters, 2006), assessment systems should seek to encapsulate the phonological, orthographic and morphological complexities associated with spelling (Bahr, 2015). Assessment in spelling based on an error analysis of written words should therefore consider utilising words that are monosyllabic, disyllabic and polysyllabic but also feature common phonological regularities, legal letters sequences and morphemic complexities.

The study

The present study forms part of a larger mixed-methods study which compared the spelling performance of low-achieving spellers and high-achieving spellers in Years 3 to 6. The primary purpose of the study discussed in this paper was to determine whether there is a relationship between performance as measured by a proofreading and editing method, and performance as measured by a dictation format. In doing so, we also build on an earlier study which sought to develop and test the reliability of a dictation-based spelling assessment tool informed by TWFT: the Components of Spelling Test (CoST) (Daffern, Mackenzie & Hemmings, 2015). A detailed description of the development and initial testing of the CoST, including the theoretical framework that informed the study, as well as the process of selecting words and items can be found in Daffern et al. (2015).

The study reported here aimed to examine the relationships between two different measures of spelling performance. In this paper, results are reported in response to the following research questions:

1. What are the relationships between students' spelling results, as measured by NAPLAN spelling and the linguistic components of spelling, as measured by the CoST, for low-achieving spellers and high-achieving spellers?

2. Are these relationships affected by gender and/or year level?

Sampling

Data reported in this paper derive from a sub-sample of students in Years 3 to 6 (see Table 1) who participated in a larger mixed-methods study (n=l,198) in 2013. For the present study, eight schools from the Australian Capital Territory (ACT) were randomly selected to participate. Following ethics approval from the researchers' university Human Research Ethics Committee and the ACT Catholic school system, informed written consent was obtained from the participating school principals, teachers, students and their parents. The sub-sample represents those students identified as low-achieving spellers (Year 3, n=71; Year 4, n=56; Year 5, n=55; Year 6, n=55) and high-achieving spellers (Year 3, n=74; Year 4, n=61; Year 5, n=78; Year 6, n=62). These groups of students were identified from within the larger study as those who performed in the bottom third and top third of the spelling measure within the NAPLAN Language Conventions Test, respectively.

Spelling achievement data

As can be seen in Table 2, students' NAPLAN Language Conventions Test results from 2012 and 2013 were collected. These data were gathered from school databases or directly from the parents of the participating student/s if the school did not hold the records. In addition, the CoST was administered by the principal researcher (first author) to participating students in October, 2013, in collaboration with the respective school principals and teachers.

A proofreading and editing format (The NAPLAN Language Conventions Test: Spelling domain)

The NAPLAN Language Conventions Test is part of a series of national standardised tests administered annually in May to all Australian students in Years 3, 5, 7 and 9 (ACARA, 2016). The 40 minute test assesses aspects of student achievement in spelling, grammar and punctuation. This test requires students to complete visually oriented tasks, decontextualised from compositional writing processes. Specifically, students identify and edit spelling errors in words presented either in isolation or within a short phrase, as well as identify and label some common grammatical and punctuation conventions such as the correct use of pronouns, conjunctions and verb forms. Student participants' results from items identified only within the spelling domain of the Language Conventions Test were utilised for the present study (see Appendix 1 for the 2012 and 2013 NAPLAN spelling domain items).

A dictation format: The Components of Spelling Test (CoST)

The CoST is a dictation test designed to measure knowledge of the linguistic components of the Standard English spelling system. Closely aligning with TWFT, the CoST provides a means from which to interrogate student knowledge of the spelling system without confining spelling achievement into a specific stage or phase of development. Strong internal consistency results for this spelling test have been reported, with Cronbach's alphas ranging from .78 to .94 (Daffern et al., 2015).

The CoST requires students to write 70 words presented to them orally, each within the context of a sentence (see Appendix 2); however, the measure comprises 15 constructs and 101 individual items across three subscales: i) Phonological Component; ii) Orthographic Component; and iii) Morphological Component. Appendix 3 contains the scoring templates for the three subscales of the CoST. Appendix 4 includes norms obtained from the sample of students who participated in the larger study.

What follows is an explanation of the administration and scoring procedures of the CoST. This information may be useful for educational researchers and practitioners seeking to utilise the CoST. However, to ensure that integrity to the validity of the tool is maintained, potential test administrators are urged to carefully consider and follow the recommended protocols, outlined below. In addition, it is strongly recommended that any potential re-testing does not occur within a timeframe of approximately twelve months.

Administration protocols for the CoST

In the study, the CoST was administered (by the first author) to groups of students in regular classroom settings. Testing time was approximately 30 minutes per group. Students were not given the opportunity to study the 70 target words in advance of testing. Prior to commencing the dictation test, the administrator orally provided the following instructions to the student/s:

'I am going to ask you to write 70 words. Some of the words may be easy to spell; some may be difficult. If you do not know how to spell a word, spell it the best you can. First, I will say the word, and then I will use the word in a sentence. If you didn't hear it, listen very carefully because I will repeat the word one last time. If you really didn't hear a word, put up your hand and wait for me to ask you what the problem is.'

The participating students were encouraged to complete the entire test, even if the target words appeared to be very difficult for them. Potential test administrators should be mindful that if a student finds the words too difficult, the student should still be encouraged to write the words. However, if a student is clearly having significant trouble with the more difficult words and has not attempted more than five words in a row, the student could be invited to stop. In addition, if a student does not hear a word, the test administrator needs to make a judgement whether or not to repeat the word. It is important to bear in mind that too much repetition could disrupt the testing situation and potentially invalidate the results if inconsistencies arise in the administration of the CoST. In some instances, the administrator needs to remind the student/s to listen carefully as words would not be repeated again.

For this study, the participating students were required to write their responses on lined paper, as illustrated in the sample shown in Figure 1. At the conclusion of testing, the administrator immediately collected the response papers from all participating students.

The sentence prompts that were used by the test administrator are provided in Appendix 2 for the reader's convenience. The administrator called each target word (indicated in italics) aloud, used it in a sentence, and then repeated the target word at the end of the sentence. Each word was dictated without deliberate or artificial emphasis of any particular feature, including phoneme or syllable.

Scoring procedure

After collecting the response papers from the students, the administrator analysed and scored the spelling of each word for each participating student. This scoring process involved an analysis of phonological, orthographic and morphological errors that a student may have made in the list of 70 words that were written. For each student, the scoring was documented on the Scoring Templates (see Appendix 3). As specified in the CoST, some words contain only one measurable item (target grapheme/s), while other words contain two or more measurable items. In the Scoring Templates, each item is located in a cell to the right of each word. The items in the CoST are designed to measure a students' representation of specific linguistic features within words (rather than the spelling of whole words). Each item is assessed dichotomously (that is, as correct or incorrect).

Mirror reversals of graphemes (letters) were accepted as correct, unless they lead to other correct graphemes (e.g., rob as rod; speaker as sqeaker). Both upper and lowercase letters were accepted as correct. If an item contained more than one grapheme, all of the graphemes were written in the order specified in the respective cell for that item to be marked as correct. For example, in the word smudged, the item containing the letters dge should have been spelled exactly as dge for the item to have been marked as correct. Any other spelling alternations such as gde, dg, or bge were incorrect.

The scoring process began by placing the Phonological Component Scoring Template beside a student's response paper. For each word listed on the far left column of the Phonological Component Scoring Template (see Appendix 3), the scorer (first author) examined whether the corresponding word part (item) was correctly spelled by the student. The items which the student correctly spelled were colour highlighted to indicate that those items were correctly written by the student. For example, if a student spelled tag as tug, the letter a listed in the corresponding 'short vowel grapheme' cell was not highlighted, while the t and g were highlighted in the corresponding 'initial and final consonants' cells to indicate correct responses. When all items in the Phonological Component Scoring Template were dichotomously assessed, the sum of correct responses for each linguistic feature (construct) was computed. Finally, the total Phonological Component raw score was then calculated.

The process described above was repeated for the Orthographic Component and Morphological Component, using the respective Scoring Templates (see Appendix 3).

Method of analysis

All achievement data sets (that is, from the NAPLAN Language Conventions Test and the CoST) were entered into SPSS (Version 20). Pearson's bivariate correlation analyses (Bryman, 2008) were then conducted to establish the relationships between NAPLAN spelling and the three linguistic components of spelling, as measured by the CoST, for low-achieving spellers and high-achieving spellers. Multiple Regression Analyses (MRAs) were also performed to determine whether these relationships were affected by gender and/or year level.

Specifically, for Years 3 and 4, bivariate correlations were carried out to examine the relationships between the Year 3 NAPLAN spelling scores and the CoST subscale scores for low-achieving and high-achieving spellers. This procedure was repeated for Years 5 and 6, however, only the students' Year 5 NAPLAN Language Conventions Test spelling scores were used for these correlations. Separate bivariate correlation analyses for males and females were conducted to examine gender differences. MRAs were then used to test if the three components of spelling, as measured by the CoST, predicted NAPLAN spelling for low-achieving and high-achieving spellers in Years 3 to 6. These were followed by another series of MRAs that were carried out to see if gender was predictive of NAPLAN spelling for low-achieving and high-achieving students in Years 3 to 6.

Results

Results for low-achievers and high-achievers across the four cohorts indicated significant positive correlations between the NAPLAN spelling results and the three subscales (components) of spelling, as measured by the CoST (see Table 3). For low-achieving spellers across all year levels, the strongest correlation was between NAPLAN spelling and the orthographic subscale, r = .64, p < .001 (Year 3); r = .69, p <.001 (Year 4); r = .67, p < .001 (Year 5); and r = .64, p <.001 (Year 6). It is worth noting, however, that in the Year 6 low-achieving group an equally strong relationship was observed with the morphological subscale. For the high-achieving spellers, the strongest relationship was between the NAPLAN spelling and the morphological subscale, with the exception of Year 5, in which the relationship with the phonological subscale was marginally stronger, r = .51, p < .001, than the morphological subscale, r = .50, p <.001.

Separate bivariate correlation analyses for males and females were also conducted to examine gender differences (see Table 4). A testing of the significance of the CoST subscale correlations between males and females, using the low- and high-achieving spelling groups, revealed no significant findings. This testing was based on a two-tailed t-test with a Bonferroni correction of .05/3. It needs to be noted, however, that there were some high correlations in spite of the truncated NAPLAN spelling scores in both spelling groups. In addition, a comparison of gender in the low-achieving and high-achieving spelling groups by year level revealed a dominance of females in the high achieving groups; however, there was no obvious pattern of gender difference in the low-achieving groups, as can be seen in Table 4.

MR As were then used to test if the three components of spelling, as measured by the CoST, predicted NAPLAN spelling for low-achieving and high-achieving spellers in Years 3 to 6. The results of the analyses indicated that the three CoST subscales, across all year levels, for low-achieving spellers and high-achieving spellers were significantly associated with NAPLAN spelling (see Table 5). For example, for low-achievers in Year 3 about 41% of the variance was explained by the CoST, [R.sup.2]adj=.413, F (3,67) =17.43, p<001; and approximately 30% for high-achievers, [R.sup.2]adj =.299, F(3,70)=11.40, p<.001.

As indicated in Table 5, the results of the MRAs for the Year 3 cohort revealed that the orthographic subscale score was the only significant predictor in this model at p=.002; for the high-achievers, the morphological subscale score was the only significant predictor at p<.001. For low-achieving spellers in Years 4 and 5, the only significant predictor of the three CoST subscale scores was the orthographic score. For high achieving spellers in Year 5, the orthographic subscale was the strongest predictor, followed by the phonological subscale. Although no CoST subscale scores independently predicted NAPLAN spelling in Year 6, the overall model was a good fit, with about 42 percent of the variance in NAPLAN spelling jointly explained by the CoST for low-achievers and 24 percent for the high-achievers.

While significant, these results do suggest, particularly for high-achieving spellers, that NAPLAN spelling involves other competencies not measured by the CoST. Further, it seems reasonable to assume that high-achieving spellers are equipped with greater linguistic agency than low-achieving spellers. It also needs to be kept in mind that the spelling component of the NAPLAN Language Conventions Test is based on a proofreading and editing task, whereas the CoST uses a dictation format.

Another series of MRAs were carried out to see if gender was predictive of NAPLAN spelling for low-achieving and high-achieving spellers in Years 3 to 6. When gender was entered in step one (that is, as a control variable) for the MRAs, none of the [R.sup.2] values reached a significant level (that is p< .05). As a second step, the three CoST subscale scores for each year level were entered. This second step not only showed that the model for each year level was significant, but permitted calculation of an R2 change value for each MRA (see Table 6). The [R.sup.2] change was calculated by subtracting the R2adj value (for gender in step one of the MRA) from the [R.sup.2]adj value (for gender, phonological subscale, orthographic subscale and the morphological subscale measures in step two of the MRA). In cases where the [R.sup.2]adj value, for step one, was less than zero, the R2adj value was treated as zero. The recording of a negative [R.sup.2]adj value occurs when only one variable is entered and this step leads to a very low value (Tabachnick & Fidell, 2001).

As can be seen in Table 6, results indicate that for the low-achieving groups of students across the year levels, the [R.sup.2] change values were relatively large, while for the high-achieving spelling groups the [R.sup.2] change values were somewhat smaller. Further, the orthographic subscale was generally the main predictor of NAPLAN spelling across year levels, with the exception of the Year 6 high-achieving group and both Year 4 groups, where no individual subscale was a significant predictor. In addition to the orthographic subscale, the phonological subscale was a significant predictor, but only in the Year 5 high-achieving group.

Discussion

In a climate where mandatory national testing of student achievement appears to be influencing educational practices in literacy (Hardy, 2013), there is a pressing need for classroom teachers to have access to student achievement data that enable them to make informed decisions regarding instructional approaches and priorities. When used in isolation, the NAPLAN results do not provide Australian classroom teachers with specific feedback about the linguistic skills that need to be addressed in order to support student learning in spelling. It has also been argued that the proofreading and editing format of the NAPLAN spelling domain may potentially 'confuse the picture of students' spelling ability' (Willett & Gardiner, 2009, p. 2). To address these issues, we sought to test whether there is a relationship between the results from the NAPLAN spelling domain and a dictation-based measure of the three linguistic constituents in spelling, namely the phonological, orthographic and morphological components.

Our findings reveal significant relationships between the two testing formats. Importantly, the results indicate that even though orthographic accuracy, as measured by the CoST seems to be a particularly strong predictor of NAPLAN spelling, a combination of accurate phonological, orthographic and morphological representations does indeed constitute success with spelling. The results also align with the fundamental principle underpinning TWFT that learning to spell 'depends on developing awareness of phonological, orthographic, and morphological word forms . and coordinating them' (Richards, Berninger & Fayol, 2009, p. 332).

The results of this study also provide evidence of criterion-related validity for the CoST. Specifically, significant relationships between the NAPLAN spelling results and the three subscale scores of the CoST, for low-achieving and high-achieving spellers, across the four cohorts were found. This suggests that students who perform poorly on a spelling measure that employs a proofreading and editing format, for example, as measured by the spelling domain of the NAPLAN Language Conventions Test, may also perform poorly on spelling tests that utilise a dictation format, such as the CoST. The same pattern of correlation can be construed for high-achieving spellers. This finding seems to contradict Willett and Gardiner (2009), who have questioned the validity of the NAPLAN format and Critten et al. (2013) who suggest that production tasks are cognitively more demanding and proofreading/editing tasks. Indeed, the finding initiates an opportunity for future research aiming to undertake further validity testing of the CoST, particularly by utilising other measures of phonological, orthographic and morphological representations in spelling. It needs to be noted that further validity testing was considered for the present study; however, it was deemed impossible because adequate measures of phonological, orthographic and morphological representations of real word spelling did not appear to exist. Indeed, future research should seek to develop parallel or alternative tests for the CoST to strengthen its validity, but also to increase its utility. Also worthy of future examination is an analysis of the relationships between the CoST subscales and non-word measures of phonology (see, for example, Wagner, Torgesen, Rashotte & Pearson, 2013), orthography (see, for example, Conrad et al., 2013) and morphology (see, for example, Nunes, Bryant & Olsson, 2003).

Another important finding revealed in this study is that for low-achieving spellers across all year levels, the strongest correlation was between NAPLAN spelling and the orthographic subscale; an equally strong relationship was observed with the morphological subscale for the Year 6 low-achieving spellers. This finding builds on the work of Conrad et al. (2013), who demonstrated that orthographic knowledge, at age seven to nine years, contributes to spelling 'over and above the contributions of phonological skills' (p. 1223). It also supports research by Rothe et al. (2014) who provided evidence that orthographic knowledge predicts spelling in German-speaking Kindergarten students. While these two studies did not include a morphological measure, the present study did, and the results showed that for the high-achieving spellers, the strongest relationship was between the NAPLAN spelling measure and the morphological subscale, with the exception of Year 5, in which the relationship with the phonological subscale was marginally stronger than the morphological subscale. In addition, a strong relationship was observed with the morphological subscale for the Year 6 low-achieving spellers.

While gender differences were examined by testing the significance of the CoST subscale correlations between males and females, using the low- and high-achieving spelling groups, no significant findings were revealed. Critically, the results suggest that the students who performed poorly in the NAPLAN spelling retained their status as poor spellers up to 18 months later, as measured by the CoST, and that gender was not an influential factor. Similarly, high-achieving spellers (as measured by NAPLAN spelling) remained as high-achieving spellers (as measured by the CoST) during the same time period. Resonating with this finding is the stability of spelling performance reported in Abbott et al.'s (2010) longitudinal research in spelling acquisition. This highlights the need to systematically utilise assessment outcomes in order to guide instruction, and in particular to provide early and effective intervention in spelling for those students experiencing difficulties with spelling. Specifically, teachers could use the CoST, in combination with other forms of assessment, to help them ascertain the linguistic skills which do require instructional attention in order to improve spelling achievements.

While differences were found between low-achieving spellers and high-achieving spellers, the correlations between the two test results do not provide causal evidence. However, the findings support the view that heightened sensitivity to sub-lexical orthographic regularities plays an important role in learning to proofread and edit spelling errors, but also in learning to spell more broadly (Richards et al., 2009; Rothe et al., 2014). It is not surprising that the orthographic subscale of the CoST appears to be such a strong predictor of NAPLAN spelling, considering that the latter is essentially a visual-spatial exercise that involves scanning word/s, and identifying and correcting orthographic anomalies. Nevertheless, future research is needed to conclusively determine why there may be differences in error types across spelling abilities.

Concluding remarks

The generalisability of the findings in the study is limited to schools across the ACT. It also needs to be acknowledged that the ACT broadly represents one of the highest performing jurisdictions in Australia in terms of academic achievement in school, as measured by NAPLAN. Further, the sample size for this study poses some limitations to the findings. Specifically, hierarchical regression analyses were limited by the truncated scores, particularly in the Year 6 cohort. As such, replication of this study involving other jurisdictions is welcomed. It may also be intriguing to explore whether or not similar correlations demonstrated in the present study are to be found, with the advent of 'online' NAPLAN testing, as opposed to paper and pen modes of delivery.

One single assessment tool cannot provide the complete answer to a child's knowledge about the spelling system and it should be accompanied by other methods of assessment. Indeed, being able to understand the extent to which children can explain and justify their spelling is insightful. Although the findings reported in this paper offer evidence that the CoST is an informative and valid assessment tool, we assert that multiple forms of assessment should still be integrated into an instructional program. These may include linguistic analyses of dictated non-words, but also spelling errors recorded in students' written compositions. In addition, conducting open-ended interviews with students, or including questionnaires and/or surveys as part of assessment regimes in classroom contexts are also very valuable.

References

Abbott, R., Berninger, V. & Fayol, M. (2010). Longitudinal relationships to levels of language in writing and between writing and reading in grades 1 to 7. Journal of Educational Psychology, 102(2), 281-291. doi:10.1037/a0019318

Al Otaiba, S. & Hosp, J. (2010). Spell it out: The need for detailed spelling assessment to inform instruction. Assessment for Effective Intervention, 36(1), 3-6.

Apel, K. (2014). A comprehensive definition of morphological awareness: Implications for assessment. Topics in Language Disorders, 34(3), 197-209. doi:10.1097/ TLD.0000000000000019

Australian Curriculum, Assessment & Reporting Authority (ACARA). (2016). NAP national assessment program. Retrieved from http://www.nap.edu.au/about/about.html

Bahr, R. (2015). Spelling strategies and word formation processes. In R. Bahr & E. Silliman (Eds.), Routledge handbook of communication disorders (pp. 193-203). London: Routledge.

Bahr, R., Silliman, E. & Berninger, V. (2009). What spelling errors have to tell about vocabulary learning. In C. Woods & V. Connelly (Eds.), Contemporary perspectives on reading and writing (pp. 109-129). New York: Routledge.

Bahr, R., Silliman, E., Berninger, V. & Dow, M. (2012). Linguistic pattern analysis of misspellings of typically developing writers in grades 1-9. Journal of Speech, Language, and Hearing Research, 55, 1587-1599.

Bahr, R., Silliman, E., Danzak, R. & Wilkinson, L. (2015). Bilingual spelling patterns in middle school: It is more than transfer. International Journal of Bilingual Education and Bilingualism, 18(1), 73-91. doi:10.1080/13670050.2013 .878304

Bear, D.R., Invernizzi, M., Templeton, S. & Johnston, F. (2012). Words their way: Word study for phonics, vocabulary, and spelling instruction (5th ed.). Upper Saddle River, NJ: Pearson Education, Inc.

Berninger, V., Abbott, R., Nagy, W. & Carlisle, J. (2010). Growth in phonological, orthographic, and morphological awareness in grades 1 to 6. Journal of Psycholinguistic Research, 39(2), 141-163. doi:10.1007/s10936-009-9130-6 Bryman, A. (2008). Social research methods (3rd ed.). New York: Oxford University Press, Inc.

Conrad, N., Harris, N. & Williams, J. (2013). Individual differences in children's literacy development: The contribution of orthographic knowledge. Reading and Writing, 26(8), 1223-1239. doi:10.1007/s11145-012-9415-2

Critten, S., Pine, K. & Messer, D. (2013). Revealing children's implicit spelling representations. British Journal of Developmental Psychology, 31(2), 198-211. doi:10.1111/ bjdp.12000

Daffern, T., Mackenzie, N.M. & Hemmings, B. (2015). The development of a spelling assessment tool informed by Triple Word Form Theory. Australian Journal of Language & Literacy, 38(2), 72-82.

Ehri, L.C. (2005). Learning to read words: Theory, findings, and issues. Scientific Studies of Reading, 9(2), 167-188. doi:10.1207/s1532799xssr0902_4

Garcia, N., Abbott, R. & Berninger, V. (2010). Predicting poor, average, and superior spellers in grades 1 to 6 from phonological, orthographic, and morphological, spelling, or reading composites. Written Language & Literacy, 13(1), 61-98.

Gentry, J.R. (2012). An analysis of developmental spelling in GNYS AT WRK. In D. Wyse (Ed.), Literacy teaching and education (Vol. 3, pp. 347-357). London: SAGE Publications.

Hardy, I. (2013). Testing that counts: Contesting national literacy assessment policy in complex schooling settings. Australian Journal of Language & Literacy, 36(2), 67-77.

Holmes, V. & Ng, E. (1993). Word-specific knowledge, word-recognition strategies, and spelling ability. Journal of Research in Reading, 31, 136-156.

Kohnen, S., Nickels, L. & Castles, A. (2009). Assessing spelling skills and strategies: A critique of available resources. Australian Journal of Learning Difficulties, 14(1), 113-150.

Nagy, W., Berninger, V. & Abbott, R. (2006). Contributions of morphology beyond phonology to literacy outcomes of upper elementary and middle-school students. Journal of Educational Psychology, 98(1), 134-147.

Nunes, T., Bryant, P. & Olsson, J. (2003). Learning morphological and phonological spelling rules: An intervention study. Scientific Studies of Reading, 7(3), 289-307. doi:10.1207/s1532799xssr0703_6

Read, C. & Treiman, R. (2013). Children's invented spelling: What we have learned in forty years. In Piattelli-Palmarini & R.C. Berwick (Eds.), Rich languages from poor inputs (pp. 197-211). New York: Oxford University Press.

Richards, T., Aylward, E., Field, K., Grimme, A., Raskind, W., Richards, A., ... Berninger, V. (2006). Converging evidence for triple word form theory in children with dyslexia. Developmental Neuropsychology, 30(1), 547-589. doi:10.1207/s15326942dn3001_3

Richards, T., Berninger, V. & Fayol, M. (2009). fMRI activation differences between 11-year-old good and poor spellers' access in working memory to temporary and longterm orthographic representations. Journal of Neurolinguistics, 22(4), 327-353.

Rothe, J., Schulte-Korne, G. & Ise, E. (2014). Does sensitivity to orthographic regularities influence reading and spelling acquisition? A 1-year prospective study. Reading and Writing, 27(7), 1141-1161. doi:10.1007/ s11145-013-9479-7

Sharp, A.C., Sinatra, G.M. & Reynolds, R.E. (2008). The development of children's orthographic knowledge: A microgenetic perspective. Reading Research Quarterly, 43(3), 206-226.

Silliman, E., Bahr, R. & Peters, M. (2006). Spelling patterns in preadolescents with atypical language skills: Phonological, morphological, and orthographic factors. Developmental Neuropsychology, 29(1), 93-123. doi:10.1207/ s15326942dn2901_6

Snyder, I. (2009). The stories that divide us: Media (mis) representations of literacy education. English in Australia, 44(1), 13-23.

Tabachnick, B.G. & Fidell, L.S. (2001). Using multivariate statistics (4th ed.). Boston, MA: Allyn and Bacon.

Treiman, R. & Kessler, B. (2006). Spelling as statistical learning: Using consonantal context to spell vowels. Journal of Educational Psychology, 98(3), 642-652.

Wagner, R.K., Torgesen, J.K., Rashotte, C.A. & Pearson, N.A. (2013). Comprehensive Test of Phonological Processing (2nd ed.). Austin, TX: Pro Ed.

Westwood, P. (2005). Spelling: Approaches to teaching and assessment. Camberwell, Vic: ACER Press.

Willett, L. & Gardiner, A. (2009). Testing spelling: Exploring NAPLAN. Paper presented at the Australian Literacy Educators' Association Conference.

Tessa Daffern

Science, Technology, Engineering and Mathematics Education Research Centre (SERC), University of Canberra

Noella Maree Mackenzie

Research Institute for Professional Practice, Learning and Education (RIPPLE), Charles Sturt University

Brian Hemmings

Research Institute for Professional Practice, Learning and Education (RIPPLE), Charles Sturt University

Tessa Daffern is an Assistant Professor in Teacher Education at the University of Canberra. She has also been a member of the Research Institute for Professional Practice, Learning and Education, and a Subject Coordinator in the Master of Education at Charles Sturt University. She is an accredited provider of professional learning with the Teacher Quality Institute, in the Australian Capital Territory, and regularly engages in professional work with school teachers. Email: tessa.daffern@canberra.edu.au

Noella Maree Mackenzie is a Senior Lecturer in literacy studies at Charles Sturt University. Noella's research has focused on the teaching and learning of writing and teacher PL. Her research informs, and is informed by, her ongoing professional work with teachers in schools and her university teaching. Noella has been recognised for teaching excellence. Her work has been published in professional (e.g. Practical Literacy) and research journals (e.g. Australian Educational Researcher). Email: nmackenzie@csu.edu.au

Brian Hemmings is currently the Sub-Dean (Graduate Studies) and Deputy Director, Research Institute for Professional Practice, Learning and Education (RIPPLE) at Charles Sturt University. He has published widely and his most recent publications appear in Professional Development in Education and the Australian Journal of Teacher Education. Email: bhemmings@csu.edu.au Appendix 1. Items measured in the NAPLAN spelling domain (2012 and 2013) 2012 NAPLAN Spelling 2013 NAPLAN Spelling Domain (items) Domain (items) Identify and Correct spelling Identify and Correct spelling correct error in correct error in spelling error specified word spelling error specified word unable restart stars yellow coming saving pretty leaf waterfall rainy pick bear juice oval white shopping dramatic climb wise winner promised quiet movement flower satisfied possible kindness neatly balance hospital glitter bowl instrument stitch waited parrot beginning revision healthy loose parcel measure battery popular valuable cylinder daily autumn sneeze technology

Appendix 2. CoST Sentence Prompting Guide

1. tag: The tag on the travel bag has my address on it. Write tag

2. gum: They like to blow bubbles with the gum. Write gum

3. rob: The thief tried to rob the bank. Write rob

4. stick: The girl was poking a stick in the mud. Write stick

5. why: I wonder why the door is open. Write why

6. thorn: A thorn from a rose is very sharp. Write thorn

7. wait: She must wait for her mother to arrive. Write wait

8. sled: The dogs pull a sled in the snow. Write sled

9. chew: Some babies like to chew on soft toys. Write chew

10. coach: The soccer players were listening to their coach. Write coach

11. smooth: The icing on the cake is smooth. Write smooth

12. carries: He carries his iPod in his pocket. Write carries

13. stripe: My shirt has a green stripe on the back. Write stripe

14. marched: The soldiers marched on Anzac Day. Write marched

15. shade: Sit in the shade so that you don't get sunburnt. Write shade

16. fright: I jumped with fright at the sound of the thunder. Write fright

17. bottle: My water bottle is empty. Write bottle

18. bait: A worm makes good bait. Write bait

19. nibble: I will nibble on the fruit. Write nibble

20. boil: Please boil the water to make tea. Write boil

21. shrink: This jumper might shrink if washed in warm water. Write shrink

22. shouted: Mum wasn't happy when I shouted at my brother. Write shouted

23. serving: The waiter is now serving our food. Write serving

24. cellar: My dad went down to the cellar to choose a bottle of wine. Write cellar

25. rail: Hold onto the rail as you walk down the stairs. Write rail

26. speaker: It is my turn to be the speaker at assembly. Write speaker

27. tunnel: A car accident blocked the tunnel. Write tunnel

28. grew: The trees in our playground grew quickly. Write grew

29. hawk: The hawk swoops and circles in search of prey. Write hawk

30. knotted: The knotted shoe lace was hard to undo. Write knotted

31. moat: The castles' moat is full of crocodiles. Write moat

32. favour: She did her friend a favour. Write favour

33. scratches: He had to paint over the scratches on the door. Write scratches

34. smudged: The wet paint smudged. Write smudged

35. doctor: The doctor gave me a prescription. Write doctor

36. incredible: The noise from the crowd was incredible. Write incredible

37. mussel: A mussel is a type of shellfish. Write mussel

38. opposition: The opposition was competitive during the game. Write opposition

39. magnitude: Events of magnitude will always be remembered. Write magnitude

40. irresponsible: It would be irresponsible to leave the rubbish on the ground. Write irresponsible

41. pleasure: It has been a pleasure working with you. Write pleasure

42. waist: The waist is between the ribs and hips. Write waist

43. dominance: There is a dominance of cane toads. Write dominance

44. soar: The bird will soar into the sky. Write soar

45. succession: She won the race for the second year in succession. Write succession

46. illuminate: A flash of light will illuminate the room. Write illuminate

47. fortunate: She was fortunate to have won the prize. Write fortunate

48. hoarse: My voice sounds hoarse because I have a cold. Write hoarse

49. correspond: Her actions do not correspond with her words. Write correspond

50. medicinal: Traditionally, people used native plants for medicinal purposes. Write medicinal

51. acclaim: Let us acclaim the wonderful performance. Write acclaim

52. annotate: If you annotate the drawing its meaning will be clear. Write annotate

53. lapse: There was a large lapse of time between the two events. Write lapse

54. genes: My eye colour comes from my parents' genes. Write genes

55. chlorine: The water in the pool contains a lot of chlorine. Write chlorine

56. serial: The new TV drama serial has been very popular. Write serial

57. perspiration: As he exercised, perspiration ran down his forehead. Write perspiration

58. irrelevant: The lessons are interesting but sometimes irrelevant. Write irrelevant

59. psychology: A person who studies psychology learns about the human mind. Write psychology

60. perpendicular: Use a ruler to draw a perpendicular line. Write perpendicular

61. substantial: She had a substantial amount of money. Write substantial

62. properly: The paper plane is folded properly. Write properly

63. equilibrium: He maintains an equilibrium between work and play. Write equilibrium

64. diagnostician: That doctor is a medical diagnostician. Write diagnostician

65. recognise: He could recognise the car easily. Write recognise

66. kinaesthetic: My muscles hurt after the kinaesthetic activity. Write kinaesthetic

67. negligence: The driver's negligence has caused an injury. Write negligence

68. precipitation: Recent precipitation has caused flooding. Write precipitation

69. rehabilitating: Thousands of dollars were spent in rehabilitating the old building. Write rehabilitating

70. aristocracy: Royal families experience a life of aristocracy. Write aristocracy Appendix 3: CoST Scoring Template CoST Scoring Sheet: Phonological Component 1. Initial and 2. Short Vowel 3. Consonant Final Consonants Gapheme Digraphs 1. tag t- -a- 1. tag -g 2. gum -m -u- 3. rob -b -o- 4. stick -i- 5. why wh- 6.thorn th- 7. wait w- 8. sled -e- 9. chew ch- 10.coach -ch 11. smooth -th 35. doctor 40. irresponsible 47. fortunate 58. irrelevant 59. psychology 60. perpendicular 61. substantial 62. properly 63. equilibrium 64. diagnostician 65. recognise 66. kinaesthetic 67. negligence 68. precipitation 69. rehabilitating 70. aristocracy Sub Item Totals /5 /5 /5 Phonological Component Total Score: /31 5. Polysyllabic-word Medial Blends 1. tag 1. tag 2. gum 3. rob 4. stick 5. why 6.thorn 7. wait 8. sled 9. chew 10.coach 11. smooth 35. doctor -oct- 40. irresponsible -spon- 47. fortunate -ort- 58. irrelevant -rel- 59. psychology -ol- 60. perpendicular -rpendi- 61. substantial -ubstan- 62. properly -roperl- 63. equilibrium -libri- 64. diagnostician -agnosti- 65. recognise -gn- 66. kinaesthetic -estheti- 67. negligence -eglig- 68. precipitation -ipita- 69. rehabilitating -habilita- 70. aristocracy -ristocr- Sub Item Totals /16 Phonological Component Total Score: /31 CoST Scoring Sheet: Orthographic Component 6. Common 7. Ambiguous 8. Complex Long Vowel Vowel Consonant Graphemes Graphemes Patterns 12. carries -ar- 13. stripe -i-e str- 14. marched -ar- 15. shade -a-e 16. fright -igh- 17. bottle 18. bait -ai- 19. nibble 20. boil -oi- 21. shrink shr- 22. shouted -ou- 23. serving -er- 24. cellar 25. rail -ai- 26. speaker -ea- 27. tunnel 28. grew -ew 29. hawk -aw- 30. knotted kn- 31. moat -oa- 32.favour 33. scratches -tch- 34. smudged -dge- 35. doctor Sub Item Totals /7 /7 /5 Orthographic Component Total Score: /29 9. Syllable 10. Unaccented Juncture Final Syllables Consonants 12. carries -rr- 13. stripe 14. marched 15. shade 16. fright 17. bottle -tt- -le 18. bait 19. nibble -bb- 20. boil 21. shrink 22. shouted 23. serving 24. cellar -ll- -ar 25. rail 26. speaker 27. tunnel -nn- -el 28. grew 29. hawk 30. knotted 31. moat 32.favour -our 33. scratches 34. smudged 35. doctor -or Sub Item Totals /5 /5 Orthographic Component Total Score: /29 CoST Scoring Sheet: Morphological Component 11. Inflected 12. Derivational 13. Morpheme Suffixes Suffixes Juncture Schwa Vowels 12. carries -ries 14. marched -ed 22. shouted -ed 23. serving -ving 30. knotted -tted 33. scratches -es 34. smudged -ed 36. incredible -ible 37. mussel 38. opposition -tion -po- 39. magnitude -tude 40. irresponsible -ible -re- 41. pleasure -ure 42. waist 43. dominance -ance -mi- 44. soar 45. succession -sion 46. illuminate 47. fortunate -ate 48. hoarse 49. correspond 50. medicinal -ci- 51. acclaim 52. annotate 53. lapse 54. genes 55. chlorine 56. serial 57. perspiration -pi- 58. irrelevant Sub Item Totals /7 /8 /5 Morphological Component Total Score: /41 14. Homophone 15. Greek and 16. Assimilated Latin Roots Prefixes 12. carries 14. marched 22. shouted 23. serving 30. knotted 33. scratches 34. smudged 36. incredible -cred- 37. mussel -ssel mus- 38. opposition -pp- 39. magnitude magn- 40. irresponsible 41. pleasure 42. waist -aist 43. dominance 44. soar -oar 45. succession -cc- 46. illuminate -lumina- -ll- 47. fortunate 48. hoarse -oarse 49. correspond -rr- 50. medicinal medic- 51. acclaim -cc- 52. annotate -nn- 53. lapse -se 54. genes gene- 55. chlorine chlor- 56. serial seria- 57. perspiration -spir- 58. irrelevant -rr- Sub Item Totals /7 /7 /7 Morphological Component Total Score: /41

Caption: Figure 1. Sample of a student's response in the CoST Table 1. Means, standard deviations and age range (in months), for participating students (Years 3 to 6) Year level Age (months) M (SD) Age (months) Min-Max Year 3 (n=145) 115 (4.2) 106-131 Year 4 (n=117) 127 (4.5) 110-138 Year 5 (n=133) 138 (4.2) 123-152 Year 6 (n=117) 150 (4.0) 140-163 Table 2. Schedule for collection of NAPLAN and CoST data 2013 Test Administration Periods Cohorts May 2012 May 2013 October 2013 Year 3 [?] [?] Year 4 [?] [?] Year 5 [?] [?] Year 6 [?] [?] Notes [?] Year 5 NAPLAN: Language Conventions Test (Raw scores for Spelling, Grammar and Punctuation) [?] Year 3 NAPLAN: Language Conventions Test (Raw scores for Spelling, Grammar and Punctuation) [?] CoST Table 3. Relationships between the NAPLAN spelling scores and the CoST subscale scores for low-achievers and high-achievers (Years 3 to 6) Year Phonological Orthographic Morphological n Level Subscale r Subscale r Subscale r Year 3 Low .54 * .64 * .51 * 71 High .31 * .38 * .57 * 74 Year 4 Low .42 * .69 * .49 * 56 High .37 * .35 * .45 * 61 Year 5 Low .50 * .67 * .51 * 55 High .51 * .48 * .50 * 78 Year 6 Low .54 * .64 * .64 * 55 High .41 * .43 * .51 * 62 Note Low Low-achieving group [bottom third] as measured by NAPLAN Spelling High High-achieving group [top third] as measured by NAPLAN Spelling * Significant at the 0.01 level (2-tailed) Table 4. Correlation analyses for males and females by spelling group Year Group Gender Phonological Orthographic subscale subscale Year 3 Low male .40 .54 female .63 .73 High male .34 .57 female .28 .29 Year 4 Low male .46 .72 female .41 .65 High male .50 .37 female .26 .34 Year 5 Low male .37 .48 female .65 .82 High male .60 .37 female .50 .54 Year 6 Low male .63 .73 female .44 .63 High male .43 .42 female .40 .47 Year Group Gender Morphological N subscale Year 3 Low male .45 37 female .54 34 High male .56 31 female .55 43 Year 4 Low male .48 28 female .56 27 High male .62 20 female .36 41 Year 5 Low male .27 25 female .71 30 High male .58 28 female .48 50 Year 6 Low male .66 24 female .60 31 High male .66 20 female .49 42 Table 5. CoST subscale scores for low- and high-achieving groups as predictors of NAPLAN spelling (Years 3 to 6) Year n Phonological Orthographic Morphological Level Subscale [beta] Subscale [beta] Subscale [beta] Year 3 Low 71 ns .266 * ns High 74 ns ns .18 * Year 4 Low 56 ns .359 * ns High 61 ns ns ns Year 5 Low 55 ns .302 * ns High 78 .234 .382 ns Year 6 Low 55 ns ns ns High 62 ns ns ns Year n Overall CoST Level Model [R.sup.2]adj Year 3 Low 71 .413 High 74 .299 Year 4 Low 56 .451 High 61 .176 Year 5 Low 55 .425 High 78 .388 Year 6 Low 55 .415 High 62 .241 Note Low Low-achieving group [bottom third] as measured by NAPLAN Spelling High High-achieving group [top third] as measured by NAPLAN Spelling * [beta] coefficient is significant at the 0.01 level (2-tailed) Table 6. [R.sup.2] change values for low- and high-achieving groups (Years 3 to 6) for a two-step entry Year Low-achieving group High-achieving group [R.sup.2] Significant IV [R.sup.2] Significant IV change change 3 .405 Orthographic .271 Orthographic 4 .444 None .162 None 5 .421 Orthographic .366 Phonological & Orthographic 6 .429 Orthographic .252 None Note IV Independent variables (CoST subscale totals)