Rethinking the achievement of the student writer.

Wyatt-Smith, Claire

Despite over 90 years of research into the means of devising and implementing testing procedures and scoring rubrics, the assessment of student writing remains problematic (White, 1990; Rothery & Macken, 1991; Cox & Haynes, 1991; Freebody, 1992; Johnston, 1992; Elbow, 1993). There is continuing uncertainty about how grading ought to occur and also about the ways raters arrive at grading decisions, including the use they make of available scoring rubrics in the reading and rating process (Huot, 1990; Wyatt-Smith, 1996). A contention currently proffered by educational linguists and other researchers working in the field of literacy assessment is that defined criteria and standards are needed to demystify the assessment process (see, for example, Macken, 1989; Macken & Slade, 1993; and Gipps, 1994).

One of the premises of this work concerns the benefits to students and teachers of setting and making available those criteria to be used in assessing student performance. Drawing on Sadler's (1992) insights into assessment criteria, Gipps (1994) presents the ethical argument that it is only fair for students and teachers to have access to published statements of criteria. She also strengthens the fairness and equity argument with claims that an assessment approach that makes available relevant criteria has `the potential for providing motivation and a clear sense of direction particularly since all pupils who meet the criteria are eligible to receive recognition for this' (p.145). Advocates of a genre-based linguistic approach to assessment, including Macken and Slade (1993), similarly present an ethical argument for using explicit or sharply defined criteria written in linguistic terms. They claim that criteria of this type provide students with a coherent, public description of the characteristics of particular contexts and of the texts that are constructed within them. From this genre-based linguistic perspective, specified criteria serve the dual purposes of supplementing teachers' `gestalt' intuitions about quality of student writing, and of making teachers accountable for their judgements in terms of stated performance expectations.

In this article I wish to introduce a further consideration into this line of argument, and consider the possible ways in which teachers can produce readings of school writing and the various forces that may serve to shape how readings actually occur. The starting point is that while there is a considerable body of literature on the evaluative criteria of good writing, some dating from the early 1900s (see, for example, Colvin, 1902; Rice, 1903; Diederich, 1974; Smith, 1989), it is not yet known whether any stated or defined criteria mirror the actual criteria used by the teacher to award grades (Gilbert, 1989). Also unknown is whether the teacher's use of set criteria when reading is implicated in other practices that involve talk, values, beliefs and interaction. As Huot noted:

... other than results that measure the importance of content and organization in rater judgment of writing quality, little is known about the way raters arrive at these [grading] decisions. We are aware that rater variability is linked to the importance of expectation in the fluent reading process, but we have little or no information on what role scoring procedures play in the reading and rating process.

This observation points to the problematic nature of the assumption that any set of stated criteria necessarily turns a teacher's private practice with student-produced text into a publicly accountable procedure for judging achievement.

The concern in what follows is less with the question `Which criteria?' than with the connection between defined criteria and other kinds of assessment insights that writing teachers may draw on and combine when reading and appraising student writing. Specifically, the article exposes some of the interpretive issues surrounding an assignment task sheet and the accompanying statement of criteria, as distributed in a final year secondary English classroom. Before proceeding, it is worth emphasising that while the term `criteria' (singular: criterion) may be used in many different ways, it is defined here and understood throughout as follows:

criterion: A distinguishing property or characteristic of any thing, by which its quality can be judged or estimated, or by which a decision or classification may be made. (From Greek kriterion, a means for judging.)

Task and criteria specifications: The explicit agenda for reading

The assignment specifications shown on the next page were supplied to students at the beginning of a four-week poetry unit that drew on a range of British, American and Australian poems from different eras.

YEAR 12 ENGLISH -- RESPONSE TO POETRY

TASK: Poetic writing

GENRE: Poetry

PURPOSE: To entertain and express feelings

AUDIENCE: Teacher

STATUS: Summative

LENGTH: 300-500 words

TASK TOPIC: Choose one poem studied that you particularly

liked or found presented interesting ideas. You must write a piece of

poetic prose as response to this poem. This could be done in a couple

of ways:

* Use the first line of the poem as the first line of writing.

* Use a powerful image from the poem as a starting point.

* Use a theme from the poem to focus your writing.

FORMAT: On your assessment, please either rewrite the poem you are

using or paste a photocopy. Write a short passage where you discuss the

poem, explaining what aspects of the poem you are using to focus your

writing (50 words).

A distinct feature of the task specifications is the emphasis given to outlining the range of approved ways for students to display the inter-relatedness of their poetic writing and the chosen poem. Some of these ways (for example, re-using the studied poet's image or theme) could be subtly suggestive of the relationship between what the student wrote and the studied poem, while others were expected to be direct and obvious. In this second category was the teacher's requirement that student poems be accompanied by a copy of the stimulus poem and a statement of how it had acted as a focussing device.

Collectively, these requirements point to the teacher's desire to read individual student's writing as being intertwined with the writing of at least one other published poet. What is noticeably missing from the specifications and the accompanying statement of criteria (see Table 1) is any mention of the activity of reading or, more specifically, of the possible multiplicity of readings that could be made of the stimulus poems. Also missing is any indication of preferred readings or of how any variations in readings are to be received by the teacher-assessor.

[TABULAR DATA NOT REPRODUCIBLE IN ASCII]

Before commenting on Table 1, some mention should be made of the assessment system in which it has its origin. In December 1970 the Junior Public Examination was held for the last time in Queensland, to be replaced by a system of moderated school-based assessment. Since then, radical and clearly observable changes have occurred in the state's secondary education. School-based assessment was introduced, firstly, in a form which followed the recommendations of the Radford Report (1970), and then in the form of ROSBA, so named after the Scott Committee's report (1978) entitled A Review of School-based Assessment in Queensland Secondary Schools. While a comprehensive analysis of these changes is beyond the scope of this article, it is worth mentioning that a major distinction between the Radford and ROSBA schemes is that teacher judgements about student work no longer rely on direct inter-student comparisons, ranking of student performances, or the aggregation or weighting of scores. The comparisons rather between the work to be assessed (either a single piece or a representative sample) and defined criteria and standards.

It is widely recognised that secondary teachers have been diligent in the implementation of the Queensland model of wholly school-based assessment, reported to be `long-known for its radical approach to assessment' (Broadfoot, 1995: p. 5). The approach taken to specifying criteria and standards shown in Table 1 is only one of many developed by teachers in the state. While an examination of the range of so-called `criteria sheets' currently in use is not appropriate here, four points are worth mentioning. Firstly, Queensland secondary school teachers of all subjects are required to make available those criteria to be used in judging performance prior to students commencing work on an assessable task. Secondly, there is no official requirement that English teachers adopt a particular approach or format when developing criteria statements. Thirdly, since the mid-80s, there has been no systematic, ongoing professional development program to assist English teachers in producing criteria- and standards-specifications and in examining their nature and function. Fourthly, underlying the various approaches that have been adopted to formulating such specifications is the analytic assumption that tasks can be accurately decomposed to allow the identification of discrete criteria and the nomination of a set of standards on each criterion along the continuum from lowest to highest proficiency. Consider how there are eight criteria identified in Table 1, each one being chunked along the continuum of A+ to E-, with qualitative descriptors ranging across five levels. The criterion, `writing style', for instance, is chunked along a continuum that ranges from use of extensive discriminating and imaginative vocabulary clearly expressed' to `very restricted vocabulary very poorly expressed'.

The teacher who developed the criteria statement above reported that specifications such as these have the advantage of freeing students from the need to `second-guess' teacher expectations. Their value, the teacher said, lay in making explicit (and hence available for scrutiny) those features or characteristics that `counted' in the sense that they contributed to the overall grade awarded to the piece. From this position, the set of stated criteria in Table 1 is understood to provide the student with essential information about a quality performance, and the teacher with a regulatory, constraining template for reading that is applicable across readers and across texts.

The merit of attempts to wholly pre-specify all criteria relevant to a future appraisal remains a hotly disputed issue and it is one I do not wish to take up here. Similarly, there is no suggestion of a challenge to the ethical argument developed by Sadler (1992) and supported by Gipps (1994) that:

... once the criteria and standards are set they are available to the teachers and pupils which is only fair. This has the potential for providing motivation and a clear sense of direction particularly since all pupils who meet the criteria are eligible to receive recognition of this. By contrast, norm-referenced grading is a `hidden' process which, if seen as unfair, can be demotivating.

Gipps' point about the potential usefulness of criteria to motivate and to provide a sense of direction is widely accepted among assessment researchers, as is the need to demystify the assessment process. However, it seems that Gipps has assumed that there is a direct correspondence between defined criteria as the official account of reading and assessment practice, and the actual practice that assessors rely on to determine achievement. As applied to student writing, the assumption is that stated criteria represent a tool that can be applied independently of judgements about the institutionally bound contexts of authorship and readership, and about the ideological content of the writing in question.

From this perspective, as mentioned earlier, there is a telling gap in the criteria supplied in Table 1. What is omitted is any clue that there may be differences in how teacher and student read the `source poem `or writing stimulus. This omission hints at an unstated, but nevertheless powerful, assumption that teacher and student will (or should) adopt the same reading position, or at least that the student will willingly occupy the reading position that is modelled by the teacher and endorsed in classroom talk. The significance of this assumption becomes clear when one considers the gradings awarded to two pieces of writing, both being responses to Randall Jarrell's poem, `The Death of The Ball Turret Gunner'. Each student complied with the format requirements and provided (i) a handwritten copy of the poem; (ii) an explanatory paragraph outlining how the chosen poem was used to focus the writing; and (iii) the `original' piece of poetic prose. Jarrell's poem (in Heaney & Hughes, 1982: p.125) is reproduced here by way of introduction to the students' writing.

The Death of The Ball Turret Gunner

From my mother's sleep I fell into the state,

And I hunched in its belly till my wet fur froze.

Six miles from earth, loosed from its dream of life,

I woke to black flak and the nightmare fighters.

When I died they washed me out of the turret with a hose.

What writing (or is it what reading) is rewarded?

Text 1 (written by a male student)'

Poetic prose

Over the moor, an' through the glenn,

twisted and tormented,

Came the Beastie from hell's end.

Silent but screaming,

Violent but kind,

Cold yet it's steaming,

It shall look but not find.

The children skating,

now under the ice,

died a death,

of the Beastie's device.

Death doth dwell,

On the old, soon to be dead.

A dusty, creased face,

forms a front, for the dull knotted head.

The sand of the hourglass,

In his head,

Is time useless to the bones of the dead.

Into the city of razors he came,

I come for your little ones,

Don't try to resist,

for in my wake you will fail to exist.

Fat little pigs, spoiled and pampered,

eating fine food, living the fat life,

The Beastie sees, strikes like the plague.

A little fat man, cries for his dead wife.

I am the hermit whom you keep,

at the back of your mind,

when you cannot sleep,

The devil has died and taken a new force,

I am the hobgoblin who stands in his place.

Explanatory paragraph

I chose Jarrell's poem because it was short and violent. This reflects the imagrey [sic] of pilots `they were mere numbers etc...' Another reason was it seemed to be alot [sic] more realistic than many of the others. I felt no grief for the unfortunate gunner. In fact, I laughed.

There is a fantasy-like, nightmare quality about the student's poetic prose that is achieved, in part, through several powerful, even brutal images. The reader is introduced, for example, to the children skating/now under the ice; the Beastie from hell's end [that] strikes like the plague, the city of razors, the hermit, the devil [that] has died and taken a new force, and the hobgoblin who stands in his place. While these images could be read as indicators of the `originality' of the writing (criterion 2), they could also be interpreted as the student's efforts to pick up on the nightmare quality of Jarrell's poem and its references to waking from a dream - `I woke to black flak and nightmare fighters'. Also, in both the poetic-prose and the explanatory paragraph, the student-writer relied, as did Jarrell, on the first person form of address to evince a tone of the personal. Taken together, these features could be read as adding to the overall impact and effectiveness of the writing. However, undercutting this is the student's revealing claim that he `felt no grief for the unfortunate gunner'. These words give a hint of an unauthorised, even unanticipated reading of Jarrell's poem in which the pilot is both murderer and victim. Whereas the student chose to characterise the pilot as a figure of laughter, perhaps even of scorn -- `I laughed' -- the teacher had spoken of the pilot's bravery and self-sacrifice, describing him during class discussion as `a victim of the state' and `deserving of sympathy'.

Of the eight criteria listed in Table 1, only the first one refers to the student's explanation of how the chosen stimulus poem was used to focus the writing. The influential nature of the explanation as a `surrounding' text and the teacher's reading of it should not be understated however. In relation to Text A, it was the explanation that highlighted an apparent discrepancy in how the teacher and student `read' the stimulus poem and, more specifically, the identity of the pilot and his fate. This observation gains significance when it is considered in conjunction with the composite assessment picture of five D+s and three C-s that appeared on the criteria statement, the overall grading of C-, and the teacher's written comment on the returned script:

This [the explanation) needs to be clearer. You need to show directly the

relationship between the chosen poem and your writing. This is not shown.

Both the letter-grading outcome and the comment could be interpreted as signalling the limited success of the student's writing in allowing itself to be read as an authorised or acceptable response to Jarrell's poem. The awarding of an overall A- grading to Text 2 (see below) indicates that it was more successful in this regard, the performance pattern on the criteria ranging from A to B+.

Text 2 (written by a female student)

Poetic prose They carried him home in a preserving jar. Small bits of the young hero, floating in The formless mess that used to be human. His socketless eyeballs, pupils dilated, float in the chunky fluid, locked in the expression of death. No longer seeing the present they fill with images of the past.

A woman, face lined with age, bathed in love, waving goodbye. A great silver bird, soaring in the sky, Pursued by a flock of spitting vultures, the nightmare fighters, their talons digging into metal, their hoarse cries, gunfire.

A hand, his hand, a dealer of death Bent like a claw it sinks lifeless to the bottom of the jar. Iridescent white, it glows through the crimson liquid that surrounds it. Its fingers, which once held a weapon now uselessly attached to the palm of the hand.

Pieces of the airman, parts of himself, all that remains of his life form. Sinewy feet, which once carried him with dignity, are now carried themselves in the liquid. Tawny hair, which once blew in the breeze move with the currents in the vessel.

Ears, which once heard the birds sing, the planes roar, the explosions of bombs now hear gentle splashing.

Cold, still life, which once erupted with a cry of pain now whisper mutely `When I died they washed me out of the turret with a hose'.

Explanatory paragraph

I mainly focus my writing on the last line of the poem:

`When I died they washed me out of the turret with a hose.'

This line, I felt, conveyed the feeling of a loss of dignity. It gave to me the feeling

of inhumanness as well as being a stark, powerful image of the carnage of war.

I tried to focus my writing on the fact that after all he had done, the ball turret

gunner, after death, was just a dead body to the military. He was no longer a

person in their eyes.

There are three features in Text 2 which make apparent how it was embedded in and arose out of Jarrell's poem. These are (i) the mention of a mother-figure; (ii) the adaptation of the poem's water image; and (iii) the repetition of Jarrell's words. The following diagram sets out how the careful embedding was achieved and displays more effectively the interconnectedness of the two texts.

Selected extract from source text Student response
From my mother's sleep I feel A woman, face lined
into the state with age, bathed in
 love, waving goodbye

They washed me out of They carried him home in
 a preserving jar/small
 bits ... floating in the
 formless mess

 his hand ... it sinks
 lifeless to
 the bottom of the jar

 it glows through the
 crimson liquid

When I died they wash When I died they washed me
me out of the turret with out of the turret with
a hose a hose

What comes to light here are fairly obvious similarities between two texts, similarities that demonstrate how the student's writing was cued into the authorised reading. Consider, for instance, how the pilot is portrayed by the student as a victim worthy of sympathy and not, as suggested in the preceding text, a figure of scorn or laughter. Also evident is the student's strategic approach to limiting the possibilities of her `new' text. There are no newly' created characters and places such as the Beastie from hell's end, the city of razors, the devil, and the hobgoblin who stands in his place, as mentioned in the C- text.

The rewarding of the rewriting' approach with a high overall grading (A-) indicates that the achievement of the student writer relied on the construction of a text that permitted itself to be read as constructed in accordance with the demands of form (poetic prose), of discourse (wartime sacrifice and bravery), and of intertextuality (links with the wording of Jarrell's poem). From this perspective, there is an irony surrounding the teacher's comment recorded on the student's criteria sheet -- Well done. I enjoyed this -- you managed to convey your message clearly. These words beg the important question about whose meaning was being conveyed in the student's writing. They also make it possible to consider the extent to which an individual student's perceived compliance (or in the case of Text 1, non-compliance) with an authorised reading predisposed the teacher to read the writing as more (or less) successful.

Where to from here? This article shows that while the set of defined criteria could in principle, and may in practice, have formed the basis of an evaluative reading, the criteria alone did not explicate fully what the teacher took into account while reading. In this way the article makes the point that while defined criteria and standards may maintain the guise of assessment as orderly, even objective, they do not necessarily prevent other unstated considerations from coming into play as classroom writing is read and appraised. Further, it provides support for Sadler's (1985) observation that a possible lack of direct correspondence may occur between the experience or act of appraising, and the words used to account for any outcome of the appraisal process.

Despite appearances, the article does not devalue the usefulness of criteria statements as tools to communicate teacher expectations of a quality performance and to aid students in learning how to self-assess. Several notable researchers, including Beach (1986) and Sadler (1985, 1989), have argued the case in favour of teacher and student application of assessment criteria as a valuable means of improving performance. While the article does not challenge this earlier work, it does indicate that wholly anticipated criteria may provide only a partial account of those factors that influence teacher appraisal of student writing.

Similarly, it is not suggested that teacher autonomy in designing and publishing criteria should be reduced, or that a standardised format should be uniformly adopted. Teachers are well placed to practise authentic criteria-based assessment, given that they have the opportunity to couple together instruction and assessment tightly and effectively. The article does make clear, however, the need to develop a fully articulated and theorised model of criteria-based assessment in which attention shifts away from the constraining, regulatory nature of criteria to an awareness of their highly intersubjective, social nature.

The potential of such a model is threefold. Firstly, it could provide a basis for teachers to consider the important distinction between how stated criteria function in the appraisal process and their usefulness in justifying and defending outcomes of that process. Secondly, it could provide an opportunity for beginning and experienced teachers to reflect on how any set of defined criteria remains constantly open to negotiation, and open to the changes that a recursive process of teaching, learning, and assessment necessarily involves. Consideration of the malleability and negotiability of defined criteria also paves the way for teachers to examine the part that criteria play in maintaining power relations in the classroom. Thirdly, the model could offer encouragement to teachers and researchers to move beyond the traditional distinction between objective and subjective judgments and to reject, once and for all, the longheld superiority of the former. This move is of fundamental importance if teachers are to stop looking to defined criteria for true or self-evident assessments and bring a critical eye to bear on the connections between themselves, what happens in the classroom world, and the positions and practices that they adopt as readers of student writing.

REFERENCES

Beach, R. (1986). Demonstrating techniques for assessing writing in the writing conference. College Composition and Communication, 37, 1. pp. 56-65.

Board of Secondary School Studies. (1978). A Review of School-based Assessment in Queensland Secondary Schools. (E. Scott, Chair.) Brisbane.

Broadfoot, B. (1995). Editorial. Assessment in Education: Principles, Policy & Practice, 2, 1. pp. 5-7.

Campbell, W., Archer, J., Beasley, W., Butler, J., Cowie, R., Cox, B., Galbraith, P., Grice, D., Joy, B., McMeniman, M., Sadler, R. & Wilkes, R. (1983). Implementation of ROSBA in schools. Unpublished report to the Minister for Education, Brisbane.

Campbell, W., Basset, G., Campbell, E., Cotterell, J., Evans, G. & Grassie, M. (1975). Some consequences of the Radford scheme for schools, teachers and students in Queensland. Final report of Project. Brisbane: Australian Advisory Committee for Research and Development in Education.

Carey, R. (1988). Evaluations and whole language Paper presented to the New York State Education Department Whole Language Conference Rochester, New York.

Colvin, S. (1902). Invention versus from in English composition: An inductive study. Pedagogical Seminary, December. pp. 393-421.

Cox, R. & Haynes, D. (1991). Effective evaluation of writing: Communicating with children. Australian Journal of Reading, 14, 1. pp. 21-28.

Diederich, P. (1974). Measuring Growth it, English. Urbana, Ill.: National Council of Teachers of English.

Elbow, P. (1993). Ranking, evaluating and liking: Sorting out three forms of judgment. College English, 5,2. pp. 187-206.

Freebody, P. (1992). Inventing cultural-capital distinctions in the assessment of HSC English papers coping with inflation in an era of `literacy crisis'. In F. Christie (ed.), Literacy in Social Processes: pp. 96-108. Darwin: Northern Territory University Press.

Gilbert, P. (1989). Writing, Schooling and Deconstruction: From voice to text in the classroom, London: Routledge and Kegan Paul.

Gipps, C. (1994). Beyond Testing: Towards a theory of educational measurement. London: The Falmer Press.

Jarrell, R. (1982). The Death of the Ball Turret Gunner.' In S. Heaney and T.

Hughes (eds), The Rattle Bag. London: Faber & Faber.

Huot, B. (1990). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60, 2. pp 237-63.

Johnston, P. (1992). Constructive Evaluation of Literate Activity. New York: Longman.

Macken, M. (1989). Assessment of students' writing. Med research paper (unpublished). Geelong, Victoria: Deakin University.

Macken, M. & Slade, D. (1993). Assessment: A foundation for effective learning in the school context. In B. Cope and M. Kalantzis (eds), The Powers of Literacy: A genre approach to teaching writing. pp. 203-230. London: Falmer Press.

Rice, J. (1903). English: The need of a new bias in education. Forum, October. pp. 269-93.

Rothery, J. & Macken, M. (1991). Developing critical literacy: An analysis of the writing task in a Year 10 reference test. In F. Christie, P. Freebody et al., Teaching English Literacy, Volume II. pp. 215-52. Report of a Project of National Significance to the Commonwealth Department of Employment, Education, and Training on the preservice preparation of teachers for teaching English literacy.

Sadler, D. (1985). The origins and functions of evaluative criteria. Educational Theory, 35, 3. pp. 285-97.

Sadler, D. (1987). Specifying and promulgating achievement standards. Oxford Review of Education, 13, 2. pp. 191-209.

Sadler, D. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18. pp. 119-44.

Sadler, D. (1992). Expert review and educational reform: The case of student assessment in Queensland secondary schools. Australian Journal of Education, 36, 3. pp. 301-317.

Smith, C. (1989). A study of standards specifications in English. Master of Education thesis (unpublished), University of Queensland.

Smith, C. (1995). Teachers' reading practices in the secondary school writing classroom: A reappraisal of the nature and function of pre-specified assessment criteria. PhD thesis (unpublished), University of Queensland.

White, E. (1990). Language and reality in writing assessment. College Composition and Communication 41, 2. pp. 187-200.

Wyatt-Smith, C. (1995). Writing Pedagogy and Competing Conceptualisations of Student Achievement. Centre for Literacy Education Research (CLER): Griffith University.

Wyatt-Smith, C. (1996). Teachers' reading practices: The interplay of pre-specified assessment criteria and other factors. Literacy Learning: Secondary Thoughts, 4, 2. pp.37-44.