Bridging multimodal literacies and national assessment programs in literacy.

Unsworth, Len ; Chan, Eveline

Introduction: Reading as integrative interpretation of image and language

The increasingly prominent role of images in combination with language in the vast majority of texts students encounter has prompted widespread advocacy over the last decade or so of the need to redefine literacy and literacy pedagogy. Many have argued that the visual-verbal interface is now a crucial dimension of literacy learning and development (e.g., Andrews, 2004; Dresang, 1999; Dresang & McClelland, 1999; Kamil, Intrator, & Kim, 2000; Kress, 2000a, 2000b; Luke, 2003; Richards, 2001; Russell, 2000) Images, and their contribution to overall meaning, vary with the type of text. Overwhelmingly however, both the information in images and their effects on readers are far from redundant or peripheral embellishments to print. Because images are being used increasingly in a complementary role to words in representing the meanings central to a text, it is no longer adequate to consider reading simply as processing information in print.

This changing concept of reading, to embrace the negotiation of multimodal texts, has been incorporated in the current planning for an Australian National Curriculum in English, as outlined in the National English Curriculum: Initial Advice Paper:

The subject of English has historically been largely about the reading and writing of printed texts. More recently there has been debate about the growing significance of visual and non-print communication such as speaking and listening, combinations of visual information with language, and the new digital developments. In considering the tasks that young learners face in school, in their further education and training, and in workplaces, the argument has been that subject English should expand its scope to include more focus on these non-print forms.

Clearly these forms of communication are expanding, in and out of formal education, and so they have an important place in a national English curriculum. (National Curriculum Board, 2008, p. 8).

The Initial Advice paper goes on to indicate that the envisaged National Curriculum in English

... will also involve the systematic exploration and production of multimodal texts throughout the school years, in turn incorporating a growing understanding of how visual texts work, their structures, interpretation, and the effects of certain features. (National Curriculum Board, 2008, p. 12)

This position reflects a number of existing syllabus documents that require the interpretation of images to be included within a broader concept of literacy (Curriculum Corporation, 1994; New South Wales Board of Studies, 1998, p. 8). However, while there seems to be widespread consensus around this newly emerging dimension of the English curriculum, relative to the extensive, long established and ongoing tradition of scholarship supporting more traditional aspects of the curriculum, work exploring the co-articulation of image and language and the literate practices entailed in negotiating the meanings so constructed, is in its infancy (Kress, Jewitt, Ogborn, & Tsatsarelis, 2001; Macken-Horarik, 2003). Nevertheless, the re-conceptualisation of literacy beyond a focus on words alone to include the increasingly multimodal nature of contemporary paper and screen based texts needs to be reflected in national assessment programs. This is essential to optimise the use of such assessments in enhancing the literate capacity of all young people growing up into an exponentially intensifying multimedia digital age.

This paper first notes the very limited attention to this multimodal conceptualisation of reading comprehension in Australia's first National Assessment Program in Literacy and Numeracy (NAPLAN) (Ministerial Council on Education, Employment, Training and Youth Affairs (MCEETYA), 2007) in comparison with the previous New South Wales (NSW) Basic Skills Tests (BST) (New South Wales Department of Education and Training, 1998-2000, 2005, 2007a). It then outlines some of the results of an ARC funded study using the BST 1, explicating the different types of image/language relations that needed to be negotiated in responding to test questions, and indicating the relative difficulty level of these different types of intermodal meanings. The paper then outlines the results of the final phase of the study, which investigated year six students' understanding of different types of image/language relations in online texts dealing with science and social studies topics for primary school. Finally, implications for the development of a National Assessment Program that takes due account of the multimodal nature of contemporary paper and screen based texts, will be discussed.

NAPLAN and multimodal reading assessment

In Australia, mandatory group reading comprehension tests conducted by the States and Territories usually for year three, year five, and year seven children in government primary schools were replaced by national tests in 2008. A number of these tests did seek to assess children's understanding of images and language in reading materials. In NSW, the BST program had been administered to children in years three and five since 1994, and items to assess the role of images had been developed over time. An earlier study (Unsworth, Thomas, & Bush, 2004) noted the proportion of test items that involved the use of images in answering test questions as shown in Table 1.

In this 2004 study, we analysed all of the questions in the BST 1998-2000 to derive an exhaustive list of all the possible strategies for answering the questions correctly. We were able to group these into eight categories so that all strategies were included. The eight categories were:

1. Read main text

2. Read supplementary text

3. Read main text and supplementary text

4. Read main text and image

5. Read supplementary text and image

6. Read image

7. Read text structure

8. Prior knowledge.

The items involving images were those in categories 4, 5 and 6 and these are the proportions shown in Table 1. An example Strategy 4: Read the main text and the image, is found in BST 1998, test item 38, referring to the text on page 14 of the magazine. The question was 'Which picture shows the marine stinger with fronds?' and has a row of four images of different stingers in the question book below this question. The answer required reading the main text to determine which marine stinger had fronds, identifying the corresponding image, and locating the correct image from the test booklet. Strategy 5: Read supplementary text and image, similarly required integration of information from the caption text and the image, and Strategy 6 required reading of the image only.

Similarly, in our recent Australian Research Council funded project involving the BST in years 2005 and 2007 i, question descriptions as indicated in the explanatory material for teachers which accompanied the test results, were used as a starting point for identifying test items that targeted images. Then each item was further described in terms of image/language dependence, that is, whether the correct answer could be derived from the language alone, from the image alone, or from a combination of meanings from language and image.

For example, item 28 in the 2005 Year 5 materials refers to a painting from Tobwabba Art Gallery. (Image may be viewed at http://www.tobwabba.com. au/escaping_the_nets.htm). The question, In this artwork which shape shows a fish trap or net?, explicitly targets the image in its wording. However, the correct answer can only be obtained by synthesising meanings from both the text (... the various fish traps and nets shown by the dark areas) and image (by identifying the dark areas in the painting). Table 2 shows for the 2005 BST year 3 and year 5 and the 2007 BST year 7, the proportion of test items involving images as described above.

Hence it would appear that from about the year 2000 the proportion of test items in the year three and year five BST in NSW that involved images was about 33%. Now if we look at the proportion test items involving images in the 2008 NAPLAN the largest proportion is 8% for the year five test as indicated in Table 3.

Overall, in the 2008 NAPLAN only six test items from a total of 168 or 4% involve images (Table 3 shows eight items but one item is common to year 3 and year 5 and another is common to year five and year seven). One question, 'What is the Chimpanzee doing?' requires the reader to link the action of the chimpanzee in the image to drinking water referred to in the text. Another of these six items only requires the use of images for those students who do not readily understand what is meant by 'chooks'. This is because in the story of 'Lacy' in the year five and year seven test, the text only refers to chooks and chook house but question 23 refers to chickens, hence if you don't know what chooks are the image clarifies this. Two of the six questions ask what the main purpose of the relevant image is--referring to the life cycle of a frog in the 'Amphibians' text for years three and five, and the image of various dinosaurs depicting their 'weapons' in the 'Attack and Defence' text in year five. The remaining two test items relating to the text 'Endemism' for year nine, require the students to read maps.

Effectively then, with less than 4% of the questions involving images, the 2008 NAPLAN does not address the range of ways in which images contribute to the meanings of the texts--either conveying meaning in and of themselves or in combination with the main and/or supplementary (caption) language components. But when one considers the variety of semantic relations between images and the related language segments of texts, there are in fact multiple ways in which image/language relations construct meanings. Our recent ARC study indicates that while some of these semantic relations in image/language interaction are easily comprehended, others were among the most challenging of the test items in the 2005 and 2007 BST.

The relative difficulty of comprehending different types of image/text relations

The relevant aspects of our recent ARC study are reported in more detail elsewhere (Unsworth & Chan, 2008). Here we will simply summarise one main set of findings.

We found that the test items in the 2005 and 2007 BST included two basic types of image-language interaction which we described in terms of relations of elaboration and extension (Figure 1). In the first type, one mode elaborates on the meaning of the other by further specifying or describing it, while no new ideational element is introduced by the text or image. Two sub-types of elaboration are: equivalence, where ideational content corresponds across modes and so there is some redundancy of meaning; and, exposition, which refers to the re-expression or reformulation of the meanings of the image or the text in the alternative mode.

[FIGURE 1 OMITTED]

An example of equivalence can be seen where a descriptive caption provides the same information as depicted in an image. For example, in the 2005 Year 3 BST text 'Water Animal Records' (p. 2) the diagram depicting a large turtle on one side of a beam balance and ten human figures on the other side is accompanied by the caption, 'One leathery turtle weighs the same as 10 humans'.

An example of exposition, where the image elaborates on aspects of the text and vice versa, can be seen in the 2007 Year 5 BST stimulus (p. 9), '10 Years of Recycling' (Figure 2). Two sentences above the image provide a direct commentary on the data displayed in the bar graph. In the image, the vertical axis represents the amount of waste produced per person in hundreds of kilograms while the main text specifies '690 kilograms' (language more specific). Similarly, the commentary states 'This was more any other country ... except the USA' while the graph specifies the individual countries compared in the study (image more specific).

With the second basic type of image-language relation, extension, we also found two sub-types in this data. Augmentation may involve an image extending or adding new meanings to the text or the text extending the meanings of the image. For example, in the 2007 BST Year three text 'Puddles', adapted from The Puddleman by Raymond Briggs (2004), students needed to infer extra information from the text that was not in the images--the comic strip depicted two characters, while the words shown in speech bubbles came from three speakers.

[FIGURE 2 OMITTED]

The second sub-type, distribution, refers to juxtaposed images and text, jointly constructing activity sequences. For example, the image(s) might depict the end result of a process described in the verbal text. This occurs in the 2007 BST year three extract from 'Mr. Archimedes' Bath' by Pamela Allen (1980), where the text states 'the water rose' while the accompanying image shows water overflowing from the bath.

A total of 64 'visual' test items were identified from the 2005 and 2007 BSTs and the 2007 English Language and Literacy Assessment (NSW Department of Education and Training (2007b). The analysis of the image-text relations associated with these items were examined with data from the School Measurement, Assessment and Reporting Toolkit (SMART) (NSW Department of Education and Training, 2006) pertaining to test item descriptors, item difficulty measured in logits (?), and state-wide performance on the items as indicated by the percentage (%) of the test population that answered the question correctly. A clustering of results was observed, which was suggestive of a relationship between the relative difficulty of a test question and the type of image-text relation involved. A one-way analysis of variance (ANOVA), confirmed a significant difference in the mean item difficulty for each of the image-text relation types--in decreasing order of difficulty: 'augmentation', 'distribution', 'exposition' then 'equivalence'.

The strategic work required of students in intermodal meaning-making when the image/language relation is one of augmentation can be seen in a closer examination of two test items that posed the greatest difficulty in the Year 5 BSTs. First, item 28 in the 2005 test, mentioned earlier, referred to the Tobwabba Art Gallery text, which included a stylised Aboriginal painting concerned with aspects of fishing customs (http://www.tobwabba.com.au/ escaping_the_nets.htm). Only 44% of the whole NSW cohort of year 5 students was able to answer question 28 correctly ('Which shape shows a fish trap or net?').

Those who selected wrong answers tried to interpret the image by itself, or they referred back to the text but could not find the answer in the words, 'the fish traps and nets shown by the dark areas' in the painting, so relied on the image instead. In the 2007 test, the comic strip stimulus text 'Puddles' adapted from The Puddleman by Raymond Briggs (2004) also required an integrated reading text and images. Only 46% of the Year 5 cohort answered Question 11 correctly ('How many characters are in this text?'), making this the most difficult item on the test. Students who answered incorrectly attempted to answer from the images alone, which only depicted an old man and a young boy. The third character, the grandma, could only be identified through her speech shown in speech bubbles.

As part of her ongoing doctoral study, Ann Daly has shown that the spread of difficulty among the reading items assessing image-language interaction was similar in the Year 5 BST to the Year 3 BST with slightly more in the difficult range for the Year 5 BST. This was discerned by using the NSW DET (2006) software program, SMART (School Measurement, Assessment and Reporting Toolkit) for reporting and analysing the test results. The 'Item Analysis' function on SMART was used to order test items according to the percentage of students who achieved each skill, that is, from easiest to most difficult. Table 4, shows the number of 2005 Year 3 and Year 5 items that assessed image-language interaction out of the total number of items located in quartiles of difficulty.

The results of this study indicate the importance of integrative reading of language and images in constructing meaning and the demanding nature of this task for many students in negotiating multimodal texts where the meanings represented in image and language are different but complementary. Clearly, a national literacy assessment program needs to attend to these issues and to do so it needs to be informed by research-based theories of intermodal reading comprehension. But this discussion so far has only addressed image/ language interaction in static hard copy texts. It is also of crucial importance to address intermodal reading comprehension online.

Reading image/language relations in online texts

In the third year of the ARC project, (1) 32 year six students (17 boys and 15 girls) were followed up for a study of online reading comprehension. The participants included high, medium and low performers on the 2005 NSW BST from four metropolitan Sydney schools. Each student worked individually with a researcher to read online a selection of web pages from sites within Australian Museum (2003), responding to orally-presented questions relating to each of the web pages. The students answered a total of twenty questions which required the negotiation of a range of image-text relations, including those described above for the BST study. However, in addition to the interaction of image and language were challenges presented by how the material is organised on the internet, such as the need to co-ordinate information from images with text segments hyperlinked through 'roll overs' or 'pop-up' links.

A more detailed account of this study is provided in Eveline Chan (2009), but what is significant to note briefly here is that the relative difficulty of the different types of image/text relations found in the BST study (augmentation [right arrow] distribution [right arrow] exposition [right arrow] equivalence) seemed to be reflected also in the students' negotiation of online texts.

For example, in an online text about the life cycle of a millipede (Australian Museum, 2003) equivalent information about how many legs a hatchling has is provided in the image of the hatchling and in the pop up text, and 91% of the students provided the correct answer of three pairs of legs (Figure 3).

In contrast, when students read a webpage about The Rainbow Serpent (Figure 4) (Australian Museum, 2003) and were asked: 'Which element of nature is represented by the Serpent shown in the painting?', only 13% students answered correctly. This item required readers to integrate information from the caption ('Many of its characteristics are similar to the Ribboned Pipefish') and second paragraph of the main text ('There are many versions of the Serpent, each representing central elements of nature: the sky is evoked by rainbows ... the land by snakes and the sea by pipefishes') with reference to an image depicting a 4000-6000-year-old representation of the Rainbow Serpent.

[FIGURE 3 OMITTED]

[FIGURE 4 OMITTED]

While some students found this item difficult for various reasons (e. g. they 'didn't understand the question'; 'didn't understand some words'), many answered incorrectly because they: 'didn't look at the caption', 'didn't look for answer in text'; 'looked for answer in the words' (but could not find); or made an association between the rock painting and land. Thus, even though students may have referred to different parts of the page for an answer, it was necessary to integrate information from the image, caption, and main text in order to infer the correct answer.

The challenges of the integrative reading of images and language in multimodal texts are clearly important in reading both traditional paper media texts and perhaps even more so in online texts, especially as further issues specific to online formats, such as the non-simultaneous, sequential display of a text window followed by an animated image, for example, add complexity to the integration of information required for coherent meaning-making from multimodal texts.

Conclusion

The results of the research reported in this paper indicate that different types of image/language relations in hard copy or online texts differ in the degree of difficulty they pose for students, and that negotiating some of these image/ language relations are among the most challenging tasks encountered in reading comprehension tasks. It is important to note that the items involving the integration of meaning from images and language in the NSW BST are very typical of routine curriculum area reading required of students in the middle to upper primary school. A national literacy assessment program that reflects a national curriculum perspective addressing the multimodal nature of contemporary literacy needs to be informed by a systematic account of the ways in which images of various kinds interact with language in different kinds of hard copy and online texts to construct the interpretive possibilities to which readers respond. This is essential to understand the nature of reading comprehension in relation to contemporary multimodal texts, to provide a basis for pedagogy to ensure students are being taught how to most effectively interpret such texts, and to inform the assessment of reading comprehension so that what is being assessed is addressing the fundamental competencies needed to negotiate the actual texts students need to read and understand. When governments and education authorities use large-scale group reading comprehension tests as key indicators of students' literacy standards, effectiveness of teaching and of school resource needs, it is essential that such tests include the negotiation of image/text relations that is a normal part of curriculum area, and community reading practice.

References

Allen, P. (1980). Mr. Archimedes' Bath. NY: Angus & Robertson/Harper Collins Publishers.

Andrews, R. (2004). Where next in research on ICT and literacies. Literacy Learning: The Middle Years, 12(1), 58-67.

Australian Museum. (2003). Wild Kids (homepage) Retrieved 5 February, 2009, from http://www.amonline.net.au/wild_kids/index.cfm.

Briggs, R. (2004). The Puddleman. London: Random House. Chan, E. (2009). Image-language interaction in online reading environments: Challenges for students' reading comprehension. Paper presented at the 44th RELC International Seminar 'The Impact of Technology on Language Learning and Teaching: What, How and Why'. Singapore, 20-22 April 2009.

Curriculum Corporation. (1994). English--A curriculum profile for Australian schools. Carlton, Victoria: Curriculum Corporation.

Daly, A. (2009). Comprehending complexities in multi-semiotic texts: Indigenous students in different geolocations. Unpublished PhD thesis, University of New England, Australia.

Dresang, E. (1999). Radical Change: Books for youth in a digital age. New York: Wilson.

Dresang, E., & McClelland, K. (1999). Radical change: Digital age literature and learning. Theory into Practice, 38(3), 160-167.

Gleeson, L. (1991). Big Dog. Sydney: Ashton Scholastic.

Kamil, M., Intrator, S., & Kim, H. (2000). The effects of other technologies on literacy and learning. In M. Kamil, P. Mosenthal, P. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. 3, pp. 771-788). Mahwah, New Jersey: Erlbaum.

Kress, G. (2000a). Multimodality. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures (pp. 182-202). Melbourne: Macmillan.

Kress, G. (2000b). Multimodality: Challenges to thinking about language. TESOL Quarterly, 34(3), 337-340.

Kress, G., Jewitt, C., Ogborn, J., & Tsatsarelis, C. (2001). Multimodal teaching and learning: The rhetorics of the science classroom. London: Continuum.

Luke, C. (2003). Pedagogy, connectivity, multimodality and interdisciplinarity. Reading Research Quarterly, 38(10), 356-385.

Macken-Horarick, M. (2003). The children overboard affair. Australian Review of Applied Linguistics, 26(2), 1-16.

Ministerial Council on Education, Employment, Training and Youth Affairs (MCEETYA) (2007). National Assessment Program Literacy and Numeracy (NAPLAN). Reading sample questions. Accessed on 18 Feb 2008 from http://www. naplan.edu.au/test_samples/test_samples.html

National Curriculum Board (2008). National English Curriculum: Initial Advice Paper. Retrieved 5 February, 2009, from http://www.acara.edu.au/engagement/past_papers. html

New South Wales Board of Studies. (1998). English K-6 syllabus and support documents. Retrieved 5 February, 2009, from http://k6.boardofstudies.nsw.edu.au/ english/english_index.html

New South Wales Department of Education and Training (1998-2000). Basic Skills Tests. Sydney: NSW Department of Education, Educational Measurement and School Accountability Directorate.

New South Wales Department of Education and Training (2005). Basic Skills Tests. (Water). Sydney: NSW Department of Education, Educational Measurement and School Accountability Directorate.

New South Wales Department of Education and Training (2007a). Basic Skills Tests. (Puzzles and problems). Sydney: NSW Department of Education, Educational Measurement and School Accountability Directorate.

NSW Department of Education and Training (2006). School Measurement, Assessment and Reporting Toolkit (SMART), Retrieved from https://www.det.nsw.edu.au/ proflearn/links/detintra.htm

New South Wales Department of Education and Training (2007b). English Language and Literacy Assessment (Places and possibilities). Sydney: NSW Department of Education, Educational Measurement and School Accountability Directorate.

Richards, C. (2001). Hypermedia, internet communication, and the challenge of redefining literacy in the electronic age. Language Learning and Technology, 4(2), 59-77.

Russell, G. (2000). Print-based and visual discourses in schools: implications for pedagogy. Discourse: Studies in the cultural politics of education, 21(2), 205- 217.

Tobwabba Art Gallery http://www.tobwabba.com.au/escaping_the_nets.htm).

Unsworth, L., & Chan, E. (2008). Assessing integrative reading of images and text in group reading comprehension tests. Curriculum Perspectives, 28(3), 71-76.

Unsworth, L., Thomas, A., & Bush. R. (2004). The role of images and image-text relations in group 'Basic Skills Tests' of literacy for children in the primary school years. Australian Journal of Language and Literacy, 27(1), 46-65.

Len Unsworth & Eveline Chan

University of New England

(1) An Australian Research Council (ARC) Linkage Grant in conjunction with the Educational Measurement and Schools Assessment Directorate of the New South Wales (NSW) Department of Education and Training (DET), 2006-2008, entitled New dimensions of group literacy tests for schools: Multimodal reading comprehension in conventional and computer-based formats.

Table 1. Proportion of Year 5 test items involving images
in the BST 1998-2000

Data on Test Items and            BST          BST          BST
their relation to Images          1998         1999         2000
                                  Year 5       Year 5       Year 5

Total number of                   11           19           14
 images in magazine
Total number                      47           46           47
 of test items
Number of test items              4            9            17
 involving the use of images
Proportion of test items          9%           20%          36%
 involving images

Table 2. Proportion of Year 3 test items involving images
in the 2005 and 2007 BST

Data on Test Items and            2005 BST      2005 BST      2007 BST
their relation to Images          Year 3        Year 5        Year 5

Total number of                   24            23            34
 images in magazine
Total number                      36            46            46
 of test items
Number of test items              12            15            14
 involving the use of images
Proportion of test                33%           33%           30%
 items involving images

Table 3. Proportion of Year 3, 5, 7 and 9 test items involving
image in the 2008 NAPLAN

NAPLAN Data on             2008        2008        2008        2008
Test Items and their       Year 3      Year 5      Year 7      Year 9
relation to Images

Total number of            11          14          20          9
 images in magazine
Total number               38          36          46          48
 of test items
Number of test items       2           3           1           2
 involving the use
 of images
Proportion of test         5%          8%          2%          4%
 items involving
 images

Table 4. Spread and location of 2005 Year 3 and Year 5 items
assessing image-language interaction

2005 BST      1st quartile    2nd           3rd           4th quartile
(number       (easiest        quartile      quartile      (hardest
of items)      items)                                      items)

Year 3        5 out of 9      2 out of 9    2 out of 9    3 out of 9
 (12 out
 of 36)
Year 5        5 out of 11     1 out of 12   6 out of 12   3 out of 11
 (15 out
 of 46)