How did English schools and pupils really perform in the 1995 international comparisons in mathematics?

Prais, S.J.

The recently published results of mathematics tests set to representative samples of pupils in over forty countries provide an important opportunity to re-assess priorities for reforms in English schooling. Five Western European countries--Austria, Belgium, France, Netherlands, Switzerland are suggested in this critical study as providing appropriate standards for England's immediate aims. Attainments there are shown to be about a year ahead of England for average pupils at age 14; the gap is larger for those of below-average attainment, suggesting some structural bias in English schooling. The gap is particularly evident in those arithmetical fundamentals which need to be mastered by all school-leavers (rather than in advanced aspects suitable only for mathematical specialists); and that gap has its clear origins at the primary level of schooling.

An immense quantity of statistical data on the schooling attainments of 13 and 14 year-olds in mathematics and science in 46 countries has now emerged from the IEA's sample surveys of pupils carried out at the beginning of 1995--known as TIMSS--and published in record time in comparison with previous similar studies.(1) Not all countries were able to carry out their surveys with equal success (there were greatly varying co-operation rates amongst countries by the randomly-selected sample of schools, and greatly varying participation rates of pupils within co-operating schools), and careful scrutiny and adjustment are necessary in reaching reliable conclusions. This note attempts to answer in critical detail some half-dozen questions of public concern in England; the attempt is based on the material so far published, namely, national reports for England by the (English) National Foundation for Educational Research (which carried out the survey in England and Wales) and the international summary reports published by the TIMSS Centre in the United States (which co-ordinated the national studies) giving summary results available so far for other countries.(2) It will be necessary to wait for the publication of full reports from other countries before a final assessment of England's position can be made; but those full national reports lie some time ahead and, in the meantime, it is worth looking carefully at what has so far become available.

It seems agreed by all commentators that English pupils' performance in mathematics was disappointing. The questions we shall consider in detail below may be outlined as follows:--

(1) In relation to the average pupil, how much are English pupils behind those of the same age in other relevant countries, measuring the lag in terms of equivalent years of schooling? The tests were administered to two school-years (approximately at ages 13 and 14), and the average rise in scores between those years helps in interpreting the average gap between England and other countries. An important issue is: which countries, of those which supplied data that are adequately reliable for this comparison, can sensibly be taken as relevant exemplars to improve and guide improvements in English pupils' attainments?

(2) Previous international comparisons indicated the main shortfall in English pupils' mathematical attainments lay in basic arithmetic, rather than in other branches of mathematics (geometry, data handling, etc.); we need to examine whether this is still SO.

(3) We then examine the achievements of our top-attaining pupils; this is clearly relevant to the country's future capabilities in research and technology. Previous international surveys had suggested that pupils at the top end of the attainment-range in England performed perhaps even better than the corresponding proportion at the top in other countries; is this still so?

(4) Next we examine what the new survey tells us about the relative position of our low-attaining pupils: does the new survey support the view that England still has a `long tail of low achievers', the employment prospects of whom are likely to become ever more precarious with the growth of automation and of industrial competition from low-wage countries? Participation in such a survey is likely to be weaker amongst low-attaining schools and pupils, because of other pre-occupations or embarrassment; we shall need to look at response-rates particularly carefully in assessing the relative extent of England's low-attaining pupils.

(5) Aside from looking at schooling outcomes in terms of pupils' mathematics scores, we need to ask what the survey tells us about causative factors that have entered policy discussions; for example, how much teaching time is devoted to mathematics, what is the predominant form of classroom organisation (mixed ability, or a number of classes organised according to attainments/ability, etc); what is the predominant teaching style; to what extent are textbooks used?

1. The average pupil's mathematical attainments at 13-14 Of the approximately forty countries which provided returns to the IEA of pupils' attainments in classes in which the majority were aged 13 and 14, corresponding to Years 8 and 9 in our schools in England, there are five neighbouring Western European countries with which it seems particularly helpful to compare England: Austria, Belgium, France, The Netherlands and Switzerland.(3) Even the returns for those five countries are subject to reservations on closer scrutiny; and those reservations will sometimes have to be taken into account below. For example, the returns for Belgium were satisfactory only for the older class aged 14; but for the younger class aged 13, some low-attaining pupils--amounting to a tenth of all pupils--were not covered by the sampling scheme (having been retained in primary school for an extra year before moving into secondary `vocational classes' in, nominally, Year 9).(4) A question mark was hung by the IEA over the returns of the Netherlands since their overall participation rate--even after including replacement schools--was only 59 per cent; however, England's overall participation, at 78 per cent, was not all that much higher: for comparisons of average attainments we shall probably not be badly misled by including the Netherlands.

A few words are in order on the relevance for direct comparison with England of some other countries participating in that survey. Of European countries, the results for Germany--which included East Germany in this survey-are of considerable interest because of radical developments in its schooling organisation in the past generation. Germany had previously participated in the first IEA international survey of 1964, when its average and below-average pupils at age 13 were well ahead of England's; those high standards provided the foundation for the repute and success of its technical and commercial apprenticeship system. Organisational reforms in schooling were gradually put into effect in subsequent decades in most West German Lander with the help of a new generation of teachers, very often amidst heated controversy and with continuing debate. Reforms affected such matters as teaching styles--a less authoritarian role for the teacher, more initiative to the child; and schooling organisation--less rigidity in the secondary selective system and the introduction of some comprehensive secondary schools. The broad intention, understandable in view of that country's grim political history, was to bring its schools closer to a `democratic' and more child-centred orientation in line with `progressive' Anglo-American educational ideals and schooling practices. In addition, there were shifts in the nature of the mathematics curriculum with the introduction of considerable formal elements of `modern mathematics' (set theory, arrow diagrams, operators, ...).(5) These changes were implemented to varying extents in the different Lander; overall they are now shown by TIMSS to have had the consequence that the outstanding mathematical performance of 1964 by German average and below-average pupils can hardly be traced today in Germany as a whole.

There are now considerable variations in average attainments amongst the constituent Lander, amounting to the equivalent of about one and a half years' learning. Those Lander that remained closer to their traditional schooling methods (mainly in SW Germany) continue to show mathematical attainments substantially ahead of England's; this appears from a limited analysis by Lander published in the German TIMSS report, and is consistent with observations by the National Institute teams on visits to mathematics classes in SW Germany.(6) Economic consequences for Germany as a whole must be expected to become increasingly evident in its standards of vocational training and, eventually, must be expected to affect the country's industrial efficiency and international competitiveness. The important exceptions are the SW adjoining Lander of Bavaria and Baden-Wurttemberg (21m. combined population, main cities--Munich, Stuttgart) with their prosperous vehicle, machine tool and general engineering industries. Bearing in mind the continuing controversy on its schooling policies, these TIMSS results indicate that the country as a whole no longer provides an easily helpful exemplar to advance English pupils' attainments in mathematics.(7)

The attainments of pupils in Scandinavian countries--Sweden, Norway, Denmark--were fairly similar to England's (only Sweden was a shade higher). Compulsory schooling in these countries begins two years later than here; it is not clear that their pupils' attainments carry any obviously valuable lessons for England. It is prudent also to retain reservations at this stage--without further direct observation of current schooling processes--regarding the relevance for our purposes of the many ex-Soviet Bloc countries covered in the survey (Czech Republic, Slovak Republic, Hungary, Russian Federation, Latvia, Lithuania, Bulgaria, Romania, Slovenia): while conditions there are changing, can we yet be satisfied that the balance of values of their system of schooling is adequately relevant to England's? And can we be as sure of their application of random sampling methods and the administration of the tests? Outside Europe, the Far Eastern countries (Singapore, Korea, Japan, Hong Kong) are clearly well ahead of Western Europe in their recorded attainments, though similar worries cannot entirely be dismissed from the mind. The United States with its expensive schooling system needs no more than a mention: its serious problems with school `drop outs' and under-achievement are well-known, and its low scores in TIMSS confirm earlier IEA surveys that it hardly provides an exemplar for England.

Turning now to the scores for England and the five neighbouring Western European countries abstracted in Table 1: it will be seen that the average score on the TIMSS mathematics test for English pupils at age 14.0 was 506, and had risen by 30 points compared with the previous school-year. The average score for the five Western-European countries at age 14.3 averaged 540; the English score was thus some 36 points lower, corresponding to just over a year's learning (at the English rate of learning). Adjusting for the small difference in average ages of the pupils sampled in England and Western Europe, the lag of average English pupils may be put at about 11 months. We can confirm that lag directly from the table, in that the English average score in Year 9 (506 points) is the same as the Swiss score for Year 8, and close to the average Western European score for Year 8 (501 points). In this overall respect the results of this 1995 survey broadly confirm England's lag of about a year behind Western European countries often noted in previous comparisons. It is perhaps of some consolation that the lag does not seem to have grown any larger at age 14; on the other hand, despite the introduction of a National Curriculum in England together with many other educational reforms, the gap has so far not obviously narrowed.

Table 1. Mathematics attainments by average 13 and 14 year-old pupils, England compared with selected other countries, 1995

 Average score Median
 Age score Annual
 Year 9(a) Year 8 Year 9 at age 13 gain(b)

England 14.0 476 506 482 30
Austria 14.3 509 539 509 30
Belgium 14.2 ..(c) 546 539 ..(c)
France 14.3 492 538 498 46
Netherlands 14.3 516 541 519 25
Switzerland 14.2 506 545 519 40

Average WE(5)(d) 14.3 501(c) 540 510 39(c)
Germany 14.8 484 509 .. 25
United States 14.2 476 500 472 24

Source: Based on TIMSS Mathematics, tables 1.1, 1.3 and 1.8.

Notes:

(a) English Year 9; international Year 8 (Swiss Year 8 for most Cantons; Year 7 elsewhere).

(b) Between average of Year 9 and Year 8.

(c) Excluding Belgium because of sampling frame deficiencies (see text; and TIMSS, p. 28, n. 8).

(d) Of 5 countries Austria--Switzerland (ie excluding England and Germany), weighted by population.

(e) Average of column (ie excluding Belgium).

(f) Not calculated by TIMSS because such a large fraction of pupils aged 13 were in a lower class, and were outside the scope of the survey.

Much the same conclusion is to be derived from a slightly more exact comparison based on the median pupil at age 13 (penultimate column in the table), calculated by TIMSS from returns for pupils in both Years 8 and 9. Both these classes include pupils in their thirteenth year of age; by choosing pupils with a calendar age of 13 from both year-groups, a tighter comparison is possible. In selecting the median pupil, allowance was made for the proportions of pupils of that age who, in Continental countries, had repeated a school-year and were in a lower class, or had skipped a year and were in a higher class. The net effect is to put the calculated gap in scores between England and the Western European average at 28 points, instead of the 36 points noted above; this again corresponds to about 11 months' learning at the average rate of learning in English schools.

It will be noticed that the average increase in pupils' attainments between classes of pupils predominantly aged 13 and 14 in the listed Western European countries is about a third higher than in England (39 points rise a year, compared to 30; see final column of Table 1).(8) The gap between England and these countries can therefore be expected to increase with age and approach almost a two years' gap for the average pupil when they reach school-leaving age. This corresponds broadly with the impressions of the National Institute's teams of school inspectors and teachers who visited Swiss (and Germany's Baden-Wurttemberg and Bavarian) classes at ages 15-16 in recent years, who frequently commented on a `one-to-two years' gap' in mathematical attainments. Some of that more rapid rise in classroom standards on the Continent is attributable to class-repetition by, say, 2-3 low-attaining pupils; those weaker pupils nevertheless often stay on until they have completed the final year of schooling so as to attain the educational requirements for an apprenticeship in their chosen vocation.

Switzerland is close to the top performing West European country, with Belgium only very slightly (but not statistically significantly) ahead; Switzerland's performance in this TIMSS survey of 1995 thus echoes the findings of the IAEP Survey of 1991 (Belgium did not take part in that earlier survey). Switzerland also applied the TIMSS tests to an additional third year, when pupils were of average age 15:1; their average score had risen a further 45 points to 590, similar to the rise of 40 points between the previous two years, and confirming a more rapid rate of learning in that country between successive secondary school classes as compared with England.(9)

For other countries which, as discussed above, are of less immediate policy interest for comparison with England, the average lead (or lag) for pupils at age 14 in terms of equivalent number of years learning at the English pace--can be summarised as follows: Germany, lag of 6 months (but Bavaria, a lead of 11 months); Scandinavia, no significant lead or lag; United States, lag of about 4 months; ex-Soviet bloc (unweighted average), lead of about 6 months; and, most notable, the Far Eastern countries with an average lead of 3.5 years.

Singapore was the world leader in these tests, at the equivalent of about a year ahead of Japan, and 4.6 years ahead of England. These attainments of pupils in the Far East may stretch the credulity of some English readers. They become intuitively just a little more understandable when expressed in terms of percentiles: the median pupil in Singapore had a score close to that of English pupils at the top fifth percentile; and the Singapore pupil at the lowest fifth percentile attained a score close to that of the English median pupil. Perhaps a further step towards reconciling these results with English experience is to note that the national tests in mathematics at age 14 in England (Key Stage 3 SATs) showed enormous variations amongst schools: at the top decile of English schools in 1995 the average pupil was at a National Curriculum Level of 5.34, while at the lowest decile of schools the average pupil was at Level 3.57. Each NC Level corresponds notionally to two years of learning, so that there is now a notional gap of 3.6 years' learning within England between average pupils in schools at the top and bottom deciles--roughly the gap between Japan and England.(10)

The question could justifiably be asked--what prevents English pupils reaching Japanese or Singaporean summits? But bearing in mind the great cultural gaps, it seems more realistic at this stage to address the more modest practical challenge and ask: what is the nature of the weaknesses which prevent English pupils attaining the standards of our leading Western European neighbours?

2. Arithmetic

Among the more striking international contrasts in pupils' mathematical attainments to emerge from the immediately preceding tests carried out in 1991 (by the IAEP) was a simple decimal multiplication: 9.2 x 2.5. This was calculated correctly by 55 per cent of 13 year-old pupils in France, Italy and Switzerland; but by a mere 13 per cent of English pupils--a lower proportion than in any of the twenty countries participating in that survey.(11) It confirmed the serious English lag in basic arithmetic detectable in the earlier IEA large-scale sample surveys of 1981 and 1964, and which had been noted with varying degrees of worry by many previous observers--going back over a century ago to the visits of Matthew Arnold (holding a position equivalent to our present-day HM Chief Inspector) to European classrooms.

From the 1995 TIMSS survey, the following very basic question in arithmetic deserves attention here (the choice of suitable questions for the diagnostician is not great, as the results for only a few of the questions administered so far have been published: as it happens, this one illustrates the issue well):-

6000

- 2369

It was a multiple choice question in which one of only four possible answers was to be ticked by the pupil (A 4369; B 3742; C 3631; D 3531). It was not a test in mental arithmetic (of the kind many Continental children would be able to do `in their head' at ages 10-11), but was set out in the test paper in a vertical fashion so that it could be calculated in the ordinary written way. While not necessarily the best question that can be devised for testing subtraction, it must surely be regarded as an adequate test of competence in the most basic of numerical skills. This kind of question is usually taught as written (not mental) work in English primary schools by Year 5 (age 10-11); however, these skills are often not well consolidated in England, and many teachers expect pupils to rely mainly on calculators.(12)

Of 13 year-olds in the five Western European countries listed above, 92 per cent answered this question correctly; Switzerland was the highest of these countries, with 96 per cent answering correctly; Belgium was at that same high level. In Germany--which otherwise did not do well, as noted above--93 per cent answered correctly; and even in the United States 88 per cent answered correctly. In England only 59 per cent did so. If it had been an `open question' (rather than multiple choice, with only four possible choices), one might expect--on the conventional 'correction for guessing'--that under half of all English children at age 13 would be able to answer this question correctly. Of the 41 countries included in the survey, average scores lower than England's were recorded only by Colombia and South Africa (both with 57 per cent correct; Kuwait joined this group of low-attaining countries for 14 year-olds). Scotland's overall scores in mathematics were generally a little below England's, yet on this basic question Scottish pupils did better--with 75 per cent answering correctly (can this advantage in arithmetical fundamentals be the basis for Scotland's reputedly better schooling?); even so, Scotland was among the lowest four of the forty participating countries for which results for this question were reported.(13)

Some small improvement by English secondary pupils took place in this basic skill in their next year at school--from 59 per cent answering the above question correctly at age 13, to 65 per cent at age 14. Not too much further progress is to be expected at these ages since this kind of exercise is taught in primary rather than in secondary schools; even if that modest rate of progress were maintained, it is clear that Western European standards of competence are quite unlikely to be achieved by English pupils before they reach school-leaving age. Indeed, we have to take the TIMSS results as telling us that we must now expect about a third of English youngsters to reach school-leaving age unable to carry out such a basic sum, compared with under one in ten in Western Europe (those looking for clues on how an under-class is created by the English schooling system, could well begin by contemplating the international contrasts at this very basic level).

A second, slightly more difficult, arithmetical question from the survey of 1995 may be quoted:-

Peter bought 70 items and Sue bought 90 items.

Each item cost the same and the items cost 800 [pounds sterling]

altogether. How much did Sue pay?

It was an open question (ie not multiple choice) concerned with proportionality; the numerical work was not complex (it involved calculating 800/160, that is, 80/16). It was answered correctly by 54 per cent of Western European pupils at age 14 (amongst whom, by 60 per cent of Swiss pupils), but by a mere 17 per cent of English pupils.(14) Pupils in both Germany and the United States were better in this question on proportionality (37 and 23 per cent correct, respectively) than those in England, but markedly below the five Western European countries considered here. The rise between ages 13 and 14 in the proportion of English pupils responding correctly to the question on proportionality was a mere 3 per cent, from 14 to 17 per cent. Only three of the forty participating countries--South Africa, Colombia and Kuwait--produced lower average scores than England.

The contrast between German pupils' exceptional competence in the subtraction question quoted above, and their weakness in this question on proportionality, points to some lag in their curriculum at the secondary stage (perhaps associated with the recent over-emphasis in secondary schools in many Lander on formalistic aspects of `modern mathematics', with an inadequate number of contextualised exercises), whereas the lag in England arises fairly clearly at the primary stage.(15) As it happens, the precise subtraction question mentioned above was also put to primary school pupils at age 9 in parallel IEA tests at that time; only 36 per cent of English pupils provided a correct answer, compared with 86 and 92 per cent in The Netherlands and Austria respectively (these were the only two of our five neighbouring Western European countries which participated at the primary stage). Allowing for the multiple-choice nature of the way this question was asked (and making the usual assumption on guessing), it seems that at age 9 only about 15 per cent of English pupils can be expected to answer correctly an open question of this kind, compared with about 90 per cent in The Netherlands and Austria. It is failings at this very basic level--failings in arithmetical fundamentals that ought to be mastered at primary and early secondary stages of schooling--which make subsequent training in technical occupations difficult for so many English school-leavers.

It might be hoped that English pupils' comparatively high attainments in science, as shown in IEA tests carried out at the same time as those in mathematics, in some ways could balance (or offset) their poor showing in mathematics; but such a hope is of limited significance. Those science questions involved recognising basic natural `scientific' concepts (temperature scale, battery circuit, an insect's body parts, ...) together with problems which did not require any serious numerical calculations (hardly more than single digits); such `scientific expertise' is clearly of limited value since applications usually require at least some basic arithmetical manipulation.

In looking at attainments in broader groups of mathematical questions included in TIMSS, it is necessary to bear in mind that broader averages may hide the nature of the true deficiencies in English teaching processes. The reason is that international agreement was necessary to ensure that questions included in the TIMSS survey related to topics and levels of difficulty covered in a fair proportion of the participating countries. Agreement is easier to reach on common fundamental topics, rather than on more advanced topics which may be taught in different countries at varying ages--if at all. For better or worse for our purposes here, the participating countries varied widely in their general economic, technological and cultural development. The inevitable result was that the tests are sounder in respect of foundation topics required by all pupils than in respect of the attainments of top pupils; and the range of questions included were inevitably not of a standard to be wholly helpful in diagnosing problems specific to England. The more difficult kind of mathematical problems which had been observed by National Institute research teams as being tackled competently by average pupils in Continental classrooms--that is to say, questions judged difficult on the basis of what would be expected of average English pupils of the same age--were not as frequent in TIMSS as we would have chosen. At the other extreme, unduly simple questions were included--such as, `write a fraction that is larger than 2/7'--which are too obvious for pupils at that age to provide scope for discrimination between countries (England, 79 per cent correct on that question; Switzerland 83 per cent both at age 14); an exercise involving manipulation of fractions would have been more to the point at this age.(16)

In the category of questions that were unduly simple we may also include some that were more in the nature of tests of vocabulary, or mere verbalisms, than tests of true mathematical competence ('what is the ratio of the length of the side of a square to its perimeter?'; for English pupils this mainly requires knowing the meaning of the--essentially Greek word--`perimeter'). Including such questions in an overall average for mathematics inevitably muffles the nature and severity of English pupils' problems in acquiring basic mathematical competences: it should be noticed that both the omission of difficult questions, and the inclusion of unduly easy or merely definitional questions, contribute to that muffling.

The full extent of the shortfall in English pupils' attainments in comparison with the Continent is thus not revealed by looking at broad average of questions set in TIMSS. No careful student of the results of the magnificent international enterprise represented by TIMSS would, of course, be misled by overall averages of items of such disparate difficulty; but some hurried commentators may have misled themselves, and then misled others in their rush to defend the way mathematics has been taught in England, by emphasizing that England's average results in TIMSS were `not really much below the international average' (and including in that `international average' many countries still in a developing stage).(17)

Table 2 presents a summary of average scores at age 14 in the six sub-categories of mathematical topics distinguished in that survey, comparing England with five Western European countries, with Switzerland separately, and with Germany, the United States and the grand average for all 41 participating countries. England is seen to lie below Western Europe for each of the six content-areas, including two areas--geometry and data representation, etc.--on which some had previously thought (on the basis of earlier international surveys) that England was ahead; the same is true when England is compared with the average of all 41 participating countries--including those still in early stages of development--with the sole exception of the content-area of `data representation, analysis and probability' (on which, however, certain reservations are in order).(18)

Table 2. Average percentage of questions answered correctly at age 14, by content-area, England compared with other countries

 Western Europe(a)

 England 5 coun- Switzerland Germany
 tries(b)
Fractions and number
 sense 54% 65% 67% 58%
Geometry 54 63 60 51
Algebra 49 55 53 48
Data representation,
 analysis and
 probability 66 71 72 64
Measurement 50 58 61 51
Proportionality 41 50 52 42

MATHEMATICS OVERALL 53 61 62 54

 Inter- No. of
 national questions
 USA average(c) in test

Fractions and number
 sense 59% 58% 51
Geometry 48 56 23
Algebra 51 52 27
Data representation,
 analysis and
 probability 65 62 21
Measurement 40 51 18
Proportionality 42 45 11

MATHEMATICS OVERALL 53 55 151

Source: Summarised from TIMSS, table 2.1, p. 41.

Notes:

(a) Excluding Germany and England.

(b) See table 1 (includes Switzerland).

(c) All 41 countries participating in the survey (including those in previous columns).

England's more serious weakness in arithmetic can be recognised in Table 2 (though, as explained, in a muffled form) in the scores for two content-areas `fractions and number sense' and `proportionality'; these are the content-areas from which the two basic questions were quoted above, namely, a subtraction sum with whole numbers of four digits (no decimals), and a simple proportionality sum involving whole numbers of not more than two digits (apart from zeros). The apparently wide range of questions provided by the TIMSS survey is undoubtedly valuable in providing a framework for our enquiries; but it may not be as helpful in guiding educational policy for a specific country, such as England, as would a narrower enquiry focused on relevant comparator countries combined with greater emphasis on diagnostically-relevant content-areas.

3. The best pupils

Whatever may previously have been thought as causing concern and requiring improvement in English mathematical schooling, there has usually been a bright star on the horizon: previous international comparisons have been consistent with the view that England's top pupils did as well as, and in some cases better than, the equivalent proportions of top pupils anywhere in the world. This was of obviously great scientific, technological and economic importance; and it reflected favourably on the schooling preparation of its top scientists and mathematicians and their subsequent excellence at university level. As recently as the 1991 survey by IAEP, the top 5th percentile of English pupils did slightly better than an average of the three Western European countries that participated in that survey (France, Italy, Switzerland).(19) The 1995 TIMSS survey provides no comfort of this sort; scores summarised in Table 3 (upper half of table) suggest that the top 5th percentile of pupils in England did not do as well as their counterparts in any of our five Western European comparator countries; the same held for the top 25th percentile, which extends to the broader group now entering our universities. This declining trend in English top pupils' attainments reinforces the worries expressed with increasing urgency by mathematics professors on the entry standards of university students to their departments.(20)

Table 3. Attainments of pupils aged 14 at top 5, top 25 per cent and median of the distribution of mathematics scores, England compared with other countries

 Top 5% Top 25% Median

England 665 570 501
Austria 693 608 537
Belgium 684 609 549
France 666 591 534
Netherlands 688 604 543
Switzerland 685 607 549

Average WE(5) 677 599 539
Germany 661 572 506
United States 653 563 494

Gap between English
and WE(5) scores(a) 12 29 43
Rise in scores between
ages 13 and 14:(b)
 England 26 30 31
WE(5) 49 43 36
Gap between England
and WE(5) measured
in equivalent no. of
years' learning
(at England's rate) 0.5 1.0 1.4
Less adjustment for
difference in
average ages(c) 0.2 0.7 1.1

Source: TIMSS Mathematics, appendix table E1.

Notes:

(a) Western Europe(5) minus England.

(b) From scores at relevant percentiles in table E1 minus table E2 (for WE, excluding Belgium, see n. 4 to text above).

(c) English pupils were aged 14.0, compared with 14.3 in WE(5) (see table 1, col. 1); the difference of 0.3 years has therefore been subtracted from the previous row.

In diagnosing the sources of Britain's schooling problems in this subject, it remains important to notice that any disadvantage of our top pupils in comparison with our five Western European comparator countries is nevertheless not as great as for median pupils (gaps of 12 and 29 points at the 5th and 25th percentiles respectively, compared with a 43 points gap for the median pupils as measured on the TIMSS scale which has an average of about 500). We can express these gaps in more readily comprehensible form in terms of an equivalent number of years of learning, derived from pupils' scores in successive classes at ages 13 and 14, as calculated in the lower half of Table 3: the top 5th percentile of English pupils were only about half a year behind their Continental equivalents, compared with a lag of about a year for the top 25th percentile, and nearly a year and a haws lag for the median pupil. Bearing in mind that English pupils at the date of the survey were about 03 years younger than pupils in these Continental countries (see Table 1), we might even say that there was a negligible gap at age 14 for the very top 5 per cent of pupils, growing to just over a year as we move towards median pupils.(21) Let us next examine the gap for English pupils who were of below average attainment.

4. How long is the tail of under-achievement?

Taking the survey's results for pupils in classes corresponding to our Year 9 at their published face value, it appears that the mathematics scores in Switzerland attained by the lowest 10th percentile of pupils--the bottom 2-3 pupils in a class--were attained by pupils in England only at the 21st percentile, or the bottom 5-6 pupils (a score of 434 on the scale used in that survey).(22) In other words, on that simple criterion it appears there were about twice as many low attainers in England as in Switzerland. We focus comparisons for this end of the attainment-range on Switzerland since, as will be seen in more detail below, participation rates in this survey by schools and pupils were particularly good for that country and the survey's results were thus less subject to worries about accuracy.

We can alternatively express the gap between these countries in terms of equivalent years of learning, as we did in the preceding section; the survey then tells us that in Year 9, Swiss pupils at the lowest 10th percentile were about 41 points ahead of English pupils at the lowest 10th percentile (433 points compared with 392 points), corresponding to a lag of about one and a half years of learning at the English rate at that percentile (there is a rise of about 25 points a year in England at that percentile). The lag at this lower extreme thus appears just a little greater than at the median.

But two adjustments need to be made to this simple account; the adjustments are a little complicated and, as we shall see, in their net outcome they largely offset each other (the hurried reader may therefore wish at this point to move to the next section). The adjustments arise from (a) the consequences of the Continental practice of class-repetition for the survey's intended coverage of low-attaining pupils; and (b) unintended lower participation in the survey by low-attaining schools and by low-attaining pupils. Improvements in the way the survey was carried out on this round permit us to examine these problems considerably more satisfactorily than previously; even so, as will be seen, certain obscurities remain.

Class repetition

At the end of each school-year in England pupils move up to their next higher class 'automatically', strictly according to age based on the twelve months' period defined for entry to obligatory schooling. On the Continent it is accepted as normal that there may be deferred entry to schooling by slow-developing children, and some subsequent class-repetition by low-attainers; only one or two pupils may be involved each year but, by the time pupils reach the ages of 13-14 (with which we are here concerned), something like a quarter of all pupils may well be a year older than their class's normal age-range. In looking at low-attainers in an English class in Year 9, and if we wish to compare pupils of the same calendar age, we should therefore be interested in low-attainers in a Continental class that is nominally a year younger, that is, in a Continental class corresponding to our Year 8. Fortunately, this latest IEA study provided data that can be used for this purpose. Of course, not all low-attaining Continental pupils in a class should be expected to be a year older, since it is a prime objective of Continental policies on late-entrants and repeaters to help low-attaining children catch up and then maintain progress with the rest of their class. Such a comparison--between our low-attaining Year 9 pupils with Continental low-attaining Year 8 pupils--is clearly worth carrying out, but it should probably be regarded as approaching an extreme assumption.

The details are as follows. In Year 9, pupils at the lowest 10th percentile in England attained a score of 392; in Switzerland, looking at pupils in a class in which they were normally a year younger, that score was attained by pupils at about the lowest 6th percentile. In other words, instead of there being twice as many low-attainers in England as in Switzerland, as appeared above on taking the published TIMSS results at their face value, there may be only two-thirds more after adjusting for age-differences resulting from class repetition and deferred entry.

We may compare, alternatively, the score at the lowest 10th percentile of English pupils in Year 9 (as said, 392 points) with the score at that percentile of Swiss pupils in the class a year lower (a score of 412); the difference between those scores (20 points) is equivalent to about one year's learning, at the rate of learning in England at that end of the attainment-range. It compares with a lag of one and a half years derived above without making an adjustment for class repetition.

Participation in the survey by low-attaining schools and pupils

That kind of adjustment for the effects of class repetition is, however, largely offset by the consequences of differences in countries' response rates to the surveys. The problem is that lower-attaining schools are more often busy grappling with a variety of social and other problems originating in their neighbourhoods, and are thus more reluctant to participate in such a survey; they may judge, for example, that their time is better occupied in teaching their children than in undertaking tests inadequately related to their work. Further, within schools which participate, weaker pupils may not participate as frequently, perhaps because there is a reluctance to answer tests which will display their failings, or because they are more frequently ill (without necessarily any encouragement by their teacher to stay away on the day of the tests).

These doubts affect the interpretation of many aspects of these surveys, but have a particular impact on low-attainers; the underlying facts on participation rates in the survey therefore need careful consideration here. At the level of the school, as it turned out, problems of participation seem not to have been overwhelmingly serious in England when compared to most other countries: 56 per cent of English schools randomly chosen (according to a stratified sampling scheme) agreed to participate, rising to 85 per cent including replacements. Of course, we cannot be quite sure that such replacements were in all respects equivalent, even if they had similar GCSE results and similar geographical characteristics.(23) But other Western European countries were not always more successful; for example, Dutch schools were particularly reluctant (only 63 per cent participated even after replacements were added). On the other hand, the co-operation of Swiss schools was exemplary--93 per cent co-operated at the initial stage, rising to 95 per cent including replacement schools.

More serious questions need to be mentioned in relation to the participation of pupils within participating schools. Pupils withdrawn or excluded from the survey because they had special educational needs, or were otherwise not suited to take the tests according to the international criteria laid down for the survey, totalled a mere 0.8 per cent of Swiss 14 year-olds.(24) The corresponding proportion of pupils withdrawn or excluded in English schools totalled 4.8 per cent. The additional 4 per cent of pupils withdrawn in England, it seems, can largely be attributed to the greater degree of integration of SEN pupils within normal classes in England (statistics for the Canton of Zurich indicate that 4.8 per cent of all 14 year-olds were in Special Classes or Special Schools in 1995, compared with 1.7 per cent in England).(25) We probably therefore need not worry too much about differences between countries in rates of withdrawal of pupils from the survey because of special educational needs.

The really important difference among countries relates to pupils who were simply absent from the tests: they varied from a forgivable 1.9 per cent in Switzerland (say, one pupil absent in every other class, on average), to a surprising 7.4 per cent in England (say, two pupils per class). How are we to take into account the additional 5.5 per cent of absentees in England? If all those additional English absentees were very low-attaining pupils then, broadly speaking, the score of the lowest 10th percentile of Swiss pupils participating in Year 8 (with a score of 412) should more properly be compared with the score recorded in England for roughly the lowest 5th percentile participating in Year 9 (a score of 361). These pupils in the two countries can be regarded as being at roughly the same true relative position in the attainment range and of approximately the same age; the gap between their TIMSS scores was 50 points, corresponding to over two years of learning in England at that end of the attainment-range. This compares with an estimated gap of only one year derived after allowing only for class-repetition (as in the previous sub-section).

If only half the additional English absentees were from that low end of the attainment range, then, of course, only about half that adjustment needs to be made; this would bring England's estimated lag back to the one and a half years derived initially from taking the TIMSS figures at their face value, as published. That, perhaps, represents a reasonable approximation to reality. In any event, we are bound to consider such heavy absences from school in England as a source of under-achievement for those pupils, and as leading to teaching difficulties for the class as a whole when they return.(26)

In short, this new survey tells us that while the mathematical attainments of average English pupils undoubtedly lag behind those of leading Western European countries at ages 13-14, very little of that lag arises amongst pupils at the top of the attainment-range: most of it arises at the middle and lower end, where the lag amounts to the equivalent of about one and a half years of schooling. In other words, there is still a `long tail of under-achievement' (rather than, as some have suggested, only a normal degree of under-achievement in distributions with much the same dispersion); it suggests that there is some structural malfunctioning of the English schooling system which is unlikely to be cured simply by providing more resources and distributing them much as at present. While those concerned with educational policy in England would therefore be justified in continuing to devote the bulk of their attention to measures to improve the attainments of average and below-average children, this does not contradict the need to consider also why attainments in mathematics of our top pupils are now no longer--as thought in earlier years--ahead of their Continental counterparts, and to develop appropriate policies.

5. Variability of educational attainments

The preceding sections have indicated that English pupils have a 'longer tail of under-achievement', and that high-attaining pupils in England are no longer as far ahead of their Western European counterparts as previously. Taking these two aspects together, a related question arises as to how the gap between low- and high-attaining pupils in England compares with other countries. Table 4 attempts an answer in terms of three measures: (a) the difference between the scores of pupils at the top and bottom 5th percentiles; (b) the same at the 25th percentiles (the interquartile range); and (c) the standard deviation.

Table 4. Variability of pupils' attainments in mathematics at age 14 (measured on the TIMSS scale)

 Range of 5th Inter-quartile Standard
 extreme range deviation
 percentiles

England 304 127 93
Austria 300 134 92
Belgium 284 124 89
France 251 107 76
Netherlands 291 127 89
Switzerland 284 122 88

Average WE (5) 270 117 83
Germany 293 124 90
United States 297 128 91

England as
 ratio of
 WE(5) 1.13 1.09 1.12

Source: Based on TIMSS, appendix tables E1, E3.

Note: All measures are on the TIMSS scale, with US average at age 14 set at 500 (see Table 1 above).

On a broad view--we shall mention exceptions in a moment--all three measures indicate that educational outcomes in England are more variable than in the five Western European countries chosen here as suitable comparators. The excess is clearest on the first measure based on the extreme 5th percentiles, and is similarly evident in the standard deviation. Austria is only slightly below England on both these measures. Focusing more at the middle-half of the distribution, as measured by the inter-quartile range, Austria is somewhat more variable than England; and the two other important countries shown in the table--Germany and the United States--are similar to England. France seems the least variable country on all three measures.

While English educational attainments are thus, in a broad sense, more variable than those of its European neighbours, the table suggests that the source of that excess is to be found in the extremes of the distribution--in the good attainments of its top pupils and the poorer attainment of its weakest pupils--rather than in the variability of the central half of its pupils. Such a conclusion is of some interest, but it must be qualified by the margin of uncertainty arising from suspected variable and lower response rates to this survey by weaker schools and weaker pupils.

6. Causative factors

Success in pupils' mathematical attainments depends on innumerable factors, some of which lie to a certain extent within the control of teachers and educational authorities; and some lie fairly clearly outside their control--such as the growth in single-parent families or the growth in television watching with their consequences for children's attention span and learning capabilities.(27) The latter factors have to be considered as more in the nature of wider challenges to the social structure, to the resolution of which schools and the educational system can contribute, if at all, only in the very long term.

Important factors that lie within the more immediate control of teachers and schools, it will probably be agreed, include the following--but surprisingly little information on these was provided by TIMSS (this selection is based largely on what struck the National Institute teams of teachers as particularly relevant to English schooling after visiting mathematics classes in Continental Europe).

Teaching time

From the previous 1991 survey by IAEP we learnt that there are considerable differences in average teaching time devoted to mathematics: for example, Switzerland--as noted above, a high-achieving country in this subject--devoted 250 minutes a week to mathematics for pupils at age 13, compared with 190 minutes in England. Many teachers are convinced that high achievement in mathematics needs not only sufficient time during the week, but the subject also has to be taught each day if it is to be consolidated efficiently in pupils' minds (especially important for low-attaining pupils): the IAEP survey told us that the majority of pupils in Switzerland (60 per cent) had mathematics lessons every day, whereas in England that held for only a small minority of pupils (17 per cent).(28) The 1995 IEA international study reported a distribution only of total weekly teaching time and only into very broad time-intervals (for example, 3.5 to 5 hours a week) and has added little to our knowledge on this important causative element.

Division of cohort into ability groups

Next, we need to know how countries vary in the way the age-cohort is divided into ability or attainment groupings. What goes on in this respect within each school at age 13 clearly depends very much on the extent to which pupils were divided between secondary schools according to their ability after leaving primary schooling: a 'comprehensive', full ability-range, system of secondary schooling such as England's provides each school with a greater range of attainments--and hence often with a greater incentive to form ability groupings--than, for example, the Dutch, German or Swiss selective systems which, starting from ages 10-12, have 3-5 types of secondary school according to pupils' academic achievements and interests. Within each such selective school the need for sub-division of pupils is obviously less, and usually no streaming or setting takes place within Continental schools. But no information on such fundamental organisational aspects of secondary schooling has been processed by TIMSS--though it has an obvious bearing on teaching styles.

Teaching styles

Consequently, it is difficult to interpret the information compiled by TIMSS on the important associated issue of whether the 'teacher teaches the whole class', whether 'pupils work in small groups with/without assistance from the teacher,' or the use made of another ten specified 'classroom organisation' approaches which were distinguished.(29) We are not even told what proportion of lesson-time is devoted to each of those approaches; instead, we are told only whether teachers 'used each organisational approach during most or every lesson': each approach was ticked, presumably, even if used for only a minimum sensible in each lesson. The real question, to put it in terms of a simple example, is not whether pupils sometimes work in small groups or are sometimes taught as a whole class: but what proportion of teaching time is devoted to these and other styles; and what is the rhythm of alternation. These questions are undoubtedly more complex; they were not adequately addressed by the survey.

Subject-specialist teachers

The role of generalist teachers, as against teachers who are mathematical specialists, was investigated in TIMSS with the following thought-provoking contrasts: England, with its low attainments, employed mainly specialist teachers of mathematics at secondary schools (69 per cent taught mathematics for more than three-quarters of their total teaching time, and 90 per cent for over half their total teaching time); whereas in high-attaining Switzerland, over half (52 per cent, according to TIMSS, 75 per cent according to IAEP) of teachers were closer to 'generalists', in that they devoted less than half their 'formally scheduled school time to teaching mathematics' (regrettably no further sub-division 'of less than half' was tabulated).(30) The underlying issue here relates to broader socio-pedagogic concerns: generalist or form-teachers understand better the overall relative strengths and weaknesses of their pupils in different subjects and any personal problems, and can motivate better particularly those pupils who are of average or below-average attainments; this has to be balanced against the more advanced knowledge that a specialist teacher can provide for high-attaining, academic, pupils. Considerations of this kind are much in the mind of practising teachers when they compare alternative schooling systems; but little at that level is to be gleaned from the TIMSS survey.

Textbooks

Anyone who visits English and Continental mathematics classes cannot avoid being struck by differences in the role of textbooks. English teachers tend to rely more on duplicated worksheets; if printed textbooks are used, teachers usually emphasise that no single textbook meets their needs, and that they need to draw on a variety of sources; if textbooks are used for some lessons, there are often not enough copies to go round, and pupils are required to share (usually two or three pupils per copy). English pupils thus do not usually have their own 'textbook for the year'; they are usually not permitted to take home any textbooks for their homework even if they are available (too many would soon 'get lost', and the 'school could not afford to replace them'). The contrast with Continental Europe (as with Japan) need not be spelt out here. What does the TIMSS survey tell on the extent of such obviously important international differences in pedagogic practice? For England, the answer is--Nothing; for other countries, there was a question only on whether teachers refer to `textbooks or curriculum guides' in deciding which topics to teach, and in deciding how to present them: the unsurprising answer for almost all countries is that most teachers refer to curriculum guides in deciding which topics to teach, and to textbooks in deciding how to present each topic.(31) But nothing emerges with clarity on the more important contrasts just mentioned amongst countries in pupils' use of textbooks--a matter of clear importance for English policy in relation to pupils' access to a systematic printed treatment of the subject, and one that is in step with their national curriculum requirements (and their changes!).

Calculators

The degree of usage of calculators in classrooms varied considerably between England and the Continent: usage was lowest in high-attaining Belgium and Switzerland (only 27-32 per cent of pupils used a calculator 'almost every day'), and was highest in England (83 per cent used a calculator).(32) Ownership of calculators is now fairly universal at secondary school ages, varying only in the narrow range of 97-100 per cent between pupils in England and in Western Europe.

Probably the more important pedagogical issue arises at primary (rather than secondary) school ages, where England needs to question whether more harm than good is done by calculators in schools; calculators were hardly ever (or never) used in mathematics classes by 85 and 98 per cent of 9 year-olds in The Netherlands and Austria respectively, but this was true for only 8 per cent of English pupils at that age (calculators were used at least once a week by 4 and I per cent of Dutch and Austrian pupils, compared with 53 per cent of English pupils).(33)

Computers were 'never or almost never' used in over 80 per cent of secondary mathematics classes in Belgium, France and Switzerland; in England that applied only to 53 per cent of classes, while in 46 per cent of English classes computers were used for 'some lessons'.(34) In short: in respect of the availability of 'high technology' mathematical equipment, and time spent in its actual usage, England is thus ahead of its neighbours: the sources of what problems there are in English mathematical attainments must clearly be sought elsewhere.

7. Summary and discussion

This latest round of international comparison of mathematical attainments of pupils aged 13-14 has been carried out in a more extensive and thorough way than previously. The findings add conviction to important inferences derived from previous rounds which are of wide policy relevance for English education; the new results also indicate one important change in England's relative standing. The following five points stand out.

1 Probably the most important finding is that English pupils' attainments in mathematics at these ages continue to be undistinguished by broad international standards, and provide no grounds for the expectation--fairly widespread a generation ago--that England's schooling is such as to set the country at the international forefront of scientific and technological progress. The attainments of English average pupils at 13-14 now lag--by the equivalent of about a year's schooling--behind such Western European countries as Austria, Belgium, France, The Netherlands and Switzerland; and are very much more behind 'Pacific rim' countries (Japan, Korea, Singapore, ...) in which schooling for a technological age-intended at first to match, and then to overtake, Western standards--became their over-riding national objective for the past generation.

Of international leading economies, only the achievements of the United States give grounds for pause and reflection: their pupils' mathematics scores at this age were similar to England's (perhaps even a shade lower). But only by a variety of expensive supplementary measures, including an average length of schooling for the equivalent of about two full-time years beyond that typical here, have they succeeded in maintaining their technological capabilities and economic performance at very high levels. It must also be said by way of reservation that the US economy is no longer advancing as rapidly as previously, and other countries are `catching up' with its level of productivity; and there are widely recognised US educational and social failings in the great variability of pupils' educational outcomes, the considerable proportion of underachieving and unemployable school-leavers ('dropouts'), and the widening disparity in income levels between low and high educational achievers. The US educational experience thus provides little by way of comfort or guidance to those looking for new directions in which English schooling should move.

2 Arithmetic is confirmed by TIMSS as continuing to be the branch of mathematics with the greatest shortfall in English pupils' attainments. The concern is not with arithmetic of a complex kind (for example, calculating the square-root of a number by a paper-and-pencil algorithm) but is at the most basic level: about half of English pupils at age 13 could not calculate correctly 6000 - 2369. Progress by pupils after that age at this basic arithmetical level--which is the proper province of the primary stage of schooling--is bound to be slow; and about a third of English pupils seem likely to reach the end of compulsory schooling unable to carry out such a calculation. England was behind almost all other forty participating countries in this respect (only in four participating countries were fewer pupils able to answer that subtraction question correctly).

3 English deficiencies in arithmetic are confirmed as arising at the primary stage of schooling by a parallel IEA survey of 8-9 year-olds. That same subtraction sum (6000 - 2369) could be answered correctly by some 91 per cent of Continental 9 year-olds, but only by 15 per cent of English pupils at that age. It is easy to discount failings in such basic arithmetic as being only a small part of the broader canvas of mathematics needed by the aspiring modern mathematical or technological specialist. But to do so would overlook the role of arithmetical competence as a pedagogical foundation stone for progress in other branches of mathematics and science. It would also undervalue the arithmetical needs of the ordinary citizen in his everyday life, and the employment needs of the broad cross-section of school-leavers who, without being technological specialists, need to work with increased precision in an increasingly automated and computerised world.

4 England's shortfall in mathematical attainments, when compared with the Western European countries mentioned above, are more severe among average and below-average pupils than at the upper end of the attainment range. The proportion of secondary school pupils with very low scores in England in these mathematics tests was about twice as great as in the Western European countries mentioned above; for example, scores attained by the lowest 10 per cent of Swiss 14 year-old pupils were attained by the lowest 20 per cent of English pupils. (In arriving at this conclusion it was possible, with the help of the greater detail provided in this latest round of tests, to take into account the Continental practice of class-repeating, as well as international differences in participation rates in the survey.) This larger proportion of low- and underachievers in England, with particularly great disabilities in basic arithmetic, leads to worries that the English schooling system is in some way malfunctioning, and is contributing to the creation of an economic and social under-class. Absenteeism by pupils was apparently high in England, but was inadequately investigated in this survey; it obviously contributed to under-achievement, and warrants further investigation.

5 The new point to emerge from this survey relates to England's top-attaining pupils. Previous international surveys were consistent with the view that-irrespective of low mathematical attainments by England's average and below-average pupils--the attainments of England's top pupils equalled, and perhaps even exceeded, the best of the corresponding top proportion in other countries: the present survey indicates that the mathematical attainments of the top 5 per cent of English pupils (those who might become mathematical specialists) and of the top 25 per cent (the broader group now eligible for university entrance in general) are below those of the Western European countries mentioned above. This finding is consistent with complaints made increasingly in recent years by English university professors of declining mathematical standards of students now entering mathematics, science and engineering departments.

Science tests were set by TIMSS at the same time as the mathematics tests; English pupils' attainments in science questions were altogether more creditable by international standards. But some important reservations need to be kept in mind. The science questions in these tests covered a very broad field, but they were not very deep; and they barely touched on the needs of everyday life, for example, in domestic electricity or health. The science attainments tested were not such as could be considered ideally relevant to the needs of the budding scientist, nor sufficiently applied to help the general citizen. Deficiencies in arithmetical competences, on the other hand, are of very wide relevance. Even routine applications of scientific knowledge usually require some numerical work; competence in science without mathematics is thus of limited value. Ail in all, it is far from clear that English pupils' apparently good results in the TIMSS science questions can be taken--as sometimes suggested--as `balancing', or in any serious way `offsetting', their poor attainments in mathematics questions.(35)

England has not been bereft in the past generation of major policy initiatives to raise schooling attainments. The secondary school system has been `comprehensivised' (virtual elimination of selective secondary schools for high-attaining pupils), there was a large-scale governmental inquiry specifically into mathematics teaching (the `Cockcroft report', Mathematics Counts, 1982), and a National Curriculum has been introduced specifying centrally-legislated attainment targets for primary and secondary schools. In that perspective the TIMSS report on mathematical attainments may be interpreted as being of a familiar disappointing sort--England `should be capable of doing better', `must try harder', `needs to re-consider basic study habits'. Little, if anything, seems to have been achieved to advance low-attaining pupils; while top-attaining pupils seem to have lost their international excellence.

Current thinking amongst English educationists who have visited mathematics classes in successful Western European countries tends to focus on pedagogic tools, school organisation and teaching methods: the availability to pupils, and the degree of usage by them, of systematic textbooks closely related to an agreed course of instruction; the availability of detailed teachers' manuals co-ordinated with those textbooks; the detailed sequencing and consolidation of sub-steps in teaching difficult mathematical topics; the use of teaching styles which, for a great part of each lesson, involve the whole class in extensive `interactive' teacher-pupil question-and-answer transactions; the ages and ability-groups for which generalist teachers are to be preferred to subject-specialists; the extent of class repetition; the grouping of pupils into classes according to attainment (how many parallel classes at each age typically span the whole attainment-range?); grouping of pupils within class according to their attainment (how many such groups are manageable by a teacher? for what fraction of each lesson are pupils subdivided in this way?). Whilst TIMSS devoted much questioning to underlying social and attitudinal factors that may affect attainments (pupils' home background, their liking for mathematics, television watching, teachers' perceptions about mathematics, etc.), very little is to be discovered in that report which is relevant to the pedagogic factors just mentioned.

In that sense, the TIMSS report may well be judged disappointing by English readers. Valuable as such a large international enquiry may be in its own way, it does not replace more tightly focused enquiries by practising teachers into the work of their counterparts in other countries--that is, countries with a similar balance of educational ideals and where schooling has yielded more satisfactory results--with observations focused on pedagogic details likely to be of help to them. An hour spent observing the practical working of a successful classroom may be worth a thousand hours of statistical analysis!

NOTES

(1) IEA = International Association for the Evaluation of Educational Achievement (established in 1959 as an international co-ordinating research centre; funded for the 1995 TIMSS centre by the US, with each participating country funding its own national study). TIMSS = Third International Mathematics and Science Study (previous studies in mathematics were carried out in 1964 and 1981; and in science in 1971 and 1984).

(2) W. Keys, S. Harris, C. Fernandes, Third International Mathematics and Science Study: First National Report Part 1 and Appendices (NFER, Slough, November 1996). A.E. Beaton et al, Mathematics Achievement in the Middle School Years: IEA's Third International Mathematics and Science Study (TIMSS International Study Center, Boston College, Chestnut Hill, MA, USA (November 1996); do for Science. Supplementary lists of questions in mathematics and science were released by the international centre in March 1997, but without the percentage of pupils giving correct answers in each country (requests for percentages correct, even for only selected questions, have gone unanswered). A parallel international enquiry into the attainments of 9 year-olds was undertaken by about half the countries involved in the 13-14 year-olds enquiry, including England but few European countries; certain of those findings are relevant in tracing the age at which England's deficiencies become evident (I.V.S. Mullis et al, Mathematics Achievement--the Primary School Years, TIMSS Center, June 1997).

(3) For convenience of the English reader, the English class-nomenclature is used throughout this Note rather than the `international nomenclature' used in the IEA report (which is one year younger; eg pupils are aged 14 in the middle of English Year 9, but are in `international class 8' in other countries where formal schooling begins one year later).

(4) TIMSS, Mathematics, p. 28, n. 8. Unfortunately the TIMSS report did not attach this caveat to the relevant tables, nor did it explicitly say in the text (aside from that text-footnote) that the very low calculated rise in average scores for Belgium between the two successive years is fundamentally misleading. Separate returns were published by TIMSS for the Flemish and French speaking parts of Belgium; for our purposes a simple average of the two parts is adequate, and has been quoted here throughout for that country.

(5) The contrast between the `modern mathematics' type of curriculum introduced in Nordrhein-Westfalen, in which `set language and arrows abound', and the more traditional emphasis of Baden-Wurttemberg on `the acquisition of competence in more limited areas', is outlined in G. Howson's National Curricula ill Mathematics (Mathematical Association, Leicester, 1991), esp. pp. 94, 97.

(6) J. Baumert, R. Lehmann, et al, TIMSS: Mathematik-Naturwissenschaftlicher Unterricht im internationalen Vergleich, Max-Planck-Institut, Berlin, 1997; esp. pp. 118 and 125, table D2 and figure D3. This gives scores for three groups of Lander according to their proportions of comprehensive schools; average scores for two particular Lander were subsequently reported in the general press (on the measures reported in table 1 below: Bavaria 534, Nordrhein-Westfalen 493). Limitations of space prevent further discussion here; a fuller (duplicated) comment by the present writer is available from the National Institute's Publications Department.

(7) Unfortunately for the reader, the Land Baden-Wurttemburg did not participate in TIMSS; there were also serious problems with the age-ranges of the classes covered (pupils were eight months older on average in Germany than in England).

(8) The sampling error of the rise in attainments between one year and the next can be expected to be quite small, since the basic approach of TIMSS was that the same schools provided the corresponding upper and lower grades in each country (TIMSS Mathematics, Appendix A, p. A-11); in other words, the sampling error of the average rise can be calculated on the well-known basis of `paired comparisons'. Unfortunately this was not noticed by the TIMSS statisticians, and they quoted (in their table 1.3) very high sampling errors, calculated as if the successive classes in each country were independent samples.

(9) Ibid, Appendix table D3. More strictly: this additional test was carried out only in the German-speaking parts of Switzerland; but since that part accounts for about two-thirds of the whole country, the comparison cannot be far out. The choice of Switzerland as an exemplar for reforms in mathematical education in the London Borough of Barking and Dagenham, carried out jointly with National Institute researchers, thus seems to be supported by these new results from TIMSS. (Incidentally, the description in TIMSS Appendix table D3 of Swiss classes aged 15.1 as being in the `eighth grade' must be regarded as an error for the `ninth grade': otherwise serious inconsistencies would occur with the main maths results reported for the seventh and eight grades.) The Flemish-speaking part of Belgium showed significantly higher scores than the French part (average scores for Year 9 of 565 and 525); it would be worth looking more closely at these results once the full national reports for that country become available (till then, given the sampling problems that arose in that country in respect of Year 8, as mentioned in n. 4 above, some caution is in order).

(10) I am indebted to Dr John Marks for these calculations based on the SATs results; see also his studies for the Social Market Foundation: Standards of English and Maths in Primary Schools for 1995; Social Market Foundation memorandum no. 24, 1996; Standards of Reading, Spelling and Maths for 7-year Olds in Primary Schools for 1995, Social Market Foundation memorandum no. 25, 1997.

(11) This was an `open' question (ie not multiple choice). For further comment see my Productivity, Education and Training (CUP, 1995), pp. 85 and 127, n. 22.

(12) See Helvia Bierhoff, Laying the foundations of numeracy: a comparison of primary school textbooks in Britain, Germany and Switzerland, Teaching Mathematics and its Applications, December 1996.

(13) See TIMSS, p. 62 for the subtraction question, and p. 58 for the `percent correct'. Scotland's total scores in mathematics were 463 and 498, at ages 13 and 14 respectively, compared with England's 476 and 506 (ibid, pp. 22, 26); Scotland was thus behind England by about a term's work. Scotland's current concern with its low attainments in basic arithmetic thus seem well borne out by the TIMSS results (Fourth Survey of Mathematics 1994, Education and Industry Department, Scottish Office, 1996), p. 7.

(14) TIMSS, pp. 94 and 97.

(15) TIMSS report on primary mathematics, op cit p. 68, table 3.3. Response rates to the primary survey were less satisfactory than for the secondary survey; where replacement schools were introduced to compensate for non-participation (28 per cent of the final sample in England, 52 per cent in the Netherlands, 23 per cent in Austria), it is not clear that any stratification by attainment-range of schools was attempted (eg in England using SATs results at Key Stages 1 and 2).

(16) Ibid, pp. 58, 62. We would need to examine closely the translated wording of that question (in all three Swiss languages) to understand why no more than 83 per cent of Swiss pupils answered it correctly.

(17) Whether English pupils' mathematical attainments have fallen (or perhaps risen?) since the previous IEA survey of 1981 is a matter of obvious interest and importance; a certain number of questions from that previous survey were therefore included in TIMSS (`the anchor items'). But so far there has been no comparative analysis of the two surveys, neither at the international level (by IEA) nor for England by (NFER), though this was done on the previous occasion (see, for example, the NFER study The Second International Mathematics Study in England and Wales, by M. Cresswell and J. Gubb, NFER-Nelson, 1987, pp. 56-66) Could a lack of funding for this important aspect reflect official worries as to what the results may show?

(18) One of the questions--on which English pupils did well--in the content-area of `data representation, etc.' required the representation in a pictograph of 55 students, in which one symbol (a head, shown as a circle in which there are two dots for eyes, and a curved line for a mouth) `represents ten students'. Should the respondent `round down' and give the answer as five symbols; or `round up' and provide six symbols? Or should he provide 5 1/2 symbols? According to the report (example 19, p. 83), the last is the only correct answer; this may well be in accord with conventions as taught in some countries, but it is not obvious to the present writer that other conventions are not equally acceptable as correct (it seems more of an exercise in what may be called `inverse palaeography'--an inference from digital arithmetic to primitive symbolism--which is surely of doubtful value to a pupil at that stage). While this content-area is at present an explicit part of the English National Curriculum, it should be noticed that English pupils did relatively well only when compared to the average of all 41 other countries; compared with our Western European neighbours, English pupils did not do as well-though the areas of `data representation, etc.' do not feature explicitly on their curricula but arise at this stage merely as common-sense applications of basic arithmetic.

(19) For an earlier comparison of mathematical attainments at ages 13-14 indicating that English Grammar School pupils were also considerably ahead of German Gymnasien pupils, see Appendix B on the 1964 IEA survey in my (1985) paper with K. Wagner (op cit), final para.

(20) A.G. Howson, chairman of a joint committee of leading mathematical institutions, Tackling the Mathematics Problem, London Mathematical Society, 1995).

(21) It would go beyond the proper ambit of this paper to do more than footnote a few factors frequently mentioned as inhibiting the progress of top pupils: reduced possibilities today for early entry of bright children to secondary school (at age 10, rather than 11 for the majority); substitution of the GCSE broader curriculum for the previous narrower but deeper O-level examination; too much time spent under National Curriculum requirements on `investigations' ...

(22) Scores for percentiles quoted in this section have been derived by graphical interpolation (using probability graph paper) from the tabulated 5th, 25th and 50th percentiles in tables E1 and E2 of TIMSS.

(23) For example, participating substitute schools but had GCSE results at ages 16+ in the same attainment bracket, but may have been stronger in attainments at the survey ages 13-14 than those schools who refused to participate. Published response rates by English schools' characteristics suggest that Metropolitan schools (including `inner city' schools) were slightly under-represented even after `replacement schools' were added (NFER, Appendix Volume p. 40). In addition, some 7 per cent of English schools were excluded because they were taking part in National Curriculum trials (ibid, p.39), and this accounts for the greater part of the special reservation attached by TIMSS to England's results (`only England exceeded the 10% limit' of exclusions; TIMSS Report, p. A 11); since those excluded schools were presumably chosen in a representative manner, that reservation is probably of no substantial significance (the critical reader may nevertheless wonder why they were not excluded from the sampling frame in the first place). The detailed international response rates are to be found in TIMSS, Appendix A, table A 4 for 14 year-olds (referred to in the text above); for 13 year-olds table A6 shows slightly lower response rates. In the IEA's primary school survey, 63 per cent of English schools responded on first approach, and an additional 25 per cent from a second replacement list; but, in contrast to arrangements for the secondary school survey, there was no provision for choosing replacement schools from the same attainment stratum (eg by relying on results from the SAT tests); the reliability of the primary survey is thus more questionable.

(24) TIMSS Mathematics, p. A 14. The proportions for 13 year-olds lead to much the same conclusions and, for simplicity of exposition, are not reproduced in the text above.

(25) Jahrbuch, op cit, calculated from pp. 66-82; Education Statistics for the United Kingdom, 1995 Edition (HMSO, 1996), table 12b.

(26) The problem posed by absentees in interpreting the results of the TIMSS tests was not adequately addressed in the international report. We must however note that absence rates above 10 per cent were recorded for Germany, Hungary and Scotland; 9 per cent for the United States; and even Japan recorded 5.5 per cent. It is not clear how absence rates as recorded for TIMSS tests compared with absence rates in the rest of the year. A more precise comparison of the length of the tail of under-achievement seems difficult in such circumstances (better sampling design, employing the familiar device of asking teachers to provide advance estimates of pupils' performance, would have permitted the calculation of response rates by attainment-stratum, and hence to less-biased estimates for the population).

(27) Questions were asked about the extent of pupils' television watching (TIMSS, p, 116), but are difficult to interpret. The survey shows that pupils who spend more time watching have lower scores in mathematics; but causation could be in the other direction--pupils who are badly taught, tend to become bored with the subject, and with homework on it, and fill in their evenings with more television watching. Many questions were asked--with equally inconclusive results--on pupils' attitudes to, and teachers' perceptions about, mathematics: 93 per cent of English pupils--with their nationally low attainments--agreed that they `usually do well in mathematics', while only 44 per cent of Japanese pupils--with their nationally high attainments--agreed that they `usually do well' (TIMSS p. 118). Does it follow that English teachers are better at spreading complacency amongst pupils? And does it follow that they are pedagogically counter-productive in doing so?

(28) A.E. Lapointe et al, Learning Mathematics, International Assessment of Educational Progress (IAEP), Princeton, NJ, 1992, p. 49.

(29) TIMSS, p. 154.

(30) TIMSS, p. 146; IAEP, p. 59. On the basis of direct observations in German-speaking Switzerland I suspect that the higher IAEP figure is correct. The employment of generalist teachers for middle and lower streams is normal in Switzerland (as the TIMSS report correctly noted); it would have been of some interest to be more precise, and to tabulate the contrasting use of generalist/specialist teachers according to the level of the mathematics attainment-group--but this was not done.

(31) IMSS, p. 157.

(32) Ibid, pp. 163-4, 166.

(33) TIMSS, Primary mathematics, op cit, table 5.14, p. 176 (based on teachers' reports; the associated table 5.16, based on pupils' reports, gives a broader and less helpful classification). The issue has attained further policy prominence (at the time of writing) with the commissioning by the new Labour Government of a fresh enquiry into the use of calculators at primary schools. A previous National Institute study (Bierhoff, op cit, pp. 152-3) emphasized the importance of training in mental agility in primary mathematics, and the negative role of calculators in that respect. A subsequent official (and anonymous!) Discussion Paper from SCAA defended the use of calculators; while discussing some of the considerations raised in the National Institute's study, the issue of mental agility was not given adequate attention (The Use of Calculators at Key Stages 1-3, Discussion Paper no. 9, SCAA, March 1997).

(34) Ibid, pp. 163,167-8. There seems to have been some ambiguity in the term `computers' in that survey, since an astonishing 89 per cent of English sample reported having `a computer in the home' (p. 163; exceeded only by Scotland's 90 per cent--in contrast to poor Switzerland's 66 per cent). But Britain's General Household Survey reported only 48 per cent of households with dependent children as owning a computer (CSO, Social Trends, HMSO, 1996, p. 119). It seems more than possible that computerised games--many of which, though intellectually challenging, have little to do with mathematical attainments as understood for the present purposes--were included by respondents to TIMSS on questions relating to the ownership of computers.

(35) The associated issues are clearly relevant to the proposed revision of the National Curriculum. Limitations of space prevent a fuller discussion here; a supplementary note is available in duplicated form from the National Institute's Publications Department.

(36) See, for example, R.G. Luxton and G. Last, `Under-achievement and Pedagogy: Experimental Reforms in the Teaching of Mathematics Using Continental Approaches in Schools in the London Borough of Barking and Dagenham', National Institute Discussion Paper no. 112, February 1997.

S.J. Prais, Correspondence should be addressed to the author at the National Institute of Economic and Social Research, 2 Dean Trench Street, London, SW1P 3HE. This Note has been prepared as part of the National Institute's research programme into international comparisons of educational attainment, carried out in association with inspectors and teachers in the London Borough of Barking and Dagenham. I am grateful to Roger Luxton and Graham Last (Principal and Senior Inspectors in that Borough), Professor Geoffrey Howson (University of Southampton), Dr John Marks (School Curriculum and Assessment Authority), Jason Tarsh (Department for Education and Employment) and to my colleagues at the National Institute for helpful discussion and comments on an earlier draft. For help in interpreting the results for Germany, I am particularly indebted to Professors Aurin (Freiburg) and W. Blum (Kassel) and to Helvia Bierhoff (previously researcher at the National Institute, now in Germany). The work was carried out with the financial support of the Gatsby Charitable Foundation. Responsibility for errors of fact and judgement remains with the author.