VOUCHERS AND TEST SCORES.

Peterson, Paul E.

What the numbers show

Students in private schools learn more and score higher on standardized tests than their counterparts in public schools. Some say this does not prove that private schools are better but only shows that children from more motivated families (who are willing and able to pay the tuition) attend private schools. As former Wisconsin state school superintendent Herbert Grover, an arch-critic of school choice, has argued, "Do private school children outperform children in public schools? It's hard to imagine that they wouldn't, given the initial advantages they enjoy from their parents."

To see whether students actually learn more as a result of attending a private school, my colleagues and I are currently evaluating a school- choice pilot program in New York City funded by the School Choice Scholarships Foundation (SCSF).

In February 1997 the SCSF offered public-school students from low-income families who were entering grades one through five a chance to win a $1,400 annual scholarship, good for at least three years, to help defray the cost of attending a private school, either religious or secular. Over 20,000 applications were received. Ninety percent of the applicants were either Latino or African American. Scholarships were awarded by means of a lottery. Some 1,200 SCSF scholarships were used to attend some 225 participating private schools. Students began school in the fall of 1997.

Because SCSF awarded scholarships by means of a lottery, it was possible to evaluate the pilot program using a scientific method regularly employed in medical research, the randomized field trial (RFT). In a medical RFT, one group is given a pill, the other a placebo. Individuals are assigned to one or another group by lot, or, in scientific parlance, at random. This method is preferred over all others, because the test and control groups, on average, can be assumed to be similar, save for the medical intervention under investigation. Positive results from RFTs are required in order to win approval of a medication from the Food and Drug Administration.

In my view, education innovations should be subjected to similar testing before being introduced on a wide scale. Unfortunately, this seldom happens, in part because public schools typically resist rigorous, independent evaluations, but also because the Department of Education, unlike the FDA, has not provided strong research leadership. Fortunately, SCSF was willing to permit a rigorous, independent evaluation of its pilot program, and my colleagues and I were able to obtain funds for the evaluation from a broad network of private foundations.

The lottery was held in mid-May 1997. The firm responsible for the evaluation, Mathematica Policy Research, administered the lottery in order to leave no doubt about its integrity; SCSF announced the winners.

To estimate the effects of attending a private school, the mathematics and reading components of the Iowa Test of Basic Skills were administered in the spring of 1997 to scholarship applicants. Each component of the test took approximately one hour to complete. Students participating in the evaluation were tested again in the spring of 1998. Both the scholarship students and students in the control group were tested in locations other than the school they were currently attending. To guarantee similar testing conditions, both for scholarship students and students in the control group, the tests were administered under the supervision of the evaluation team.

Each student's performance was given a national percentile ranking between one and one hundred. The national average is 50. The data indicate that these students are educationally disadvantaged: overall, average test scores were below the 30th percentile. Results were collected from approximately 85 percent of the participants in the evaluation, an unusually high response rate from a low-income, inner-city population.

Our evaluation focused on students entering grades two through five, because only from them were we able to obtain baseline test-score data. Baseline test-scores were unavailable for those entering first grade, because those children were still in kindergarten at the time of application.

After one year, the national percentile ranking of students attending private schools was, on average, two points higher in reading and mathematics than the ranking of the comparison group that remained in public schools. Differences were uneven in grades two and three, but choice students in grades four and five achieved substantially higher scores, six percentile points more in math, and four points more in reading.

Long-term Rewards

When reporting these effects of school choice, Education Week headlined them as "modest" the New York Times found them "slight." Whether or not these gains after one year are slight or substantial depends in part on what happens in later years. Nonetheless, there is reason to conclude that the effects of choice on the performance of students in their middle years is already sizable enough to merit careful consideration.

Scholars typically calculate effect sizes in standard deviations. As an indication of both the average score and the degree of variation from it, standard deviation allows us to compare results across different data sets. One can grasp its essential quality by keeping in mind that one standard deviation is approximately the current difference between the average test scores of blacks and whites nationwide. The effects of school choice on students in fourth and fifth grade are roughly one fifth of a standard deviation. If similar effects occur in subsequent years, these are large enough to bring the scores of minority students up to the levels currently attained by whites. This would be taking a large step toward achieving equal educational opportunity across ethnic groups, something most people would regard as a major accomplishment.

More to the point, these test scores are not a triviality, or a hobgoblin only of interest to academic researchers. Students who score higher on standardized tests are more likely to remain in school, more likely to achieve a college degree, more likely to remain married and avoid welfare dependency, and more likely to enjoy a higher family income. According to the best available estimates, a gain of one standard deviation in test scores will later in life increase that person's family income by over 20 percent. If students in the choice program in New York City simply hold the gains they have already achieved, one could expect their family income, on average, to be 4 percent higher than it otherwise would have been. Assuming a modest annual income of $30,000, that's an increase of $1,200 a year. If these estimates are reasonably accurate, the philanthropists in New York will realize an ample return on their charity dollar, once these students enter the labor force.

Another way to consider the effects of the SCSF program is to compare them to the results of a different intervention. Very few education innovations of interest have been subjected to a random field trial, but one. Class size reduction from 25 to 15 students has been rigorously evaluated by this method. It is worth comparing the results of a school choice field trial with the results from a class-size reduction experiment, because both innovations can be introduced rather straight-forwardly by legislative action. (Other reforms, such as requiring students to do more homework, are much more difficult to mandate by legislative fiat.)

The class size RFT was conducted in Tennessee, where students were randomly assigned to classes of different sizes. No incremental effects on student learning were observed for students after the first grade. Among first graders, effect sizes varied between .15 and .30 standard deviations. Fred Mosteller, one of those involved in the experiment, observed, "although effect sizes of the magnitude of 0.1, 0.2, or 0.3 may not seem to be impressive gains for a single individual, for a population they can be quite substantial."

Congress was apparently persuaded by such reasoning and by the results from the effect sizes observed in Tennessee. After extensive policy deliberations in which the Tennessee evaluation was frequently mentioned, in 1998 Congress enacted legislation appropriating $1.1 billion for the purpose of reducing the size of elementary school classes.

The effect sizes observed in our evaluation of the New York scholarship program in grades four and five do not differ materially from those observed in Tennessee in grade one. The effects among fourth and fifth graders of attendance at a private school were, on average, .23 and .18, not much different from the .15 to .3 effects observed in the first grade of the Tennessee Study-the only grade where incremental class size effects were detected. Following Mosteller's guidelines, these effect sizes, observed after just one year in the program, can be said to be "quite substantial."

From a cost-benefit perspective, school choice seems a better intervention than reductions in class size. To get effects of about .2 standard deviations, class sizes in the Tennessee study were reduced from approximately 25 to approximately 15 students. If such reductions were introduced as a school reform more generally, it would increase the size of both the teaching staff and classroom space by 40 percent. Per pupil costs could be expected to increase by approximately 20 percent (if it is assumed that classroom costs constitute about half the cost of public schooling). By comparison, the per pupil cost of school choice is minimal; the taxpayer may in fact enjoy some savings, if eventually competition among schools leads to more effective education at lower cost.

Moreover, the incremental benefits of class size reduction disappear after first-grade. If larger differences between the test scores of scholarship students and those in the control group appear in subsequent years in New York City, the benefits of school choice will clearly outstrip those obtained through large reductions in class size.

When we initially announced our findings, Sandra Feldman, president of the American Federation of Teachers, offered the interesting hypothesis that class size and school size probably accounts for the results that we observed. "I see it as a validation of the need for small class sizes, and for smaller schools that are orderly and disciplined," she said. To test this hypothesis, we conducted a statistical test in order to ascertain whether any of the following characteristics could account for the higher test scores in the private schools: (1) class size (2) school size (3) discipline problems (4) school-parent communications (5) school resources.

Data on these potential explanatory factors were available from information contained in the questionnaires administered to parents when the students were tested. Although parents reported that private schools were superior in all five respects, only discipline problems had a large and consistently positive effect on both the math and reading scores of the two older grades. Class size had no significant effect. Students in larger (not smaller) schools did slightly better in math (but not reading). Improved parental communications had a positive effect on math (but not reading) scores, and additional resources had a positive effect on reading (but not math) scores. Most importantly, none of these factors, nor all of them, reduced the size of the effects of receiving a scholarship to attend a private school in fourth and fifth grade.

These new pilot programs provide new opportunities to find out whether students learn more when families are given a choice of schools.

Perhaps the program's impact comes from the sheer fact of choice: the opportunity to better match older students with an appropriate school. But, more likely, it is some constellation of many factors that affect scores in ways not easily captured by a statistical model. In any case, the advantage of attending a private school is not readily reduced to any one or single set of factors.

As we have pointed out, the advantages of attending a private school in New York City are not clearly evident until a student enters fourth and fifth grade. This finding is consistent with other indications that in American education problems begin during the middle years of schooling. According to the National Assessment of Educational Progress (NAEP), students in fourth grade are performing at higher levels than their counterparts a generation ago. Gains over the past two decades have been particularly large for students from minority groups. But NAEP data also show that, after fourth grade, initial gains disappear. In fact, students nationwide learned less between fourth and eighth grade in the 1990s than they did in the '70s. The slippage seems even greater in high school. Similarly, international comparisons reveal that U.S. fourth-grade students keep up in science and math with most of their peers abroad (though not with the Japanese and Koreans). But by eighth grade U.S. students trail those in all other leading industrial nations, and by twelfth grade they fall to near the bottom of all participating countries. If the problems in American education develop in the middle years of schooling, perhaps it is at this point that the advantages that come with school choice are particularly evident.

Of course, the findings from New York City are simply first-year results. Our evaluation is scheduled to continue for two more years, and only time will tell whether the initial gains are maintained in the future. It remains to be seen whether school choice, if generalized to a larger population, will yield comparable gains. But it does seem time to begin larger-scale experiments.

An Historical Perspective

The unique quality of the SCSF pilot program can be appreciated by situating its evaluation within the long-running controversy over research on public and private schools. In the early 1980s two nationwide studies, one conducted by a team headed by sociologist James Coleman, the other conducted by John Chubb and Terry Moe, reported that high school students learned more in private than in public schools. School choice critics questioned the findings from both studies on the grounds that the students in private schools came from families more committed to their children's education.

Both studies had anticipated this argument by taking into account family background characteristics, such as education and income. But critics say that no amount of statistical tinkering can ever fully correct for the selection effect: families who pay to send their child to private school are almost certainly more involved in and concerned about their child's education, even after adjusting for demographic characteristics. Even the Coleman research team admitted, the "difference between parents, by its very nature, is not something on which students in public and private schools can be equated" in a statistical analysis.

Beginning in the late 1980s, a number of publicly and privately-funded school choice pilot programs began providing researchers with opportunities to consider the question anew. Educational outcome information is currently available from programs in San Antonio, Indianapolis, Milwaukee, and Cleveland. In the next few years, still more information will become available not only from New York City but also from other pilot programs that are getting underway in many other cities, including Washington, D.C., Dayton, and San Antonio.

These new pilot programs provide new opportunities to find out whether students learn more when families are given a choice of school. For one thing, differences in family background have been reduced, compared to the national surveys mentioned above, because most of these programs are limited to inner-city children from low-income families. More importantly, from a research perspective, these scholarships are often awarded by lottery whenever the number of applicants exceeds the number of scholarships available. Because a lottery is used to award the scholarships, these programs can be evaluated by means of an RFT.

Unfortunately, many of the school-choice pilot programs conducted thus far do not permit an RFT. Privately funded programs in Indianapolis and San Antonio admitted students on a first-come, first-serve basis. Such admission procedures have a fairness of their own, and they are easy to administer, but any findings from these programs may be contaminated by the selection effect. After all, those families who are quick, clever and well-connected enough to get a first-come, first-serve scholarship are likely to have other attributes that favorably affect their child's educational attainment. Nonetheless, test score results from these experiments are mainly positive. For example, the scores of students participating in the school choice program in San Antonio increased between 1991-92 and 1993-94, while those of the public-school comparison group fell. In Indianapolis, students in private schools did better than students in public schools, particularly in grades six through eight.

Inconclusive Cleveland

Much the same can be said for the disparate findings that have emerged from research on the state-funded pilot program that began in Cleveland in the fall of 1996. Although the state required that the scholarships be awarded by lot, various legal, political, and administrative problems made it impossible to gather data necessary to conduct an RFT. As a result, both the research team that I headed and other researchers were forced to rely upon less precise research techniques.

In 1997 my colleagues and I found that students attending the Hope schools, two newly established choice schools serving 25 percent of the students previously attending public schools, gained 9 national percentile rank points in math and 6 percentile points in reading. But because no control group was available for comparison purposes, we cannot be sure that a comparable group of students would not have achieved similar gains in Cleveland's public schools.

All in all the evidence that school choice enhances achievement of low- income students has now become quite substantial.

Another evaluation by Indiana University's School of Education found no programmatic effects on the test scores of 94 third grade choice students. The Indiana University evaluation suffers from a number of limitations:

1. The study analyzed only third-grade test scores; no information is available for students in kindergarten, first or second grades.

2. To control for student achievement prior to the beginning of the scholarship program, the evaluation used implausible second-grade scores collected by the Cleveland Public Schools before the beginning of the choice experiment when students were still in public school. These dubious second-grade scores tell us that students from central-city, low-income, largely one-parent families were performing in second grade, on average, at approximately the national average. Yet in an independently proctored test administered one year later, the same students scored, on average, 40 percentile points in reading. Clearly, the previous second-grade test scores were inflated.

3. The evaluation excluded Hope school students from the evaluation, despite the availability of comparable test-score data.

In the end, firm conclusions cannot be drawn from the studies of the scholarship program in Cleveland. In neither our research nor that of the Indiana evaluation team was it possible to compare similar groups of students by means of an RFT.

The state-funded program begun in Milwaukee in 1990 also required that scholarships be awarded by means of a lottery, if applicants exceeded places available. In this case, the lottery was successfully conducted; as a result, data are available from an RFT for the first four years after the program was started (school years 1990-91 to 1994-95). Unfortunately, no data are available after 1995.

The original evaluation of the Milwaukee choice program did not carefully analyze the data from the randomized field trial but instead compared students from low-income families with a cross section of public school students whose parents were motivated enough to return a mailed questionnaire. Although this research reported no systematic achievement effects of enrollment in a private school, its findings are problematic because the study compared choice students with public-school students enjoying much more advantaged families.

When these data were released to the general public, my colleagues and I analyzed the data from the RFT. Although the data collection was less complete in Milwaukee than in New York City, making the findings less definitive, they are nonetheless of interest. We found that enrollment in the program had about the same modest effects for all students (regardless of grade) during the first year of the program, just as was observed in New York City. But we also found that choice students scored much higher in years three and four. The differences in these years were as much as one quarter of a standard deviation in reading and one third of a standard deviation in mathematics. Once again, these gains are large enough that, if similar gains are made in the remaining years of education, they have the potential of bringing minority students up to the level currently achieved by white students.

That choice students did not demonstrate improved performance until the third and fourth years is quite consistent with a common-sense understanding of the educational process. Choice schools are not magic bullets that transform children overnight. It takes time to adjust to a new teaching and learning environment. The disruption of switching schools and adjusting to new routines and expectations may hinder improvement in test scores in the first year or two of being in a choice school. Educational benefits accumulate and multiply with the passage of time. As Indianapolis choice parent Barbara Lewis explains the process: "I must admit there was a period of transition, culture shock you might call it. He had to get used to the discipline and the homework. . . . But Alphonso began to learn about learning, to respect the kids around him and be respected, to learn about citizenship, discipline, and doing your lessons. . . . My son has blossomed into an honor roll student."

Note to Government: More Choice

School choice programs are too recent to provide information on their effects on college attendance, though the private school choice program in Milwaukee (PAVE) reports that 75 percent of those who have graduated from high school have gone on to college. More systematic information on the effects of attendance at a Catholic high school are contained in a recent University of Chicago analysis of the National Longitudinal Survey of Youth, conducted by the Department of Education, a survey of over 12,000 young people. Students from all racial and ethnic groups are more likely to go to college if they attend a Catholic school, but the effects are the greatest for urban minorities. The probability of graduating from college rises from 11 to 27 percent, if such a student attends a Catholic high school.

The University of Chicago study confirms results from two other analyses that show positive effects for low-income and minority students of attendance at Catholic schools on high school completion and college enrollment. As one researcher summarized one of these studies, it "indicates a substantial private school advantage in terms of completing high school and enrolling in college, both very important events in predicting future income and well-being. Moreover . . . the effects were most pronounced for students with achievement test scores in the bottom half of the distribution."

All in all the evidence that school choice enhances the achievement of low-income students has now become quite substantial. Although additional RFTs are desirable, the results from the first year of the New York City evaluation suggest that, at least for children in grades four and five, there are clear benefits for low-income minority students that come from attendance in private schools.

The results from New York tend to confirm findings from a wide variety of previous studies that used less definitive research methods. Only time will tell if the choice students in this program score much higher in later years as they did in Milwaukee.

If Congress regards the research evidence sufficient to justify the $1.1 billion federal intervention to reduce class size appropriated in 1998, then the evidence is equally sufficient to justify comparable state and federal expenditures on school-choice experiments.

Paul E. Peterson is the Henry Lee Shattuck Professor of Government and the director of the Program on Education Policy and Governance at Harvard University.