The economics of education.

Levitt, Steven D.

In recent years, I have written a number of papers related to the economics of education. This research agenda has three distinct strands. One set of papers analyzes the impact of school choice on student outcomes. A second line of research investigates teacher and administrator cheating on standardized tests, and explores how such behavior responds to the introduction of high-stakes testing. Third, I have examined Black-White test score differentials and the role that the educational system may play in contributing to those differences. I discuss these three sets of papers in turn.

The Impact of Public School Choice on Student Outcomes

In recent years, school choice has become an increasingly prominent feature of primary and secondary school education. With the passage of new federal legislation (No Child Left Behind), there is little doubt that the trend will continue. School choice comes in a variety of flavors. Vouchers and charter schools are two types of school choice which have received a great deal of both academic and media attention. A third type of school choice, open enrollment, is actually far more prevalent than either vouchers or charter schools. Under open enrollment, students within a public school district are able to attend schools other than their neighborhood school, including specially designated magnet schools. As of 1996, open enrollment was available in more than one in every seven school districts nationally, and in more than a third of large districts. Moreover, No Child Left Behind mandates that students in underperforming schools be provided the option to attend other schools in the district.

Along with co-authors Julie B. Cullen and Brian Jacob, I have written two papers that analyze the impact of open enrollment policies on student outcomes in the Chicago Public Schools (ChiPS). ChiPS represents an excellent laboratory for studying the impact of open enrollment. Chicago has been among the most aggressive cities in implementing this form of school choice, with more than half of the students in the system presently opting out of their neighborhood schools. Thus it may provide a window into what the future holds for other districts that are moving in the same direction. The Chicago data are also exceptionally rich, including not only detailed administrative records on attainment and test scores, but also attitudinal surveys administered periodically to students.

The first of these papers (1) starts with the observation that students who opt out of their local school to take advantage of open enrollment are 7.6 percentage points more likely to graduate from high school than peers who are observationally equivalent in eighth grade--off of a baseline graduation rate of 50 percent. This increment to graduation is the same order of magnitude as the gap between students at Catholic and non-Catholic schools in previous studies.

There are several competing explanations for why students who opt out of their assigned school outperform those who stay behind. Higher graduation rates among those who opt out may be the result of these students attending better schools or finding a school that better matches their preferences. In either of these cases, the increased graduation rates represent the true benefits of open enrollment. There are, however, scenarios in which the students who take advantage of school choice outperform students who do not, but the differences in outcomes do not actually reflect real benefits of open enrollment. Higher graduation rates among those who opt out may be spurious if those who opt out are better on unobserved dimensions (for example, student motivation, parental involvement). In other words, the students who opt out may have systematically done better than other students, even if they had not left their assigned schools. Also, it is possible that the graduation gap is attributable not to the students who opt out doing better, but rather to the students who remain behind doing worse, since they have less able and motivated peers.

Our results suggest that, with the exception of career academies (that is, vocational schools that focus on practical skills), the benefits of school choice to students who opt out are illusory. There are three primary pieces of evidence supporting this claim. First, in a survey administered in eighth grade that asks students a wide range of questions about their expectations for the future, past educational record, and parental involvement, the responses are strongly correlated with both the likelihood of graduation and with the decision to opt out. This suggests that students who opt out would be expected to do better, even if they had to remain in their local school. The second piece of evidence is that students who live in areas with many nearby schools on average should derive the greatest benefit from the availability, of school choice, because distance to nearby schools is a strong predictor of the likelihood that a student will opt out of the assigned school. Empirically, we find that easy access to a career academy is associated with substantial increases in graduation likelihood, but the same is not true for other types of schools, including high-achieving schools. Finally, when we compare student outcomes within a given school (in most schools in ChiPS some students are assigned and some opt in), we find that those opting in do the same as those assigned at career academies, but do much better at other schools. Since all students at a school experience similar peers and teacher quality, the fact that those opting in far outperform those assigned to the school reinforces the idea that those who opt in are systematically better than observationally similar students who make other schooling choices and would outperform them regardless, except at career academies.

Our second paper on this topic (2) exploits the fact that school choice causes desirable schools in ChiPS to be oversubscribed, and many of these schools use randomized lotteries to determine which students gain admission. We analyze data from 194 separate lotteries held to gain access to high school. One drawback of the data is that we only observe student outcomes if they enroll in ChiPS. To the extent that there is selective attrition, the inferences drawn from a simple comparison of outcomes of lottery winners and losers will be misleading. Relative to past studies (for example, the Milwaukee voucher experiment), however, attrition rates are low, with over 90 percent of the students remaining in ChiPS.

Empirically, we find that those students who win the lotteries attend what appear to be substantially better high schools--for example, schools with higher achievement levels and graduation rates and lower levels of poverty. Nonetheless, consistent with our first paper discussed earlier, we find little evidence that attending these sought-after programs provides any benefit on a wide variety of traditional achievement measures, including standardized test scores, attendance rates, course-taking patterns, credit accumulation, or grades. We do, however, find evidence that attendance at such schools may improve non-traditional outcome measures, such as self-reported enjoyment of school, availability of computers, expectations for college attendance, and arrest rates. This suggests that schools may be influencing children in a variety of ways not generally captured by test scores. To the extent that these non-traditional measures help to predict life outcomes such as college attendance, labor market attachment, wages, and criminal involvement, an exclusive focus on test scores will be misleading.

An important caveat to interpreting the results of both of these papers is that we are only able to evaluate how access to a particular school affects educational outcomes for a student, holding constant the existence of a school choice program. We cannot estimate the overall impact of introducing a system of school choice, which might induce changes in residential location choice or in overall school quality due to increased competition.

Teacher Cheating

High-stakes testing, like school choice, has become an increasingly prominent feature of the educational landscape. Every state in the country, except Iowa, currently administers state wide assessment tests to students in elementary and secondary school. Federal legislation requires states to test students annually in third through eighth grade and to judge the performance of schools based on student achievement scores.

The debate over high-stakes testing traditionally has pitted proponents arguing that such tests increase incentives for learning and hold schools accountable for their students' performance against opponents who argue that the emphasis on testing will lead teachers to substitute away

from teaching other skills or topics not directly tested on the exam. Along with Brian Jacob, I have written two papers that explore a very different concern regarding high-stakes testing--cheating on the part of teachers and administrators. As incentives for high test scores increase, unscrupulous teachers may be more likely to engage in a range of illicit activities, such as changing student responses on answer sheets, or filling in the blanks when a student falls to complete a section. Our work in this area represents the first systematic attempt to identify empirically the overall prevalence of teacher cheating and to analyze the factors that predict cheating.

To address these questions, we once again turn to data from the Chicago Public Schools, for which we have the question-by-question answers given by every student in grades 3-7 taking the Iowa Test of Basic Skills (ITBS) over an eight year period. In the first paper, (3) we develop and test an algorithm for detecting cheating. Our approach uses two types of cheating indicators: unexpected test score fluctuations and unusual patterns of answers for students within a classroom. Teacher cheating increases the likelihood that students in a classroom will experience large, unexpected increases in test scores one year, followed by very small test score gains (or even declines) the following year. Teacher cheating, especially if done in an unsophisticated manner, is also likely to leave tell-tale signs in the form of blocks of identical answers, unusual patterns of correlations across student answers within the classroom, or unusual response patterns within a student's exam (for example, a student who answers a number of very difficult questions correctly while missing many simple questions).

Empirically, we find evidence of cheating in approximately 4 to 5 percent of the classes in our sample. For two reasons, this estimate is likely to be a lower bound on the true incidence of cheating. First, we focus only on the most egregious type of cheating, where teachers systematically alter student test forms. There are other more subtle ways in which teachers can cheat, such as providing extra time to students, that our algorithm is unlikely to detect. Second, even when test forms are altered, our approach is only partially successful in detecting illicit behavior. We then demonstrate that the prevalence of cheating responds to relatively minor changes in teacher incentives. The importance of standardized tests in the ChiPS increased substantially with a change in leadership in 1996. Schools that scored low on reading tests were placed on probation and faced the threat of reconstitution. Following the introduction of this policy, the prevalence of cheating rose sharply in classrooms with large numbers of low-achieving students. In contrast, schools with average or higher-achieving students, which were at low risk for probation, showed no increase in cheating.

Our second paper on this topic (4) reports on the results of an unusual policy implementation of our cheating detection tools. We were invited by ChiPS to design and implement auditing and retesting procedures implementing our methods. Using that cheating detection algorithm, we selected roughly 120 classrooms to be retested on the Spring 2002 ITBS. The classrooms retested include not only cases suspected of cheating, but also classrooms that had achieved large gains but were not suspected of cheating, as well as a randomly selected control group. As a consequence, the implementation also allowed a prospective test of the validity of the tools we developed in our first paper on the subject.

The results of the retesting provided strong support for the effectiveness of the cheating detection algorithm. Classrooms suspected of cheating experienced large declines in test scores (on average about one grade equivalent, although in some cases the fall in mean classroom test scores was over three grade equivalents) when retested under controlled conditions. In contrast, classrooms not suspected of cheating a priori maintained virtually all of their gains on the retest. As a consequence of these audits and subsequent investigations, disciplinary action was brought against a substantial number of teachers, test administrators, and principals.

Black-White Test Score Gaps Early in Life and the Contribution of Schools

The Black-White test score gap is a robust empirical regularity. A simple comparison of mean test scores typically finds Black students scoring roughly one standard deviation below White students on standardized tests. Even after controlling for a wide range of covariates including family structure, socioeconomic status, measures of school quality, and neighborhood characteristics, a substantial racial gap in test scores persists.

In a paper joint with Roland Fryer, (5) I revisit this topic with a newly collected data set, the Early Childhood Longitudinal Study Kindergarten Cohort (ECLS-K). The survey covers a sample of more than 20,000 children entering kindergarten in the fall of 1998. The original sample of students has subsequently been re-interviewed in the spring of kindergarten and first grade.

The results we obtain using these new data are informative and in some cases quite surprising. As in previous datasets, we observe substantial racial differences in test scores in the raw data: Black kindergartners score on average .64 standard deviations worse than Whites. In stark contrast to earlier studies (including those looking at kindergartners), however, after controlling for a small number of other observable characteristics (children's age, child's birth weight, a socio-economic status measure, WIC participation, mother's age at first birth, and number of children's books in the home), we essentially eliminate the Black-White test score gap in math and reading for students entering kindergarten. While there are numerous possible explanations for why our results differ so sharply from earlier research, we conclude that real gains by recent cohorts of Blacks are likely to he an important part of the explanation.

Despite the fact that we see no difference in initial test scores for observationally equivalent Black and White children when they enter kindergarten, their paths diverge once they are in school. Between the beginning of kindergarten and the end of first grade, Black students lose .20 standard deviations (approximately .10 standard deviation each year) relative to White students with similar characteristics. The leading explanation for the worse trajectory of Black students in our sample Is that they attend lower quality schools. When we compare the change in test scores over time for Blacks and Whites attending the same school, Black students lose only a third as much ground as they do relative to Whites in the overall sample. This result suggests that differences in quality across schools attended by Whites and Blacks is likely to be an important part of the story. Interestingly, along "traditional" dimensions of school quality (class size, teacher education, computer-to-student ratio, and so on), Blacks and Whites attend schools that are similar. On a wide range of "non standard" school inputs (for example, gang problems in school, percent of students on free lunch, amount of loitering in front of school by non-students, amount of litter around the school, whether or not students need hall passes, and PTA funding), Blacks do appear to be attending much worse schools. Other explanations for the divergence in Black White test scores, such as a greater "summer setback" for Blacks when school is not in session, or discrimination by teachers against Blacks, find no support in our data.

(1) J. B. Cullen, B. Jacob, and S. D. Levitt, "The Impact of School Choice on Student Outcomes: An Analysis of the Chicago Public Schools," NBER Working Paper No. 7888, September 2000, forthcoming in Journal of Public Economics.

(2) J. B. Cullen, B. Jamb, and S. D. Levitt, "The Effect of School Choice on Student Outcomes: Evidence from Randomized Lotteries," forthcoming as an NBER Working Paper.

(3) B. Jacob and S. D. Levitt, "Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating," NBER Working Paper No. 9413, January 2003, and Quarterly Journal of Economics, 117 (August 2003), pp. 843-77.

(4) B. Jacob and S. D. Levitt, "Catching Cheating Teachers: The Results of an Unusual Experiment in Implementing Theory," NBER Working Paper No. 9414, January, 2003, and Brookings-Wharton Papers on Urban Affairs, 2003.

(5) R. Fryer and S. D. Levitt, "Understanding the Black-While Test Score Gap in the First Two Years of School," NBER Working Paper No. 8975, June 2002, forthcoming in Review of Economics and Statistics.

Steven D. Levitt, Levitt is a Research Associate in the NBER's Programs on Public Economics, law and Economics, Children, and Education. He is also a Professor of Economics at the University of Chicago.