New York City Charter Schools: how well are they teaching their students?

Hoxby, Caroline M. ; Murarka, Sonali

The 60 charter schools operating in New York City have provided a unique opportunity for the New York City Charter Schools Evaluation Project, of which we are a part, to con-duct a randomized field trial of the impact of charter schools on student achievement. The study reported here thus differs from virtually all other published research on charter schools in its reliance on experimental methods to determine the schools' effectiveness. In particular, we take advantage of the lottery-based admissions process for charter schools to compare the academic performance of two groups of students: those who wanted to attend a charter school and were randomly admitted and those who wanted to attend but were not admitted and remained in traditional public schools. In this article, we present findings from the first year of what will be a multiyear study.

We address two main questions about charter schools in the city. First, who enrolls in New York City's charter schools? And, second, how well are the schools educating students? What we found is that, compared with other students in the traditional public schools, charter school applicants are more likely to be black and poor but are otherwise fairly similar. We also found that charter school students benefit academically from their charter school education. Charter school students in grades 3 through 8 perform better than we would expect, based on the performance of comparable students in traditional public schools, on both the math and reading portions of New York's statewide achievement tests. There is not yet a sufficient number of charter school students in grades 9 through 12 for us to report achievement effects for this group.

Data Collection

Forty-seven charter schools were operating in New York City in the 2005-06 school year, the most recent for which we have test-score results, and all but five are included in the analysis presented here. Two schools, Manhattan Charter School and South Bronx Charter School for International Cultures and the Arts, are participating in our ongoing study but are not included in the analysis because they do not yet have any students in test-taking grades. One school, Read-Net Bronx Charter School, was in the process of closing in 2005-06. The absence of ReadNet Bronx from our evaluation is likely to have only a small impact on our assessment of student achievement because the school had only two years of test-taking students before it closed. The New York Center for Autism Charter School is not included in the study because it serves a very special population and is not compatible with many elements of the study. The United Federation of Teachers Elementary Charter School has declined to participate in the study so far, but it does not yet have any students in test-taking grades.

[FIGURE 1 OMITTED]

Charter schools must advertise their availability to all students eligible to attend public schools and are not allowed to select their students from among applicants. Instead, if a charter school in New York receives more applicants than it has places, it must enroll students based on a random lottery. Each spring, charter schools that are oversubscribed hold admissions lotteries.

Our study data are collected as follows: First, the information from each charter school application is sent to the New York City Department of Education for inclusion in its administrative database. This database contains entries for all students who attend New York City's traditional public schools and for all students who attend New York City's charter schools. A contractor for the department uses the maximum amount of information possible--for example, the student's name, birth date, and Social Security number, if available--to match each applicant to a corresponding existing entry in the department's database. The contractor then extracts information on each student's demographic characteristics, enrollment, test scores, and certification for and participation in various programs such as free and reduced-price lunch, special education, and English-language services. This information is gathered from both the years before and the years after the application to a charter school and sent to us with an encrypted student identification number.

We first obtained application data on the lottery conducted in the spring of 2005 for the 2005-06 school year, and we requested application data from earlier years as well. Not all schools had archived this information or had requested all of the elements that would prove helpful in matching up their applicants. The 2005-06 application data therefore have the most complete coverage of schools and the most information on which to match. In order to be as representative as possible, the analysis of the characteristics of charter school applicants described below is based on the data from that year. In our achievement analysis, however, we use data from all lotteries for which we have application data.

The Applicants

Who applies to New York City's charter schools? In answering this question, it is important to recognize that the charter schools are located in neighborhoods that are substantially poorer than and almost twice as black as the average New York City neighborhood. Charter school neighborhoods contain only one-third as many whites and Asians as the average New York City neighborhood. In fact, it is no exaggeration to say that if the charter schools draw from their neighborhoods, they will draw students who are 90 to 95 percent black or Hispanic. The charter schools are thus in a situation that people some-times find confusing. Normally, if we say that a traditional public school is "more black" or "more Hispanic," we mean to imply that the school has fewer white students. However, for New York City's charter schools, "more black" or "more Hispanic" cannot imply "less white" because there are hardly any whites (or Asians) to be displaced. Instead, when we say a New York City charter school is "more black" than surrounding schools, it is automatically "less Hispanic" (and vice versa). Any school that disproportionately serves black students will disproportionately not serve Hispanic students. These are not two independent comments: they are the same comment!

As one might predict based on their neighborhoods, applicants to New York City's charter schools are twice as likely to be black (64 percent versus 32 percent) and much less likely to be white or Asian (7 percent versus 28 percent) than the average public school student in New York City. Because charter school students are disproportionately likely to be black, they are somewhat less likely to be Hispanic (27 per-cent versus 39 percent). About half of charter school applicants are female, just like students in the traditional public schools (see Figure 2).

[FIGURE 2 OMITTED]

There is no simple explanation for the disproportionate appeal of charter schools to blacks. While a couple of character schools--Harriet Tubman and Sisulu-Walker--are named after a black person, most of the charter schools, not a few, disproportionately draw black students. Nor does the explanation seem to involve strong language barriers for Hispanics. Traditional public schools and charter schools located in areas with significant Hispanic populations provide the same level of Spanish-language translation for school materials. In both sets of schools, key materials, such as applications, school calendars, and school descriptions are usually available in Spanish. A more complex story is needed. For instance, black parents may feel more comfortable "disagreeing" with their regular school assignment than Hispanic parents do, particularly if the parents in question are recent immigrants.

A common proxy for poverty is a student being certified to receive a free or reduced-price lunch. (To get certified, a student's household income must be less than 185 percent of the federal poverty line.) Using this proxy, we find that the applicants to charter schools are much more likely to be poor than is the average New York City student (93 percent versus 74 percent).

Unfortunately, charter schools and regular public schools have some information recorded differently in the New York City database, and these differences cause charter schools' numbers of special education and English language learner students to be understated. Nevertheless, the data that we have suggest that, at the time they applied, 11.1 percent of charter school applicants were participating in special education. This is about the same percentage as in the New York City schools overall (12.5 percent). The data we have also suggest that, at the time they applied, 4.2 percent of charter school applicants were classified as English language learners, while 13.6 percent of New York City's students were classified as such. Because of our concerns about the differences in the recording of English proficiency status, we cannot draw the conclusion that charter schools appeal disproportionately to students who are proficient in English. But the fact that charter schools appeal disproportionately to black students is probably reflected in applicants being more likely to be English speakers.

We do not have good data that would help answer the question of whether charter schools disproportionately draw high or low achievers. Because most students enter charter schools before the 3rd grade when state-mandated testing begins, only 36 percent of applicants in our study have prior test scores on record and this group is not representative of all applicants.

Student Achievement

The basic strategy we use to evaluate the effect of charter schools on student achievement is to compare students who are awarded a seat in a charter school through a lottery with students who enter the lottery but are not awarded a seat. About 91 percent of all charter school applicants participated in lotteries. The random assignment to the two separate groups of students who are otherwise similar--in their measured characteristics and the fact that they expressed a desire to attend a charter school--enables us to isolate the impact of attending a charter school.

We first wanted to confirm that the two groups contained similar students. As expected, when we compared students who were awarded a seat in a charter school to those who were not, we found no statistically significant differences on any of the demographic or predetermined program eligibility characteristics we could measure.

We use common statistical procedures to estimate the effect on math and reading test scores of each additional year of actual attendance at a charter school. Our results therefore reflect the performance of students who, if offered a seat in a charter school, choose to enroll--that is, those who comply with the experimental treatment. In some applications, having an estimate of a program effect that is valid only for compliers is problematic, because it would be useful to know what would happen if the program were expanded to other populations. In the case of charter schools, however, an estimate of their effect on students who enroll is exactly what we want, as the basic idea behind charter school reform is that only students who want to should attend them. Our present approach also assumes that each year of charter schooling has the same effect on student achievement. When we investigated whether each year of attendance at a charter school had a different effect, we found no evidence to support the idea of different effects in different years. However, we plan to return to the question in subsequent analyses when we will have more variation in the number of years students attend charter schools.

We use test-score data from the years 2000-01 to 2005-06 from the 36 charter schools that enroll students in grades 3 through 12. However, because the number of students in grades 9 through 12 is too small to produce statistically significant results at this time, our discussion will focus on the results for the 32 schools that enrolled 3rd through 8th graders in the relevant years. For them, the number of test-score observations included in the analysis ranges from almost 7,800 in grade 5 to 3,000 in grade 8.

We first present our results in the way most often used by researchers: standard scores. These scores, which are generated by dividing a scale score by its standard deviation, are helpful because they allow researchers to compare the effects of charter schools to the effects of other interventions, like class-size reductions. Our results indicate that, on average, New York City's charter schools raise their 3rd through 8th graders' math achievement by 0.09 of a standard score and reading achievement by 0.04 of a standard score, compared with what would have happened had they remained in traditional public schools (see Figure 3). We find no evidence that the improvement in achievement differs between boys and girls or between blacks and Hispanics.

To put these results in context, consider the Tennessee STAR Experiment, which produced some of the literature's highest estimated effects for class-size reduction. The Tennessee experiment suggested that a 10 percent reduction in class size in grades K-3 raised students' standard scores by 0.06. Furthermore, this was a one-time effect: even if students stayed in smaller classes for multiple years, their achievement rose only once, by 0.06. In contrast, the average charter school student improved by 0.09 in math and 0.04 in reading for each year of charter school attendance.

Another way to present the results is in terms of New York State's performance levels. In 2005-06, depending on the grade, a student's math scale score had to rise by an average of 32 points to go from the top of the Performance Level 1 range ("failing" or not meeting learning standards) to the bottom of the Performance Level 3 range ("proficient" or meeting learning standards). The equivalent required rise in a student's reading score was 44 points.

We estimate that, depending on the grade, students' math scale scores rise by 3.75 to 3.98 points and their reading scale scores rise by 1.53 to 1.61 points for every year they spend in charter schools. Again, these improvements are measured relative to what would have happened to the same students in traditional public schools. Another way to think about these gains is to understand that, for every year they spend in a charter school, students make up 12 percent of the distance from failing to proficient in math. They make up 3.5 percent of the distance from failing to proficient in reading.

There are several possible explanations for the effects of charter schools being larger in math than in reading. The most likely explanation, we believe, is that schools largely control math education, but that both families and schools exert strong influence over reading skills. If, for instance, the families of students who were and were not awarded a seat through a lottery had the same effect on reading and families controlled half the gains in reading, then the difference between the estimated math and reading effects would be fully explained.

Keep in mind, these annual gains are relative to whatever gains the students would have been expected to make in the traditional public schools had they not been awarded a seat through the lottery. Because most of the students in our study have been attending a charter school for between one and three years and no student has attended for more than six years, we are uncomfortable extrapolating our finding beyond four years of enrollment in a charter school.

We also estimated a separate effect on achievement for each of the 32 charter schools with students in grades 3 through 8. The results for about one-third of these schools are very imprecise, usually because they had very few students in test-taking grades during the analysis years. Based on the remaining schools for which we have reasonably precise estimates, however, we found a good deal of variation in achievement effects. About 19 percent of charter school students attend a school that is estimated to have a positive effect on math that is very large: greater than 0.3 of a standard score per year. Another 56 percent attend a school that is estimated to have a positive effect that is large: between 0.1 and 0.3 of a standard score. 18 percent attend a school with a positive but small to moderate effect. Only 6 percent attend a school that is estimated to have a negative effect on math, and these estimated effects are all small. The effects on reading are similarly distributed across a range, with 80 percent being positive and only 8 percent being negative.

School Policies

The variation in achievement effects among charter schools raises the question of whether one can identify specific policies that are associated with charter school success (see side-bar, page 60). To provide hints at possible answers, we conducted some preliminary analysis on the question using the math and reading results from the 32 schools that enrolled elementary and middle school students.

We want to be clear that our analysis cannot establish definitively whether the policies of charter schools cause changes in student achievement. We can describe only associations between policies and achievement effects, and the distinction between association and causation is very important in practice in the charter school context. Charter schools may adopt policies for reasons that we do not observe and it may be that it is these unobserved reasons that actually affect achievement. For instance, suppose that charismatic school leaders were a key cause of positive achievement effects, and suppose that charismatic leaders just happened to like long school years. We cannot measure charisma, but we can measure the length of the school year. Therefore, we might find an association between a long school year and positive achievement effects, even if the charisma, and not the long school year, caused higher achievement. A school that lengthened its school year would be disappointed in the results, not realizing that what it had really needed to do was to hire a charismatic leader.

That caution given, there are a few clear and interesting associations to be noted. We find no relationship between how long a charter school has been in operation and student achievement after controlling for school policies. However, if we do not control for school policies and look at the simple correlation between a charter school's years in operation and student achievement, we find that older schools have more positive achievement effects. The fact that this correlation disappears when we include such policies in our analysis suggests that the reason older schools have more positive achievement effects is that they adopt more effective policies.

A long school year is associated with positive achievement effects, and we estimate that schools with years that are 10 days longer are associated with average student achievement that is 0.2 standard deviations greater. This is a large effect, and a 10-day difference among school calendars is quite common. In fact, 12 days is the standard deviation in the length of the school year among charter schools. We should note, however, that a long school year tends to go part and parcel with several other policies, such as a longer school day and Saturday school, and this should make us cautious about assigning too much importance to a longer school year in and of itself. A more conservative conclusion would be to think of the package of the three policies having a positive association with student achievement.

We also find that class size, optional afterschool programs, and most math and reading curricula seem to have no relationship to student achievement. Everyday Math and Open Court reading curricula did have negative and statistically significant associations with achievement effects. We discourage readers from interpreting these as causal effects, however, since an equally plausible interpretation is that these are curricula that schools adopt when their students are struggling.

Conclusion

In sum, in the largest lottery-based evaluation of charter schools to date, we find that charter schools in New York City are having positive effects on the academic progress of the students who attend them. These effects are largest in charter schools that have extended the length of the school year, though we cannot establish definitively that this is the reason for their exceptional performance. We also find that the students applying to charter schools in New York City are more likely to be black and eligible for a free or reduced-price lunch program than students in the public schools in the district.

While it is reasonable to extrapolate the findings to other urban students who are similar to New York City applicants, we would argue against these results being applied to students who differ substantially from applicants to the charter schools. In particular, the results should not be applied to students who are substantially more advantaged or to students who would not be interested in applying to the types of charter schools available in New York City, even if they were conveniently located in the students' area.

That said, our results provide a strong basis for recommending the continued expansion of charter schooling in the Big Apple and in other large cities with similar student populations.

Charters Score (Figure 3)

New York City charter school students in grades 3-8 scored
substantially higher on state reading and math tests than would
have been expected had they remained in traditional district schools.

Impact of attending a charter school ...
 Math Reading
...for one year .09 .04
...for 4 year .36 .16

SOURCE: Authors
Note: Table made from bar graph.

[FIGURE 3 OMITTED]

RELATED ARTICLE: New York City Charter Basics

New York City has three charter school authorizers. of the schools covered in this report, the State University of New York authorized 20, the chancellor of the New York City schools authorized 19, and the New York State Board of Regents authorized 3. Three types of organizations operate charter schools in New York City: nonprofit community-grown organizations (CGOs), nonprofit charter management organizations (CMOs), and for-profit education management organizations (EMOs). CMOs and EMOs are formal organizations that exist to manage charter schools, and they function somewhat like firms that have a strong brand and that establish fairly independent branches or franchises (see "Brand-Name Charters," features, page 28). COMs EMOs typically make overarching curricular and policy decisions, conduct back-office activities, and provide something of a career ladder for teachers and administrators within their network of schools. The CMO with the most schools in New York City in 2005-06 was the KIPP Foundation, and the EMO with the most schools was Victory Schools. CGO schools may be founded by a group of parents, a group of teachers, a community organization that provides local social services, one or more philanthropists, or the teachers union. More often than not, the founding group combines people from a few of the groups listed above.

Fifty-six percent of the charter school students covered by this report attend 23 schools operated by CGOs; 19 percent attend 12 schools that are affiliated with CMOs; and 25 percent attend 7 schools run by EMOs. As these percentages suggest, the average school operated by an EMO has considerably larger enrollment than the average school operated by a CGO or a CMO.

Missions and Policies

Every charter school describes itself in a carefully crafted mission statement that sets out its vision, educational philosophy, and focus. Based on these statements, we can categorize the schools roughly into five groups: those that have a child-centered or pro-gressive educational philosophy and typically seek to develop stu-dents' love of learning, respect for others, and creativity (29 percent of students); those with a general or traditional educational mission and a focus on students' core skills (28 percent of students); those with a rigorous academic emphasis, which have mission statements that focus almost exclusively on academic goals auch as excelling in school and going to college (25 percent of students); those that target a particular population of students, such as low-income students, special needs students, likely dropouts, male students, and female students (11 percent of students); and those in which a certain aspect of the curriculum, such as science or the arts, is paramount (7 percent of students).

There are a number of reasons to expect that charter schools will choose different policies and practices: They are independent and fairly autonomous. Their operating agencies have a variety of histories and priorities. All are young schools and more likely to experiment with new policies than are established schools. At the same time, there are reasons to think that New York City's charter schools will share certain policies. They commonly serve disadvantaged students; they are all under pressure to attract parents and to satisfy a small number of authorizers; one school may deliberately imitate another by adopting a policy that seems to be working in the other school; schools may also imitate one another unconsciously (as when teachers who have worked at one school are hired by another and bring their knowledge with them).

The common characteristics of charter schools reveal which innovations seem most promising to urban school leaders empowered to set their own policies (see Figure 4). About 64 percent of students attend a charter school with a school year of 190 days or longer, and 20 percent attend a school with a school year of 200 days or longer. By way of comparison, the modal school year in the United States is 180 days or 36 weeks. About 55 percent of students attend a charter school with a day that lasts eight hours or longer, 67 percent attend one with an optional afterschool program, and about 57 percent attend one with Saturday school that is mandatory for all or at least some students (for instance, stu-dents who are struggling academically).

About 49 percent of students attend a charter school that has a system of bonuses for successful teachers, and 17 percent of students attend a charter school whose teachers are unionized. Most of the students in charter schools whose teachers are unionized attend one of the five charter schools that were formerly traditional public schools but converted to charter status.

In addition, about 91 percent of charter school students attend schools that require uniforms, and about 95 percent attend schools that voluntarily administer standardized exams on a regular basis for diagnostic purposes. The advisory system is used by nearly all the charter schools that serve middle or high school grades. In an advisory system, a teacher or pair of teachers is assigned to a group of students for an entire school year. Teachers meet frequently (often daily) with their students and are responsible for tracking their progress and preventing them from "falling through the cracks." Because students in elementary grades are assigned to one teacher for most of the school day, advisory systems would be duplicative and are therefore not used by elementary schools.

[FIGURE 4 OMITTED]

About 52 percent of students attend charter schools that ask their parents to sign "contracts." Because these contracts are not enforceable, it is best to think of them as a method of trying to ensure that parents know about the school's policies and expectations. Some parents may also feel morally bound to abide by the contract. Just over half the students attend a charter school that reserves one or more seats on its board for parents. About 21 percent attend one with a disciplinary policy that fits the "no broken windows" school of thinking, which holds that encouraging small courtesies and punishing small infractions (usually at the classroom level) are important. This is in contrast to disciplinary strategies that focus more on preventing or punishing large infractions (often at an administrative level above the classroom).

The charter schools employ a variety of math and reading curricula, with no curriculum being dominant. The most popular are Saxon Math (41 percent of students) and Core Knowledge (38 percent of students.) Fifty-four percent of students have an extended English or language arts period of 90 minutes or more, and the same percentage have an extended math period. While the Children First initiative in New York City mandates a daily "literacy block" of 90 minutes for elementary school grades, the city requires that traditional public elementary schools have between 60 and 75 minutes of math instruction daily, depending on the grade.

Caroline M. Hoxby is professor of economics at Stanford University and director of the Economics of Education program at the National Bureau of Economic Research. Sonali Murarka is a project manager at the National Bureau of Economic Research. They are, respectively, principal investigator and project manager of the New York City Charter Schools Evaluation Project.