Questioning the quality of virtual schools: NEPC report on K12 uses flawed measures of school performance.

Chingos, Matthew M.

Gary Miron and Jessica L. Urschel, "Understanding and Improving Full-Time Virtual Schools: A study of student characteristics, school finance, and school performance in schools operated by K12 Inc.," National Education Po/icy Center, School of Education, University of Colorado-Boulder (July 2012)

Proponents of school choice have sought for at least two decades to expand the education options available to families who lack the financial means to move to a neighborhood with high-quality public schools or to pay private-school tuition. Forty-one states and the District of Columbia now allow the founding of charter schools, which enrolled just over 2 million students in 2011-12, or about 4 percent of students nationwide, more than triple the number a decade earlier. Some states have voucher-type programs that enable children to use public funding to attend private schools, and some districts allow students to attend a traditional public school other than the one in their neighborhood.

Families certainly have more education options for their children than they did 20 years ago, but the growth of high-quality alternatives to the neighborhood school has often been constrained by geography: a student may not live within a reasonable distance of a desirable charter school or may lack reliable transportation to a school of choice if the district does not provide it. In rural communities, it may not make financial sense to have more than one school, and even populous areas may not have enough students to support a range of schools targeted at students with different needs and interests.

The potential to eliminate such geographic constraints on school choice at both the course and school levels may lie in digital learning. For instance, a student at a small high school that does not have enough students to justify offering an Advanced Placement course in physics can now take a course through an online provider if her school permits and funds such opportunities. In 31 states, students can enroll in a full-time virtual school, often from anywhere in the state, free of limitations based on geography or the physical constraints of a building.

Full-time virtual schools have gone from barely a blip on the radar screen a decade ago to enrolling approximately 275,000 students in 2011-12, according to one estimate. The schools have attracted the kind of scrutiny that most new innovations receive before they have an established track record of success (or fail and die out). The fact that many virtual schools are operated by for-profit education management organizations (EM0s) has surely contributed to the degree of scrutiny, prompting such publications as a recent report by the National Education Policy Center (NEPC) on the largest operator of these schools, K12 Inc.

[ILLUSTRATION OMITTED]

The NEPC report presents data from a variety of public sources on a portion of the schools operated by K12 Inc. (referred to henceforth as "K12"), including 48 full-time virtual schools that served more than 65,000 students in 2010-11. The report contains some useful descriptive information on the population of K12 schools across the country but is ultimately of little use to policymakers or researchers. The NEPC report uses badly flawed measures of school performance that provide little information about how much students learn as a result of attending K12 schools. Consequently, it is unclear how to interpret the report's comparisons of school finances without knowing whether K12's schools are performing well, poorly, or in between.

The NEPC Report

Written by Gary Miron and Jessica Urschel, NEPC's July 2012 report, "Understanding and Improving Full-Time Virtual Schools," is billed as a "systematic review and analysis of student characteristics, school finance, and school performance of K12-operated schools." These three sections of the report use publicly available data to compare K12-operated schools with all public schools in the same states.

The report first examines students' demographic characteristics using data from the 2010-11 school year. Compared to all students in the same states, students at K12-operated schools are more likely to be white (75 vs. 55 percent), less likely to be Hispanic (10 vs. 28 percent), and about equally likely to be black (11 percent). K12 students are modestly less likely to participate in the federal free or reduced-price lunch program (40 vs. 47 percent), roughly as likely to be classified as having a learning disability (9 vs. 12 percent), and much less likely to be English language learners (less than 1 vs. 14 percent). K12 students are disproportionately enrolled in the middle grades rather than in the elementary or high-school grades.

The NEPC report's analysis of revenues and spending in 2008-09 is limited to seven K12 schools in five states (representing approximately 60 percent of K12 enrollment nationwide) due to data constraints. The available data indicate that this subset of K12 schools received an average of $7,393 in public revenue per student, which is 20 percent less than the charter school average ($9,258) and 37 percent less than the district school average ($11,708) for the same states. K12 schools spend more on instructional costs but less on teacher salaries and benefits, and more on administration but less on administrator salaries and benefits. The NEPC report refers to these differences as cost advantages and disadvantages. For example, the fact that K12 schools spend $715 per student less on support services than public schools in the same states is interpreted as a "cost advantage" for the virtual schools.

Finally, the NEPC report summarizes a number of measures of what it calls "school performance." In 2010-11, 28 percent of K12 schools made Adequate Yearly Progress (AYP) under the federal No Child Left Behind accountability law, compared to 52 percent of schools nationwide. In the same year, only 19 percent of K12 schools rated by state education agencies (7 out of 36) received satisfactory grades. Many of these ratings reflect the fact that K12 students are less likely to score at the "proficient" level or above on statewide assessments, with differences (compared to the state average) varying by grade from 2 to 11 percentage points in reading and 14 to 36 points in math. High-school students at K12 schools have an on-time graduation rate of 49 percent, compared to 79 percent at schools in the same states.

Measuring School Quality

The NEPC report paints a dismal picture of student learning at K12-operated schools, but the fatal flaw of the report is that the measures of "performance" it employs are based primarily on outcomes such as test scores that may reveal more about student background than about the quality of the school, and on inappropriate comparisons between virtual schools and all schools in the same state. What parents and policymakers need to know about a school is how much its students learn relative to what they would have learned at the school they would otherwise have attended. In the case of virtual schools, policymakers need to know how well the students at those schools do relative to how they would have done if the virtual schools didn't exist.

The measures used in the NEPC report--whether schools make AYP, state accountability system ratings, the percentage of students that score proficient on state tests, and high-school graduation rates--are at best rough proxies for the quality of education provided by any school. Using these metrics to compare one group of schools to another is as potentially misleading as inferring that private schools are better simply because their students score higher than their public-school counterparts on the National Assessment of Educational Progress.

Rigorous efforts to measure school quality focus instead on the growth in individual students' scores on standardized tests from one year to the next. These "value-added" measures are subject to some of the same problems, but by focusing on what students learn over the course of the year, they are a significant improvement over a simple average test score (or, worse yet, the percentage of students that score above an arbitrary "proficiency" threshold). These measures can be adjusted for student background characteristics. However, such adjustments are particularly challenging in the case of virtual schools, because their students may be less likely to participate in some of the programs that are used to measure student backgrounds, such as the federal lunch program.

In addition to using poor performance measures, the NEPC report makes highly questionable comparisons between K12 students and all students in the same state. Parents don't choose between a virtual school and any school in the state, but rather between a virtual school and the schools in the vicinity of where they live. A credible measure of the effectiveness of a virtual school would compare the achievement growth of students at that school to the performance of students in the schools those students would have attended otherwise. These comparison schools may look very different from the average school in the state, especially if families are most likely to choose the virtual option when their traditional options are =satisfactory.

Measures of school performance based on carefully constructed comparisons of student achievement growth, and other important outcomes, such as high-school graduation and college enrollment rates, require student-level data that are not publicly available. Most states now have such information in their longitudinal databases, but no published studies have used these data to compare the achievement growth of students at virtual schools with demographically similar students at carefully selected comparison schools.

Research that painstakingly tries to separate out the actual effects of schools clearly has value, but it is important to bear in mind that, in the absence of random assignment of students to schools (such as occurs via charter school lotteries), families that choose for their children to be educated in their home (through virtual schools) are likely to be very different from other families. The parents of virtual-school students need to provide (or arrange for) supervision of their children during the school day. These families may use virtual schools as a form of home-schooling, or as a way to provide stability for students whose parents frequently relocate, for example.

Assembling descriptive information about the students attending virtual schools is a necessary first step to designing such careful comparisons. The NEPC report provides some basic demographic information, such as race/ethnicity, and data on participation in programs, such as free and reduced-price lunch and special education. These data are a useful starting point, but may be confounded by comparisons to statewide averages instead of to the other schools in these students' neighborhoods as well as the differences in program participation discussed earlier. A useful addition would be data based on surveys of parents with children enrolled in virtual schools and in their brick-and-mortar counterparts.

Comparing Finances

The NEPC report presents information comparing the finances of a subset of K12-operated schools with other schools in the same states, but it is hard to interpret the spending data without good information on the performance of K12 schools. If a rigorous study found that K12 schools produced equivalent (or superior) learning outcomes to traditional schools, then it would be useful to determine whether the virtual schools were able to achieve the same (or better) outcomes at lower costs. But the NEPC report contains no information that can be used to accurately measure the effect of K12 schools on how much their students learn.

The comparison of specific categories of expenditures is also difficult to interpret, in large part due to the fundamentally different instructional and operational models of virtual and brick-and-mortar schools. It is misleading to refer to all differences in spending as "cost advantages and disadvantages," when many of them reflect choices made by schools. The unsurprising fact that virtual schools do not spend much on transportation or food services likely reflects a true cost advantage of the virtual model. But differences in spending on teacher salaries as compared to student support services are not necessarily cost advantages or disadvantages, but rather decisions made by the school.

Describing differences in expenditures in this way is also confounded by differences in the overall amounts of funding provided to virtual and traditional schools. Unless states' school-finance formulas are perfectly calibrated to reflect costs, variations in spending between groups of schools will reflect both differences in costs and differences in available funding. Describing reduced spending on various categories of expenditures as cost advantages when overall spending levels differ is like telling a poor person that he has a "cost advantage" relative to a wealthier individual.

Describing the different models of education offered by virtual and traditional schools, and the implications for different categories of costs, would certainly be a useful endeavor. For example, how much can student-teacher ratios be increased, and at what cost savings, by leveraging technology in the virtual education model? But the NEPC report's conclusion that virtual schools have a cost advantage because they spend less money, when they receive less money, is simply a tautology. The publicly available data do not allow one to calculate the profits made by for-profit education providers such as K12.

The NEPC report recommends that schools be provided with funding based on the costs of educating students. This is sensible to the degree that funding is adjusted to reflect the challenge of educating certain kinds of students, such as those with special needs. But a broader policy that ties funding to costs creates perverse incentives for schools to drive up costs in order to increase their public funding. A better solution is to provide the same allocation to all schools that serve similar student populations, and then allow them to compete on quality. If parents can choose among schools and new schools can enter the market, then schools that provide a subpar education in order to increase profits would be driven from the market by higher-quality providers.

Policy Implications

Full-time virtual schools, in which students learn primarily from their own homes, clearly are not for everyone. Even after their recent enrollment growth, only one-half of 1 percent of public-school students in the U.S. attend full-time virtual schools. The key question for policymakers is whether virtual schools should be among the choices available to families deciding how best to educate their children. The NEPC report argues they should not be, calling for states to "slow or put a moratorium on the growth of full-time virtual schools." But policymakers only control the growth of enrollment in virtual schools when they decide whether or not to allow them to exist and what cap, if any, to put on their enrollments. Once those decisions are made, enrollment in virtual schools is mostly up to parents.

The success or failure of virtual schools therefore depends on the ability of policymakers and parents to evaluate their quality. Policymakers need to know whether a given virtual school meets some minimum standard so as to be acceptable as a choice for parents dissatisfied with their traditional options. Parents need to have information on which to base decisions about what school is best for their child. It is simply not possible to make these sorts of decisions with the data in the NEPC report. For example, the report tells us that 70 percent of 8th-grade students at K12-operated schools met proficiency standards in reading, as compared to 77 percent in all public schools in the same states. But we have no idea what the scores are at the neighborhood schools of the K12 students, much less what the actual effect is of attending one school or another.

[ILLUSTRATION OMITTED]

The NEPC report gets one important point right: the need for better information on school quality, especially when it comes to nontraditional schools. Acknowledging that some of the measures it uses to judge the quality of K12 schools are "inadequate or inappropriate," the report calls for states to develop new and better instruments. Some states, such as Florida, already incorporate measures of student learning growth into their accountability metrics. But much more sophisticated measures will be needed to allow policymakers and parents to adequately judge the quality of the expanding diversity of education options.

Checked by Matthew M. Chingos

Matthew Chingos is a fellow in the Brookings Institution's Brown Center on Education Policy.