View source for On negative effects of vouchers

<!-- you can have any number of categories here -->
[[Category:Mark Dynarski]]
[[Category:Vouchers]]
<!-- 1 URL must be followed by >= 0 Other URL and Old URL and 1 End URL.-->
{{URL | url = http://www.brookings.edu/research/reports/2016/05/26-on-negative-effects-of-school-vouchers-dynarski?rssid=LatestFromBrookings&utm_source=feedblitz&utm_medium=FeedBlitzRss&utm_campaign=FeedBlitzRss&utm_content=On+negative+effects+of+vouchers}}
<!-- {{Other URL | url = }} -->
<!-- {{Old URL | url = }} -->
{{End URL}}
{{DES | des = Recent research on statewide voucher programs in Louisiana and Indiana has found that public school students that received vouchers to attend private schools subsequently scored lower on reading and math tests compared to similar students that remained in public schools. The magnitudes of the negative impacts were large. These studies used rigorous research designs that allow for strong causal conclusions. | show=}}
<!-- insert wiki page text here -->
<!-- DPL has problems with categories that have a single quote in them.  Use these explicit workarounds. -->
<!-- otherwise, we would use {{Links}} and {{Quotes}}  -->
{{List|title=On negative effects of vouchers|links=true}}
{{Quotations|title=On negative effects of vouchers|quotes=true}}

{{Text |
Executive summary

Recent research on statewide voucher programs in Louisiana and Indiana has found that public school students that received vouchers to attend private schools subsequently scored lower on reading and math tests compared to similar students that remained in public schools. The magnitudes of the negative impacts were large. These studies used rigorous research designs that allow for strong causal conclusions. And they showed that the results were not explained by the particular tests that were used or the possibility that students receiving vouchers transferred out of above-average public schools.

Another explanation is that our historical understanding of the superior performance of private schools is no longer accurate. Since the nineties, public schools have been under heavy pressure to improve test scores. Private schools were exempt from these accountability requirements. A recent study showed that public schools closed the score gap with private schools. That study did not look specifically at Louisiana and Indiana, but trends in scores on the National Assessment of Educational Progress for public school students in those states are similar to national trends.

In education as in medicine, ‘first, do no harm’ is a powerful guiding principle. A case to use taxpayer funds to send children of low-income parents to private schools is based on an expectation that the outcome will be positive. These recent findings point in the other direction. More needs to be known about long-term outcomes from these recently implemented voucher programs to make the case that they are a good investment of public funds. As well, we need to know if private schools would up their game in a scenario in which their performance with voucher students is reported publicly and subject to both regulatory and market accountability.

Recent research on statewide voucher programs in Louisiana and Indiana has found that public school students that received vouchers to attend private schools scored lower compared to similar students who did not attend private schools.[i] This is the kind of research finding that generates a reaction of ‘wait, what?’ Negative effects are rare in education research.

The magnitudes of the negative impacts were large, too. In Louisiana, a public school student who was average in math (at the 50th percentile) and began attending a private school using a voucher declined to the 34th percentile after one year. If that student was in third, fourth, or fifth grade, the decline was steeper, to the 26th percentile. Reading declined, too: a student at the 50th percentile in reading declined to about the 46th percentile. In Indiana, a student who had entered a private school with a math score at the 50th percentile declined to the 44th percentile after one year.

Voucher programs emerged in the nineties in Milwaukee, and later in New York City, Dayton, Ohio, and Washington, DC.[ii] The Milwaukee, Dayton, and New York programs were supported by state and local funds or philanthropies, and the DC program was supported by Federal funds. In each location, students applied to receive a voucher, which would be paid to the private school they chose to attend.

Since then, vouchers and voucher-like mechanisms for supporting private school choice with public funds (tax-credit scholarships, education savings accounts, and individual tax-credits) have exploded at the state level.[iii]

In publicly funded voucher programs, many of the private schools that are recipients are religious organizations. This generates thorny legal issues in the 38 states that have so-called Blaine amendments to their constitutions (which prohibit direct government aid to educational institutions that have a religious affiliation and were originally intended to target Catholic schools in states with large immigrant populations).

Legal issues aside, the present rationale for vouchers—perhaps the rationale—is to help low-income students attend private schools because they will learn more. And if parents look at data from the National Assessment of Educational Progress (NAEP), they would see that, historically, private school students regularly score higher than public school students by 15 to 20 points in reading and math, which is a big difference (on average, NAEP scores rise about 10 points for each grade level). So, a parent might surmise, if their child attends private school, their child’s scores will be higher too. But some part of that score difference arises because higher-performing students select into private schools. Parents do not know how their child would score in private school.

And, in fact, studies of vouchers have reported mixed results on scores. Scores improved for some students in some places, and scores did not improve for other students in other places.[iv] The evidence showed higher reading and math scores for black students in New York City, and higher reading scores in the third follow-up year of the DC program. The evidence showed no differences in scores for Milwaukee students in reading, for Hispanic students in New York City in reading and math, and for DC students in math. What little research there has been on other education outcomes reported higher rates of high school graduation in New York City and DC, and higher rates of college attendance in New York City.

None of these earlier studies reported significant negative effects on test scores, which adds to interest in what might be happening in Louisiana and Indiana that could explain negative effects. Is it something about how the research studies were designed, how the programs were structured, the quality of the public schools against which private schools were being compared?

The research designs are unlikely to explain the findings. The Louisiana study used a random assignment experimental design, which is a high research standard for measuring effects. Louisiana private schools that had more applicants than available seats operated lotteries to choose applicants randomly. Students who won the lottery constituted the treatment group, whereas those that lost were the control group. Random assignment experiments are central in medical research and are increasingly used in education. For example, studies of the effects of charter schools have relied on experiments created by school lotteries.[v] And two different teams of researchers studying the Louisiana data reached similar conclusions.

In Indiana, effects were measured by comparing time trends of test scores for students who initially attended public schools and then received vouchers to time trends for students who continued to attend public schools, a ‘fixed-effects’ estimator. This is a lower research standard for measuring effects than experimental designs, but sample sizes were quite large (more than 3,000 voucher students and 500,000 public school students) and the findings are consistent with findings from Louisiana.

There was an oddity in the structure of testing in Louisiana. Tests used there were mandated by the state to be the ones used for the Louisiana Education Assessment Program (LEAP) carried out in public schools. Students that used their voucher to attend private schools had to take LEAP tests, though other students in those schools did not. Because LEAP tests are part of the state’s accountability structure for its public schools, the tests are ‘high stakes’ for those schools but low stakes for private schools. Perhaps students in the voucher group did worse than those who remained in public schools because the public schools taught to the test, whereas the private schools did not.

To address this hypothesis, the Mills and Wolf study exploited the fact that some LEAP tests had more content related to Louisiana state standards, and those tests might favor public school students. The study reported that private school voucher students did worse than public school counterparts regardless of the degree of alignment of the tests with the state standards guiding instruction in public schools.

Maybe students that applied for vouchers were attending excellent public schools rather than mediocre or bad ones, so the private schools that received voucher students were up against the cream of Louisiana public schools. The Abdulkdiroglu et al. study rejected this hypothesis with the finding that Louisiana public schools attended by students applying for a voucher were below average in the state. On the recent NAEP, Louisiana ranked 43rd among states in reading scores of its fourth graders and 49th among states in math scores of its eighth graders. To test the effectiveness of the instruction of voucher students in private schools by comparing the students’ learning outcomes with those of similar students in below-average public schools in Louisiana is not a high bar.

Another possibility that these studies did not investigate is that our historical understanding of the superior performance of private schools is no longer accurate. Since the nineties, public schools have been under heavy pressure to improve test scores. No Child Left Behind and conditional waivers from it under the Obama administration are visible manifestations of this pressure, with various kinds of sanctions for not meeting targets for score growth. Private schools were exempt from these accountability requirements.

A recent study by Wong et al. used NAEP data to compare public schools to private schools from the nineties to 2011.[vi] The study reported that private schools started with higher scores in the nineties—this is the difference between private school students and public school students mentioned at the outset, above. By 2011, the gap had closed significantly, especially in math. The authors concluded that No Child Left Behind improved scores. But a broader conclusion is relevant here. Public schools improved relative to private schools, for whatever reason.

The figure showing national private and public school scores on fourth-grade math indicates some of what the authors found. In 2000, scores of private school students are well above scores of public school students, a gap of 14 points. By 2013, the gap had closed to 5 points. 



The same contrast of public and private schools cannot be done for Louisiana or Indiana per se. Their samples of private schools are too small to meet reporting standards of the National Center of Education Statistics. But Louisiana and Indiana public schools show trends on NAEP similar to national trends. The second figure uses the same national trends as the previous figure (the dashed lines) and adds score trends for Louisiana and Indiana. General improvements in the trend that were evident nationally also are evident in Louisiana and Indiana. In fact, by 2013, Indiana’s public school fourth graders score above the national private school average.



Improvements in scores also are evident for the low-income population of students most likely to apply for vouchers. The next figure compares fourth-grade math scores for all public school students in Louisiana and students eligible for a free or reduced-price school lunch, which is a common indicator for poverty. It shows a lower score level for students in poverty but a nearly identical trend. Indiana score trends for low-income students also mirror its general trend.



It is at least plausible that Louisiana and Indiana public schools have surpassed their private schools (for low-income students). Without data for private schools in each state, we can’t know for sure. But to the extent that voucher programs are based on the premise that students from low-income families receiving a voucher will have access to private schools that are, on average, substantially better than public schools, the equity rationale for vouchers is in doubt.

Based on what we have learned from the recent research I’ve reviewed, the decision facing parents who are considering applying to these voucher programs is complex. Suppose a parent is told their child will likely perform less well in reading and math (possibly much less, if they are in earlier grades), but might be more likely to graduate and go to college. How many would take that risk? Doing so requires a leap of faith that lower skills in reading and math will not affect their child’s future path through school, which is not intuitive to say the least. If a parent were enrolling their child in kindergarten, it would be quite a leap of faith, amounting to ‘in 13 years, it will all work out.’

In education as in medicine, ‘first, do no harm’ is a powerful guiding principle. A case to use taxpayer funds to send children of low-income parents to private schools is based on an expectation that the outcome will be positive. These recent findings point in the other direction.

More needs to be known about long-term outcomes from these recently implemented voucher programs to make the case that they are a good or bad investment of public funds. As well, we need to know if private schools would up their game in a scenario in which their performance with voucher students is reported publicly and subject to both regulatory and market accountability. And, finally, there are difficult philosophical and political questions that voters and elected officials need to mindfully address, including whether families with the personal wealth to pay tuition at a private school should have the opportunity to choose that school even if it is underperforming on traditional academic test measures whereas low-income families should be denied that right.


[i] Two studies have examined the effects of vouchers in Louisiana, one from the National Bureau of Economic Research (Atila Abdulkdiroglu, Parag A. Pathak, and Christopher R. Walters, “School Vouchers and Student Achievement: Evidence form the Louisiana Scholarship Program,” NBER Working Paper 21839 (Cambridge, MA: December 2015)) and one from the University of Arkansas (Jonathan N. Mills and Patrick J. Wolf, “The Effects of the Louisiana Scholarship Program on Student Achievement After Two Years,” (Fayetteville, AR: University of Arkansas department of Education Reform, 2016)). The study of Indiana is by Joseph Waddington and Mark Berends, “Vouchers in the Crossroads, Heterogeneous Impacts on Student Achievement and Attendance across Private Schools in Indiana” (nd).

[ii] Wolf (2012) summarizes the numerous studies of the Milwaukee program (http://www.uaedreform.org/downloads/2012/02/report-36-the-comprehensive-longitudinal-evaluation-of-the-milwaukee-parental-choice-program.pdf). Results of the New York City program are summarized at https://www.mathematica-mpr.com/our-publications-and-findings/projects/an-evaluation-of-new-york-city-school-vouchers and results of the DC program are at http://ies.ed.gov/ncee/pubs/20104018/pdf/20104018.pdf. A summary and comparison of the New York, Dayton, and DC results is at http://educationnext.org/vouchersinnewyorkdaytonanddc/.

[iii] http://www.edchoice.org/school-choice/school-choice-in-america/  

[iv] Matthew Chingos and Paul Peterson, “The Impact of School Vouchers on College Enrollment,” Education Next 13, no. 3 (Summer 2013).

[v] Melissa A. Clark, Philip Gleason, Christina Clark Tuttle, and Marsha K. Silverberg, “Do Charter Schools Improve Student Achievement? Evidence from a National Randomized Study,” (Princeton: Mathematica Policy Research, December 2011); Joshua D. Angrist, Sarah R. Cohodes, Susan M. Dynarski, Parag A. Pathak, and Christopher R. Walters, “Stand and Deliver: Effects of Boston’s Charter High Schools on College Preparation, Entry, and Choice,” NBER Working Paper 19275 (Cambridge, MA: National Bureau of Economic Research, July 2013); Caroline M. Hoxby, Sonali Murarka, and Jenny Kang, “How New York City’s Charter Schools Affect Achievement,” (Cambridge, MA: New York City Charter Schools Evaluation Project, September 2009).

[vi] http://www.tandfonline.com/doi/full/10.1080/19345747.2013.878011
}}