Major review of No Child Left Behind. Kids moved ahead in math, but in little else.

A much-anticipated evaluation of the No Child Left Behind Act — the sweeping federal law that imposed consequential accountability on all states and schools — found strong evidence that math achievement improved in the earlier grades as a result of the decades-long law.

A new evaluation of President Bush's signature law No Child Left Behind found improvement to math performance in lower grades, but to little else.

But  the controversial law had no impact on reading.

One of the challenges to analyzing the law has been separating out what changes in student performance were related to other variables including the improvements to the economy during the decade. All those moving parts made it hard for studies to offer any definitive claims about the law’s impact, said researcher Thomas Dee. (I just finished a phone interview with Dee and will be writing that for my Monday education column. I will also post here.)

Dee and his colleague Brian Jacob sought out a credible control group. The foundation of their comparison became those states that imposed the earliest generation of statewide school-level accountability systems. In a podcast on Education Next,  Dee notes that  NCLB was just tinkering for many states since they already had accountability in place. “In other word, it was largely irrelevant for them,” he said.

Using those pioneer states as a a control group, the researchers looked at the states that never had introduced any school-wide accountability.

And the results show mixed results: Improvements were concentrated in the earlier grades, most notably in grade 4 NAEP math scores and mostly among Hispanic and low-income students.  In that respect, Dee says the law fell short of its so-called “moonshot rhetoric” that it would remake American education.

The study does not conclude why the improvements came in math over reading, suggesting that more work needs to be done to see whether there is something unique about teaching math in the early grades.

The revised NCLB plan under consideration by the Obama administration would get rid of the serious consequences for many of the nation’s schools and instead limit those consequences for underperformance to the 5 to 10 percent of the lowest performing schools.

Under the Obama plan, the other 90 percent of the nation’s schools would still have testing and public release of testing data, but not the consequences now imposed for failing to show progress among a wide range of student subgroups. Dee said getting data out for public consumption has not been shown to be sufficient to drive improvements to schools.  If there are no consequences for schools, Dee predicts a reduction in improvements gained through policies like NCLB.

Now, here is the official release on the study:

Just as the Obama administration has signaled that it has made reauthorizing the landmark No Child Left Behind  federal law a priority in 2010, an Education Next analysis by professors Thomas Dee and Brian Jacob shows that NCLB is responsible for marked gains in math skills, particularly among Latino and low-income students, but produced no improvements in reading achievement.

The impact NCLB has had on student achievement since its implementation in 2002 has always been difficult to gauge. Since the law applied to all public school students, there was no comparison group and it was impossible to determine which of countless factors contributed to student achievement.

However, authors Dee and Jacob conducted groundbreaking research, to be published in the summer issue of Education Next and available now online, comparing test score changes in states that did not have NCLB-style accountability systems (both publicizing performance and attaching consequences to the performance) in place before 2002 to changes in those that already did when NCLB was implemented.

Dee’s and Jacob’s findings suggest that “the accountability provisions of NCLB generated large and statistically significant increases in the math achievement of 4th graders and that these gains were concentrated among Hispanic and low-income students.”

“Specifically, we find evidence that the accountability provisions of NCLB generated large and broad gains in the math achievement of 4th grad¬ers and somewhat smaller gains for 8th graders,” said the authors.  “Our results suggest that NCLB accountability had no impact on read¬ing achievement for either group.”

The study relied on test-score data from the National Assessment of Educational Progress (NAEP), also known as “The Nation’s Report Card.”  Data were available in 39 states for 4th-grade math, 38 states for 8th-grade math, 37 states for 4th-grade reading and 34 states for 8th-grade reading. The scholars found that NCLB raised the percentage of students who reached a basic level of proficiency by 10 percentage points in 4th grade math and by 6 percentage points in 8th grade math.  The percentages reaching full proficiency in math increased by 6 percentage points in 4th grade, but no detectable gains were identified for the percentage reaching full proficiency in 8th grade math.  Those identified as fully proficient in 4th grade reading increased by 2.5 percentage points, but no other significant reading impacts were identified. NCLB impacts on Hispanic math performance were even greater.

The research also found that NCLB increased achievement among higher-achieving students, casting doubt on concerns that the law has harmed this group.

The authors say that as lawmakers consider a redesign of NCLB, they may need to pay more specific attention to understanding what causes differing results by grade and subject.

“Understanding these differences, according to the analysis, will be critical as policymakers discuss the future design of NCLB,” Jacob said. “Our results, much like earlier evaluations of state-level school accountability policies, show that we need to look closely at what’s happening within our schools that can cause these changes in achievement.”

Thomas Dee, currently an associate professor of economics at Swarthmore College, will be professor of public policy and economics at the University of Virginia this fall and Brian Jacob is professor of education policy and economics at the University of Michigan.