Prior to his appointment as Under Secretary, Smith was a professor of education and Dean of the Graduate School of Education at Stanford University. Previously, he was an associate professor at Harvard Graduate School of Education and a professor at the University of Wisconsin at Madison, where he also served as the Director of the Wisconsin Center for Education Research. Smith earned both a master’s (1963) and a doctoral (1970) degree in measurement and statistics from the Harvard Graduate School of Education
In this guest column, he discusses an oddity of the APS cheating scandal: The system was showing notable progress on the National Assessment of Educational Progress, which is known as the Nation’s Report Card. It wasn’t that Atlanta was leading the nation, but its progress was significant.
When we have discussed this in the past, skeptics contend that there was likely cheating in Atlanta or skewed sampling, but NAEP stands by the Atlanta scores. (To see the AJC’s take on this question, read this PolitiFact Georgia column.)
If we accept Smith’s contention that the scores were genuine, what was happening in Atlanta?
In the indictments of 35 educators two weeks ago, the Fulton DA says monetary rewards and job security led educators to cheat on the CRCT. But neither was at stake with NAEP as the scores are not used to judge school systems or educators. Nor are the scores sent home to parents, most of whom who have no idea what NAEP is or what it does. So, there is no apparent motive for APS to cheat.
It’s an interesting issue.
By Marshall S. Smith
Charges that adults in the Atlanta school system systematically altered student answers on Georgia’s annual high-stakes tests have made front pages nationwide. It’s not a new story: federal and state high pressure accountability based on narrow quantitative measures, with bonuses for some and pink slips for others, has had the unintended effect of leading some educators across the country (in Chicago, New York City, Washington DC, and Texas, among others) to allegedly fix test results.
Atlanta is the most famous example because of the intensity of the investigation and the number of indictments, reaching from teachers and principals to former Superintendent Beverly Hall. The indicted in Atlanta deserve their day in court. If they are guilty, they deserve to be appropriately penalized.
But there was something else going on in Atlanta during Hall’s tenure, which appears at odds with the picture of Atlanta painted by the investigators and the media. Atlanta’s scores on the National Assessment of Educational Progress, the Nation’s Report Card also rose very substantially between 2003 and 2011, when Hall was school chief.
Others have pointed to Atlanta’s gains on NAEP but failed to point out how extraordinary they were and did not ask how Atlanta gained so much while other districts did not.
The NAEP assessments, which are overseen by the federal government and carefully designed and administered by independent contractors, provide a valuable way to compare achievement across time and among states and districts.
Shortly after the original news of the Atlanta scandal broke the federal government made its own investigation of whether the NAEP scores from Atlanta were valid. They found that there was no evidence of cheating. Because of the careful procedures used by independent contractors who oversee the administration and transportation of the assessments it would be practically impossible for teachers and principals to alter the students work on these tests.
Students in Atlanta and nine other urban districts, New York, LA, Boston, DC, Chicago, Cleveland and Houston among them, took the NEAP Mathematics and Reading assessments at grades 4 and 8 in 2003 and then every two years until 2011, the period All 50 states also took the NAEP each of those years.
Atlanta students improved more on the 4th and 8th Reading Assessments than the students in every one of the other nine districts from 2003 to 2011. They also gained more than every one of the 50 states. In Mathematics, Atlanta’s fourth graders improved more than seven of the nine districts and more than or equal to the gains of 48 states. Eighth-grade Atlanta gains easily topped the list of all nine districts and all 50 states.
How big are the gains?
In reading in grade four, Atlanta youngsters gained 15 points, considerably more than a grade level; in grade eight, they gained 13 points. In fourth-grade math, Atlanta students gained 12 points, a little over a grade level, and in eighth grade 22 points, almost two grade levels.
Atlanta’s gains exceeded those of Georgia in every comparison. In 8th grade, Atlanta’s gains exceeded Georgia’s by almost a grade level in reading (9 points) and well over a grade level in math (14 points). Eighth grade gains are an important measure of the cumulative effect of 8 years of schooling and of students’ readiness for high school. The 8th grade reading gains for other districts were well below Atlanta’s gains of 13 points — for example, NYC gained two points, Boston three , DC school system minus 2, Cleveland zero, Chicago five.
Atlanta still has a long way to go especially. But these gains from 2003 to 2011 are substantial.
What was going on in Atlanta during these years? Clearly, the students in 2011 knew more than their predecessors. Moreover, they must have been motivated to do as well as they did on the NAEP, a test that is not consequential for them.
If the reason for the NAEP gains was the federal and state high stakes testing and accountability, why didn’t all of the districts and states succeed as well as Atlanta? Or, maybe the reason is that Atlanta has a high poverty rate and had more to gain? But Atlanta’s poverty rate is about the same as many of the districts that gained less on the NAEP.
Research on effective districts finds that key factors in explaining success include strong leadership, a professional culture, intensive training, clear and challenging goals, a quality curriculum, a respectful relationship among administrators, teachers and students, and an understanding that gains that continuously accumulate over time are more lasting than “big bang” gains.
Continuous improvement is the pattern of Atlanta’s NAEP gains from 2003-2011. The American Association of School Superintendents national award to Hall in 2009 used as criteria the quality of leadership, communication, professionalism and community involvement. AASA did not base its decision solely on test scores.
Contrast that with the view of one of the investigators as quoted in the press, “under Dr. Hall’s leadership there was a single-minded purpose and that is to cheat.”
Yet. the children in Atlanta school district achieved the largest gains in the country on an independent assessment during the same period of time. What’s happening here? The investigators were only after evidence of cheating. Perhaps a broader investigation would yield a better understanding of what was going on.
–From Maureen Downey for the AJC Get Schooled blog