CRCT tampering: Crisis in confidence, character and conscience for APS

In a jaw-dropping meeting with Gov. Sonny Perdue and his staff Tuesday, the AJC learned that a review of all 2009 CRCT answer sheets identified 74 “severe” elementary and middle schools schools in which there appeared to be widespread answer-sheet tampering. The worst offender was Atlanta Public Schools in which cheating appears to have occurred in 37 of its 55 elementary schools.

Here is the link to the AJC searchable database of all schools. Here is a link to the list of severe-only schools. Here is a link to the detailed news story.And this takes you to a map. Here is a new map created Friday.

Responding to earlier evidence of cheating, including an analysis by the AJC, the state had every 2009 answer sheet reviewed to measure how often kids changed wrong answers to right by virtue of erasures on the sheets. Because every test sheet was checked, the state was able to develop a reliable index of how often test answers were changed from wrong to right and flag schools that had inordinate occurrences of answer changes, right down to the classroom level.

It then flagged schools that had higher-than-average numbers of wrong -to-right answers, and found troubling patterns, most of which occurred in Atlanta schools and in Dougherty County schools. To understand, look at third grade math scores. Reviewing the answer sheets of 125,000 third graders, the state found that the average student changed 1.87 answers from wrong to right.

If there was a third-grade classroom in which the students on average changed 4.8 answers from wrong to right, a flag went up, said Kathleen Mathers, executive director of the Governor’s Office of Student Achievement. “That change was so much bigger than what we saw in classrooms across the state.”

APS Superintendent Beverly Hall must address a culture of cheating in her schools that threatens all her reform efforts.

Then, the state examined how each individual class performed on the test and how many answers went from incorrect to correct. They compared each classroom to the state average. To be flagged, the changes from wrong to right answers had to be well above state average, so much so that it could not be a matter of chance. Then, the state looked at the schools as a whole and found widespread instances of improbable answer changes.

On its list of the 74 schools with the highest number of classrooms with questionable – unbelievable, in fact – erasures from wrong to right, APS has 43 schools and Dougherty has eight schools.

Out of the 74 flagged schools, DeKalb had six schools including the much acclaimed DeKalb Path Academy Charter School. Fulton has three schools. Clayton has two schools, including Lewis Academy of Excellence, a charter school that appeared before the state board of education this morning to plead for a reconsideration of state charter status, citing its academic achievement and its performance on state tests.

No Gwinnett, Cobb, Fayette, Cherokee, Rockdale, Decatur, Forsyth or Henry schools are on the list of 74.

With evidence suggesting that tampering of test sheets took place in 67 percent of its 55 elementary schools, Atlanta Public Schools is now facing a crisis in culture, confidence, conscience and character.

This data show that a culture of cheating exists in Atlanta schools, a culture that may have taken root before reform-minded Superintendent Beverly Hall arrived a decade ago or may be a result of her relentless pressure on her schools to improve and do it quickly.

Either way, this culture cannot be tolerated and must be banished, even if it means a wholesale firing of staff. (I suspect that some will call for Hall’s firing. If the cheating traces back to her, she will have to resign. There may well be a case to be made that she should have known that this was going on in her district.)

This is not a few bad apples. This is rot to the core of APS and it cannot be addressed with training or memos. The shakeup at APS should bounce desks off the floor and rattle pictures off the walls.

Otherwise, how can parents know if their kids are learning if test results are not valid?

On the issue of conscience, how could schools – including some in which eight out of 10 classrooms had compelling evidence of cheating — promote students to the next grade who were not able to do the work, yet had soaring CRCT scores?

This is educational malpractice of the worst kind.

It not only hurts the children, but it victimizes the next teacher in the chain who can’t understand why her student who scored proficient in reading the year before now can’t sound out a sentence. Under the federal No Child Left Behind law, those students may have qualified for tutoring based on their undoctored scores, says Mathers, of the Governor’s Office of Student Achievement. “Students were deprived of those opportunities.”

Superintendent Hall should not waste a minute arguing with the state’s findings. As a clearly concerned Gov. Perdue told us yesterday in an hour-long meeting, “The facts are what the facts are. Trust me, we will not allow this to be whitewashed. I can’t think of any superintendent who has the students’ best interests at heart who won’t want to find out what happened and where.”

Gov. Perdue met with the AJC Tuesday to share a stunning study of how often test answers were changed on last year's CRCTs.

Perdue said he is counting on systems to investigate and right any wrongs, including getting true assessments of their students and providing academic help to kids who aren’t really proficient.

“There is strong evidence here,” he said. “It is not my job to impugn or indict any one person or school. I may change my mind as I go down the line.”

When you look at the lists – and our AJC technical staffers have been working all night to get this data from the state in easy-to-use form for you — you will see that the schools with the greatest instances of tampering are poor and minority. These are the schools that have the farthest to go to get their kids to proficiency on state tests.

You do not see the suburban powerhouse counties on the list. I also have to note that some districts with high poverty enrollments are not on the list. (Along with the 74 truly problematic schools, the state assembled a list of 117 schools with moderately troubling test irregularities.) So, between the severe- and moderate-concern schools, there are 191 or 10 percent of the state’s elementary and middle schools that have test results that merit monitoring.

But the searing findings raise so many questions. Where did the cheating take place — in the classroom or after the answer sheets were turned into the school offices?

In some instances, as many as 48 answers on a test sheet were changed from wrong to right. Why would a teacher or principal go that far to ensure a single student passed? How desperate were they?

Are the honest teachers turning a blind eye or are their complaints ignored?

How does APS rebuild after this? Why did so many Atlanta schools resort to cheating when national testing showed that APS was, in fact, raising achievement? Or, are those test results now in doubt, too?

How can teachers at the 43 APS schools on the state’s most extreme list go to work tomorrow knowing that all their good work – and there is clearly some good work amid this wreckage – is now in question?

We need to talk about whether we are asking too much of students or too little of schools?

Let’s start.

(I just received the official statement from GOSA on this and am tacking it on here as it gives more information on the process:

The Governor’s Office of Student Achievement (GOSA) today released the results of a spring 2009 Criterion Referenced Competency Test (CRCT) erasure analysis. GOSA partnered with CTB-McGraw Hill (CTB), the state’s testing vendor in charge of developing and scoring CRCT exams, to conduct a comprehensive examination of all statewide CRCT answer documents for grades 1 through 8. The analysis focused on the number of wrong answers that had been changed to right answers on individual student answer sheets in Reading, English-Language Arts, and Mathematics.

“The analysis looked on average at 125,000 test takers in every subject and grade level at which the CRCT was administered and provided a clear picture of typical student test behavior against which all schools could be compared,” said GOSA Executive Director, Kathleen Mathers. “Our recommendations are intended to eliminate future problems and help students who have been adversely affected by test tampering.”

In the analysis, CTB psychometricians scanned answer documents to identify total erasures per classroom, flagging those classrooms in which the number of wrong-to-right changes proved to be three standard deviations (SDs) or more above the state average. Less than 0.15% of test takers would be expected to fall in that range naturally.

Based on the analysis, schools were placed in varying categories according to their percentage of flagged classrooms. 80% of Georgia’s elementary and middle schools fell into the “Clear” category, meaning less than 6% of the classes within a given school were flagged; 10% fell into the “Minimal Concern” category with 6%-10% of classes flagged; 6% were determined to be in the “Moderate Concern” category with 11%-24% of classes flagged; and only 4% were termed “Severe Concern” as defined by a school having 25% or more of its classes flagged for wrong-to-right changes.

Recommendations on which the State Board of Education will vote range from requiring local Superintendents to conduct internal investigations to determine the causes of testing irregularities to schools rotating teachers during the 2010 CRCT test administration so that they administer the test to students they have not taught. In addition, state monitors will be placed in all schools in the severe concern category during this spring’s test.

“Important decisions will be made from this data that are critical to the future of Georgia’s children,” said GOSA Deputy Director, Dr. Eric Wearne. “Overall, Georgia’s schools are performing well and continue to excel in student achievement.”

The CRCT is a standardized assessment given to students in grades 1-8 in Georgia. The test is designed to measure how well students at each grade level have learned the state’s curriculum. CRCT results are used to determine whether schools have made Adequate Yearly Progress (AYP) as required by No Child Left Behind (NCLB).

GOSA plans future analyses of standardized test scores, possibly including End of Course Tests (EOCT) and Georgia High School Graduation Tests (GHSGT) and will also examine graduation and dropout rates and other factors that determine student achievement. Please visit www.gaosa.org to see the full 2009 CRCT erasure analysis report.