Testing reviewer in CRCT APS probe: We did state-of-the-art testing analyses

I interviewed the testing expert John Fremer on the phone for an hour a few weeks ago and have shared some of his comments with you here, largely his remarks that the erasure analysis done by the state the snared many APS schools was a first step but that more thorough evaluation and measures ought to be performed before people are indicted for cheating.

His own review led to the same 12 schools that the state deemed the top offenders, but he said he needed more information — some of which the state would not give to him — to make judgments on other schools that the state flagged in APS. When the governor and his staff met with the AJC, I asked Kathleen Mathers of the Governor’s Office of Student Achievement about Fremer’s contention that he needed more data and she said the state gave him everything he wanted.

When I asked him about the contention — later voiced by Perdue — that these results were designed from the start to exonerate APS, Fremer told me that there is no reason for his company to fudge results as this is their business and they do it on a global basis so their integrity is crucial

With that background, here is what Fremer wrote in today’s paper:

As students return to school this fall, Gov. Sonny Perdue and the Governor’s Office of Student Achievement, or GOSA, continue to criticize the work of my company, Caveon Test Security. The governor’s position is misguided with respect to Caveon, but he is calling attention to an ugly reality: Wholesale, organized cheating in some Atlanta public schools occurred and must be addressed. His statements about Caveon, though, do not rest on well-established and scientific cheating detection methods.

The governor’s public statements about our analyses of 2009 statewide Criterion Referenced Competency Tests, or CRCT, data ignore that we confirmed GOSA’s conclusion that cheating occurred at some APS schools. Caveon is the leading test security company in the field, with experience examining more than 15 million test instances for the largest, most esteemed test organizations in the world, including 15 state departments of education and the Department of Defense.

The state’s vendor used a “preliminary screening” method of counting wrong-to-right CRCT erasures. Caveon would never recommend that our clients launch full-scale investigations solely on the basis of wrong-to-right erasures. Such an approach flies in the face of industry best practices.

A key concept in our work is helping clients focus on the “worst of the worst.” We also believe that minimizing false accusations is highly desirable. This means every school and class that is investigated shows clear evidence of serious problems.

What the governor seems unwilling to recognize is that many of the Atlanta schools flagged in the original GOSA analysis are almost certainly on the list of “schools of concern” because of factors unrelated to cheating. These factors include common missteps such as students getting off track and making mistakes marking their answer sheets. “Losing your place” on answer sheets occurs on a fairly common basis. The CRCT is no exception, and when a student realizes a miscue, erases the misaligned marks and corrects the answer sheet, wrong-to-right erasures are created.

No credible measurement expert would be satisfied with the original GOSA analyses as a basis for deciding that a teacher or school had serious problems unless the results were very extreme, as was the case with some of the schools flagged by GOSA and confirmed by Caveon’s in-depth follow-up.

To the detriment of our ability to analyze tests to the fullest extent possible, the GOSA provided data only for the 58 schools that had been designated as being “of concern,” not all APS schools. Most importantly, we did not receive student-by-student response data for every question. This omission prevented us from conducting these other cheating analyses:

● Collusion: Which schools and classes had “unusual agreement” among students, far beyond what could reasonably have occurred by chance alone? This is the most basic cheating analysis, one any expert would insist on doing.

● Unusual patterns: Which schools and classes had very odd patterns of student responses, such as missing easy questions and answering hard questions? This outcome is very likely to have been caused by students receiving help before or during an exam as well as tampering with answer sheets or student records after testing.

From Caveon’s perspective, we would very much appreciate discontinuation of uninformed and wrongheaded criticism of the state-of-the-art cheating analyses that we did. Any contention that we jeopardized our personal and professional reputations to please a client has no basis in fact whatsoever.

John Fremer is president and COO of Caveon Testing Security