A consultant hired by the Dougherty County School System criticized the Governor’s Office of Student Achievement’s erasure analysis on the 2009 Criterion-Referenced Competency Tests, which had raised questions about many classrooms in the southwest Georgia county.
In what I suspect will be the findings of most systems asked by the state to review their CRCT practices, Dougherty’s consultant James Wilson maintains that the higher number of erasures from wrong to right reflects the test-taking strategies taught to students.
I do agree with Wilson’s comment that an easy fix to this issue would be online testing.
I am not sure how the state will react to these clean of bills of health that counties are giving themselves. I do think that the position of Dougherty — and likely other systems — still falls short of addressing a critical point. Why would students in Dougherty have such different test-taking approaches than their peers around the state?
In other words, are these counties with high wrong to right erasures full of kids who take tests differently than their peers in Gwinnett or Cobb, which both had very few classrooms with erasure anomalies?
I would love a plausible explanation as I think this is the sticking point.
According to good piece in today’s Albany Herald:
Wilson, who has worked with the board for 11 months and is on contract, said the state relied too heavily on one indicator — wrong-to-right answers — to be able to substantiate its case that Dougherty County schools had an exceptionally high number of test sections with a large number of wrong-to-right
“It’s really kind of sad to be honest,” he said of the GOSA’s efforts. “I honestly think the Office of Student Achievement got what they were looking for. They were looking for a trigger, and that’s their term, to create controversy without doing a full investigation. They only used one indicator going from wrong-to-right and they could’ve used multiple indicators.”
Wilson even noted that test vendor, “CTB-McGraw Hill clearly suggests that these types of analyses must be supported by additional, collateral information.”
As a 35-year educator who has owned Education Planners of Marietta for six years, Wilson spearheaded a three-person team that represented Dougherty County when it viewed the CRCT test documents in CTB-McGraw Hill’s secure warehouse in Indianapolis on March 22. The team of Wilson, Gerald Eads, the coordinator of evaluation and research for the Georgia Professional Standards Commission, and DCSS Test Coordinator Renee Bridges studied the actual scoring sheets of third- to eighth-grade students.
Sheets for the first grade and second grade were not available since some of the test questions could be used again the next two years. Of Dougherty’s flagged test sections, 63 percent of those in the eight elementary schools in the Severe Concern category and five elementary schools in the Moderate category were in first or second grade test sections.
Four percent of state schools made the Severe Concern list by having a school with 25 percent or more of its test sections flagged for wrong-to-right answers. Six percent of state schools were listed at the Moderate concern level with 11 percent to 24 percent of test sections flagged. Eighty percent of schools in the state were placed into the Clear of Concern category, which is less than 6 percent of the test sections within a given school were flagged.
“The erasure analysis cannot tell who, when, or why erasures were made, only that a high number of erasures of wrong to right answers,” Wilson states in his 13-page DCSS CRCT Erasure Study. “Viewing the actual test answer sheets, we were able to validate that with almost every student there were erasures from wrong to right and erasures from right to wrong and wrong to wrong.
“Many classroom scoring sheets were reviewed,” he continued. “In one fifth-grade class, English/Language Arts, with 22 students, we reviewed every erasure mark. The erasure marks of the individual 22 students on this particular test: 9, 8, 7, 7, 6, 4, 4, 3, 3, 3, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0. This class was flagged.”
Wilson said that some questions caused students to mark all the answer options and then erase all but one choice. Wilson said this may have been caused by teachers encouraging students, especially first- and second-graders, to answer all questions and then going back later to check for possible mistakes. He also said some teachers suggested making marks on questions you’re not sure about, Wilson said of conversations he had with teachers.
Dougherty Supt. Sally Whatley said students were taught to use a process of elimination, which caused a lot of erasures. As a result of the state’s findings, Whatley said students were instructed when they took this year’s test to make as few marks as possible on the answer sheets and that excessive erasures were discouraged.
“In such a case, if the erasure of multiple items were of the same depth, CTB clearly stated that the scanner could have picked these up as wrong to right changes,” Wilson stated in his report. “There were several lines where students had made small tick marks with their pencil. CTB stated that in such cases, the scanner could have picked these up as wrong to right changes. There were instances where the student erased so hard that it almost wore through the paper. Several students tried to draw the circles and letters back. CTB stated that in such cases, the scanner could have picked these up as wrong to right changes.”
After giving his presentation, board member Michael Windom provided his assessment of the state’s erasure analysis investigation. “The results weren’t valid because you need other forms of judgment,” he said. “I would just conclude the Office of Student Achievement just isn’t very accountable. If we’re going to be accused of something, at least prove it.”
Whatley, who had to file an Open Records request to get some information from the state, said that if the first and second grade erasure marks were taken out of 12 of the 13 schools, the percentage of test sections flagged decreases significantly. At the end of his presentation, Wilson said adapting an online testing system would eliminate entirely the use of erasure analysis investigations.
“The state ought to let the students take the tests online,” he said. “You take it, you finish it, you’re done with it. That’s where we ought to be today is working with technology instead of accusing second graders of cheating because of their erasure marks.”
Our AJC database reporter John Perry – who was part of the AJC team that first reported on the improbable CRCT score swings that sparked the state audit — sent me two notes about the Dougherty review that I am sharing:
Apart from the erasure analysis, our analysis of test score change points to at least three Dougherty County elementary schools with unusual gains in 09: Wet Town, MLK and Jackson Heights. They all had at least one test that was at least 3 standard deviations above their predicted score. Jackson Heights had 3 tests with gains above 2 standard deviations above the predicted score, and MLK and West Town each had 2.
All three schools were in GOSA’s severe concern category because of erasures. And even with Whatley’s “corrected” count of flagged classes, West Town is still in the severe category. Jackson Height and MLK both dropped to the moderate concern category.
Also, your blog might be a good place to ask if those test-taking tips are really different from what teachers statewide tell their students. If those techniques are common, then they don’t explain why one school has more erasures than another