Look ma — No cheating! Dougherty review finds no evidence of tampering. What does state do now?
A consultant hired by the Dougherty County School System criticized the Governor’s Office of Student Achievement’s erasure analysis on the 2009 Criterion-Referenced Competency Tests, which had raised questions about many classrooms in the southwest Georgia county.
In what I suspect will be the findings of most systems asked by the state to review their CRCT practices, Dougherty’s consultant James Wilson maintains that the higher number of erasures from wrong to right reflects the test-taking strategies taught to students.
I do agree with Wilson’s comment that an easy fix to this issue would be online testing.
I am not sure how the state will react to these clean of bills of health that counties are giving themselves. I do think that the position of Dougherty — and likely other systems — still falls short of addressing a critical point. Why would students in Dougherty have such different test-taking approaches than their peers around the state?
In other words, are these counties with high wrong to right erasures full of kids who take tests differently than their peers in Gwinnett or Cobb, which both had very few classrooms with erasure anomalies?
I would love a plausible explanation as I think this is the sticking point.
According to good piece in today’s Albany Herald:
Wilson, who has worked with the board for 11 months and is on contract, said the state relied too heavily on one indicator — wrong-to-right answers — to be able to substantiate its case that Dougherty County schools had an exceptionally high number of test sections with a large number of wrong-to-right
“It’s really kind of sad to be honest,” he said of the GOSA’s efforts. “I honestly think the Office of Student Achievement got what they were looking for. They were looking for a trigger, and that’s their term, to create controversy without doing a full investigation. They only used one indicator going from wrong-to-right and they could’ve used multiple indicators.”
Wilson even noted that test vendor, “CTB-McGraw Hill clearly suggests that these types of analyses must be supported by additional, collateral information.”
As a 35-year educator who has owned Education Planners of Marietta for six years, Wilson spearheaded a three-person team that represented Dougherty County when it viewed the CRCT test documents in CTB-McGraw Hill’s secure warehouse in Indianapolis on March 22. The team of Wilson, Gerald Eads, the coordinator of evaluation and research for the Georgia Professional Standards Commission, and DCSS Test Coordinator Renee Bridges studied the actual scoring sheets of third- to eighth-grade students.
Sheets for the first grade and second grade were not available since some of the test questions could be used again the next two years. Of Dougherty’s flagged test sections, 63 percent of those in the eight elementary schools in the Severe Concern category and five elementary schools in the Moderate category were in first or second grade test sections.
Four percent of state schools made the Severe Concern list by having a school with 25 percent or more of its test sections flagged for wrong-to-right answers. Six percent of state schools were listed at the Moderate concern level with 11 percent to 24 percent of test sections flagged. Eighty percent of schools in the state were placed into the Clear of Concern category, which is less than 6 percent of the test sections within a given school were flagged.
“The erasure analysis cannot tell who, when, or why erasures were made, only that a high number of erasures of wrong to right answers,” Wilson states in his 13-page DCSS CRCT Erasure Study. “Viewing the actual test answer sheets, we were able to validate that with almost every student there were erasures from wrong to right and erasures from right to wrong and wrong to wrong.
“Many classroom scoring sheets were reviewed,” he continued. “In one fifth-grade class, English/Language Arts, with 22 students, we reviewed every erasure mark. The erasure marks of the individual 22 students on this particular test: 9, 8, 7, 7, 6, 4, 4, 3, 3, 3, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0. This class was flagged.”
Wilson said that some questions caused students to mark all the answer options and then erase all but one choice. Wilson said this may have been caused by teachers encouraging students, especially first- and second-graders, to answer all questions and then going back later to check for possible mistakes. He also said some teachers suggested making marks on questions you’re not sure about, Wilson said of conversations he had with teachers.
Dougherty Supt. Sally Whatley said students were taught to use a process of elimination, which caused a lot of erasures. As a result of the state’s findings, Whatley said students were instructed when they took this year’s test to make as few marks as possible on the answer sheets and that excessive erasures were discouraged.
“In such a case, if the erasure of multiple items were of the same depth, CTB clearly stated that the scanner could have picked these up as wrong to right changes,” Wilson stated in his report. “There were several lines where students had made small tick marks with their pencil. CTB stated that in such cases, the scanner could have picked these up as wrong to right changes. There were instances where the student erased so hard that it almost wore through the paper. Several students tried to draw the circles and letters back. CTB stated that in such cases, the scanner could have picked these up as wrong to right changes.”
After giving his presentation, board member Michael Windom provided his assessment of the state’s erasure analysis investigation. “The results weren’t valid because you need other forms of judgment,” he said. “I would just conclude the Office of Student Achievement just isn’t very accountable. If we’re going to be accused of something, at least prove it.”
Whatley, who had to file an Open Records request to get some information from the state, said that if the first and second grade erasure marks were taken out of 12 of the 13 schools, the percentage of test sections flagged decreases significantly. At the end of his presentation, Wilson said adapting an online testing system would eliminate entirely the use of erasure analysis investigations.
“The state ought to let the students take the tests online,” he said. “You take it, you finish it, you’re done with it. That’s where we ought to be today is working with technology instead of accusing second graders of cheating because of their erasure marks.”
Our AJC database reporter John Perry – who was part of the AJC team that first reported on the improbable CRCT score swings that sparked the state audit — sent me two notes about the Dougherty review that I am sharing:
Apart from the erasure analysis, our analysis of test score change points to at least three Dougherty County elementary schools with unusual gains in 09: Wet Town, MLK and Jackson Heights. They all had at least one test that was at least 3 standard deviations above their predicted score. Jackson Heights had 3 tests with gains above 2 standard deviations above the predicted score, and MLK and West Town each had 2.
All three schools were in GOSA’s severe concern category because of erasures. And even with Whatley’s “corrected” count of flagged classes, West Town is still in the severe category. Jackson Height and MLK both dropped to the moderate concern category.
Also, your blog might be a good place to ask if those test-taking tips are really different from what teachers statewide tell their students. If those techniques are common, then they don’t explain why one school has more erasures than another
Tell us what you think about the site
Customer Care »
Vacation stops, manage subscriptions and more
AJC Services
Sections
Other Editions
Visitor Agreement | Privacy Statement
© 2013 The Atlanta Journal-Constitution
62 comments Add your comment
Allen
April 30th, 2010
12:25 pm
For the record–I DO think some cheating has occurred; I have doubts about the way it was assessed.
It saddens me to think that we’ve come to this: we’re arguing over the validity of a review of alleged cheating on a test that is itself an invalid measure, of anything, but that so threatens some systems & teachers they feel compelled to take unethical measures to “succeed” on it. In other words, since the test is meaningless, the fact that some systems cheated while other systems were “cheat-free” (or, more accurately, “cheat-lite”) is also meaningless, in terms of educational quality.
If we had a DOE, a governor, and a legislature who actually cared about education, and not just about passing measures to make it look like they care maybe education in GA could get somewhere.
Ed Johnson
April 30th, 2010
12:45 pm
catlady, expect for APS a finding of “indeterminate.”
john konop
April 30th, 2010
4:10 pm
James Wilson,
In all due respect nothing in his background demonstrates Mr. Wilson has knowledge in research methods. I would love to look at the paid by the accused school district methodology used by Mr. Wilson.
@ APS parent
May 3rd, 2010
5:27 am
APS Parent,
Let me tell you that Gwinnett county does cheat on the tests! In my school, the classes flagged for abnormal erasures were 7th grade ESOL LA, 8th grade ESOL LA, and both 7th and 8th grade “intervention” teams. You see, Gwinnett just lumps all the students who need “help” into like groups and the AP over testing corrects the answers over the weekend! GCPS manipulates the testing environments to facilitate cheating! I have seen this done, but when I spoke up, the negative evals began!
Laurie
May 4th, 2010
11:02 pm
“On the GOSA site are charts that show all erasures, as well as wrong to right ones. So, you can look at those charts to figure out the right to wrong [sic - actually, from that info you can get the TOTAL of WTR PLUS WTW answers, not the breakdown between those two, but anyway...]. I will post a link to those charts shortly.”
Yes….?
Maureen Downey
May 5th, 2010
8:03 am
Laurie, If you go to http://www.gaosa.org/, you will see a list of the reports in a prominent box on the lower right of the screen. The reports have the erasure charts in them.
Teachasa2ndCareer
May 5th, 2010
11:40 pm
If some of you pay attention to kids’ conversations, you will also learn that they are listening to the news as well and are asking a ton of questions. Do you think they cheated? Should I erase on my test? If I do, will I be considered as cheating? These are fifth grade students who were so bombarded with the news that they brought the conversation into the classroom. Just like I tell my own children, I said you have every right to erase. If you feel you got it wrong after review and especially if you realize after looking for your answer in a reading section, please erase. Check your answers! By the way, the teacher manual for test is on the GA DOE site. Teachers are instructed especially this year to read it word for word. One of the parts is for us to tell the students (& I am paraphrasing) to go ahead an answer if you are not sure to answer it with what you think it is and go back and change it. I asked the same questions at our first teacher’s meeting about any analysis performed that shows the percentage of wrong to wrong and right to wrong. No one had this answer either. So, I will be definitely looking at the above link.
Also, the schools in the northern part of the state that are low may have more exposure and a higher percentage of parents who are high school and college educated even today. Furthermore, statements from parents and students who attend there say they soley teach to the test and not do those wonderful performance tasks that the state say do. Finally, has anyone looked at maybe certain people have access to the test prior to the CRCT…that is a form of cheating as well. Please don’t assume just because a school system is in the low are of erasures this means that they don’t cheat. We are still talking about human nature. If you are saying that it happens in one school district, then it happens in all of them.
For me, I have a conscious. I don’t expect my children to cheat or be helped on a standardized test and neither do I. I teach, assist, support, reteach in every possible way I can and work my butt off at home and in the classroom. They know what they know when the test arrive and I try to not make their days anymore stressful than they already are.
I agree, look at all the analyses first and other factors before making the ridiculous statement that erasures alone is an indication of cheating. Apparently, you don’t have children, you haven’t taught at schools with diverse populations, or you haven’t taught at various grade levels for a more in-depth knowledge base.
Laurie
May 6th, 2010
7:40 am
Maureen, there are a number of reports (summaries, etc.) and a number of charts within. Could you please let us know specifically the name or number of the chart (and in which report) contains the number of total erasures classroom-by-classroom? Thanks.
Laurie
May 11th, 2010
12:59 pm
Allen wrote: “Actually there’s no way to know about the statistical probability if OSA didn’t look at all the statistics: right to wrong, wrong to wrong, and wrong to right.
I’m not saying no cheating happened, I’m saying relying on the GA DOE’s to do a good job of analysis may not be the soundest strategy for determining what did happen.”
This is EXACTLY right.
Maureen Downey has posted: “To all, On the GOSA site are charts that show all erasures, as well as wrong to right ones. So, you can look at those charts to figure out the right to wrong.”
This, as it turns out, is incorrect.
I have asked on this blog about 5 times for a pointer to a chart that shows total erasures by classroom (or, alternatively, RTW and WTW erasures, as well as the WTR erasures on which the state relies to flag schools). Neither Maureen Downey nor anyone else has pointed me to them. I have read and reread and reread and reread the reports and the charts therein, looking for the information that is supposed to be there.
Finally, I emailed AJC database reporter John Perry to ask him. He has confirmed in an email to me that the state has NOT released that data, to his knowledge. The state HAS the data (the reports themselves so state), but it hasn’t released the data to the public. (Hopefully, it has released that data to APS and the other school systems it has charged with investigating the matter – although apparently Dougherty had to get some of the data it needed to conduct its investigation with an Open Records request!!) The publicly available charts on the GOSA site show the data on the total erasures STATEWIDE. But there is simply no way to determine from the data the state has released whether, say, in a class that was flagged because the average child had 3 WTR (wrong to right) answers, the average child also had 4 RTW (right to wrong) answers, and 4 WTW (wrong to wrong) answers, for a total of 11 erasures per test. I’ll repeat: there is NO WAY to determine from the data publicly released whether in fact Atlanta kids (or kids in any of the other school systems, or in any particular classroom) in fact do simply erase more frequently than the state average. And there is NO WAY to determine, for any particular flagged classroom, whether the erasures that occurred raised the students’ scores or LOWERED them. Don’t you think that information is, at the least, quite relevant to whether cheating occurred? Don’t you think, if you’re going to rely solely on the high number of erasures in a classroom as proof of cheating, you’d also want to know whether the erasures raised or lowered a classroom’s CRCT scores?
Like Allen, I’m not saying that cheating didn’t occur. (In fact, I think it’s highly unlikely that any school district in the whole state is 100% free from cheating – in a huge group of apples, there are going to be some bad apples.) I’m not even saying that cheating didn’t occur on a widespread basis in APS or any other school district.
I’m saying that looking at only the number of WTR erasures, while refusing to look at the total number of erasures and the relative proportions of WTR, RTW, and WTW answers means that you can’t reasonably draw inferences about the likelihood of cheating from erasure data. Period.
Some, including Ms. Downey, have speculated about the plausibility that Atlanta students, or those in other flagged schools, erase far more than the state average. Well, this isn’t something we should have to be speculating about, because that information is already available to the state. It may be true. It may be false. If true, it may seem strange. If true, it may indicate, as Catlady says, a less than ideal test coaching strategy (or not – I am neither an administrator nor a teacher – I’m a parent who took standardized tests as a child and was never instructed to cross out answers in my answer book either). But IF true, if high WTR erasures is just part of a pattern of high erasures of all kinds (whether in a particular classroom, or in a school district), that DRASTICALLY decreases the chances that the high rate of WTR erasures indicates cheating, and increases the chances that there is some other explanation that some students are erasing more than others.
My question is: why are we speculating about what is “plausible”, when the data has already been collected?
And that’s what is really strange (fishy? or maybe just indicative of very, very sloppy work): the state HAS this data. It says so in the reports themselves. Why is it not releasing this data? Why, even though it’s now been months since spokespeople and others from APS started suggesting that its students were instructed differently and/or may have erased more frequently than some others, why is this data still not public? Those who keep saying that it’s terrible that APS is taking so long to investigate the situation (and apparently just want, what, for Bev Hall and every principal and teacher at every APS school to be summarily fired without any actual investigation?), I ask you to remember that the state itself had almost a year to compile this data and prepare a report – and then ask yourselves, why, many months after the state was on notice (if it had not already been on notice, as I think it should have been) that analysis of all types of erasures was extremely relevant to determining whether cheating occurred, why has it still not released data it has in its possession, much less undertaken to analyze the data?
And – why doesn’t the AJC care?
Maureen Downey
May 11th, 2010
2:18 pm
@laurie, I didn’t get that you were searching for total erasures by classroom. I am hoping that you did see the state breakdown of total erasures by grade, which are online.
John Perry and I just discussed your comment and he has sent a note to GOSA.
Maureen
Laurie
May 11th, 2010
5:56 pm
“I am hoping that you did see the state breakdown of total erasures by grade, which are online.”
I have read all of the GOSA reports and charts, carefully, several times. And, yes, I have seen the breakdown of total erasures in GEORGIA by grade, but I’m not sure why you are referencing them. Those charts are completely irrelevant to the question Dougherty is raising, to the question Allen and Catlady and some others are discussing in the comments to this blog, and to the question that I have repeatedly raised. The issue is not erasures of third graders versus the erasures of fifth graders. The issue is how often APS students (and those in other vilified districts) erase versus how often Georgia students in general erase. We already know that they likely change their answers from wrong to right more often than Georgia students in general do. But do they also make more erasures in general (not just wrong to right ones erasures, which one might expect to be the case if teachers or administrators are doing the erasing in hopes of raising CRCT scores)? And if so, to what degree is this the case? Charts with Georgia data provide only half the answer. To compare APS with Georgia as a whole, you have to have data about APS. (Similarly, to compare an individual classroom with Georgia as a whole, you have to have data about an individual classroom.) You can’t compare two sets of data if you have only one of them! And breaking down the Georgia by grade level, as is done in the charts you reference, doesn’t change that. (I’m not sure why that grade-by-grade chart is even in that report, since the data it contains is statewide only, and thus, in the absence of breakdown by district, school, or classroom, irrelevant to whether any district, school, or classroom is cheating. Perhaps it’s just intended by the report drafters as a distractor, in which case, I can see that it has unfortunately fulfilled its purpose.)
“I didn’t get that you were searching for total erasures by classroom”
Given that AJC.com previously had a “check your child’s classroom and see if the teacher cheated” chart online, I’d love to see data by individual classroom, so that individual teachers have a chance of exonerating themselves publicly. (Yes, I know it didn’t identify the classrooms by name, but in some cases, it was possible to determine anyway, e.g., when all 4 First Grade classes in a particular school were flagged.) And given that GOSA’s WTR analysis was a classroom-by-classroom analysis, ultimately, that’s the kind of data that will be needed to correct their analysis.
But for now, I’ll point out that there is not even any publicly available ***district-by-district*** data on total erasures, so there is no basis for claiming that Atlanta students DON’T erase more than Georgia students as a whole. And no basis for asserting that APS students “def[ied]the odds that kids will sometimes change right answers to wrong ones,” as you stated in a previous blog post.
“John Perry and I just discussed your comment and he has sent a note to GOSA.”
Yay. Thank you.
Maureen Downey
May 11th, 2010
6:01 pm
Laurie, In response to the issues you raised, John sent GOSA a note today. GOSA has sent over additional data and John is going through it today and tomorrow. I asked John if he could write a summary once he takes a look at it. (John is our database expert.)
Maureen