Why is state throwing up roadblocks in CRCT investigation?
Longtime educator Herb Garrett of the Georgia School Superintendents Association sent me this piece, which I hope to run on the Monday education page in the next week or so. But I am putting it up here as I think it raises good questions, especially in light of today’s AJC story that the data security firm hired to investigate APS suggests the state isn’t being all that helpful.
By Herb Garrett
As I have watched the CRCT “cheating scandal” unfold, I have been amazed at the number of people willing to jump to conclusions before gathering necessary evidence.
At the same time, I find myself dumbfounded at the roadblocks that are being thrown in front of those who wish to do the evidence-gathering so as to do a complete and thorough investigation.
Now, let me be clear: There is a lot of smoke around this issue, so there is the likelihood that there will be some cheating “fire.”
However, there is also some smoke being blown over this, and it is imperative that educators who had no business being accused of wrongdoing have a complete chance to exonerate themselves.
According to the Governor’s Office of Student Achievement, 44 percent of the classes that were “flagged” for excessive erasures were first and second grade classes. Apparently, no one thought to ask the question: “Does that make any sense?”
I would offer the experienced opinion that the answer is “no.” There is absolutely no motivation to cheat at those grade levels because neither grade’s test scores count in the computation of the all-important Adequate Yearly Progress score. Scores don’t begin to count until third grade.
Neither first nor second graders are required to pass the tests to be promoted to the next grade.
With only the rarest of exceptions, first and second grade teachers do not cheat. It is not a part of their culture. I think I know what I’m talking about on this one; that’s where my 42 years of experience in public education comes in.
First and second grade students mark everywhere on their answer documents. These 6- and 7-year-olds draw happy faces, puppy dogs and even random lines across the page. The job of the teacher is to “clean up” the documents after they are handed in, which could account for extra erasure marks.
Now, the only way that this can be investigated adequately is for the actual testing documents to be viewed by those responsible for proper fact-finding. But GOSA has limited access to the documents. School systems are given the option of viewing only a sample of the answer sheets at the GOSA offices or can choose to spend taxpayers’ money to fly to Indianapolis to view documents at the headquarters of the testing company.
Further, GOSA has declared that first and second grade answer booklets cannot be viewed since some of the test questions will be used again next year Never mind that the first and second grade teachers read the test questions aloud to their students during the testing sessions and thus have already seen the questions. Aren’t these documents student records that were originated in the local school system, thus paving the way for local system personnel to view them??
It has been apparent from the outset of this event that the GOSA staff is supremely confident in their statistical methodology. It is just as obvious that those responsible for investigating alleged wrongdoing want more than a “trust us” answer from GOSA to questions they have on the methodology.
GOSA’s reluctance to provide an opportunity for no more than a cursory glance at the “evidence” of possible cheating leaves eyebrows raised and questions in the minds of many local system leaders as to what their next steps should be in determining the truth about what may (or may not) have happened.
Herb Garrett is of the executive director of the Georgia School Superintendents Association.
Tell us what you think about the site
Customer Care »
Vacation stops, manage subscriptions and more
AJC Services
Sections
Other Editions
Guides
Visitor Agreement | Privacy Statement
© 2011 The Atlanta Journal-Constitution
80 comments Add your comment
Dunwoody Mom
March 19th, 2010
4:18 pm
Good question Maureen. I have been asking that same question since I first read the article regarding Doughtery County’s frustration with getting the “evidence” from GOSA. Seems strange to me, to say the least, that the school systems are not getting any assistance, but yet are expected to pay for, with the investigations.
HOLY MOLLY
March 19th, 2010
4:43 pm
Sound like somebody has made a mistake and trying to CYA!I would look at the Governor’s office or the state superintendent office
Learning all the wrong stuff
March 19th, 2010
5:22 pm
Herb, Herb, Herb, don’t you think that they’ve used other forensic methods on this evidence by now? They probably HAVE fingerprint evidence, they probably have ALREADY tested for similarities or other characteristics of the eraser residue. It doesn’t take NCIS or CSI to come up with these techniques.
Garry Owen
March 19th, 2010
5:34 pm
Rep Bishop says he will vote “yes” on health care, as he read the final version?! Reports indicate the House version is very fluid and continues to change to satisfy House members and get their votes! Shame on Rep Bishop if he thinks he has read the final version.
Garry Owen
March 19th, 2010
5:47 pm
Before taking schools to task for the CRCT scandal go volunteer to be a monitor during the state testing period. Watch for students who skip a number on the answer sheet, see thier mistake, go back and erase many answers to get back on track. It happens. If the student does not catch their mistake many answers will be wrong because they do not match the question number in the test booklet -even though the students answer is correct, just on the wrong number on the answer sheet. Also, how many of you have volunteered in a First or Second Grade classroom? Go do this for more than several days and you will have a much better understanding and appreciation of the teacher in that clasroom and the dedication she/he has for her/his students.
Alex
March 19th, 2010
5:49 pm
The Office of Student Achievement is a joke. Like the rest of the Governor’s Office, it is staffed by a bunch on campaign workers and Sonny worshipers who know absolutely nothing. Kathleen Mathers, Jennifer Rippner, Martha Reichrath…..all OSA directors more interested in making a name than improving education.
Lynn43
March 19th, 2010
5:51 pm
Garry, This is about the CRCT and schools. Get on the right page for your complaints.
Learning all the wrong stuff
March 19th, 2010
5:53 pm
No doubt in my mind that of every gov’t dollar spent, 20% is WASTE and 30% is FRAUD. All gov’t, minus the possible exception of the military, should be cut by at least half NOW.
Central Limit Theorem and small samples
March 19th, 2010
5:57 pm
Is this erasure analysis even valid for classes smaller than 30 students? This analysis is based on the Central Limit Theorem that works well with large enough samples. Along with its findings, OSA stated that “the normal distribution holds only for large classes”, although they fail for declare what is the class size limit for which this theorem works.
What is considered a large enough sample? In statistics that number is 30 or greater. What happens when you have smaller samples, such as homerooms of 5 to 10 students who were tested on the CRCT? The probability that such a sample is even 3 standard deviations above the mean needs to be adjusted because small samples are not necessarily normally distributed. For small sample sizes, the Central Limit Theorem may give a poor approximation, resulting in confidence intervals that are misleading. Adjustments in computing probabilities for small sample sizes need to be made. Statisticians use instead a more conservative distribution called the T-distribution that uses the sample deviation of the sample instead of that of the population.
This has not been done in this erasure analysis. Lots of classrooms that were very small in size during testing could have been unnecessarily flagged. Nobody seems to notice or care about this fact. Unfortunely, lots of special ed. or ESOL classes were small in size, and their erasure averages were analyzed with a theorem that does not even work for such small class sizes.
Learning all the wrong stuff
March 19th, 2010
5:58 pm
Lynn43, what Bishop AND the CRCT alleged cheaters are doing IS THE SAME THING, from the same rotted roots.
Ed Johnson
March 19th, 2010
6:23 pm
Visual examination of properly selected, stratified by grade, random samples of answer sheets will suffice to confirm or deny GOSA’s facts. This will be especially easy with APS, given its extraordinary numbers of erasures — so many that quibbling over right-to-wrong and wrong-to-wrong erasures is pointless. So, Blue Ribbon Commission and Caveon, get samples from GOSA, already; no need to travel. And investigating testing protocol and trying to come up with recommendations about that will contribute nothing to establishing veracity of GOSA’s wrong-to-right facts. Too, looking at testing protocal and such are but distractions, as Mathers suggests. If the APS school board is incapable to make recommendations after the findings come in, they should step down.
@ Gary Owen
March 19th, 2010
6:58 pm
Only certified teachers can proctor the CRCT. You have to be certified to escort a child to the bathroom!
Ed Johnson
March 19th, 2010
7:29 pm
@Central Limit Theorem and small samples said: “Statisticians use instead a more conservative distribution called the T-distribution that uses the sample deviation of the sample instead of that of the population. This has not been done in this erasure analysis. Lots of classrooms that were very small in size during testing could have been unnecessarily flagged.”
Indeed, by putting GOSA’s statewide, classroom-level wrong-to-right data by subject and grade through the computations reqired of a particular kind of “process behavior chart,” about ten percent fewer classrooms were detected than GOSA flagged. Still, for APS it makes no material difference; APS lights up like the preverbal Christmas tree in the dead of night.
The process behavior chart works no matter how the data are distributed, and creating and using one does not depend on the central limit theorem, probability, T-tests, tests of hypothesis, standard deviation, etc. They work by computing upper and lower control limits from the data themselves, so as to characterize the behavior of the process from which the data come. Is the process stable, unstable, on the brink of chaos, or what? The data say APS is deep in a state of chaos.
A point either above the upper control limit or below the lower control limit signals something special or unusual is going on that’s likely worth investigating. A point within the upper and lower control limits signals just random variation due to common causes and nothing special or unusual is going on, no matter how great or small the variation. In this case, it will likely prove economically wasteful to single out any one point for special treatment, blame or investigation.
Importantly, process behavior charts take a holistic view, so get rid of ranking and answers the question, when are two different numbers the same? It’s a paradigm shift, and such odd questioning can be ruinous.
Still, even elementary kids are known to make and use process behavior charts to help them improve and be accountable (hate that word) for their learning. Now how about that? It only takes simple arithmetic and sometimes it takes just the “eyeball test” and no computation at all.
Ed Johnson
March 19th, 2010
7:35 pm
Maureen, the filter, please…
sped teacher
March 19th, 2010
7:36 pm
To eliminate any possible cheating and guarantee clean data (is there really such a thing?) a third party who has no stake in the outcome should administer the test. There should be trained personnel contracted by the company that created the CRCT to consistently administer the test at each school. If teachers are being furloughed for three to five days, let us have that time away from the school so there can be no question about the results, and let someone else foot the bill for administering the test. Frankly, it is quite sad that everyone teaches to the test now. What happened to good teaching, giving students time to MASTER a concept then build skills, not just EXPOSE them to a concept so they can pass a test.
Teach
March 19th, 2010
8:05 pm
I know Herb Garrett professionally and hold great respect for the work he does on behalf of Georgia education. His comments here are completely on target. There is a power struggle going on here and we all know the allure of power. Having administered the CRCT test (and equivalents) for decades I know that 1) teachers have very little time with the tests 2) students are encouraged to eliminate impossible answers, a practice that can create “stray” marks needing to be erased by students later 3) young children make many more stray marks 4) this is a mess brought on by the desire for accountability (do we get what we pay for, what our kids deserve), a fair goal without a fair method of implementation. And some want to pay me based on this mess?
Jennifer
March 19th, 2010
8:16 pm
@ @Garry Owen- not true. You have to be a certified teacher to ADMINISTER the test, not to proctor it.
Jim
March 19th, 2010
8:30 pm
I had the pleasure of working as a principal under Herb Garrett when he was supt. in Henry Co. He is a totally fair and honest man who does not suffer fools or cheaters at all. His analysis is “spot on”. Stop playing politics Sonny and lets get the facts out and move on. The roadblocks are not necessary. First and second graders do in fact write all over their tests and it is also true that those grades do not count toward AYP in any fashion.
Thanks for the timely info.
Tony
March 19th, 2010
8:32 pm
Thank you to Herb Garrett for speaking up on this matter. Ed Johnson is partially correct in his assertion that a random sample should yield some supporting evidence, however the state has analyzed the erasures at the classroom level. To pull random answer sheets from throughout the school system will not yield the same results. Next, Central Limit Theorem poster is also correct. When analyzing at the classroom level where there are fewer than 30 students it is difficult to apply this theorem to all classrooms. For instance, many of the classrooms that were flagged had fewer than 20 students and the standards deviation that was used was based on the statewide average. There is a huge problem with that.
Next, by blocking access to the answer sheets, there will remain a mystery about the truth of the erasures. Without bringing these answer sheets into the light of day, schools will be able to hide behind that mystery. Of course, the state gets to hide behind it, too. Everyone loses.
To truly test the claims made by the testing company and the Governor’s Office of Student Achievement, someone needs to be allowed to check the documents by hand, tabulate the erasures, and check the machine count. Until that is done, the mystery will remain and this problem will not be resolved. The state only obfuscates the truth by preventing the school systems’ access to the answer documents.
One last remark to @Gary Owen – the proctor does not have to be a certified person. The test adminstrator must be, but the proctor can be a parent volunteer. Some school districts may have differing rules for who can be a proctor, but the limit is not as severe as you suggest.
Jim
March 19th, 2010
8:33 pm
Worked with Herb in Henry co. Mr. Garrett is a great educator and does not suffer fools or cheaters at all. His analysis is “spot on” regarding the first and second graders. Scores do not count at all toward AYP and those kids do write all over their tests. The culture of first and second grade teachers is also no cheating and neatness counts! Most teachers, especially at that level of education, are not about to let messy work go anywhere.
Thanks for the info. here.
Tony
March 19th, 2010
8:39 pm
In the article by Torres, Kathleen Mathers is quoted as saying that schools need to investigate chain-of-evidence rather than number crunching. She is wrong. The schools systems must be allowed to crunch the numbers related to their answer sheets in order to get a better understanding of the depth of the problem they face. By hiding that information, the state will prevent systems from truly understanding where there are real problems. Yes, procedures must be evaluated as well, but attempting to direct everyone away from the numbers only adds to the mistrust.
Reality
March 19th, 2010
9:09 pm
One can make excuses for erasures…. but come on – statistically there is no feasible excuse for that MANY erasures. And, therein lies the problem.
If you can casually speak to a teacher in either Atlanta or Dougherty County where they can be honest – you will find the answer. The teachers know how much their students know and they also will tell you the likelihood of those students passing any standardized test without cheating.
I have relatives that teach in both systems. The answer is yes – there is a high likelihood that cheating occurred. The students in those systems don’t know the content well enough to score as high as they did (or have in recent years). The teachers know this.
But who ever asks the teachers?
why not
March 19th, 2010
9:20 pm
What happen to the blog about changing graduation requirements? Guess that the feedback wasn’t as positive. Graduation rates will fall way back in two years when these “everybody must be ready to go to a 4 year college” requirements first class are seniors. Well, we hope they are seniors. A lot of money is being spent on these new requirements that could be spent much wiser to benefit all students.
why not
March 19th, 2010
9:24 pm
New rules this year for testing. Teachers are assumed to be cheaters and will need to watched to prove they are not cheating.
Wonderful! A few bad apples now lets spend more money to make this system worse in order to benefit no one. Spend, Spend, Spend, Test, Test, Test!
Oh, this is another way to make teachers feel awful about the profession.
The beatings will continue until moral improves!
Pompano
March 19th, 2010
9:51 pm
Reality – good comment. They also appear to be ignoring the statistically impossible gains made by many of these schools as well.
Cut to the chase
March 19th, 2010
10:24 pm
You can quote of the figures lie and liars figure stats you want, and it still boils down to this: if it looks like she-yat, and it smells like she-yat, and there’s she-yat on the eraser, it’s probably she-yat.
Especially when someone tries to tell you it’s potpourri.
Off topic but,
March 19th, 2010
10:28 pm
@ sped teacher – I wish you wouldn’t use that term. I’ve heard teachers refer to their students as “SPEDs”. I find it offensive to call special education students SPEDs. It’s so condescending and disrespectful. Please consider their feelings and the feelings of their parents.
And, we all know that the only way true cheating can occur on a large scale is in the principal’s office.
catlady
March 19th, 2010
10:59 pm
Sounds like a case of the Hanging Chads to me.
Yes, little kids do lots of stray marks, but how many of those stray marks change an answer from wrong to right? Must be some heck of a great smiley face to change wrongs to right in statistically impossible degrees!
Perhaps a federal judge could babysit the scoresheets while they are being examined?
And on why first and second graders work might have been tampered with: Don’t you think it would be crazy, if you wanted to cheat, if you only changed answers for 3rd and 5th graders? I mean, wouldn’t that be a “tell?” Wouldn’t you want it to appear that the incredible gains happened over time, rather than only in 2 grades? I mean, if your special programs were really working.
Wish they’d get this thing taken care of. Test the kids who made the spectacular gains last yea,r with special monitoring, and see if their gains hold up. That should be what we call a “clue.” Let the GBI hold the tests when not in the classroom, and take them immediately upon completing the week’s testing. Let the APs on cleanup crew do the cleaning up at GBI headquarters under supervision. Let’s see how these same kids do this year.
ScienceTeacher671
March 19th, 2010
11:16 pm
@Off topic but, if you’ll look at the certificate of a special education teacher, no matter how many subjects that teacher is certified to teach, each subject is prefaced with “Sp Ed”….so maybe you should blame the state….
catlady
March 19th, 2010
11:24 pm
The filter is working overtime tonight. “Please release me, let me go!” (old country song)
What roadblocks have been thrown up? GOSA wants maximum security when the items are examined. That does not seen unwarranted, given that the allegations are serious.
While little kids mark on their test papers a lot, what are the chances that they would have smiley faces in the right place to turn wrong answers to right answers? That dog don’t hunt.
And why would any administrator want ONLY the 3rd and 5th graders to stand out as exceptional? Wouldn’t that look too obvious? They want to show that the whole school is benefiting from these miracle programs instituted by the miracle superintendent.
Let’s shorten this drama: put the principals, APs, and testing coordinators on the rack. I am sure some of them would squeal pretty quickly.
Ed Johnson
March 19th, 2010
11:36 pm
The control chart works with as few as three data to detect a chaotic, unstable process (but not to detect a predictable, stable process). The Central Limit Theorem is irrelevant re control charts.
Slade
March 20th, 2010
2:04 am
One thing I don’t think was mentioned here is filing a suit or filing a Federal Free Information Act against the State. I am unsure of the technicalities of this thought, but it is that, a thought. A simple subpoena for the test answer sheets will answer many questions. I agree with the Central Limit Theorem topic and know that it is an effective process, however we are dealing with apparently an incompetent division of the state where accountability comes from the next level and on down (they are excluded). It is my belief that they went about the situation however they wanted to go to make the ones look bad that needed to look bad and the ones look good who needed to look good.
Suggestions for future testing would be let the kids use golf pencils with no erasers. Then let the Governor and his staff personally make the corrections of stray marks on every test on their own dime not mine. I would vouch to say if we pulled previos years of tests and reviewed them we would most likely find them same issue that is just now being brought out now for political gain. I am not anti-government, back stabbers, or thief’s in the name of my child’s education. I want what is right to be done.
Teaching in FL is worse
March 20th, 2010
8:29 am
SpED teacher- you know better than that! While you idea has merit, if you have some of the students I have, you know some kids need to test with a familiar teacher as an accomodation. If there is a proctor with you, there is less chance for suspicion.
Teacher
March 20th, 2010
9:43 am
This is just part of the theatre brought to GA education to keep the citizens from facing the real problems and throwing every adminstrator and politician out. Until GA quits buying the garbage that flows from Glenn Brock and his bunch of banker/lawyer/developer/politician/Republican friends; there will never be a true effort at educating students in GA. The whole episode with Pope and DeKalb county is from Brock. This is exactly how they took over Fulton County. Walker and the Chairmoan of DBOE are the inside people. This is also the same people and similar methods that were used to destroy Clayton. There are numerous people that should be ashamed of themselves. Wake up GA.
bart
March 20th, 2010
9:55 am
Both the Governor’s office and Superintendent’s office are bad jokes. Their level of incompetence is staggering.
Dr. Phil
March 20th, 2010
10:22 am
Under Sonny Perdue’s “leadership” it has been common practice for instances of wrongdoing to be “investigated” by the State Attorney General, who has proven to be a lackey and “washing machine” for Sonny’s administration. The State AG typically allows issues to die in his office without written comment. This was the case with several charges against Michael Adams at UGA and a number of other matters that would be embarrassing to Sonny and his menions in state government. This is probably the case with the cheating issue.
M2teacher
March 20th, 2010
10:28 am
What about that wonderful thing call “No Child left Behind”? Maybe Sonny is just trying to the wishes of His god the fed.
Amanda
March 20th, 2010
10:32 am
What are the qualifications of the personnel in the Governor’s Office of Student Achievement?
Who did the analysis that showed a relationship between erasures and tests? When you get the document from the testing company on the OSA web-site, it says in the conclusion that the study can’t show evidence of cheating- it could be other variables. What???? Our school systems are paying for investigations into something that was never substantiated in the first place??
Did the Governor’s staff at OSA not have the statistics and measurement backround in order to know that the first rule is to apply the question: Does this make sense?
We’re spending tax payers money so that the Governnor’s minions in the OSA office can embarrass our state in the view of the business community nationwide? How do we attract and retain business if the Governor makes the schools look like cheaters? And, when it the investigation results are all over with, will he and the OSA (and all of those who were so quick to rush and judge) look like idiots?
Jennifer
March 20th, 2010
10:35 am
I have never trusted GOSA’s response to the public. They were arrogant when it came to their own errors in negotiating alternative education facilities first into and then out of IE2 in Gwinnett. And the public in Gwinnett brought the errors to their attention. They were also way too confident in their RTI data which has frozen for 10 years the academic expectations in Gwinnett to a 2008 level for special education students. Not one time did they or Gwinnett publish data to support that move for IE2 – but they were happy to pronounce it like it came from the heavens at the state board meeting when IE2 was being voted on.
Nappytop
March 20th, 2010
10:46 am
They know that educators that have been erroneously accused will file a class action suit! LOL
M2teacher
March 20th, 2010
10:49 am
The teachers hands are tied. They are overloaded with useless paperwork and not allowed to have discipline in their class. If they say the wrong thing to a child they are heaped on by the parents. Back when I went to school the teachers ruled and if you didn’t do what they said you got punished.
ScienceTeacher671
March 20th, 2010
11:14 am
The state has been proclaiming that “Georgia will lead the nation in student achievement” and also that Georgia is making progress in achieving this goal.
If some or most of that progress turns out to be the result of cheating, it makes Kathy Cox & Co. look kind of bad, doesn’t it?
PappyHappy
March 20th, 2010
11:53 am
Wasn’t Kathy Cox the one who cautioned ‘not to jump to conclusions’? Those cheating trained first and second graders really pulled one on the teachers and administrators didn’t they? (Guess they were all trained in the pre-K years how to ‘beat the man’!!)
Education in this state is not going to change until we get some innovative management in the state’s head office — who will focus on K-12 with tough standards. This is pretty sad! Someone should pay!
tom kat
March 20th, 2010
12:16 pm
I beleive Herb knew of the cheating or should have known before it was made public.He is just trying to muddy the water and hoping he will not be fired.
kathy gump
March 20th, 2010
1:05 pm
@off topic, but FYI, isn’t the current PC term SWD’s (Students With Disabilities)? And calling these students ‘SPEDS’ must be local or at most regional…. never heard it before in other system’s terms of familiarity. Regardless, my students R O C K !!!
Principle
March 20th, 2010
1:28 pm
Let the fog machines blow. The apologists for the teachers will pump as much fog into the investigation as they can to undermine any conclusions before they are reached. Lets criminalize cheating on statewide tests so that the investigation will be done by law enforcement, not some panel picked by the school board.
The evidence was pretty clear as well as being circumstantial. More investigation is necessary but the data presents a picture of enormous concern. If the schools are to have a shred of credibility with the taxpayer, the investigation must be wide open and above reproach.
If the name fits
March 20th, 2010
1:30 pm
The only way you could believe cheating isn’t widespread in this state is that you would have to be smoking so much herb that you can’t see the eraser marks through the haze.
Purple haze, all in my eye
Lately things don’t seem the same
Scores look funny, but I don’t know why
Excuse me, while I tell a lie
catlady
March 20th, 2010
1:42 pm
RTI is a travesty. Ms. Downey, you should look into how it really works (how systems use it to deny services). It is being used to violate IDEA, which is illegal, and should be punished with heavy fines!
anon teacher
March 20th, 2010
5:04 pm
“RTI is a travesty. Ms. Downey, you should look into how it really works (how systems use it to deny services). It is being used to violate IDEA, which is illegal, and should be punished with heavy fines!”
I agree wholeheartedly with catlady. It is almost impossible to get a child the services he/she needs. Our system denied a child services, even though he had a doctor’s diagnosis! It took him 2 years through the RTI process to get services. That child suffered for two years. Also, the other children in his classes suffered because of his violent outbursts and physical aggression toward them.
It is SO frustrating!
ScienceTeacher671
March 20th, 2010
5:21 pm
I agree with catlady and anon teacher.