A primer on Erase to the Top by our expert

In reponse to comments on the blog from Tony and SCIFI about the small number of erasures that could land a school on the state’s “trouble” list, I asked John Perry, the AJC’s database expert, to reply. (I sent him the two comments at the end of this entry as a guide on what to address.)

Here is what he sent me:

The criteria for flagging a class as unusual was created by comparing the average number of wrong-to-right erasures for a classroom on a particular test with the actual numbers of erasures statewide for the same grade and test. The statistical analysis was designed by test publisher CBT/McGraw Hill to identify classrooms with too many erasures to be explained as random occurrence.

It was base on the central limit theorem in probability, which says that if you take a bunch of samples, and if those samples are random, then the averages for those samples should be distributed in a very particular way around the actual overall average. This is called a normal distribution, and it is the same idea that gives us the margin or error for survey results.

The analysis also considered the number of students in a class. Smaller classes, just like smaller survey samples, will naturally have more variability, Larger classes should have averages that trend closer to the state average. Statisticians call this the rule of large numbers. So to avoid unfairly flagging schools with smaller class sizes, the analysis adjusted the criteria for class size differences.

Then to allow for a few unusual but innocent circumstances, such as students accidently answering questions out of sequence then erasing to correct their mistakes, the state only looked at schools with more than 5 percent of classrooms flagged.

So while 2.5 erasures per students – in a classroom – that would be 50 erasures in a class of 20 – may seems like a small number from a subjective point of view, it can be still highly unusual when compared with the number of erasures found in other similar classrooms across the state. And it takes more than one unusual classroom to place a school on the concern list. Of all the tests taken in all the classrooms statewide, about 4 percent (5,505) were flagged as unusual. About 42 percent of schools had no classrooms flagged, and 80 percent had 5 percent or fewer flagged.

John Perry, database specialist, the Atlanta Journal-Constitution

Comment 1 from SCIFI:

I guess kids today can only erase 1.8 answers. Look at he facts the flagged classrooms indicated if 29 students were tested the average class had 1.64 more erasures than the alloted 1.8 for the state. So Inquess because the average erasures and changes were 4.4 instead of 1.8 the kids are stupid and do not have the right to rethink and change an answer. The report makes it appear thatt 20 answers were changed for every child insted of 4. What is up with that statistic. Research shows that some kids erase more than average. I quess they are cheating huh!

Comment 2 from Tony:

Since there has not been an answer to my earlier query, I’ll go ahead and post the answer. To get flagged, it only took a class average erasure count of 2.50 (give or take a couple of 100ths). This is the magic number that is 3 standard deviations above the mean. Amazingly low, don’t you think?

Now before some of you go off half-cocked on me, I do not condone cheating and I think our ethics rules can be used to deal with the problems effectively. There were some rather egregious counts of erasures in some places and there is probably something fishy about those incidents. However, I’m beginning to see and hear about schools being put on the list where the erasure counts are quite low.

59 comments Add your comment

flagged classroom

February 17th, 2010
4:46 pm

My class has been flagged and I didn’t do it. Being a non-AYP grade level, I don’t believe admins did it either. So now what? does my 21 year teaching career end here? I don’t know what to think.

flagged classroom

February 17th, 2010
4:48 pm

To clarify, I meant I didn’t erase any student answers.

Attentive Parent

February 17th, 2010
5:05 pm

Maureen,

You may want to respond to this post today on the scandal at the Core Knowledge blog.

http://blog.coreknowledge.org/2010/02/17/to-catch-a-cheat/

Maureen Downey

February 17th, 2010
5:09 pm

Attentive, I am not sure why the posters to that site think the erasure scan didn’t find wrong to wrong. The GOSA stuff shows that they found both, and then examined the wrong to right.
Maureen

Attentive Parent

February 17th, 2010
5:33 pm

I do not know either but I’m sure Robert Pondiscio would love it if you would clear up the confusion.

He changed his initial reaction to the Seattle lawsuit once he became aware of the actual ruling and the factual evidence on record.

Just a head’s up.

john konop

February 17th, 2010
5:49 pm

What would be interesting to look at the concentration (50 erasures) levels of changed answers by student in the classrooms that are on the list. If the changes were focused on particular students in a class by classroom, and than compare the spike rate between normal classrooms by student (20 erasures). And if the students were targeted you could than find out if students with academic problem were the target because the changes would be focused on a fewer students and easy to spot skike rate.

Maureen Downey

February 17th, 2010
5:54 pm

Flagged, I sent your comment to John Perry and to GOSA. Haven’t heard back from GOSA, but this is from John:

Both the CBT statisticians in their written report and the GOSA officials have been careful to say that an unusual number of erasures on one test in one classroom is not a definitive indicator of cheating. It’s the accumulation of unusual cases at a particular school that begins to raise suspicions. If fewer than 11 percent of classrooms are flagged, then the state is not requiring an investigation, only more careful monitoring for the next round of testing. If flagged classes are over 5 percent. For schools where the state is requiring an investigation, it will be up to the local district how that takes place.

But...

February 17th, 2010
7:07 pm

If a class is flagged and no wrong doing was done, that teacher will still face needless scrutiny and suspicion. I feel for Flagged.

at this point in the game

February 17th, 2010
7:09 pm

I feel like this cheating scandal, while important and telling of the existing problems in some schools, has totally taken the focus off the dismal budget and the fact that there are teachers that will not be able to afford to teach in Georgia schools due to the furloughs, lack of supplies, etc. Let’s not forget the debilitating measures our legislature is going to take to support other, less valuable projects (Go Fish, anyone?)

Also, Maureen – thanks for providing profiles of the new gubernatorial candidates, how about profiles for the new state superintendents of education? THATS who really need to be talking right now!

Where is GAE?

February 17th, 2010
7:10 pm

Why is Arne Duncan allowed to go to the national media and say that it’s teacher, not administrators but teachers, who are cheating, and GAE or NEA won’t demand a retraction?

How can anybody believe anything Arne Duncan says again when he’s covering for administrators, when administrators have already been sanctioned?

Dr. John Trotter

February 17th, 2010
7:11 pm

Teachers Teach. Administrators Cheat.

By John R. Alston Trotter, EdD, JD, http://www.theteachersadvocate.com

I wish that the Governor and the General Assembly would balance the budget by chopping away at the administrative bloat in public education in Georgia. We could get rid of one-half of the useless administrators in the State, and the school systems would get along just fine because so many of these administrators are worthless and counter-productive. They hinder learning, not facilitate it. Hey, I like this slogan on a good picket sign: “Teachers Teach. Administrators Cheat.” Or, “Teachers Teacher. Administrators Snoop.” Or, “Next Election: Teachers With Pitchforks!” Finally, “Let Teachers Evaluate Administrators.” Right now, bad and evil (yes, evil!) administrators can do a lot a damage in the educational process, including destroying and getting rid of good, dedicated, and effective teachers, but a good teacher has no recourse against an angry and abusive administrator. The Georgia Code permits the teacher evaluation of administrators but school boards and superintendents don’t want to know about the terrible administrators; they choose not to exercise this option. This “option” ought to be mandated by the State. You would see many administrators “get religion,” and the teachers would at least appreciate this small effort to mollify their situation in these very tough economic times.

If I were running for Governor, I think that I would tap into this huge frustration and try to actually do some things for teachers that cost the State virtually no money…mandate that school systems allow the teachers to evaluate the administrators and that the compilation of the scores be presented to the school board. Also, the State should just simply chop in half the administrative bloat so that teachers would not have to be furloughed. We have way too many useless, ineffective, and abusive administrators in Georgia. (c) MACE, February 17, 2010.

Where indeed is GAE?

February 17th, 2010
7:24 pm

If our Constitution has provided checks and balances, and it’s been good enough for our country for over 200 years, why can’t our admittedly broken educational system provide checks and balances to restore some integrity to our schools?

Once again MACE has offered a common sense effective way to empower teachers and put a check on administrative abuses-such as trying to pressure teachers to cheat-by suggesting we follow what state law allows when it comes to having staff evaluate administrators. Is GAE too scared to follow MACE’s lead, because they have to protect some of the same administrators who are engaging in the abuses?

Has GAE even gathered the intestinal fortitude to utter the word cheating, or are they kowtowing to Kathy Cox and spinelessly referring to it as irregularities, as if the problem is just a mere lack of fiber in the diet?

Scooter

February 17th, 2010
7:54 pm

Where’s the comment made by Arne Duncan that it was the teachers and not the administrators that were cheating?

It seems to me that if there are a lot of classrooms with high wrong-to-right erasures within one school, that would indicate a problem with an administrative person. You couldn’t get a bunch of teachers together and all agree to cheat and then keep their mouths shut. Or if they did cheat, then they would do so “quietly” and not yak about it. If that were so, test security was not followed . And if it wasn’t followed, then would the administration be hoping that teachers would cheat? It’s very typical to blame teachers for any problem in education, but it is indicative of a person who likes to spout off and not think.

What's does John Perry think?

February 17th, 2010
7:58 pm

Maureen, how ironic that John Perry used the term the law of large numbers as Dr. Trotter has been referencing this term for years to talk about the cheating.

Not that I expect you to put him on the spot, but I wonder what John Perry thinks is the most statistically likely reason for these erasures; kids who have been superbly trained, like a ninja, in the secret art of stealth erasing, or administrators who have been completely immersed in a culture of corruption so deep that they thought they were above the law?

Here's the scoop Scooter

February 17th, 2010
8:02 pm

The reporters name was Richard Fausett, and it was in the LA Times

In a phone interview Friday, Education Secretary Arne Duncan said it was likely that “a tiny, tiny number of teachers” changed scores. If they’re caught, he said, “you’ve got to get them out.”

Not one word about administrators in the article. Duncan clearly scapegoated teachers; did he compromise his integrity because he wants to use the supposed successes of APS and their leaders for political gain?

Why should teachers have any faith in anything Duncan says after such a vicious assault against teachers?

Dr. John Trotter

February 17th, 2010
8:08 pm

Yes, I’ve noted that a few of our phrases have been adopted. Law (or, rule) of large numbers, systematic cheating, etc. Anyone with a lick of common sense and who can pour p_ss out of a boot knows that students who come to your 4th grade classroom without the ability to read a sentence but whose scores were off the chart on the CRCT in the third grade did not truly earn those scores. This is what has been happening for years in Atlanta under Beverly Hall. I have been saying for years that Beverly Hall (and SACS’s Mark Elgart) are the two biggest phonies in public education in Georgia. I have always said that Crawford “High Octane” Lewis is just “a clown of a superintendent.” Anyone can go to our website and read what we have been saying for years at MACE. http://www.theteachersadvocate.com

Dr. John Trotter

February 17th, 2010
9:24 pm

Yes, I’ve noted that a few of our phrases have been adopted. Law (or, rule) of large numbers, systematic cheating, etc. Anyone with a lick of common sense and who can pour p_ss out of a boot knows that students who come to your 4th grade classroom without the ability to read a sentence but whose scores were off the chart on the CRCT in the third grade did not truly earn those scores. This is what has been happening for years in Atlanta under Beverly Hall. I have been saying for years that Beverly Hall (and SACS’s Mark Elgart) are the two biggest phonies in public education in Georgia. I have always said that Crawford “High Octane” Lewis is just “a clown of a superintendent.” Anyone can go to our website and read what we have been saying for years at MACE. http://www.theteachersadvocate.com

This is another attempt to post. What’s wrong witht he filter. We can’t mention “Mark Elgart” or something?

Maureen Downey

February 17th, 2010
9:25 pm

Take a look at the AJC tomorrow as Heather Vogell spoke to Duncan this week and the story should run tomorrow. Duncan is also speaking in Atlanta Friday.
Maureen

Lynn

February 17th, 2010
9:30 pm

Maureen

Where is Arne Duncan speaking on Friday?

Ed Johnson

February 17th, 2010
9:34 pm

tcherlady77

February 17th, 2010
9:36 pm

At my school for the first time we didn’t allow kids to do anything after the test – no books, no homework, nothing. Could that account for the fact that there more erasures..that the kids were actually following our advice and rechecking their work? Also, what’s up with the schools that have a 0%? No kids in any of those classes made changes on their CRCT?

Common cents

February 17th, 2010
10:00 pm

Why are students taking standardized tests on paper? I would be beneficial for the students to have the test administered on a computer as most graduate school tests are in that format.

Mandella

February 17th, 2010
10:01 pm

Maureen – From all of your and the AJC’s investigative reporting, why is it that no one is asking what was the rationale for looking at the test scores of so many GA students? I know that the quick, first answer will be because of the four schools that the AJC questioned last year; however, recall that those scores reflected a wide variation between the Spring scores and the Summer retakes, not a “wrong to right” analysis of Spring scores. Part of your title of this blog may begin to point you in the correct direction. Mr. Trotter, I think you should run for Governor – in Montana

Dr. John Trotter

February 17th, 2010
10:08 pm

Arne Duncan is also a joke. He has never taught school a day in his life. (We don’t count his momma’s after-school program.) Where do these bozos come from? It would be like Hewlett Packard (is this the correct spelling?) hiring me to run the computer company. Heck, I just learned how to cut and paste. Would I know the nuances and intricacies of running a computer company? Arne, Welcome to Atlanta! (You still, however, don’t know what you are talking about when it comes to the public schooling process. My name is “John Trotter,” and I challenge you to any debate on the public schooling process.)

Mandella

February 17th, 2010
10:17 pm

Maureen – This is the second attempt at this posting. Out of your and the AJC’s investigative reporting, where is the questioning regarding the rationale for this investigation? Of course, the quick response is that it was because of the four schools that were reporting on in the AJC last year; however, recall that those issues were related to Spring CRCTs against Summer retakes, not “wrong to right” erasures (this was just the piece that was used to validiated the AJC’s concerns). The state decided to look at all tests of Spring results only. Why? I believe that a part of your blog’s title might be a good starting point. Also, I think that it would be a great idea for Mr. Trotter to run for Governor – of Montana

Here's the scoop Scooter

February 17th, 2010
10:23 pm

I certainly hope that, as knowledgeable on the story as Heather Vogell is, she called Arne Duncan on it if he tried to scapegoat teachers for cheating, as he did in the LA Times.

Tony

February 17th, 2010
10:37 pm

I am fully aware of central tendencies, the statistical calculations utilized in the analysis, the methodology, etc. I have taken masters level stats classes. It is still difficult for me to stomach the fact that 2.5 average erasures was sufficient cause to flag a class.

Common cents – for students to takes tests on computers would require schools to have sufficient computers in the first place. There is no reliable funding mechanism for computer replacement.

And whether coincidence or planned, the release of this report and the attention it’s getting is certainly distracting people away from the budget issues facing our state and the short-changing the teachers are about to experience.

Bottom line

February 17th, 2010
10:46 pm

Tony, just because some houses were left standing didn’t mean Haiti didn’t have an earthquake!

Get Real

February 17th, 2010
11:08 pm

F.Y.I. APS has stated through faculty meetings (in which all staff members were required to sign-in) that under no circumstances were staff members supposed to speak with Heather Vogell. If she contacted someone then we are to give her the APS publicist’s #. We were also told to call the school and let them know about her inquiries.

Legal beagles and newshounds

February 17th, 2010
11:15 pm

Does anybody know if a school system can legally mandate that its employees not talk to reporters?

If there is proof that a school system did this, wouldn’t that allow, as far as journalistic ethics goes, the AJC to have more leeway when using off the record sources as far as follow up reporting goes?

GAE and PAGE joint press conference

February 17th, 2010
11:43 pm

There are reports that there is a GAE and PAGE joint press conference tomorrow. The word is that they didn’t wanted to speak out on recent events until they had gathered more facts.

But now they are ready to come out with a joint statement, and go on record that Truman defeated Dewey.

Fact

February 18th, 2010
2:50 am

Preach on Dr. Trotter!! I sure do wish I could sit down and tell you about the FACTS that I know about bad and evil (YES EVIL) administrators. Liers, Cheaters, swingers, arrogant ________!!!
Believe that my story will be public very soon! I can’t wait to see his/their mug shots!!!!

APS Employee

February 18th, 2010
6:12 am

Get Real – Really?! I’m out on leave right now and have no clue what’s going on at my school. I’ve been wondering if our principal had toned down the “get those scores up or you’re fired” routine.
I DID call Heather and I’d love for one of those jack —es to try and do something about it. I’m no lawyer, but I know that’s not right. In fact, I think all of us should call Heather. She said she needs more people to call her if she’s going to use anonymous sources.
That makes me furious! I personally know of several administrators who I can absolutely see holding a faculty meeting and giving a big fire and brimstone sermon (they are soooo good at that, must be part of Bev Hall’s training program) and making all the little kids, I mean teachers sign the paper.
Puh-lease.

gwinnett educator

February 18th, 2010
6:14 am

There will be a rally at the state capitol this Saturday.
RALLY AT THE STATE CAPITOL
FEBRUARY 20TH, HIGH NOON
SPREAD THE WORD!!!

APS Employee

February 18th, 2010
6:17 am

Rally for what, Gwinnett?

APS Employee

February 18th, 2010
6:20 am

The filter aspect of this blog is quite annoying.

gwinnett educator

February 18th, 2010
6:34 am

To stop furloughs and to fully fund schools. They had a rally last month (if I am not mistaken) that many people did not know about. I received the flier/email yesterday from a coworker. This is a copy and paste of some of the info…
STOP THE FURLOUGHS
FULLY FUND SCHOOLS
Stop Cutting & Bleeding Our Children ‐ Fully Fund Public Schools
• Only two states furloughed teacher in 2009. Hawaii (the only state‐wide
school district in the U.S.) and Georgia!
• Over the past 7 years, more than $3.5 billion in direct instructional funding
has been eliminated by state leaders.
• $90 million, money from the initial phase of the stimulus package is still
available for use in FY ‘10, but is being deferred to FY ‘11. These funds are
enough to avoid adding 3 more furlough days to the current school year.

While the state is financing $25 million Go‐Fish Ponds and $9 million dollar horse
stables (in the Governor’s home county of Houston) Peach County Schools’ have
had to resort to a 4 day work week.

Been There. . . Done, well. . . just done!

February 18th, 2010
9:10 am

Maureen & other bloggers, if you have a time and an area near the Capitol where the rally will start Saturday, many other readers would be interested in attending. Also, bloggers, if someone would be able to find the link for the joint GAE-PAGE news conference, that would also be a help. Maureen, perhaps their response(s) would be a good topic for one of your daily columns – tomorrow, maybe?

Attentive Parent

February 18th, 2010
9:26 am

Maureen

Can you get me out of the filter on the Parent Math Letter posting? That’s still an active discussion and disinformation always needs a prompt response.

Thanks.

James

February 18th, 2010
12:33 pm

@Legal beagles – of course a school system can mandate that employees not talk to reporters. It’s almost routinely done in most any company and government job that you should not talk to media about the company unless you have specific permission.

James

February 18th, 2010
12:36 pm

@at this point in the game – If there are teachers that can no longer afford to teach in Georgia because of pay then they can surely go to one of the other 30+ states that pay *LESS* than Georgia. We are in the top 18 for both starting teacher salary and average teacher salary.

http://teacherportal.com/teacher-salaries-by-state

James

February 18th, 2010
12:50 pm

@gwinnett educator – I don’t believe that GA and Hawaii are the only places doing furloughs. Rhode Island, New Mexico, Florida and California all have districts doing furloughs as well. And stop the drama please – “cutting and bleeding our children”?

If people really want to cut spending then cut the whole Nanny-State spending budget. Cut the entire school transportation budget – there’s no reason that parents can’t take their kids to school and it doesn’t need to be the state’s job to do this. Also cut the entire school lunch budget. Even poor people can send a sandwich with their kids for lunch. Cut all after-school sports; leave PE and during school athletics. We don’t need to be paying for football, coaches, and fields. Cut those things and we’ll get a TON of money back and not need furlough days at all.

gwinnett educator

February 18th, 2010
1:44 pm

@James…I said it was a COPY and PASTE of the email that was sent to me. Please go back and read and comprehend what was said. Those are NOT my words…

Again, I will COPY and PASTE for you to soak in…
***This is a copy and paste of some of the info…
STOP THE FURLOUGHS
FULLY FUND SCHOOLS
Stop Cutting & Bleeding Our Children ‐ Fully Fund Public Schools***

NOT MY WORDS..but the words of the information that was included in the email and flier attachment.

(lunch break)

gwinnett educator

February 18th, 2010
1:46 pm

Maureen..my response to James is caught in the filter.

James…I said that those were NOT my words so there is no drama to stop. If you go back and READ and let it digest what was typed earlier…I said that it was a COPY and PASTE of some information that was in the email and flier attachment.

APS Employee

February 18th, 2010
2:35 pm

James, APS is not a corporation. It is a government agency that is funded by tax payers. I live in Atlanta. I am a tax payer. Just because I work there makes absolutely no difference in whether or not I can talk to whoever I darn well please about what goes on there.
It’s just more APS b.s. of threatening staff. It’s obvious they think very little of their employees. Who but an idiot would believe they cannot speak to the newspaper?

Joy in Teaching

February 18th, 2010
2:50 pm

There is also a scheduled PAGE rally on Tuesday, February 23 at the Capital. I am taking a personal day in order to go with some friends of mine. I’ll be the one dressed in black.

Tony

February 18th, 2010
4:56 pm

Bottom Line – and just because the seismograph registered vibrations does not mean there was an earthquake. It takes multiple sources of information to confirm what data are telling us. This report creates a whole lot of circumstantial evidence and probably points to some areas that must be addressed, but it does not provide substantial proof that can stand alone.

ScienceTeacher671

February 18th, 2010
10:31 pm

Late breaking: PAGE rally postponed til the General Assembly is back in session.

[...] Atlanta Journal and Constitution eduscribe Maureen Downey has dubbed the standardized test cheating scandal in Georgia “Erase to the [...]

fbguy

February 20th, 2010
1:01 pm

I’m not from your state, but have a question that doesn’t seem to be addressed anywhere here. How do they distinguish between the different kinds of erasures (i.e. wrong-to-wrong, wrong-to-right, right-to-wrong)? To all in GA, good luck.