In reponse to comments on the blog from Tony and SCIFI about the small number of erasures that could land a school on the state’s “trouble” list, I asked John Perry, the AJC’s database expert, to reply. (I sent him the two comments at the end of this entry as a guide on what to address.)
Here is what he sent me:
The criteria for flagging a class as unusual was created by comparing the average number of wrong-to-right erasures for a classroom on a particular test with the actual numbers of erasures statewide for the same grade and test. The statistical analysis was designed by test publisher CBT/McGraw Hill to identify classrooms with too many erasures to be explained as random occurrence.
It was base on the central limit theorem in probability, which says that if you take a bunch of samples, and if those samples are random, then the averages for those samples should be distributed in a very particular way around the actual overall average. This is called a normal distribution, and it is the same idea that gives us the margin or error for survey results.
The analysis also considered the number of students in a class. Smaller classes, just like smaller survey samples, will naturally have more variability, Larger classes should have averages that trend closer to the state average. Statisticians call this the rule of large numbers. So to avoid unfairly flagging schools with smaller class sizes, the analysis adjusted the criteria for class size differences.
Then to allow for a few unusual but innocent circumstances, such as students accidently answering questions out of sequence then erasing to correct their mistakes, the state only looked at schools with more than 5 percent of classrooms flagged.
So while 2.5 erasures per students – in a classroom – that would be 50 erasures in a class of 20 – may seems like a small number from a subjective point of view, it can be still highly unusual when compared with the number of erasures found in other similar classrooms across the state. And it takes more than one unusual classroom to place a school on the concern list. Of all the tests taken in all the classrooms statewide, about 4 percent (5,505) were flagged as unusual. About 42 percent of schools had no classrooms flagged, and 80 percent had 5 percent or fewer flagged.
John Perry, database specialist, the Atlanta Journal-Constitution
Comment 1 from SCIFI:
I guess kids today can only erase 1.8 answers. Look at he facts the flagged classrooms indicated if 29 students were tested the average class had 1.64 more erasures than the alloted 1.8 for the state. So Inquess because the average erasures and changes were 4.4 instead of 1.8 the kids are stupid and do not have the right to rethink and change an answer. The report makes it appear thatt 20 answers were changed for every child insted of 4. What is up with that statistic. Research shows that some kids erase more than average. I quess they are cheating huh!
Comment 2 from Tony:
Since there has not been an answer to my earlier query, I’ll go ahead and post the answer. To get flagged, it only took a class average erasure count of 2.50 (give or take a couple of 100ths). This is the magic number that is 3 standard deviations above the mean. Amazingly low, don’t you think?
Now before some of you go off half-cocked on me, I do not condone cheating and I think our ethics rules can be used to deal with the problems effectively. There were some rather egregious counts of erasures in some places and there is probably something fishy about those incidents. However, I’m beginning to see and hear about schools being put on the list where the erasure counts are quite low.