Higher CRCT failure rates should erase doubts about whether test sheet tampering occurred last year

By using open records to obtain CRCT scores from the major metro districts and diving into the data, the AJC investigative team examined how schools under suspicion for test tampering in 2009 fared this year when testing protocols were tightened and answer sheets were guarded like gold bullion. (You can see the results for yourselves here as we have just posted them.)

Not very well, it turns out.

The newspaper’s analysis by AJC reporters John Perry and Heather Vogell found that score drops in the 39 “severe” Atlanta schools drove up the district’s overall failure rate in reading, English and math.

For all of you who have insisted that this was a witch hunt, consider this: Most schools that did not have unusual erasure patterns showed a slight gain in their CRCT scores. Is this also random?

Those schools with suspicious erasure rates last year and jump in failures this year ought to be doing some soul searching. And Superintendent Beverly Hall ought to searching for the reasons for the plummet in scores. And for the people responsible for any tampering.

Yes, there are fluctuations in test scores year to year. But abnormal fluctuations can’t be ignored in light of the state erasure analysis.

While the state won’t release final school results until mid July, the AJC used the state’s Open Records Act to obtain preliminary scores for nearly 500 schools in the six biggest metro districts. In general, the scores showed that students at schools where state officials said cheating concerns were “severe” suffered average score drops of as much as 11 points, while students at schools considered “clear” made slight gains.

Responding to evidence of cheating uncovered by an AJC investigation in 2008, the state reviewed every 2009 CRCT answer sheet to measure how often kids changed wrong answers to right by virtue of erasures on the sheets. Because every test sheet was checked, the state was able to develop a reliable index of how often test answers were changed from wrong to right and flag schools that had inordinate occurrences of answer changes, right down to the classroom level.

The worst incidents occurred in Atlanta.

For some APS  schools, the drop from last year was stark. At Gideons Elementary, 92 percent of fifth-graders passed math in 2009. This year, 39 percent did. And at Dunbar Elementary, about 87 percent of fourth-graders passed math last year, compared to 49 percent this spring.

In a statement prior to the AJC’s story, Hall acknowledged the scores dropped but maintained the results overall demonstrated the district’s “continued academic progress.”

I am assuming that she will come out with a stronger statement soon and that it will not gloss over the very real evidence of test tampering.

According to the AJC:

The scores showed that overall, students at schools where state officials found cheating concerns were “severe” suffered average score drops of as much as 11 points, while students at schools considered “clear” made slight gains.

Passing rates mirrored the sagging scores. In severe schools, for instance, the percent of students failing math rose roughly 12 percentage points. In cleared schools, the percent of failing students declined by 1 percentage point this year.

School districts should view stark score drops at severe schools as potential evidence of tampering, said Kathleen Mathers, executive director of the Governor’s Office of Student Achievement.

“Generally, in schools where there proves to be a dramatic drop, we have concerns about what would cause that drop,” she said. “If it happens to be a school that had a high number of answers changed last year from wrong to right, that could be an indication that there had been intentional wrongdoing.”

130 comments Add your comment

Atlanta mom

June 10th, 2010
7:57 am

Seems to me, the best way to look at this problem, is to compare last year’s 5th grade scores to this year’s 6th grade scores. I’ve never understood why we compare one group of 5th graders to a different group of 5th graders the following year.
That said, this is very disappointing. I had been holding out hope. Color me foolish.

Nicole

June 10th, 2010
8:09 am

I, for one, would like you to also post actual NUMBERS of students rather than just percentages. Many of these schools are extremely small. Much larger schools have astronomically higher numbers of erasures but lower “percentages.” It’s easy to say 50% of classrooms when there are only two teachers in a grade level. I’ll be looking forward to the numbers.

Dunwoody Mom

June 10th, 2010
8:11 am

I agree with Atlanta mom. In comparing same grades, different students, what are you really comparing? You cannot compare last year’s 3rd grade students, because 4th grade students do not take the CRCT. How do you compare last year’s 5th grade students? They are in middle school now and the scores would be intermingled with those of students from different elementary schools.

Gwinnett Parent

June 10th, 2010
8:13 am

Wow 39% failure rate in math and they are only in elementary school. These children don’t have a chance. How can parents trust public schools to educate their children when only 39% of them can pass a 5th grade math exam? No wonder there are so many parents like myself scrambling to buy workbooks and drill their children in academics over the summer. We know that in the fall it’s going to be one large grab bag and god only knows if our children will learn anything at all. Look at how much Sylvan has grown over the past decade. That alone should tell us how much of a disconnect there is in the classroom.

Gwinnett Parent

June 10th, 2010
8:14 am

Oops, I meant to say “39% passing rate”.

Maureen Downey

June 10th, 2010
8:19 am

@Dunwoody and Atlanta moms, I am driving to Savannah today for the Georgia School Boards Association, but will ask John Perry to address these issues either here on the blog or in an e-mail that I can post.
Also, the AJC’s information specialist Matt Dempsey just posted the CRCT results that we obtained through Open Records. Anyone who is interested can look at these 2010 results and then look at the 2009 results on the GOSA site

http://www.ajc.com/news/crct-scores-show-troubling-545144.html

Thanks, Maureen.

Rick

June 10th, 2010
8:22 am

Makes you wonder about the ethics of teachers and school administrators!

Dunwoody Mom

June 10th, 2010
8:28 am

Thanks, Maureen. I don’t mean to imply I don’t believe there was cheating, I will wait for the “experts” to weigh-in. Anyway, those scores are troubling, nonetheless.

APS Parent

June 10th, 2010
8:30 am

I would also like to note that the article didn’t mention that students were tested with different teachers. At my daughters school, she was tested by a Kindergarten teacher that let a child sleep during the test because she didn’t want to be accused of cheating.ALso your need to see how many of these kids were the same. Dunbar is not like Brandon, my daughters classmates are totally different this year than last year. There are more kids ( from 44 to 60) and more special ed. kids. Her grade didn’t drop from 92% to 39%. Her teacher said they were still in the 70’s.

Northview (Ex)Teacher

June 10th, 2010
8:31 am

Why would anyone be surprised to learn this? Education in Georgia is in shambles, and the repukes in the General Assembly have declared war on public schools. They make it impossible for schools and especially teachers to be successful and then blame them for failure.

Counting down the days!

Beverly Hall's "New" Record Deserves National Attention

June 10th, 2010
8:35 am

Of course there was cheating in many of the APS schools. The question is how so many schools decided to cheat at the same time unless there was pressure from Beverly Hall or her minions at the top. The hope is that someone caught cheating will try to cut a deal and come forward with the whole scenario. This is not over.

Maureen Downey

June 10th, 2010
8:36 am

APS Parent, I don’t think that we can cite these scores as proof that cheating occurred in every school. Nor do I think it did. And certainly I want to be clear that there are some high-achieving APS schools.
But here is what bothers me: The trendlines. The metro area schools that were found last year to have no evidence of tampering showed slight gains as a group on the 2010 CRCT.
But the schools that were on the severe list for signs of tampering posted a decline.
It is hard to ignore that.
Also, to be clear. Even looking at the severe list, not every classroom in all those schools were flagged.
Maureen

ADO

June 10th, 2010
8:41 am

You know, as much as I have come to dislike our governor and his handling of state government, “Sonny” should step up now and take over the APS System. The first thing he should do is “Fire” Beverly Hall……this will not cure what’s wrong with the system, but it’s a good first step…..she obviously is not up to the task for which she was hired.

APS Parent

June 10th, 2010
8:43 am

Yes, but as a parent it really bothered me that five pepople were in and out of my childs classroom during the test. I also was bothered that a teacher that she did not know tested them. I think a 5% or 10% drop under those conditions would be expected. I mean really? In the past schools only had 1 or two monitors, but 4 or 5 is excessive. My daughter complained that everytime she looked up someone was staring at them. Her school dropped about 10% but to me that looks like something you would expect to happen in a totally different testing environment than usual.

Vince

June 10th, 2010
8:43 am

Maureen…. The number of schools not making AYP will go up this year and every year due to the increase in the annual measurable objectives as NCLB requires us to have every student perfroming on grade level.

Having said that, thank you and the AJC staff very much for the handy data you put together. I have digested our numbers and made comparisons in many ways since receiving scores a few weeks ago, but your data was a new slant for me.

Maureen Downey

June 10th, 2010
8:48 am

@APS Parent. Not sure why your school had so much traffic during one testing session. Even if you had monitors, it would seem that five people were three too many.
My kids were in a “clear” school and did not report any unusual things during CRCT. (They hate CRCT testing weeks, but they didn’t tell me anything out of the ordinary happened this year.)
Maureen

APS

June 10th, 2010
8:53 am

As an educator in APS I plead with school board to move swiftly to remove permanently the teachers and administrators who are involved in cheating. This small percentage of employees is killing the rest of us who are busting our behinds to meet targets and goals through legitimate means. The pressure is incredible.

APS Teacher

June 10th, 2010
8:54 am

I agree to a point that yes it is clear that some cheating is obvious. I also agree with the APS parent. Changing the classroom teacher who is testing the students in the past was considered a testing irregularity. A new environment can give a student advantages and disadvantages during testing. I think the AJC is targting the innocent with the guilty. My school was severe, my class was flagged, my kids did really well this year…not a well as last year. APS has been corupt for decades, Beverly Hall didn’t make it that way, and Sonny is just as bad as she is. I think they should do away with “targets”. You are comparing apples to oranges. My last years class was smaller and more advance than this years class. So how do you expect me to make the same scores + better when you increase the number of PEC students in my class and give me students that have hopped around from school to school.

APS Parent

June 10th, 2010
9:02 am

Maureen,

My daughter’s school was on the “Severe” list. She said that there were four people that walked around that were not from the school. She said they either walked around the room, or stood in the door way and watched them. Her exact words were ” it was creepy”. I was told they were from the state and the district. It was excessive. Like I said, a 40% drop sounds a totally different alarm than 10% if those same kids pasted the previous year.

Just Saying...

June 10th, 2010
9:03 am

I’m with Atlanta Mom. With the exception of the dramatic drops, the -50% to –30%, you cannot compare the class of 2009 to the class of 2010. In order for me to make an informed decision, I would need to know the number of students that was taking the test this year compared to last year.

Class A with 200 students taking the exam could have the following passing percentage Math: 80%, Reading 90%, Science: 50%, Social Studies: 75%. This would mean that Class A had the following number of students to passed each exam: Math: 160, Reading 180, Science: 100, Social Studies: 150.

Class B with 300 students taking the exam could have the following passing percentage Math: 70%, Reading 80%, Science: 51%, Social Studies: 65%. This would mean that Class B had the following number of students to passed each exam: Math: 210, Reading 240, Science: 153, Social Studies: 195.

When looking at the percentages alone it would seem as if Class B is failing at a far greater rate, however when you look at the actual number of students taking the exam then you can see that Class B actually had more students to pass the exam. On the other hand, you also have a greater number of students failing the exam. Comparing Class A to Class B is like comparing apples to oranges, both are fruit and have seeds but those are the only similar qualities.

I would like to see a comparison of the number of students taking the exam in 2009 and 2010 with the percentage passing respectively.

Understanding Atlanta

June 10th, 2010
9:07 am

Test scores are not indicative of a teachers ability. It doesn’t matter how “bad” a school is, when parents are actively involved in their child’s education process, including helping with homework, meeting with teachers, ensuring the child has discipline. I would like to know in these schools with high failure rates, what does parent participation look like, in the school and at home.

I went to a school that was a “failing” school and I had great test scores because my parents made sure I did my homework, studied, and met with my teachers to discuss my performance. Teachers play a role, but if more parent’s would supplement the educational process these test scores would instantly improve

Editing the Editor

June 10th, 2010
9:24 am

Since you are writing about education, please allow a minor correction.

You wrote: “The worst incidences occurred in Atlanta.”

You probably meant to say “incidents” since the context of your article focuses on the number of occurrences. But giving you the benefit of the doubt, you may have meant to qualify the extent of those same occurrences, in which case “incidence” (singular) could have worked for you.

After all, the AJC certainly wants to avoid any “instance” (example) of a writing error, particularly when holding the APS to account – an endeavor which we enthusiastically support.

Beatrix Turr
Asst. Curator
The Society for the Preservation of the Mother Tongue

Maureen Downey

June 10th, 2010
9:29 am

@Editing, Duly noted and corrected. Thanks, Maureen

redweather

June 10th, 2010
9:36 am

Test scores not indicative of a teacher’s ability? So in your view there is no correlation between content based apptitude test scores and the instruction students receive? I don’t know. Granted, the test scores are not the only thing that should be evaluated in deciding whether a teacher is competent.

Dunwoody Mom

June 10th, 2010
9:40 am

Maureen, I noticed that the scores for Dunwoody Elementary were not included. Is there someone I can contact to get those?

justbrowsing

June 10th, 2010
9:43 am

While scores can vacillate from year to year- rarely does a 50 to 30 point drop happen after steady gains. Scores within 10 pts of last years scores are not a significant cause for concern in my opinion. Compelling drops in scores such as the aforementioned should remain the focus of investigation. Sad that teachers and administrators are pressured. Someone should conduct a work climate study in all metropolitan schools. Some experiences I have heard about border on horrific.

Would Be Nice

June 10th, 2010
9:43 am

I think that it should be noted if there was a change in performance if the school had a DIFFERENT principal in 2009 than in 2008 and earlier.

Shout

June 10th, 2010
9:44 am

Maureen would there be an investigation of the mis-use of federal funds by APS? E-rates, Bidding etc.

Maureen Downey

June 10th, 2010
9:45 am

@Just saying and @Dunwoody and Atlanta moms, (And to all interested in the AJC analysis methodology.)
I asked AJC investigative reporter Heather Vogell to read over these comments and she wants folks to know that John Perry, the AJC database expert, did address these points.
She notes:
1. John weighted the average according to the number of kids in the school.
2. John did look at it both from the cohort perspective (one year to the next) and the same grade-to-same-grade.
I expect to hear from John shortly and will post his response as well.
Maureen

Maureen Downey

June 10th, 2010
9:47 am

@Shout. That is ongoing. The paper is seeking more data and contracts through Open Records. I will keep you posted on that but I think there are more stories to come.
Maureen

justbrowsing

June 10th, 2010
9:49 am

@redweather- you are right- the instruction they are able to receive. How much can they receive if when it is provided there are blatant disciplinary infractions in the classroom and students are unwilling to be ACCOUNTABLE for attempting to learn. This is and will always be an issue. Don’t assume students are being denied anything or it correlates to teacher preparation. APS has some great teachers- I know how hard they work- and often under threats, intimidation, low parental support, and excessive brow-beating along with students who are not aware of the values of education. I do not know how they do it. Can anybody?

Maureen Downey

June 10th, 2010
9:50 am

@Dunwoody, Let me check on that. Heather and John used open records to get school-level scores. For some reason, the state doesn’t plan to release them to the media and public until mid July. (They said it is to give schools time to validate their scores, but the DOE released state-wide data based on those school scores yesterday so I am unsure why there is a delay. And the state DOE will release county data in two weeks, also based on those scores.)
Maureen

d

June 10th, 2010
9:51 am

I know Arne Duncan still wants to use the mantra “test, baby, test,” but I hope Congress hurries up and gets ESEA reauthorized and gets rid of the whole AYP concept before it further destroys our schools.

@ APS Mom

June 10th, 2010
9:53 am

I teach in Dekalb and every year, the county sends county officials to look in on the classes and my principal also looks in, EVERY day of testing. My kids know this before the test and I stress that they should focus on the task at hand. It does not negatively impact their test. I also spent considerable time at Dunbar years ago in a volunteer capacity. You seem to be very involved in your child’s education. If they are still implementing a certain reading reform, (DI) you should advocate for changing that. It is not an effective tool for teaching reading.

catlady

June 10th, 2010
9:58 am

As we noted yesterday, many of us teachers think this year’s test was markedly easier than last year’s version, making these schools’ drops even more egregious. The reading passages seemed simpler, the questions’ wording seemed easier, and the language used in the math problems seemed easier.

Of course, deny-ers (those who think this is a witch hunt) will also say 1)these are different students whose scores will differ from last year’s (true, there will be some difference, but statistically marked difference?), 2) the statistical manipulation may have “set up” these schools to fail (yet why would that not have happened across the state) 3) there was so much pressure NOT to erase that kids left wrong answers (kids given correct test taking tips do not change many answers. Instead, they are taught to carefully consider all the answers THE FIRST TIME), and 4) teachers were too distracted by last year’s troubles to teach their best this year (instead of that last year’s troubles would make them even more careful to drill the GPS).

Taxpayers should demand an independent audit of scores for the schools under suspicion. If a child did not pass in first, did not pass in second, and then, miraculously passed highly in third, but yet in 4th dropped to did not pass, that might be a “tell.” Especially if there is a pattern of classmates doing the same. Don’t taxpayers, the ones funding this, have any power? Assign the kids random, ficticious names or numbers AND COMPARE THE DATA!

justbrowsing

June 10th, 2010
10:02 am

I hope that Arne Duncan is paying attention to this.

Dunwoody Mom

June 10th, 2010
10:08 am

Sorry, if this is duplicated, but,Thanks Maureen. I am assuming the State BOE is waiting for the CRCT re-test results before releasing the “final” scores and AYP results, but releasing these preliminary scores should have been done. It should not take a Open Records request to get that information.

Maureen Downey

June 10th, 2010
10:10 am

@Dunwoody Mom, Because Dunwoody Elementary opened in 2009, it was not in our comparison report.
Maureen

Dunwoody Mom

June 10th, 2010
10:14 am

Duh, that makes sense. Thanks for checking.

APS IS CORRUPT

June 10th, 2010
10:16 am

For those who said to fire Beverly Hall: It won’t make a bit of difference to her nor will it have any impact on the classroom. She’s already swindled her millions from the taxpayers and the Superintendent has no direct affect on the classroom. You have to understand that every administrative decision made at APS is made by incompetent bureaucrats who’s only objective is to “appear” to be doing something while enriching themselves and their friends. If APS were a business, they would be out of business within six months.

Everything they do there “for the children” is a farce. In the midst of all of the budget cuts/layoffs, cheating and declining student performance they’ve quadrupled the size of their Communications/Propaganda Dept. to sell their b.s. to the public. Anyone remember the old “APS 2007″ performance campaign? The slogan was “By 2007 70% of APS schools will have met 100% of their targets” or something like that. When they failed to live up to it, the slogan was quietly changed to “By the end of 2007″. Of course, even though they failed, Beverly Hall still got her big bonus that year and continued jet-setting around the country on the taxpayers dime to receive her personal accolades.

Fact is, the people at the bottom of the totem pole, such as teachers and support staff, who actually care about the organization get screwed so the people at the top who only care about themselves get rewarded.

Maureen Downey

June 10th, 2010
10:22 am

To all interested in what the AJC did with these CRCT scores: Here is a note from John Perry, our database expert who is co-authoring the CRCT series with Heather Vogell:

In the analysis used in the story, to keep things as simple as possible, we collapsed the numbers down to the school site level, so we compared all the kids in one school last year to all the kids in that same school this year. But in our first round of analysis, we did compare grade cohorts as Atlanta mom suggested — last year’s third graders to this year’s fourth grade, for example. (And that is possible because kids do take the CRCT every year from first grade through eighth.) That analysis resulted in an even more dramatic difference between the cleared schools and the severe concern schools.

Also, in answer to Nicole, the averages we used were weighted by the number of kids at a school, so they are actually the per-student averages, not the school by school averages. And I don’t know that really it answers Nicole’s concern, but the GOSA erasure study did try to account for differences in class sizes by adjusting the bar for flagging a classroom. (I just looked on the GOSA site for their methodology statement, but I couldn’t find it. I’ll keep looking.)

– john

Dunwoody Mom

June 10th, 2010
10:25 am

Well, I was wrong. Why did I think 4th Graders did not take the CRCT? It hasn’t been that long ago that I had a 4th grade student?

cricket

June 10th, 2010
10:42 am

Dunwoody mom- You got confused because 4th is not a “gateway” grade like 3rd and 5th.

Dingy

June 10th, 2010
10:49 am

Maureen, please tell the reporters who posted the original article that “peal” relates to sound as in ringing a bell or saying something loudly. The tape was “peeled” back. A grammatical error like that discredits the story about education. And, I am not an English major – just a product of Alabama public schools.

Back to your regularly scheduled programming…

Maureen Downey

June 10th, 2010
10:50 am

@Dingy, Thanks. I will send to the copy desk.
Maureen

catlady

June 10th, 2010
10:55 am

Dunwoody mom: probably because, unlike 3rd and 5th, it doesn’t “count” in 4th. Don’t worry about it.

Our 4rh grade teachers and students are much more relaxed about the CRCt, BTW.

Susan

June 10th, 2010
10:58 am

I have some serious problems with your story. While no one doubts there is a need to look at the “severe” schools from last year, I think you need to wait until all schools’ scores are available. A few weeks ago, the AJC ran a story on the abysmal math failure rate among 9th graders across the state, due to the new math curriculum.
In our part of the state, schools are reporting high failure rates by 8th graders on the CRCT math, due to the new math curriculum, poor training of teachers on the new curriculum, etc. And, to add insult to injury, the practice questions for the math test were totally different from the actual test.
Perhaps the “severe” schools have high failure rates due to other reasons, in addition to the scandal.

Show Me the REAL Math

June 10th, 2010
11:24 am

Maybe John Perry could also figure out what the state’s new cuts scores are. It is one thing to say that x students did better/worse than y students, but I really resent Kathy Cox patting herself on the back for positive gains when we all know the cut scores have been manipulated. Rather than say HOW MANY students met/exceeded expectations, the AJC should figure out what the average number of CORRECT answers were per topic, per school. That, I promise you, would be an eye opener.
CRCT scores are easily manipulated… and I’m not talking about cheating. That’s why parents should rely more on ITBS scores when evaluating your child’s progress. At least Cox and crew couldn’t get their ham-fists into that cookie jar.

Atlanta mom

June 10th, 2010
11:30 am

“But in our first round of analysis, we did compare grade cohorts as Atlanta mom suggested — last year’s third graders to this year’s fourth grade, for example. (And that is possible because kids do take the CRCT every year from first grade through eighth.) That analysis resulted in an even more dramatic difference between the cleared schools and the severe concern schools.”

This is the worst possible result. It seems to me, this is what APS needs to do. Any child who failed this year’s test and “passed” last year test, needs to retake the test the child “passed.” If said child cannot pass the previous year’s test, that child deserves individual tutoring paid for by APS, until he/she is up to grade level.

catlady

June 10th, 2010
11:42 am

That is something I would feel concern about: What i perceive to be a drop-off of scores in 4th, since it “doesn’t count”. So, comparing 2nd to 3rd is problematic, given the difference in how the kids take the test (read to vs read yourself), and the difference from 3rd to 4th, given that it is not a gateway grade. Has anyone documented this possible phenomena at the state level, and “compensated” for it statistically in scaling for the test?