“By requiring minimum competency testing accountability, we demand mediocrity.”

In the “keeping us honest” department, here is an e-mail from our resident testing expert Jerry Eads in response to the AJC story on exceptional CRCT gains at five APS elementary school, gains that fell far outside the expected range.

The AJC data analyst — the same person who originally did the CRCT analysis in 2008 that led to the exposure of widespread cheating — said  the odds of such increases range from about one in 700 to one in 21,000. APS is now looking at the gains, which principals and parents credit to extraordinary efforts.

Jerry is the former coordinator, research and evaluation, for the Georgia Professional Standards Commission:

Here is Jerry’s response to the AJC reporting on this issue:

Oh, where to start.

Those of you who catch my posts know I have fits about the AJC continuing to mislead readers by saying “scores increase” or “scores decrease.” That’s because the so-called “scale” scores on minimum competency tests are virtually meaningless. The ONLY thing that should be said of minimum competency tests is whether PASS RATES go up or down.

I’ve noted many times here that there are ONLY TWO even remotely meaningful “score points” on the CRCT “scale”: 800 & 850. The shape of the raw score ‘curves’ are extremely skewed (they don’t even begin to look like a bell curve). What that shows is that the tests are BY DESIGN very easy for average and above average students – they all get most of the questions correct, and we learn NOTHING worthwhile about those students. That also should tell you that the PR about “high standards” is nothing but, well, PR. “Standards” are fine when you’re fitting doors onto cars on an assembly line, but they have no place in an educational system that should be doing its best to meet the needs of ALL students.

On any state’s tests, the difficulty of the individual questions changes a bit each year, so that a “pass” might be 30 questions one year, 32 the next, and 29 the year after that. What THAT means is that the contracted test makers have to “massage” (i.e., beat the heck out of) the scale each year so that the difficulty represented (in our case) by 800 and 850 (pass and exceeds) is pretty close to the same each year. Nothing else matters. The difficulty of each year’s tests at 800 and 850 still DOES vary a bit (sometimes a lot – remember social studies); that’s one part of why some of the change in district and state pass rates has NOTHING to do with whether students performed better or worse from one year to the next.

For all the millions we spend on accountability testing, most of you should no longer be surprised that while we (testing geeks) can sometimes be pretty good at measuring student learning, we’re totally worthless at splitting hairs with individual students (like the difference between 799 and 800).

Most of us expect that a point on a “scale” on a test, whether it’s from 50 to 51, 500 to 501, or 800 to 801 to be akin to inches on a ruler. That’s pretty much true for something like the SAT. The difference between 4 and 5 inches is the same as between 11 and 12 inches. Nothing could be further from the truth in minimum competency testing. It’s quite possible that the difference between 852 and 858 could actually be LESS than between 858 and 871 in terms of actual student learning.

SO, even though the school’s average reading CRCT “scale” score increased from one year to the next, IF the test actually tried to adequately measure ALL students (like the norm-referenced tests from days of yore), it’s VERY likely the average score would actually go DOWN, because minimum competency testing forces teachers to focus only on those students around the “pass” difficulty level, at the expense of all other students. This is especially true in areas with high proportions of very disadvantaged students.

Are the changes in pass rates at the school referenced in Maureen’s recent post possible? Yes. There are MANY ways to change pass rates – not the least of which is to transfer or at least label as transfers low-performing students. Another is to change answers after the fact. But another is to put all (or most) of your resources into doing NOTHING else but teaching the students performing close to the pass point to pass the tests – to the exclusion of real education. While the last is perhaps the best of three miserable options, it is certainly far from what we should want for our kids. By requiring minimum competency testing accountability, we demand mediocrity.

I’ve heard rumors there may be changes afoot for Georgia’s school accountability. For the sake of our kids, our state, and our state’s economy, I certainly hope so.

–From Maureen Downey, for the AJC Get Schooled blog