Are there ever errors on the CRCT that proofreaders miss?

A parent sent me a note about the CRCT and how carefully the tests are reviewed for errors that could impact a student’s score.

I know that DOE proofs the tests, but assume that a mistake could be missed. Has anyone had a similar experience to this parent?

With the parent’s permission, here is her note to me:

My son took the CRCT last week and brought up something I thought would interest you. He is in fifth grade and has always done very well on the tests. I am questioning the proofing of the tests themselves. What first caught my attention was when he was taking the online practice OAS tests. The results showed he answered many of the math problems incorrectly. Yet, when I looked at the questions, he appeared to have the correct answer.

The next day, his math teacher told him some of the answers the OAS provided were incorrect. Not sure how they’re supposed to learn from a flawed practice test.

During the week the CRCT was given, I’d ask him each day how the test went. Every day seemed to be fine – until he got to science. He told me it was “strange.” He claimed several of the questions were not grammatically correct and some of the diagrams were off. When I asked for details, he explained a diagram of a cell, with a question asking what part of the cell the arrow was pointing to. Yet, the arrow was in the blank space outside of the cell.

He told me several other students also were talking about how strange the science portion of the test was. I understand teachers are not allowed to look at these tests, but how do we know what they are putting on the tests is even correct? Who oversees this?

It seems like once the tests are administered, they should be made available for parents and teachers to view. I can fully understand typos and some mistakes slipping through. However, when you place such a high importance on passing a test like this, I would think it would go through several rounds to make sure no errors exist.

–From Maureen Downey, for the AJC Get Schooled blog

95 comments Add your comment

Private School Guy

April 27th, 2011
5:38 am

As an educator we are told we can discuss nothing about the test under threat of losing our certification and being sued by the testing corporation. This entire business is a national disgrace. Each year I hear more and more parents are considering boycotting the test. I hope this comes to pass.

ScienceTeacher671

April 27th, 2011
5:56 am

If our HS students think there is an error on a state test we are allowed to write down the question number and the test version so that the testing coordinator can ask the state about it, but we’re not supposed to look at the tests.

Tucker Guy

April 27th, 2011
6:03 am

I am reading the CRCT to a 4th grader and some of the questions are so badly written that I don’t know which answer they are looking for.

Sam

April 27th, 2011
6:10 am

While I can understand your frustration, I do think that it’s important to point out that OAS is primarily composed of questions created by people in-house and/or a group of teachers gathered for the specific purpose of creating questions. You will see more errors there because they are inputed by someone at the GaDOE and those are rarely checked for human-error. They are not regularly updated and/or proofed. In fact, I have been told by someone at the GaDOE when complaints were being made that bc of budgeting and cuts, OAS and other resources unrelated to Math and ELA/Reading were at a low priority.

d2

April 27th, 2011
6:52 am

When you even have the coach books given the wrong slope formula for students to study for 8th grade–know wonder–The CRCT is not out to test what you know –it is to trick what you know.

Tony

April 27th, 2011
6:54 am

Yes. The CRCT has errors and poorly written questions, but we as educators are not supposed to know this because it is strictly against the rules for us to look at the questions. While monitoring the tests, I have seen a few of these questions. Most questions are good, well written items based on the grade level curriculum.

One year I tried to report a question with a serious flaw and the folks at DOE tried to turn me in for violating test security. There really should be a means for us to work together to make sure our tests can be improved.

catlady

April 27th, 2011
7:16 am

We can’t talk about it. Hummm, hummm, hummm.

Hard to miss when you have to read it to those with that accomodation.

We have been saying this for a long time. It is a poorly written test that has limited validity and reliability. How much and who have profited from this test, its construction, grading, reporting, remediating, publishing?

goodforkids

April 27th, 2011
7:36 am

My kids and I have seen errors on OAS practice tests too. Not a lot, but some.

Yes, there are always going to be kids in the building that require someone to read the test to them, so teachers will see the items.

Crazy that such a teacher would not be permitted to provide feedback on a flawed item.

At a quality testing company, even basic item analysis would bring potentially bad items to the attention of those who are in charge, who can make then decisions to examine and/or toss those items.

I don’t know if the CRCT is administered and monitored by a quality testing company…

YES, we have been wondering about all of that for quite some time, catlady…so it bears repeating,
How much and who have profited from this test, its construction, grading, reporting, remediating, publishing? I have tried to find info on line before but without much luck. You would think that would be transparent…

another view

April 27th, 2011
7:41 am

ALWAYS remember that state (ALL states) minimum competency (euphemistically referred to as “criterion” referenced – they’re not) tests have one thing in common with the space program.


Low bid.
The people who write the actual questions (FAR away from here) are often people who have minimal or no experience with children or schools. The questions (”items”) they write are reviewed by people who have (usually) a fair amount of technical training in test development, but not necessarily with teaching, kids or schools. The questions (most of the time) are tested in front of real students the year before actually being used to fail kids or teachers or schools, but they’re only screened for their “quality” in terms of (usually) a statistical technique known as item-response theory, and reviewed (usually) by people (who rarely have expertise in testing) for “bias.” All this has to be done at the lowest cost and in the least time possible – to get ready to start all over again.
We’ve been doing this in the country since the late ’70’s and only now are people just beginning to understand how incredibly, terribly poor these tests are. The papers gleefully report whether pass rates (NOT SCORES) go up or down – anybody ever asked what that has to do with your child’s education and whether or not they’re actually being well taught, or whether they’ll do well in college and in life?
We should dearly hope that Dr. Barge has some success in both cleaning his house from the damage done by the last two residents of that office, and in countering the damage that will be further perpetrated on Georgia teachers and students – and Georgia – by the misguided Race to the Top.

another view

April 27th, 2011
7:43 am

Oh, yes. If the actual tests are that bad, it doesn’t take much reasoning power to figure out how poor somebody’s hastily tossed together (also low-bid) “practice” tests might be.

Michael Moore

April 27th, 2011
8:02 am

I think it is wise to remember that our tests (crct) are provided by McGraw Hill. This company also provides tests for sixteen other states. These really aren’t Georgia’s tests, they are provided by a very powerful testing company.

EdDawg

April 27th, 2011
8:08 am

I have been reading the CRCT for about the last 7 years since the start of the read aloud accommodation. Yes there are mistakes & culturally biased questions. That’s all I can legally say about that.

Dr NO

April 27th, 2011
8:10 am

Im sure the test isnt flawed as Lil Einstein couldnt possibly make an error nor could Mother. ;)

HS Public Teacher

April 27th, 2011
8:14 am

Please STOP blaming teachers. We are disposable commodities in the eyes of the school systems in Georgia and simply do as we are told at the risk of losing our jobs and our certification.

If YOU don’t like this, then you should support a real teacher union that allows classroom teachers a voice to improve education.

What's best for kids?

April 27th, 2011
8:20 am

Since we are not allowed to read them, how are we supposed to acknowlege the errors? I think that was the plan in the first place.
EOCTs are just as bad. Checkpoints are even worse.

lala

April 27th, 2011
8:31 am

When I was on a school council, we used to read old test questions and test prep OAS questions. It was always my favorite meeting of the year b/c we would end up in hysterics because some of the questions were such a complete mess. My kids come home every day during the CRCT with an amusing story about a screwed up question. The whole thing is a complete joke.

student

April 27th, 2011
8:36 am

I think when a question is missed by an overwhelming majority, it is usually taken out for scoring. Mistakes are bound to happen in test writing processes. A grammatically incorrect or unclear question never killed anyone. Parents overreact too often. Students have taken these tests for years, no one is hurt.

What's best for kids?

April 27th, 2011
8:42 am

@student,
But a grammatically incorrect or unclear question CAN impact the school’s AYP status, especially for ELL and Special Ed Students. There need to be more teachers for test generators.

teacher&mom

April 27th, 2011
8:54 am

@Tony: after going through the process of reporting, and then suffering through the subsequent inquisition…there is still no guarantee the test item is even changed :(

Below is a link to several articles written by Todd Farley, a former employee of the standardized testing machine. He offers an insider’s perspective of the testing industry.

http://www.huffingtonpost.com/todd-farley

@Maureen: any chance you could give us the amount of money GA budgets for standardized testing? I’d also like to know the amount spent on test support materials. I think the general public would be shocked.

Let me add this: As a parent….any time you hear the words “this will improve test scores”, you should immediately go on the defensive. Nothing in education should be about improving test scores. Instead, it should be ALL about improving learning. No one talks about learning anymore. Too much of the conversation is devoted to test scores.

student

April 27th, 2011
9:03 am

@What’s best

I see your point, but from what I’ve seen, questions are never THAT bad. Also, I’m pretty sure if a significant majority of students miss a question, it will be thrown out, therefore, the questions are not used in calculation of the scores – which I think correspond to AYP.

A Conservative Voice

April 27th, 2011
9:14 am

I think I would go to McGraw-Hill and look at their hiring statistics :)

Atlanta Media Guy

April 27th, 2011
9:19 am

Heard from my son that a child in his class had the same issue with the science part of the test. He raised his hand, told the teacher and then the fun began. The student was escorted from the room the principal, asst. principal, teacher in charge of testing and another observer my son had never seen all huddled together to discuss the question. My son told me his friend thought it seemed a bit ridiculous. Plus, he was not given extra time to complete the test since he had to go outside of the room as well. It was his insistence that allowed him to finish the test. He said he wanted to finish the test and should be allowed to.. When he was told he would not be able to he asked for a phone to call his parents. Well the principal allowed him to finish the test in the hall under her watchful eye. Common sense would help this process, like it did in this instance.

I don’t think unions would help this issue, however I do believe teachers should be involved in the test prep process. Teachers! You know the people who actually spend time in a CLASSROOM and do the job that the the leaders, of many districts, seem they can’t do right now

teacher&mom

April 27th, 2011
9:20 am

@A Conservative Voice: I’d recommend going to McGraw-Hill and Pearson and look at their political contributions :)

mystery poster

April 27th, 2011
9:27 am

The test is NOT available to discuss right after it’s given because the same test is reused. IMHO, this has a major threat to test security.

NG

April 27th, 2011
9:48 am

Years ago, a good friend of mine was teaching first grade and his students were all upset about a math question…it seems that this friend had correctly taught his students that a square was also a rectangle (by the mathematical definition) and the question aksed which one was a rectangle…but had a “standard” rectangle and a square as two of the possible answers. His well-taught students knew that the question actually had 2 answers and didn’t know which one to choose. I think this was when the test was in development and they were allowed to discuss and turn in problem questions, so he went to the other first grade teachers to see if they had the same problem and got blank stares…seems his colleagues didn’t understand that a square is a rectangle. I don’t know if that question is still on the test in that format, but it is a shame that the test writers are not able to write questions that aren’t confusing.

cost?

April 27th, 2011
9:50 am

what is the cost of all this testing? How much tax payer money is going to pay for CRCT, EOCT, and which company provides the materials? Who are their lobbist and are they former politicans?

AJC sounds like a story that is worth going after…

Roach

April 27th, 2011
9:51 am

It would do a world of good to have a citizens commission reviewing standardized tests in a public forum. To have improvement, there must be feedback. Measurement theory says there are an infinite number of ways to ask most any question, so it is not as though the company will run out of questions. If Georgia can’t do this alone, then our governor should partner with influential states like Texas. It would cost much–I’d volunteer just for the amusement value.
. The regents tests that our college students had to pass were just as bad, and for years and years. International students in our universities sometimes had a terrible time, because the tests were loaded with U.S. pop culture references (not what these students had come here to study) and because of bad grammar. Strong classroom scholars had to take some of these tests repeatedly, to overcome the weaknesses of the tests. Thank goodness the regents tests have been discontinued.

Roach

April 27th, 2011
9:52 am

“wouldn’t” cost much . . . sigh

student

April 27th, 2011
10:13 am

@Roach

I’d volunteer too. That’d be a good idea.

Just A Teacher

April 27th, 2011
10:33 am

I agree with “cost?”. I would love to know how much money the state spends on standardized testing every year. That would include paying for the testing materials, the salaries of those administering the tests, the salaries of those who grade them at the state level, shipping costs between the schools and the state and, now, the salaries of those who are investigating erasure / cheating scandals. I’m sure the total would be tens of millions of dollars at least. All for a flawed system which does nothing but take away instructional time from our students.

d2

April 27th, 2011
10:48 am

The State could care less what anyone thinks–but I really wish our Assemblymen and women would read these and do something–espcially when the want teachers to be evaluated one the test. Oh by the way why can’t we make the test have some accountability for the students like the EOCT counts as part of the high schoolers grade. Whatever happen to the growth model? Give a pre-test and a post-test to measure growth?

Veteran teacher, 2

April 27th, 2011
10:49 am

Here’s the real question: Is it in the testing company’s best interest for students to do well on the tests?? Politicians have bought into the testing game hook, line, and sinker. Why? Following the money would probably prove interesting.

Now, what are we going to do about this? The politicians work for US!! It will take work, but we can stop all of this testing foolishness.

d2

April 27th, 2011
10:56 am

pardon my typo errors–it should be want the teachers to be evaluated on the test and high schoolers should have an ” ‘ ” and Oh should have a comma, and a few others–I have to say that because the state employs paid bloggers to attack posts on websites and most of the time they attack typo errors instead of addressing the real issues

Stacey

April 27th, 2011
10:56 am

My son (4th grade) mentioned that there were questions on both the math and science sections that he felt like could have had two answers or that none of the answers were correct. An example he gave was an identify the shape type questions where the choices included both rhombus and parallelogram (sp) but he didn’t have a choice that included both. He said he chose rhombus since he felt it was the best answer. He said he tried to ask his teacher afterwards if he made the correct choice and was told not to worry about it and just focus on the review for the next section. Based on the responses from other teachers on here, she was probably afraid to express anything that could possibly be interpreted as an opinion. Thankfully they are taking the last section today.

student

April 27th, 2011
11:00 am

The directions are “Choose the best answer.”

Clueless

April 27th, 2011
11:01 am

The AJC has done great work showing how many students fail and are socially promoted, exposing the cheating in APS, and showing that test grades and class grades don’t always align. Would love to see them “follow the money” in testing all the way to the top.

Clueless

April 27th, 2011
11:03 am

Teachers aren’t allowed to discuss specific questions with students.

Ann

April 27th, 2011
11:10 am

These tests are terrible, in almost every way. Students who happen to have trouble testing are afraid of them, and who can blame the students for their fear. One test, on one day and the child winds up labeled for years, if not retained. Students who learn easily and/or test well and laugh at how easy they are. Ditto for the graduation tests! Teachers fear them – one test, on one day, and a couple of sick students = intense analyzing of all they do. Students and teachers should be held accountable, and should have tests results matter, but no one test is infallible. Even SAT’s and ACT’s can be retaken.

Ann

April 27th, 2011
11:12 am

I know there is one chance to retake the CRCT and administrators and parents can override the results if they agree, but the perception by some of the students affects their ability to test well.

Teach

April 27th, 2011
11:18 am

I no longer teach math, but a few years ago I noticed the same mistakes two years in a row. Separate questions asked for the square root of 49 and the square root of 64. Each question included two correct answers (7 and -7) (8 and -8) in separate answer choices. My testing coordinator contacted the state, but the glaring error appeared the next year.

Ed Johnson

April 27th, 2011
11:31 am

Has anyone any idea what percentage of CRCT items are field test items? In other words, non-scored items put on the tests to see how well they can be answered or not answered and by whom?

Susan

April 27th, 2011
11:31 am

@student: If the question asks, “Which figure represents a rectangle?” and you have a rectangle and a square, there is NO “best answer”…both answers are perfectly correct. Same thing with “Which figure represents a parallelogram?” If there is a square, a rectangle, a non-rectangle/rhombus/square parallelogram, and a rhombus, again, there is NO best answer. All answers are correct. I think that’s the problem with those questions.

Teaching in FL is worse

April 27th, 2011
11:41 am

As a FORMER proctor of the GACE, I see the companies cutting corners everywhere they can. I say former because the proctor pay stayed the same, while shortening lunch and increasing the number of people who need to be monitored.

As far as CRCT goes, it’s like FIGHT CLUB-I don’t talk about it.

Angela

April 27th, 2011
11:51 am

What is the purpose of standardized testing?

Cindy Lutenbacher

April 27th, 2011
12:15 pm

Amen and thank you to “another view,” “HS Public Teacher,” and “teacher and mom.” The practice test questions I’ve seen have been horribly written and ignore the fact that a kid can be viewing things from a wonderfully smart and knowledgeable point of view, but if it’s not the one view the test-maker perceived, then the answer is as wrong as that of a child who knew and understood nothing. Furthermore, I believe it was Todd Farley (my memory could be wrong) who revealed that test questions were chosen based upon which ones received more “correct” answers from upper and upper-middle class kids; questions which more lower class kids got right (and upper classes got wrong) were tossed out.

We desperately need a strong teachers’ union. Most teachers I know understand the lunacy of NCLB, Race to the Top, and all their dreadful spawns, but they can’t afford to be fired for speaking out. It may be against the law to fire someone for speaking out against these bureaucratic abuses, but we all know how a firing can be manipulated to avoid that law.

Also, I once did a quick calculation (based on info from California public schools) of the cost of the graduation test (and that’s just ONE test) for Georgia schools, and it was more than our application for Race to the Top (at the time, we’d not received this nightmare grant).

No matter what we believe or know about test questions, standardized testing is a huge boon for companies like McGraw-Hill (longtime supporters of the Bush family) and a huge boondoggle for our kids.

We need to make sure that the teachers in our schools are tops and we need to support our teachers. Most of the public school teachers I’ve observed are wonderful; just a handful of teachers [in the many public schools (all of which are Title I) in which I've volunteered] need to be replaced or retired.

Roach

April 27th, 2011
12:16 pm

Don’t worry about the cost of testing–our legislators just deduct that from what we would have spent on education.

LS

April 27th, 2011
12:24 pm

@ Angela….to pigeon hole children’s potential! Sorry…just bitter. Standardized testing is meant to ensure a certain level of understanding. Each student should be able to reiterate a certain amount of learned information.

They are seriously flawed. There are some who do very well in a classroom environment and bomb a test such as the CRCT. Not fair.

I’d much prefer to see something that actually shows what my child has learned rather than what he’s memorized for the purpose of testing and then forgets!!

Rationaled

April 27th, 2011
12:27 pm

The state spent approximately $30 million on standardized testing last year. I’m not sure if that is first administration only, or not.

Field test items are there, however, we are never given any information as to which items are field items…and scores are reported in totality. In other words, if the test has 70 questions, the teacher finds out that the student answered 50/70 correctly – never knowing if any of the 20 (or the 50) are field test items. The raw score that follows is a JOKE of a method to concoct: scores are held until a sample size large enough yields data that allows the state to make a judgment on the # that is designated Exceeds, Meets, Does Not Meet, the last being somewhere around 45-48% correct.

GoGoPsychometricians!

And no, we are not allowed to look at the tests we administer for fear we might….find out how truly awful the tests really are. I’m one for making the state release the tests EACH YEAR after the final administration window has closed. What are they afraid of? It’s not like we can teach any MORE to the test than we already do….

student

April 27th, 2011
12:33 pm

If the question asks, “Which figure represents a rectangle?” and you have a rectangle and a square, there is NO “best answer”…both answers are perfectly correct.

Yes, they are both correct, but it is obvious that the “rectangle” is the answer they are looking for with the question. I’m not saying the tests are perfect, but I do think some sort of standardized testing is necessary to gain a representation of the state’s schools. So what if there is a few “less-than-ideal” questions? Everyone gets the same test. No one is being singled out. The student just has to use his/her best judgement on each question. I’m pretty sure that’s the purpose of TESTING anyway. I took a whole buch of CRCT, ITBS, EOCT, GHSGT, SAT, ACT and AP tests. There were confusing questions, qusetions with mutliple “correct” answers, and intentionally tricky questions on them. Guess what I did? I bubbled in the correct answer, and went on with it! One question in 100 doesn’t do much…

big picture

April 27th, 2011
12:39 pm

I remember a practice question from last year. It asked something like: Which of the following would best be a descriptor that Ann would use to describe her horse? Choices include: black, good, and a couple of others that clearly didn’t fit. My child picked “good” as the answer, but got it wrong because the correct answer was “black.” I find this ironic because the school works very hard to get students to just others on the basis of their character rather than their outward traits, but then a question like this emerges. I have often wondered how many questions like this are on the exam.