You asked. DOE responded that CRCT does measure higher-order thinking

Last week, I sent some questions to DOE from posters, including this one: Does the CRCT measure higher-order thinking?

DOE says “yes.” Here is the detailed response:

Each of Georgia’s test programs contains a range of test items in terms of both difficulty (rigor) and complexity. To gauge the complexity  or cognitive demand of the test items, Georgia uses a model called “Depth of Knowledge” (DOK), which was developed by Norman Webb at the University  of Wisconsin.

DOK consists of four levels of complexity ranging from recall (level 1), skill/concept (level 2), strategic thinking (level 3) to extended thinking (level 4).  Each item that appears on a Georgia test has been assigned a DOK rating by a committee of Georgia teachers during item review.  One of the strengths of Webb’s model is that in addition to rating the items, the curriculum standards themselves are also rated using the DOK rubric.

Utilizing DOK helps to ensure alignment of a test (e.g., CRCT) to a curriculum (e.g., Georgia Performance Standards (GPS)).  Alignment is more than simply ensuring an individual item reflects the content or skill of a  particular curriculum standard.  It also involves ensuring the collection of items that comprise a test also reflect the cognitive demand outlined in the curriculum.

While every test will have a range of items, a well-aligned test will ensure the average DOK of the collection of items matches the average DOK for the curriculum standards measured on the test.

Ensuring the match between the standards and items is an important aspect of test construction.

Finally, it is important to note that both the Mathematical Process Standards and the Characteristics of Science standards are assessed. These standards are integrated with the content standards and included on the assessments.
Matt Cardoza
Director of Communications

46 comments Add your comment


July 15th, 2010
1:14 pm

Did you expect the DOE to say otherwise? Of course they will defend this idiotic test.

South Ga Teacher180

July 15th, 2010
1:21 pm

I have been on those item reviews and let me just tell you, the testing moderator at each table gets rated on rejection rate. So whatever the teachers say, you can bet most of that is taken with a grain of salt. Especailly, when the modarator is not a DOE employee. It is a testing company employer such as Pearson or McGraw-Hill or a subsidiary company.


July 15th, 2010
1:55 pm

Ask for examples of these levels of questions from actual questions that have been retired from the CRCt. Then ask for a breakdown of what percentage of each are on the test, specifically. (You would expect, from reading above, that more Level 4 questions would appear on the 8th grade test than on the 3rd,for example.

I’d love to see actual examples of level 3 and 4 questions for 3rd and 5th!

[...] This post was mentioned on Twitter by Neil Sullivan, Maureen Downey. Maureen Downey said: You asked. DOE responded that CRCT does measure higher-order thinking [...]


July 15th, 2010
1:58 pm

New blog topic: You asked and DOE answered: Every person working at the DOE is absolutely necessary for the success of our students, and each one is paid a pittance of what they are worth!


July 15th, 2010
2:02 pm

Guess that makes the DOE a bunch of libs.


July 15th, 2010
2:02 pm

I received this same information from the DOE – WORD FOR WORD – in an email earlier this week. Unfortunately, the person who corresponded with me did not understand that a recall question (Level 1) is a recall question even if the question requires recalling lengthy, memorized formulas, a little known president, or recognizing the characters in a story. The standards themselves are not tested on CRCT – the test items focus on the standard’s elements, which are generally written on a recall or skill level. The Mathematics process standards of making connections (among mathematical ideas, concepts, strategies, procedures), communication (explaining problem-solving processes, procedures, ideas, etc), problem-solving (using maths understandings to solve a problem that does not have 3 answers already given to choose from) are NOT tested on CRCT. Developing mental math strategies is a requirement for 2nd grade and 3rd grade – that is not tested (can’t be as each person’s strategies may be different but still efficient and correct). There is an element for kids 1st-3rd grade knowing basic math facts with fluency and understanding – how can understanding (the part that is above level 1) be assessed on a multiple choice test? The standards for reading require children at different grade levels to be able to determine themes in a book – NOT read a one page (many time shorter) story, and choose an answer from 3 possible choices.

The days of simply putting out stuff that “sounds good” are coming to an end. My questions remain. What are the percentages of questions at each level of DOK for each grade level? I teach 1st grade and read the questions and answers to my children. I am very familiar with Webb’s DOK – and do know the level of the questions and answers for the 1st grade test. I invite the DOE CRCT folks to release the test items for CRCT at all levels and let us be those teachers who assign the DOK levels to the questions….I won’t hold my breath. But, y’know, the online practice tests might give everyone a chance to do this…I believe last year’s practice tests are still available.

Again, I have no problem with a state common assessment – however, this is the ONLY accountability that exists in the state for GPS – what and how you test shows what you value. I simply have problems with how the scores are used.

why bother

July 15th, 2010
2:08 pm

So people suggest we should ask DOE, but when DOE responds they just simply dismiss the response. Do they really want any answer, or do they want to hear the answer they want/believe only?

Teach 10

July 15th, 2010
2:09 pm

From…..”Evaluation: judging or deciding according to some set of criteria, without real right or wrong answers.” Everything we are told as teachers is to expand critical-thinking in our students and to go as deep as possible and incorporate the 21st Century Skills (technology, real-world application, innovation, creativity, collaboration, critical thinking, communication), which is a magnificent way of engaging, reaching, and teaching students. It’s difficult to understand how ANY of these can be judged on a multiple choice, standardized test. If you want to have the CRCT and standardized test scores be the end-all-be-all then fine. Just don’t pretend they’re something that they’re not.

1st Grade Teacher

July 15th, 2010
2:09 pm

I had an ESOL student that did not know his letters of the alphabet (not even his own name) yet he passed the reading section of the CRCT. Either he’s a very good guesser, a lucky guesser, or his scores were changed (no I’m not in APS). With three answers, and one of them usually being obviously wrong, that gives them a 50/50 shot at a correct answer.

I think it’s strange the CRCT would have 6 questions on commas for 1st graders. I would rather it focused on more basic language arts skills.

The CRCT does not tell me where my kids are because the baseline is set too low to make an assessment. Has anyone seen the breakdown of what’s a passing score for each subject. 800 may be passing, but how is it broken down. The scores mean little for the assessment of my class.


July 15th, 2010
2:12 pm

The Georgia Standards are about 40% writing, yet the CRCT has no writing portion. How, then, do the questions really align with the standards?


July 15th, 2010
2:15 pm

I just posted a long reply and it must have been caught in the filter – aaarrrggghh!!!!

catlady, your questions are exactly the ones I asked the DOE related to this very topic. I received a reply that is word-for-word the same as this statement. I pointed out to them that it is not the standard that frames the test item questions but the elements of standards which are often quite content or skill based. Also, the mathematics process standards are NOT assessed on CRCT – they can’t be. How can a child’s ability to independently make connections to/among math concepts and skills be assessed using a multiple choice format? How can independently solving a problem and communicating (explaining) how the problem was solved be assessed with multiple choice? Then there’s the elements for knowing basic facts with fluency and UNDERSTANDING – how do you assess a child’s understanding with multiple choice? Now, think about reading – some grades require that a child be able to determine themes in a reading selection (and we hope the element is referring to at least a 5 page short story rather than a one page standardized test passage)…ability to infer a reasonable theme would not be assessed by providing 3 answers.


July 15th, 2010
2:27 pm

another aspect of this idea is I believe that there is nothing wrong with CRCT – we need a common assessment for the children in our public schools. The rub is that no one tells the public that the scores from trhe test do NOT mean that children can actually perform the elements of the standard independently. The public is NOT apprised of the standards and elements NOT assessed by CRCT. Nor is the public told that school systems are not held accountable for making sure that children CAN do these things independently.


July 15th, 2010
2:30 pm

Finally, release the test items. Release the actual percentages of test items believed to be at each level and let other teachers take a shot at assigning DOK levels. I gave the 1st grade CRCT and therefore read every single question and answer…I USE the DOK to develop assessments, instruction, and tasks. I already know the how the first grade test would be evaluated…

Teaching in FL is worse

July 15th, 2010
2:40 pm

teacher/learner @ 2:15. You nailed it-a mult choice test CANNOT truly assess higher thinking. As much as I hate to admit it, the FL test (FCAT) has selective response questions IN ADDITION TO the multiple choice questions. They are closer to the ideal (such as it is)

check check

July 15th, 2010
2:50 pm

the GA governor did not want to give up the money going to his test making cronies and VETOED 1st and 2nd graders from taking the test. Cut scores changed, more questions thrown out,

these people dont care about student learning; critical thinking (which has more to do with brain maturity than what they are taught),

experinece and age have more to do with critical thinking that what goes on in a classroom

Hell, the founding fathers knew this, why do you think that put in the 35 year old age limit for running for President?

RE: check check

July 15th, 2010
2:55 pm

we won’t be seeing “check check” on Smarter Than A 5th Grader anytime soon.


July 15th, 2010
3:05 pm

Why bother, I was thinking the exact same thing.


July 15th, 2010
3:25 pm

When given a situation with only four options as possible answers, the value of DOK is decreased substantially. The student is forced to think within the framework or predetermined responses – NO MATTER HOW WELL CONSTRUCTED the test may be.

john konop

July 15th, 2010
4:06 pm

How is this measured when you need less than 50% of the questions to be right for a pass? I am very curious what standard taught in research methods the DOE is using?

Larry Major

July 15th, 2010
4:24 pm

Somewhat off topic but, Maureen do you know what “vacation” means?

Take some time off to enjoy yourself. Drink more than you would. Sleep more than you would. Forget us and spend time with your family. The world will still be here when you get back.


July 15th, 2010
4:36 pm

They had me right up until “committee of Georgia teachers”. (paragraph 2)

A Teacher

July 15th, 2010
4:49 pm

Anyone who trys to argue that a multiple choice test requires higher order thinking lacks credibility. I am a teacher and have writen and reviewed standardized benchmark tests using DOK for my county (metro Atlanta county).

Standardized Mutlple choice tests are a politically convient way to compare students-to-students and schools-to-schools.

Do they help faciliate a love for learning within students? Ask a student to find out. There are far more meaningful ways to assess learning than through multiple choice tests.

By the way, it is impossible for a m/c test to provide a level 4 DOK. The CRCT is designed to be 50% level 1, 30% level 2, 20% level 3.

Way to go Georgia.

Freedom Education

July 15th, 2010
5:07 pm

Very good A Teacher. The first thing that came to mind was that multiple choice tests are lower order thinking in and of themselves.

Gwinnett Parent

July 15th, 2010
5:16 pm

If the teacher has to read the test, it should be void. It is called reading comprehension for a reason. I am glad that they scraped the CRCT for 1st and 2nd grade. The students should read these tests independently. Obviously if the student cannot read the comprehension section, he or she should not go to the next grade. No wonder the teachers at the upper level are pulling their hair out in frustration.


July 15th, 2010
5:41 pm

It is utterly amazing the amount of wasted spending that comes with standardized testing. From research and development to paying publishers, to paying proctors, to scoring. This is a huge monetary investment.

I believe the hundreds of millions used for testing statewide could be better used to take care of teachers.


July 15th, 2010
6:24 pm

@Gwinnet Parent – no, the 1st and 2nd grade CRCT is now back – Sonny vetoed that legislation saying that the little ones need early standardized testing to get ready for the 3rd grade “gateway” tests…yes, absurd.

@a Teacher – thank you for your wise remarks – where did you get the percentages for each level of questions? :)

@why bother

July 15th, 2010
6:48 pm

“Do they really want any answer, or do they want to hear the answer they want/believe only?”

They don’t want any answer, they want a real answer. Unfortunately what most people fail to see is that the question itself wasn’t a a question designed to get real accountability from the DOE.

A real question would have been, “Does the CRCT measure higher order thinking, and if so what percent of the questions measure higher order thinking, and can you provide some specific examples of how the CRCT multiple choice format tests higher order thinking?”

And this isn’t to say the DOE couldn’t provide a good answer. Maybe they could. But once again, readers let yet another aspect of the AJC’s coverage go unchallenged, demonstrating their own lack of critical thinking skills.

It is what it is.


July 15th, 2010
7:24 pm

Give me a break, when my 8th graders who read on the 4th grade level passed the Reading CRCT where is the higher order thinking? I believe it resides in political expediency.


July 15th, 2010
8:11 pm

The question would be whether students have to pass any questions about level 1 to pass the test.


July 15th, 2010
8:11 pm

about = above


July 15th, 2010
8:16 pm

CRCT = another example of a government agency being given over to corporate America in the 2000’s . Cha – to the – ching.


July 15th, 2010
8:29 pm

I am willing to bet two breakfast meals at Denny’s that neither the members of the DOE nor 1/3 of our elected state house and senate would pass the test.

Ole Guy

July 15th, 2010
9:11 pm

Could it be at all possible that, somewhere in these remarks, we’ll find some answers to why educators are being imported to this Country, and into Georgia?

While I don’t have any numbers or stats, I would imagine that vast numbers of kids/young adults in “advanced” countries around the globe, speak multiple languages. While kids from the so-called developing countries…those countries from where many immigrants, both documented and undocs, hail…have only our illustrious ESOL on which to learn English, the countries of the European Nation and the Pacific Rim seem to have realized, for decades upon decades, the direct correlations between foreign language skills and applied math and science disciplines as they relate to the critical thinking skills.

Maybe this is why foreign teachers are being recruited into the educational wasteland of the USA.

HS Teacher

July 15th, 2010
9:17 pm

Everyone – we need to just ‘give up’ on GA education until the republicans in the state are no longer in power. They will continue to beat the drum of their party line and drive education in this state into the ground. They will continue to decrease education spending, increase state DOE personnel (to support this testing, etc.), and lead us down the path of destruction in order to gain popularity of the ‘voucher’ program.

I really feel for the students across this state whose lives will be forever crippled because of this political move. At least the children of these republican politicans won’t be impacted – you KNOW they send THEIR kids to private school.


July 15th, 2010
9:37 pm

I understand that a m/c test cannot test level 4 DOK. But does that matter so much when the students can’t answer the lower-level questions? So many students who may be able to understand how to properly work an algebra or probability problem are unable to get the right answer because they can’t do the arithmetic required. I get the concern about the tests, but I think we need to take the first things first and worry about the more abstract concerns with the tests next.

@Writing? – The “writing” portion of the tests focuses not on how well one can write an essay but on grammar, mechanics, etc. using m/c questions as well.

Team Teacher

July 15th, 2010
10:47 pm

I miss common sense and “MacGyver” skills in my students the most. I worry about this more and more. Is it apathy to thinking? Why are they such passive learners? Something is not right. Each year they seem more and more off a mark, some mark, I don’t know exactly what the mark is… The “okay” mark, the “safe” mark, the mark that says they are tall enough to ride the roller coaster…it’s a feeling, like a gut feeling from an anxiety, like you are allowing them to go off on a journey unprepared. Alone. In Danger. Something is wrong. We, the collective “us” in education, past and present, did this. Some of us made it the law, some dressed it up and made it attractive with money, some demanded it, required it, or held up each new hoop for the rest of us, some just did what they were told, some just stayed silent waiting and hoping for its fruition or the winds of each new best thing in education to come blow it away. Did all of us in education lose our collectively minds? What are we making progress, as a nation, towards if our children can’t make the cap for the BP oil gusher faster and better next time? What are we making progress towards, as a nation, if our children can’t solve our problem of dependence on natural resources for energy & fuel? I’m worried that somehow leaving no child behind has ended up meaning no child moves forward. Have we made them unfit for the future?

check check

July 16th, 2010
12:04 am


thats right I will not be on any show where Kathy Cox has been

[...] This post was mentioned on Twitter by Beth Friese, MONISE SEWARD. MONISE SEWARD said: From @AJCGetSchooled You asked. DOE responded that #CRCT does measure higher-order thinking. #education #edreform [...]


July 16th, 2010
5:42 pm

The top level thinking skills of synthesis and evaluation require thought. Since students’ thinking processes are not measured and the students are limited to the four predetermined choices, it is IMPOSSIBLE to measure at these levels with multiple choice tests. Even at the third level of Bloom’s taxonomy, analysis, students must be able to share reasons, comparisons and contrasts to successfully measure this level of thought. Again, multiple choice tests can not do this as long as students are limited to four predetermined answers.

For the state leaders and the testing department to make claims otherwise undermines their credibility with me and other educators as another poster has already stated. Assessments have their place but let’s not make claims about the tests that are impossible.

Since the pressure of testing has limited the curriculum in our schools, we are losing a generation of children in terms of developing thinkers. I have seen evidence that even in our gifted programs – where all lessons should focus on the top levels of Bloom’s taxonomy – students are enduring more of the basic levels of thought. This is very alarming.

Parents and educators must act through the political process to help our leaders understand that it is time to reduce the emphasis on standardized testing of our children. Next Tuesday will be a good opportunity to begin sending that message.

john konop

July 18th, 2010
8:23 am

Tony good post all should read!


July 18th, 2010
5:29 pm

I wonder if the AJC would be interested in sitting in on the committee meetings while teachers conduct the item review? What does the item review process look like? How is it conducted? Who is in charge of the meeting….the DOE or a Pearson representative? Is it really a review that allows meaningful feedback or just a powerpoint presentation.

We should insist that the DOE release an item-by-item analysis of the CRCT’s/EOCT’s/GHSGT. That analysis should correlate to the exact GPS standard, DOK, and the percentage of students that answered the question correctly. This analysis should be broken down into the following: state level, district level, school level, and classroom level.


July 18th, 2010
7:30 pm

Teacher&mom, an excellent suggestion!

Toto: exposing the man behind the curtain

July 19th, 2010
12:24 am

The original CRCT’s were based on (Benjamin) Bloom’s Taxonomy of Educational Objectives Cognitive Domain. Many HOTS items (higher order thinking skills) were based on Book 2 Affective Domain. ETS had two representatives on this classification project, as well as Warren Findley from the University of Georgia. Many of these HOTS items were not just about measuring academics, but about measuring attitudes, values and beliefs. These HOTS questions were farmed out to the “professionals”. Use the FOA to request the RFP’s (Request for Proposal) to see who got the bid for writing these test items. The testing company holds the copyright for all test items, even though taxpayers foot the bill for them. You can, however request to see the CRCT item “writing guides”, which are downtown at the GDE tower. Gleaning information from the test taker is quite sophisticated. Don’t assume certain testing parameters are measured by only one question. They use something called “strands”. Just looking at test questions will not give a true picture of what and how a student is being assessed. You would also need the “item specifications”. This would tell what level and domain a test item/strand is measuring.

Private School Guy

July 19th, 2010
8:27 am

Mandatory universal testing must be eliminated. Random testing to ascertain that the grading schools are doing is valid is really what is needed. Coca Cola does not open every bottle of Coke and taste it to make sure they are making a good product.
Testing is just a payoff to the test makers who have politicos and others in their pocket.


July 19th, 2010
9:58 am

Oh, PLEASE!!! Every single answer on the CRCT comes from kids parroting what they have been taught by teachers deathly afraid for their job if the kids cannot regurgitate the answers well enough. Our kids are memorizing answers, not being taught how to think.