DOE response on ITBS versus CRCT: Testing basics

Several of you have asked for an explanation from DOE about state testing and why we don’t see how Georgia students performed on the Iowa Test of Basic Skills

Here is the response from Melissa Fincher, associate superintendent of assessment and accountability:

The assessments that comprise the Georgia testing program are mandated by Georgia law. For years the Georgia program has included a national norm-referenced test (NRT), which historically has been the Iowa Test of Basic Skills (ITBS).

During the 2008 legislative season, the law was amended to make school district participation in the NRT optional. In addition, the law allows districts to select two grades – one in grade band 3 – 5 and one in grade band 6 – 8 – to assess with the NRT.

This option went into effect during the 2008-2009 school year. Prior to this time, school districts were required to test students in grade 3, 5, and 8 annually.

Before we discuss why statewide performance on the NRT is not available, it is important to discuss what an NRT is. Like most nationally-normed tests, the ITBS reflects content and skills that are commonly taught in each grade across all fifty states.

While certainly the content and skills taught in each state are similar, there are differences in terms of when things are taught and the degree to which they are stressed within a state’s curriculum.

Therefore, there is some degree of misalignment between any national NRT and what is taught in any given state for any given grade.

The purpose of norm-referenced testing is to obtain insight into a student’s performance or achievement relative to a nationally representative sample that was administered the test at a point in time (called a norming sample). Most NRTs have fall and spring norms and depending on when the test is administered in a school, student performance is reported relative to the spring or fall norms.

Specifically, results of an NRT depict how a student’s academic skills compare with students in the national sample—the obtained scores highlight differences between and among students across an achievement continuum.

This differs from the purpose of criterion-referenced tests, such as Georgia’s Criterion-Referenced Competency Tests (CRCT), which is to depict whether students have met a particular standard of performance.

In Georgia, performance on the ITBS has been moderately well correlated with performance on the CRCT. This means that students with high performance on one measure tend to have high performance on the other and vice versa.

Statewide performance on the NRT is unavailable for a couple of reasons:
1) Although prior to 2008-2009 districts were required to test all students in the mandated grades, districts were allowed to select a testing window that best met their scheduling needs. This means that different norms were applied to district testing in the fall versus spring. In addition, the norms depend on when in the fall or spring the district tested. When different norms are utilized, it is inappropriate to compare or aggregate.
2) As discussed earlier, last school year the NRT became optional. Many districts elected not to administer the NRT while others elected to test in grades other than 3, 5, and 8.

39 comments Add your comment

ScienceTeacher671

November 20th, 2009
6:21 am

Maureen, thanks for posting this! I appreciate Melissa’s thoughtful answer. I’m rushing to get ready for work now, but will look forward to reading it again later today.

PHD mom

November 20th, 2009
7:22 am

If the state were tracking this data, they would see that ITBS scores are dropping and CRCT scores are going up. A prominent, international scholar in the education world has an excellent explanation of this phenomenon. To paraphrase, he says that if you are specific about what you want to measure, you will get it. The question is, are you measuring anything that is worthwhile. In Georgia, we only care about CRCT scores, a test of minimum competency that all students are expected to pass. We force our teachers to spend a large proportion of time teaching to this meaningless test, and we ignore other crucial learning information (which may be measured by the ITBS). The fact that our state is loosening requirements that districts use nationally normed tests to evaluate students should frighten everyone!

catlady

November 20th, 2009
7:24 am

When I read the title of this, I could tell you exactly what it was going to say.

The truth is, our second graders SHOULD BE able to do well on the ITBS for second grade. There is not THAT much divergence in what a 7 year old should be able to do in Georgia vs. Oklahoma vs. South Dekota. Sentences still end with periods, question marks, or exclamation marks, and begin with capital letters, for example.

The way our county gave the ITBS each year was, in third grade the EXACT SAME VERSION of the test was given in fall and spring. Valid measure?

The problem isn’t the differences in the goals of the test, IMHO. It is in the construction and validity of the tests. The CRCT has little to recommend it in terms of this. It is largely poorly designed and written, and cannot stand up to close scrutiny. In addition, would we rather our Georgia third graders be able to handle vocabulary, or tell you the state bird of Georgia or discuss the beaks of different kinds of birds, both 3rd grade CRCT material every year? I also note that a thinking person can frequently find questions for which more than one plausable answer is given. Finally, you have the problem of “cut” scores. They are set rediculously low on the CRCT.

A good score on the CRCT may mean you have learned the GPS for the grade, but does that mean anything for the student? Is knowing the state bird critical to the child’s success in 4th grade? And if success” is 38/70 correct, what does that say? To much “play” in the old steering wheel.

Obfusticate! Obfusticate!

Cobb County Parent

November 20th, 2009
8:02 am

Well said, CatLady. I’d like to see GA ditch the CRCT, but they won’t. Why? Because they can keep playing with the cut scores and the DOE and county BOEs can tell everyone how great a job they are doing in educating our children. What bunk. I’ve personally met with 3 of the 4 candidates to replace Cox for State School Superintendent (Brian Westlake, Richard Woods, and Beth Farohki). All three state they want to change how testing is done and to reverse the math crap curriculum away from the smoothie it has become. I hope to meet with Roger HInes shortly to get his opinions. I wouldn’t give Cox the time of day.

DeKalb County Parent

November 20th, 2009
8:35 am

Another thing to add to this bit of information is that, although the ITBS is not “required” in 1st grade, it is essentialy required to have your child eligible for gifted services for the remainder of the 1st grade year (in DeKalb) and picking up again in 2nd grade. If you do not take the ITBS in 1st grade, in DeKalb county, you will not be offered the opportunity to qualified for gifted (which is an issue all on its own) until 3rd grade.

oldtimer

November 20th, 2009
10:25 am

Catlady you are so correct.

Attentive Parent

November 20th, 2009
12:10 pm

“This means that different norms were applied to district testing in the fall versus spring. In addition, the norms depend on when in the fall or spring the district tested. When different norms are utilized, it is inappropriate to compare or aggregate.”

This explanation does not make sense. Maureen are you quoting someone at the state or is this your summary of their explanations? That it was given suggests that state officials remain desperate to avoid nationally normed comparisons for the state, individual districts, and particular schools for as long as possible. They really don’t want attention called to what textbooks and instructional practices are effective or not in becoming literate, knowledgeable, and mathematically proficient with a variety of materials.

You adjust the # of correct answers required for a percentile or stanine to adjust for spring or fall administration and to accommodate a spread of grades taking the same test. More is expected of the spring test taker than the fall and the 5th grader than the 4th. That explanation is yet another obfuscation indeed.

Next excuse? Perhaps Ga will next abolish use of the SAT in USG admissions or decide not to participate in the National Merit program as they don’t measure what Ga wishes for students to be doing in its classrooms and campuses.

Maureen Downey

November 20th, 2009
12:15 pm

Attentive Parent, I am not quoting. DOE wrote the piece. I did not interview the author. I plan to ask DOE to look at these responses and see if they want to follow up any of them.
Maureen

Attentive Parent

November 20th, 2009
1:21 pm

Thank you for looking into the offered explanation.

Since we don’t want Ga’s schoolchildren forever shackled to living in this state or that P-16 in Georgia is like the “Hotel California”, we need the nationally normed data to know what is working especially in districts where commercial tutoring is not available or is not the norm.

Singing to the Choir

November 20th, 2009
1:47 pm

Cobb County decided this year they would give the ITBS to 7th graders instead of 8th graders. Most Middle Schools are sitting on the results. It makes you wonder if those numbers are down as well.

wondering

November 20th, 2009
3:38 pm

Maureen – while you are at it can you ask the State DOE if they count drop out numbers for all students that were enrolled at the age of 16 and beyond. Isn’t it by age 16 that a student can choose to be in school or not and can choose to drop out.

Reality 2

November 20th, 2009
3:41 pm

I am just wondering for what purpose people are wanting the ITBS or any other nationally normed test. Is it to evaluate schools/teachers?

If teachers can teach to the CRCT, whose problems aren’t released and the main guidelines is the state standards, wouldn’t they do the same for the ITBS? As the name suggest, the ITBS is a test of “basic skills.” Do we want our schools to just focus on basic skills – if the test is used to evaluate schools and teachers, that’s what will happen for sure. Cheating will happen, too.

catlady criticizes the CRCT question about the state bird, but is it a science question or a social studies question? Also, the CRCT is supposed to be testing whether or not students are learning the standards. So, if there is a question about the state bird (what is it, anyway?) then, it must be an item in the standard. Maybe we should be questioning the standards, too.

Also, Cobb county parent, I am perfectly happy with the HS math curriculum my son is going through – he is in the group for whom everything since Grade 6 is “new” for their teachers. So, if you don’t like it, fine. But, please look for a way to allow other parents who are happy to keep the new program going.

Larry

November 20th, 2009
4:04 pm

Attentive Parent, The DOE actually made some valid points for not publishing NRT results.

Lacking a statewide schedule, individual school systems administer NRTs only in grades they select, if they bother at all, which would make any state level report incomplete at best.

As you correctly noted, different norms applied to the fall/spring administration thing would allow valid comparisons, but this is only true in some areas. For example, percentile ranking could be reasonably aggregated, but not grade equivalents.

I don’t know how other systems handle it, but Gwinnett publishes ITBS results every year with the PR and GE broken down by individual schools. The web sites of school systems in which you’re interested may provide similar detailed information.

Singing to the Choir

November 20th, 2009
4:19 pm

Reality 2 you are the first person I’ve heard say they are pleased with the new math. What do you like about it? Do you think it is a better approach to math? If so then why?

Ed Johnson

November 20th, 2009
6:14 pm

Here’s Douglas Reeves, in The Learning Leader, on norm-referenced tests such as the ITBS…

“Norm-referenced tests often fail to provide consumers with raw scores, and they give only a relative indication of student performance. While performance in the ‘60th percentile’ may sound somewhat reassuring (the work in the top half of a theoretical nationally representative group of students), we would be unhappy if we learned that such a score represented accurate answers to fewer than half the test questions. The claim that a student is not as bad as 59 percent of the students in the comparison group is hardly a compelling case for the use of the term ‘proficient.’”

In other words, as is no secret, the ITBS is a zero-sum game, purposely designed to put roughly 50 percent of students above average and roughly 50 percent of students below average. Is it impossible for all kids to ever get above average by the ITBS because when too many of thenm start becoming above average, the ITBS gets “re-normed” to reestablish the 50-50 divide. Too, the ITBS has been much criticized for comprising mostly test items so-called “big house” kids are likely to get right and othet kids are likely to get wrong, and for omitting test items less advantaged kids are likely to get right and the “big house” kids are likely to get wrong.

Better and fairer if the CRCT truly were criterion-referenced free of political baggage rather than try to push the pendulum back to the norm-referenced ITBS.

Reality 2

November 20th, 2009
6:19 pm

Singing,

Apparently you didn’t read this blog about a month ago when there was a discussion of the new math standards. There were even math teachers, imagine that, who liked the new standards. I actually know several of them myself. I think whenever we try something new, it is hard at first. But, I think the new standards are much more cohesive and reflect the nature of mathematics very well. Mathematics is about patterns and relationships, and specially at K-12 level, arbitrarily separating the topic into algebra, geometry, etc. doesn’t really make sense.

I think whenever we have these hot issues, we tend to develop a tunnel vision/hearing because we associate ourselves to those who are like-minded. There is nothing wrong with that, but it is probably a good thing not to think that EVERYONE thinks like us. There are always different viewpoints.

ScienceTeacher671

November 20th, 2009
6:21 pm

I agree with the comments above, particularly those by catlady, Cobb County Parent and Attentive Parent about obfustication. If you start with the two stated reasons that the results are not available, last year systems may have given the tests in different grades, but in previous years they did not, and results still were not published.

I find it hard to believe that Georgia state law requires nationally normed tests to be given, yet the General Assembly did not intend for some use to be made of the test results. Why are we requiring districts to give this test, and yet apparently making no use whatsoever of the results?

As the DOE spokesperson and others have noted, the ITBS is a test of basic skills, and by no means encompasses all of the skills we would wish for students at a particular grade level to master.

If our students were near or above average, it is hard to believe that the DOE would not be trumpeting our students’ results from the tops of the highest skyscrapers in Atlanta. Instead, the reasonable conclusion that might be drawn is that comparison of ITBS results with CRCT results shows (as comparison with NAEP results also shows) that Georgia’s standards are abysmally low. Indeed, during the last two years that our 8th grade students took the ITBS, those students who passed the CRCT with a minimal score were reading and doing math at between a 4th and 5th grade level, according to their ITBS grade-equivalent scores.

Aggregation of ITBS scores certainly makes more sense than aggregation of SAT scores, even though there are limitations on the use of the data.

catlady

November 20th, 2009
6:34 pm

Oh, Yeah! You are right, Reality 2: one year that item was on one form of the test twice–on social studies AND on science, as I recall. I think some of the GPS are quite hokey. We need to think: do our GPS reflect IMPORTANT knowledge/skills that kids should have? And, as another example of its problems, in 2nd grade virtually all the test is read to the kids. In third grade, suddenly, they are on their own. They only get to have the directions read to them. In another example, the CRCT seems to claim it is testing math, but first and foremost, at least in grades 3-5, it is testing READING SKILLS, not computation, not math concepts. I think there should be a computation section, as in some tests like the ITBS, to tease out if the student can manipulate numbers appropriately.

Regarding why the state’s argument that because different counties give the ITBS at different grades: I am not sure that parents care about how ITBS scores track from year to year. What they want to see is, if a kid has an 800 on the CRCT and a 14th percentile in the same subject on the ITBS, WHAT DOES THAT SAY ABOUT THE CRCT IN TERMS OF BASIC, IMPORTANT KNOWLEDGE?

Finally, if we think that the CRCT measures critical information/skills appropriately, WHY DON’T WE ENFORCE RETENTION FOR THE KIDS FAILING IT!? And if it doesn’t, why do we pretend it does? Has the DOE done data analysis to see if the kids who pass with an 800 (a manipulated number, by the way) are successful in the next grade but those with a 785 are not? ‘Cause if passing it or not passing it does not mean anything for future success, why do it at all?

catlady

November 20th, 2009
6:40 pm

Guess my 6:35 got vaccumed by the spam filter.

David S

November 20th, 2009
7:38 pm

Homeschool and never have to worry about this crap. You kid will get a better education, real morals, and learn how to be an individual rather than a compliant drone for the government.

Judy

November 20th, 2009
8:48 pm

Ed Johnson,
ITBS does include a raw score for each section of the test- I usually discuss raw scores with parents.

Reality 2

November 20th, 2009
9:43 pm

Since the CRCT problems aren’t released (in general), I will just have to take your word for it. So, what is the state bird anyway?

Again, though, for what purpose do you want to have the ITBS administered? To evaluate schools/teachers? I’m assuming that’s where you are going since if the only focus is for parental information, there is no need to get the aggregated state results, right?

If you were to use the ITBS to make any decision about retention, someone will have to set the cut scores. How do you set the cut scores? Does the ITBS developer have any data suggesting the minimum score below which the chance of the students doing ok in the following grade is not good?

Also, there is no elementary science or social studies ITBS tests, is there? Do we make all judgment about students based on language and math?

cut score teacher

November 20th, 2009
10:32 pm

I had an opportunity to be on a committee that selected cut scores for a CRCT subject. The process is intense with teachers from all around the state working with the test company making item analysis, data reviewing and recommendations. That is only one stage of the process. Please take the time to see how the cut score is determined and not just disrespect the process. If you have a research based way of determining how to best test Georgia students then by all means share that.

ScienceTeacher671

November 20th, 2009
10:33 pm

Reality2, the ITBS actually gives a better picture of a student’s skill levels, strengths, and weaknesses than the CRCT. For instance, rather than just a reading test and a language arts test, which is what you have with the CRCT, the ITBS results are broken down into subcategories such as reading comprehension, spelling, punctuation skills, etc. Some students might be very good at spelling and capitalization, but not so good at reading comprehension.

I agree that using the test for retention purposes might be problematic, but it’s not as if we’re really using the CRCT for retention purposes now. I’ve had regular education students who haven’t passed a single section of the CRCT in years sitting in 9th grade…and they haven’t been retained in a single grade either.

And, in case you haven’t noticed, according to NCLB, we do supposedly make all judgments about students based on language and math.

ScienceTeacher671

November 21st, 2009
8:49 am

By the way, the ITBS does have tests in social studies and science at the elementary level.

@cut score teacher, I have not been involved with the process of setting cut scores, but I would be interested in knowing why a committee with teachers from all over the state would agree that getting 45% of the answers correct demonstrates mastery of the standards.

Reality2

November 21st, 2009
11:52 am

I believe the NCLB says science and social studies will be a part of “report cards” — I can’t remember when. The last time I checked the ITBS web page, they said science and social studies are offered at grades 7&8.

As I said before, and you affirm, that the ITBS is a test of “basic skills.” If that becomes the “high stake” test, I can guarantee you that schools will be focusing on teaching for the ITBS – just the basic skills. I don’t discount the importance of skills, but there are many things that go far beyond just having skills. A skill focused science and social studies course may simply be reduced to memorizing names/terms and that’s what I hated about my school science and social studies.

Furthermore, if skills are the only things we teach, we don’t need teachers. We just need good curriculum materials, perhaps computerized learning modules, and adult supervision. Anyone will be able to “teach.”

ScienceTeacher671

November 21st, 2009
12:23 pm

Reality2, why is it that many students who are “proficient” on the CRCT, which you apparently think tests higher level skills, can’t do well on the ITBS, which just tests basic skills?

If you have the higher-level skills, shouldn’t you also be adept at basic skills?

ScienceTeacher671

November 21st, 2009
12:28 pm

According to the ITBS website, science and social studies testing begin at levels 7 and 8, which corresponds to first and second grades.

live.love.eat.

November 21st, 2009
2:29 pm

Reality 2 what do you wish to be taught other than skills?

Reality 2

November 21st, 2009
2:40 pm

Learn to reason, solve problems, communicate effectively, …

ScienceTeacher… I stand corrected. But exactly what are they testing in science?

Attentive Parent

November 21st, 2009
3:40 pm

In determining the cut scores for the CRCT and EOCT the state seems to think that the results should approximate a bell curve outcome. That makes no sense as you are testing knowledge and there’s nothing random about which schools or group of students do well on tests measuring knowledge and academic skills. It’s influenced by what is learned at home and especially by the teachers and curriculum at given school. Normal distributions result from independent factors and there’s nothing independent about the factors that influence how a student does on a test of knowledge like the EOCT or CRCT.

If we call the bell curve by its proper names – “the distribution of random errors” or the “normal curve of error”, the outcome from the raw data makes more sense. More importantly it lets us look to see which schools do well and find out what type of textbooks, instruction, and teacher training they are using.

A missed question, especially if the error is widespread, may be telling us that a concept is not being taught or that the predecessor knowledge was never properly learned. For example a question about irrational numbers could be missed because it’s not adequately addressed in the textbook, the worked examples were insufficient, or fractions were never mastered in arithmetic. A student who still thinks of fractions in terms of pizza will have trouble with irrational numbers.

The answer if there is a high percentage of incorrect answers is not to decide that the concept is not important and delete it from the Standards or decide the question needs to be easier to fit a normal distribution or not covered at all. All those incorrect answers should be telling us what must be covered more fully.

ScienceTeacher671

November 21st, 2009
9:21 pm

Students need to learn basic skills before they can “learn to reason, solve problems, [and]communicate effectively.” Unfortunately, too many don’t learn basic skills or the more advanced ones.

Reality 2

November 21st, 2009
9:48 pm

Science teacher,

I guess you and I just have to agree to disagree. Children, even before they enter a school, can reason and solve problems. Sometimes their logic isn’t quite valid from adult’s perspective, and they don’t have some skills to communicate effectively. However, saying that they have to “learn basic skills” before they can “learn to reason, solve problems, etc.” is a huge disservice to children.

ScienceTeacher671

November 21st, 2009
10:29 pm

Reality 2, if they can reason and solve problems even before they go to school, why do you say that’s what schools should be teaching them to do? And surely you don’t mean to suggest that students have no need of basic skills?

Reality 2

November 22nd, 2009
8:58 am

I never said kids don’t need basic skills – it’s just a part of schooling. I didn’t say kids’ reasoning and problem solving is perfect. Schooling is for kids to sharpen/broaden/deepen their ability to reason, problem solve, communicate, etc. Some, only some, of that can be accomplished through learning some skills. If you want to communicate through writing, you do need to learn to spell, use correct punctuations, etc. However, if you can’t put together the ideas logically and cohesively, then your writing skills will not help.

parent

November 23rd, 2009
10:30 am

Reality 2 – Who are you?

ScienceTeacher671

November 23rd, 2009
11:25 am

If you haven’t learned some basic facts and information, you won’t have anything to write about, and you won’t know whether or not someone has already solved the problem you’re facing, either.

Reality 2

November 23rd, 2009
12:30 pm

I see pre-school children talking with each other about so many different things. When you face a problem, one way to solve it is by trying to find out if anyone else has solved it. Another way is to solve it using what you know. I want schools to help students get better at the latter, without discounting necessarily the usefulness of the former. Re-inventing wheels may not be the most efficient approach, but sometimes it is important for children to experience the process of inventing, too.

parent,

I’m not sure what you want to know. I am a parent, too. I am not a classroom teacher, but I do know a lot of them. I do spend some time in classrooms as volunteers (usually elementary).

Reality 2

November 23rd, 2009
1:58 pm

That should have been “a volunteer,” obviously…