FYI on AYP: Federal law makes it harder for diverse schools

After our recent discussions about the adequate yearly progress or AYP standards established by the federal No Child Left Behind Act  — see yesterday’s media conference call with Arne Duncan on how it might change –  Jordan sent me this note:

I find it fascinating (especially after your most recent article) that very few people are aware how AYP calculations punish heterogeneous schools. The more diverse your school, the more sub groups you have. The more sub groups you have, the more you have to meet. It’s almost as if No Child Left Behind encourages schools to segregate. That hardly seems in line with the spirit of American education.

So too, the SWD (students with disabilities) affect AYP in poorly designed ways.

“Federal law requires states and local districts to improve the performance of students with disabilities on standardized assessments. The current measure typically being used to calculate an achievement gap uses the percentage of nondisabled students performing at the proficient and above level vs. the percentage of students with disabilities performing at proficient and above. This measure tends to obscure the demographic factors both within and between school districts that can affect the performance of students with disabilities. In addition, since a performance gap is often the defining characteristic of a disability on the individual level, particularly for disabilities such as SLD (specific learning disabilities), it may not appear particularly innovative or useful to establish an overall achievement gap between population of students with disabilities and nondisabled students in a districts.” (Goldschmidt 2009). Lecture notes can be here.

If you want to delve deeply into this issue, Jordan recommends this article from the Philadelphia Notebook on how the education law is tougher on diverse schools and this research paper. The Notebook piece is a great primer on the issue.

–From Maureen Downey, for the AJC Get Schooled blog

30 comments Add your comment

Pluto

August 9th, 2011
10:10 am

You mean all of these years that we’ve been told that our diversity makes us stronger (which I have never really understood what that means) have been a lie?

A Conservative Voice

August 9th, 2011
10:11 am

Sounds like a repeal of our forced integration policy would be a cure for our testing woes :) And, to all of you, I didn’t say we should not have integrated schools…….I said “forced integration” is not working, never has worked and never will work. Neighborhood schools work……forced integration of these schools does not!!!!!!!

John

August 9th, 2011
10:14 am

This is true. Look at the school disricts that always make AYP. Most are small systems like Bremen, Trion, and Chickamauga that have small minority student bodies and that don’t have to meet the standards for minority subgroups. That gives them an unfair advantage over more diverse systems that are likely doing a better job of educating their students. Every school system should have the same standards. If one school system is required to meet AYP for a subgroup of a certain race, every system should be–whether there are 2, 20, or 200 students in that subgroup. If one system is exempt from such requirements, all should be exempt.

sarahph

August 9th, 2011
10:27 am

Maureen… how about a blog/article on Georgia being one of the states applying for waivers under NCLB. This was announced yesterday. Good news!!!

Maureen Downey

August 9th, 2011
10:29 am

@Sarah, I did write about the waiver program. Maureen

Ed

August 9th, 2011
10:31 am

Put another way, maybe AYP should be based on one score for the entire school based on how ALL kids do, not multiple scores for multiple groups (with failure for any 1 group failing the entire school).

But maybe that’s just crazy talk.

sarahph

August 9th, 2011
10:36 am

Oops… sorry… I saw that after I posted. Lesson learned… always finish coffee before attempting to read blogs! :-)

sloboffthestreet

August 9th, 2011
2:21 pm

Someone please correct me if I am wrong. I was under the impression the reason the scores are sent back to each system before any data is released to the public is so each school can decide if a student can be added or deleted from a group depending on if their score passes AYP or not. If it is true that a system can move students in or out of a subgroup to attain their favorable AYP status, then the diversity appears to only be problematic when a system does not meet the standard.

How do schools or school districts make AYP?

To make AYP, each school and district must meet the following criteria:

95% Participation: Each school, as a whole, and all student groups with at least 40 members must have a participation rate of 95% or above on selected state assessments in Reading/English Language Arts and Mathematics.
Annual Measurable Objectives: Each school, as a whole, and each student group meeting the minimum group size must meet or exceed the State’s Annual Measurable Objectives (AMO) regarding the percentage of students scoring proficient or advanced on State assessments in Reading/English Language Arts and Mathematics. For AMO, the minimum group size is 40 or 10% of the students enrolled in AYP grades, whichever is greater (with a 75 student cap).
Second Indicator: Each school, as a whole and as subgroups, must meet the standard or show progress on a Second Indicator. For Second Indicator, the minimum group size is 40 or 10% of the students enrolled in AYP grades, whichever is greater (with a 75 student cap).
In defining AYP, each state sets the minimum levels of improvement, based on student performance on state standardized tests, that school districts and schools must achieve within time frames specified in law in order to meet the 100% proficiency goal. These levels of improvement are known as Annual Measurable Objectives (AMO) to ensure that all student groups, schools, school districts, and the State as a whole reach this goal by 2013-2014.

Georgia’s plan for AYP allows great flexibility in how schools can demonstrate AMO. There are four ways:

1. direct comparison of student performance to AMO;
2. confidence interval;
3. multiyear averaging and;
4. safe harbor.

If anyone would care to research and report back about these four variables allowed each system in order to demonstrate AMO, this would bring a new conversation to the front of the line. I never hear anyone discuss these four variables. Could it be that the state set it’s “Minimum levels of Improvement” too low for the first 10 years of the program, not demanding greater achivements from both students and educators and now find themselves with failure as their only option? The use of a state authored test along with a state set cut score would suggest this very possibility. Each state has been left to set and measure their own improvement. Just like magic, much of what we see and are told appears to be well disguised deception. Laws are in place to enforce the second indicator for elementary grades, which is attendence. High Schools second indicator was graduation rate. It is no wonder they have removed it in place of EOCT’s. I think it is less about diversity and more about the failures of the Georgia Public Education System as a whole. No one wants to take responsibility for anything just like our President. When there are awards to hand out and pictures to be taken, people are tripping over each other to make the shot shouting out their names to the reporter dressed in their Sunday best though. Funny how that works. Funny queer, not funny ha-ha.

Jordan Kohanim

August 9th, 2011
2:29 pm

Ed — OR we could get rid of AYP’s over-reliance on test scores and actually do as president Obama promised in his campaign speech–focus on MORE than test scores.

“On the campaign trail, President Obama declared, ‘We should not be forced to spend the academic year preparing students to fill in bubbles on standardized tests,” and he called for ‘a broader range of assessments that can evaluate higher-order skills.’”

Source: (http://fairtest.org/sites/default/file /root%20&%20branch%20fall%20-%20MN%20testing%20only.pdf)

Jordan Kohanim

August 9th, 2011
2:32 pm

Slob,
“Someone please correct me if I am wrong. I was under the impression the reason the scores are sent back to each system before any data is released to the public is so each school can decide if a student can be added or deleted from a group depending on if their score passes AYP or not.”

I have never heard of that practice, and I don’t think it is legal. Maureen, do you know?

I know, for example, our school missed AYP by 5 students.

Cindy Lutenbacher

August 9th, 2011
2:42 pm

Let us all try to remember that the standardized tests do not reveal learning, ability, or teaching effectiveness.

Also, with astounding consistency, the standardized tests have been shown to be absurdly biased toward the upper and upper-middle classes. They are not objective in any way at all. For example, when new questions are written and piloted, questions on which lower class kids score better than upper class kids are routinely tossed out; the questions on which the wealthier kids score better are kept.

We’ve been sold a very dangerous, very rotten bill of goods with the standardized tests. To me, they are analogous to the “weapons of mass destruction” that led the people of this country to support a war that had nothing to do with security and 9/11 and everything to do with oil. Standardized tests have nothing to do with kids and everything to do with throwing tax dollars into private corporations, everything to do with destroying public schools.

sloboffthestreet

August 9th, 2011
3:00 pm

Jordan wrote,

I have never heard of that practice, and I don’t think it is legal. Maureen, do you know?

I know, for example, our school missed AYP by 5 students.

Could someone please explain why the tests are sent back to each system for review before the scores are released then? Hey Jordan, It’s not the crime, it’s if you get caught. Or as a very wise man once said, “A nod is as good as a wink to a blind horse.”

sloboffthestreet

August 9th, 2011
3:18 pm

Cindy Lutenbacher,

Would you please elaborate on how you know this is true?

“Also, with astounding consistency, the standardized tests have been shown to be absurdly biased toward the upper and upper-middle classes. They are not objective in any way at all. For example, when new questions are written and piloted, questions on which lower class kids score better than upper class kids are routinely tossed out; the questions on which the wealthier kids score better are kept.”

Maureen Downey

August 9th, 2011
3:20 pm

@Jordan, I have never heard of that, either. Have sent Slob’s entire comment to DOE to ask for someone there to clarify.
Maureen

Jordan Kohanim

August 9th, 2011
3:26 pm

Cindy Lutenbacher

August 9th, 2011
4:44 pm

Dear Slob,
I must heartily apologize that I cannot find the link to the articles written by folks “on the inside” of testing, folks who explained this matter. I’m not sure, but I believe it’s fairly widely known that pilot-testing new questions involves making sure that the test-takers who scored high on previous tests are used as the “standard” by which new questions are chosen. Thus, it’s a self-feeding cycle of bias.

But that issue is only one of the multitude of problems with standardized testing.

long time educator

August 9th, 2011
5:03 pm

Test results are sent back to the system before they are made public to give the system a week or so to catch errors in student data or appeal some mistake in the numbers. Only mistakes can be corrected and must be proven to the state; such as a mistake in a student number, gender, race, or special ed classification. We also double check that those who fail did attend a full academic year from the first FTE count to the last. As a principal I was always sent the numbers to check; I never found a mistake. I think my system may have appealed some error one year at one school.

long time educator

August 9th, 2011
5:23 pm

I have said it before, but after 9 years of NCLB, we see alot of unintended consequences. Way too much time is spent in massaging these numbers, verifying them, and figuring out legal tricks to keep making AYP (make sure your school has less than 40 of the offending subgroup, if possible). My school had over 70 SWD bacause we had a countywide homeroom for a particular disability, but other elementary schools were able to stay just under 40 students and make AYP even though their subgroup scores were lower than the goal. They had a real incentive to transfer SWD out if they could. I guess the incentive with ELL and racial subgroups would be similar if the numbers hovered around 40. It is ALL such a waste of time when we are supposed to be about educating children, but this is how it has evolved. No one wants to make the paper as a failure and the general public only read the headlines, not the details about subgroups.

A Conservative Voice

August 9th, 2011
5:35 pm

@Jordan Kohanim

August 9th, 2011
2:29 pm

“On the campaign trail, President Obama declared, ‘We should not be forced to spend the academic year preparing students to fill in bubbles on standardized tests,” and he called for ‘a broader range of assessments that can evaluate higher-order skills.’”

Yeah, another one of his “LIES” to get elected…….remember November 6, 2012

Jerry Eads

August 9th, 2011
5:49 pm

Hi all
@slob, nicely done, that said knowing we have our disagreements. THAT said, I’m now not in the middle of it and can’t do you a decent answer that would do your post justice. We’ll see what the state has to say.

There have been and continue to be some pretty cute moves to, um, influence how AYP comes out at the local level. BUT the law is complicated enough (mixed with a great deal of ‘interpretation’ by fed ed) that it takes pretty sophisticated gaming to come out ahead. VERY much like tax law but, unlike tax law, there’s usually insufficient resources to play.

@Cyndy (& @slob) – at the individual student level, any test – teacher or multimillion $ CRCT/EOCT/GHSGT/etcT or something actually well made like the SAT or ACT (or ITBS or CTBS, etc.) – are potentially stupendously inaccurate, because they are dependent virtually entirely on a WIDE array of influences – only one of them being the cognitive skill of the test taker. That averages out (gets more accurate ), as research has shown over and over and over and over again, only a TINY bit with a classroom full of kids, and a tad more at the school level with, say, a hundred or two hundred kids. My own analyses (which I couldn’t publish) showed that most (WELL over half) of the variation in (that is influence on) pass rate on the state tests was a function of whether or not a kid was on free or reduced lunch. Only a tiny bit of the variation in test performance could possibly be a function of teacher effect. MANY dozens of published refereed (meaning, likely legitimate) research reports say the same thing with other tests, including the SAT and ACT. Bottom line: money talks, especially for test scores.

Jerry Eads

August 9th, 2011
5:51 pm

oops, sorry, meant to note that’s why teachers take MANY dozens of measures in a year on a student – quizzes, observations, notes, tests, etc. etc., knowing that any ONE test by itself is just about totally worthlless.

Lee

August 9th, 2011
8:12 pm

No, there is an IQ deficit that the “diverse” students bring to the table.

MS Principal

August 9th, 2011
9:39 pm

Long time educator is right, the data gets sent back to make sure that it is accurate. The data for each student is pulled/sent electronically from the student information system of each district. Sometimes students are mislabeled with a subgroup label or they are counted FAY when they shouldn’t be or aren’t counted when they should be. The subgroup minimum of 40 does harm schools with small subgroups, but it is statistically not practical or viable measure to use a group smaller than 40 for statistical purposes. In addtion, in GA, if you have between 400 and 750 students who count for AYP grades, your smallest subgroup size is ten percent of your total population. So if you have 500 students, then you subgroup must have 50 kids to be counted for AYP. This does cause some less than scrupulous districts and schools to massage numbers so that there are not subgroups. I know, also, that many school in GA have made AYP because they do not have enough SWD to make a subgroup.

The three “second looks” that slobofthestreet mentioned are very important ways that schools can meet AYP. Confidence interval, Multi-Year Average, and Safe Harbor are ways that schools can make AYP by subgroup if they don’t make the AMO or absolute bar. The first is a statistical measure looking at average expected growth and attaching a z value. It is designed to give schools the benefit of the doubt with student cohort groups changing from year to year and allowing for natural variances. The multi-year average allows a school to have a slighlty down year mixed in with a long history of successful years. The final calculation, Safe Harbor, is when schools decrease the percent of students who Did Not Meet by ten percent from the previous year.

It is confusing and requires a lot of study to figure out how it works. This is clearly time better spent on accelerating gifted students and remediating students in need of help.

Jordan Kohanim

August 9th, 2011
9:49 pm

MS Principal- Thank you so much for clarifying! It definitely gets confusing.

You said:
“I know, also, that many school in GA have made AYP because they do not have enough SWD to make a subgroup.”

That is true, but it brings up substantial ethics questions.

Would schools who then take on SWD (Students With Disabilities) have a higher probability of not making AYP? Doesn’t that then discourage schools from accepting ALL students? Does that not also fly in the face of the goal and purpose of education?

@Jordan

August 9th, 2011
10:31 pm

That’s a large reason why teachers suspect the RTI process was put in place…to keep kids from an official SPED designation. It’s not written that way, but it sure ends up that way. The hoops to jump through to get a kid tested is ridiculous, and the paperwork mind-numbing and confusing. If a kid doesn’t have a strong parent advocate, they will likely get passed over and moved on. Sad, but true. Bureaucracy at its finest.

Jordan Kohanim

August 9th, 2011
10:42 pm

@@Jordan

That is sad.

You know, with the experience and training I had with RtI (Response to Intervention for those who don’t know), I never understood why it is implemented in high school. Wasn’t it primarily designed for elementary schools ?

Jennifer

August 9th, 2011
10:42 pm

In any of your research did you find an assessment system that identified different racial subgroups to different %s of proficiency on the assessment tool/test being used ?

CRCT Hater

August 9th, 2011
10:54 pm

I’d love to see a public school that is solely for SWD. I personally found the high achieving middle school my child attended a complete joke. After the course work was watered down to parts per million to accommodate the IEP, the only focus was preparing for the CRCT. I found it crazy that my child was ‘too smart’ for the CRCT-M but not smart enough to be main streamed and did not pass the science and math sections. After cramming my 8th grader for a week prior to retaking the Math CRCT, the school excused her (and other students) from any other final exams while others students went through the normal end of the year process. I think it would just be easier to separate SWD and exempt them from these types tests since so much time is spent accommodating their needs.

On another note, I found it quite interesting that the first Math CRCT score was 786 and then passed with 800 the second time around. My child’s scores have never been over 812 on any portion of CRCT tests. Hmmm.

Jennifer

August 9th, 2011
10:57 pm

All RTI folks – google “RTI & Dear Colleague Letter” from the Office of Special Education – you are looking for the one published in 2011. You will see your answers there. And @ Jordan – you are so right.

Subgroups – keep in mind the rule is 10% of your student population or 40 whichever is GREATER. Only one Gwinnett County High School actually had subgroups large enough in SWD to even count on AYP – so don’t listen to all the bemoaning about SWD issues in high school. We are by far the largest school system – so I am pretty sure that only a handful of high schools across the state have subgroups at the high school level that even count SWD.

I analyzed subgroups at the high school level 3 years ago and you would be surprised at how many of our high schools just ‘missed’ the subgroup size by one or two or three. Ironic, no ? There is nothing funny about this at all – I would rest my hand on a stack of bibles that schools are gaming the system when it comes to subgroup size.

carla roark

August 11th, 2011
3:03 pm

I am trying to find out what AYS (the intials themselves)stand for I am doing this for my son who is very busy with work he is looking for info on this particular organization. Also need info on AYS and SWD. My children have been out of school for years now and I am not fimilar with any of the new organizations available for our children. I would appreciate a quick note back. Thank you for your time and concideration.

carla Roark