AJC addresses question over whether its national test score investigation considered student mobility. It did.

In the Washington Post Answer Sheet blog, Gary Miron, professor of education at Western Michigan University, questions whether the AJC investigation into test score disparities nationwide considered student mobility.

Reporter Heather Vogell, a member of the AJC investigative team into test scores, responds here to that concern:

By Heather Vogell

Some school district officials and education consultants have raised the issue of whether high student mobility would lead a district to be highlighted in our analysis even if they had no cheating problem.

A high rate of mobility is a characteristic of virtually all inner city high-poverty districts. If it were true that our methodology just flagged mobility instead of potential cheating, then you would expect all urban districts with high mobility to be flagged.

This was not the case. For example, Cleveland schools, with a better than 30 percent mobility rate, had an average 4 percent of classes flagged by our analysis in 2008-2011. Statewide, about 5 percent of classes were flagged in those years. Chicago, Fresno and Amarillo, Tx., are other examples of districts grappling with high mobility which did not have high concentrations of suspect scores.

Before publishing the analysis, we also looked for large changes in the percentage of poor students. They were so rare we did not eliminate any data because of them. We also looked for changes in the number of students in a grade that were greater than 25 percent, suggesting a massive change at a school. We did remove from our data those with such big changes, which amounted to about 6 percent of classes.

University of Georgia statistics professor Jaxk Reeves, who is also director of the Statistical Consulting Center, reviewed the AJC’s analysis and said that absent a radical change in the makeup of a school — such as a sudden, large influx of poor or wealthy students — mobility should not have a major impact on how a district fares in the analysis.

Atlanta school district officials also raised this issue when the AJC first began analyzing their scores. The newspaper took an in-depth look at student mobility in Atlanta and in other, similar districts nearby. It showed that the other districts did not have nearly the same concentration of suspicious scores as Atlanta, despite having similar populations of students

–From Maureen Downey, for the AJC Get Schooled blog

28 comments Add your comment

Ed Johnson

March 27th, 2012
2:58 pm

While congratulations are in order for a job well done, the work must be able to pass the muster of Donald J. Wheeler, Ph.D., IMHO.

AJC may contact Dr. Wheeler at,

SPC Press
5908 Toole Drive Ste C
Knoxville TN 37919
(865) 584-5005
http://www.spcpress.com/
spcservice@spcpress.com

“Dr. Wheeler has been teaching people how to use their data more effectively for over 40 years. He was a student and colleague of Dr. [W. Edwards] Deming for 21 years and is a recipient of the Deming Medal. He is a Fellow of the American Statistical Association and also a Fellow of the American Society for Quality. He has conducted over 1000 seminars in seventeen countries on five continents. He is author or co-author of 25 books and over 200 articles. He has been a columnist for both Quality Digest and Quality magazines. Through these seminars, books, and articles he has had a profound impact on companies and organizations around the world.

“Dr. Wheeler graduated from the University of Texas with a Bachelor’s Degree in Physics and Mathematics, and he holds M.S. and Ph.D. Degrees in Statistics from Southern Methodist University. From 1970 to 1982 he taught in the Statistics Department at the University of Tennessee where he was an Associate Professor. In 1982 he formed Statistical Process Controls, Inc. and has worked as a consulting statistician ever since.”

Some of Dr. Wheeler’s highly accessible and acclaimed books are:

* Understanding Variation: The Key to Managing Chaos
(this one, short and sweet, will rock your world!)
* Building Continual Improvement: A Guide for Business
* Understanding Statistical Process Control
* Making Sense of Data

Beverly Fraud

March 27th, 2012
3:54 pm

We all know the entire focus of the AJC’s investigation for the past several years was to discredit Beverly Hall, because they were jealous of her successes.

Andy and Shirley will tell ya!

Beverly Fraud

March 27th, 2012
3:56 pm

That some people might read the above post and think it is serious goes to illustrate the level of denial some have about this issue.

Cheating is MASSIVE, cheating is WIDESPREAD, and what we are doing in education is NOT working.

But are we willing to change?

Ed Johnson

March 27th, 2012
4:05 pm

@Beverly, if you will, please describe again the application of rigor. Recently I tried quoting you but ended up flubbing it so badly I had to back off.

Beverly Fraud

March 27th, 2012
4:35 pm

Well of course Ed, any true discussion of the rigorousness of rigor must include the rigorous application of jargon, paying particular (rigorous) attention to applying said jargon in a STRICTLY dogmatic manner, absent of any critical thinking (though ironically, when applying jargon, one must rigorously use the term “critical thinking” in any and all discussions of rigor.)

And it goes without saying, to create the current climate of rigorousness we have achieved, one must rigorously, even studiously, avoid the application of common sense to the thought process.

Dr. Craig Spinks/ Georgians for Educational Excellence

March 27th, 2012
6:31 pm

Have the apologists for the miasma engulfing GAPubEd begun: To resort to “ad hominem” attacks against those who question the efficacy and integrity of their system? To attacks upon the desirability of the use of nationally-normed tests to measure our kids’ basic academic skills? To attacks upon the efficacy of such academic measurements? To elicit sympathy for the folks whose cheating/”mistake” in CRCT administrations has called into question the value of all educational records maintained by our state’s elementary and middle schools?

Prof

March 27th, 2012
7:25 pm

@ Dr. Craig Spinks: You always provide the moral perspective on this terrible unfolding debacle. Thank you.

Really?

March 27th, 2012
7:26 pm

@Beverly Fraud.. please take you meds.

jezel

March 27th, 2012
7:48 pm

Spinks….Educational Excellence in Ga. ….you should be fired.

jezel

March 27th, 2012
7:50 pm

maybe we need more more smiley faces….right MAUREEN

brad

March 27th, 2012
10:01 pm

Great job Heather. Thank goodness you didn’t listen to the humanist Maureen and just take at face value that Atlanta was succeeding because money was being spent and the scores looked good. Maureen thought Hall and money could make any kids excel but we now know beyond any doubt that broken homes, welfare, parental neglect are more important to learning than liberals think. Secular humanism has destroyed many schools.

bootney farnsworth

March 27th, 2012
10:13 pm

some of the most mobile kids in America, military kids, tend to do better than their civilian counterparts.

but military kids are raised with a understanding of disipline

catlady

March 28th, 2012
7:05 am

bootney, while what you say is probably true in the grand scheme of things, some of the most messed up kids I ever taught had a daddy in the military or a daddy who was a preacher. Now, I do admit this is a small sample, at lease in terms of the military daddies; in our area, any man without a job can declare himself “called” and open up a church and take the tithes and offerings for himself! LOL but sadly true

teacher&mom

March 28th, 2012
7:14 am

Larry C

March 28th, 2012
8:52 am

For me at least, this is an inadequate answer to the question of how student mobility impacted the analysis.
It is an error to assume that high mobility would mean that all schools with high mobility would have been flagged. In statistical terminology this is a variance issue. The Bureau of Labor Statistics in their monthly labor force surveys ensures a 75% overlap in sample households for the explicit purpose of reducing the variability on month to month change estimates. A smaller overlap would result in greater variability in the estimates they produce of the month to month change in the nation’s unemployment rate. In the same way with greater mobility there will be a greater variability in the change estimates as you follow the “cohorts.” Unless that variability is accounted for explicitly in the models it will impact the results.
What does this mean? Unless mobility is included in the modeling process schools with greater mobility would be more likely to be flagged. They would be just a likely to be flagged for a decrease as for an increase. But a variance increase does not mean that all schools with high mobility will be flagged. Just the probability of being flagged is increased. How big an increase this will have depends very much on how much variability there is in the student mobility rate among the schools in the study.
To single out Cleveland to make the point that mobility does not matter is an error. Statistical cases are rarely made on the basis of only a few examples. You have cited Cleveland because it supports your hypothesis. But if Cleveland had not turned out so well it would have happened in another school district and you would be citing that school district for your example. In a statistical analysis there are almost always cases that will support a given claim.
I am not saying student mobility caused school districts to be flagged. I am just saying you have not made the case that this is not true. I am curious why Professor Reeves believes mobility is not an issue.

AJC is NOT Credible

March 28th, 2012
9:00 am

LARRY C…YOU.ARE CORRECT…IF THEY ARE NOT TRACKING THE STUDENTS…INSTEAD THEY LOOK AT.THE NEXT GRADE…THOSE COULD BE TOTALLY DIFFERENT KIDS…INNOCENT TEACHERS ARE GOING TO GET CAUGHT UP IN THIS MADNESS WITH THE WRONGDOERS.

A Conservative Voice

March 28th, 2012
9:21 am

@Beverly Fraud

March 27th, 2012
3:54 pm

We all know the entire focus of the AJC’s investigation for the past several years was to discredit Beverly Hall, because they were jealous of her successes.

Actually, IMHO it goes a little deeper than that, BF. I think most of us (make that some of us) (OK, a few of us) know fully well that the AJC has slipped down to the lower rungs of major metropolitan newspapers. The owners knew they needed a boost, saw an opportunity with an expose’ of cheating and went for the jugular. They should be applauded for the fine, professional journalism exhibited; however, I’m still not convinced that their motives had anything to do with helping the multitudes and a generation of school children who will suffer for a long time. I think it had more to do with saving their *** and receiving awards and platitudes for their reporting to increase their readership and standing. Please forgive my negativity, but I have seen the City of Atlanta and the AJC go from being a great city and newspaper to struggling to attain mediocrity.

Dekalbite@ AJC and Larry C

March 28th, 2012
3:09 pm

While it is true that on a classroom level for remediation purposes, it makes no sense to compare your students from this year to next years students, that’s not true on a macro or system level.

If for example, the 4th graders in your school system year after year cannot compute double digit multiplication or division, fractions or decimals, then that shows a problem with the content mastery of math for 4th graders. Most students will score very low on the 4th grade criterion referenced math test if they cannot perform these math functions that are taught in almost every 4th grade classroom. That is actually what happens when school systems have a problem with content mastery for students. The students will not be able to compute these problems correctly so they will “not meet expectations” (which are that in the 4th grade they should be able to solve problems that require double digit multiplication and division and convert fractions to decimals, etc.). And the greater the universe (number) of students tested, the less variance you would expect. That’s just statistics.

My understanding from listening to the video is that within each school system, the AJC looked school by school at – for example – the variance between what students in a particular school scored in 4th grade and then what they scored in 5th grade in math and reading. If the variance (increase or decrease) was beyond an expected level (statistically computed using a very large universe of students), then they “flagged” it as being outside the norm or an outlier. They looked for large increases or large decreases from year to year.

This is EXTREMELY important for teachers who are facing paychecks based on test scores.

For example, 40% of our 4th graders come into my class and other 4th grade teachers’ classes reading on a 2nd grade level from 3rd grade, and when they leave our classes, many of them score on the standardized reading test like they are on a 5th grade level (big increase in scores). They have advanced far beyond what they would be expected to advance based on other students that far behind. Now the next year in 5th grade when they take the reading test, most of these 40% score on a 2nd or 3rd grade level (big decrease in scores). The principal asks the 5th grade teachers how they fell behind 3 years in reading with these students who had such problems in 1st, 2nd and 3rd grade but most of whom were completely remediated by the 4th grade teachers. If any 5th grade teacher says the scores from 4th grade were not valid, the principal punishes that employee. He only wants them to say – next year the scores will be very high. No excuses.

Some of the scores and actions of the outlier schools in APS were just that egregious. What are the statistical probability that those 5th graders would regress 3 years in reading over a 10 month period of time? And what if this happened consistently every year? Would you blame the 5th grade teachers? That is what happened in APS. Those 5th grade teachers had pressure to ratchet up the scores as well to hide what was happening in the 4th grade. Because cheating was taken off the table as a possible reason (and penalties were placed on anyone who said differently), it could only be the 5th grade teachers who were pulling the students down. So the pressure on the 5th grade teachers became unbearable. If the principal says it’s the 5th grade teachers, and lets a few go (now they are unemployable anywhere so he has ruined their careers), then the others will feel the pressure to do exactly what the 4th grade teachers are doing – ensure the scores are up appropriately – or they know they will be let go as well and no one will hire them. When they go to the Central Office to complain, they are punished or terminated. When they go to the newspaper, the Central Office hears of it and so they are punished or let go. When they go to the state, they are punished or let go. The business community has a lot vested in high test scores to attract other businesses so they place pressure on the state and the newspaper and anyone who dares question the “miracle”. The Board of Education does not want to hear this either. If anyone goes to them, the Central Office will make sure they are punished or terminated.

Meanwhile, that principal who is busy ensuring the score are looking better and better by purging his school of anyone who doesn’t go along is going year by year hoping that he will get a promotion and end up in the Central Office as a coordinator with the same or better pay but no test score responsibility. If he is lucky, he may parlay that position into a higher position in another school system – say an Area Director or a Deputy Superintendent in that school system or another school system where the big pay, perks, and prestige are. Now he is far enough removed that he doesn’t have to dirty his hands. He can truthfully say – I upped test scores in my former school district and you should be able to do the same. Advancement is the name of the game, and many of them hope to be somewhere else when the balls they are juggling fall to the ground.

The AJC obviously had statistical mathematical models developed to handle the massive amount of data, but really the concept is fairly simple. Outliers in any school need to be looked at. Testing needs to be valid and secure. Expected advancements in scores need to be realistic.

Instituting teacher compensation based on invalid data is a recipe for disaster for students. That’s why this study is so important for children.

bu2

March 28th, 2012
7:12 pm

Bootney
There have been studies that show that kids who move more, not just military kids, tend to do better.

Now I read about those studies a while back before we had so many migrant kids who move 3 times in a school year. I doubt that is helpful. But moving each year can be stimulating to a young mind (even if its difficult socially).

Dekalbite@bu2

March 28th, 2012
10:09 pm

Could you provide some links to those studies? That’s was not my personal experience over 30 years of teaching in a number of different schools – low income, middle income and affluent.

Teach2Learn

March 29th, 2012
6:53 am

Dekalbite@ AJC and Larry C
March 28th, 2012
3:09 pm

Excellent description about how it all can play out.

Being Censored by @Maureen

March 29th, 2012
8:34 am

You are being misled. The response from the AJC does not fully address Gary Miron’s concerns. I have a list of TONS of articles from researches who are bashing the AJC study is incomplete, misleading, and sensational an issue that should not be sensationalized. But your author will not print them.

Anytime Ms. Downey or Diane Ravitch have an opening to steer the education reform debate to the wrong issues, such as high stakes testing, they take it! Testing is only a PART of the problem, not the root cause of the problems plaguing our public schools!

Standardized tests are not the main problem of our public education system – the problems are much more complex than that. But the AJC has just inflicted serious damage to a system in need of systemic change. By continuing to respond out of “fear,” it is clear that real reform in public education will never happen, not until the “digital natives” are the ones making policy decisions.

[...] Get Schooled blog. A response to criticism on their handling of the issue of student mobility is posted at that blog. My primary concern is that the methods have not been disclosed in sufficient detail to evaluate [...]

Being Censored by @Maureen

March 29th, 2012
4:17 pm

With all due respect, Mr. Riley’s comments on this issue, as editor of the AJC, do not serve the paper well. This reckless disregard for safety and security has put the entire newspaper’s future on the line. His opinion doesn’t matter to me, because it is not coming from an independent third party. Nashville has already bashed the methodology with solid facts, as have others, and, as it relates specifically to Dr. Miron’s concerns about the methodology, the AJC has not refuted each and every point that Dr. Miron makes – only the mobility issue.

This study should NOT have been released until it was bullet-proof. I have worked in the media industry for many years, and you NEVER release a study to the public unless it is BULLETPROOF. There are several problems with the AJC study, and it has served to create a major panic in the public education system, and distract the nation from the systemic dysfunction in public education. Standardized testing is an issue, but NOT the root cause of what plagues our schools. Just because a study unraveled APS doesn’t mean the same variables are relevant in other markets.

I will be advising a colleague to author a blog post on a nationally recognized blog that unravels the entire study for the world to see.

For the first time since I moved to Atlanta, I am embarrassed to say I live here.

Being Censored by @Maureen

March 29th, 2012
6:40 pm

So Maureen, are you going to censor this one too? Seems that the AJC is already backing down from its methodology. Nashville has already reported that any references to Two Rivers Elementary School have been removed from the original story. Seems they’re onto something, and pretty soon, the AJC will have to issue a retraction on the entire piece.

Shame on you! All of you.

Being Censored by @Maureen

March 29th, 2012
6:45 pm

To all readers of this blog. Follow this link to see that the AJC is study will be retracted very soon because it is materially flawed. http://oldcenteres.mnps.org/Page76595.aspx?Panel=Listing&ListingType=tag&tag=atlanta

Albert Crosby

March 30th, 2012
8:52 am

The simple claims about how mobility appears to impact the results of the study are not enough. Without information on how the various states released or summarized data, we cannot know if results of mobile students were even available to the AJC analysts. Some states do not include the scores of mobile students in their score summaries. Others calculate more than one measurement, but use only one.

The model, as described, assumes that looking at Wayside Elementary’s 3rd grade in 2009 and their 4th grade in 2010 is looking at the same students. In many districts, it isn’t. This is another meaning of mobility — students who change schools inside a district. It happens because people move, overcrowding, classroom balancing, construction, redistricting, and a host of other reasons.

I’m at least equally curious about why the AJC analysts used simple linear regression at a statewide level. This implies that the issues traditionally considered barriers to learning, such as free/ reduced participation, limited English proficiency, mobility, special education, ethnicity and race are homogenous at a statewide level. Extremely unlikely. And it’s fairly easy to show that such differences between school districts, schools, and even classes within a school have a statistically significant, even substantial influence on predicting the mean scaled score on a benchmark exam.

Some states also use criterion referenced tests for measuring growth. This means that the 3rd grade test focuses on 3rd grade curriculum standards. 5th grade tests focus on 5th grade standards. While not an excuse for settings in learning, it does counter the journalist’s statement that learning rarely disappears. That may be true, but learning multiplication tables by rote doesn’t always adequately prepare a student who has limited English skills to solve word problems or do algebra. We hope — and even expect — that a strong early performance in literacy or math translates into sustained strong learning. But I’ve seen too much student data to believe that’s true across the board, especially when the standards (necessarily) shift to expecting higher level thinking skills in 6th or 7th grade.

Since I don’t have access to the data supplied by my state to the AJC, I can’t even begin to replicate our revere engineer their study. But it does give me a lot to think about and presents several new studies to run.

I do think, that especially for districts with changing demographics, it’s easy for their “cohorts” as defined for this study to vary from the growth predicted by a simple statewide linear regression model. Thus, it’s not right to label districts as cheaters in the public’s eye on such a limited-depth study based on such a simple model.

I hope that it’s not an attempt to enjoy journalistic accolades while educators try to figure out what you did, and explain “this doesn’t prove cheating, but…” to angered patrons.

Being Censored by @Maureen

March 30th, 2012
9:20 am

Great points, Albert. Maureen has censored my blog posts that provide links to other sources that have completely invalidated their research methodology. I know from experience that you don’t release a study to the public unless it’s BULLETPROOF.