Now, Nashville issues rebuttal to AJC national testing probe

With the AJC series on national test score disparities minutes away from publication, a second  district with implicated  schools has chosen to issue an preemptive strike. Houston released a statement Friday.

The series is now online at AJC.com

From Nashville Metro Schools:

Metro Schools has several, significant concerns with the data presented by the Atlanta Journal Constitution regarding our district and believe that Nashville has no place in the story they have prepared. Some of those concerns include:

• The AJC analysis assumes that students in one grade level at a school one year are the same students that were there the previous year in the previous grade.

• Our district has mobility rates between 35%-40%.

• Zoning changes have impacted student enrollment.

• A significant number of ALCs (Alternative Learning Centers) and special schools (and “homebound”) are flagged in our data; these schools often have very fluid populations – ALC populations have 100% annual turnover.

• This AJC information flags schools with unusually high numbers of discrepancies from predicted scores – both high or low.

• 77% of the cases flagged in MNPS were due to low performance (significantly below predictions).

• The negative scores we saw were not part of a pattern of high gain one year followed by significant loss the next (as would be expected if cheating was involved).

• There was not a single case of unusually high gains flagged for 2011.

• Obvious errors in data provided where children who were absent from testing were assigned a “zero” rather than being excluded from the analysis – resulting in average scores that were below the minimum score possible.

• Tennessee adopted much more difficult academic standards in 2010, resulting in a significant decrease in student test scores statewide.

• The AJC methodology will automatically identify 5% of the cases/classes analyzed statewide, but schools with changing populations (see previous bullet) or higher than usual numbers of highly effective or highly ineffective teachers (research supports that teacher effectiveness make the greatest difference in test score gains) are likely to have higher percentages.

• Dozens of districts in Tennessee had higher percentages of classes flagged in a given year than did MNPS or Memphis, but those districts had too few cases to show statistical significance; the methodology is much more likely to identify only larger school districts.

• Metro Schools has the highest population of English Learners in the state, with approximately 1 in 4 of our students coming from non-English Language Backgrounds. We also educate about 30% of the state’s EL students.

• National data show EL students typically make among the largest gains from year to year as they learn the English language; several of the schools flagged for positive gains have high EL populations.

• Regression to the mean could be a factor in why some of the relatively highest (e.g., Meigs) and relatively lowest (ALCs) scoring schools are flagged.

• The Tennessee Value Added Assessment System (TVAAS) uses much more sophisticated analytical techniques (Henderson’s mixed model methodology) and follow’s actual cohorts of students to analyze gains of teachers, schools, and districts and TVAAS results have typically been comparable to statewide trends for several years.

• Analysis of test gains of MNPS middle school math teachers over three years (2006-07 through 2008-09) by Dr. Brian Jacobs of the University of Chicago, on behalf of Vanderbilit University, did not show any unusual or suspicious patterns.

• The analysis was conducted as part of the Project on Incentives in Teaching (POINT) study, to determine whether tying significant financial incentives for teachers (up to $15,000 per teacher per year) to student achievement made an impact.

•The analysis conducted by Dr. Jacobs was specifically to determine if evidence existed of invalid test scores (cheating).

•Dr. Dale Ballou, Associate Professor of Public Policy and Education at Vanderbilt University, is prepared to speak to this study. We have also shared the data AJC provided with Dr. Ballou and he has found some statistically impossible anomalies in the data. He is also willing to speak to these findings.

• The district takes test security very seriously

• Meetings with principals and test coordinators are held annually prior to TCAP testing and test security guidelines and procedures are reviewed.

• Principals and test coordinators annually provide training to teachers and other staff involved in testing.

• Every individual involved in test administration and handling is required to read and sign the district’s State-Mandated and District-Wide Test Security Agreement form, which highlights key test security guidelines and includes the state law regarding test security (stating that violations could lead to immediate suspension and potential dismissal).

• Test administration is monitored in a sample of schools each year by both district staff and the Tennessee Department of Education; annual reports from the TDOE have been positive.

• All allegations of test security violations are reported to the Tennessee Department of Education and investigated jointly by district staff in the departments of Research, Assessment & Evaluation and Human Resources.

• Action is pursued when violations of test security are confirmed, and educators have been suspended or terminated due to these violations

–From Maureen Downey, for the AJC Get Schooled blog

15 comments Add your comment

Digger

March 24th, 2012
11:51 am

They doth protest too much.

Scott

March 24th, 2012
12:08 pm

Having teachers test their own students is just asking for trouble. Most teachers are honest, but not all… as we saw in APS. Same goes for administrators, who can get focused on politics instead of actual student learning. So, who can you trust? Only an independent third party. This is why national standards and independent testing might be helpful… to keep the foxes out of the hen house.

Steven

March 24th, 2012
5:39 pm

Compare this response to Houston’s response – Houston responded by admitting that problems have occurred, and the district has a commitment to putting policies in place to make sure it doesn’t happen again and that teachers are held accountable.

Nashville is just making excuses.

Ron F.

March 25th, 2012
12:19 am

“Our district has mobility rates between 35%-40%”

And places like Baltimore, Houston, Atlanta, and basically any major city doesn’t? I imagine, given the recent economic meltdown, that most school districts have seen an uptick in the percentage of students moving around, both within and out of districts. Ironically, they shoot themselves in the foot with that line. If a child moves within the district, which I would assume would count as “mobility”, shouldn’t the effect of that be pretty neglible if the entire district follows the same curriculum?

Beverly Fraud

March 25th, 2012
7:27 am

Nashville’s (unspoken) response:

Office Depot had a sale on ERASERS. Whaddya expect us to do, NOT support our local bidness?
(Besides, they promised if we got caught, they’d help us “finesse the investigation past the governor.”)

They didn’t say nothin ’bout the AJC! Waaaaaaa!

Dave Shearon

March 25th, 2012
8:22 am

As a former school board member in Nashville and someone with a lot of background in TN’s accountability system, http://www.shearonforschools.com/TVAAS.html, I suspect Nashville’s more right about this that AJC. Analyzing school performance from test scores is difficult – that’s why TVAAS (or EVAAS as it’s called in other districts and states) exists. It can help school and policy leaders ask good questions and assess progress. But it also does a great deal to protect against both test design screw ups and cheating.

Over a decade ago, New York was going to fail a lot of kids because their reading scores were too low on a nationally used standardized test. TN used the same test, but the TVAAS analysis showed a systematic downward deviation in reading scores that was traced to a mistake in form design by the tester. New York ended up having to back off their move, and TN’s scores were released late that year because the national testing company had to go back and revise.

AJC would do well to pay attention to what Nashville is saying. I suspect the statistical tools and analytical approaches they are using are, as one independent reviewer has said, still “light years” ahead of anything else in the country.

Brandy

March 25th, 2012
1:54 pm

@Maureen, Has Baltimore or any other major district also issued a rebuttal (other than Houston)? I know for a fact that both schools I taught at in Baltimore were flagged in previous analyses, though one was shut down due to facility maintenance issues.

AJC is NOT Credible

March 25th, 2012
5:01 pm

I think Nashville is right…the data is flawed…Look at AJC’s database for APS…2010 has the highest flagged schools yet, that was the year with the highest testing procedures and the state helped monitor the test…I emailed several people at the AJC and received no response. Explain Maureen.

Stop Stealing Dreams

March 25th, 2012
8:51 pm

Shame on the AJC for trying to find a way to exonerate the APS Cheating Scandal. I applaud Nashville Metro Schools for responding so quickly to refute the allegations. I have spent time with their most senior educators and public policy officials. From my research, Nashville has had one of the most remarkable turnarounds of any urban school district in the United States, and they are doing all the right things. While I am still in the process of reviewing this article, I felt compelled to write a note of support for what they are doing.

I am embarrassed for the city of Atlanta, that this publication has written the story it has, and I look forward to writing a much more detailed, research-driven critique on my own blog, because I will not give this publication or any of its blog authors such respect, since they were certainly more than willing to disrespect other urban school districts without ANY direct evidence of cheating.

The AJC has singlehandedly destroyed its brand and credibility. This story was disgusting to read.

Maureen Downey

March 25th, 2012
9:24 pm

@Stealing, I should know better than to attempt to respond to you with actual facts, but for the record:
A team of three AJC reporters and two database specialists spent months of unprecedented data gathering and hundreds of hours of sophisticated computer analysis collecting databases of standardized test scores in approximately 70,000 schools, in more than 14,000 districts in 49 states.

· With the data in hand, we used a method similar to the analysis used to find suspicious test scores in Atlanta (the AJC’s previous investigative report that led to a state investigation and found cheating up to the highest levels in the Atlanta Public Schools).

· Our research compared test scores achieved by a “cohort” of students: that is, when a third-grade class in a school moves on to fourth grade, the group is likely to remain similar and so test scores won’t vary significantly.

· By plotting large changes in scores for a cohort, for better or worse, an analyst can identify test results that are highly unlikely to happen by chance. When scores go up as much as our research found, it suggests some intervention to manipulate the expected results. Scores that drop are meaningful, too. Test scores can rise or fall dramatically because one may have been manipulated and the next one does not, or vice versa.

· We showed our methodology and results to independent experts at three major universities specializing in testing and data analysis to confirm our findings. We also consulted with the American Institutes for Research. Our analysis identified districts nationwide with clusters of suspicious score changes.

Experts Who Confirmed Our Published Report

· Gary Phillips, vice president and chief scientist, American Institutes for Research, advised on methodology.

· Jaxk Reeves, director of the University of Georgia Statistical Consulting Center, conducted a detailed independent review of our data and methodology, replicating the analysis for multiple states from scratch.

· James Wollack, director of testing and evaluation services, University of Wisconsin-Madison, reviewed methodology and a sample of the findings.

· Edward Rothman, professor of statistics and director of the Center for Statistical Consultation and Research, University of Michigan, reviewed methodology.

TimeOut

March 25th, 2012
11:48 pm

So much data……..so little relevance………I don’t care about standardized test scores. I’m not even sure that I care about the cheating. The idiocy of NCLB and AYP is a repetition of prior idiotic moves perpetrated by the powerful against defenseless school children who so desperately need the examples of ethical conduct from those same power-mongers. Ain’t gonna happen. Instead of calling them bureaucrats or the socioeconomic or political ‘elite’, I find myself agreeing with the use of the term ‘pathocrat.’ Normal people are better off with minimalist interference by government because government attracts too much scum to its rosters. This scum then does whatever is necessary to improve its socioeconomic and political power. The children don’t matter. Their parents don’t matter. The TESTS were used to cheat us all. The emphasis on ‘measuring’ and on obtaining ‘data-driven’ drivel sickens me. Minimal competency on tests of ‘recognition’ and ‘recall’ is just so much useless drivel. Every week when a high school student in France,
in groups of three, we survived oral exams administered in two two-hour sessions by two of our professors. We never took a multiple choice test. Every evaluation required application in what we would have called ‘blue book’ fashion here in the states. So, the pathocrats created a system that resulted in mass cheating. That’s what happens when self-absorbed pathocrats make self-serving decisions that affect us all. We could try to vote them out of office. Who will take their place? ………more of the same……..better to change the laws and remove as much non-local legislative power as possible from the classroom. At least the local yokels have to worry about their neighbors’ response to their selfish acts.

dan

March 26th, 2012
3:00 am

Does anybody have a link to the actual data where one can see which schools were flagged?

Sharon Pitts must Go

March 26th, 2012
7:05 am

Nashville?Houston superintendents need to borrow a page from Bev Hall and take a trip to Hawaii

East Cobb Parent

March 26th, 2012
12:24 pm

IMHO, this is a valid argument, ” The AJC analysis assumes that students in one grade level at a school one year are the same students that were there the previous year in the previous grade.” I’ve always felt we compared apples to oranges when comparing this year’s 3rd graders to last years. I feel you should follow the grade, so this year’s 3rd graders to their scores as 4th graders. Of course that has it’s own issues.

c williams

March 26th, 2012
11:02 pm

You really should check Coffee County High School I over heard a student say her teacher took her online Gatway test for her so she could Grad. in May she is on an IEP this year in grade and this student went to school maybe 20 days the whole yr and she is going to be able to walk for Grad. she don’t deserve to