With the AJC series on national test score disparities minutes away from publication, a second district with implicated schools has chosen to issue an preemptive strike. Houston released a statement Friday.
From Nashville Metro Schools:
Metro Schools has several, significant concerns with the data presented by the Atlanta Journal Constitution regarding our district and believe that Nashville has no place in the story they have prepared. Some of those concerns include:
• The AJC analysis assumes that students in one grade level at a school one year are the same students that were there the previous year in the previous grade.
• Our district has mobility rates between 35%-40%.
• Zoning changes have impacted student enrollment.
• A significant number of ALCs (Alternative Learning Centers) and special schools (and “homebound”) are flagged in our data; these schools often have very fluid populations – ALC populations have 100% annual turnover.
• This AJC information flags schools with unusually high numbers of discrepancies from predicted scores – both high or low.
• 77% of the cases flagged in MNPS were due to low performance (significantly below predictions).
• The negative scores we saw were not part of a pattern of high gain one year followed by significant loss the next (as would be expected if cheating was involved).
• There was not a single case of unusually high gains flagged for 2011.
• Obvious errors in data provided where children who were absent from testing were assigned a “zero” rather than being excluded from the analysis – resulting in average scores that were below the minimum score possible.
• Tennessee adopted much more difficult academic standards in 2010, resulting in a significant decrease in student test scores statewide.
• The AJC methodology will automatically identify 5% of the cases/classes analyzed statewide, but schools with changing populations (see previous bullet) or higher than usual numbers of highly effective or highly ineffective teachers (research supports that teacher effectiveness make the greatest difference in test score gains) are likely to have higher percentages.
• Dozens of districts in Tennessee had higher percentages of classes flagged in a given year than did MNPS or Memphis, but those districts had too few cases to show statistical significance; the methodology is much more likely to identify only larger school districts.
• Metro Schools has the highest population of English Learners in the state, with approximately 1 in 4 of our students coming from non-English Language Backgrounds. We also educate about 30% of the state’s EL students.
• National data show EL students typically make among the largest gains from year to year as they learn the English language; several of the schools flagged for positive gains have high EL populations.
• Regression to the mean could be a factor in why some of the relatively highest (e.g., Meigs) and relatively lowest (ALCs) scoring schools are flagged.
• The Tennessee Value Added Assessment System (TVAAS) uses much more sophisticated analytical techniques (Henderson’s mixed model methodology) and follow’s actual cohorts of students to analyze gains of teachers, schools, and districts and TVAAS results have typically been comparable to statewide trends for several years.
• Analysis of test gains of MNPS middle school math teachers over three years (2006-07 through 2008-09) by Dr. Brian Jacobs of the University of Chicago, on behalf of Vanderbilit University, did not show any unusual or suspicious patterns.
• The analysis was conducted as part of the Project on Incentives in Teaching (POINT) study, to determine whether tying significant financial incentives for teachers (up to $15,000 per teacher per year) to student achievement made an impact.
•The analysis conducted by Dr. Jacobs was specifically to determine if evidence existed of invalid test scores (cheating).
•Dr. Dale Ballou, Associate Professor of Public Policy and Education at Vanderbilt University, is prepared to speak to this study. We have also shared the data AJC provided with Dr. Ballou and he has found some statistically impossible anomalies in the data. He is also willing to speak to these findings.
• The district takes test security very seriously
• Meetings with principals and test coordinators are held annually prior to TCAP testing and test security guidelines and procedures are reviewed.
• Principals and test coordinators annually provide training to teachers and other staff involved in testing.
• Every individual involved in test administration and handling is required to read and sign the district’s State-Mandated and District-Wide Test Security Agreement form, which highlights key test security guidelines and includes the state law regarding test security (stating that violations could lead to immediate suspension and potential dismissal).
• Test administration is monitored in a sample of schools each year by both district staff and the Tennessee Department of Education; annual reports from the TDOE have been positive.
• All allegations of test security violations are reported to the Tennessee Department of Education and investigated jointly by district staff in the departments of Research, Assessment & Evaluation and Human Resources.
• Action is pursued when violations of test security are confirmed, and educators have been suspended or terminated due to these violations
–From Maureen Downey, for the AJC Get Schooled blog