I ran a letter a few days ago from the principal of an Atlanta charter school expressing concerns about the value-added scores assigned to his school.
Atlanta is looking at both teacher and school-level value-added as part of its Effective Teacher in Every Classroom initiative. Using test scores, researchers are calculating how much “learning” Atlanta students gain in the standard school year. This sort of calculation is being made for school districts and teachers nationwide and will ultimately be done for every school system in Georgia as we move to accountability models that measure student progress over time.
There is great debate over whether any value-added system — and Atlanta has hired some of the nation’s top experts to help it develop accurate value-added metrics — can be trusted.
Under Atlanta’s analysis, students at Atlanta Neighborhood Charter were found to only gain gain 5.2 months of learning in a year, one of the lowest scores in the district.
In a letter to parents, Neighborhood Charter principal Matt Underwood wrote: “As much as these efforts try to boil “success” down to a single number or letter, assessing students or teachers or schools is substantially more complicated than that, a fact that has been on my mind these past few weeks…All students–whether at ANCS or elsewhere–are complex individuals with differing strengths and weaknesses. They deserve ways of assessing their performance (and, by extension, the performance of their teachers or schools) that acknowledge just how unique they are as human beings and recognition that there is value in many skills and knowledge that cannot be shown by filling in a bubble.
APS has written a response to Underwood. It is from Rubye Sullivan, director of research and evaluation for School Improvement. I received several emails from researchers on how APS created its scores, and I hope Dr. Sullivan’s letter helps them.
Here it is:
By Dr. Rubye Sullivan
I read Matt Underwood’s letter on value added with great interest. I am very impressed by Matt’s efforts to understand both value-added and multiple measures of his students’ learning. I would like to clear up a few questions raised about value-added.
First, a technical concern: Matt raises the issue that not all CRCT exams have content alignment from year-to-year. When two exams do not have content alignment, we can think of value added as measuring whether students illustrate the same amount of knowledge one year later as students who scored similarly the year before. Although Social Studies exams measure different content areas in different grade levels, the predictive relationship between current and previous social studies exams is as high as or higher than the relationship for English or math exams. This allows for accurate Social Studies value-added models.
Atlanta Public Schools is also careful to be forthcoming about the accuracy of value-added results. Schools receive a value-added report that includes a point estimate and a confidence interval for each result. The confidence interval is similar to the plus/minus numbers associated with public opinion polls. If teacher or school effects are difficult to measure in a particular grade or subject, the confidence intervals will be wider. This helps prevent over-interpreting the data.
APS also uses information external to test scores to validate value-added models. Social Studies value-added scores are positively correlated with CLASS Keys evaluation scores; the same teachers that have high-value added tend to have high evaluation scores. The correlation is similar in magnitude to other subjects and external academic research in other school districts.
Mr. Underwood also highlights the very important point that test scores are only one facet of student outcomes. APS stresses during the distribution of value-added scores this very same point. However, we do consider test scores and test score growth to be important. As long as our students are held accountable for meeting proficiency standards, we are obligated to highlight our schools and teachers who are able to grow their students towards proficiency and beyond. Value-added is how we are able to understand the successful instructional practices of our schools and teachers and replicate those practices such that all of our students experience academic growth.
I do not want to misconstrue Mr. Underwood’s point. I know he is very concerned about students with low test scores as well. Rather, his concern is that the increased emphasis on student test scores weakens instruction and other attempts to enrich student’s lives.
Recent studies by the Gates Foundation have worked to better understand exactly what value-added is telling us, including whether high-value added is a result of teaching to the test or real, conceptual learning. In addition to state exams, students in six large public school districts were also administered exams that attempt to measure more higher -order conceptual thinking. The study found that teachers who increase learning on state exams tend to be the same teachers that increased learning on more conceptual exams. For example, the correlation of value-added between the state exam and the Stanford 9 Open Ended Reading assessment was 0.59. (Correlation values range from -1 to 1. A value of 1 means two measures are the same, zero means they are unrelated, and -1 means they are exact opposites.) If “teaching to the test” or memorization was the way to maximize value added, we would expect a negative correlation. The positive correlation is encouraging, although, because it is not 1, also emphasizes that accountability exams must measure what we expect our students to be learning. The current CRCT and EOCT exams clearly define curriculum objectives and Georgia will be adopting the PARCC exams in the 2014-2015 school year, which seek to better measure higher-order conceptual learning.
There are also concerns that value-added does not measure a school’s impact on other important results such as a student’s motivation or behavior. However, these outcomes are closely tied to test scores. A school that is unable to motivate its students or teach good behavior will also be ineffective in raising academic outcomes. Value-added is a powerful tool for a district to understand learning patterns across schools or a principal to understand learning within schools. It will help recognize areas of strength but will not diagnose what specific strategies are improving student learning. This requires additional investigation by school staff, a step that Mr. Underwood appears to already be taking.
We also know that raising test scores now has important long-run impacts for children. A recent study by researchers at Harvard and Columbia has helped us understand the long-term impacts of teachers who raise test scores. The paper links student test scores and teacher and school assignments with IRS wage data to show that the difference between a single year with a low performing teacher versus a high-performing teacher has a substantial effect on student’s future earnings as well as outcomes such as college attendance and reduced likelihood of teenage pregnancy.
In Atlanta Public Schools, we are using value-added information for school improvement, not accountability. Please be reminded that similar to achievement data as reported in terms of proficiency rates, these data only tell one part of the complex story of teaching and learning. We will continue to improve our data systems such that our school leaders have access to multiple sources of data to utilize in the school improvement process.
–From Maureen Downey, for the AJC Get Schooled blog