Atlanta Weather

| Traffic

Search
Get Schooled

Atlanta explains how it calculates value-added scores. Using scores for “improvement, not accountability.”

7:19 am October 18, 2012, by Maureen Downey

I ran a letter a few days ago from the principal of an Atlanta charter school expressing concerns about the value-added scores assigned to his school.

Atlanta is looking at both teacher and school-level value-added as part of its Effective Teacher in Every Classroom initiative. Using test scores, researchers are calculating how much “learning” Atlanta students gain in the standard school year. This sort of calculation is being made for school districts and teachers nationwide and will ultimately be done for every school system in Georgia  as we move to accountability models that measure student progress over time.

There is great debate over whether any value-added system — and Atlanta has hired some of the nation’s top experts to help it develop accurate value-added metrics — can be trusted.

Under Atlanta’s analysis, students at Atlanta Neighborhood Charter were found to only gain gain 5.2 months of learning in a year, one of the lowest scores in the district.

In a letter to parents, Neighborhood Charter principal Matt Underwood wrote: “As much as these efforts try to boil “success” down to a single number or letter, assessing students or teachers or schools is substantially more complicated than that, a fact that has been on my mind these past few weeks…All students–whether at ANCS or elsewhere–are complex individuals with differing strengths and weaknesses. They deserve ways of assessing their performance (and, by extension, the performance of their teachers or schools) that acknowledge just how unique they are as human beings and recognition that there is value in many skills and knowledge that cannot be shown by filling in a bubble.

APS has written a response to Underwood. It is from Rubye Sullivan,  director of research and evaluation for School Improvement.  I received several emails from researchers on how APS created its scores, and I hope Dr. Sullivan’s letter helps them.

Here it is:

By Dr. Rubye Sullivan

I read Matt Underwood’s letter on value added with great interest. I am very impressed by Matt’s efforts to understand both value-added and multiple measures of his students’ learning. I would like to clear up a few questions raised about value-added.

First, a technical concern: Matt raises the issue that not all CRCT exams have content alignment from year-to-year. When two exams do not have content alignment, we can think of value added as measuring whether students illustrate the same amount of knowledge one year later as students who scored similarly the year before. Although Social Studies exams measure different content areas in different grade levels, the predictive relationship between current and previous social studies exams is as high as or higher than the relationship for English or math exams. This allows for accurate Social Studies value-added models.

Atlanta Public Schools is also careful to be forthcoming about the accuracy of value-added results. Schools receive a value-added report that includes a point estimate and a confidence interval for each result. The confidence interval is similar to the plus/minus numbers associated with public opinion polls. If teacher or school effects are difficult to measure in a particular grade or subject, the confidence intervals will be wider. This helps prevent over-interpreting the data.

APS also uses information external to test scores to validate value-added models. Social Studies value-added scores are positively correlated with CLASS Keys evaluation scores; the same teachers that have high-value added tend to have high evaluation scores. The correlation is similar in magnitude to other subjects and external academic research in other school districts.

Mr. Underwood also highlights the very important point that test scores are only one facet of student outcomes. APS stresses during the distribution of value-added scores this very same point. However, we do consider test scores and test score growth to be important. As long as our students are held accountable for meeting proficiency standards, we are obligated to highlight our schools and teachers who are able to grow their students towards proficiency and beyond. Value-added is how we are able to understand the successful instructional practices of our schools and teachers and replicate those practices such that all of our students experience academic growth.

I do not want to misconstrue Mr. Underwood’s point. I know he is very concerned about students with low test scores as well. Rather, his concern is that the increased emphasis on student test scores weakens instruction and other attempts to enrich student’s lives.

Recent studies by the Gates Foundation have worked to better understand exactly what value-added is telling us, including whether high-value added is a result of teaching to the test or real, conceptual learning. In addition to state exams, students in six large public school districts were also administered exams that attempt to measure more higher -order conceptual thinking. The study found that teachers who increase learning on state exams tend to be the same teachers that increased learning on more conceptual exams. For example, the correlation of value-added between the state exam and the Stanford 9 Open Ended Reading assessment was 0.59. (Correlation values range from -1 to 1. A value of 1 means two measures are the same, zero means they are unrelated, and -1 means they are exact opposites.) If “teaching to the test” or memorization was the way to maximize value added, we would expect a negative correlation. The positive correlation is encouraging, although, because it is not 1, also emphasizes that accountability exams must measure what we expect our students to be learning. The current CRCT and EOCT exams clearly define curriculum objectives and Georgia will be adopting the PARCC exams in the 2014-2015 school year, which seek to better measure higher-order conceptual learning.

There are also concerns that value-added does not measure a school’s impact on other important results such as a student’s motivation or behavior. However, these outcomes are closely tied to test scores. A school that is unable to motivate its students or teach good behavior will also be ineffective in raising academic outcomes. Value-added is a powerful tool for a district to understand learning patterns across schools or a principal to understand learning within schools. It will help recognize areas of strength but will not diagnose what specific strategies are improving student learning. This requires additional investigation by school staff, a step that Mr. Underwood appears to already be taking.

We also know that raising test scores now has important long-run impacts for children. A recent study by researchers at Harvard and Columbia has helped us understand the long-term impacts of teachers who raise test scores. The paper links student test scores and teacher and school assignments with IRS wage data to show that the difference between a single year with a low performing teacher versus a high-performing teacher has a substantial effect on student’s future earnings as well as outcomes such as college attendance and reduced likelihood of teenage pregnancy.

In Atlanta Public Schools, we are using value-added information for school improvement, not accountability. Please be reminded that similar to achievement data as reported in terms of proficiency rates, these data only tell one part of the complex story of teaching and learning. We will continue to improve our data systems such that our school leaders have access to multiple sources of data to utilize in the school improvement process.

–From Maureen Downey, for the AJC Get Schooled blog

  • Print

71 comments Add your comment

3schoolkids

October 18th, 2012
6:23 pm

The author says regarding the Harvard/Columbia study “The paper links student test scores and teacher and school assignments with IRS wage data to show that the difference between a single year with a low performing teacher versus a high-performing teacher has a substantial effect on student’s future earnings as well as outcomes such as college attendance and reduced likelihood of teenage pregnancy.” Did the researchers have access to such sensitive personal information that they are able to tie even one year of low test scores/poor teacher performance with an individual’s earnings, college attendance or teen pregnancy? Or was this a demographic/geographic study? Sounds like they were looking at invidual data…how Big Brother like.

Private Citizen

October 18th, 2012
7:37 pm

Educators work in the classroom and except for maybe 1% of them, the income box on their tax return does not have more $100k.

It has become acceptable in Georgia public schools to have central office staff with salaries over $100k and there is absolute army of these persons. It would be interesting to see a number, how many persons in Georgia public school systems are paid over $100k/year. 500? 1000? It is categorical that (almost) none of these persons being paid over $100k.year work in the classroom teaching students. In the school building, principals and assistant principals are generally paid in the $100k+ range. Based on income disparity alone, this creates a management caste that is wholly different from the classroom teachers. This is not sour apples, it is recognition of the sheer quantity of this activity in the State of Georgia. It would interesting to see a distribution based on the numbers and income of salaries of workers in the Georgia public schools. Certainly it is a finite data set.

City of Atlanta Schools
Principal pay 100-135k
Executive Director pay 89-144k
Teacher pay 55-111k
http://www.atlanta.k12.ga.us/site/Default.aspx?PageID=1172

I think the vast majority of the teachers in the state earn 40-50k.

Private Citizen

October 18th, 2012
7:55 pm

Add to that teacher spending their own money on supplies simply to be able to do their jobs. Not personal supplies, but supplies for students. You try teaching a 110+ kids without textbooks, officially told go to the internet to get your materials. Recently told of a local high school teacher, 170 students every day, get home at 8pm each night. Central office do not keep these kind of hours and the only thing they spend their own money on is leaving a tip when they go out for lunch. Try spending a year leave home when it is dark outside, return home when it is dark outside. Try that for several years. And then sit in the meetings and listen to the redelivery of the new thing, not told directly where it came from, just given a skeleton description and a bunch of demands. Went over to visit with a teacher put in charge of some of this stuff. Teacher said, “there’s no instruction set. Nobody will tell you the specifics, -can’t get any answers.” I said, “Can you get the documents from the prior year? Something to model it on?” [NOTE. FULL STOP. DUE TO TEACHER DISPLACEMENT THE PERSONNEL FROM THE PRIOR YEAR ARE NO LONGER IN THE BUILDING.] I said, “Well, you’ve just got to find something to feed the dragon – give them what they want.”

And produce detailed lesson plans EVERY WEEK. Start from scratch and “guidelines” and invent it yourself, by yourself, skipping around in the internet. You ever stay up half the night to make one “power point” presentation to teach the specified “guidelines, i.e. “standards.” Many of the subjects there is no teacher support materials, none, nothing. Go ask John Barge. Go ask a principal. Go ask someone getting paid $100k+ at the central office. They’ll act like they’re hard of hearing and you do not exist. Lots of guidelines. Just go over and look at the “Georgia Standards” of “Common Core” or what have you. You’ll find no link to the tools to get the job done and you won’t find them in the classroom either. Want to teach something? Buy it yourself or use the photocopier.

Private Citizen

October 18th, 2012
8:05 pm

PS The minute a Georgia teacher initiates any communication with the central office, they’ll regret it. They central office will then know who you are and they’ll come after you. They’ll make a phone call and send one of the highly paid “specialists” to check on you and after that they will never leave you alone.

Private Citizen

October 18th, 2012
8:19 pm

@Matt Underwood, excellent concept on how to test for social studies, testing on skills as opposed to jeopardy questions on the 10,000 details associated with social studies.

Please continue because you have good relevant ideas. I recall the year that the state DOE simply did not return the social studies CRCT scores.

Lisa Roberson

October 18th, 2012
8:42 pm

Matt,

Thank you for your wonderful response and all you and your teachers and staff do for our students and our school. I remain grateful each and every day for the wonderful education my three boys are getting at ANCS.

Private Citizen

October 18th, 2012
9:12 pm

PS The VAM concepts for the lower performing schools sound great. I want these kids to succeed as much as anyone but it would be helpful if at the same that VAM is being applied where it is appropriate, it would be nice if some subversion jackal from a foundation or the fed or the state was not ambushing high performing teachers and students solely because they are effective and outscore everybody. I would be d^%& insensed if I was a parent of color of a child and preparing them for college and I found that their college-prep teachers were getting rooted around in this manner. And it would be nice if the state and local management, the superintendents and redelivery artists did not so thoroughly front the jackals and subvert the very city where they live by applying the “all / everything” mandates so freely to the point of disassembling high performing sectors of the schools. The principals have to eat these directives even if it means telling everybody to hang upside down like bats from the ceiling. I’ve seen it.

I’ve personally been blind-sided with some of this and I’m ready to bite back, so if you see my teeth and I go “chomp,” well, you deserve it.

Chris Murphy

October 18th, 2012
9:15 pm

” When two exams do not have content alignment, we can think of value added as measuring whether students illustrate the same amount of knowledge one year later as students who scored similarly the year before. Although Social Studies exams measure different content areas in different grade levels, the predictive relationship between current and previous social studies exams is as high as or higher than the relationship for English or math exams. This allows for accurate Social Studies value-added models.”

Can anyone translate that into English? Matt, I always appreciated your willingness to deal with middle schoolers, but now I really admire your patience.

Private Citizen

October 18th, 2012
9:21 pm

@Chris Murphy, If grade 6 social studies teaches Georgia history, and grade 7 social studies teaches China and Mongolia, the content of the two years of studies or “exams” does not “line up.”

Some subject areas are what is called “recursive.” That means that from year to year the subject material sort of circles back around and reteaching the prior fits in with teaching the present. For example, math and languages arts could be considered “recursive” in nature as the content is similar from year to year. In this manner, there may be some utility in comparing two sequential years of testing from one year to the next.

Another example of non-recursive content occurs in science, where students might study rocks and weather one year, and cell biology and “living things” the next year. In this way, it would difficult to determine continuity of learning based on test scores.

Ed Johnson

October 18th, 2012
9:24 pm

Value-add? Effective Teacher in Every Classroom Initiative” (ETEC)? Teacher Effectiveness Dashboard (TED)?

Let’s not forget so quickly. Think Beverly Hall. Think Bill Gates.

http://blogs.ajc.com/get-schooled-blog/2010/06/07/crct-probe-in-atlanta-narrows-to-a-dozen-schools/?cp=all#comment-62382

http://blogs.ajc.com/get-schooled-blog/2011/06/11/aps-board-chair-downtown-business-district-setting-citys-agenda-to-detriment-of-students/?cp=all#comment-134530

http://blogs.ajc.com/get-schooled-blog/2011/11/12/georgias-only-milken-winner-this-year-i-am-my-students/?cp=all#comment-164005

Chris Murphy

October 18th, 2012
9:52 pm

@Private citizen: I appreciate the effort, but calling different courses recursive means that the VAM is about as subjective as a measure can get. Although subject material in any given year would depend somewhat on previous courses- either being built on that previous knowledge or relating to it- one year’s learning doesn’t “align” with any other. That’s the semantics of a meaningless measure.

I personally don’t think it’s a bad idea to test students- maybe because that is what I know from my own education. I wasn’t exposed to pseudo-tests like the CRCT though, where an easy test is then dicked with by using “cut scores,” where they basically lower the ‘passing’ score. I seriously doubt any accurate evaluation can be done from GA’s CRCT, and thus anything based on it is similarly flawed. Does keep the central office folks off the streets, and those Ed. consultants in fine fettle.

jlmdra

October 18th, 2012
10:25 pm

As with any industry, there’s a place in American health care where big money and big ego cross paths. But in medicine, that intersection is often found in the body and mind of individual Americans.

A distinguished ophthalmologist and clinical professor at Emory, Tom Harbin provides the authoritative account of the rise and rise of Dwight Cavanagh. Performing eye surgeries in impressive numbers, Cavanagh made himself into a money machine for Emory. Not only did the institution receive reimbursement for the procedures; Cavanagh was also adept at winning grants. The whole department prospered. The University built state-of-the art facilities. Everybody seemed to win. Cavanagh was the ophthalmological equivalent of a rock star.

Except that whispers began to spread about whether the patients really needed all those operations. In one case, Cavanagh operated on the wrong eye, blinding a poor man who hadn’t clearly needed surgery in the first place. After too many operations on too many borderline patients, the hard-working, honest physicians alongside Cavanagh finally mustered the courage to question the rock star’s practices. Cynically, the Emory administration closed ranks, and it was the honest critics whose careers were stunted.

Harbin tells this true story with a novelist’s pace and an insider’s authority. Waking Up Blind succeeds because it’s a gripping story told by an authoritative physician with a graceful and unobtrusive style. It’s also an engaging account of how Big Ego and Big Health Care can actually compromise patient outcomes. Arriving in the midst of the national health care debate, Waking Up Blind couldn’t be more timely. By Horace Nalle

Private Citizen

October 18th, 2012
10:37 pm

Chris, I’m not “calling” anything recursive. What is recursive is recursive, what is not is not. Maybe I should not have said anything but I was trying to shed light on the mechanics of your questions.

I do not know enough about the fine points of VAM to really comment on it except that this concept seems to be being used to chop down high-end programs and villify high performance sectors of the schools. This is rather serious. I guess the people who do this must thing someone somewhere else is going to train the doctors and then the doctors are going to move to Georgia because they just happen to love the place.

I do not know who started this “glamorize and dote upon the lower tier” initiative but it seems they have gotten to the point of it is about as safe as a runaway train and has just as much inertia, momentum and mass. Point is these initiatives come from outside foundations, are ported in by the fed and it is not unjust to say that Georgia is the Whore of Babylon when it comes to re-delivering this concepts as policy in return for receiving some money. I wouldn’t be so harsh if I didn’t think it was serious. It is a leech system where they must expect the education of doctors and engineers to take place somewhere else.

i hope the charter initiative passes. I went out this evening and passed a church that runs a little school there. The church had told me that the county messes with them over any little thing, hassling them on any kind of permits and such, because the county resents that they are operating a school there in place of the county system. I have no misconception that the charter amendment may be used to bring in big brother and Wal-Mart and ALEC and Koch kin and anything else you can think of, but I am ready for some relief from the management bullying of the public schools. Someone said that by roughing up legitimate talent it makes more room for nepotism.

i suggest there is some good to Value Added concepts and that this good is limited to low performing schools who need the leg up (as might be said in regard to horsemanship.) Don’t come mess with the F1 drivers because we will take you apart. We didn’t get to be F1 drivers by passive resistance.

Leslie Grant

October 18th, 2012
11:14 pm

Just have to say that I’m so thankful to be part of a system that, come hell or high water, has allowed folks like Matt Underwood AND Rubye Sullivan to flourish. They are both incredibly gifted people who do care deeply about truly finding the best ways to educate children. I am so grateful that Matt Underwood found his way to ANCS and I look forward to the day that APS finds a productive way to look beyond the metrics in order to find out what is working so well for so many.

Concerned

October 18th, 2012
11:37 pm

Data can say whatever you want it to; just change the lens of purpose. Was Rubye on the team to determine NAHS was ineffective? Did she provide some of this value-added data to show growth of students there? It seems as if APS will tell whatever story they want.

Concerned

October 18th, 2012
11:40 pm

You know APS creates additional work that makes high quality teaching and learning impossible. Why was there an entire job dedicated to this work??? Why not use this money to provide support – true, authentic support to teachers and schools which will support achievement to the school.

Private Citizen

October 19th, 2012
6:47 am

“The Gates Foundation was described as a shell for tax avoidance by philanthropist and accounting expert, Sheldon Drobny.” http://techrights.org/wiki/index.php/Gates_Foundation_Critique

Private Citizen

October 19th, 2012
7:17 am

” They’ll get sort of addicted, and then we’ll somehow figure out how to collect sometime in the next decade.” http://en.wikiquote.org/wiki/Bill_Gates

Nikole

October 19th, 2012
8:38 am

As I have been saying for years, VAMs are just more faulty information. My first graders will ALWAYS improve, but if you have a gifted class taking a first grade test, how much value will be added? They mastered the per test and post test. Because they’re both first grade tests, no growth will be shown. And if you have the inclusion class, should they be expected to grow at the same rate as others? We’ve already determined that they have developmental delays, right? It all comes down to the luck of the draw. Research shows teachers being highly effective one year and ineffective the next. What changed? Their students.

Southside Parent

October 19th, 2012
3:30 pm

@Nikole: The full VAM reports provide more data. My favorite subreport was the academic subreport – comparing growth for kids based on their prior year CRCT scores. Does your school add extra value to the kids who in prior years “failed to meet”? The “meets expectations” performers? The “exceeds” performers?

Flawed information. Many valid issues being raised with VAM. But with the potential to be incredibly helpful if we can keep the flaws in mind.

dbow

October 19th, 2012
5:50 pm

She mentioned the Gates Foundation. That’s really all I needed to read.

« Previous 1 2 Show All