Can we really measure effective teaching? Yes, says new Gates Foundation study.
After three years of research and an investment of $45 million, the Bill & Melinda Gates Foundation believes it now knows how schools can fairly and reliably measure effective teachers.
While student test scores are part of the solution, scores alone are not enough to gauge how well a teacher is performing, according to the Gates-funded Measures of Effective Teaching Project.
Released Tuesday, the final report from the MET Project says a three-prong approach, multiple classroom observations, student surveys and student growth as measured by state test scores, provides a good picture of how effective a teacher is. The project found that an accurate observation rating for a teacher requires two or more lessons, each scored by a different certified observer.
The report will likely resonate in Georgia, which is in the midst of rolling out a new teacher evaluation system funded by the state’s Race to the Top grant. Georgia is spending millions on its new evaluation system, which will consider student performance in rating whether a teacher is exemplary, proficient, developing/needs improvement or unsatisfactory.
The MET findings examined the performance of students of the 3,000 teachers from Charlotte-Mecklenberg, Dallas, Denver, Hillsborough County, Fla., Memphis, Pittsburgh and New York City who volunteered to be part of the project.
Through a randomized experiment, the research sought to answer the question: Are seemingly more effective teachers really better than other teachers at improving student learning, or do they simply have better students?
Ultimately, the only way to resolve that question was by randomly assigning students to teachers to see if teachers previously identified as more effective actually caused those students to learn more. That is what we did for a subset of MET project teachers.
Based on data we collected during the 2009–10 school year, we produced estimates of teaching effectiveness for each teacher. We adjusted our estimates to account for student differences in prior test scores, demographics, and other traits. We then randomly assigned a classroom of students to each participating teacher for 2010–11.
In fact, we learned that the adjusted measures did identify teachers who produced higher (and lower) average student achievement gains following random assignment in 2010–11. The data show that we can identify groups of teachers who are more effective in helping students learn. Moreover, the magnitude of the achievement gains that teachers generated was consistent with expectations.
In addition, we found that more effective teachers not only caused students to perform better on state tests, but they also caused students to score higher on other, more cognitively challenging assessments in math and English
On average, the 2009–10 composite measure of effective teaching accurately predicted 2010–11 student performance. The research confirmed that, as a group, teachers previously identified as more effective caused students to learn more. Groups of teachers who had been identified as less effective caused students to learn less. We can say they “caused” more (or less) student learning because when we randomly assigned teachers to students during the second year, we could be confident that any subsequent differences in achievement were being driven by the teachers, not by the unmeasured characteristics of their students. In addition, the magnitude of the gains they caused was consistent with our expectations.
A practical concern of many readers of this blog is the time and expense associated with classroom observations. The new MET report — with its endorsement of multiple observers including someone from outside the school to counter bias — will probably inflate those concerns.
The study addressed the time constraints:
Our analysis from Hillsborough County showed observations based on the first 15 minutes of lessons were about 60 percent as reliable as full lesson observations, while requiring one-third as much observer time.
Therefore, one way to increase reliability is to expose a given teacher’s practice to multiple perspectives. Having three different observers each observe for 15 minutes may be a more economical way to improve reliability than having one additional observer sit in for 45 minutes. Our results also suggest that it is important to have at least one or two full-length observations, given that some aspects of teaching scored on the Framework for Teaching (Danielson’s instrument) were frequently not observed during the first 15 minutes of class.
American Federation of Teachers President Randi Weingarten issued this statement in response to Tuesday’s release of the final MET report:
The Gates Foundation’s study makes clear that evaluation of teachers must start with genuine feedback, which means the days of haphazard or check-list observation of teachers must end. Just dropping by a teacher’s classroom and writing up an evaluation must be replaced with a more serious process that actually helps improve teacher practice and student learning.
The MET study of thousands of teachers reaffirms that teacher evaluation is both an art and a science that requires time, tools, training and trust — ingredients that teachers and principals should have but too often don’t. This study underscores the fact that teacher evaluations must be about improving teaching, not just a mere snapshot assessment.
The MET findings reinforce the importance of evaluating teachers based on a balance of multiple measures of teaching effectiveness, in contrast to the limitations of focusing on student test scores, value-added scores or any other single measure.
To read more responses to the MET report, check out this Ed Week story.
–From Maureen Downey, for the AJC Get Schooled blog
Tell us what you think about the site
Customer Care »
Vacation stops, manage subscriptions and more
AJC Services
Sections
Other Editions
Visitor Agreement | Privacy Statement
© 2013 The Atlanta Journal-Constitution
74 comments Add your comment
WillinRoswell
January 9th, 2013
11:20 am
I’m not sure about measuring effective teaching, but ineffective teaching can certainly be measured.
Well learned Feller
January 9th, 2013
11:25 am
If learning ability can be measured then teaching ability can be measured.
Pluto
January 9th, 2013
11:32 am
Think of a classroom as a two way street; effective teaching coupled with active learning succeeds everytime. Why is the emphasis only on “teaching”? I suggest effective learning strategies that go unheeded only to discover I am a crappy teacher. Where is the joint accountablity?
Michael Moore
January 9th, 2013
11:42 am
@Michelle – I’d say the school knows who the good teachers are and the bad ones. Not necessarily the administrators. I worked for too many clueless principals.
Private business concerns have a history in education reform. Between 1896 and 1920, a small group of industrialists led by John D. Rockefeller outspent the US government with private funds for education to create an American Education System bent to the will of industry and the state. In 1906 he created The Education Board and gave 180 million dollars to fund education policy. “The task we set before ourselves is simple…We will organize children…and teach them to do in a perfect way the things their fathers and mothers are doing in an imperfect way” (Gates, 1913). This, at best, was an effort to get kids away from their parents and supported Eugenics theory. (Frederick T. Gates. “The Country School Of To-Morrow, “Occasional Papers, No. 1 (1913) online). All based on the Prussian model.
living in an outdated ed system
January 9th, 2013
12:51 pm
A very interesting report and one that I will study very closely. We can measure the effectiveness of just about any employee in any line of work, but why not a teacher? The fact that Randi Weingarten did NOT discount it “out of hand” is surprising and could mean that there is finally hope that we can make progress on a teacher evaluation system that is fair and balanced.
Jerry Eads
January 9th, 2013
1:07 pm
Thanx for the ‘honorable mention’ Bev. I try.
Over the next few days I’ll dig into the report. As with many such ‘tank’ reports of research, it may or may not have been peer reviewed yet (if ever), so I’ll look to see if I can find such feeback.
I do believe, from what I’ve seen, that the Gates Foundation is honestly – if from their perspective – trying to help public ed rather than – apparently like some of our elected representatives – trying to destroy it. Do remember that Bill himself is a LONG, LONG way from the day to day of his foundation, which is actually LOTS of different people doing LOTS of different things. This is ONE piece.
That said, let’s all acknowledge that (a) most teachers are not only good at what they do, they are ALWAYS working to get better and (b) some teachers need a lot of help, ESPECIALLY the newbies (but we don’t, much). Some no doubt shouldn’t be there. Let’s also acknowledge that teacher evaluation has long been recognized to be useless. All too often it’s done haphazardly and ineffectively as Maureen noted, and done all too often by people who couldn’t teach (or hated it) – and couldn’t do it well if they had to – so they became administrators.
Can we get better at identifying good teaching? I hope so, but the best of my colleagues in the research and measurement bizzes – not to mention teachers – have forever (to no effect) been telling those who make policy that factoid recognition tests do not and CANNOT measure anything but getting kids to recognize factoids, which isn’t teaching.
As long as we have (a) “school leaders” who can’t lead, much less teach (my apologies to all of you who are fantastic), (b) poorly made low-bid tests that look at little more than the aforementioned factoid recognition, and (c) schools that are run like factories or concentration camps where teachers are treated like assembly-line workers (or inmates) instead of respected professionals, we’ll have a tough row to hoe measuring actual teaching, good or bad.
Do I think we’ll get better at schooling in general? I hope so, in spite of our (don’t’ forget it’s OUR fault) electing some legislators who seem happy to throw your kids under the bus (sacrificing the long term health of society for the short term gain of getting you to pay for their own kids’ “charter” schools), and in spite of some school leaders who may well take a teacher evaluation instrument that might be accurate within, say, plus or minus 50 points and fire someone with a 69 and keep someone with a 70. And then say something like “See how we got rid of “bad” teachers!” while spending relatively no effort to make anyone better.
“The beatings will continue until morale improves” doesn’t improve anything in the public OR private sectors. The teacher haters, including the many who post here, apparently don’t get that (or they’re so wrapped up in loving to hate they don’t care).
Focusing solely on unloading a few bad eggs will work JUST like minimum competency kid testing. People will work to avoid being “most worst” but not focus on being “best” because the system focuses on punishing bad rather than rewarding getting better.
BT
January 9th, 2013
3:47 pm
As an experienced educator, we teach way too much at one time and assess to death. All this testing is absolutely rediculious!!! The more assessments you give the less you teach.
beteachin
January 9th, 2013
3:59 pm
The only thing standardized tests prove is whether or not a student can bubble in an oval.
The only thing teacher observations by administrators prove is whether the teacher can jump through the hoops set forth by that administrator.
Learning has NOTHING to do with standardized testing; good teaching has NOTHING to do with Charlotte Danielson, etc.
Teachers know if their students are making progress; if they aren’t, teachers must find another way to reach them.
Administrators know if teachers are engaging students and offering rigorous instruction; if they aren’t, administrators must find someone else to reach them.
Linda
January 9th, 2013
4:43 pm
The correct spelling is Hillsborough, not Hillsboro. Journalistic typos are embarrassing.
Proud Teacher
January 9th, 2013
5:38 pm
I think Bill and Melinda Gates have been a tremendous success with The Bill and Melinda Gates Foundation. They are not teachers with actual experience in the classroom. They sell technology and Georgia boards of education and administrators buy their toys at an alarming rate and bandy them around their schools with pride. All the while our scores decline and our students real needs are neglected. We are not trying to educate numbers; we are trying to educate children. I’m so tired of being criticized and being told what to do by those who have never tried to do what I do: teach!
Private Citizen
January 9th, 2013
6:20 pm
Speaking of “free markets” I went to the Comcast public counter today and made a u-turn and walked right back out of there. About thirty people in one line, many holding their dvd player sized tv boxes I guess to turn them in. Where I live, for wired internet there is no alternative and no competition to Comcast. I wish more Americans were a little keener on their loss of markets. Literally I have had no living breathing mammal mention the Comcast monopoly to me. It’s like I need to visit Moscow or Seoul to find some human who even knows what a market environment is, much less the effect of no market or a vibrant market.
Private Citizen
January 9th, 2013
6:27 pm
When I die, please put on my tombstone “Bill Gates makes me want to die.”
This is the guy who temporarily monopolised desktop computing in the United States. Now his computing is obsolete so he is monopolising education. He is literally a born monopolist. It’s in his DNA. There’s a TED talk of him speaking his big ideas. It leaves me cold and vacant.
Note: “Microsoft” is still milking their cash cow “Office.” God only knows why every government and office in the United States is compelled to give them money.
catlady
January 9th, 2013
6:34 pm
cris: When I started teaching (1973) some state universities required potential ed admins to have 5-7 years in the classroom BEFORE APPLYING to a master’s program for administrators. (This same school required potential teachers to pass tests in use of correct speech and writing before being graduated.) Can we sing it? Those were the days…
Private Citizen
January 9th, 2013
7:10 pm
catlady, Can we sing it? Those were the days… http://www.youtube.com/watch?v=ObeJpmdeyQ4
Beverly Fraud
January 9th, 2013
7:25 pm
It’s time to call complete bs on Charlotte Danielson and her Framework based on the following.
In addition to the characteristics of “Effective”:
· Student behavior is entirely appropriate; no evidence of student misbehavior.
· The teacher monitors student behavior without speaking – just moving about.
· Students respectfully intervene as appropriate with classmates to ensure compliance with standards of conduct.
So Charlotte, you have a child born with fetal alcohol syndrome, with an incarcerated father and alcoholic mother, who was locked in his room and not fed dinner last night, who has been diagnosed with Oppositional Defiant Disorder, but his mother hasn’t giving him his medicine for a week and he has a meltdown which the teacher handles with all appropriate intervention methods but you downgrade the teacher because there was “evidence of misbehavior” that didn’t magically dissipate when the teacher walked by?
Walk into a behavior disorder class and show us how it’s done Charlotte. Seriously, how can anyone take her seriously after reading the above nonsense that is part of her rubric?
Private Citizen
January 9th, 2013
7:59 pm
BF, that’s called “fantasy” and then someone signs it into law.
Private Citizen
January 9th, 2013
8:31 pm
“When the longhairs came to town, that was a major crisis.”
BBC documentary (2012) about southern U. S. music. http://www.youtube.com/watch?v=IRHN5so_Ijo Lots of real politik sociology / history in this video.
Gemeinschaft and Gesellschaft
January 9th, 2013
8:43 pm
Any student, parent, or administrator can tell you which teachers are effective in a school and which are not. We don’t need to spend millions of dollars figuring out how to ‘measure’ it. What we need to do is find a better way to spend all that dough.
1. Working technology in classrooms
2. Working toilets in bathrooms
3. 21st century media centers
4. Books
5. Authentic staff development
6. Subsidized test fees
7. Transportation
8. Give back furlough days to teachers
9. Better pay for teachers
10. Figuring out the REAL problem in American schools.
Pride and Joy
January 9th, 2013
8:56 pm
Of COURSE we can measure teacher effectiveness and parent feedback absolutely needs to be part of that evaluation. All of we Americans are very well suited to evaluate teachers because we all have so much experience with the profession. ALL of us have had, at a minimu, twelve different teachers. We know which teachers are good and what their strengths are. We also know those who should be fired and never are.
We all evaluate a president of the U.S. and none of us on this blog has ever been one. If we can evaluate the leader of the free world, we are certainly qualified to evaluate our kids’ teacher.
Jerry Eads
January 9th, 2013
9:02 pm
PT: And many of the folks who run the projects and grants from the Gates group are not teachers either. Some probably are.
As we know, I’m a geek, not a teacher (okay, I work with teachers-to-be now, but I certainly don’t pretend to know anything other than experimental design, policy research, statistics and measurement). That put up front, I have to take two stands:
1: After having been beat up so unfairly for so many years from so many corners, it’s easy for some folks teaching to see things as yea or nay, black or white. While I understand that, I think from the evidence I see I’d say THAT’s unfair too. Gates got lucky (in addition to being brilliant). He made and is making an incredibly huge fortune. He could spend it only on yachts and 747s like an oil sheik, but instead he dumps a bunch of it into a foundation to do research and grants on education. I know it’ll come as a shock to you, but some of it’s pretty good stuff. For example, go ask some of your fellow Georgia teachers what they’ve been getting from something called LDC. At the same time, your very own elected state government stole $1.7 billion from our schools for, among other things, fishing contests. There are two absolutes. One is that everything is relative. The other is that there are no absolutes.
2. I’ve been looking at the tech on the MET study a bit this aft. Is it perfect? Of course not. It’s the real world. AND THEY’RE HONEST ABOUT IT. Is it an interesting step toward trying to understand how we might do teacher evaluation so it’s not totally worthless? A WHOLE lot more useful than what we’ve seen from some other corners.
Temper thy opinions with reason and information, everyone.
Private Citizen
January 10th, 2013
6:14 am
Gates got lucky (in addition to being brilliant). He made and is making an incredibly huge fortune. He could spend it only on yachts and 747s like an oil sheik, but instead he dumps a bunch of it into a foundation to do research and grants on education.
Mr. Gates did not “get lucky.” He is a shrewd hard-nosed and sly monopolist. Did you miss the part about the company he created getting fined a bilion dollars in europe for being monopolists? Europe still protects the idea of “market” and the USA does not, evidenced by that I have no choice where to buy wired internet service, which is completely outrageous. Anyway, my point is that Gates is a strategist and my guess is that every bit of his education massaging is connected to a business strategy to extract money for services of some kind. Pretty keen when he and ilk will be providing the services.
Jerry, from the perspective of a teacher (me), let me ask you something. How come not a single one of these education initiatives includes materials for teachers to work with? Every body wants to “grade the teacher” meanwhile, in my experience, some teachers (math and science) have materials (books and software) to work with and basically, the rest of the teachers have either none, minimal, or patchy obsolete resources. I have not seen a single soul anywhere address that teachers are demanded to do all of these things, so to speak, meanwhile, as someone indicated, let’s hope the toilets are flushing in the antiquate bathrooms, and that the one copying machine in the building is working, since many teachers “steal stuff from the internet” and use a photocopier to reproduce it. There is just a gigantic huge inefficiency going on when so many teachers are required to work on their own, like an island, sourcing their own teaching materials. It is like it is the biggest lie of omission from big power. It makes me crazy in the head, as I have spent so many hours having to source and make materials when I should have organized high quality materials available and instead have to hobble things together. Until any of these educa
Private Citizen
January 10th, 2013
6:18 am
Until any of these education foundations, etc. get real about the tools teachers need to work with, I think it is appropriate to question the legitimacy of their assertions, which seem to be limited in relevance to teaching, and specific to some business objective to benefit (set the hook, get the contract) the foundation or it’s associated businesses.
Jerry Eads
January 10th, 2013
8:26 pm
PC: I repeat, ask your fellow teachers about LDC.
AND, there’s no doubt that for the past 40 years (more, but that was the advent of mincomp testing) the joy of teaching for most folks has been harder and harder to find. There’s LOTS to be angry about. All I’m saying, again, is that everything is relative. Few things are all bad (even, maybe, Georgia government). The Gates Foundation would be very difficult to position as nothing but a money grub for Microsoft from the actual data. I DO understand your anger, but, I again offer, temper your anger with reason and information, not just opinion.
Joe'smom
January 11th, 2013
8:43 am
As an educator of many years I am so sick of those individuals who have NEVER so much as seen a grade book telling me what I should be doing to “perform” more effectively! As teachers it is time for us to rise up and take back our profession if there will ever be any hope for our students.