How has the Measures of Effective Teaching project helped show how effective teaching can be measured fairly and consistently?
1. WHAT IS THE MET PROJECT?
The MET project was a research partnership between 3,000 teacher volunteers and dozens of independent research teams. The project’s goal was to build and test measures of effective teaching to find out how evaluation methods could best be used to tell teachers more about the skills that make them most effective and to help districts identify and develop great teaching. Launched in 2009, the study has identified multiple measures and tools that – taken together – can provide an accurate and reliable picture of teaching effectiveness. By understanding what great teachers do and by improving the ways teachers gain insight into their practice, we can help more teachers develop their practice and achieve success for their students.
Research shows that a teacher’s contribution matters more than anything else within a school. More than class size. More than school funding. More than technology. For decades, most initiatives to improve public education have focused on improving poor performing schools. But studies show that there are bigger differences in teaching quality within schools than there are between schools. This means that in the same school, a child taught by a less effective teacher can receive an education of vastly different quality than a student just down the hall who is taught by a more effective teacher. And the way evaluations are currently conducted don’t provide a teacher who is struggling with a roadmap to improve.
Because teaching is complex, no single measure can capture the complete picture of a teacher’s impact; yet many evaluation systems use tools that measure only a few aspects of teaching. The information that results provides teachers with very limited, occasional feedback to help develop their practice. Multiple measures are needed to help school leaders understand how teaching contributes to student success. By evaluating multiple aspects of teaching, instructors and school leaders can create better professional development programs that promote proven techniques and practices that help students learn, and can make better-informed hiring and tenure decisions.
The project was funded by the Bill & Melinda Gates Foundation as part of ongoing efforts to give teachers the tools they need to be successful and to improve student achievement in public schools across the United States.
2. WHO WAS INVOLVED IN THE MET PROJECT?
Truly understanding the skills and techniques that are hallmarks of great teaching requires collaboration between educators and researchers. The MET project was unprecedented: it was a partnership among thousands of teacher volunteers and administrators and union leaders from school districts across the country and dozens of independent researchers and education organizations.
Teachers – more than 3,000 of them – were at the heart of the study. The teacher volunteers were recruited from Charlotte-Mecklenburg Schools, N.C.; Dallas Independent School District, Texas; Denver Public Schools, Colo.; Hillsborough County Public Schools, Fla.; Memphis City Schools, Tenn.; New York City Department of Education, N.Y.; and Pittsburgh Public Schools, Pa. Pittsburgh served as the project’s pilot district, but no data from this district was analyzed.
Lead researchers involved in the project were affiliated with Dartmouth College, Harvard University, the University of Michigan, the University of Virginia and Stanford University. Participating non-profits and education companies included RAND Corporation, Educational Testing Service, Teachscape, The Danielson Group, The New Teacher Center, National Math & Science Initiative and Westat.
The project was funded by the Bill & Melinda Gates Foundation.
3. WHAT MEASURES OF EFFECTIVE TEACHING DID THE MET PROJECT INCLUDE IN ITS ANALYSIS?
Because great teaching is multi-faceted, it takes multiple measures to give a complete picture of a teacher’s effectiveness. The goal of the MET project was to identify which measures give the best and most accurate information about how well a teacher helps his or her students learn – and how these measures should be used together to see the whole picture of a teacher’s effectiveness.
Many of the measures that were studied by the MET project were well established, though some were new. They included:
- The Tripod Student Perception Survey, developed by Ron Ferguson at Harvard and administered by Cambridge Education. This survey, developed and refined over the past ten years, captures students’ perceptions of their classroom experiences, providing teachers with actionable feedback about how to improve teaching practices that lead to student learning.
- The Content Knowledge for Teaching (CKT) test, a new written assessment developed by the Educational Testing Service meant to gauge teachers’ understanding of teaching strategies.
- Protocols for classroom observation designed to help school leaders assess teaching effectiveness by watching teachers in action.
- The Framework for Teaching, developed by Charlotte Danielson
- Classroom Assessment Scoring System (CLASS), developed at the University of Virginia
- Mathematical Quality of Instruction (MQI), developed at the University of Michigan
- Protocol for Language Arts Teaching Observations (PLATO), developed at Stanford University
- Quality of Science Teaching (QST), developed at Stanford University
- UTeach Teacher Observation Protocol (UTOP), developed at the University of Texas-Austin for assessing math and science instruction
4. HOW HAS THE MET PROJECT DETERMINED WHICH MEASURES ARE RELIABLE AND VALID?
Teaching evaluations are only valuable if they can be counted on to accurately reflect what a teacher is doing in his or her classroom that is helping students succeed – in research terms, they must be “reliable” and “valid.”
A teaching effectiveness measure is considered “reliable” when its results for a teacher are reflective of his or her typical performance, and not the idiosyncrasies of a particular observer, lesson or group of students.
“Valid” teaching effectiveness measures are those that are proven to lead to student learning. Many current evaluation systems, by contrast, are subjective observations based on checklists of teaching practices that have not been demonstrably linked to promoting student learning.
Reliability and validity represent the key trustworthy tests that teachers and education leaders are applying as they create new systems of teacher evaluation, so they also represent the key questions the MET project was designed to investigate.
The 3,000 teacher volunteers who participated in the project agreed to be evaluated — for research purposes only — using the teaching effectiveness measures being studied. (Data on individual teachers that was collected for the project is confidential and will never be shared with principals or other school or district personnel.) MET project researchers have studied how well each measure identifies effective teaching by studying the relationship between each teacher’s results on the measures utilized through the study and his or her students’ learning. The researchers then determined what combination of the measures best predicted teacher and student success.
5. WHY IS IT IMPORTANT FOR A TEACHING EVALUATION SYSTEM TO INCLUDE MULTIPLE MEASURES?
Teachers, school leaders and education researchers agree that teaching is too complex to be fully captured by a single measure like student test scores. Just as importantly, student test scores alone do not give teachers constructive feedback about how they can improve in the classroom and better help their students succeed. Feedback that captures the true range of professional skills and competencies that teachers must employ can be gathered from several sources, including student surveys, classroom observations and student test scores. Together these sources can give teachers the information they need to continue to confirm what they’re doing right and where they need to improve. The information can also help districts develop evaluation systems and professional development programs that support great teaching and promote student achievement.
6. WHY IS IT IMPORTANT TO VALIDATE MEASURES OF EFFECTIVE TEACHING AGAINST STUDENT ACHIEVEMENT?
Effective teaching is teaching that leads to student success. The feedback school leaders give to teachers must help them improve in ways that are linked to student learning. Validation is about testing the alignment among measures and outcomes to ensure that they’re helping provide teachers with useful information that allows for a path forward, and not information that pulls them in different directions. Validation also keeps teacher evaluation focused on supporting the core business of schools: student learning. By testing different measures of teaching effectiveness to see how well each one links to student learning, the MET project has helped ensure that school systems can identify and support great teaching.
Teachers need to know that the feedback they receive from evaluation can help them achieve greater success for their students in terms of learning outcomes.
7. HOW HAS THE MET PROJECT MEASURED STUDENT ACHIEVEMENT?
The MET project validated all its measures against the difference between actual and expected student achievement on standardized tests using a method sometimes called “value-added.” A teacher’s contribution to student achievement gains is estimated by comparing the achievement gains made by his or her students to those made by teachers with students with similar characteristics, including similar prior achievement levels. By comparing a teacher’s student achievement gains with his or her results on the other measures of teaching effectiveness that were examined by the study, MET project researchers investigated which teaching practices, and which teaching effectiveness measures, best predict future achievement gains.
8. WHICH STUDENT TESTS DID THE MET PROJECT USE TO MEASURE STUDENT ACHIEVEMENT?
The MET project measured student achievement using two types of standardized tests: the state standardized test and a supplemental test. The supplemental tests used by the MET project were:
- Stanford 9 Open-Ended Reading Assessment, in grades 4-8
- Balanced Assessment of Mathematics (BAM), in grades 4-8
- ACT QualityCare series for Algebra I, English 9, and Biology
While the state tests are designed to measure how well students have learned the in-state standards, the supplemental tests tend to measure more reasoning skills and conceptual understanding. The two types of tests together provide a more complete picture of student achievement than either one alone.
9. WHICH FINDINGS HAS THE MET PROJECT RELEASED?
Through its close collaboration with teachers, school districts, research organizations and advocacy groups across the country, the MET project has shared practical insights and tools throughout the course of the study. These insights are helping to support teachers and students in classrooms today.
School districts across the country continue to examine ways to improve their teaching evaluation systems. To help inform this work, the MET project released progress reports as research and analysis progressed. Through these reports, practitioners and policy makers had quick access to the data developed through the project in order to begin thinking through the practical implications of the work prior to release of the final report in January 2013.
- The MET project’s first preliminary findings (released December 2010) showed that surveying students about their perceptions of their classroom environment provides important information about teaching effectiveness, as well as concrete feedback that can help teachers improve.
- The project’s second set of preliminary findings, released in January 2012, examined classroom observations and offered key considerations for creating high-quality observations systems.
- The MET project’s final analysis, released in January 2013, showed, though a large-scale study that randomly assigned teachers to classrooms of students, that it is indeed possible for measures that account for students’ different starting points to identify teaching that better helps students learn. Also in January 2013, the MET project added to earlier findings by reporting that:
- Multiple measures produce more consistent performance ratings than student achievement measures alone.
- Composites that combine individual measures can help indicate multiple aspects of effective teaching while avoiding too-narrow a focus on any one aspect, when they are balanced so that between half and one-third of the weight is assigned to student achievement gains.
- The MET team and its research partners have been part of ongoing national, state and district-level conversations among educators about the latest findings and advances in this field.
10. WHAT ARE EXAMPLES OF THE KINDS OF TOOLS THAT THE MET PROJECT HAS DEVELOPED?
First and foremost, the final report for the MET project has not set forth a recommended formula or system for the measurement or evaluation of teaching effectiveness. Instead, the final report explains the researchers’ findings about the way in which each measurement tool studied yields valuable information about teaching effectiveness, with insights about how districts might think about using that information to help teachers improve and support student success.
In addition, the MET project has developed a set of tools, resources and implementation guides to support schools and districts as they design or refine evaluation systems that accurately and reliably identify great teaching. The tools developed include:
- Tools for testing observation raters and instruments. Recognizing the pressing need within districts to ensure that classroom observations are valid and reliable, the MET project used its extensive data and analytical resources to develop a rater “certification” tool districts can use to assess whether the individuals they train can make accurate judgments. “Validation engine” for testing observation instruments. States and districts can use this online software to determine for themselves if the tools they use for classroom observations are valid and reliable. Trained raters in a school district can score videotaped lessons from the validation engine’s online library. School leaders can then see whether their observation tool accurately identified the effective teachers, and whether there is consistency among the school’s raters.
- Panoramic camera for taping classroom practice. To capture videotaped lessons for the classroom observations portion of the study, the MET project developed a small, easy-to-use camera that records a nearly 360-degree view of a teacher’s classroom. The camera does not interfere with classroom activity and can be set up easily by teachers who then upload their videos to a secure website. The camera allows teachers and observers to see how the entire class was reacting during the lesson and provides greater perspective and insight into the teaching dynamic. Designed specifically for this study by MET project partner Teachscape, the camera is available for purchase by schools and districts and is already being used by schools across the country for teacher professional development.
The MET project is a large-scale, multi-year study. The data set that has been collected through this process is being made available to the research and practitioner community, free of charge, through the University of Michigan. Our hope is that access to this data will encourage other researchers to conduct and share their own analysis, helping to expand knowledge in the field overall.
In addition, in the 2011-12 school year, the MET project embarked on a new round of data collection aimed at creating an online video Library of Practice. Approximately 360 teachers are helping to build this collection for use in professional development. They will record more than 50 lessons by the end of the 2012-13 school year and will make these lessons available to states, school districts, and other organizations committed to improving effective teaching.
Videos from this project are being made available online for professional learning through the SOE Teaching and Learning Exploratory, run by the University of Michigan School of Education. Click here for more information.