Brookings Institution: Never judge a book by its cover—use student achievement instead


By Thomas J. Kane, Brookings Institution

They say never judge a book by its cover. We need to start judging textbooks and other instructional materials using student achievement instead.

In February and March of last year, as teachers were preparing for the first administration of the end-of-year assessments from the Partnership for Assessment of Readiness for College and Career (PARCC) and the Smarter Balanced Assessment Consortia (SBAC), the Center for Education Policy Research at Harvard surveyed a representative sample of 1,500 teachers and 142 principals across five states.[i] Our report focused on the instructional changes teachers were making and the effectiveness of the training and support they were receiving. However, we also asked about the textbooks they were using.

We matched each teacher to the students they were teaching and assembled data on students’ demographic characteristics, performance on prior state tests, and the averages of such characteristics for the peers in their classroom. We also estimated each teacher’s impact on student performance in the prior school year (2013-14) to use as a control. (We wanted to account for the fact that more effective teachers may choose to use particular textbooks.) After controlling for the measures of student, peer, and teacher influences above, we estimated the variance in student outcomes on the new assessments associated with the textbook used.[ii]

The textbook effects were substantial, especially in math. In 4th and 5th grade math classrooms, we estimated that a standard deviation in textbook effectiveness was equivalent to .10 standard deviations in achievement at the student level.[iii] That means that if all schools could be persuaded to switch to one of the top quartile textbooks, student achievement would rise overall by roughly .127 student-level standard deviations or an average of 3.6 percentile points. Although it might sound small, such a boost in the average teacher’s effectiveness would be larger than the improvement the typical teacher experiences in their first three years on the job, as they are just learning to teach.

As advocated by Russ Whitehurst and Matt Chingos, the search for more effective curriculum materials can yield outsized bang-for-the-buck, because schools are already buying textbooks and better textbooks do not cost more on average than less effective ones. We estimate that such a study would need to collect data from roughly 1,800 schools to have the statistical power necessary to detect a .10 impact for any textbook representing at least 10 percent of the market. Some states, such as Indiana and Florida, already collect data on textbook adoptions, but even if it were necessary to collect data from a new sample of schools, the study would likely cost less than $2 million annually.[iv] Across the PARCC and SBAC states, there are approximately 3 million 4th and 5th grade students. Assuming the more effective textbooks have a similar cost to the textbooks they replace, the incremental cost of the 3.6 percentile point gain in achievement would be the cost of the study itself—roughly 67 cents per student.

Read the whole story by visiting the Brookings Institution >

[i] The five states were Maryland, Massachusetts, Delaware, New Mexico, and Nevada. Unfortunately, we could not use the testing outcomes in Nevada due to problems with the SBAC assessment in the spring of 2015.
[ii] We dropped the 20 percent of students whose teachers reported they did not use a textbook, an additional 30 percent whose teachers did not use their textbook as their primary curriculum, and limited the sample to textbooks that at least two teachers reported using.
[iii] Others have found big effects of textbooks and curricula on student achievement. For instance, the What Works Clearinghouse (WWC) currently contains ten evaluations of elementary school math textbooks and software, with effect sizes ranging from roughly −2 to 17 percentile points.
[iv] Chingos and Whitehurst advocated for a similar approach, suggesting that the Institute of Education Sciences should develop standards for collecting data on instructional materials. Rachana Bhatt and Cory Koedel have studied the effect of elementary math curricula by matching schools on baseline characteristics and comparing performance on the test in Indiana. See Bhatt, Rachana and Cory Koedel, “Large-Scale Evaluations of Curricular Effectiveness: The Case of Elementary Mathematics in Indiana” (2012).