Tennessee Value Added Assessment System
Comments and links by Sherman Dorn
Table of Contents
Introduction
In the early 1990s, Tennessee Governor Ned McWherter wanted the legislature to reform the financing of public education in the state. His first idea, which floundered, was the creation of a state income tax. He finally settled for a half-cent increase in the state sales tax to bring many county systems into line with the rest of the state (Educational Improvement Act, 1992). (The fact that many of the counties had brought a suit against the state's educational financing system helped spur McWherter's efforts.)
Along with the financing reform, however, came a broad array of other efforts to improve education in an omnibus bill passed in 1992. One such measure was the creation of a statistical system for measuring student gains on achievement tests from one year to the next, the Tennessee Value Added Assessment System (or TVAAS). Several legislators wrote the TVAAS into the bill based on advice from University of Tennessee statistician William Sanders, who had been testing a small version using data from a few Tennessee cities in the 1980s.
When I moved to Nashville for a postdoctoral position at Peabody College of Vanderbilt University, TVAAS was just starting its first year of reporting scores for systems and schools. The creation of a mechanistic system for producing "effect sizes" for individual schools—and eventually individual teachers—struck me as the epitome of distrust of teachers. The hysteria among newspapers to publish the scores and then derive school rankings also struck me as the wrong way to make schools accountable, by some technocratic mechanism. We want teachers to ask the hard questions, every day, of how to help students. Giving them abstract scores of "this is how your students learned, based on a few hours' of testing" would neither help them nor encourage them to ask the hard questions. It creates, instead, a very high stakes environment that makes many teachers defensive.
I was also part of an Internet discussion of educational policy (EDPOLYAN, now defunct), and at some point in the fall of 1994, Michael Scriven (another participant in EDPOLYAN), made an off-hand remark about value added assessment in general. I made an equally off-hand remark stating that I hoped his confidence was not based on Tennessee. Later that fall, Sandra Horn and William Sanders of the University of Tennessee Value Added Research and Assessment Center (UT VARAC) sent some responses back to the list. I read the suggested materials (e.g., Sanders & Horn, 1994), corresponded by e-mail with Ms. Horn, and decided that, while Sanders had been very well-intentioned, my original impression was largely correct. Then again, I am an educational historian, and the assumption that statistical systems can perfect school administration is an error from the Progressive Era and no surprise to me. In December 1994, I posted a long description of my concerns to EDPOLYAN, UT VARAC and others responded, and we had a hearty discussion for more than a month. Below is a very small sampling of that discussion and the core of my criticism.
Since the EDPOLYAN debate, TVAAS has continued, though with some substantial criticism in the state. Prompted by questions about TVAAS by teachers and administrators, the state comptroller investigated some of the results and suggested an external, independent review. Bock and Wolfe (1996) concluded that the basic statistical system was sound but that estimates of school effects could vary widely, that some of the tests used for TVAAS had too few items for reliability, and suggested that the use of teacher scores wait until the state could verify that teacher scores confirmed principal and other administrative judgment of excellent and poor teachers. The state is delaying the use of teacher scores for evaluation until the state has more research, and according to Bock (in discussion at the 1997 American Educational Research Association meeting), the state is adding more questions to the social studies and science tests. However, as noted below, the state has ignored others. Fisher (1996) also criticized the politics of TVAAS, but I know of no evidence that his criticism of TVAAS-as-policy has made any difference in Tennessee at all.
In early December 1996, some legislators wanted to dismantle TVAAS, but some business leaders were making clear that their support of educational reform depended on the maintenance of TVAAS. In the spring of 1997, Education Commissioner Walters recommended eliminating all but the 10th-grade subject tests in high school and making the 2nd grade tests voluntary, as well as scaling back the probationary measures for school systems. The legislature eventually passed amendments which delayed the subject tests in high school for at least a year, made the 2nd grade tests voluntary (with the promise to replace them with diagnostic tests in reading and "other basic skills"), and postponed formal probation for school systems until a year of being put "on notice" that their performance was inadequate.
Back to the Table of Contents.
Original sources
The following are links to some sources on TVAAS. I cannot vouch for their longevity. They may have expired at some point without my knowledge.
- Gene Glass's compilation of EDPOLYAN discussion
- 1996 TVAAS report (http://www.state.tn.us/education/rptcrd96/valuadd.htm), the state's official "report card" on education.
- Sample newspaper report on one school (http://edge.edge.net/cgi-bin/schrpt?M%Davidson+County%Goodlettsville+Elem+School). The Nashville Tennessean is still making point estimates of school scores, including scores from the social studies and science tests (which the external evaluators noted had too few items for reliable results). The staff at the newspaper took the state's figures and juggled them to come up with point estimates of percentile rankings of schools; here is one example. This journalistic exercise in statistical manipulation shows the danger when the state ignores the external evaluators' strongest suggestions for change (eliminating point estimates for schools and not using the social studies and science tests).
Back to the Table of Contents.
References
Bock, R. D., & Wolfe, R. (1996, March 15). A review and analysis of the Tennessee Value-Added Assessment System. Part I: Audit and review of the Tennessee Value-Added Assessment System (TVAAS): Final report. Nashville, TN: Comptroller of the Treasury.
Camilli, G. (1996). Standard errors in educational assessment: A policy analysis perspective. Educational Policy Analysis Archives, 4(4). URL: http://olam.ed.asu.edu/epaa/v4n4.html
Educational Improvement Act
, Chapter No. 353 enacted March 11, 1992, by the Tennessee Legislature.Fisher, T. H. (1996, January). A review and analysis of the Tennessee Value-Added Assessment System. Part II. Nashville, TN: Comptroller of the Treasury.
Sanders, W. L., & Horn, S. (1994). The Tennessee Value-Added Assessment System (TVAAS): Mixed-model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8, 299-311.