Every two weeks, Vermont Education Commissioner Richard Mills visits a different region of the state to see what's happening in classrooms and to talk with local educators and community members. Although guided tours are de rigueur for most outside visitors to schools, Mills' excursions are more purposeful: He likes to look through portfolios of student work. His reason? "You get an immediate perception of what's important in that school and in that classroom" by examining the portfolios, Mills says.
Talking with pupils about their portfolios, Mills is impressed with their insights into how good their work is and why. Recalling one recent conversation with a group of 8th graders, he says, "What was notable was that they could talk about their own performance in relation to standards. They were able to talk about specific standards that they could meet with relative ease, and other standards for which no amount of sweat seemed to work. It was quite a sophisticated conversation, and I was quite taken by this group."
In 1991–92, Vermont became the first state to use portfolios in a statewide assessment program, requiring all students in grades 4 and 8 to keep portfolios in writing and mathematics. The state's efforts are being closely watched by proponents of portfolios and performance assessment generally. Vermont is testing whether performance assessment can fuel improvements in classroom practice. Moreover, Vermont's experiences will shed light on whether the psychometric properties of performance assessments are rigorous enough to warrant the curtailment of traditional standardized exams, which are widely used outside Vermont.
- Teachers score their students' portfolios according to established criteria. A sample of portfolios is then drawn from each class, to be scored by an external group of trained raters in a process known as "moderation."
- In writing, all students select a "best piece" from their folders to represent their best work as writers. That piece, as well as the remainder of the portfolio (which must include samples of different kinds of writing), is evaluated. The writing samples are judged against five criteria: purpose, organization, detail, voice/tone, and grammar/usage/mechanics.
- In mathematics, each portfolio is expected to include from 10 to 20 items. From these, students and their teachers cull 5–7 "best pieces" for evaluation. Four of the rating criteria are related to problem solving, and three are related to mathematical communication.
- In addition to the portfolio work, Vermont students in grades 4 and 8 take a "uniform test" (a timed writing sample and a series of mathematics problems). The uniform test and the portfolio work constitute the state assessment program.
Mixed Results
Results from the 1992–93 statewide assessment will not be ready until this fall. But the results from the 1991–92 portfolio assessment program—and a subsequent analysis by the RAND Corporation—provide some preliminary evidence on the issues of instructional impact and scoring.
The evidence so far suggests that portfolios can be a potent force for improving classroom instruction. Portfolios "have had the biggest impact on helping teachers improve instruction," says Doug Harris, superintendent of the Franklin Northwest Supervisory Union, which is a member of ASCD's Assessment Consortium. "I've seen some real changes in teacher behavior—probably as dramatic as anything I've seen in 22 years."
An evaluation by the RAND Corporation of the 1991–92 portfolio process found that both teachers and principals felt that the use of portfolios had positive effects on instruction. For example, in mathematics, teachers increased their emphasis on problem-solving strategies and mathematical communication—the focus of the scoring criteria and the national standards developed by the National Council of Teachers of Mathematics. "I now put more emphasis on problem solving and writing," says Aldo Bianchi, an 8th grade math teacher at Missisquoi Valley Union High School. For example, students are called on to explain their reasoning and to use representations such as charts or graphs.
Another widely touted benefit of the Vermont plan is that students, educators, and parents are all more familiar with criteria for good work in mathematics and writing. The tough standards by which portfolios are judged—the result of months of work by educators—show that the profession holds high standards for itself, says Mills. "There's pressure coming from the profession itself to make the standards common and high, and that is a dramatically different practice."
Students and parents also are learning about the new standards and judging criteria. For example, Sandy Williams, a 4th grade teacher at Highgate School, teaches students the criteria by which their work will be judged and provides exemplars of actual student work. Students begin to internalize these criteria as they do their work, she says. Many teachers also are using the portfolios to communicate with parents about their child's work during open houses or conferences, adds Clare Forseth, who teaches 6th grade at Marion Cross School in Norwich. Sometimes, students themselves explain their work, and parents "are very taken aback, surprised, and impressed" by students' abilities to analyze their strengths and weaknesses, she says.
Technical Questions
Vermont has major obstacles to surmount, however, if it is to meet the second goal of its statewide assessment program—providing accurate data for policy decisions, program planning, and evaluation.
The 1991–92 statewide assessment process yielded low reliability ratings that limited the amount of useful data. Reliability coefficients, which express the extent to which two raters rank students' work the same on a scale of 0.00 (no agreement among raters) to 1.00 (perfect agreement), are frequently .70 or higher for standardized performance assessments in writing, a RAND study states. In Vermont, reliability coefficients for portfolio scoring in writing and math ranged from .33 to .43 during the 1991–92 assessment. As a result, assessment results could be issued only at the state level (where the larger number of students assessed makes variation in scoring individual pupils less important). Vermont officials were unable to release scores for each supervisory union (a group of school districts) as expected, and plans to release scores at the school level depend on obtaining better reliability in scoring.
In 1992–93, the second year of statewide implementation of portfolios, Vermont took several steps to improve the process and bolster its technical soundness, says Mills. For example, fewer teachers were selected to do statewide scoring, and those who were received more training. Checks for reliability also were made during the scoring. The new approach is "much more tightly controlled and much more intensely supported," Mills adds.
Even if technical limitations prevent some of the intended uses of the data, some Vermont educators say the portfolio program already has had a positive effect on classrooms. "I don't worry about the scoring results," says Bianchi. The new directions in his math classes, brought on with the portfolio program, have helped students to think more deeply and creatively about their work. Students are considering several ways to solve a problem and justifying their reasoning "instead of just pushing numbers around on the paper," he says. "I'm really excited about this."