Introduction

I believe that most teachers, in fact, do understand this reality. But we often don't carry it through into our assessment practices. Studies analyzing classroom tests, over many decades, have found that most teacher-made tests require only recall of information (Marso & Pigge, 1993). However, when teachers are surveyed about how often they think they assess application, reasoning, and higher-order thinking, both elementary (McMillan, Myron, & Workman, 2002) and secondary (McMillan, 2001) teachers claim they assess these cognitive levels quite a bit. Although some of this discrepancy may come from recent advances in classroom practices that emphasize higher-order thinking, it is also clear that many teachers believe they are assessing higher-order thinking when, in fact, they are not.

The reason that recall-level test questions are so prevalent is that they are the easiest kind to write. They are also the easiest kind of question to ask off the top of your head in class. Teachers who do not specifically plan classroom discussion questions ahead of time to tap particular higher-order thinking skills, but rather ask extemporaneous questions "on their feet," are likely to ask recall questions.

This situation is true for even the best teachers. After participating in professional development about questioning, one high school social studies teacher wrote the following:

Upon reflection, it became obvious that many of the questions I have asked were at a lower-order thinking, or simply recall or factual response, level. [I am now …] more aware of the necessity for higher-order or open-ended questions in class. Many of the students also now understand the importance of the many different types of questions that can be asked.

The same thing happens on classroom tests. Teachers who put together tests quickly, or who use published tests without reviewing them to see what thinking skills are required, are likely to end up asking fewer higher-order-thinking questions than they intended. Contrary to some teachers' beliefs, the same thing also happens with performance assessments. Students can make posters or prepare presentation slides listing facts about elements, planets, or stars without using higher-order thinking, for example. Of course, what amount and what kind of higher-order thinking should be required on a classroom assessment depend on the particular learning goals to be assessed.

Most state standards and district curriculum documents list goals for learning that include both knowledge of facts and concepts and the ability to use them in thinking, reasoning, and problem solving. The purpose of this book is to clarify what is involved in several different aspects of higher-order thinking, and, for each, to show how to write good-quality, well-planned assessments.

What Is Knowledge?

The nature of human thought and reason is the subject of a field of philosophy called epistemology. Epistemologists still debate the definition of knowledge. A classic definition, based on ideas in Plato's dialogue Theaetetus, is that for something to count as knowledge it must be justified, true, and believed. Branches of philosophy have developed to describe what count as reasonable and plausible justifications, what counts as truth, and the nature of belief.

I use this tidbit about Plato to make what I consider an important point. Even seemingly simple knowledge rests on some historical higher-order thinking. Facts and concepts did not just fall out of the sky—or out of a textbook. They were discovered and debated until they came to be widely held as true, and widely believed. When we teach students to do higher-order thinking, we are not just teaching them some fancy skills useful for the flexibility and adaptability required for life in our 21st century "information age." We are teaching them to be human.

What Is Higher-Order Thinking?

If we agree to stay grounded in this important purpose, our definitions of higher-order thinking for the purposes of this book can be much more modest and practical. In this Introduction, we consider the kinds of higher-order thinking that are (or should be) stated or implied in state content standards and classroom learning objectives. Definitions that I find helpful fall into three categories: (1) those that define higher-order thinking in terms of transfer, (2) those that define it in terms of critical thinking, and (3) those that define it in terms of problem solving.

Here is a definition in the transfer category:

Two of the most important educational goals are to promote retention and to promote transfer (which, when it occurs, indicates meaningful learning) … retention requires that students remember what they have learned, whereas transfer requires students not only to remember but also to make sense of and be able to use what they have learned. (Anderson & Krathwohl, 2001, p. 63)

The critical thinking category includes this definition:

Critical thinking is reasonable, reflective thinking that is focused on deciding what to believe or do. (Norris & Ennis, 1989, p. 3)

Another example in this category comes from Barahal (2008), who defines critical thinking as "artful thinking" (p. 299), which includes reasoning, questioning and investigating, observing and describing, comparing and connecting, finding complexity, and exploring viewpoints.

In the problem solving category are these two definitions:

A student incurs a problem when the student wants to reach a specific outcome or goal but does not automatically recognize the proper path or solution to use to reach it. The problem to solve is how to reach the desired goal. Because a student cannot automatically recognize the proper way to reach the desired goal, she must use one or more higher-order thinking processes. These thinking processes are called problem solving. (Nitko & Brookhart, 2007, p. 215)

As you explore new domains you will need to remember information, learn with understanding, critically evaluate ideas, formulate creative alternatives, and communicate effectively. [A problem-solving] model can be applied to each of these problems … to help you to continue to learn on your own. (Bransford & Stein, 1984, p. 122)

Of course, the first thing that may strike you as you read these definitions is that there is a lot of overlap. In the discussion here, and in the chapters that follow, this overlap will be apparent as well. I discuss the definitions separately in the following sections and give practical advice for assessment of these different aspects of higher-order thinking in Chapters 2 through 6, for analytical reasons. As any taxonomy of higher-order thinking skills shows, pulling a concept apart and discussing its various aspects is one way of understanding it. Think of this book as an analysis of classroom assessment of higher-order thinking.

Higher-Order Thinking as Transfer

The most general of the approaches to higher-order thinking is the Anderson and Krathwohl (2001) division of learning into learning for recall and learning for transfer. Learning for recall certainly requires a type of thinking, but it is learning for transfer that Anderson, Krathwohl, and their colleagues consider "meaningful learning." This approach has informed their construction of the Cognitive dimension of the revised Bloom's taxonomy.

For many teachers, operating with their state standards and curriculum documents, higher-order thinking is approached as the "top end" of Bloom's (or any other) taxonomy: Analyze, Evaluate, and Create, or, in the older language, Analysis, Synthesis, and Evaluation (Anderson & Krathwohl, 2001). Chapter 2 discusses assessing higher-order thinking conceived of as the top end of a cognitive taxonomy.

The teaching goal behind any of the cognitive taxonomies is equipping students to be able to do transfer. "Being able to think" means students can apply the knowledge and skills they developed during their learning to new contexts. "New" here means applications that the student has not thought of before, not necessarily something universally new. Higher-order thinking is conceived as students being able to relate their learning to other elements beyond those they were taught to associate with it.

There is a sense in which teaching for transfer is a general goal of education. Many teachers use the phrase "What are you going to do when I'm not here?" Most of the time, this reflects teachers' appreciation of the fact that their job is to prepare students to go into the world ready to do their own thinking, in various contexts, without depending on the teacher to give them a task to do. Life outside of school is better characterized as a series of transfer opportunities than as a series of recall assignments to be done.

Higher-Order Thinking as Critical Thinking

Critical thinking, in the sense of reasonable, reflective thinking focused on deciding what to believe or do (Norris & Ennis, 1989) is another general ability that is sometimes described as the goal of teaching. In this case, "being able to think" means students can apply wise judgment or produce a reasoned critique. An educated citizen is someone who can be counted on to understand civic, personal, and professional issues and exercise wisdom in deciding what to do about them. As we all learned in American history class, Thomas Jefferson argued this point explicitly. He believed that education was necessary for freedom, that having a citizenry that could think and reason was necessary for a democratic government.

The goal of teaching here is seen as equipping students to be able to reason, reflect, and make sound decisions. Higher-order thinking means students can do this. One of the characteristics of "educated" people is that they reason, reflect, and make sound decisions on their own without prompting from teachers or assignments.

Wisdom and judgment are particularly important in higher-order thinking tasks like judging the credibility of a source, always an important skill but newly emphasized in the era of ever-expanding, electronically available information. Identifying assumptions, a classic skill, also is very relevant today. As school and society become increasingly diverse, it is less likely that everyone's assumptions will be similar. Identifying the assumptions behind points of view—what students might call "seeing where you're coming from"—is a true life skill.

Examples of the importance of critical judgment occur in all disciplines. Literary criticism involves both analyzing works of literature and evaluating to what degree the piece of writing succeeds in accomplishing the author's purpose. Advertisers estimate the effect of various advertising strategies on different audiences. Closer to home, students estimate the effects various arguments might have in persuading their parents of their point of view. All of these involve critical judgment about purposes and assumptions and about the relative effectiveness of various strategies used to meet these purposes.

To help students learn to think by looking at works of art, Project Zero at Harvard University developed the "Artful Thinking Palette" (Barahal, 2008). Six thinking dispositions are listed around the image of a paint palette: exploring viewpoints, reasoning, questioning and investigating, observing and describing, comparing and connecting, and finding complexity. Although these dispositions were developed in the context of learning from visual art, they are good ways to approach other critical-thinking tasks as well. For example, try thinking about how these six approaches apply in the study of literature, history, or science.

Higher-Order Thinking as Problem Solving

A problem is a goal that cannot be met with a memorized solution. The broad definition of problem solving as the nonautomatic strategizing required for reaching a goal (Nitko & Brookhart, 2007) can also be seen as a broad goal of education. Every academic discipline has problems. Some are closed problems, like a set of math problems designed to elicit repeated practice with a particular algorithm. But many problems are open-ended, could have many correct solutions or multiple paths to the same solution, or are genuine questions for which answers are not known. Economists, mathematicians, scientists, historians, engineers—all are looking for effective or efficient solutions to both practical and theoretical problems. Educators are, too. Teachers propose a solution strategy for a complex problem—how to effectively teach a particular learning target to particular students in a given amount of time and with the materials available—every time they write a lesson plan. Many life problems are open-ended. For example, planning for and living within a budget is an open-ended problem most households deal with. People solve problems in many different ways, depending on the values and assumptions they bring to the task.

Bransford and Stein (1984) noted that problem solving broadly conceived—in a model they call the IDEAL problem solver, which I'll describe in Chapter 5—is the mechanism behind learning for understanding. This is a similar position to Anderson and Krathwohl's (2001) discussion of "meaningful learning." Bransford and Stein also point out that problem solving is the general mechanism behind all thinking, even recall. This may seem ironic, but think of it this way. To recall something, students have to identify it as a problem ("I need to memorize the capitals of all 50 states. How can I do that?") and devise a solution that works for them.

In fact, Bransford and Stein say that in addition to driving both recall and learning, problem solving is necessary for critical thinking, creative thinking, and effective communication. The role of problem solving in critical thinking (for example, "How well did this movie director accomplish his purpose with this film?") and communication (for example, "How can I write this review so that readers will be interested in seeing the movie?") seems pretty obvious. But does problem solving have a role in creativity? Isn't creativity the free-spirit, whatever-you-want kind of thinking? Actually, no. Most human creations, both inventions of things and inventions of social customs, were conceived to solve some sort of problem. The proverbial invention of the wheel, for example, solves a problem that can be expressed as "How do I get this heavy stuff from here to there?"

If you think of higher-order thinking as problem solving, the goal of teaching is equipping students to be able to identify and solve problems in their academic work and in life. This includes solving problems that are set for them (the kind of problem solving we usually think of in school) and solving new problems that they define themselves, creating something new as the solution. In this case, "being able to think" means students can solve problems and work creatively.

What Is the Effect of Assessing Thinking Skills?

When you teach and assess higher-order thinking regularly, over time you should see benefits to your students. Your understanding of how your students are thinking and processing what they are learning should improve as you use assessments specifically designed to show students' thinking. Ultimately, their thinking skills should improve, and so should their overall performance. Students learn by constructing meaning, incorporating new content into their existing mental representations; therefore, improving thinking skills should actually improve content knowledge and understanding as well. How large can we expect this effect to be?

Higgins, Hall, Baumfield, and Moseley (2005) did a meta-analysis of studies of thinking-skills interventions on student cognition, achievement, and attitudes. A meta-analysis is a quantitative synthesis of studies that reports effect sizes, or amount of change in standard-deviation units. Standardizing the effects from different studies means researchers can average effect sizes across studies, which yields a more stable estimate of the size of an effect—in this case, the effect of thinking-skills interventions—than any one study alone could provide. For their review, Higgins and his colleagues defined thinking-skills interventions as "approaches or programmes which identify for learners translatable mental processes and/or which require learners to plan, describe, and evaluate their thinking and learning" (p. 7).

Higgins and his colleagues found 29 studies, from all over the world but mostly from the United States and the United Kingdom, that were published in English and that reported enough data to calculate effect sizes. Nine of the studies were conducted in primary schools and 20 in secondary schools; most were in the curriculum areas of literacy (7 studies), mathematics (9 studies), and science (9 studies). Their purpose in doing the meta-analysis was to estimate the size of effects of teaching and assessing thinking skills, and they found very strong effects. The average effect of thinking-skills instruction was as follows:

• 0.62 on cognitive outcomes (for example, verbal and nonverbal reasoning tests), over 29 studies.
• 0.62 on achievement of curricular outcomes (for example, reading, math, or science tests), over 19 studies.
• 1.44 on affective outcomes (attitudes and motivation), over 6 studies.

Because of the small number of effect sizes of motivational outcomes, the average effect size estimate of 1.44 may be less reliable than the other two effect sizes. But even 0.62 is a large effect for an educational intervention, equivalent to moving an "average" class of students from the 50th percentile to the 73rd percentile on a standardized measure.

Overall, then, Higgins and colleagues' meta-analysis supports the conclusion that thinking-skills interventions are effective in supporting student improvement in thinking, content area achievement, and motivation. In the next sections I describe some specific studies from the United States that support this conclusion. The studies described only scratch the surface of research in this area, and I encourage readers who are interested to look up additional works.

Assessing Higher-Order Thinking Increases Student Achievement

Using assignments and assessments that require intellectual work and critical thinking is associated with increased student achievement. These increases have been shown on a variety of achievement outcomes, including standardized test scores, classroom grades, and research instruments, as the studies described here illustrate. These increases have been demonstrated in reading, mathematics, science, and social studies. And they have been documented particularly for low-achieving students.

Evidence from NAEP and TIMSS. Wenglinsky (2004) reviewed studies of the relationships between student performance on large-scale measures and instruction emphasizing higher-order thinking, projects, and multiple-solution problems. He reported clear evidence from both the National Assessment of Educational Progress (NAEP) and the Trends in International Mathematics and Science Study (TIMSS) that, in mathematics and science, instruction emphasizing reasoning was associated with higher scores in all grade levels tested. In reading, teaching for meaning (including thinking about main ideas, author's purpose, and theme, and using real texts) was associated with higher NAEP performance as well, although Wenglinsky reminds his readers that NAEP testing begins in 4th grade, so it does not shed light on approaches to teaching beginning reading. In civics, 4th graders who studied basic information about how government works performed better on NAEP, but by 8th grade, students whose instruction also included active involvement and thinking did better.

Evidence from an urban district. Newmann, Bryk, and Nagaoka (2001) studied the mathematics and writing assignments of Chicago teachers in grades 3, 6, and 8. Students who received assignments requiring "authentic intellectual work" (p. 2) made greater-than-average gains in reading and mathematics on the Iowa Tests of Basic Skills (ITBS), and in reading, mathematics, and writing on the Illinois Goals Assessment Program (IGAP). As the name suggests, the ITBS is a basic skills test. The IGAP was the state test in place in Illinois at the time of the study.

To do their study, Newmann, Bryk, and Nagaoka had to define what they meant by "authentic intellectual work." They contrasted two kinds of instruction: didactic and interactive. By "didactic" instruction, they meant the kind of instruction in which students learn facts, algorithms, definitions, and such. In didactic instruction, students are tested with "right-answer," recall-level questions or with problems that require application or problem solving just like what was done in class.

However, in "interactive" instruction, "students are often asked to formulate problems, to organize their knowledge and experiences in new ways to solve them, to test their ideas with other students, and to express themselves using elaborated statements, both orally and in writing" (Newmann et al., 2001, pp. 10–11). Readers will hear in this definition the kind of higher-order thinking discussed in this book. In this kind of instruction, students are assessed with nonroutine application of knowledge and skills. The researchers defined "authentic intellectual work" as requiring "construction of knowledge, through the use of disciplined inquiry, to produce discourse, products, or performances that have value beyond school" (p. 14). This kind of work was associated with one-year learning gains on the ITBS that were 20 percent greater than the national average. On the IGAP, students from classes that did this kind of work performed about half a standard deviation above students from classes whose work was very didactic. Students with both high and low prior achievement benefited.

Evidence for disadvantaged students. Pogrow (2005) designed the Higher Order Thinking Skills (HOTS) program specifically for educationally disadvantaged students, both Title I students and students with learning disabilities. The program specifically works on four kinds of thinking skills: (1) metacognition, or the ability to think about thinking; (2) making inferences; (3) transfer, or generalizing ideas across contexts; and (4) synthesizing information. In its 25-year history, the HOTS program has produced gains on nationally normed standardized tests, on state tests, on measures of metacognition, in writing, in problem solving, and in grade point average.

Two things make these results for the HOTS program particularly impressive. For one, in several of the evaluations, teaching thinking skills has been contrasted with enhanced content instruction. The thinking-skills instruction did a much better job of setting up the students to be flexible, allowing them to "understand understanding" (p. 70) and to handle all sorts of different content. For another, these results hold for about 80 percent of students who have been identified as Title I or learning disabled students, as long as they have a verbal IQ of 80 or above. It takes time, though. Pogrow (2005) reports that with these students, "It takes about four months before students will give a reason for a response without being asked, and it takes about six months before they will disconfirm a prior answer" (p. 71). But they do!

Assessing Higher-Order Thinking Increases Student Motivation

Studies have shown that holding students accountable for higher-order thinking by using assignments and assessments that require intellectual work and critical thinking increases student motivation as well as achievement. Students do not become engaged with their studies in the abstract, nor do they become motivated in the abstract. Rather, they become engaged in thinking about particular things and motivated to learn particular things. Higher-order thinking increases students' sense of control over ideas. Thinking is much more fun than memorizing.

A study of 3rd grade language arts. Meece and Miller (1999) studied elementary students' goal orientations (interest in mastery and interest in performing well), perceived competence, and strategy use in reading and writing. During the research project, some of the 3rd grade teachers expressed concern that their students showed mastery of skills and strategies on reading and writing tests but did not transfer those skills to actual reading and writing beyond the tests. Meece and Miller evaluated the 3rd grade assignments and found that most of them focused on individual skills, recall, and teacher control. Many assignments required one-word answers, for example. Meece and Miller helped teachers learn to devise assignments that required students to read extended material, write more than one paragraph, and collaborate with classmates. Students in classes where teachers gave these kinds of assignments regularly declined in their performance-goal orientation (meaning they were less inclined to want to do assignments for the sake of gaining the approval of others).

More interesting, work-avoidance scores of low-achieving students in these classes (from student questionnaires about schoolwork) decreased, whereas work-avoidance scores of low achievers in the regular classes stayed the same. This finding may seem like a conundrum. Arguably, work that required more reading and writing could have been more, not less, off-putting, especially to low achievers. But the opposite was the case. Low-achieving students were more motivated to do the thoughtful work than the one-word-answer drill work.

A study of 5th grade social studies. In a much smaller-scale study—but one very similar to something you could do in your own classroom—Carroll and Leander (2001) were concerned that their own 5th grade social studies students lacked interest in the topic and that many perceived it as difficult and not fun. Their master's thesis reported on a 14-week project to teach students learning strategies designed to improve higher-order thinking. They also instituted cooperative learning to allow students to think together.

Observations before the program suggested the average student was off-task during class about 20 percent of the time and inactive about 10 percent of the time. In a survey, less than half (47 percent) agreed that they were excited about learning, and less than half (47 percent) agreed that social studies assignments were easy. After a 14-week program that included teaching students questioning strategies, using graphic organizers, cooperative-learning research projects, and portfolio construction, the measures were repeated. This time, observations suggested the average student was off-task during class only about 10 percent of the time and inactive about 8 percent of the time. In the survey, 95 percent agreed that they were excited about learning, and 89 percent agreed that social studies assignments were easy. Students' grades on chapter-comprehension assignments improved as well.

A study of teacher and student perceptions of learner-centered practices. Meece (2003) reported on a study of 109 middle school teachers and 2,200 middle school students in urban, suburban, and rural communities. Both teachers and students completed surveys to assess the use of learner-centered teaching practices that stress higher-order thinking. For teachers, the only ratings correlated with student motivation and achievement were related to teachers' reported support for higher-order thinking. For students, ratings on all the learner-centered practice dimensions (including practices supporting higher-order thinking) were correlated with motivation and achievement. Higher-order thinking practices were the only practices found to be related to motivation from both teachers' and students' perspectives.

The Contents of This Book

This book is intended to help teachers assess the kind of complex thinking emphasized by current content standards in various disciplines. I first lay out principles for assessment in general and for assessment of higher-order thinking in particular (Chapter 1). Then I define and describe aspects of higher-order thinking emphasized in classroom learning and give examples of how to assess each aspect (Chapters 2 through 6).

The focus of the book is on assessment of higher-order thinking. I describe how to design assessments that require students to do higher-order thinking in an explicit enough form that the thinking becomes visible for appraisal, feedback, and discussion with the student. I describe both how to write and how to score questions and assessment tasks. The scoring is part of the assessment: if a question requires higher-order thinking but the scoring scheme only gives points for correct recall of facts, the assessment fails as a measure of higher-order thinking.

Of course, assessment of higher-order thinking assumes teaching of higher-order thinking. Although teaching these skills is not the subject of this book, it is worth noting that working through tasks like those in this book, with lots of feedback, could be part of such instruction. The ultimate goal is for students to learn to do more higher-order thinking, and do it better.

For ease of illustration, I use the following categories of higher-order thinking in the chapters illustrating ways to assess various aspects of such thinking:

• Analysis, evaluation, and creation (the "top end" of Bloom's taxonomy).
• Logical reasoning.
• Judgment and critical thinking.
• Problem solving.
• Creativity and creative thinking.

Chapters 2 through 6 describe in more detail the specific category, give guidelines for how to assess it, and provide some examples. These categories are consistent with the discussions of higher-order thinking as transfer, reasoned judgment, and problem solving. They also make a useful framework for talking about assessment (and instruction, too, for that matter), because slightly different strategies are used to assess each one. And as I have already said, there is overlap.

Throughout the book I have included many examples of assessments of higher-order thinking. The examples come from several sources. Some are particular examples that teachers have given me permission to share. Others are examples I have written myself that are based on many real examples but are not exact reproductions of any one of them. I have also used examples from NAEP because this is a good public source of well-written assessment items. The focus here is on individual items and tasks, not NAEP results or any of their uses in your state. This book is about classroom assessment of higher-order thinking.

Some readers may be surprised to see that some of the examples are multiple-choice test items. We often think of essays and performance assessments when we think of assessing higher-order thinking. But well-written multiple-choice items, especially those with introductory material, can also assess higher-order thinking. You wouldn't rely on multiple-choice items alone for such assessment, but it is important to be able to include on multiple-choice tests questions that tap thinking as well as recall. For example, in districts where banks of multiple-choice test items are used for benchmarking, if such items are not in the bank, then student thinking will not be part of the benchmark information. This book shows how to write both test items and performance assessments that tap higher-order thinking.

I have chosen each example to illustrate assessment of the particular aspect of higher-order thinking discussed in the various sections of the book. Because this is a book for K–12 teachers in all subjects, I have tried to select examples from a variety of subjects and grade levels. I encourage readers not to think, "This assessment is a good example," but rather, "What kind of thinking is this assessment a good example of?"