We see some of this conceptual uncertainty in the Taxonomy of Educational Objectives: Cognitive Domain. The book was written in 1956 by Benjamin Bloom and his colleagues to classify and clarify the range of possible intellectual objectives, from the cognitively easy to the difficult; it was meant to classify degrees of understanding, in effect. As the authors often note, the writing of the book was driven by persistent problems in testing: Just how should educational objectives or teacher goals be measured in light of the fact that there was (and is) no clear meaning of, or agreement about the meaning of, objectives such as “critical grasp of” and “thorough knowledge of”—phrases that have to be used by test developers?
In the introduction to the Taxonomy, Bloom (1956) and his colleagues refer to understanding as a commonly sought but ill-defined objective:
For example, some teachers believe their students should “really understand,” others desire their students to “internalize knowledge,” still others want their students to “grasp the core or essence.” Do they all mean the same thing? Specifically, what does a student do who “really understands” which he does not do when he does not understand? Through reference to the Taxonomy . . . teachers should be able to define such nebulous terms. (p. 1)
Recall that when our health teacher, Bob James, was thinking about his nutrition unit (see Chapter 1), he seemed unsure about what an understanding was and how it differed from knowledge. In fact, two generations of curriculum writers have been warned to avoid the term understand in their frameworks as a result of the cautions in the Taxonomy. For example, in the Benchmarks for Science Literacy from the American Association for the Advancement of Science (AAAS), the authors succinctly describe the problem they faced in framing benchmarks for science teaching and assessing:
Benchmarks uses “know” and “know how” to lead into each set of benchmarks. The alternative would have been to use a finely graded series of verbs, including “recognize, be familiar with, appreciate, grasp, know, comprehend, understand,” and others, each implying a somewhat greater degree of sophistication and completeness than the one before. The problem with the graded series is that different readers have different opinions of what the proper order is. (1993, p. 312)
Yet the idea of understanding is surely distinct from the idea of knowing something. We frequently say things like, “Well, he knows a lot of math, but he doesn't really understand its basis,” or, “She knows the meaning of the words but doesn't understand the sentence.” A further indication is that, 50 years after Bloom, many state standards now specify understandings separate from knowledge. Consider these examples from the California standards in science, which make the distinction explicit, with knowledge subsumed under the broader understanding:
Newton's laws predict the motion of most objects. As a basis for understanding this concept:
- Students know how to solve problems that involve constant speed and average speed.
- Students know that when forces are balanced, no acceleration occurs; thus an object continues to move at a constant speed or stays at rest (Newton's first law).
- Students know how to apply the law F = ma to solve one-dimensional motion problems that involve constant forces (Newton's second law).
- Students know that when one object exerts a force on a second object, the second object always exerts a force of equal magnitude and in the opposite direction (Newton's third law). . . .
Scientific progress is made by asking meaningful questions and conducting careful investigations. As a basis for understanding this concept and addressing the content in the other four strands, students should develop their own questions and perform investigations. Students will:
- Select and use appropriate tools and technology (such as computer-linked probes, spreadsheets, and graphing calculators) to perform tests, collect data, analyze relationships, and display data.
- Identify and communicate sources of unavoidable experimental error.
- Identify possible reasons for inconsistent results, such as sources of error or uncontrolled conditions. . . .
Although we might quibble as to whether the statement “Scientific progress is made by asking meaningful questions and conducting careful investigations” is a concept, the implication of the standard is clear enough: An understanding is a mental construct, an abstraction made by the human mind to make sense of many distinct pieces of knowledge. The standard further suggests that if students understand, then they can provide evidence of that understanding by showing that they know and can do certain specific things.
Understanding as meaningful inferences
But how are understanding and knowledge related? The standard still leaves the relationship murky in the phrase “As a basis for understanding this concept . . .” Is understanding simply a more complex form of knowledge, or is it something separate from but related to content knowledge?
Making matters worse is our tendency to use the terms know, know how, and understand interchangeably in everyday speech. Many of us would say that we “know” that Newton's Laws predict the motion of objects. And we may say we “know how” to fix our car and “understand” how to fix our car as if the two statements expressed the same idea. Our usage has a developmental aspect, too: What we once struggled to “understand” we say we now “know.” The implication is that something that once required a chain of reasoning to grasp hold of no longer does: We just “see it.”
Mindful of our tendency to use the words understand and know interchangeably, what worthy conceptual distinctions should we safeguard in talking about the difference between knowledge and understanding? Figure 2.1 presents some useful distinctions between the terms.
Figure 2.1. Knowledge Versus Understanding
- The facts
- A body of coherent facts
- Verifiable claims
- Right or wrong
- I know something to be true
- I respond on cue with what I know
- The meaning of the facts
- The “theory” that provides coherence and meaning to those facts
- Fallible, in-process theories
- A matter of degree or sophistication
- I understand why it is, what makes it knowledge
- I judge when to and when not to use what I know
John Dewey (1933) summarized the idea most clearly in How We Think. Understanding is the result of facts acquiring meaning for the learner:
To grasp the meaning of a thing, an event, or a situation is to see it in its relations to other things: to see how it operates or functions, what consequences follow from it, what causes it, what uses it can be put to. In contrast, what we have called the brute thing, the thing without meaning to us, is something whose relations are not grasped. . . . The relation of means-consequence is the center and heart of all understanding. (pp. 137, 146)
Consider an analogy to highlight these similarities and differences: tiling a floor with only black and white tiles. All our factual knowledge is found in the tiles. Each tile has definite traits that can be identified with relative precision and without much argument. Each tile is a fact. An understanding is a pattern visible across many tiles. There are many different patterns, some of them encompassing many or few tiles. Aha! Suddenly we see that small patterns can be grouped into sets of larger patterns—that was not apparent to us at first. And you may see the patterns differently than we do, so we argue about which is the “best” way to describe what we see. The pattern is not really “there” in an important sense, then. We infer it; we project it onto the tiles. The person laying the tiles merely positioned a black one next to a white one; he need not have had any pattern in mind: We may be the first to have seen it.
Let's move the analogy closer to intellectual life. The words on the page are the “facts” of a story. We can look up each word in the dictionary and say we know it. But the meaning of the story remains open for discussion and argument. The “facts” of any story are the agreed-upon details; the understanding of the story is what we mean by the phrase “reading between the lines.” (The author may not have “meant” what we can insightfully “infer”—just as in the tiling example; this is one of the debates in modern literary criticism—which view, if any, is privileged.) A well-known example from literacy studies makes the point elegantly:
First you arrange things into groups. Of course one pile may be enough, depending on how much there is to do; but some things definitely need to be separated from the others. A mistake here can be expensive; it is better to do too few things at once than too many. The procedure does not take long; when it is finished, you arrange the things into different groups again, so that they can be put away where they belong. (Bransford & Johnson, 1972, in Chapman, 1993, p. 6)
As a writer referring to this passage notes in a book on critical reading skills,
There is a point which varies depending on the individual reader, at which readers who monitor their own understanding realize that they are not “getting it” even though they know the meanings of all the words, the individual sentences make sense, and there is a coherent sequence of events. . . . At that point, critical readers who want to understand typically slow down, sharpen their attention, and try different reading strategies. (Chapman, 1993, p. 7)
The first passage is a vague account of doing laundry. More generally, the goal in understanding is to take whatever you are given to produce or find something of significance—to use what we have in memory but to go beyond the facts and approaches to use them mindfully. By contrast, when we want students to “know” the key events of medieval history, to be effective touch typists, or to be competent players of specific musical pieces, the focus is on a set of facts, skills, and procedures that must be “learned by heart”—a revealing phrase!
Understanding thus involves meeting a challenge for thought. We encounter a mental problem, an experience with puzzling or no meaning. We use judgment to draw upon our repertoire of skill and knowledge to solve it. As Bloom (1956) put it, understanding is the ability to marshal skills and facts wisely and appropriately, through effective application, analysis, synthesis, and evaluation. Doing something correctly, therefore, is not, by itself, evidence of understanding. It might have been an accident or done by rote. To understand is to have done it in the right way, often reflected in being able to explain why a particular skill, approach, or body of knowledge is or is not appropriate in a particular situation.
Understanding as transferability
It would be impossible to over-estimate the educational importance of arriving at conceptions: that is, meanings that are general because applicable in a great variety of different instances in spite of their difference. . . . They are known points of reference by which we get our bearings when we are plunged into the strange and unknown. . . . Without this conceptualizing, nothing is gained that can be carried over to the better understanding of new experiences.
—John Dewey, How We Think, 1933, p. 153
Baking without an understanding of the ingredients and how they work is like baking blindfold[ed] . . . sometimes everything works. But when it doesn't you have to guess at how to change it. . . . It is this understanding which enables me to both creative and successful.
—Rose Levy Berenbaum, The Cake Bible, 1988, p. 469
To know which fact to use when requires more than another fact. It requires understanding—insight into essentials, purpose, audience, strategy, and tactics. Drill and direct instruction can develop discrete skills and facts into automaticity (knowing “by heart”), but they cannot make us truly able.
Understanding is about transfer, in other words. To be truly able requires the ability to transfer what we have learned to new and sometimes confusing settings. The ability to transfer our knowledge and skill effectively involves the capacity to take what we know and use it creatively, flexibly, fluently, in different settings or problems, on our own. Transferability is not mere plugging in of previously learned knowledge and skill. In Bruner's famous phrase, understanding is about “going beyond the information given”; we can create new knowledge and arrive at further understandings if we have learned with understanding some key ideas and strategies.
What is transfer, and why does it matter? We are expected to take what we learned in one lesson and be able to apply it to other, related but different situations. Developing the ability to transfer one's learning is key to a good education (see Bransford, Brown, & Cocking, 2000, pp. 51ff). It is an essential ability because teachers can only help students learn a relatively small number of ideas, examples, facts, and skills in the entire field of study; so we need to help them transfer their inherently limited learning to many other settings, issues, and problems.
Consider a simple example from sports. When we grasp the idea that on defense we need to close up available space for the offense, we can use that understanding to adapt to almost any move members of the other team make, not just be limited to the one or two positionings we were taught in a three-on-three drill. We can handle entire classes of offensive problems, not just familiar instances. Failure to grasp and apply this idea in context is costly:
“When I got the ball in midfield and I started dribbling,” said Lavrinenko, the [NCAA men's soccer] championship tournament's outstanding offensive player, “I was looking to pass right away. But my teammates opened up space, and I continued running. When I played the ball to Alexei, 2 players went to him and opened up more space for me.” (New York Times, December 13, 1999, sec. D, p. 2)
And because the big idea of “constraining offensive space” transfers across sports, it is equally applicable in soccer, basketball, hockey, water polo, football, and lacrosse. The same is true in math or reading: To get beyond mere rote learning and recall, we have to be taught and be assessed on an ability to see patterns, so that we come to see many “new” problems we encounter as variants of problems and techniques we are familiar with. That requires an education in how to problem solve using big ideas and transferable strategies, not merely how to plug in specific facts or formulas.
Big ideas are essential because they provide the basis for the transfer. You must learn that a single strategy underlies all possible combinations of specific moves and settings, for example. The strategy is to get someone on your team open, using various moves and fakes—regardless of what the other team does or whether it looks exactly like what you did in practice. In academics, you must learn to transfer intellectual knowledge and skill:
Transfer is affected by the degree to which people learn with understanding rather than merely memorize sets of facts or follow a fixed set of procedures. . . . Attempts to cover too many topics too quickly may hinder learning and subsequent transfer. (Bransford, Brown, & Cocking, 2000, pp. 55, 58)
This is an old idea, famously framed by Whitehead (1929) almost 100 years ago in his complaint about “inert ideas” in education:
In training a child to activity of thought, above all things we must beware of what I will call “inert ideas”—that is to say, ideas that are merely received into the mind without being utilized or tested, or thrown into fresh combinations. . . . Education with inert ideas is not only useless: it is above all things, harmful. . . . Let the main ideas which are introduced be few and important, and let them be thrown into every combination possible. (pp. 1–2)
In reading, we may not have previously read this book by this author, but if we understand “reading” and “romantic poetry,” we transfer our prior knowledge and skill without much difficulty. If we learned to read by repeated drill and memorization only, and by thinking of reading as only decoding, making sense of a new book can be a monumental challenge. The same is true for advanced readers at the college level, by the way. If we learned to “read” a philosophy text by a literal reading, supplemented by what the professor said about it, and if we have not learned to actively ask and answer questions of meaning as we read, reading the next book will be no easier. (For more on this topic, see Adler and Van Doren, 1940.)
Transfer is the essence of what Bloom and his colleagues meant by application. The challenge is not to “plug in” what was learned, from memory, but modify, adjust, and adapt an (inherently general) idea to the particulars of a situation:
Students should not be able to solve the new problems and situations merely by remembering the solution to or the precise method of solving a similar problem in class. It is not a new problem or situation if it is exactly like the others solved in class except that new quantities or symbols are used. . . . It is a new problem or situation if the student has not been given instruction or help on a given problem and must do some of the following. . . . 1. The statement of the problem must be modified in some way before it can be attacked. . . . 2. The statement of the problem must be put in the form of some model before the student can bring the generalizations previously learned to bear on it. . . . 3. The statement of the problem requires the student to search through memory for relevant generalizations. (Bloom, Madaus, & Hastings, 1981, p. 233)
Knowledge and skill, then, are necessary elements of understanding, but not sufficient in themselves. Understanding requires more: the ability to thoughtfully and actively “do” the work with discernment, as well as the ability to self-assess, justify, and critique such “doings.” Transfer involves figuring out which knowledge and skill matters here and often adapting what we know to address the challenge at hand.
Here's an amusing transfer task to illustrate the point one more time. See if you can use your knowledge of French pronunciation and English rhymes to “translate” the following song. Say it out loud, at a normal speaking speed:
Oh, Anne, doux
But. Cueilles ma chou.Trille fort,
Chatte dort.Faveux Sikhs,
Pie coupe Styx.Sève nette,
Les dèmes se traitent.N'a ne d'haine,
Écoute, fée daine.*
* This is a fake song in which French words are used to make up a familiar rhyme about numbers—in accented English. Hint: the book is called Mots d'Heures: Gousses, Rames. (Mots d'Heures: Gousses Rames, by Luis d'Antin Van Rooten [Penguin Books, 1980; first published by Grossman Publishers, 1967]).
All of the cases we've discussed here illustrate the importance of confronting students with a real problem for thought if understanding is to be called for and awakened. This is very different from giving students lessons and tests that merely require taking in and recalling from memory, based on highly cued exercises in which learners simply plug in what is unambiguously required. (See Chapters 6 through 8 for further discussions on crafting understandings and meaningful assessments.)
The failure of even our best students to transfer their learning is evident in many areas but is most striking in mathematics. Consider the following examples of test items, all of which are testing the same idea (in each case, approximately two-thirds of the tested students did not correctly answer the question):
From the New York State Regents Exam:
To get from his high school to his home, Jamal travels 5.0 miles east and then 4.0 miles north. When Sheila goes to her home from the same high school, she travels 8.0 miles east and 2.0 miles south. What is the measure of the shortest distance, to the nearest tenth of a mile, between Jamal's home and Sheila's home? (The use of the accompanying grid is optional.)
From the NAEP 12th grade mathematics test:
What is the distance between the points (2,10) and (-4, 2) in the xy plane?
From a Boston Globe article on the Massachusetts MCAS 10th grade math scores:
The hardest question on the math section, which just 33 percent got right, asked students to calculate the distance between two points. It was a cinch—if students knew that they could plot the points and use the Pythagorean theorem, a well-known formula to calculate the hypotenuse of a right triangle if the lengths of two legs are given. The sixth-hardest math question, which only 41 percent of students got right, also required use of the Pythagorean theorem. “It seems applying the Pythagorean theorem was a weakness for kids,” said William Kendall, director of math for the Braintree public schools. “These weren't straightforward Pythagorean theorem questions. They had to do a little bit more.” (Vaishnav, 2003)
All three problems require students to transfer their understanding of the Pythagorean theorem to a new situation. It is likely that most students in the United States could not do it, despite the fact that every set of state standards identifies a grasp of the Pythagorean theorem as a key desired result.
We can apply our understanding to this news without too much difficulty, based on what has been said thus far. We surmise that the A2 + B2 = C2 theorem is taught as a fact, a rule for making certain calculations when confronted with a known right triangle and simple tasks. Remove a few blatant cues, however, and students cannot transfer their learning to perform with understanding. Is it any wonder, then, that students do not understand what they supposedly know? And what few educators seem to realize, therefore, is that drilling students for state tests is a failing strategy.
Understanding as a noun
Note again that the word understand(ing) has a verb meaning and a noun meaning. To understand a topic or subject is to be able to use (or “apply,” in Bloom's sense) knowledge and skill wisely and effectively. An understanding is the successful result of trying to understand—the resultant grasp of an unobvious idea, an inference that makes meaning of many discrete (and perhaps seemingly insignificant) elements of knowledge.
A genuine understanding involves another kind of transfer. We go beyond what we see, using big ideas, to make meaning of it, as Dewey noted in the quotation from How We Think cited earlier. “Oh, that's just like what we saw when the pioneers headed west!” a student excitedly realizes, when considering 20th-century immigration. That's the kind of transfer we seek! The challenge is to make it more likely by design rather than by luck or by natural disposition. With deliberate and explicit instruction in how to transfer (and assessments that constantly demand such transfer), the learner must take what were initially bits of knowledge with no clear structure or power and come to see them as part of a larger, more meaningful, and more useful system. Without lessons designed to bring ideas to life, concepts such as honor, manifest destiny, or the water cycle remain empty phrases to be memorized, depriving learners of the realization that ideas have power.
Here is a link, then, between the discussion in Chapter 1 on priorities in design and the specific goal of student understanding. Designing around big ideas makes learning more effective and efficient. As the authors of How People Learn note,
Teaching specific topics or skills without making clear their context in the broader fundamental structure of a field of knowledge is uneconomical. . . . An understanding of fundamental principles and ideas appears to be the main road to adequate transfer of training. To understand something as a specific instance of a more general case—which is what understanding a more fundamental structure means—is to have learned not only a specific thing but also a model for understanding other things like it that one may encounter. (Bransford, Brown, & Cocking, 2000, pp. 25, 31)
Transfer must be the aim of all teaching in school—it is not an option—because when we teach, we can address only a relatively small sample of the entire subject matter. All teachers have said to themselves after a lesson, “Oh, if we only had more time! This is just a drop in the bucket.” We can never have enough time. Transfer is our great and difficult mission because we need to put students in a position to learn far more, on their own, than they can ever learn from us.
Paradoxically, transfer heads in the opposite direction from “new” knowledge. An education for understanding asks us to more closely examine prior knowledge and the assumptions by which we claim something to be knowledge. Socrates is the model here. He questioned knowledge claims in order to understand and learn far more. When we are helped to ask certain questions—Why is that so? Why do we think that? What justifies such a view? What's the evidence? What's the argument? What is being assumed?—we learn a different kind of powerful transfer: the ability to grasp what makes knowledge knowledge rather than mere belief, hence putting us in a far better position to increase our knowledge and understanding.
The Expert Blind Spot
Teaching specific topics or skills without making clear their context in the broader fundamental structure of a field of knowledge is uneconomical.
—Jerome Bruner, The Process of Education, 1960, p. 31
Understanding the importance of transfer can help us make sense, then, of those educators, like Bruner, who claim that typical coverage is “uneconomical.” How can he say this? It seems so manifestly false: Teaching for understanding is perhaps more effective, but how can it possibly be more efficient? Can't we address far more content through didactic teaching and textbook coverage than we can by setting up inquiry-based work to help students come to deeper understanding of the material on their own?
But this confuses the teaching with the learning. Consider Bruner's three reasons for why a traditional coverage approach is uneconomical in the long run:
Such teaching makes it exceedingly difficult for the student to generalize from what he has learned to what he will encounter later. In the second place, [such] learning . . . has little reward in terms of intellectual excitement. . . . Third, knowledge one has acquired without sufficient structure to tie it together is knowledge that is likely to be forgotten. An unconnected set of facts has a pitiably short half-life in memory. (Bruner, 1960, p. 31)
In other words, we as educators fail to understand understanding when we think that coverage works. What we call the Expert Blind Spot is hard at work, causing us to confuse what we (or textbook authors) talk about with the active meaning-making required by the learner to grasp and use meaning. This habitual response by so many of us amounts to saying, “If I cover it clearly, they will ‘get it’ and be able to call upon it in the future. The more I cover, therefore, the more they will learn, and the better they'll do on the tests.”
What we hope you see by the book's end, however, is that this widely held assumption is false; the “yield” from coverage is quite low for most students:
More than 30 years ago, medical educators conducted a study on what first-year medical students remembered of the thousands of new terms that they'd memorized in their first-year gross anatomy course. They were tested and retested over time. The curve that matched most closely to their forgetting of gross anatomy was the same shape as discovered in Ebbinghaus's classic study of memory for nonsense syllables a century ago.The publication of data like these made a mark in the world of medical education. The teaching of anatomy has since changed radically in schools of medicine. (Shulman, 1999, p. 13 [emphasis added])
To cover everything is like quickly talking through a connect-the-dots puzzle in which the teacher further confuses students into thinking that understandings are merely more dots to be added to the page, thereby causing the picture to be even less clear and more confusing than it might be. Coverage leaves students with no sense of the whole that seems so obvious to the expert—all but the few most able students will get lost, and perhaps alienated.
Teachers do not optimize performance, even on external tests, by covering everything superficially. Students end up forgetting or misunderstanding far more than is necessary, so that reteaching is needed throughout the school experience. (How often have you said to your students, “My goodness, didn't they teach you that in grade X?”) So we end up with what we see in so many schools (as verified by NAEP test results): Students in general can do low-level tasks but are universally weak in higher-order work that requires transfer.
The research on learning (considered in greater detail in Chapter 13) merely supports the sobering truth of common sense: If learning is to endure in a flexible, adaptable way for future use, coverage cannot work. It leaves us with only easily confused or easily forgotten facts, definitions, and formulas to plug into rigid questions that look just like the ones covered. Furthermore, we have thereby made it far more difficult for students to learn the “same” things in more sophisticated and fluent ways later. They will be completely puzzled by and often resistant to the need to rethink earlier knowledge. In short, as Lee Shulman, president of the Carnegie Center for the Advancement of Teaching, put it so well, conventional teaching abets the three “pathologies of mislearning: we forget, we don't understand that we misunderstand, and we are unable to use what we learned. I have dubbed these conditions amnesia, fantasia, and inertia” (Shulman, 1999, p. 12).
Our analysis thus far suggests, then, the need for three types of “uncoverage” in designing and teaching for understanding to avoid forgetfulness, misconception, and lack of transfer:
- Uncovering students' potential misunderstandings (through focused questions, feedback, diagnostic assessment)
- Uncovering the questions, issues, assumptions, and gray areas lurking underneath the black and white of surface accounts
- Uncovering the core ideas at the heart of understanding a subject, ideas that are not obvious—and perhaps are counterintuitive or baffling—to the novice
The evidence of understanding
What differentiates revolutionary thinkers from non-revolutionary ones is almost never a greater knowledge of the facts. Darwin knew far less about the various species he collected on the Beagle voyage than did experts back in England who classified these organisms for him. Yet expert after expert missed the revolutionary significance of what Darwin had collected. Darwin, who knew less, somehow understood more.
—Frank J. Sulloway, Born to Rebel, 1996, p. 20
If understanding is about making meaning of facts and transferring knowledge to other problems, tasks, and domains, what does such understanding (or lack of it) look like? What should we be seeing if our students are getting better at understanding what they are learning? To pose this question is to shift from talking about our aims to talking about the evidence of whether our aims have been met.
The Sulloway comment about Darwin suggests one line of inquiry. Consider the words we use in describing understanding at the highest levels of research. We often describe understanding as “deep” or “in depth” as opposed to superficial knowledge. You have to “dig” below the “surface” (i.e., the “cover”) to “uncover” unobvious “core” insights. Understanding “takes time and practice.” Understandings are “hard won,” not immediate—maybe even overlooked or unseen by those with lots of knowledge, as Sulloway suggests. The emphasis in all these connotations is on getting below the surface, to the hidden gems of insight. We cannot cover concepts and expect them thereby to be understood; we have to uncover their value—the fact that concepts are the results of inquiry and argument.
Notice, then, the difference in the two questions at the heart of grappling with goals related to understanding (and all educational goals more generally) via backward design—the questions for the first two of the three stages:
Stage 1: What should students come away understanding?
Stage 2: What will count as evidence of that understanding?
The first question concerns important ideas about content and what should be learned. It asks the designer to be specific about what the student should take away, given the ideas, facts, and skills encountered. (Specifying the understandings we seek is surprisingly difficult, as we discuss in Chapter 6.) The second question is different. It doesn't speak to what should be learned; it concerns acceptable embodiment of those goals: What constitutes appropriate performance and products—output—from students of that learning, determined through assessment.
The second question actually encompasses distinct questions that make up the second stage of backward design:
- Where should we look for evidence? What is the type of student work we need to see done well, given the stated standard?
- What should we look for specifically in student performance, regardless of the particular approach, for us to judge the degree to which the student understands?
Loosely speaking, the first question about the evidence involves a design standard for assessment of the work (i.e., what are valid tasks, tests, observations?), and the second question about the evidence concerns the actual evaluation of the work produced, via rubrics or other criteria-related guidelines.
The argument for backward design is predicated on the view that we are not likely to achieve our target of understanding—however we define the term—unless we are clear about what counts as evidence of that understanding. And the more we ask that nitty-gritty assessment question, the more many teachers come to understand that they may not have adequately understood understanding.
A standard is different from a performance indicator. A standard represents a goal and belongs in Stage 1. A performance indicator, such as those found often in bulleted lists under state content standards, represents possible assessment evidence. Making matters more confusing, sometimes the standards also refer to learning activities like those we would put in Stage 3. (See standard in the Glossary.)
Why might we be unsure about what constitutes good evidence of understanding? Because the evidence we tend to focus on or that stands out more readily can easily mislead us if we are not careful. When students provide the answer we seek, it is easy to conflate such recall with understanding. Bloom and his colleagues (1956) remind us of the distinction when they recount a famous story about John Dewey:
Almost everyone has had the experience of being unable to answer a question involving recall when the question is stated in one form, and then having little difficulty . . . when the question is stated in another form. This is well illustrated by John Dewey's story in which he asked a class, “What would you find if you dug a hole in the earth?” Getting no response, he repeated the question; again he obtained nothing but silence. The teacher chided Dr. Dewey, “You're asking the wrong question.” Turning to the class, she asked, “What is the state of the center of the earth?” The class replied in unison, “Igneous fusion.” (p. 29)
The story beautifully illustrates the need to distinguish the content goal from the evidence, as well as the need to stress transferability in the requirements for evidence. Children cannot be said to understand their own answer, even though it is correct, if they can only answer a question phrased just so. Furthermore, they will not be able to use what they “know” on any test or challenge that frames the same question differently, as apparently happened in the state tests mentioned earlier.
Getting evidence of understanding means crafting assessments to evoke transferability: finding out if students can take their learning and use it wisely, flexibly, creatively. The authors of the Taxonomy note, for example, that “real” knowledge involves using learning in new ways. They call this “intellectual ability” and distinguish it from “knowledge” based on recall and scripted use. Similarly, David Perkins in the book Teaching for Understanding defines understanding as “the ability to think and act flexibly with what one knows . . . a flexible performance capability,” as opposed to rote recall or “plugging in” of answers (Wiske, 1998, p. 40). A person who has understanding can cope far better than others with ambiguous—that is, real-world—challenges in which what is required does not come packaged as a straightforward cue to stimulate a single response. (Recall the vignette in the Introduction about the class valedictorian who admitted a lack of understanding despite high marks on tests of recall.)
Evidence of understanding that is transferable involves assessing for students' capacity to use their knowledge thoughtfully and to apply it effectively in diverse settings—that is, to do the subject. As the authors of How People Learn (Bransford, Brown, & Cocking, 2000) write,
Students' abilities to transfer what they have learned to new situations provides an important index of adaptive, flexible learning. . . . Many approaches to instruction look equivalent when the only measure of learning is memory. . . . Instructional differences become more apparent when evaluated from the perspective of how well the learning transfers to new problems and settings. (p. 235)
Students develop flexible understanding of when, where, why, and how to use their knowledge to solve new problems if they learn how to extract underlying principles and themes from their learning exercises. (p. 224 [emphasis added])
The point is nothing new. Bloom and his colleagues (1956) made the same point about “application” in the Taxonomy 50 years ago. An assessment of application had to involve a novel task, requiring transfer; and it ideally involved contextualized and practical use of ideas:
If the situations . . . are to involve application as we are defining it here, then they must either be situations new to the student or situations containing new elements. . . . Ideally we are seeking a problem which will test the extent to which an individual has learned to apply the abstraction in a practical way. (p. 125)
Evidence of understanding requires that we test quite differently, then. We need to see evidence of students' ability to “extract” understandings and apply them in situated problems, in performance—something quite different from merely seeing if they can recall and plug in the underlying principles the teacher or textbook gave them.
This requires us to anchor our assessments in prototypical performances in each area, success at which indicates understanding; for example, the ability to design a science experiment, debug it, and revise it in order to determine the chemical content of a substance; the ability to use the facts and skills learned in history to write a credible narrative about a period in local history. (We refer to these two examples as two of many “core tasks” in a field of study, and we propose that curriculum frameworks and programs be designed around such core tasks, along with the big ideas. For a more detailed discussion of core tasks, see Chapters 7 and 12.) We need to see if students with understandably limited ability can nonetheless transfer—that is, recognize what in their repertoire might be useful here, in this novel situation, and use it effectively. Thus, we would use far fewer narrow prompts that are intended to elicit the “correct” answer to a familiar question.
The “igneous fusion” example is extreme, but the problem strikes home more than most of us may see or care to admit. We are often too ready to attribute understanding when we see correct and intelligent-sounding answers on our own tests. What may trip us up more than we realize is apparent understanding, in other words. And that difficulty is likely exacerbated in a world of high-stakes testing and grading. For as long as education promotes a cat-and-mouse game whereby students have incentive to both please us and appear to understand what they are supposed to learn (irrespective of whether they do or not), the challenge of assessing for real understanding becomes greater.
In short, we must be careful: It doesn't matter how we term the difference between knowing and understanding as long as we safeguard the real difference. What we call understanding is not a matter of mere semantics. It is a matter of conceptual clarity whereby we distinguish between a borrowed expert opinion and an internalized flexible idea. If our assessments are too superficial and fact-centered, we may miss the distinction in the evidence we collect. It does not matter in the end what we call understanding-related targets, but it matters greatly that we safeguard the distinction between “understand” and “know the right answer when prompted.” What matters is that we grasp the challenge of assessing for transfer.
We have to be sharper at specifying what kinds of student work and assessment evidence are required if we are to judge a student as really understanding. The authors of the AAAS Benchmarks for Science Literacy (1993) cited earlier say that they decided against specifying action verbs or observable behaviors to clarify what kinds of evidence were required to reveal understanding, because “the choice among them is arbitrary” and using particular verbs “would be limiting and might imply a unique performance that was not intended” (pp. 312–313).
Although we concede that there is no unique or inherently perfect assessment task for an understanding target, certain kinds of challenges are more appropriate than others. Knowing what kinds of assessments embody the standards is precisely what many teachers need. Recall that this is why Bloom's Taxonomy was written in the first place. Without specificity concerning what counts as appropriate evidence for meeting the standards, a teacher might well be satisfied by a factual test of knowledge, whereas only a complex piece of inquiry and defense of methods and result will truly do justice to the standard.
If “correct” answers may yield inadequate evidence of understanding, what should we do to make our assessments better distinguish between real and apparent understanding? Before we answer that question, we must deal with another problem first: Sometimes a correct answer hides misunderstanding. How is that possible? And what are the implications for assessment of understanding? The irony is that we can gain significant insight into designing, assessing, and teaching for understanding by considering the phenomenon of misunderstanding.
Student mis understanding and what we can learn from it
Somehow, well-intentioned, able, and attentive students can take away lessons that we never intended. What are we complaining about when we say of students, “They know all the facts, but they put them together all wrong” or, “They just aren't thinking about what they are saying”? The Catcher in the Rye is a fixture of high school English courses in the United States, for example, yet many students come away believing the book to be about Holden's “excellent adventure” (to borrow from a recent movie title), the larklike days in the life of a hooky-playing prep school student. Somehow, the fact that Holden is in great emotional pain—and tells the story from his psychiatric hospital bed—is unseen by many students. Similarly, in mathematics, many elementary students struggle mightily with the multiplication of fractions, given the oddity of the answers being smaller than the numbers they started with. Or consider the great challenge of reading: Simple decoding is not so simple. We pronounce “lose” as “loze” and the teacher tells us we are mistaken. But we thought we understood the rule! Why isn't the pronunciation of “lose” consistent with the long-vowel rule about words that end in a consonant and e (e.g., close, doze, home)?
Misunderstanding is not ignorance, therefore. It is the mapping of a working idea in a plausible but incorrect way in a new situation. Here are some examples:
- One of our children asked: “Dad, are Spanish and English using the same words, but just pronouncing them differently?”
- The same child complained a few years later, “How can 4.28 + 2.72 = 7? Seven isn't a decimal!”
- A high school history student asked her teacher quietly at the end of a unit, “So just what did Louisiana purchase?”
- An elementary teacher reported the irritation of one of her 4th grade students at not ever seeing lines of longitude and latitude as she flew cross-country with her family.
- A very bright and learned boy, with advanced placement science courses in his background, thought “error” in science was a function of avoidable mistakes, rather than a principle inherent in the enterprise of induction.
Paradoxically, you have to have knowledge and the ability to transfer in order to misunderstand things.
Thus evidence of misunderstanding is incredibly valuable to teachers, not a mere mistake to be corrected. It signifies an attempted and plausible but unsuccessful transfer. The challenge is to reward the try without reinforcing the mistake or dampening future transfer attempts. In fact, many teachers not only fail to see the value in the feedback of student misunderstanding, they are somewhat threatened or irritated by it. A teacher who loses patience with students who don't “get” the lesson is, ironically, failing to understand—the Expert Blind Spot again. For attentive students not to “get it” is to show us that what we thought was clear was really not so. For some teachers, perpetual student misunderstanding is therefore threatening, understandably, because it seems to call into question our methods and implied goals. What the naïve teacher may be overlooking, of course, is that the big ideas are rarely obvious. Indeed, they are often counterintuitive, as we noted in Chapter 1. A word to the wise, then: If you hear yourself saying to a class, “But it's so obvious!” you are most likely falling prey to the Expert Blind Spot! Take time to ponder: Hmmm, what is not obvious to the novices here? What am I taking for granted that is easily misunderstood? Why did they draw the conclusion they did?
Making the matter of greater urgency is the fact that research over the past 20 years confirms the surprising depth and breadth of the phenomenon. Many students, even the best and most advanced, can seem to understand their work (as revealed by tests and in-class discussion) only to later reveal significant misunderstanding of what they “learned” when follow-up questions to probe understanding are asked or application of learning is required. Indeed, it is not only our view but also the view of leading cognitive researchers that ferreting out student conceptions and misconceptions and being mindful of them when designing learning is key to better results. (A summary of the research on learning and teaching for understanding is presented in Chapter 13.) Howard Gardner, David Perkins, and their Harvard colleagues at Project Zero have summarized these findings eloquently and thoroughly in the past decade, though the misconception research goes back to work done in science education in the 1970s. As Gardner (1991) explains in summing up the research,
[What] an extensive research literature now documents is that an ordinary degree of understanding is routinely missing in many, perhaps most students. It is reasonable to expect a college student to be able to apply in new context a law of physics, or a proof in geometry, or the concept in history of which she has just demonstrated acceptable mastery in her class. If, when the circumstances of testing are slightly altered, the sought-after competence can no longer be documented, then understanding—in any reasonable sense of the term—has simply not been achieved. (p. 6)
Testing of even a conventional kind can provide evidence of such failures to understand if the tests are designed with misunderstanding in mind. In the Introduction we noted the NAEP math example in which a large minority of students answered “32, remainder 12” buses. Consider this result more generally. Most U.S. teenagers study Algebra I and get passing grades. Yet NAEP (1988) results show that only 5 percent of U.S. adolescents perform well at tasks requiring higher-order use of Algebra I knowledge. The Third International Mathematics and Science Study (TIMSS, 1998) reached a similar conclusion for science in one of the most exhaustive studies to date (Trenton Times, 1997). And so did NAEP's recent test, showing “a stark gap between the ability of students in general to learn basic principles, and their ability to apply knowledge or explain what they learned” (New York Times, 1997). (The test was a mixture of multiple-choice, constructed response, and performance-task questions.)
For more than a decade in physics, specific tests have been developed and used as assessments targeting key misconceptions. The most widely used test, the Force Concept Inventory, provides a pre- and post-test instrument for measuring progress in overcoming the most common (and surprisingly persistent) misconceptions.
AAAS, in its Benchmarks (1993) and Atlas of Science Literacy (2001), has provided a rich account of desired understandings in the sciences, coupled with key misunderstandings connected with them:
When a relationship is represented in symbols, numbers can be substituted for all but one of the symbols, and the possible value of the remaining symbol computed. Sometimes the relationship may be satisfied by one value, sometimes more than one, and sometimes not at all.
- Students have difficulty understanding how symbols are used in algebra. They are often unaware of the arbitrariness of the letters chosen. These difficulties persist even after instruction in algebra and into college. Students of all ages often do not view the equal sign of equations as a symbol of equivalence but rather interpret it as a sign to begin calculating—the right side should show the “answer.”
Comparison of data from two groups should involve comparing both their middles and the spreads around them.
The middle of a data distribution may be misleading—when the data are not distributed symmetrically, or when there are extreme high or low values, or when the distribution is not reasonably smooth.
- The concept of the mean is quite difficult for students of all ages to understand even after years of formal instruction. . . . Research suggests that a good notion of “representativeness” may be a prerequisite to grasping the definitions of mean, median and mode. . . . Premature introduction of the algorithm for computing the mean divorced from a meaningful context may block students from understanding what averages are. (AAAS, 2001, pp. 122–123)
To see how easy it is to misunderstand things we think we all know, consider this more basic science question: Why is it colder in winter and warmer in summer? Just about every student in the United States has been taught basic astronomy. We “know” that the Earth travels around the sun, that the orbit is elliptical, and that the Earth tilts at about 20 degrees off its north-south axis. But when graduating Harvard seniors were asked the question (as documented in a video on the misunderstanding phenomenon produced by the Harvard-Smithsonian Center for Astrophysics), few could correctly explain why (Schneps, 1994).1
They either had no adequate explanation for what they claimed to know or they provided a plausible but erroneous view (such as, the weather changes are due to the earth being closer or farther from the sun).
Similar findings occur when we ask adults to explain the phases of the moon: Many well-educated people describe the phases as lunar eclipses. In a follow-up video series on misconceptions in science entitled Minds of Their Own, the Harvard astrophysics group documented how a physics student who can do the same electric circuit problems we give to 4th graders, and describe what is occurring, has a flawed understanding when the question is cast in a novel way (can you light the bulb with only batteries and wires?).
The recognition of inevitable learner misunderstanding in even the best minds, in disciplines as seemingly straightforward and logical as science and mathematics, is actually quite old. Plato's dialogues vividly portray the interplay between the quest for understanding and the habits of mind and misconceptions that may be subconsciously shaping or inhibiting our thinking. Francis Bacon (1620/1960) provided a sobering account of the misunderstandings unwittingly introduced by our own intellectual tendencies operating unawares in the Organon 400 years ago. He noted that we project categories, assumptions, rules, priorities, attitudes, and matters of style onto our “reality” and then develop countless ways of “proving” our instinctive ideas to be true: “The human understanding . . . when it has once adopted an opinion draws all things else to support and agree with it” (pp. 45–49). Philosophers and psychologists from Kant and Wittgenstein to Piaget and other modern cognitive researchers have attempted to figure out the puzzle of persistent misunderstanding and the naïve conviction that typically accompanies it—and the selfassessment and self-discipline needed to move beyond both.
Practically speaking, we must begin to design assessments in recognition of the need for conceptual benchmarks, not just performance abilities. We need to design assessments mindful of not only the big ideas but also the likelihood that those ideas will be misconceived—and will resist being overcome, as in this biology example cited by Shulman (1999):
Biology teachers must wrestle with the durability of student misconceptions of evolution and natural selection. Most students in courses that emphasize evolution and natural selection enter these courses as intuitive Lamarckians. They are convinced that any characteristics acquired by one generation are then transmitted to the next generation. The formal instruction emphasizes the Darwinian refutation of that position. These students may earn A's and B's in the course, demonstrating that they now understand the Darwinian perspective, but quiz them three months later and they're once again dedicated intuitive Lamarckians—as indeed are many of the rest of us. I suspect that forms of fantasia are endemic among students and graduates of higher education, many lying in wait for years before manifesting themselves at critical moments. (p. 12)
Here are some examples of common misunderstandings for some important ideas, and understandings that reflect the overcoming of them:
- Impressionism is art in which the painter offers a subjective impression or feeling evoked by the scene. The opposite is the case: Impressionism was an attempt to paint scenes realistically, not abstractly or by feeling. Impressionism refers to a technical term in philosophy whereby direct sensory impressions are distinguished from the mind's placing of those impressions into ideas.
- Each month there is a lunar eclipse when the moon is not visible. The phases of the moon depend on the relative position of the earth, the sun, and the moon, so that we see the part of the moon that is lit by the sun. Ongoing lunar eclipses are not the cause of the phases.
- Science is about finding causes. Scientists find correlations; talk of “causes” is viewed as too philosophical and unscientific. Modern science, economics, and medicine search for statistical patterns. That's why asking “What caused it?” is not necessarily a question doctors can answer, even as they prescribe effective medicines.
- When you multiply two numbers, the answer is bigger. Multiplication is not repeated addition. Fractions when multiplied yield a smaller answer, and when divided, a larger answer. How can that be? Students often see fractions and decimals as separate number systems; learning to see them as alternate means of representing the “same” quantities is the understanding.
- History is about the facts, what happened. A historian is a storyteller, not a mere gatherer and purveyor of facts. Why, then, do so few students realize that there can be and are very different stories of the same important history?
- You should cup your hands when swimming in order to “catch the water” to move faster. The greater the surface area, the greater the force. Thus, you should swim with a flat palm to maximize the amount of water being pulled and pushed.
- Light is light and dark is dark. Not true. Two light beams intersecting at crest and trough can cancel each other out and cause darkness! Noise-canceling headphones use sound to produce silence. Similiarly, mirror-image waves of light or sound cancel each other out.
- Negative and imaginary numbers are unreal. Negative and imaginary numbers are no less and no more real than ordinary numbers. They exist to provide the symmetry and continuity needed for essential arithmetic and algebraic laws.
- Evolution is a controversial idea. No, the theory of natural selection as the engine of evolution is what is controversial. Theories of evolution predated Darwin by centuries and were not seen as being in conflict with religious doctrine.
- Our founders were liberals. The American revolutionaries held that individuals, not governments, had natural rights applied through labor (based on John Locke's views about property). Thus, in one sense, they were “conservatives” (i.e., the right to personal property is fundamental).
- Irony is coincidence. Irony is not mere coincidence, though almost every sportscaster misuses the word! Irony is what the wiser person sees that another seemingly wise person does not. The audience sees what Oedipus does not, and the tension between the latter's pride and what we know is the truth is the source of the drama's power.
Given the likelihood of deeply rooted misconceptions and the potential for misunderstanding, a proactive and, for most of us, unfamiliar approach to assessment design is required. To successfully engineer understanding, we have to think backward: What does understanding look like when it is there or not there? We have to be able to describe what it looks like, how it manifests itself, how apparent understanding (or misunderstanding) differs from genuine understanding, which misunderstandings are most likely to arise (thus interfering with our goal), and whether we are making headway in ferreting out and eradicating the key impediments to future understanding. In other words, we have to think through our assessments before we think through our teaching and learning.
Any design depends upon clear purposes, as we have said. Yet the matter is complicated by the mixture of many externally imposed goals (e.g., state content standards) and self-selected goals. How should we prioritize? How do we select wisely from so many obligations to ensure an effective and coherent design? How can we design coherent units while remaining constantly mindful of the many and overlapping course and program goals? We now turn to these questions.
Information on the Science Education Project is available from the Harvard-Smithsonian Center for Astrophysics, 60 Garden Street, Cambridge, MA 02138,