Conference Countdown
Dallas, Tex.
June 27-29, 2014
  • membership
  • my account
  • help

    We are here to help!

    1703 North Beauregard Street
    Alexandria, VA 22311-1714
    Tel: 1-800-933-ASCD (2723)
    Fax: 703-575-5400

    8:00 a.m. to 6:00 p.m. Eastern time, Monday through Friday

    Local to the D.C. area, 703-578-9600, press 2

    Toll-free from U.S. and Canada, 1-800-933-ASCD (2723), press 2

    All other countries (International Access Code) + 1-703-578-9600, press 2

  • Log In
  • Forgot Password?

 

Share
2014 ASCD Conference on Teaching Excellence

2014 ASCD Conference on Teaching Excellence

June 2729, 2014
Dallas, Tex.

Explore ways to make excellent teaching the reality in every classroom.

 

Permissions

ASCD respects intellectual property rights and adheres to the laws governing them. Learn more about our permissions policy and submit your request online.

Policies and Requests

Translations Rights

Books in Translation

Sale Book (Oct 2004)
Related Topics

Using Data to Assess Your Reading Program

by Emily Calhoun

Table of Contents

Chapter 1. Using Action Research for Reading Program Assessment

Action research can be conducted on almost any process, but for school improvement purposes, it should be directed primarily toward improving student learning. This is broad territory, involving not only student learning in all the academic domains but also the study of curriculum, instructional practices, assessment procedures, and the organization of the learning environment by teachers, administrators, and others engaged in supporting student learning. In this book, we will focus on reading, applying the action research process to improving student proficiency in reading and to examining the quality of our reading program.

Along with the No Child Left Behind Act (http://www.ed.gov/nclb), passed in 2001, the flood of new books on teaching reading, and the proliferation of new assessment packages, there are numerous articles and books reminding us that we need to improve student performance, increase teachers' knowledge of content and pedagogy, improve leadership skills, and increase the learning occurring at all levels of the organization (e.g., Bransford, Brown, & Cocking, 1999; Bryk, Sebring, Kerbow, Rollow, & Easton, 1998; Darling-Hammond & Ball, 1998; and Elmore, 2000). Those of us who work in and with schools can respond to the federal regulations, the reminders from our colleagues, and the goal of the No Child Left Behind Act by using our collective experiences and our knowledge base in reading to develop more effective curriculum and instruction for students and more successful professional development for ourselves.

Action research is one of the tools we can use to help us respond in an integrative and informed manner, instead of a reactive or merely compliant manner. The action research process provides a structure for continuously assessing the effects of actions and seeking more effective ones if needed. When used organization-wide by school and district staff, the process can increase individual professional expertise and build strong professional communities. Aimed directly at student learning and accompanied by training to expand instructional and curricular practices, this technique can also yield improved student achievement (Joyce et al., 1996; Joyce, Calhoun, & Hopkins, 1999; Joyce, Hrycauk, Calhoun, & Northern Lights Kindergarten Teachers, 2003).

The nice thing about using an action research approach is that a school team or faculty can use it to study students' progress in reading and the breadth, depth, and multidimensionality of their program, whatever materials are being used—be they commercially published (such as High/Scope, Scholastic, Open Court, Guided Reading) or a locally developed curriculum. For action research is simply a structured process for addressing a problem or answering questions with the intention of using the resulting information to make improvements. Additionally, the research process can be conducted individually by a single teacher in a classroom or collaboratively among several persons with similar interests. I am going to write as if you and I were engaging in schoolwide or organization-wide action research. This allows us to look at what is happening to all students in our setting and to consider more fully the curriculum and instructional experiences they encounter.

The Action Research Framework

Because the process for examining reading programs described in this book uses a schoolwide or districtwide action research framework, it's important to understand what that framework is, how it works, and when it's useful. We need to ask ourselves the following questions: Who are our students, and what programs are currently available in the school to support their development as readers? How are they responding to the learning environment of the school? How well are our students performing in literacy, especially in reading? Do we want to improve the current levels of performance? Are we satisfied with their rate of progress (gain)? And are there changes we can make in curriculum, instruction, assessment, or the organization of the learning environment that are likely to improve literacy learning in our setting?

To help us pursue the information we need to answer these questions, we have the Action Research Matrix (Figure 1.1), a graphic organizer that focuses simultaneously on student learning and the learning environment provided for students. Faculties can use the six cells, which represent lines of inquiry, as a framework for structuring collective inquiry into student learning. Some of you may be familiar with using this matrix to design plans for schoolwide action research (Calhoun, 1994, 2002).


Figure 1.1. Action Research Matrix What Do We Know? What Do We Do?


Date: _______________________________________

Schoolwide Focus:__________________________________________

(Academic student learning goal in a curriculum area)

Learner (Students)

Learning Environment (School)

On-Site Information (Information at the school level)

1. Current student information

3. Student performance and responses we would like to see

4. Information about the current learning environment in our school

6. Learning environment we would like to see

External Information (Study of literature, standards, & best practices)

2. External information about learners/students

5. External information about the learning environment

Source: Adapted from Calhoun, E. F. (1998). Action research matrix. St. Simons, GA: The Phoenix Alliance.


The matrix includes a place to identify the student learning goal that a faculty selects for its collective focus as well as six domains, or cells, of inquiry and action. The structure of the matrix is designed to help groups study and use on-site and external information about student learning and the learning environment. Cells 1, 2, and 3 focus on learners' current performance and what is possible in terms of student achievement. Cells 4, 5, and 6 focus on the formal learning environment, which in this case comprises the reading program elements: curriculum, instruction, assessment, and the organization of students and staff.

School and district staff are asked to use both on-site and external information as they set their benchmarks and desired levels of performance for students (Cell 3) and as they select actions to study and implement in their classrooms and schools (Cell 6). The sequence of the matrix is designed to help staff explore the research base and move beyond what is currently known or done in their school or setting. However, it is only a guide to domains of inquiry and action, not a rigid set of steps.

Let's visit Elwood Elementary, where the staff are learning to use action research to study the development of reading vocabulary. The actions of the Elwood staff help illustrate the six domains of inquiry in the Action Research Matrix (Figure 1.1).

The Research Matrix in Action

Elwood Elementary is a prekindergarten through grade 5 school in a small town near a large metropolitan area. Sixty percent of its 622 students participate in the free and reduced-price lunch program. The race/ethnicity composition of the student population is 43 percent white, 38 percent black, 16 percent Hispanic, and 3 percent other ethnicity (mostly from Asia and some of the former Russian republics).

Two years ago, a new principal, Helen, came to Elwood. She has put a lot of energy into studying student results from the past five years, meeting with parents and community members, inviting board members for regular visits to the school, and reading books or team teaching in classrooms almost every day. Helen is determined to improve student achievement, and she and the staff have decided to tackle reading first and use action research to structure their collective work. Susie, the curriculum director for the district, is a skilled action research facilitator, and Helen has asked Susie to work closely with them as they get started.

The year after she arrived, Helen formed a School Leadership Team (made up of seven teachers, the counselor, and the principal) that facilitates across classroom study and action, and she has asked teachers to form study groups. Each leadership team member serves as team leader of a study group, which is composed of four staff members who work and plan as peer partners for some tasks and as a group at other times. Helen believes these group structures provide social support, facilitate communication, distribute leadership, and make whole-school improvement possible.

The School Leadership Team has worked with the district office, the community, and the staff as they have developed and implemented steps toward improving literacy. They began using the High/Scope Active Learning Practices materials (Hohmann et al., 2002) in their prekindergarten program the previous year because of its promised effects and its use of reading and writing in building students' literacy and communication skills, and they also added the Preschool Child Observation Record (High/Scope, 2003) to help them monitor student progress. This year all the kindergarten and 1st grade teachers have been participating in a yearlong staff development course offered by their intermediate service agency. The course focuses on instructional techniques that help accelerate the development of oral and written language, including a lot of work with phonological awareness activities. First and 2nd grade teachers are seeing some improvements in students' use of language, and their impressions are confirmed by the data from informal reading inventories. Third through 5th grade teachers have been working with reciprocal teaching to improve reading comprehension; however, most of their students still struggle with any but the simplest text. The staff know they must do more. They still have far too many students reading far below grade level.

Their students' needs are so great that the staff could begin with any priority student learning goal or major curriculum standard and make progress. However, as they began their action research process, they decided that their school focus would be on “expanding reading vocabulary and word analysis skills” because they think bringing that about would make the most difference in accelerating literacy for the most students. Observe how the Elwood staff, with some support from Susie, uses the structure of the Action Research Matrix to guide their collective work.

Cell 1—Current Student Information

One of the first things Elwood's leadership team did was organize the existing and easily available student achievement data from the previous year's reading results on the norm-referenced tests administered statewide at the 3rd and 5th grades. Figure 1.2 shows these results.


Figure 1.2. Elwood Elementary Students' Achievement Test Scores: Reading


3rd Grade (87 Students Enrolled)

5th Grade (82 Students Enrolled)

Percentile Range

Number of Students

Percent

Number of Students

Percent

75th–99th

5

6%

10

12%

50th–74th

20

23%

17

21%

25th–49th

27

31%

23

28%

1st–24th

32

37%

24

29%

Not tested

3

3%

8

10%


As you can see, at 3rd grade, 37 percent of Elwood students were in the bottom quartile in reading overall, with 68 percent of the students scoring below the 50th percentile. The results for Elwood's 5th grade students were a little better, with 29 percent in the bottom quartile and 57 percent scoring below the 50th percentile. However, when these data were shared with the staff, one of the teachers noted that 10 percent of the 5th grade students did not take the tests, and if that number were eliminated in calculating percentages, then 64 percent of their students were below the 50th percentile. Another teacher said, “Well, our scores below the 50th percentile pretty much match our demographics. We do have 60 percent of our students on free and reduced lunch.” Both of these comments generated some heated discussions among staff members.

The leadership team also organized the data on the vocabulary and comprehension sections of the local curriculum assessments from the previous three years, and the staff could see how similar the results were from grade to grade, with approximately 60–65 percent of the students struggling below the “Proficiency” level. (These tests are administered in grades 2 through 5.) In reviewing these results, staff also noticed that after the first year, scores were somewhat higher on vocabulary than on reading comprehension. Questions were raised about whether this was related to the content of the tests, to the fact that the same tests had been used for three years, or to the possibility that students' understanding of vocabulary had exceeded their reading comprehension skills. There were many opinions voiced, but no answers to these questions. Susie and Lisa, the 3rd grade team leader, decided to evaluate the content of the tests, and Susie told them that she would look up the districtwide scores to see if there were similar patterns in the other elementary schools. Helen was pleased that the questions were raised in the consciousness of the staff, but she felt students were performing so poorly in both dimensions that small increases in vocabulary over two years didn't seem of great moment and thus were not worthy of much of their precious staff development time.

Several teachers brought up the issue of high student turnover at Elwood, so the principal and two leadership team members decided to use student ID numbers and organize results from the previous year's 5th graders who had been in the school for at least three years. When these data were shared with all staff members at their next meeting, they found that about two-thirds of the students tested on both state and local assessments had been with them for three or more years. Some of the 3rd and 4th grade teachers wanted to look at the same kind of data for the previous year's 3rd graders, but the leadership team wanted staff to start looking at some results other than their own. They had several reasons. They could feel the defensiveness of some colleagues, and they wanted to counter the frequent comments of, “Well, look at our children. What else can we expect from so many one-parent families and poor neighborhoods?”

Cell 2—External Information About Learners/Students

In an attempt to learn more about how vocabulary develops and to provide a base for high expectations, the leadership team sought resources on how vocabulary is acquired and on student achievement in other settings. They located and studied the research synthesis “Conditions of Vocabulary Acquisition” by Beck and McKeown (1991), “The Vocabulary Conundrum” by Anderson and Nagy (1992), and several pieces by Linnea Ehri on how students learn to read words (they especially liked the directness and clarity of the 1999 AERA paper). The curriculum director used the Web and state report card information to organize and share data from five elementary schools with similar demographics that had made significant improvements in vocabulary and comprehension for three years in a row.

The leadership team and their study groups also reviewed local curriculum documents. Although they found vocabulary mentioned frequently, there was not much guidance on what was most worth teaching and when. At the next leadership team meeting, members reported by grade level and special needs. As Susie listened to their comments about what was missing, she realized that she might be able to use some of their work to improve the district curriculum standards. She also realized that the district's curriculum documents needed to provide teachers with a better conceptual map of how reading, vocabulary, and word analysis skills develop.

Cell 3—Student Performance and Responses We Would Like To See

The leadership team led the staff through an exercise to help everyone reflect on the student results they had looked at earlier, on information about how students develop vocabulary, and on student achievement data from comparable schools. Then they led the staff in setting some targets for the next year.

The principal opened the session by encouraging the staff to set high expectations for students. She was concerned that they—including her—had fallen into a pattern of setting low targets so they would be more likely to attain them. By the time the discussions were over, the staff had decided that for the state's norm-referenced tests, they would aim for increasing the school mean by 10 percentile points and having at least 40 percent of their students at the 50th percentile or higher. They would also attempt to move at least 20 percent of their students out of the bottom quartile of the vocabulary and comprehension portions of the local curriculum assessments by the time the tests were administered the next spring.

There was some discussion about how to involve students in assessing their own reading and vocabulary growth, but neither the leadership team nor the staff wanted to set any performance benchmarks yet. Teachers who were especially interested in pursuing student selfassessment were asked to put together some ideas and try them out. As this effort evolved, three pairs of teachers for grades 1, 3, and 4 were going to try different assessment techniques: (1) have students keep vocabulary boxes of new words and words they were learning; (2) have students keep vocabulary journals about new words and words they were learning; and (3) interview students about the strategies they used when they encountered an unfamiliar word. These teachers would report the results and work with colleagues to resolve issues before any recommendations were made.

Cell 4—Information About the Current Learning Environment in Our School

Staff members reported on the instructional techniques they used in teaching vocabulary and provided specific examples of assessments. The members of the leadership team also decided to ask their study groups for especially good resources they had used in teaching vocabulary and word analysis skills. However, when the team evaluated the collection, some of which were their own favorites, they discovered, with Helen's help, that most of the resources were good activities but not very good conceptually. For example, there were a number of games and worksheets (both manual and computerized) that called for students to blend sounds and form words, match words and meanings, and find words in rows and columns of letters, as well as some simple crossword puzzles for older students. Some of the more promising techniques shared were using phonetic spelling and writing activities; using books on CD-ROM, such as those by Broderbund that allow students to listen to a story on the computer and also click on words they want repeated; and keeping the American Heritage Talking Dictionary CD-ROM available in the classroom writing center.

Cell 5—External Information About the Learning Environment

The team had to go beyond their current school resources to find out more about what elements would make up an optimal vocabulary development curriculum, which instructional techniques and models of teaching were most effective, and which ways were most effective for teachers to assess growth. Lisa had just finished her master's thesis, titled “Strategies for Building Vocabulary for Kindergarten Through Third Grade.” The team asked her to present a summary of her research review, the best references and teaching resources she had found, and her student results. The following points made by Lisa generated quite a bit of discussion among team members:

  • A variety of approaches to teaching phonics for decoding unfamiliar words are effective as long as they are systematic.
  • Asking students to locate and write definitions is not an effective approach to building vocabulary or word meaning.
  • Requiring struggling and average readers to use print dictionaries is not effective in building vocabulary.
  • Beginning around grade 3, the amount a student reads is the major determinant of his vocabulary growth.
  • Most textbooks do not include enough information about key concept words to help students understand the concepts or add the words to their sight vocabulary.

Lisa shared some of the resources on vocabulary instruction she had used to design interventions in her individual teacher research. She had found several strategies that were effective with her students in Stahl's (1999) Vocabulary Development, and she had found Beck, McKeown, and Kucan's (2002) descriptions of how to develop “student friendly explanations” of words in Bringing Words to Life especially helpful. Other resources that were brought to the team as part of this initial screening included two that the 1st grade teachers had used during their yearlong staff development: an article by the Cunninghams (1992) titled “Making Words: Enhancing the Invented Spelling-Decoding Connection,” and a short book by Calhoun (1999) titled Teaching Beginning Reading and Writing with the Picture Word Inductive Model. The curriculum director also found two resources she particularly liked: a chapter by Graves (1992) titled “The Elementary Vocabulary Curriculum: What Should It Be?” and a research synthesis by Ehri, Nunes, Stahl, and Willows (2001) titled “Systematic Phonics Instruction Helps Students Learn to Read: Evidence from the National Reading Panel's Meta-Analysis.”

Elwood's staff had four weeks left before their February 15 faculty development retreat. The principal and curriculum director arranged for multiple copies of each resource so that each would be read by at least four team members. Team members signed up to evaluate the resources and complete a structured response sheet prior to the retreat. On the response sheets, they are to identify the curriculum emphases and instructional techniques that, according to the authors, hold the most promise for accelerating literacy, especially for building reading vocabulary.

Cell 6—Learning Environment We Would Like to See

At the culmination of the February retreat, after a strenuous day of sifting through many apparently effective techniques and identifying recommendations and resources to share with the staff, the team felt comfortable with the student learning focus and the plan to work on expanding sight vocabulary and teaching students word recognition strategies they could use to add words to their reading and writing storehouses. After some debates and compromises, the team finally agreed to five curriculum, instruction, and assessment initiatives to recommend for schoolwide support and study in their focus area of expanding reading vocabulary and word analysis skills: (1) ask every teacher to teach students at least two strategies for building reading vocabulary and to help students assess their use of these strategies; (2) increase read-alouds to students from prekindergarten through grade 5; (3) increase the amount of reading by all students, K–5, from picture books to chapter books; (4) use the picture-word inductive model with beginning readers K–2 and as part of social studies units in grades 3 through 5; and (5) select or begin to build a common set of up-close assessments of students' sight vocabulary and word analysis skills in both reading and writing.

The principal and team leader agreed to draft these five initiatives into an action and staff development plan along with timelines, resources, and a budget, and to get it back to the team within two weeks. (The principal was concerned about the March 22 district deadline for the next year's school plans and budgets.) The curriculum director and Lisa planned to contact the regional intermediate service agency consultant to see if she could provide staff development on strategies for building reading vocabulary and on assessing vocabulary growth. Other staff members would be asked to lead staff development and planning sessions on their other three initiatives (read-alouds, wide reading, and the picture-word inductive model).

Once the team reviewed the action plan and made necessary modifications, it went to all staff members for review, along with several resources and excerpts that forecast what the staff would be studying and implementing under these initiatives.

As the school action plan was submitted to the staff and commitments were made, a number of social and school culture issues began to swirl around Elwood's next steps. Leadership team members worried about getting their study group members to read articles and research. The six teachers who would be providing staff development on readalouds, wide reading, and the picture word inductive model worried about how to get their colleagues to study implementation and plan lessons and instructional moves the way they did. In as many ways as she could, Helen thanked those concerned for their willingness to lead and reassured them that she would be participating in everything— the reading and study group discussions, the staff development, and learning and implementing the strategies— and that time would be protected at least twice a month for staff to work together, not counting their regular study group sessions.

* * * * *

Of course, this was just the beginning of Elwood's journey toward accelerating literacy for all students. Action research provides them with a structure for cycles of continuous inquiry into student and staff learning, and the will of the leaders—the principal, the teachers, and the curriculum director—provides the energy that will make it possible. Let's juxtapose a few of the actions of the Elwood faculty to attributes of successful school improvement efforts: focus on student learning, focus on staff learning, use of data, whole-school participation, use of the research base, use of external technical assistance, cross-role learning by all members of the organization, and leadership that elevates the organization.

Elwood focused its collective study on student learning, specifically student development in reading. While faculty members examined existing student data on both vocabulary and reading comprehension, they decided to work first on improving reading vocabulary. The other dimensions of reading development were not ignored; they were just not in the foreground for systematic formal study by the faculty as a unit.

Faculty members used their existing school and district data and sought out assessment tools that would give them more information about students' vocabulary development and use of word analysis skills. They are familiar with the demographic data that describe their student population, and they used these data as they analyzed their test results and thought about the implications. They looked for ways to involve students in assessing their own vocabulary acquisition processes.

Elwood's leadership team is made up of teachers, the principal, and the counselor. The district curriculum director works with the team as a technical assistant to model the action research process and help the team learn to use it. However, it is not just team members involved in studying the data, analyzing instructional and assessment practices, and studying the knowledge base in reading; all faculty members participate in the collective study. Leadership team members are simply willing to try things first and provide support to colleagues as the faculty struggles to become a more powerful learning community. The principal and many teachers—but not all of them—believe that student achievement in reading can improve. They have arguments—most of which are resolved civilly—but this does not stop the school improvement momentum. The strength and skills of the organizationally designated leaders, the principal and the curriculum director, work in tandem to make this kind of professional study and discourse possible.

And finally, Elwood uses its school action plan to organize collective study and serve as a binding agreement for actions. Tasks are scheduled; materials and human resources are secured. Obstacles to the development of a healthy learning community have not disappeared; they are almost always present in human organizations. But—at least while these leaders and teachers are supported—collective, disciplined inquiry into student learning and educational practices will be a part of the fabric of school life at Elwood.




Loading Comments...