What does it really mean to “measure impact”? For K-12 educators and out-of-school time (OST) providers alike, this question goes beyond compliance and data dashboards. It’s about understanding how young people learn—and how to tell that story in ways that are meaningful.
In a recent ASCD webinar, Measuring Impact: Tracking Youth Development Beyond School Hours, Zakia Redd, research scientist at Child Trends, and Isaac Castillo, senior advisor for learning and evaluation at the nonprofit Kaboom, unpacked this question, drawing on findings from a Wallace Foundation report on OST program data practices.
These programs, which include before- and after-school, weekend, and summer learning initiatives, play a vital role in the broader learning ecosystem. As summer approaches and enrichment programs gear up, it’s a good time to better understand how these programs measure impact and improve student learning outcomes.
What Counts as "Success" in Out-of-School Programs?
The Wallace study gathered insights from 28 OST programs across the U.S., with a focus on those serving marginalized youth, supporting social-emotional learning, promoting equity, and working with systems-involved children and adolescents, such as those in juvenile justice programs or other public systems. Program content areas included civic engagement and social justice, career readiness, STEM programs, the arts, and more.
For educators, it’s crucial to understand just how diverse OST programs are in practice. Programs operate at different times (before/after school, weekends, summer), in different settings (in schools, community centers, online), and under different umbrellas (public agencies, private organizations). Each of these factors influences what kind of data is collected, how it’s used, and what counts as “success” when it comes to student learning and development.
A key takeaway? Though these programs vary widely, they share a common challenge: collecting the right data, in the right ways, to create their own story of student learning. But that’s often easier said than done. With no one-size-fits-all approach and limited staff capacity, many OST programs are left navigating tough decisions about what to measure, how to measure it, and how to keep the process manageable.
Programs share a common challenge: collecting the right data, in the right ways, to create their own story of student learning.
Shared Priorities—and Pain Points
Despite their differences, programs in the study shared a clear focus on measuring outcomes aligned to their mission, whether that was SEL competencies like empathy and self-confidence or program-specific goals like job readiness or technical skill development. Many also disaggregated their data—examining results by race and other demographic factors—to uncover disparities, ensure outcomes reflected students’ lived experiences, and select collection tools that felt authentic and relevant to the young people they served. They also tracked participation and program quality—important output data that can signal whether a program is on the right path.
While many relied on familiar methods, others got more creative: Alongside traditional surveys and interviews, OST programs used portfolios, student journaling, individual and group check-ins with students, and even games and award systems to track progress—especially in areas that can be hard to quantify.
Despite their intentional approaches, OST programs grappled with a common set of challenges. Programs reported:
- Difficulty measuring long-term outcomes like college and career success.
- Limited access to school data, such as test scores.
- Staff burnout from the demands of collecting, analyzing, and reporting data.
- Participant burnout, especially when data collection feels disconnected from the program experience.
As the report co-authors explain, “While many of the staff members we interviewed got into the OST field to work with young people, not data, their desire to help young people develop new skills and achieve success and well-being motivated them to learn a lot about how to collect and use data.”
That mindset—driven by care for students and strengthened by practical, sustainable strategies—is a key throughline running across the study’s findings. And for K-12 educators, it offers useful guidance on how to rethink data use in ways that reflect real-world constraints while staying rooted in student growth.
What Educators Can Learn from OST Programs
What can leaders and teachers take away from this research, especially as summer programming launches and conversations about student growth get underway? Redd and Castillo highlighted several strategies that can help schools and OST programs build more intentional approaches to measurement:
Start with the end in mind. Programs that map out their intended impact—starting with the skills and experiences they hope to cultivate—are better equipped to decide what to measure and how. Working backward by identifying outcomes first helps programs focus their efforts and avoid collecting data they won’t use. While there’s no one right way to do this, building a clear logic model makes it easier to connect the dots among programming, outcomes, and the ultimate story you want your data to tell.
Get specific. Broad goals like “academic improvement” can be hard to measure in a meaningful way. Going a few levels deeper—zeroing in on “improved math proficiency” or defining even more targeted outcomes like “larger year-to-year increases on math standardized tests”—makes it easier to collect relevant data. The more clearly you define your outcomes, the more clearly you can identify—and communicate—progress.
The more clearly you define your outcomes, the more clearly you can identify—and communicate—progress.
Use research to lighten the lift. You don’t have to document every single link between your programming and the outcomes it aims to achieve. Using existing literature to help make these connections can reduce the burden on staff while strengthening your evidence base. For example, the nonprofit Kaboom, which builds play spaces in under-served communities, drew on national research linking nature access to positive mental health outcomes to frame how their playgrounds contribute to child well-being—even when they weren’t measuring every step themselves.
Keep it usable and sustainable. To reduce burden on staff, identify a small number of high-priority outcomes and focus your efforts there. Ensure data collection stays manageable and meaningful by choosing measures that fit your team’s capacity, align with your logic model, and are usable in practice—for example, data that can inform decisions, improve programming, or be shared with stakeholders. Collecting data at a few strategic points in the calendar can also lighten the load and reduce burnout.
Make room for student voice. Some of the most meaningful data can come from asking students what they hope to gain from a program or initiative and how they perceive their own growth. Programs that involve youth in shaping what’s measured—and how—can gather richer insights. Whether through text surveys, emoji-based responses, or good old-fashioned conversations, prioritizing the perspectives of learners on data collection can make the process much more relevant and engaging.
Some of the most meaningful data can come from asking students what they hope to gain from a program or initiative and how they perceive their own growth.
Measuring What Matters
OST programs aren’t just add-ons to the school day. They’re essential partners in helping students develop skills, confidence, and connection beyond the classroom. As summer begins, many schools are launching enrichment initiatives or partnering with OST providers to keep students learning. This moment offers an opportunity to rethink how best to define and measure success—and to apply strategies that are more intentional, equitable, and reflective of what students truly need.
To learn more, read the full report and watch the webinar recording.
