Collecting and analyzing data on students' overall literacy performance is important to the evaluation of both core (Tier I) and supplemental (Tiers II and III) instruction. As part of your data-analysis process, you will want to examine what the data is telling you about instruction in all tiers.
Quality, evidence-based Tier I instruction is at the foundation of an effective RTI framework, and without it in place, it is likely that the need for Tiers II and III instruction will exceed available resources. Thus, using summative data to evaluate Tier I instruction should be considered a critical step in evaluating the RTI process at your campus. For example, if 40% of students at your campus are unsuccessful on the EOC STAAR assessment, you might consider the ways in which you can adjust Tier I instruction.
Likewise, it is important that the analysis of summative data be extended to the evaluation of Tiers II and III instruction, and you and your team will want to ensure that this activity is integrated in the decision-making process for students receiving additional support (McDougal, Graney, Wright, & Ardoin, 2010). Disaggregating outcome data for students receiving Tiers II and III interventions will help you identify trends within and across these groups. If there are significant numbers of students in Tier II or III not making adequate gains, you have reason to further evaluate the efficacy of that intervention instruction. In turn, if there are significant gains made by particular groups of students in specific interventions, you will want to identify why those practices were successful and coordinate resources to ensure those practices are implemented across intervention groups.
Once your team has analyzed patterns in the data and identified possible causes for performance gaps, you will want to extend this process to evaluate the instruction being delivered to students in all tiers. You may consider the following questions when evaluating Tiers I, II, and III instruction and addressing performance gaps:
You may want to download a copy of the entire flow chart Analyzing Outcome Data to Inform Action: Questions to Guide the Process.
While analyzing outcome data is essential to understanding program effectiveness and making decisions for the next school year, it is also important to understand how effective a program is while it is being implemented. This is especially true for students in intervention programs. It may not be acceptable to wait until the end of the year to know if a yearlong intervention class is effective. You might want to know at several points throughout the year to what degree the intervention is working. In this case, you might consider ways to measure program effectiveness throughout the year. Many schools use district-created, curriculum-based assessments to measure Tier I program effectiveness. These assessments are closely aligned to the goals of Tier I instruction (i.e., mastery of state standards) and enable teachers to adjust their instruction if it was not effective. This kind of assessment data should be shared with interventionists.
In addition, interventionists should consider what kinds of assessments they could administer to understand how effective the intervention is for most students. You may find it useful to look at progress monitoring data or use other assessments that are likewise closely tied to the goals of the intervention in determining the effectiveness of intervention instruction throughout the year. In this case, you are looking through the lens of program evaluation with a focus on overall effectiveness of intervention instruction rather than looking at individual student responses to the instruction. Switching between these "lenses" may take place at the same progress monitoring intervals but allows you to look at the bigger picture of Tiers II and III instruction overall (The High School Tiered Interventions Initiative: Progress Monitoring, Center on Intervention).
Within an RTI framework, intervention integrity is a critical factor to consider in the problem-solving process. This pertains to how well tiered instruction is delivered "with regard to content, quantity, and process" as part of your RTI program (Ball & Christ, 2012, p. 237). Another common term used for this characteristic is fidelity. Fidelity of the intervention program refers to the quality of its implementation and the degree to which practitioners "implement programs as intended by the program developers" (Using Fidelity to Enhance Program Implementation Within an RTI Framework, Center on Intervention).
When determining the degree to which students have met goals, it is important to measure and evaluate the fidelity of instruction delivered to those students and identify any departures or modifications made to intervention instruction (e.g., modified pacing, inconsistency in time allotted or daily schedule). You and your team will need to determine how you can measure the fidelity of implementation and what impact any deviations have had on student learning, and then you can plan for ways to address any gaps in the delivery of instruction within your RTI framework.
As part of your instructional planning, you may consider ways to better measure fidelity of instruction to ensure that students are receiving high-quality, evidence-based intervention instruction before making decisions about students' future needs. There are a variety of tools that can help measure and monitor fidelity. The Monitoring Fidelity in RTI webinar, available on the Center on Response to Intervention site, is one helpful resource for understanding fidelity in RTI. The webinar also discusses specific tools that may help you monitor the integrity of intervention instruction.
In sum, as you engage in data analysis and critical discussion of these issues, you and your team will want to determine action steps to address any identified gaps. As part of the cycle of data-based decision making, you will want to ensure that instructional staff collaborate to set program-wide goals for the next year and create a plan to meet those. As the cycle continues, you will use your balanced assessment plan to determine the impact of your instructional plan and the degree to which goals are being met.
This cycle of data analysis and decision making is part of the practice of creating, evaluating, and revising your data-informed plan for improving literacy instruction, as introduced in the Leadership module of this course.
TO LEARN MORE: Extensive research and resources are available on the topic of using data to inform educational reform efforts. These resources can provide guidance as you establish and maintain this process at your campus in a way that best aligns with the needs of educators and students in your particular context.
The Center on Response to Intervention's Implementation & Evaluation section provides information about RTI implementation, implementing RTI with fidelity, and evaluating your RTI framework. It contains a variety of guidance documents and training modules related to these topics.
The Regional Educational Laboratory (REL) Pacific has put together an excellent resource on logic models, “Logic Models: A Tool for Designing and Monitoring Program Evaluations.” Logic models can help educators understand the causes, effects, and systems surrounding programs (such as Tier II interventions or the implementation of a new Tier I curriculum) on their campuses. This introduction to logic models as a tool for designing program evaluations defines the major components of education programs—resources, activities, outputs, and short-, mid-, and long-term outcomes—and uses an example to demonstrate the relationships among them.
REL Pacific has also put together a brief, “Five Steps for Structuring Data-Informed Conversations and Action in Education,” which outlines guiding questions, suggests activities, and provides a framework and tools needed to support tough conversations about data. The guide also outlines five key steps in using data to make decisions about program effectiveness: setting the stage, examining the data, understanding the findings, developing an action plan, and monitoring progress and measuring success.
Brown and Doolittle's (2008) practitioner brief titled “A Cultural, Linguistic, and Ecological Approach to Response to Intervention with English Language Learners” provides guidance in ensuring that the RTI framework is implemented to meet the unique needs of English learners. Practitioners can find guiding questions to consider when evaluating intervention decisions made about English learners and the integrity of instruction in each tier. It is available on the Center on Response to Intervention site.
REL Mid-Atlantic conducted a webinar, Root Cause Analysis, which provides an outline of root cause analysis and several resources to help educators implement root cause analyses for problems that arise on their campuses. Root cause analysis can be a powerful method for educators to analyze data in new ways and identify the root causes of events. This type of analysis can help you avoid the pitfall of trying to solve the "wrong" problems and missing the root causes of the issues at hand.
NEXT STEPS: Depending on your progress in using assessment data to evaluate overall literacy performance, you may want to consider the following next steps:
- Determine how data are being used to evaluate the effectiveness of programs as they are being implemented.
- Discuss how outcome data are used to make instructional and programmatic decisions. You might use the third and fourth columns of the Assessment Audit to think through these issues.
- Gather and review the intervention materials (e.g., teachers' guide, instructional manuals) to determine if interventions are being delivered as designed.
- Determine which staff members have been trained in analyzing summative data and plan necessary professional development to address staff needs in this area.
- Determine procedures for gathering and sharing additional data to support valid decisions for all students and student groups, such as English learners.
