Use of I-SWIS

 

Use of I-SWIS by Elementary Schools to Monitor Tier 3 Behavior Supports


Kathleen M. Conley, Robert H. Horner, and Kent McIntosh


A core feature of Positive Behavioral Interventions and Supports (PBIS) is the collection, summary, and use of data for iterative decision-making. The initial design of support and the adaptations that make behavior support match cultural, organizational, and personal needs require that a support team have functional information to guide decisions. This process has been described frequently for Tier 1 (universal) and Tier 2 (targeted) support systems. In this Evaluation Brief we focus on the extent to which Tier 3 (individual) support teams collect, summarize, and use data to guide implementation of behavior support plans. Our specific focus is a descriptive summary of how elementary school teams use the Individual School Wide Information System (I-SWIS) online application (May et al., 2017) to guide their Tier 3 decision-making as well as general use of PBIS-focused data systems that support overall fidelity to critical components of the PBIS framework. By analyzing data from schools currently engaged in efforts to implement PBIS at Tier 3, we hope to spur further attention to best practices for coordinating evaluation data effectively and efficiently across all tiers and interventions.

What is I-SWIS?

The Individual School-Wide Information System is a webbased computer application, offered through PBIS Apps (pbisapps.org). I-SWIS is designed to promote an organized and team-based approach to monitoring individualized student support. Use of I-SWIS has been demonstrated to improve both the fidelity of individualized behavior support plan implementation and the impact of the plan on student behavior (Pinkelman & Horner, 2016). I-SWIS is typically used to collect and display the extent to which a support plan is being implemented as proposed and change in targeted student outcome data. Figure 1 demonstrates a sample report of data entered into I-SWIS for a fictional student, Carly Johnson, whose case file was designed to collect data daily on staff implementation of the behavior support plan as well as Carly’s rates of disruptive outbursts during the day. The report indicates that during the most recent 30-day period the fidelity of support plan implementation has remained high and disruption has decreased steadily. There are indications of plan changes and notes that can be referenced to provide context to the quantitative data.

Figure 1: Simulated I-SWIS report for student Carly Johnson indexing daily rates of disruption and level of support plan implementation fidelity.

Given existing research documenting that use of I-SWIS improves Tier 3 supports, this Evaluation Brief focuses on the patterns and contexts of I-SWIS use by Tier 3 school teams. The specific questions examined were:

  1. Are schools that use I-SWIS implementing PBIS (at Tiers 1, 2, and 3) with fidelity?
  2. What patterns of office discipline referrals exist for students monitored in I-SWIS?
  3. Are students monitored in I-SWIS likely to also have Check-In/ Check-out data?
  4. Do Tier 3 support plans monitored in I-SWIS include measures of support plan implementation fidelity and student outcome data?

Method

School Sample

A group of 50 elementary schools (K-5 grades) across 16 states in the U.S. were included in the sample. Schools were included if they were (a) using the I-SWIS application, (b) had at least one student case file with one day of fidelity or outcome data, (c) no more than 10% of the reported total student enrollment had an I-SWIS case file, and (d) agreed to share their data with the National Technical Assistance Center on PBIS for research purposes during the 2016-17 school year. Average enrollment was 500.54 students (range 265 to 1,117). Additional demographic data was available for 48 schools in the sample. The average reported percent of students receiving free or reduced-price meals was 50.67% (range 10% to 100%). School urbanicity identified schools in rural (n = 3), town (n = 9), suburban (n = 26), and city (n = 10) locales.

Student Sample

The sample included 368 students with case files monitored in I-SWIS across the 50 schools. Case files were included if there was at least one day of quantitative fidelity or outcome data entered during the 2016-17 school year. The number of student case files per school ranged from 1 to 40 with a median of 4.5 students, or 1% of the average total school enrollment. Demographic data were entered into I-SWIS for 304 (82.60%) of the students in the sample. Student gender was 71.70% male. There were 122 (33.20%) students with an active IEP and 6 (1.60%) students with an active 504 plan in place.

Measures

Demographics

Data on school and student characteristics was obtained from the National Center for Educational Statistics (NCES) and the PBISAssessment.org website.

PBIS Fidelity: The Tiered Fidelity Inventory (TFI) was used to assess the extent to which participating schools were using PBIS across Tiers 1, 2, and 3 (Algozzine et al., 2014). The TFI scores are collected within the PBIS Assessment online application and allow documentation of the extent to which core PBIS practices are implemented with a level of fidelity associated with improved student outcomes. Additionally, the subscale scores were analyzed for the Tier 3 section of the TFI to further break down the specific efforts toward individualized and intensive behavioral supports. Subscales for Tier 3 include teaming, resources, support, and evaluation.

Office Discipline Referrals (ODRs)

Student office discipline referrals were reported by school personnel using the SWIS application (May et al., 2017; reviewable at pbisapps.org), and summarized here to answer question 2, regarding ODR patterns per participating student receiving Tier 3 support.

Check-In Check-Out (CICO; Crone, Hawken, & Horner, 2010)

The participation of students in the Tier 2, CICO intervention was assessed through data provided by school personnel in the CICO-SWIS online application, also part of PBISApps.org. These data were used to summarize the number of days students participated in CICO as well as receiving Tier 3 supports.

Participation in Tier 3 Supports

The I-SWIS online application was used by school staff to enter information about (a) fidelity of support plan implementation, (b) decisions made by the support team to continue, modify, or fade support plans, and (c) impact of the plan on student behavior. The I-SWIS application also allowed summary of the number of measures and the number of days of data per student.

Results

Are schools that use I-SWIS implementing PBIS with fidelity at Tiers 1, 2, and 3?

A total of 27 schools (54%) entered TFI data during the 2016-17 school year. Schools could assess their fidelity at each of the three tiers, or only with one or two tiers. This resulted in different numbers of schools with TFI data per tier (N = 24 for Tier 1; N = 19 for Tier 2; N = 16 for Tier 3). Figure 2 provides the TFI scores across the sample by tier. Across the sample the 24 schools with Tier 1 fidelity data reported an average of 86.53% (Sd = 11.61%). The 19 schools with Tier 2 fidelity data averaged 85.43% (Sd = 17.79%), and the 16 schools with Tier 3 fidelity data reported an average of 76.84% (Sd = 17.61%). A score of > 70% on the Tier 1 scale is considered acceptable fidelity to the PBIS framework (McIntosh et al., 2017). Tiers 2 and 3 do not currently have a validated threshold for acceptable fidelity. However, a recent evaluation brief by the national PBIS Technical Assistance Center provided a summary of over 8,000 schools that completed one or more section of the TFI during the 2016-17 school year (Kittelman, Eliason, Dickey, & McIntosh, 2018). The current sample of schools using I-SWIS reported scores above the average across each tier compared to the larger sample of schools that completed the TFI during the same school year. For the larger sample described in this brief, the average Tier 1 score was 74% (Sd = 22%), the average Tier 2 score was 69% (Sd = 26%), and the average Tier 3 score was 62% (Sd = 28%).

For the 16 schools that reported Tier 3 PBIS fidelity, figure 3 provides the subscale scores to provide further insight on implementation efforts. Subscales group items by topic and provide a more detailed categorical score. For the subscale of items related to Tier 3 teaming, schools reported an average score of 82.81% (Sd = 20.35%). For the subscale related to resource allocation, schools reported an average score of 73.96% (Sd = 19.22%). For the subscale related to individualized student supports, the average score was 79.17% (Sd = 22.57%). For the final subscale related to evaluation practices, the average score was 69.53% (Sd = 20.40%).

Figure 2: TFI Scores Across Sample Schools for Tier 1, Tier 2, and Tier 3

Figure 3: TFI Subscale Scores for Tier 3

What patterns of office discipline referrals exist for students monitored in I-SWIS?

Among students with I-SWIS cases, there were 124 students (33.70%) who received 0 to 1 major ODRs during the 2016-17 school year. There were 90 students (24.50%) who received between 2 and 5 major ODRs. A total of 154 students (41.80%) of the sample received 6 or more major ODRs during the year. For students who received 6 or more ODRs, the total number for the year ranged from 6 to 126 ODRs with a median of 15 ODRs per student.

Are students monitored in I-SWIS more likely to also have Check-In/ Check-Out data?

CICO data for the students in I-SWIS was analyzed as a potential indicator that the student received Tier 2 supports during the same school year they received Tier 3 supports. There were 64 students (17.39%) across the sample who had CICO data recorded during the 2016-17 school year. The number of days students participated in CICO ranged from one to 149 days with a median of 25 days.

Do Tier 3 support plans monitored in I-SWIS include measures of support plan implementation fidelity and student outcome data?

Across the sample of 368 students with I-SWIS case files there were 202 students (54.89%) with Tier 3 support plan implementation fidelity data entered during the 2016-17 school year. The total days with support plan fidelity data for these case files ranged from one to 112 days, with a median of 6.00 days. A total of 360 students (97.83%) had student outcome data. The total days with outcome data for the case files ranged from one to 209 days, with a median of 59 days. This brief focuses on whether schools monitored fidelity and how often. Future attention is needed to determine the quality of fidelity measures, methods of collection, and extent that team-identified criteria for acceptable fidelity are met.

Discussion

This Evaluation Brief is descriptive in nature and is intended to increase attention to current and best practices in use of data by teams providing Tier 3 supports. Although the sample size is small, these are among the first descriptive data summaries related to use of individual student fidelity and outcome data across schools using PBIS.

PBIS Implementation Fidelity

Just over half the schools completed the TFI measure, and only 32% of the 50 schools completed the Tier 3 items. Average scores indicate that schools with TFI data scored 75% or higher across all tiers of PBIS and at least 69% across subscales. The average scores were higher compared to the broader national average scores that submitted the TFI during the same school year (Kittelman et al., 2018). The use of I-SWIS was specifically designed to support the evaluation subscale of Tier 3. Although this sub-sample is very small, the average Tier 3 evaluation score was 76.84%, higher than the reported national average of 62%, indicating that schools using I-SWIS are invested in efforts to implement systematic Tier 3 supports.

Students Receiving Tier 3 Supports

Students with I-SWIS case files represented a median of 1% of the reported total enrollment, which is expected given the intensive nature of Tier 3 supports. The PBIS literature often points to a range of 1% to 5% of students receiving individualized supports as an indicator that Tier 1 and 2 supports are effective. The use of I-SWIS may not be inclusive of all students receiving Tier 3 supports at a school, and further research is needed to determine how student support teams would determine when I-SWIS is the most effective and efficient data system to monitor Tier 3 supports and progress. Additionally, further exploration is needed on the measures a school team would use to determine whether their proportions of students served at each Tier are sustainable and aligned to the PBIS framework.

Fidelity to the Documented Behavior Support Plan

A major challenge within advanced tiers of PBIS is the need to monitor fidelity to the interventions purported to be implemented (Wehby & Kern, 2014). For universal and even many targeted interventions the fidelity measures can be easily standardized. For Tier 3 supports, each plan is individualized and requires adults to adjust the student’s routines, instruction, and response to behaviors. These activities require individualized measures of fidelity that align with the student’s unique support plan. Much more work is needed on guidelines and examples of fidelity measures that are effective in providing valid and reliable indicators of implementation as well as efficient for staff to collect within the time and resources allotted to deliver the support plan. It is encouraging that just over half of the cases included some assessment of individual support plan fidelity. More attention to this variable is warranted.

Monitoring Student Outcomes

Similarly, there are questions about the outcome measures used to demonstrate that a student is benefiting from current supports and making adequate progress toward the specific goals set within the support plan. Student support teams need guidance to answer questions about progress monitoring, such as (a) When is a single measure of progress sufficiently sensitive to change and when are multiple measures needed? and (b) What types of measures (e.g., rate, Likerttype scale, percent) are best suited to which behaviors?

Future discussions and research are needed to fill in gaps regarding current and best practices in advanced tiers of support, especially at Tier 3. Next steps in the evolution of PBIS will be to support schools as they balance the individualized and often intensive nature of Tier 3 supports with a need for standards and guidelines that establish minimum quality and equity. The present brief serves as one small step in drawing attention to the questions and practices used by schools using I-SWIS as a progress monitoring tool within Tier 3 supports.

References

Algozzine, R. F., Barrett, S., Eber, L., George, H., Horner, R. H., Lewis, T. J., Sugai, G. (2014). PBIS Tiered Fidelity Inventory (TFI). Eugene, OR: OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. Available at www.pbis.org.

Crone, D. A., Hawken, L. S., & Horner, R. H. (2010). Responding to problem behavior in schools: The Behavior Education Program (2nd ed.). New York: Guilford.

Kittelman, A., Eliason, B. M., Dickey, C. R., & McIntosh, K. (2018). How are schools using the SWPBIS Tiered Fidelity Inventory (TFI)? OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports www.pbis.org.

Pinkelman, S. E., & Horner, R. H. (2016). Improving implementation of function-based interventions: Self-mon- itoring, data collection, and data review. Advanced online publication. Journal of Positive Behavior Interventions. doi:10.1177/1098300716683634

Wehby, J. H., & Kern, L. (2014). Intensive Behavior Interven- tion: What Is It, What Is Its Evidence Base, and Why Do We Need to Implement Now? TEACHING Exceptional Children, 46(4), 38-44. doi:10.1177/0040059914523956

Suggested Citation for This Publication

Conley, K. M., Horner, R. H., & McIntosh, K. (2019). Use of I-SWIS by Elementary Schools to Monitor Tier 3 Behavior Supports. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. www.pbis.org.