Fidelity in High Schools

 

Fidelity of SW-PBIS in High Schools: Patterns of Implementation Strengths and Needs


Jessica Swain-Bradway, Midwest PBIS Network
Jen Freeman, University of Connecticut
Angus Kittelman, University of Oregon
Rhonda Nese, University of Oregon


School-wide Positive Behavioral Interventions and Supports (SW-PBIS) is an evidence-based, multi-tiered framework designed to support all students across all school settings (Horner, Sugai, & Anderson, 2010). The implementation of SW-PBIS is associated with improved student outcomes across elementary, middle, and high schools. For high schools especially, the implementation of SW-PBIS is associated with reductions in student dropouts, office discipline referrals, the numbers of students needing more intensive supports, as well as increased student attendance (Bohanon et al., 2006; Flannery, Fenning, Kato, & McIntosh, 2014; Freeman, Simonsen, et al., 2016).


Although the number of schools adopting SW-PBIS continues to increase in the U.S., the rates of high schools adopting SW-PBIS is slower compared to elementary schools. High schools represent approximately 34% of all schools in the US, but only 7% of those implement SW-PBIS. Out of all schools implementing SW-PBIS, only 13% of those are high schools (Freeman, Wilkinson, & Vanlone, 2016). Recent research suggests that the latency between initial training and reaching adequate SW-PBIS implementing fidelity is longer on average for high schools, compared to both elementary and middle schools (Nese, Nese, McIntosh, Mercer, & Kittelman, in press).

To increase the number of high schools adopting SW-PBIS and implementing to fidelity, more attention is needed to identifying specific implementation components that are likely to be challenging for high school staff to implement. Therefore, the purpose of this brief is to summarize patterns in SW-PBIS implementation fidelity in 996 high schools across 31 U.S. states. We specially aimed to evaluate how high schools are implementing the components of SW-PBIS, and identify implementation strengths and needs.

Research Question: What are the patterns of Tier I SWPBIS implementation for high schools at low fidelity, partial fidelity, and high fidelity?

Method

Sample

The sample consisted of 996 high schools in 31 states that met the following criteria: (1) a public, four-year school with a grade range of 9-12 (alternative schools, charter and juvenile justice facilities, and those with alternate grade level range (e.g., 8-12, 9-10, 11-12) were excluded), (2) at least 1 implementation fidelity score using the School-wide Evaluation Tool (SET; Sugai, Lewis-Palmer, Todd, & Horner, 2001) ranging from 0–100 reported to the National Technical Assistance Center on PBIS between the 2005-2006 and 2012-2013 school years, (3) complete NCES school demographic data, and (4) agreement to the use of their data for research purposes. Schools in this dataset entered their SW-PBIS fidelity data through a free online application (PBIS Assessment; pbisapps.org). These data are available from the OSEP National Technical Assistance Center on SW-PBIS, maintained at the University of Oregon.

Measure

Implementation fidelity. The SET is a 28-item research tool to assess SW-PBIS critical features reflected in seven subscales: Expectations Defined, Expectations Taught, Reward System, Violation System, Monitoring and Evaluation, Management, and District Support (Horner et al., 2004). A trained external evaluator typically completes the SET, although in some districts, a district or school-based coach may complete the SET. A school is considered implementing Tier 1 at fidelity when they meet or exceed a total score of 80% and a score of 80% on the Expectations Taught subscale. The SET is a widely used measure of SW-PBIS implementation fidelity and contains strong psychometric properties (Horner et al., 2004; Mercer, McIntosh, & Hoselton, 2017).

Analysis

Fidelity categories. To examine differences in implementation scores, we pooled the high schools into three implementation fidelity groups. Groups were based on each high school’s most recent SET total implementation fidelity score. Schools were coded as being (1) High Fidelity (SET total scores equal to or greater than 80), (2) Partial Fidelity (SET total scores equal to-or-greater than 40 and less than 79), and (3) Low Fidelity (SET scores less than 40). This process resulted in a sample of 264 high schools in the High Fidelity group, 334 high schools in the Partial Fidelity group, and 86 high schools in the Low Fidelity group. To examine group differences in the three implementation groups, we calculated descriptive statistics of the high schools’ total SET Tier 1 implementation fidelity and seven subscales scores on the SET. Average across the implementation fidelity groups were compared to determine which subscales were rated highest and lowest per group.

Results

Figure 1 shows the average scores for the total score and individual subscales on the SET for each of the three implementation fidelity groups.

High Fidelity. Average SET subscale scores ranged from 83% to 95% for high schools in the High Fidelity group. The lowest subscale score was for the Rewards Systems (M = 83%). Monitoring Evaluation (M = 94.9%) and Expectations Defined (M = 93.2%) were scored highest on average.

Partial Fidelity. Average SET subscale scores ranged from 60.5% to 73.1% for the Partial Fidelity group. Rewards System (M = 43.3%) and Expectations Taught (M = 45.6%) represented the lowest subscale scores. Four subscales clustered closely at the higher range of subscale scores: Violations Systems (M = 73.1%), District Support (M = 72.9%), Leadership (M = 72.4%) and Monitoring Evaluation (M = 72%).

Low Fidelity. Average subscale scores for schools in the Low Fidelity group ranged from 9.3% to 52%. Expectations Taught (M = 9.3%) and Rewards System (M = 9.5%) had the lowest average scores. District Support (M = 51.2%) and Violation Systems (M = 52%) were the highest.

Discussion

By examining patterns of Tier 1 SW-PBIS implementation in high schools, we can adjustment training and technical assistance to address common areas of implementation weakness. Understanding which components are typically more difficult to implement to fidelity using the SET assists teams, coaches, and trainers to problem solve and action plan during the early stages of implementation.


The four highest scored subscales for high schools in the Partial Fidelity and Low Fidelity groups were 1) Violation Systems, 2) District Support, 3) Leadership, and 4) Monitoring Evaluation. The Violation Systems and Monitoring Evaluation subscales may represent systems already in place, as part of typical high school operations. This is congruent with the national rate of suspensions in high schools (Losen & Gillespie, 2012), as well as mandatory data reporting to the state and/or district. Expectations Defined, Expectations Taught, and Rewards Systems, which represent direct delivery of SW-PBIS strategies to students were all scored lowest for high schools in the Partial Fidelity and Low Fidelity groups.

All implementation groups had the lowest comparative averages for the Rewards Systems subscale. Even among high schools in the High Fidelity group, the implementation of Rewards Systems was an area of lower average implementation. Lower average Rewards Systems subscale scores across all three fidelity groups is not surprising given the difficulty some high school teams report when encouraging staff to deliver rewards (Swain-Bradway, Loman, & Vincent, 2014). This is also not surprising given the relatively high average scores for Violation Systems across three groups, with average implementation scores of 88.8% for the High Fidelity, 73% for the Partial Fidelity, and 52% for the Low Fidelity group. Teams may have a difficult task establishing a strong Rewards System in the presence of well-established violations practices and protocols, the cornerstones of “traditional” high school discipline systems. Teams may struggle to realign Violations Systems with an instructional, inclusionary approach, and more research on how to align these systems is needed.

Conclusion

There is growing evidence that SW-PBIS is an effective, systematic approach to improving outcomes for high school students (Freeman et al., 2016a; 2016b). Implementing this framework with high Tier 1 implementation fidelity is critical for realizing these improvements. The results of this descriptive study indicate that high schools are likely to struggle more to implement specific components critical to SW-PBIS Tier 1 implementation. These findings may be used to guide the training and other technical assistance provided to high school teams in order to focus implementation efforts to address weaknesses for robust implementation. High schools in the High Fidelity group in this sample, demonstrated that (a) defining and teaching school-wide practices are critical for reaching high fidelity, (b) high schools struggle to implement Rewards Systems, and (c) high schools have fairly robust Violations Systems.

References

Bohanon, H., Fenning, P., Carney, K. L., Minnis-Kim, M. J., Anderson-Harriss, S., Moroz, K. B., & Sailor, W.  (2006). Schoolwide application of positive behavior support in an urban high school: A case study. Journal of Positive Behavior Interventions, 8, 131–145.

Flannery, K. B., Fenning, P., Kato, M. M., & McIntosh, K.   (2014). Effects of school-wide positive behavioral interventions and supports and fidelity of implementation on problem behavior in high schools. School Psychology Quarterly, 29(2), 111–124. doi: 10.1037/spq0000039

Freeman, J., Simonsen, B., McCoach, D. B., Sugai, G., Lombardi, A., & Horner, R.  (2016). Relationship Between School-Wide Positive Behavior Interventions and Supports and Academic, Attendance, and Behavior Outcomes in High Schools. Journal of Positive Behavior Interventions, 18(1), 41–51. doi: 10.1177/1098300715580992

Freeman, J., Wilkinson, S., & Vanlone, J.  (2016). Status of high school PBIS implementation in the U.S. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports.

Horner, R. H., Sugai, G., & Anderson, C. M.  (2010). Examining the evidence base for School-wide Positive Behavior Support. Focus on Exceptional Children, 42(8), 1–14.

Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B.  (2004). The School-wide Evaluation Tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12.

Losen, D. J., & Gillespie, J.  (2012). Opportunities suspended: The disparate impact of disciplinary exclusion from school. Los Angeles, CA: Center for Civil Rights Remedies at The Civil Rights Project at UCLA.

Mercer, S. H., McIntosh, K., & Hoselton, R.  (2017). Comparability of Fidelity Measures for Assessing Tier 1 School-Wide Positive Behavioral Interventions and Supports. Journal of Positive Behavior Interventions, 19(4), 195–204. doi: 10.1177/1098300717693384

Nese, R. N. T., Nese, J., McIntosh, K., Mercer, S. H., & Kittelman, A. (in press). Predicting latency of reaching adequate implementation of school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions.

Sugai, G., Lewis-Palmer, T. L., Todd, A. W., & Horner, R. H. (2001). School-wide Evaluation Tool (SET). Educational and Community Supports. Retrieved from http://www.pbis.org

Swain-Bradway, J., Loman, S. L., & Vincent, C. G. (2014). Systematically addressing discipline disproportionality through the application of a school-wide framework. Multiple Voices for Ethnically Diverse Exceptional Learners, 14, 3–17.