Use of TFI

How are Schools Using the SWPBIS Tiered Fidelity Inventory (TFI)?

Angus Kittelman, Bert M. Eliason, Celeste Rossetto Dickey, and Kent McIntosh
University of Oregon

PDF Download

School-Wide Positive Behavioral Interventions and Supports (SWPBIS) is an evidence-based framework that provides a continuum of supports for students across multiple tiers (Tiers 1, 2, and 3). The implementation of SWPBIS is associated with improvements in student behavior, academic outcomes, and school climate (Bradshaw, Koth, Bevans, Ialongo, & Leaf, 2008; Bradshaw, Mitchell, & Leaf, 2010). SWPBIS relies on school planning teams regularly assessing fidelity of implementation to ensure the critical features of SWPBIS are implemented as recommended to achieve meaningful outcomes.


School planning teams are encouraged to assess and evaluate the fidelity with which SWPBIS is implemented using one of the free PBIS Assessment surveys (www. pbisapps.org). The SWPBIS Tiered Fidelity Inventory (TFI; Algozzine et al., 2014) is the newest, research-validated instrument for assessing SWPBIS implementation fidelity available on PBIS Assessment. The measure is comprehensive and unique because it measures the implementation of the core SWPBIS features across all three tiers. In a recent empirical study, McIntosh and colleagues (2017) found the TFI to be strongly and significantly related to other SWPBIS fidelity assessment measures, especially when a school or district coach facilitated the team's assessment process. When school teams complete the TFI, data collected can inform team action planning to improve implementation fidelity at specific tiers. In each administration, teams can assess implementation of their SWPBIS systems at any one, two, or all three tiers of the framework. We recommend school teams assess all three tiers of the TFI as an initial evaluation to determine whether schools are already implementing core features of SWPBIS and focus later assessments during the school year on individual tiers as a more frequent progress monitoring and evaluation tool. Although the TFI is relatively new, a large number of schools are already using it for assessment and to inform action plans for improvement. To understand how schools are using the TFI, this evaluation brief aims to answer the following research questions:

  1. How often do schools complete the TFI per year?
  2. How many tiers are assessed during each TFI administration?
  3. What are the average scores at each tier of the TFI?
  4. What subscales and items on the TFI have the highest and lowest scores?

Method

For this evaluation brief, the sample consisted of 8,640 schools that assessed at least one tier of the TFI during the 2016-17 school year. Schools must have completed at least one item in a tier to have that tier assessment included in the analyses. For research questions 3 and 4, if schools completed the TFI more than one time during the year, we used their TFI with the highest overall score to capture the most tiers assessed during a single administration.

School Characteristics

School demographic information was gathered from the National Center for Education Statistics (NCES) for the most recent school year with data (2013-14). Of the 8,640 schools that completed the TFI, 2,785 schools (32.2%) were located in cities, 2,861 (33.1%) were located in suburban areas, 1,017 (11.8%) were located in towns, 1,391 (16.1%) were located in rural areas, and 586 schools (6.8%) did not specify a regional area. A total of 63% of the schools were elementary (n = 5,443), 16.4% were middle schools (n = 1,417), 11.4% were high schools (n = 981), 2.1% were labeled as "other" (n = 185), and 7.1% of the schools (n = 614) did not identify grade levels served. The sample included 5,946 schools (68.8%) that were Title I eligible.

The TFI Fidelity Measure

The TFI includes 45 items scored by the school's PBIS team(s) as either 0 = not implementing, 1 = partially implementing, or 2 = fully implementing. There are 15 items measuring implementation at Tier 1, 13 items at Tier 2, and 17 items at Tier 3. For schools to be considered implementing at or above adequate implementation, an overall score of 70% or higher at Tier 1 is recommended (Mercer, McIntosh, & Hoselton, 2017). At this time, there has been no benchmark technically established for implementation fidelity for Tiers 2 & 3. As a result, users may be using the same 70% benchmark to establish a goal for implementation fidelity at those tiers as well.

In a recently published article evaluating the psychometric properties of the TFI, McIntosh and colleagues (2017) found the TFI to (a) show strong content validity in capturing items associated with SWPBIS implementation, (b) be highly reliable across raters and over time, (c) have strong internal consistency across tiers, and (d) produce scores strongly related to other measures of SWPBIS implementation. They also found the TFI to be efficient to administer and complete, averaging less than 15 minutes at each tier. In addition, Massar, McIntosh, and Mercer (in press) conducted further technical validation on the TFI and found the measure to be comprehensive in measuring different aspects of SWPBIS, including overall implementation, as well as implementation at individual tiers and subscales within tiers.

Results

How Often do Schools Complete the TFI per Year?

The 8,640 schools completed 13,470 TFI surveys during the 2016-17 school year (see Figure 1). A total of 5,241 of the schools (60.7%) completed the TFI once, 2,201 (25.5%) completed the TFI twice, 993 (11.5%) completed the TFI three times, 184 (2.1%) completed it four times, 15 (0.2%) completed it five times, 5 (0.1%) completed it six times, and 1 school (0.01%) completed the TFI seven times during that school year.

Figure 1. Number of Times the 8,640 Schools Completed the TFI

image: 0_Volumes_DataDocs_Hoon_Doc_PBIS_IV_PBIS_org_Ev___June2018_Figures_evalbrief_june2018_Figure1.png

How Many Tiers are Assessed During Each TFI Administration?

Of the 13,470 TFIs that were completed by the 8,640 schools, 12,958 (96%) of the TFIs assessed Tier 1, 7,713 (57%) assessed Tier 2, and 4,411 (33%) assessed Tier 3. In addition, 4,209 (31%) of the TFIs submitted assessed all three tiers during a single administration. Because schools could take any one, two, or all three tiers of the TFI multiple times throughout the year, we used each school's highest overall TFI score to determine the following information. Of the 8,640 schools, 8,467 (98%) assessed Tier 1 at some time during the school year; 5,343 (62%) assessed Tier 2, and 3,327 (39%) schools assessed Tier 3. In addition, 3,236 (37%) of the schools submitted a single assessment that included all three tiers at some time during the school year.

What are the Average Scores at Each Tier of the TFI?

For the 8467 schools that assessed Tier 1 at some time during the school year, the average score on the Tier 1 scale was 74% (SD = 22%). Of the 5,343 schools that assessed Tier 2, the average score on the Tier 2 scale was 69% (SD = 26%). Of the 3,327 schools that assessed Tier 3, the average Tier 3 score was 62% (SD = 28%). Individual item scores are presented in Figures 2, 3, and 4.

What Subscales and Items on the TFI have the Highest and Lowest Scores?

Tier 1: Teams. The average score for the 8,467 schools completing the Tier 1: Teams subscale was 78% (SD = 24%). Mean scores for individual items on this subscale were 1.60 for Team Operating Procedures and 1.54 for Team Composition.

Tier 1: Implementation. The average score for the Tier 1: Implementation subscale was 74% ( SD = 23%). Mean scores for individual items on this subscale ranged from 1.75 for Behavioral Expectations to 1.08 for Student/Family/Community Involvement.

Tier 1: Evaluation. The average score on the Tier 1: Evaluation subscale was 73% ( SD = 28%). Mean scores for individual items on this subscale ranged from 1.64 for Fidelity Data to 1.28 for Annual Evaluation.

Figure 2. Average Scores Across Items Within Tier 1 Subscales

image: 1_Volumes_DataDocs_Hoon_Doc_PBIS_IV_PBIS_org_Ev___June2018_Figures_evalbrief_june2018_Figure2.png

Average Scores for Tier 1 Items
TeamsScore
1.1 Team Composition 1.54
1.2 Team Operating Procedures 1.60
Interventions Score
1.3 Behavioral Expectations 1.75
1.4 Teaching Expectations 1.61
1.5 Problem Behavior Definitions 1.54
1.6 Discipline Policies 1.47
1.7 Professional Development 1.42
1.8 Classroom Procedures 1.55
1.9 Feedback and Acknowledgement 1.49
1.10 Faculty Involvement 1.37
1.11 Student/Family/Community Involvement 1.08
Evaluation Score
1.12 Discipline Data 1.60
1.13 Data-based Decision Making 1.31
1.14 Fidelity Data 1.64
1.15 Annual Evaluation 1.28

Tier 2: Teams. The average score for the 5,343 schools completing the Tier 2: Teams subscale was 76% ( SD = 25%). Mean scores for individual items on this subscale ranged from 1.73 for Team Composition to 1.37 for Request for Assistance.

Tier 2: Interventions. The average score on the Tier 2: Interventions subscale was 72% ( SD = 27%). Mean scores for individual items on this subscale ranged from 1.72 for Access to Tier 1 Supports to 1.24 for Professional Development.

Tier 2: Evaluation. The average score on the Tier 2: Evaluation subscale was 59.3% ( SD = 32%). Mean scores for individual items on this subscale ranged from 1.27 for Student Performance Data to 1.00 for Fidelity Data.

Figure 3. Average Scores Across Items Within Tier 2 Subscales

image: 2_Volumes_DataDocs_Hoon_Doc_PBIS_IV_PBIS_org_Ev___June2018_Figures_evalbrief_june2018_Figure3.png

Average Scores for Tier 2 Items
TeamsScore
2.1 Team Composition 1.73
2.2 Team Operating Procedures 1.50
2.3 Screening 1.50
2.4 Request for Assistance 1.37
Interventions Score
2.5 Options for Tier 2 Interventions 1.41
2.6 Tire 2 Critical Features 1.39
2.7 Practices Matched to Student Need 1.41
2.8 Access to Tier 1 Supports 1.72
2.9 Professional Development 1.24
Evaluation Score
2.10 Level of Use 1.26
2.11 Student Performance Data 1.27
2.12 Fidelity Data 1.00
2.13 Annual Evaluation 1.22

Tier 3: Teams. The average score for the 3,327 schools completing the Tier 3: Teams subscale was 70% (SD = 30%). Mean scores for individual items on this subscale ranged from 1.50 for Team Composition to 1.32 for Team Operating Procedures.

Tier 3: Resources. The average score on the Tier 3: Resources subscale was 64% (SD = 30%). Mean scores for individual items on this subscale ranged from 1.42 for Student/Family/Community Involvement to 1.02 for Professional Development.

Tier 3: Support Plans. The average score on the Tier 3: Support Plans subscale was 63% (SD = 33%). Mean scores for individual items on this subscale ranged from 1.58 for Access to Tier 1 and Tier 2 Supports to 1.11 for Formal and Natural Supports.

Tier 3: Evaluation. The average score on the Tier 3: Evaluation subscale was 54% (SD = 33%). Mean scores for individual items on this subscale ranged from 1.16 for Level of Use to 0.92 for Data System.

Figure 4 Average Scores Across Items Within Tier 3 Subscales

image: 3_Volumes_DataDocs_Hoon_Doc_PBIS_IV_PBIS_org_Ev___June2018_Figures_evalbrief_june2018_Figure4.png

Average Scores for Tier 3 Items
Teams Score
3.1 Team Composition 1.50
3.2 Team Operating Procedures 1.32
3.3 Screening 1.40
3.4 Student Support Team 1.34
Interventions Score
3.5 Staffing 1.40
3.6 Student/Family/Community Involvement 1.42
3.7 Professional Development 1.02
Support Plans Score
3.8 Quality of Life Indicators 1.14
3.9 Academic, Social, and Physical Indicators 1.34
3.10 Hypothesis Statement 1.21
3.11 Comprehensive Support 1.12
3.12 Formal and Natural Supports 1.11
3.13 Access to Tier 1 and Tier 2 Supports 1.58
Evaluation Score
3.14 Data System 0.92
3.15 Data-Based Decision Making 1.10
3.16 Level of Use 1.16
3.17 Annual Evaluation 1.15

Discussion

The TFI is the newest and one of the most comprehensive and empirically-validated measures of SWPBIS implementation fidelity (McIntosh et al., 2017; Mercer et al., 2017). Considering its short history of availability, and as more schools continue to focus on scaling up their implementation of Tier 2 and 3 supports, the number of schools using the TFI for assessment and to inform action plans is likely to increase.

At the beginning of the school year, or the first time the TFI is administered, we recommend that schools assess all three tiers with the assistance of a school or district/ regional coach. School teams are then encouraged to complete an action plan focusing on one to three areas for improvement. We recommend that progress monitoring be completed every three to four months for the school's tier of focus (e.g., Tier 1 for newly implementing schools). At each team meeting, teams are encouraged to review the action plan to determine whether progress was made in terms of implementing critical features with fidelity and whether other items should be added to the action plan for improvement. At the end of the school year, similar to the beginning, SWPBIS teams may re-assess all three tiers to evaluate overall implementation and progress made. Schools with multiple SWPBIS teams (e.g., Tier 1 team, Tier 2-3 team) completing the TFI might benefit from first entering their respective tier assessment scores on paper and later inputting the data into PBIS Assessment at one time after information has been gathered for all three tiers. This method provides one date for the completion of all tiers on the TFI and allows for the data to be compiled more efficiently for viewing reports and action planning.

Of the 8,640 schools in this evaluation brief, about 61% completed the TFI just once during the 2016-17 school year. In addition, only 37% of the schools submitted a TFI that included all three tiers being assessed at one time. There are likely several reasons that these school teams did not assess as often and as fully as recommended: (a) school districts or states may require only one assessment per year, (b) school teams need more training and coaching on when to schedule and administer the TFI, (c) school teams are already meeting a TFI benchmark and do not see the need to re-assess during the remainder of the school year, or (d) school teams do not have the time or resources to complete the TFI as often as recommended.

Recommendations for Teams: Tier 1

Tier 1 items with the lowest implementation were: 1.11 Student/Community/Family Involvement, 1.13 Data Based Decision Making, and 1.15 Annual Evaluation. Teams can improve upon these items by accessing professional development focusing on (a) providing concrete examples of ways SWPBIS teams can include student, family, and community involvement (e.g., solicit input from stakeholders at an annual review of and discussion about school-wide expectations); (b) using the annual evaluation as a snapshot of information that should be shared with stakeholders regarding the implementation of SWPBIS at Tier 1 and its effects on student behavior, academics, and attendance (e.g., 1-2 page newsletter); and (c) focusing on efficient ways to use data to assess fidelity of implementation and student outcomes for action planning, such as using the Team-Initiated Problem Solving (TIPS) process (Todd et al., 2017; Todd et al., 2011).

Recommendations for Teams: Tier 2

Tier 2 items with the lowest implementation were: 2.9 Professional Development, 2.12 Fidelity, and 2.13 Annual Evaluation. Teams can strengthen these items by emphasizing professional development focusing on (a) providing school teams with efficient ways to assess fidelity of Tier 2 interventions (e.g., Check-In/Check-Out Fidelity Checklist; Horner et al., 2004), (b) training Tier 2 teams on how to effectively and efficiently evaluate Tier 2 practices and systems being implemented in order to improve upon implementation and build capacity, and (c) providing easily accessible SWPBIS professional development at the school, district, or regional levels.

Recommendations for Teams: Tier 3

Items 3.7 Professional Development, 3.14 Data Systems, and 3.15 Data-Based Decision Making were among the lowest scored within the Tier 3 scale. These items could be improved with ongoing and accessible professional development for Tier 3 teams focusing on (a) establishing and consistently using data systems for progress monitoring of Tier 3 supports (e.g. Individual School Wide Information System [I-SWIS]) and (b) effectively using information from these data systems for decision making during bi-weekly or monthly Tier 3 team meetings to improve the quality of Tier 3 supports for students with intensive behavior support needs.

Conclusion

School teams are increasingly using the TFI to measure fidelity of SWPBIS implementation. The TFI is a comprehensive SWPBIS fidelity measure that assesses critical SWPBIS features across all three tiers. The aim of this evaluation brief was to (a) demonstrate how and how often a large number of schools are using the TFI for evaluation and progress monitoring and (b) to identify and offer suggestions to school teams on how to improve implementation of core features.

References

Algozzine, R. F., Barrett, S., Eber, L., George, H., Horner, R. H., Lewis, T. J., . . . Sugai, G. (2014). School-wide PBIS Tiered Fidelity Inventory. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. www.pbis.org.

Bradshaw, C. P., Koth, K., Bevans, K. B., Ialongo, N., & Leaf, P. J. (2008). The impact of school-wide positive behavioral interventions and supports on the organizational health of elementary schools. School Psychology Quarterly, 23, 462-473.

Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133-148. doi:10.1177/1098300709334798

Horner, R. H., Todd, A. W., Filter, K. J., McKenna, M. K., Benedict, E. A., & Hawken, L. S. (2004). Check-In/Check-Out Fidelity Checklist. Eugene, OR: University of Oregon, Educational and Community Supports

Massar, M. M., McIntosh, K., & Mercer, S. H. (in press). Factor validation of a fidelity of implementation measure for social behavior systems. Remedial and Special Education.

McIntosh, K., Massar, M. M., Algozzine, R. F., George, P. G., Horner, R. H., Lewis, T. J., & Swain-Bradway, J. (2017). Technical adequacy of the SWPBIS tiered fidelity inventory. Journal of Positive Behavior Interventions, 19, 3-13. doi:10.1177/1098300716637193

Mercer, S. H., McIntosh, K., & Hoselton, R. (2017). Comparability of fidelity measures for assessing tier I school- wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 19, 195-204. doi:10.1177/1098300717693384

Todd, A. W., Algozzine, R. F., Horner, R. H., Preston, A. I., Cusumano, D., & Algozzine, K. (2017). A descriptive study of school-based problem-solving. Journal of Emotional and Behavioral Disorders. doi:10.1177/1063426617733717

Todd, A. W., Horner, R. H., Newton, J. S., Algozzine, R. F., Algozzine, K. M., & Frank, J. L. (2011). Effects of team-initiated problem solving on decision making by schoolwide behavior support teams. Journal of Applied School Psychology, 27, 42-59. doi:10.1080/15377903.2011.540510