Is BoQ Stable?

Stability in Variant Administration Methods of the School-Wide PBS Benchmarks of Quality (BoQ)

Karen Elfner Childs, Heather Peshak George, Don Kincaid

The School-Wide PBS Benchmarks of Quality is now being used by thousands of schools across the country.  One of the most frequently asked questions regarding the BoQ has been related to the procedures used for scoring.  As a result, we decided to investigate the following question, “Is the BoQ stable and will similar scores result if administered in the method used in the validation study versus other methods?”

The Benchmarks of Quality was developed and validated to be administered by the Coach (otherwise known as Facilitator) utilizing the Scoring Guide to complete the Scoring Form with feedback from members through the Team Member Rating (which was designed to be a more efficient means of considering the Team’s estimation of implementation).  However, many PBIS Teams and some statewide PBIS Projects desired access and input using the more comprehensive Scoring Guide/Scoring Form.  Other than being a more arduous process, there did not intuitively seem to be any inherent harm in a variant administration such as the full team using the Scoring Guide to reach consensus on each item’s score.  

Method

A statistical analysis was conducted with data from 398 administrations of the BoQ where 57% of schools used the validated method and 43% used some variant of the validated method.  The first analysis conducted was an unpaired t-test comparing BoQ factor components scores between the two methods.  
An unpaired t-test was used to compare BoQ factor component response scores between the two methods.  A Cochran-Mantel-Haenszel Statistics test was used to determine the general association between each BoQ item and the methods of administration.  The significant level α=0.05 had been adjusted to α=0.05/53 ≈0.001 since we test 53 BoQ question items simultaneously.

Results

The t-test results show that there is no significant difference on any factors of the BoQ between the methods of administration. 

Factors

Mean Difference Mean1 - Mean2

Equality of Variance

DF

t Value

Pr > |t|

boq1

0.325

Equal

395

0.36

0.7195

boq2

0.017

Equal

395

0.03

0.9735

boq3

0.0815

Equal

395

0.34

0.7319

boq4

0.1047

Equal

395

0.43

0.6642

boq5

0.082

Equal

395

0.37

0.715

boq6

0.0699

Equal

395

0.89

0.3731

boq7

0.0866

Unequal

*184

1.54

0.1247

boq8

0.0375

Equal

395

0.53

0.5969

boq9

0.0289

Unequal

*187

0.79

0.4319

boq10

0.0183

Equal

395

0.3

0.7606

The Chi-square tests also showed no significant difference between the methods of administration with only 2 of the 53 items receiving a significance level of less than α=0.05 (BoQ Question 14; .0494 and BoQ Question 33; 0.0185).

Discussion

The BoQ was found to be a valid instrument even when it is administered in diverse methods adding confidence to the utility of the BoQ.  It should be noted that each variation of administration involved use of the Scoring Rubric in some phase of the scoring process.