Date of Award
2013
Document Type
Dissertation
Degree Name
Ph.D. in Counselor Education
Department
Leadership and Counselor Education
First Advisor
Kevin Stoltz
Second Advisor
Marc Showalter
Third Advisor
Matthew Reysen
Relational Format
dissertation/thesis
Abstract
The purpose of this study was to explore the effect of instrument-specific rater training on interrater reliability (IRR) and counseling skills performance differentiation. Strong IRR is of primary concern to effective program evaluation (McCullough, Kuhn, Andrews, Valen, Hatch, & Osimo, 2003; Schanche, Nielsen, McCullough, Valen, & Mykletun, 2010) and counselor education (Baker, Daniels, & Greeley, 1990; Jennings, Goh, Skovholt, & Banerje-Steevens, 2003; Lepkowski, Packman, Smaby, & Maddux, 2009). The ability to differentiate between low and high performances of counseling skills is central to informing the classroom instruction of counseling students and the supervision of early clinical experiences (Byrne & Hartley, 2010; Fitch, Gillam, & Baltimore, 2004; Paladino, Barrio-Minton, & Kern, 2011). Participants were randomly assigned to one of four groups defined by whether they received instrument-specific training and the performance level of the counseling skills they assessed. Data was collected using the Universal Counseling Skills Assessment (UCSA) administered traditionally and through the Dynamic Scoring Interface (DSI). The researcher used a 2 X 2 factorial ANOVA, independent samples t-tests, intraclass correlation coefficients, and Fisher’s r to z transformations to analyze the data’s validity across the groups and reliability within the groups. Results that brief instrument-specific training and a structure scoring procedure can significantly strengthen IRR. The results of the analyses are discussed within the context of their implications for counselor education and future research possibilities.
Recommended Citation
Meacham, Paul Douglas, "The Effect Of Instrument-Specific Rater Training On Interrater Reliability And Counseling Skills Performance Differentiation" (2013). Electronic Theses and Dissertations. 463.
https://egrove.olemiss.edu/etd/463