What can a supervisor’s rating tell us?
What does "competent" mean to a supervisor? Dimmick, Callahan, and Cox (2023) compared supervisors’ ratings of trainees with two things a clinic could observe: whether clients returned after their first appointment and how their symptoms changed.
The Study
Published in Training and Education in Professional Psychology, the study examined 92 trainees and 192 clients at a psychology training clinic. The researchers used the Practicum Evaluation Form (PEF), a structured supervisor rating of trainee competency, and linked those ratings to actual client outcomes measured by the OQ-45.2, a widely used therapy outcome measure.
The Findings
- Each one-point increase in the rating for professional values, attitudes, and behaviors was associated with a 42% lower relative risk that a client would not return after the first session (Risk Ratio = 0.58, 95% CI: 0.37 to 0.88)
- Competency ratings accounted for more than 10% of the variance in client symptom change in this sample.
- Foundational competencies (research skills and interpersonal skills) accounted for 7.4% of outcome variance
- Functional competencies (assessment, supervision, consultation) accounted for 4.5%
What this adds to a knowledge exam
The Practicum Evaluation Form captures parts of clinical work a multiple-choice answer cannot show, including how a trainee works with a client in session.
Reliability in this study
For the Practicum Evaluation Form, the researchers reported person reliability of .99 and item reliability of .92. These estimates describe how the ratings performed in this sample.
References
Dimmick, A. A., Callahan, J. L., & Cox, R. J. (2023). Further validation of competency assessment: The practicum evaluation form. Training and Education in Professional Psychology, 17(3), 269–276.
Wampold, B. E., & Imel, Z. E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work (2nd ed.). Routledge.