Evaluating Programs

A pilot occupational therapy community skills group created to enable individuals living with severe, chronic mental illness to have better outcomes. The program was intended to enable the clients to identify and improve their performance in personally important goals in the areas of self-care and home management including skills in cooking, healthy eating, budgeting time, socializing, shopping and budgeting. The therapists were in need of a tool that would both support the identification of personally meaningful goals and their evaluation (Brown, Shiels, & Hall, 2001).

One of the primary purposes for which the COPM was designed was for use in program evaluation, as an outcome measure. However this application involves a number of changes from the usual way of thinking about occupational therapy practice. First, it requires that we think in terms of program goals rather than individual therapy goals. Program goals set out targets and expectations for a group of clients who typically share a particular diagnosis or problem. Clients are referred to the program because they have these problems, and there is the expectation that by participating in the program, they can more effectively and efficiently address their problems than would be possible in individual therapy. When we use the word program in occupational therapy, we usually mean the regime of therapy that we have designed for a particular client. However, in program evaluation, the meaning of the word program is more specific. It refers to a set of goals, a set of activities and a set of resources (both human and non-human), all aimed at solving a shared health problem.

Second, the programmatic approach requires that success be measured not on the basis of individual goal attainment, but rather on the basis of average change for all participants in the program. Thus, there is the expectation that some individuals will do better in the program than others, and in fact, it is possible that some may not improve at all. However, if the average change is positive, then the program may be deemed a success.

Third, it is important for program evaluation to be clear about the criterion for success – how much average change do we need to see to believe that the program is worthwhile? There are two ways of asserting that a margin of change is meaningful — statistical and clinical significance. Statistical significance tells us that the average change seen among members of the program is unlikely to have happened by chance, and clinical significance tells us the average change seen has meaningful implications for function. Both of these are important indicators of a successful program, and they should not be confused with one another. Previous research has provided evidence that a change of 2 or more points is clinically significant (Law et al., 2014).

The literature offers several useful examples of program evaluations where average change in COPM scores (performance and satisfaction) demonstrated the worth or value of an occupational therapy program. In the case example above, using the COPM allowed Brown and colleagues (2001) to show that clients with chronic mental illness were able to identify goals and improve their performance and/or satisfaction in a number of areas of self-care, home management, personal safety and social functioning. In another example, Persson and colleagues (2004) used the COPM to evaluate changes in occupational performance for clients who participated in a pain management program. Finally, Tam and colleagues (2005) used the COPM measured changes after intervention using an assistive technology program to develop writing skills.

Brown, F., Shiels, M., & Hall, C. (2001). A pilot community living skills group: An evaluation. British Journal of Occupational Therapy, 64, 144 – 150.

Law, M., Baptiste, S., Carswell, A., McColl, M. A., Polatajko, H., & Pollock, N. (2014). Canadian Occupational Performance Measure (5th ed.). Ottawa: CAOT Publications ACE.

Persson, E., Rivano-Fischer, M., & Eklund, M. (2004). Evaluation of changes in occupational performance among patients in a pain management program. Journal of Rehabilitation Medicine, 36(2), 85-91.

Tam, C., Archer, J., Mays, J., & Skidmore, G. (2005). Measuring the outcomes of word cueing technology. Canadian Journal of Occupational Therapy, 72(5), 301-308.

Leave a Reply

Your email address will not be published. Required fields are marked *