American Journal of Evaluation current issue

American Evaluation Association: Guiding Principles for Evaluators

2009-02-19

Evaluators' Decision Making: The Relationship Between Theory, Practice, and Experience

Tourmen, C. — 2009-02-19

How do evaluation practitioners make choices when they evaluate a program? What function do evaluation theories play in practice? In this article, I report on an exploratory study that examined evaluation practices in France. The research began with observations of practitioners' activities, with a particular focus on the phases of evaluation design. The purpose of the study was to examine the logic underlying their choices and the role that evaluation theory played in those choices. In this study, I used activity theory and ergonomics methods. I highlight various ways in which working evaluators make evaluation choices in real situations, explained by different sets of conceptual resources they hold related to their level of experience. Finally, I show how this research contributes toward clarifying the role of theories in practice and enriching an evaluation theory

Gilding the Outcome by Tarnishing the Past: Inflationary Biases in Retrospective Pretests

Taylor, P. J., Russ-Eft, D. F., Taylor, H. — 2009-02-19

We tested for inflationary bias introduced through retrospective pretests by analyzing traditional pretest, retrospective pretest, and posttest evaluation data collected on a first-line supervisory leadership training program, involving 196 supervisors and their subordinates, across 17 organizational settings. Retrospective pretest ratings by both trained (supervisors) and untrained (subordinates) respondents were significantly lower than traditional pretest ratings, resulting in substantially inflated training effect sizes when posttests were compared with retrospective pretests rather than with traditional pretests. Further analysis revealed evidence of both respondents' application of an implicit theory of change (i.e., assumption that posttraining scores should generally be higher than pretraining scores) and a tendency to rate their own improvement as greater than that of others. Implications for program evaluation are discussed.

Treatment Fidelity in Multisite Evaluation: A Multilevel Longitudinal Examination of Provider Adherence Status and Change

Zvoch, K. — 2009-02-19

Program implementation data obtained from the repeated observation of teachers delivering one of two early childhood literacy programs to economically disadvantaged students in a large southwestern school district were analyzed to estimate protocol adherence levels at the onset of the intervention as well as the change in adherence over the intervention period. Application of multilevel growth models to the classroom observation data revealed that fidelity to program protocol varied within and between treatment sites during the initial observation and over time. An exploratory examination of select teacher, classroom, and site characteristics indicated that the background characteristics of teachers and contextual factors in the treatment environment were associated with the fidelity outcomes. These results provide some insight into the range of factors that are associated with protocol adherence and highlight the challenge of achieving and maintaining fidelity to a treatment intervention that is delivered by multiple providers over multiple treatment sites.

The Development of Theory-Driven Evaluation in the Military: Theory on the Front Line

Williams, A. P., Morris, J. C. — 2009-02-19

The use of theory-driven evaluation is an emerging practice in the military—an aspect generally unknown in the civilian evaluation community. First developed during the 1991 Gulf War and applied in both the Balkans and Afghanistan, these techniques are now being examined in the North Atlantic Treaty Organisation (NATO) as a means to evaluate the effects of military operations in complex, asymmetric conflict environments. In spite of these practices, theory-driven evaluation in the military is still in the developmental stages. This article traces the development to date of theory-driven evaluation in NATO and assesses its strengths and weaknesses in the military context. We conclude that a cross-pollination of ideas between military and civilian evaluators is urgently needed to improve the quality and effectiveness of military evaluation.

Adding a Time-Series Design Element to the Success Case Method to Improve Methodological Rigor: An Application for Nonprofit Program Evaluation

Coryn, C. L. S., Schroter, D. C., Hanssen, C. E. — 2009-02-19

Brinkerhoff's Success Case Method (SCM) was developed with the specific purpose of assessing the impact of organizational interventions (e.g., training and coaching) on business goals by analyzing extreme groups using case study techniques and storytelling. As an efficient and cost-effective method of evaluative inquiry, SCM is attractive in other contexts as well, although few examples of such uses are to be found in the published literature. However, modifications of the SCM concept and design are sometimes necessary for implementing the approach in nonprofit environments where business goals are not necessarily an explicit objective. This method note demonstrates how SCM was modified and extended to a social service context, in which the program evaluated was aimed at reducing chronic homelessness and unemployment. Modifications included defining success in a nonprofit setting and adding a time-series element to the design features of traditional SCM to increase methodological rigor.

Do Self-Assessments Work to Detect Workshop Success?: An Analysis of Argument and Recommendation by D'Eon et al

Lam, T. C. M. — 2009-02-19

D'Eon et al. concluded that change in performance self-assessment means from before to after a workshop can detect workshop success in their and other situations. In this commentary, their recommendation is refuted by showing that (a) self-assessments with balanced over- and underestimations are still biased and should not be used to evaluate workshops, even though the means of self-assessments and criterion measures are artificially equal; (b) participants' performance should not be attributed directly to training, even if the self-assessments are psychometrically valid and obtained prior to the workshop as well; (c) self-assessment findings should not be generalized to other situations without further analysis and caution, even if the participants' performance can be attributed to training. For clarifying the recommendation by D'Eon et al. to use ``aggregated self-assessments'' to evaluate workshops, analysis of multilevel data is explained and discussed. Finally, nine rules of thumb in using self-assessments for evaluating training are provided.

Call for Nominations for Editorship of the American Journal of Evaluation (2010--2012)

2009-02-19

AJE Contribution Categories

2009-02-19