American Journal of Evaluation recent issues

From the Editor

Miller, R. L. — Tue, 10 Nov 2009 13:20:48 PST

A Review and Synthesis of Current Research on Cross-Cultural Evaluation

Chouinard, J. A., Cousins, J. B. — Tue, 10 Nov 2009 13:20:48 PST

As a fairly new and emergent construct, there remain many gaps in our knowledge about how to integrate notions of culture and cultural context into evaluation theory and practice, as well as gaps in our knowledge about how to conduct and implement evaluations in immigrant and indigenous communities. In this article, the authors provide a comprehensive review of the empirical literature on evaluations conducted in cultural communities, with an emphasis on the relationship between evaluators and stakeholders in the cross-cultural program context. The analysis of the literature selected for review leads to the development of a theoretical framework describing the inter-related and multi-textual dimensions (relational, ecological, methodological, organizational and personal) that interweave throughout the evaluation, and that ultimately inform the relationship between evaluators and stakeholders in the cross-cultural program context. The article concludes with an agenda for future research.

Finding the Impact in a Messy Intervention: Using an Integrated Design to Evaluate a Comprehensive Citywide Health Initiative

Weitzman, B. C., Mijanovich, T., Silver, D., Brecher, C. — Tue, 10 Nov 2009 13:20:48 PST

This article uses the evaluation of the Robert Wood Johnson Foundation’s (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites’ plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.

Six Guiding Principles for Evaluating Mode-2 Strategies for Sustainable Development

Tue, 10 Nov 2009 13:20:48 PST

A new generation of public programs emerges, which specifically addresses complex societal problems we witness today. Programs for these types of complex issues—in this article, we consider more closely the challenge of sustainable development—are characterized by emergent design, learning processes between diverse actors, and adaptive management. Managers of these kinds of programs have new demands for evaluation and evaluators. This article describes prevailing evaluation methods for sustainable development (progress assessment, goal-oriented program evaluation, and program theory evaluation) and the challenges they meet when confronted with the complexity of designing and conducting systemic intervention programs for sustainable development. The evaluation framework that we propose offers guiding principles to assist evaluators in evaluating complex programs.

The Role of Evaluation in Research--Practice Integration Working Toward the ''Golden Spike''

Brown Urban, J., Trochim, W. — Tue, 10 Nov 2009 13:20:48 PST

Program evaluation and planning is at the heart of efforts to integrate the domains of practice and research. Traditionally, research and practice have operated in independent spheres with practitioners focused on the implementation of programs that affect individual behavior and researchers focused on the development and testing of theory. Evidence-based practice (EBP), practice-based evidence, and translational research have attempted to unite these worlds, and although significant advances have been made, there is a continued need to find mechanisms that enable a seamless connection between knowledge generation and application. We propose a method that builds on the traditions of theory-driven evaluation, logic modeling, and systems science and uses evaluation and program planning as the bridging mechanism between research and practice. Included in this approach are methods that aid in the explicit expression of implicit theories, management of evaluation resources, and linkage of program theory and evaluation measures to a research evidence base.

Exploring the Intervention-- Context Interface: A Case From a School-Based Nutrition Intervention

Bisset, S., Daniel, M., Potvin, L. — Tue, 10 Nov 2009 13:20:48 PST

It has been acknowledged for several decades that programs interact with context. The nature of this interactivity, and how it defines a program, has not been adequately addressed. We view this lacuna as a function of the dominant theoretical perspectives guiding knowledge of program operations. We propose the actor-network theory (ANT) and its conceptual apparatus, the sociotechnical network, as suitable for guiding the acquisition of general knowledge on program operations. We tested this proposition with an instrumental case study of health professional practices during the implementation of a nutrition program into an elementary school setting. Data collection and analysis were guided by the ANT. Data were derived from semistructured interviews completed with six health professionals (nutritionists). Analysis procedures focused on the nutritionists’ collective representation of the microprocesses by which they aimed to build a sociotechnical network of alliances with educational stakeholders. Findings identified nutritionists as preoccupied with three overarching goals during the implementation of the nutrition program, whereby goals were found to take form interactively with the interests of the program participants (primarily students) and stakeholders (primarily teachers). Nutritionists strategically translated program components as a means of negotiating with participants and stakeholders. The findings of this study support the theoretical proposition that program implementation is a process of expanding a sociotechnical network. Beyond simply reaffirming that programs do indeed adapt to context, we interpret this adaptation through the lens of a social theory that suggests why and how adaptation is an inevitable component of program implementation.

Evaluators' Reflections on the Ethical Implications of Their Early Experiences

Cooksy, L. J. — Tue, 10 Nov 2009 13:20:48 PST

What kinds of ethical dilemmas do people who are new to the profession of evaluation face? Are they different from the challenges faced by more experienced evaluators? What strategies do they use to resolve them? These questions were the starting point for the essays included in this issue of the Ethical Challenges section. The essays were written by people who responded to information sent to them by faculty who were in turn responding to the following request from the section editor: ‘‘As part of the American Journal of Evaluation’s continued effort to help young professionals publish, I am putting together an Ethical Challenges section that will feature challenges faced by new evaluators. I am writing to ask for your help in soliciting essays from advanced graduate students or recent graduates about ethical challenges that they have faced. Ideally, the author will describe a specific dilemma (using the American Evaluation Association’s [AEA] Guiding Principles as a reference point) and then discuss how his or her newness in the profession affected its evolution and/or resolution.’’ Of the essays received in response to this request, three were selected.

Methodological Changes and Respecting Stakeholder Dignity

Kallemeyn, L. M. — Tue, 10 Nov 2009 13:20:48 PST

An Evaluation of an Alternative Teacher Certification Program A Matter of Trust

Grubbs, S. T. — Tue, 10 Nov 2009 13:20:48 PST

Evaluation Team Dynamics: Intragroup Ethical Challenges

Urias, D. — Tue, 10 Nov 2009 13:20:48 PST

Applied Research Consultants (ARC): A Vertical Practicum Model of Training Applied Research

Nadler, J. T., Cundiff, N. L. — Tue, 10 Nov 2009 13:20:48 PST

The demand for highly trained evaluation consultants is increasing. Furthermore, the gap between job seekers’ evaluation competencies and job recruiters’ expectations suggests a need for providing practical training experiences. A model using a vertical practicum (advanced students assisting in the training of newer students) is suggested as an ideal training solution. Applied Research Consultants (ARC) is a 2-year vertical practicum conceptually built around a student-run consulting firm. ARC provides opportunities for graduate students to apply what they have learned in the classroom to real-world situations, thereby gaining invaluable experience. ARC’s history, structure, and successes are discussed in the context of suggested goals in teaching, training, and hiring competent evaluation consultants. Additional methods of providing practical training in evaluation and consultation including class projects, mentored projects, internships, and practicum programs are discussed. The resources, institutional commitment, and possible limitations of a vertical practicum training model are also discussed.

Book Review: V. Sue and L. Ritter Conducting Online Surveys Thousand Oaks, CA: Sage, 2007. 208 pp. $39.95

Hartmann, D. — Tue, 10 Nov 2009 13:20:48 PST

Book Review: Debra J. Holden & Marc Zimmerman, (2009). A Practical Guide to Program Evaluation Planning. Thousand Oaks, CA: Sage. $29.95, 184 pp

Evergreen, S. D. H. — Tue, 10 Nov 2009 13:20:48 PST

Acknowledgement of Reviewers

Tue, 10 Nov 2009 13:20:48 PST

American Evaluation Association: Guiding Principles for Evaluators

Wed, 05 Aug 2009 14:02:40 PDT

Reconceptualizing Evaluator Roles

Skolits, G. J., Morrow, J. A., Burr, E. M. — Wed, 05 Aug 2009 14:02:40 PDT

The current evaluation literature tends to conceptualize evaluator roles as a single, overarching orientation toward an evaluation, an orientation largely driven by evaluation methods, models, or stakeholder orientations. Roles identified range from a social transformer or a neutral social scientist to that of an educator or even a power merchant. We argue that these single, broadly construed role orientations do not reflect the multiple roles evaluators actually assume as they complete the activities encompassing an external evaluation. In contrast to the current literature, this article suggests that typical evaluation activities create functional demands on evaluators, and that evaluators respond to these demands through a limited number of specified evaluator roles. This depiction of a set of specific multiple evaluator roles, generated in response to particular evaluation activities and their associated demands, has implications regarding how evaluation is conceptualized, practiced, and studied. This article concludes with a discussion of these implications.

From Program to Network: The Evaluator's Role in Today's Public Problem-Solving Environment

Benjamin, L. M., Greene, J. C. — Wed, 05 Aug 2009 14:02:40 PDT

Today’s public policy discussions increasingly focus on how networks of public and private actors collaborate across organizational, sectoral, and geographical boundaries to solve increasingly complex problems. Yet, many of evaluation’s key concepts, including the evaluator’s role, assume an evaluand that is programmatically or organizationally defined and bounded. This article explores the implications of this changing public policy environment for the evaluator’s role by examining one case: an evaluand that was a loose collaboration of four individuals in dispersed organizations working to reframe public policy and to change professional practice in early care and education. We describe this evaluand and the dimensions of it that challenged our evaluative role. We conclude by suggesting an alternative conception of the evaluator’s role that can serve evaluators in this changing policy environment.

Using Mixed-Method Design and Network Analysis to Measure Development of Interagency Collaboration

Cross, J. E., Dickmann, E., Newman-Gonchar, R., Fagan, J. M. — Wed, 05 Aug 2009 14:02:40 PDT

In recent years, there has been increasing attention to the importance of interagency collaboration for improving community well-being, environmental and public health, and educational outcomes. This article uses a mixed-methods approach including network analysis to examine the changes in interagency collaboration in one site funded by the Safe Schools/Healthy Students Initiative (SS/HS). Results of the evaluation demonstrated that although intraproject collaboration peaked in the middle of the grant and began to decline during the last year, interagency collaborations continued to grow during the last year of the grant. These results illustrate how network data can easily be collected and used to assess the development of interagency relationships.

Toward Accurate Measurement of Participation: Rethinking the Conceptualization and Operationalization of Participatory Evaluation

Daigneault, P.-M., Jacob, S. — Wed, 05 Aug 2009 14:02:40 PDT

While participatory evaluation (PE) constitutes an important trend in the field of evaluation, its ontology has not been systematically analyzed. As a result, the concept of PE is ambiguous and inadequately theorized. Furthermore, no existing instrument accurately measures stakeholder participation. First, this article attempts to overcome these problems by using the works of G. Goertz (2006) and J. Gerring (1999) on concept formation and evaluation to assess current conceptualizations of PE. Second, an amended version of the framework developed by J. B. Cousins and E. Whitmore (1998) is proposed as an alternative to current conceptualizations. This amended framework is then operationalized and adapted in a participation measurement instrument. The proposed conceptualization and instrument have the potential to contribute to the production of sound empirical knowledge about evaluation and to reflections on PE practice.

The Validation of the Evaluation Involvement Scale for Use in Multisite Settings

Toal, S. A. — Wed, 05 Aug 2009 14:02:40 PDT

Evaluation researchers and practitioners acknowledge that involving stakeholders in the planning and implementation of an evaluation increases buy-in, understanding, and use. With the recent increase in multi-site evaluations of large federal programs, evaluators must think differently about how to encourage meaningful collaboration by stakeholders. To date, there has been no published measure of such involvement, despite recent calls for more systematic, replicable research. The purpose of this study was to validate the Evaluation Involvement Scale for use in multi-site evaluations. Between the fall of 2006 and spring of 2007 data were collected from an electronic survey and phone interviews of evaluators and principal investigators of four National Science Foundation program evaluations. Using Messick’s unitary concept of validity as a framework, theoretical, statistical, and rational evidence is provided to support the use of the Evaluation Involvement Scale to measure stakeholder involvement in multi-site evaluations.

The Evaluator's Role in Recommending Program Closure: A Model for Decision Making and Professional Responsibility

Eddy, R. M., Berry, T. — Wed, 05 Aug 2009 14:02:40 PDT

Evaluators face challenges when programs consistently fail to meet expectations for performance or improvement and consequently, evaluators may recommend that closing a program is the most prudent course of action. However, the evaluation literature provides little guidance regarding when an evaluator might recommend program closure. Given evaluators’ ethical responsibilities to the public good coupled with the inadequacy of methods currently used to justify program closure, we posit the development of a heuristic model designed to identify the point at which a program might be recommended for closure. The heuristic model depends on whether implementation factors are under the control of program administrators (i.e., flexible factors) or out of their control (i.e., immutable factors) and are potentially moderated by facilitating or inhibitory characteristics (e.g., political advocacy, community need, length of time in operation, performance quality). By delineating whether a poorly functioning program could be substantially improved, evaluators should not consider recommendation for or support of a program closure a personal or professional defeat, but rather an opportunity to exercise their professional responsibility to the evaluation profession and the public good. Caveats regarding the political context of an evaluation and the information required to make heuristic judgments about program closure are also discussed.

Research on Evaluation Use: A Review of the Empirical Literature From 1986 to 2005

Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., Volkov, B. — Wed, 05 Aug 2009 14:02:40 PDT

This paper reviews empirical research on the use of evaluation from 1986 to 2005 using Cousins and Leithwood’s 1986 framework for categorizing empirical studies of evaluation use conducted since that time. The literature review located 41 empirical studies of evaluation use conducted between 1986 and 2005 that met minimum quality standards. The Cousins and Leithwood framework allowed a comparison over time. After initially grouping these studies according to Cousins and Leithwood’s two categories and twelve characteristics, one additional category and one new characteristic were added to their framework. The new category is stakeholder involvement, and the new characteristic is evaluator competence (under the category of evaluation implementation). Findings point to the importance of stakeholder involvement in facilitating evaluation use and suggest that engagement, interaction, and communication between evaluation clients and evaluators is critical to the meaningful use of evaluations.

Analyzing the Assumptions of a Policy Program: An Ex-ante Evaluation of ''Educational Governance'' in the Netherlands

Janssens, F. J. G., de Wolf, I. F. — Wed, 05 Aug 2009 14:02:40 PDT

This article demonstrates the value of theoretical ex-ante evaluation of policy programs. We show how a specific policy might not achieve its objectives and illustrate the elements of the policy, which need improvement. These conclusions are based on a reconstruction and evaluation of the theory behind a policy program that aims to increase productivity of schools in the Netherlands, the policy of ‘‘educational governance.’’ We chose this program for its interesting combination of measures on internal quality assurances as well as accountability and supervision. The evaluation clearly shows a flaw in the theory underlying the policy program, which threatens its potential effectiveness. Furthermore, we show that there is a risk of contrary and incompatible interests among actors, as well as some practical reasons why the program might not work. Adjustment of the program is recommended before implementation.

Book Review: Spaulding, D. T. (2008). Program Evaluation in Practice: Core Concepts and Examples for Discussion and Analysis. San Francisco: Jossey-Bass, 176 pp., $40.00

Wallace, T. L. — Wed, 05 Aug 2009 14:02:40 PDT

Retrospective Deflation vs. Situational Inflation

Hawkins, A. J. — Wed, 05 Aug 2009 14:02:40 PDT

AJE Contribution Categories

Wed, 05 Aug 2009 14:02:40 PDT

American Evaluation Association: Guiding Principles for Evaluators

Mon, 13 Jul 2009 14:52:40 PDT

The Strength of the Methodological Warrants for the Findings of Research on Program Evaluation Use

Brandon, P. R., Singh, J. M. — Mon, 13 Jul 2009 14:52:40 PDT

Considerable research has been conducted on the use of the findings of program evaluation, but little, if any, attention has been paid to the soundness of the methods of this research. If the methods are not sound or not well described in the research, the strength of the conclusions of the research is unknown. The authors examine the empirical studies reported in five widely cited reviews of the literature on program evaluation use and summarize the types of methods used, the professions for which they were conducted, and the extent to which they met two basic criteria for soundness. It was concluded that, by and large, the body of studies does not adequately address the criteria. With care, the findings of the research literature on the use of evaluations might provide guidance for evaluators, but they lack sufficient scientific credibility for other uses.

Evaluation Use: Results From a Survey of U.S. American Evaluation Association Members

Fleischer, D. N., Christie, C. A. — Mon, 13 Jul 2009 14:52:40 PDT

This paper presents the results of a cross-sectional survey on evaluation use completed by 1,140 U.S. American Evaluation Association members. This study had three foci: evaluators' current attitudes, perceptions, and experiences related to evaluation use theory and practice, how these data are similar to those reported in a previous study conducted by Preskill and Caracelli (1997), and to identify characteristics that distinguish high endorsers of use items from others in the sample. Findings suggest a fair level of agreement on several dimensions of use including stakeholder involvement, factors that influence use, and the varied roles of the evaluator. Logistic regression results indicated that external evaluators were less likely to be high item endorsers while those who reported being members of the Evaluation Use Topical Interest Group of the American Evaluation Association were more likely to be high item endorsers.

Emphasizing Cultural Competence in Evaluation: A Process-Oriented Approach

Botcheva, L., Shih, J., Huffman, L. C. — Mon, 13 Jul 2009 14:52:40 PDT

This paper describes a process-oriented approach to culturally competent evaluation, focusing on a case study of an evaluation of an HIV/AIDS educational program in Bulawayo, Zimbabwe. We suggest that cultural competency in evaluation is not a function of a static set of prescribed steps but is achieved via ongoing reflection, correction, and adaptation. The aim of these processes is to attain the ``best fit'' possible between evaluation goals, methods, and cultural context. Three main ingredients in a process-oriented approach to culturally competent evaluation are discussed: collaboration, reflective adaptation, and contextual analysis. In addition, since evaluators face constraints set by funders and other stakeholders, we suggest that cultural competence is best viewed as a continuum. An evaluator's goal should be to ``move across the continuum'' in order to achieve the highest level of cultural competency possible given the unique parameters of every evaluation.

Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design

Moore, D., Tananis, C. A. — Mon, 13 Jul 2009 14:52:40 PDT

The Pennsylvania Governor's School for International Studies is an intensive summer program designed to give talented high school students a challenging introduction to the study of international affairs. One focus of the evaluation seeks to understand the effect of the program on the students' perception of their knowledge concerning core issues. Across the long history of the program, a variety of measures were used (and subsequently discarded) to assess changes in knowledge and perception of competence. Four years ago the program instituted a retrospective pre-post design. Results from these years, indicate that these students have consistently overestimated their pre-test understanding of core competencies emphasized in the program and that they seem better able to assess their knowledge gains and their initial inflated sense of knowledge as a result of the program. This article offers an overview of the development, application, use and analysis of a retrospective pre-post instrument to address response shift bias.

A Reliability Analysis of Goal Attainment Scaling (GAS) Weights

Marson, S. M., Guo Wei, , Wasserman, D. — Mon, 13 Jul 2009 14:52:40 PDT

Goal attainment scaling (GAS) has been considered to be one of the most versatile and appealing evaluation protocols available for human services. Aspects of the protocol that make the method so appealing to practitioners—that is, collaboratively working with individual clients to identify and assign weights to goals they will work to achieve—have produced critical psychometric challenges that have threatened the method's acceptance by funders and researchers. This interrater reliability study of weighted goals contributes to their psychometric validation and therefore to the continued use of a methodology so attractive to practitioners. The subjective clinical impressions of 43 students trained in using GAS has statistically significant scorer reliability. These findings suggest that use of GAS composite scores (weight times the problem level) is a reliable tool and therefore not a reason to discourage the use of GAS as a means for monitoring a client's progress over time.

Reflections on Guiding Principle E: Responsibilities for General and Public Welfare

Cooksy, L. J. — Mon, 13 Jul 2009 14:52:40 PDT

The Fifth Guiding Principle: Beacon, Banality, or Pandora's Box?

Morris, M. — Mon, 13 Jul 2009 14:52:40 PDT

Human Resource Development (HRD) Evaluation and Principles Related to the Public Interest

Russ-Eft, D. F. — Mon, 13 Jul 2009 14:52:40 PDT

This paper examines the issues involved in the use of ethical standards related to social responsibility using the two ethical codes: the American Evaluation Association Guiding Principles for Evaluators and the Academy of Human Resource Development Standards on Ethics and Integrity. This examination will take the perspective of an internal evaluator working within a private organization. Using that perspective, the discussion will consider the complex issues involved in following such standards, particularly when undertaking evaluations within organizational settings.

The Oral History of Evaluation: The Professional Development of Eleanor Chelimsky

Oral History Project Team, — Mon, 13 Jul 2009 14:52:40 PDT

Book Review: Mertens, D. M. (2009). Transformative research and evaluation. New York: Guilford, 402 pp., $39

Nelson, G. — Mon, 13 Jul 2009 14:52:40 PDT

Book Review: Kellaghan, T., Stufflebeam, D.L., & Wingate, L.A. (2003). International handbook of educational evaluation. Dordrecht: Kluwer, 1040 pp., $699

Clementz, A. R., Greene, J. C. — Mon, 13 Jul 2009 14:52:40 PDT

Book Review: Russ-Eft, D., Bober, M. J., de la Teja, I., Foxon, M. J., & Koszalka, T. A. (2008). Evaluator competencies standards for the practice of evaluation in organizations. San Francisco: Jossey-Bass, 240 pp., $50

Maicher, B. — Mon, 13 Jul 2009 14:52:40 PDT

Book Review: Donaldson, S. I., Christie, C. A., & Mark, M. M. (2009). What counts as credible evidence in applied research and evaluation practice? Thousand Oaks, CA: Sage, 288 pp., $39.95

Fournier, D. M. — Mon, 13 Jul 2009 14:52:40 PDT

Self-Assessments for Workshop Evaluations

D'Eon, M. F., Eva, K. W. — Mon, 13 Jul 2009 14:52:40 PDT

AJE Contribution Categories

Mon, 13 Jul 2009 14:52:40 PDT

American Evaluation Association: Guiding Principles for Evaluators

Thu, 19 Feb 2009 12:35:51 PST

Evaluators' Decision Making: The Relationship Between Theory, Practice, and Experience

Tourmen, C. — Thu, 19 Feb 2009 12:35:51 PST

How do evaluation practitioners make choices when they evaluate a program? What function do evaluation theories play in practice? In this article, I report on an exploratory study that examined evaluation practices in France. The research began with observations of practitioners' activities, with a particular focus on the phases of evaluation design. The purpose of the study was to examine the logic underlying their choices and the role that evaluation theory played in those choices. In this study, I used activity theory and ergonomics methods. I highlight various ways in which working evaluators make evaluation choices in real situations, explained by different sets of conceptual resources they hold related to their level of experience. Finally, I show how this research contributes toward clarifying the role of theories in practice and enriching an evaluation theory

Gilding the Outcome by Tarnishing the Past: Inflationary Biases in Retrospective Pretests

Taylor, P. J., Russ-Eft, D. F., Taylor, H. — Thu, 19 Feb 2009 12:35:51 PST

We tested for inflationary bias introduced through retrospective pretests by analyzing traditional pretest, retrospective pretest, and posttest evaluation data collected on a first-line supervisory leadership training program, involving 196 supervisors and their subordinates, across 17 organizational settings. Retrospective pretest ratings by both trained (supervisors) and untrained (subordinates) respondents were significantly lower than traditional pretest ratings, resulting in substantially inflated training effect sizes when posttests were compared with retrospective pretests rather than with traditional pretests. Further analysis revealed evidence of both respondents' application of an implicit theory of change (i.e., assumption that posttraining scores should generally be higher than pretraining scores) and a tendency to rate their own improvement as greater than that of others. Implications for program evaluation are discussed.

Treatment Fidelity in Multisite Evaluation: A Multilevel Longitudinal Examination of Provider Adherence Status and Change

Zvoch, K. — Thu, 19 Feb 2009 12:35:51 PST

Program implementation data obtained from the repeated observation of teachers delivering one of two early childhood literacy programs to economically disadvantaged students in a large southwestern school district were analyzed to estimate protocol adherence levels at the onset of the intervention as well as the change in adherence over the intervention period. Application of multilevel growth models to the classroom observation data revealed that fidelity to program protocol varied within and between treatment sites during the initial observation and over time. An exploratory examination of select teacher, classroom, and site characteristics indicated that the background characteristics of teachers and contextual factors in the treatment environment were associated with the fidelity outcomes. These results provide some insight into the range of factors that are associated with protocol adherence and highlight the challenge of achieving and maintaining fidelity to a treatment intervention that is delivered by multiple providers over multiple treatment sites.

The Development of Theory-Driven Evaluation in the Military: Theory on the Front Line

Williams, A. P., Morris, J. C. — Thu, 19 Feb 2009 12:35:51 PST

The use of theory-driven evaluation is an emerging practice in the military—an aspect generally unknown in the civilian evaluation community. First developed during the 1991 Gulf War and applied in both the Balkans and Afghanistan, these techniques are now being examined in the North Atlantic Treaty Organisation (NATO) as a means to evaluate the effects of military operations in complex, asymmetric conflict environments. In spite of these practices, theory-driven evaluation in the military is still in the developmental stages. This article traces the development to date of theory-driven evaluation in NATO and assesses its strengths and weaknesses in the military context. We conclude that a cross-pollination of ideas between military and civilian evaluators is urgently needed to improve the quality and effectiveness of military evaluation.

Adding a Time-Series Design Element to the Success Case Method to Improve Methodological Rigor: An Application for Nonprofit Program Evaluation

Coryn, C. L. S., Schroter, D. C., Hanssen, C. E. — Thu, 19 Feb 2009 12:35:51 PST

Brinkerhoff's Success Case Method (SCM) was developed with the specific purpose of assessing the impact of organizational interventions (e.g., training and coaching) on business goals by analyzing extreme groups using case study techniques and storytelling. As an efficient and cost-effective method of evaluative inquiry, SCM is attractive in other contexts as well, although few examples of such uses are to be found in the published literature. However, modifications of the SCM concept and design are sometimes necessary for implementing the approach in nonprofit environments where business goals are not necessarily an explicit objective. This method note demonstrates how SCM was modified and extended to a social service context, in which the program evaluated was aimed at reducing chronic homelessness and unemployment. Modifications included defining success in a nonprofit setting and adding a time-series element to the design features of traditional SCM to increase methodological rigor.

Do Self-Assessments Work to Detect Workshop Success?: An Analysis of Argument and Recommendation by D'Eon et al

Lam, T. C. M. — Thu, 19 Feb 2009 12:35:51 PST

D'Eon et al. concluded that change in performance self-assessment means from before to after a workshop can detect workshop success in their and other situations. In this commentary, their recommendation is refuted by showing that (a) self-assessments with balanced over- and underestimations are still biased and should not be used to evaluate workshops, even though the means of self-assessments and criterion measures are artificially equal; (b) participants' performance should not be attributed directly to training, even if the self-assessments are psychometrically valid and obtained prior to the workshop as well; (c) self-assessment findings should not be generalized to other situations without further analysis and caution, even if the participants' performance can be attributed to training. For clarifying the recommendation by D'Eon et al. to use ``aggregated self-assessments'' to evaluate workshops, analysis of multilevel data is explained and discussed. Finally, nine rules of thumb in using self-assessments for evaluating training are provided.

Call for Nominations for Editorship of the American Journal of Evaluation (2010--2012)

Thu, 19 Feb 2009 12:35:51 PST

AJE Contribution Categories

Thu, 19 Feb 2009 12:35:51 PST