American Journal of Evaluation current issue

From the Editor

Miller, R. L. — Tue, 10 Nov 2009 13:20:48 PST

A Review and Synthesis of Current Research on Cross-Cultural Evaluation

Chouinard, J. A., Cousins, J. B. — Tue, 10 Nov 2009 13:20:48 PST

As a fairly new and emergent construct, there remain many gaps in our knowledge about how to integrate notions of culture and cultural context into evaluation theory and practice, as well as gaps in our knowledge about how to conduct and implement evaluations in immigrant and indigenous communities. In this article, the authors provide a comprehensive review of the empirical literature on evaluations conducted in cultural communities, with an emphasis on the relationship between evaluators and stakeholders in the cross-cultural program context. The analysis of the literature selected for review leads to the development of a theoretical framework describing the inter-related and multi-textual dimensions (relational, ecological, methodological, organizational and personal) that interweave throughout the evaluation, and that ultimately inform the relationship between evaluators and stakeholders in the cross-cultural program context. The article concludes with an agenda for future research.

Finding the Impact in a Messy Intervention: Using an Integrated Design to Evaluate a Comprehensive Citywide Health Initiative

Weitzman, B. C., Mijanovich, T., Silver, D., Brecher, C. — Tue, 10 Nov 2009 13:20:48 PST

This article uses the evaluation of the Robert Wood Johnson Foundation’s (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites’ plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.

Six Guiding Principles for Evaluating Mode-2 Strategies for Sustainable Development

Tue, 10 Nov 2009 13:20:48 PST

A new generation of public programs emerges, which specifically addresses complex societal problems we witness today. Programs for these types of complex issues—in this article, we consider more closely the challenge of sustainable development—are characterized by emergent design, learning processes between diverse actors, and adaptive management. Managers of these kinds of programs have new demands for evaluation and evaluators. This article describes prevailing evaluation methods for sustainable development (progress assessment, goal-oriented program evaluation, and program theory evaluation) and the challenges they meet when confronted with the complexity of designing and conducting systemic intervention programs for sustainable development. The evaluation framework that we propose offers guiding principles to assist evaluators in evaluating complex programs.

The Role of Evaluation in Research--Practice Integration Working Toward the ''Golden Spike''

Brown Urban, J., Trochim, W. — Tue, 10 Nov 2009 13:20:48 PST

Program evaluation and planning is at the heart of efforts to integrate the domains of practice and research. Traditionally, research and practice have operated in independent spheres with practitioners focused on the implementation of programs that affect individual behavior and researchers focused on the development and testing of theory. Evidence-based practice (EBP), practice-based evidence, and translational research have attempted to unite these worlds, and although significant advances have been made, there is a continued need to find mechanisms that enable a seamless connection between knowledge generation and application. We propose a method that builds on the traditions of theory-driven evaluation, logic modeling, and systems science and uses evaluation and program planning as the bridging mechanism between research and practice. Included in this approach are methods that aid in the explicit expression of implicit theories, management of evaluation resources, and linkage of program theory and evaluation measures to a research evidence base.

Exploring the Intervention-- Context Interface: A Case From a School-Based Nutrition Intervention

Bisset, S., Daniel, M., Potvin, L. — Tue, 10 Nov 2009 13:20:48 PST

It has been acknowledged for several decades that programs interact with context. The nature of this interactivity, and how it defines a program, has not been adequately addressed. We view this lacuna as a function of the dominant theoretical perspectives guiding knowledge of program operations. We propose the actor-network theory (ANT) and its conceptual apparatus, the sociotechnical network, as suitable for guiding the acquisition of general knowledge on program operations. We tested this proposition with an instrumental case study of health professional practices during the implementation of a nutrition program into an elementary school setting. Data collection and analysis were guided by the ANT. Data were derived from semistructured interviews completed with six health professionals (nutritionists). Analysis procedures focused on the nutritionists’ collective representation of the microprocesses by which they aimed to build a sociotechnical network of alliances with educational stakeholders. Findings identified nutritionists as preoccupied with three overarching goals during the implementation of the nutrition program, whereby goals were found to take form interactively with the interests of the program participants (primarily students) and stakeholders (primarily teachers). Nutritionists strategically translated program components as a means of negotiating with participants and stakeholders. The findings of this study support the theoretical proposition that program implementation is a process of expanding a sociotechnical network. Beyond simply reaffirming that programs do indeed adapt to context, we interpret this adaptation through the lens of a social theory that suggests why and how adaptation is an inevitable component of program implementation.

Evaluators' Reflections on the Ethical Implications of Their Early Experiences

Cooksy, L. J. — Tue, 10 Nov 2009 13:20:48 PST

What kinds of ethical dilemmas do people who are new to the profession of evaluation face? Are they different from the challenges faced by more experienced evaluators? What strategies do they use to resolve them? These questions were the starting point for the essays included in this issue of the Ethical Challenges section. The essays were written by people who responded to information sent to them by faculty who were in turn responding to the following request from the section editor: ‘‘As part of the American Journal of Evaluation’s continued effort to help young professionals publish, I am putting together an Ethical Challenges section that will feature challenges faced by new evaluators. I am writing to ask for your help in soliciting essays from advanced graduate students or recent graduates about ethical challenges that they have faced. Ideally, the author will describe a specific dilemma (using the American Evaluation Association’s [AEA] Guiding Principles as a reference point) and then discuss how his or her newness in the profession affected its evolution and/or resolution.’’ Of the essays received in response to this request, three were selected.

Methodological Changes and Respecting Stakeholder Dignity

Kallemeyn, L. M. — Tue, 10 Nov 2009 13:20:48 PST

An Evaluation of an Alternative Teacher Certification Program A Matter of Trust

Grubbs, S. T. — Tue, 10 Nov 2009 13:20:48 PST

Evaluation Team Dynamics: Intragroup Ethical Challenges

Urias, D. — Tue, 10 Nov 2009 13:20:48 PST

Applied Research Consultants (ARC): A Vertical Practicum Model of Training Applied Research

Nadler, J. T., Cundiff, N. L. — Tue, 10 Nov 2009 13:20:48 PST

The demand for highly trained evaluation consultants is increasing. Furthermore, the gap between job seekers’ evaluation competencies and job recruiters’ expectations suggests a need for providing practical training experiences. A model using a vertical practicum (advanced students assisting in the training of newer students) is suggested as an ideal training solution. Applied Research Consultants (ARC) is a 2-year vertical practicum conceptually built around a student-run consulting firm. ARC provides opportunities for graduate students to apply what they have learned in the classroom to real-world situations, thereby gaining invaluable experience. ARC’s history, structure, and successes are discussed in the context of suggested goals in teaching, training, and hiring competent evaluation consultants. Additional methods of providing practical training in evaluation and consultation including class projects, mentored projects, internships, and practicum programs are discussed. The resources, institutional commitment, and possible limitations of a vertical practicum training model are also discussed.

Book Review: V. Sue and L. Ritter Conducting Online Surveys Thousand Oaks, CA: Sage, 2007. 208 pp. $39.95

Hartmann, D. — Tue, 10 Nov 2009 13:20:48 PST

Book Review: Debra J. Holden & Marc Zimmerman, (2009). A Practical Guide to Program Evaluation Planning. Thousand Oaks, CA: Sage. $29.95, 184 pp

Evergreen, S. D. H. — Tue, 10 Nov 2009 13:20:48 PST

Acknowledgement of Reviewers

Tue, 10 Nov 2009 13:20:48 PST