American Journal of Evaluation recent issues

American Evaluation Association Guiding Principles for Evaluators

2008-08-18

The Complexity of Measuring the Quality of Program Implementation With Observations: The Case of Middle School Inquiry-Based Science

Brandon, P. R., Taum, A. K. H., Young, D. B., Pottenger, F. M., Speitel, T. W. — 2008-08-18

In the growing literature on the evaluation of program implementation, less has been said about evaluating program quality than about evaluating other aspects of program implementation. Furthermore, most articles and reports in the program-implementation evaluation literature have presented only brief descriptions of how implementation instruments have been developed. In this article, the authors describe a method for evaluating the quality of implementation of middle school inquiry-based science using data from observations scaled with paired comparison judgments. The authors show the complexities of developing and applying the method, describe how they tried it out, present the results of validity and reliability analyses, and describe the method's strengths and weaknesses.

Using System Dynamics as an Evaluation Tool: Experience From a Demonstration Program

Fredericks, K. A., Deegan, M., Carman, J. G. — 2008-08-18

Evaluators are often faced with many challenges in the design and implementation of a program's evaluation. Because programs are entangled in complex networks of structures and stakeholders, they can be challenging to understand, and they often pose issues of competing and conflicting goals. However, by using a systems mapping approach to evaluation design, evaluators can identify and diagram the relationships between different variables and learn about their potential impact and unintended consequences on a program's processes. In this article, we describe how using a systems approach and causal loop mapping approach during an evaluation of a demonstration program for people with developmental disabilities helped evaluators to better understand the program's implementation and the variation in the program's outcomes.

Evaluator Competencies: What's Taught Versus What's Sought

Dewey, J. D., Montrosse, B. E., Schroter, D. C., Sullins, C. D., Mattox, J. R. — 2008-08-18

This article explores the overlaps and disconnects between the competencies evaluators acquire during graduate school and those required and desired by employers. To investigate this relationship, two different surveys are administered, one for job seekers and the other for employers; 205 postings in the American Evaluation Association job bank were analyzed. The findings suggest that employers, job seekers, and job posters generally agree on the importance of some competencies, such as quantitative analyses and data management. However, some skills desired by employers, such as interpersonal, project management, and presentation skills, differ from skills that job seekers acquire in graduate school. Opportunities for additional experiences in real-world evaluation settings could fill these gaps. Implications for training and future research on training in evaluation are discussed.

Don't Throw Out the Baby With the Bath Water: A Novel Way of Evaluating Outcomes in the Healthy Families America Programs

Kessler, S. R., Nixon, A., Nelson, C. — 2008-08-18

The current study examines the inherent challenges of the wide-scale implementation and replication of program models. Using the Healthy Family America (HFA) program model, the study reviews the adaptation/adoption debate and highlights the ways in which program adoption and program adaptation could coexist to facilitate successful program implementations. Specifically, 103 HFA program sites were evaluated based on their adherence to HFA's flexible program model. Although the results are mixed, the study presents a new way of addressing existing program implementation and replication issues.

Evaluating Intangible Outcomes: Using Multiattribute Utility Analysis to Compare the Benefits and Costs of Social Programs

Selameab, T., Yeh, S. S. — 2008-08-18

Increasingly, program managers are required to evaluate programs with multiple outcomes against costs. This is difficult when program outcomes are not easily translated into a common metric. This article compares cost benefit analysis with Multiple Attribute Utility Technology and discusses the application of judgment in both analytical techniques. Multiple Attribute Utility Technology is illustrated with a case example to demonstrate how it may be applied to evaluate the cost-effectiveness of programs with multiple intangible outcomes.

In Memoriam: Egon Guba (1924--2008)

Schwandt, T. A. — 2008-08-18

R-E-S-P-E-C-T: Remembering Egon Guba

Quinn Patton, M. — 2008-08-18

A Letter From Egon

Stake, R. — 2008-08-18

Memories of a Novice, Learning From a Master

Greene, J. C. — 2008-08-18

A Tribute to Egon Guba

Stufflebeam, D. — 2008-08-18

Remembering my Teacher

Schwandt, T. A. — 2008-08-18

Teaching Program Evaluation Three Selected Pillars of Pedagogy

Oliver, D. E., Casiraghi, A. M., Henderson, J. L., Brooks, A. M., Mulsow, M. — 2008-08-18

Two challenges often associated with teaching program evaluation at the graduate level are the need to incorporate practical skills development and being limited to a one-semester-long course offering. The existing literature provides some information concerning a practical application component; however, there is almost no discussion of pedagogy or, more specifically, the selection of teaching strategies for a program evaluation course. This article explains and analyzes a pedagogical framework that has been used to teach program evaluation at a research university. The concepts and practices described may help those who plan to teach about program evaluation and other courses involving practical application components, particularly in an adult learner context. The article consists of four sections: (a) analysis of the pedagogical framework, (b) brief overview of the course, (c) explanation of how the pedagogy was integrated into the course, and (d) conclusions.

Challenges and Opportunities in Experiential Learning

Cooksy, L. J. — 2008-08-18

Principles, Promises, and a Personal Plea: What Is an Evaluator to Do?

McDonald, K. E., Myrick, S. E. — 2008-08-18

The client of a student evaluation team has requested that the evaluators provide confidential identifying information gathered in the course of the evaluation. Here, the authors consider their response to the client's request. Specifically, they draw from professional principles developed to guide ethical decision making for evaluators and educators to consider the advantages and disadvantages of denying or meeting this request from a variety of stakeholder perspectives. The authors then use this stakeholder analysis to engage in imaginative thinking about the possible responses from which the student evaluation team can select. They conclude with a review of lessons learned from this challenging situation.

A Reaction to and Mental Metaevaluation of the Experiential Learning Evaluation Project

Perry, K. M. — 2008-08-18

Many professors of program evaluation courses look for ways to merge their students' theoretical understanding of program evaluation with practical experience. In this scenario, a professor arranges for his students to evaluate a series of workshops in campus residence halls sponsored by a university diversity center (DC). What follows in this article is a reaction to and mental metaevaluation of both the professor's and students' actions as they engage in this evaluation experience. Their actions are assessed within the context of the Joint Committee on Standards for Educational Evaluation (JCSEE) and the American Evaluation Association (AEA) guiding principles. It concludes with implications this experience has for teachers of program evaluation.

A Collaborative Immersio Approach to Evaluation Capacity Building

Huffman, D., Thomas, K., Lawrenz, F. — 2008-08-18

The purpose of this article is to describe a new collaborative immersion approach for developing evaluation capacity that was used in kindergarten through Grade 12 (K-12) schools and to place this new approach on a continuum of existing capacity-building methods. The continuum extends from individualistic training-oriented methods to collaborative real-world methods of building evaluation capacity. The strengths and weaknesses of various evaluation capacity building methods are analyzed and critiqued. Included are examples of capacity building focused on technical assistance, workshops, fellowships, and collaborative immersion. The authors make the case for considering a collaborative immersion approach to evaluation capacity building.

Methodological Challenges of Collecting Evaluation Data From Traumatized Clients/Consumers: A Comparison of Three Methods

Campbell, R., Adams, A. E., Patterson, D. — 2008-08-18

This project integrates elements of responsive evaluation and participatory evaluation to compare three evaluation data collection methods for use with a hard-to-find (HTF), traumatized, vulnerable population: rape victims seeking postassault medical forensic care. The first method involves on-site, in-person data collection, immediately postservices; the second, telephone follow-up assessments, 1 week postservices; and the third, private, self-administered surveys completed immediately postservices. There are significant differences in response rates across methods: 88% in-person, 17% telephone, and 41% self-administered. Across all phases, clients gave positive feedback about the services they received and about all three methods of data collection. Follow-up analyses suggested that nonresponders did not differ with respect to client characteristics, assault characteristics, or nursing care provided. These findings suggest that evaluations with HTF service clients may need to be integrated into on-site services because other methods may not yield sufficient response rates.

Book Review: Barber, W. (2008). Gunnar Myrdal: An Intellectual Biography. New York: Palgrave Macmillan

Datta, L.-e. — 2008-08-18

AJE Contribution Categories

2008-08-18

American Evaluation Association: Guiding Principles for Evaluators

2008-05-20

Evaluation's Second Act: A Spotlight on Learning

Preskill, H. — 2008-05-20

Imagine a world where evaluation is a social epidemic . . . where individuals, groups, organizations, and communities are constantly learning about and from evaluations. The author believes that we are well on our way to creating a "global cascade" of evaluative thinking and practice. Evidence of this phenomenon can be seen in the field's increasing commitment to evaluation capacity building and participatory evaluation approaches and the expanding applications of technology for accessing and disseminating information. If the old saying is "The second act is the best" then we might ask ourselves, "How can evaluation realize its potential for making a difference in our increasingly complex and changing world?" The author argues that addressing today's critical evaluation theory, practice, and policy issues requires us to understand the ways in which evaluation facilitates learning, how evaluators can maximize learning from and about evaluation, and what it would mean if the evaluation profession was a sustainable learning community.

Educating for Intelligent Belief in Evaluation

Schwandt, T. A. — 2008-05-20

We are facing a rather worrying brew of developments affecting practical intellectual life in modern society and the very well-being of society itself. These developments threaten to degrade the central role that the cognitive endeavor, known as reasoned evaluative criticism, plays in the achievement, maintenance, and enhancement of the good society. In this climate, we cannot simply teach about the ways and means of doing evaluation; we must also convey the idea of evaluation as a practical, intellectual disposition and outlook on social and political life. In sum, there is a pressing need to educate both aspiring evaluators and the citizenry for intelligent belief in evaluation. This article discusses several components of such a belief.

Bringing Evaluative Learning to Life

King, J. A. — 2008-05-20

This excerpt from the opening plenary asks evaluators to consider two questions regarding learning and evaluation: (a) How do evaluators know if, how, when, and what people are learning during an evaluation? and (b) In what ways can evaluation be a learning experience? To answer the first question, evaluators can apply the commonplaces of evaluative learning, where, in a given evaluative context, the evaluator is a teacher, the clients/participants are students, and the process and results of the evaluation are the curriculum. To answer the second question, evaluators can consider two ideas for understanding evaluative learning: (a) evaluation for accountability and control and (b) evaluation for program development.

Random Assignment and Informed Consent: A Case Study of Multiple Perspectives

Walker, R., Hoggart, L., Hamilton, G. — 2008-05-20

Although random assignment is generally the preferred methodology in impact evaluations, it raises numerous ethical concerns, some of which are addressed by securing participants' informed consent. However, there has been little investigation of how consent is obtained in social experiments and the amount of information that can be conveyed—and absorbed—prior to consent. This article reports on the implementation of the Employment Retention and Advancement (ERA) program, a large social experiment in the United Kingdom. Procedures to elicit informed consent were generally scrupulously followed, but even so many participants appeared to have only limited understanding of the experiment and their involvement in it. The reasons for this are articulated and recommendations are made to help ensure, to the extent possible, that potential random assignment study participants adequately understand their situation and their choices.

Cross-Disciplinarization: A New Talisman for Evaluation?

Jacob, S. — 2008-05-20

Reflections on crossing disciplinary lines abound in the scientific community. Can cross-disciplinary approaches, with all their complexity and particularities, provide the way forward in the search for practical solutions to real-world problems? In this article, the author addresses how the debate on cross-disciplinarization pertains to the field of policy evaluation. Evaluation is appropriate terrain for such a discussion as this particular field of social science seeks to produce useful knowledge for both managers and policy makers. As such, the author offers a general account of the advantages and disadvantages of cross-disciplinary evaluation. Because evaluation requires close collaboration between individuals from different domains and backgrounds, the author further outlines the specific challenges that face the practitioner when conducting a cross-disciplinary evaluation.

Reducing World Poverty by Improving Evaluation of Development Aid

Clements, P., Chianca, T., Sasaki, R. — 2008-05-20

This article argues that given its structural conditions, development aid bears a particularly heavy burden of learning and accountability. Unfortunately, however, the organization of evaluation guarantees that evaluations will be inconsistent and it creates incentives for positive bias. This article presents evidence from organizational studies of aid agencies, from the public choice literature, from eight development projects in Africa, and from one in India, that demonstrates positive bias and inconsistency in monitoring and evaluation. It proposes that the evaluation function should be professionalized through an approach titled "monitoring and evaluation for cost effectiveness," and it compares this approach to the World Bank's results-based monitoring and evaluation, the Development Assistance Committee's five evaluation criteria, and evaluations based on randomized trials. This article explains the analytics of the proposed approach and suggests directions for further research.

Book Review: Program Theory-Driven Evaluation Science: Strategies and Applications, by Stewart I. Donaldson. Mahwah, NJ: Lawrence Erlbaum, 2007

Coryn, C. L. S. — 2008-05-20

Book Review: Logic Modeling Methods in Program Evaluation by Joy A. Frechtling. San Francisco: Jossey-Bass, 2007

Taut, S. — 2008-05-20

Book Review: Reframing Evaluation Through Appreciative Inquiry, by Hallie Preskill and Tessie Tzavaras Catsambas. Thousand Oaks, CA: Sage, 2006

Rodriguez-Campos, L. — 2008-05-20

AEA Award Citations 2007

2008-05-20

AJE Contribution Categories

2008-05-20

From the Editor

Miller, R. L. — 2008-02-06

The Evaluation of Large Research Initiatives: A Participatory Integrative Mixed-Methods Approach

Trochim, W. M., Marcus, S. E., Masse, L. C., Moser, R. P., Weld, P. C. — 2008-02-06

Over the past few decades there has been a rise in the number of federally funded large scientific research initiatives, with increased calls to evaluate their processes and outcomes. This article describes efforts to evaluate such initiatives in one agency within the U.S. federal government. The authors introduce the Evaluation of Large Initiatives (ELI) project, a preliminary effort to explore how to accomplish such evaluation. They describe a pilot effort of this project to evaluate the Transdisciplinary Tobacco Use Research Center (TTURC) initiative of the National Cancer Institute. They present a summary of this pilot evaluation including the methods used (concept mapping, logic modeling, a detailed researcher survey, content analysis and systematic peer-evaluation of progress reports, bibliometric analysis and peer evaluation of publications and citations, and financial expenditures analysis) and a brief overview of results. Finally, they discuss several important lessons and recommendations that emerged from this work.

The Fairy Godmother--and Her Warts: Making the Dream of Evidence-Based Policy Come True

Weiss, C. H., Murphy-Graham, E., Petrosino, A., Gandhi, A. G. — 2008-02-06

Evaluators sometimes wish for a Fairy Godmother who would make decision makers pay attention to evaluation findings when choosing programs to implement. The U.S. Department of Education came close to creating such a Fairy Godmother when it required school districts to choose drug abuse prevention programs only if their effectiveness was supported by "scientific" evidence. The experience showed advantages of such a procedure (e.g., reduction in support for D.A.R.E., which evaluation had found wanting) but also shortcomings (limited and in some cases questionable evaluation evidence in support of other programs). Federal procedures for identifying successful programs appeared biased. In addition, the Fairy Godmother discounted the professional judgment of local educators and did little to improve the fit of programs to local conditions. Nevertheless, giving evaluation more clout is a worthwhile way to increase the rationality of decision making. The authors recommend research on procedures used by other agencies to achieve similar aims.

Combining Campbell Standards and the Realist Evaluation Approach: The Best of Two Worlds?

van der Knaap, L. M., Leeuw, F. L., Bogaerts, S., Nijssen, L. T. J. — 2008-02-06

This article presents an approach to systematic reviews that combines the Campbell Collaboration Crime and Justice standards and the realist notion of contexts-mechanisms-outcomes (CMO) configurations. Both approaches have their advantages and drawbacks, and the authors will make a case for combining both approaches to profit from their advantages without having to deal with their separate weaknesses. They proceed to discuss how their approach should be put into practice and illustrate its use by giving a concise description of an example from their own work on the effects of prevention of violence. The authors conclude with a discussion of the strengths and weaknesses of their suggested approach and state that combining Campbell standards and the realist evaluation approach should be done more often.

Using an Evaluation Hotline to Promote Stakeholder Involvement

Skolits, G. J., Boser, J. A. — 2008-02-06

This article addresses the design and application of a hotline to promote broader community-wide participation in a public school evaluation. Evaluations of community resources such as public schools present evaluators with challenges from the perspective of promoting stakeholder involvement. Although many evaluation stakeholders are readily identifiable, there are potentially many unknown and hard-to-reach community leaders and residents who may want to participate in the evaluation. An evaluation hotline offers a mechanism for potentially identifying and reaching some of these interested community stakeholders and enabling their participation. This article introduces an evaluation hotline design and implementation, along with perceived hotline strengths and weaknesses observed in its use.

Indigenous Fieldworkers in Evaluation: An Exploration of Ethical Considerations

Cooksy, L. J. — 2008-02-06

Ethical Dilemmas in Evaluations Using Indigenous Research Workers

Alexander, L. B., Richman, K. A. — 2008-02-06

This article addresses ethical dilemmas experienced by street-level research and evaluation workers recruiting and gathering data in community-based research projects. The authors focus on a subgroup of street-level research workers, whom they call research extenders (REs), employed because they share important characteristics with the target population. Like other street-level research workers, some REs are single role (only do research work) and others are dual role (do research work and provide services). They discuss relevant literature and findings from pilot focus groups exploring how REs understand responsible ethical conduct in their research work in the community. REs face ethical issues somewhat different from those of more traditional, non-indigenous street-level research workers. These issues increase social risk for study participants and could compromise the validity of data. Researchers need to be aware of and address the distinctive set of ethical issues raised by research and evaluation projects employing indigenous, street-level research workers.

Identifying Unique Ethical Challenges of Indigenous Field-Workers: A Commentary on Alexander and Richman's "Ethical Dilemmas in Evaluations Using Indigenous Research Workers"

Smith, N. L. — 2008-02-06

In contrast with nonindigenous workers, to what extent do unique ethical problems arise when indigenous field-workers participate in field studies? Three aspects of study design and operation are considered: data integrity issues, risk issues, and protection issues. Although many of the data quality issues that arise with the use of indigenous workers appear similar to those found in conventional studies, unique problems may arise in terms of risk and protection of confidentiality, especially when field-workers occupy dual roles of both data collectors and service providers. Much is known about ethical problems in the use of conventional field-workers, but the use of indigenous field-workers has been less well studied. It is important to understand which ethical issues may be unique to their work so that specialized training, study design, and management procedures can be developed.

Using Self-Assessments to Detect Workshop Success: Do They Work?

D'Eon, M., Sadownik, L., Harrison, A., Nation, J. — 2008-02-06

An accepted gold standard for measuring change in participant behavior is third-party observation. This method is highly resource intensive, and many small-scale evaluations may not be in a position to use this approach. This study was designed to assess the validity and reliably of aggregated group self-assessments as one way to measure workshop effectiveness. In this study, participants completed a pre-, post-, and retrospective self-assessment on their perceived skill level in delivering feedback. Trained raters scored recorded role-play episodes. A statistically and practically significant difference in feedback skills was detected in both the self-assessments and observer ratings. The instruments used to assess participants' feedback skills had acceptable reliability. Those charged with workshop evaluation should have some confidence that aggregated self-assessments can be used to help determine workshop effectiveness.

Response Rates for Mixed-Mode Surveys Using Mail and E-mail/Web

Converse, P. D., Wolfe, E. W., Xiaoting Huang, , Oswald, F. L. — 2008-02-06

This study examines response rates for mixed-mode survey implementation involving mail and e-mail/Web components. Using Dillman's Tailored Design Method, 1,500 participants were sent a survey either (a) via mail with a follow-up contact via e-mail that directed them to a Web-based questionnaire or (b) via e-mail that directed them to a Web-based questionnaire with a follow-up contact via mail. Results indicate that these mixed-mode procedures produce moderately high response rates. However, the mail survey tended to be more effective than the e-mail/Web survey, when serving either as the initial contact or as the follow-up contact. These results suggest that survey implementation involving mail followed by e-mail/Web, or even mail-only approaches, may result in larger samples than implementation involving e-mail/Web followed by mail.

Book Review: Program Evaluation: Forms and Approaches (3rd ed.), by John M. Owen. New York: Guilford Press, 2006

Cooksy, L. J. — 2008-02-06

Book Review: The ABCs of Evaluation: Timeless Techniques for Program and Project Managers (2nd ed.), by John Boulmetis and Phyllis Dutwin. San Francisco: Jossey-Bass, 2005

Montrosse, B. E. — 2008-02-06

AJE Contribution Categories

2008-02-06

Book Review: Running the Obstacle Course to Sexual and Reproductive Health: Lessons from Latin America, by Bonnie Shepard. Westport, CT: Praeger, 2006

Guzman, B. — 2008-05-22

American Evaluation Association: Guiding Principles for Evaluators

2007-11-20

Crossing National, Continental, and Linguistic Boundaries: Toward a Worldwide Evaluation Research Community in Journals of Evaluation

Diaz-Puente, J. M., Cazorla, A., Dorrego, A. — 2007-11-20

In an attempt to build a worldwide evaluation community, English evaluation journals are best positioned to promote international dialogue and increasingly provide international exchanges that deepen and enrich the evaluation field. This article analyzes the degree of internationalization achieved by these journals by looking at the extent of international authorship of the contributions they publish. Results show that—in spite of the efforts made—national, continental, and linguistic boundaries are still hindering the international exchange in the journals. Discussion is presented concerning the implications of these results and the efforts required to take greater advantage of the English journals to bridge national and regional traditions in the evaluation field.

The Cost-Effectiveness of Five Policies for Improving Student Achievement

Yeh, S. S. — 2007-11-20

Comparisons of student achievement effect sizes suggest that systems in which student performance in math and reading is rapidly assessed between 2 and 5 times per week are 4 times as effective as a 10% increase in per pupil expenditure, 6 times as effective as voucher programs, 64 times as effective as charter schools, and 6 times as effective as increased accountability. Achievement gains per dollar from rapid assessment are even greater—193 times the gains that accrue from increasing preexisting patterns of educational expenditures, 2,424 times the gains from vouchers, 23,166 times the gains from charter schools, and 57 times the gains from increased accountability. Two sensitivity analyses suggest that the relative advantage for rapid assessment is not sensitive to the particular parameter estimates.

Models and Mechanisms for Evaluating Government-Funded Research: An International Comparison

Coryn, C. L. S., Hattie, J. A., Scriven, M., Hartmann, D. J. — 2007-11-20

This research describes, classifies, and comparatively evaluates national models and mechanisms used to evaluate research and allocate research funding in 16 countries. Although these models and mechanisms vary widely in terms of how research is evaluated and financed, nearly all share the common characteristic of relating funding to some measure of past performance. Each of these 16 national models and mechanisms were rated by independent, blinded panels of professional researchers and evaluators in two countries on more than 25 quality indicators. The national models were then ranked using the panels' ratings, in terms of their validity, credibility, utility, cost-effectiveness, and ethicality. The highest ratings were received by nations using large-scale research assessment exercises. Bulk funding and indicator-driven models received substantially lower ratings. Implications for research evaluation practice and policy are considered and discussed.

Assessment Beyond Performance: Phenomenography in Educational Evaluation

Micari, M., Light, G., Calkins, S., Streitwieser, B. — 2007-11-20

Increasing calls for accountability in education have promoted improvements in quantitative evaluation approaches that measure student performance; however, this has often been to the detriment of qualitative approaches, reducing the richness of educational evaluation as an enterprise. In this article the authors assert that it is not merely performance but also how learners think and how their thinking changes that we should be measuring in educational program evaluation. They describe a mixed-methods evaluation model based on the qualitative method phenomenography that can be used to evaluate how learners think in multiple contexts, from skills training to employee development to higher education, and how their thinking may change over time. They then describe two evaluation studies making use of this approach and provide suggestions for evaluators interested in using the phenomenographic model.

Evaluations That Consider the Cost of Educational Programs: The Contribution of High-Quality Studies

Ross, J. A., Barkaoui, K., Scott, G. — 2007-11-20

Cost studies are program evaluations that judge program worth by relating program costs to program benefits. There are three sets of strategies: cost—benefit, cost-effectiveness, and cost-utility analysis, although the last appears infrequently. The authors searched relevant databases to identify 103 cost studies in education and then reduced the set to 31 using criteria focused on rigor in determining program effects and assessment of costs. They found that cost studies provide evidence of the worth of educational spending at the macro and individual program levels, information that is not provided by other evaluation approaches; provide direction for program improvement that differs from recommendations based solely on effect sizes; and contribute to knowledge development by constructing and testing models that link spending to student learning.

Horizontal Evaluation: Fostering Knowledge Sharing and Program Improvement Within a Network

Thiele, G., Devaux, A., Velasco, C., Horton, D. — 2007-11-20

Horizontal evaluation combines self-assessment and external evaluation by peers. Papa Andina, a regional network that works to reduce rural poverty in the Andean region by fostering innovation in potato production and marketing, has used horizontal evaluations to improve the work of local project teams and to share knowledge within the network. In a horizontal evaluation workshop, a project team and peers from other organizations independently assess the strengths and weaknesses of a research and development (R&D) approach being developed and then compare the assessments. Project team members formulate recommendations for improving the R&D approach, and peers consider ways to apply it back home. Practical results of horizontal evaluation have included strengthening the R&D approaches being developed, experimenting with their use at new sites, improvements in other areas of work, and strengthened interpersonal relations among network members.

The New Environment for Development Evaluation

Picciotto, R. — 2007-11-20

The millennium development goals have created new challenges for development evaluation. The main unit of account has shifted to the country level. Evaluation ownership must move from donor agencies to developing countries. The recognition that rich countries have development obligations is opening up evaluation frontiers beyond aid. A transformation of evaluation priorities is needed: (a) Evaluation frameworks should give more weight to alignment with the millennium development goals, (b) impact measures of development programs should be aggregated to the country and global levels, (c) accountability should be enhanced by sharper attributions of results according to the distinctive accountabilities of development partners, (d) attribution of results to aid should be examined using methods appropriate to the situation, and (e) the asymmetry of the development evaluation agenda should be remedied by a sharper focus on the impact of rich countries' policies on global poverty reduction.

Evaluation of the Fun With Books Program: An Interview With Katrina Bledsoe

Fitzpatrick, J. L. — 2007-11-20

Using Problem-Based Learning to Train Evaluators

Lee, J., LeBaron Wallace, T., Alkin, M. — 2007-11-20

As the evaluation profession has continued to grow and develop, there has been a corresponding concern about how to properly train future evaluation practitioners. Those who teach evaluation strive to develop training opportunities that create the appropriate balance of the practical, how-to knowledge of evaluation with the burgeoning theoretical knowledge that undergirds responsible evaluation practice. There is a recognized need to move beyond traditional teaching methods to ones that are more engaging and "hands on" to help students understand the interactive nature of program evaluation. Problem-based learning is an experiential learning approach that can integrate the need to balance self-study of theory and practice, along with the need to familiarize students with the dynamic, interactive nature of program evaluation. This article serves as an introduction to the problem-based learning approach and describes this instructional method applied to teaching a graduate-level course in evaluation procedures.

Teaching Program Evaluation in the Visual and Performing Arts

Johnson, R. L., McGuinness, C., McCorkendale, S. C., Laney, M. A. — 2007-11-20

The literature indicates that the demand for program evaluations is likely to increase and that those evaluations will be conducted across diverse disciplines. Preparation of discipline-specific evaluators may present challenges. This study reviews the experiences of a group of arts educators who participated in a discipline-specific evaluation course that combined a summer residence experience with Saturday meetings. The primary aim of the course is for participants to develop an evaluation plan for implementation in their settings. Through use of a reflective survey and a focus group, the authors investigate the extent to which the course addressed participants' needs as arts educators in program evaluation. In their feedback, the arts educators indicate that the development of an evaluation plan was one of the strongest aspects of the course; however, they also indicate that future offerings of the course should provide models of evaluation plans and reports specific to the arts.

Photolanguage: How a Picture Can Inspire a Thousand Words

Bessell, A. G., Deese, W. B., Medina, A. L. — 2007-11-20

Program evaluators need methods that can be used in a variety of situations to help gather data. Photolanguage is one such technique that uses black-and-white photographs to elicit responses from individuals. It is particularly useful in situations where the respondents may give restricted or only minimal data. This article provides 1the step-by-step process for implementation of Photolanguage, explores the possibility of uses for Photolanguage as a tool for small-group evaluation, and supplies data showing its benefits as a method of evaluation.

Book Review: Multiple Case Study Analysis, by Robert E. Stake. New York: Guilford Press, 2006

Barela, E. — 2007-11-20

Evaluation Theory, Models, and Applications, by Daniel L. Stufflebeam and Anthony J. Shinkfield. San Francisco: Jossey-Bass, 2007. 768 pp. $70.00

Datta, L.-e. — 2007-11-20

Public Program Evaluation: A Statistical Guide by Laura Langbein with Claire L. Felbinger. Armonk, NY: M. E. Sharp, 2006

Shipman, S. — 2007-11-20

Taking Stock Again Results in the Same Conclusions: A Reply to Fetterman and Wandersman's Defense of Empowerment Evaluation

Lin Miller, R., Campbell, R. — 2007-11-20

Acknowledgement of Reviewers

2007-11-20

AJE Contribution Categories

2007-11-20