This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Hanssen, C. E. Articles by Dunet, D. O. Search for Related Content Social Bookmarking                         What's this?

Concurrent Meta-Evaluation

A Critique

Hanssen Consulting, LLC, carlh{at}hanssenconsulting.com

Frances Lawrenz

University of Minnesota

Diane O. Dunet

Centers for Disease Control and Prevention

Meta-evaluations reported in the literature, although rare, often have focused on retrospective assessment of completed evaluations. Conducting a meta-evaluation concurrently with the evaluation modifies this approach. This method provides the opportunity for the meta-evaluators to advise the evaluators and provides the basis for a summative judgment about the quality of the evaluation. The authors conducted a concurrent meta-evaluation of a new evaluation technique being developed by a federal governmental agency; the new evaluation technique was expected to be highly visible and widely applied. The differences between concurrent meta-evaluation and other meta-evaluations were continuous involvement, attendance at data collection events, and external verification of the evaluation data. The authors' experience conducting the concurrent meta-evaluation is described and challenges are discussed in this critique. The authors conclude that concurrent meta-evaluation holds promise for improving the practice of evaluation and of meta-evaluation.

Key Words: evaluation methodology • meta-evaluation • site visits • evaluation standards • public health

References

• Ashworth, A., Cebulla, A., Greenberg, D., & Walker, A. (2004). Meta evaluation: Discovering what works best in welfare provision. Evaluation, 10, 193-216.[Abstract]
• Baizerman, M., Compton, D.W., & Stockdill, S. (2002). New directions for ECB. In D. W. Compton, M. Baizerman, & S. Stockdill (Eds.), The art, craft and science of evaluation capacity building (New Directions for Evaluation, No. 93, pp. 109-116). San Francisco: Jossey-Bass.
• Brandon, P. (1998). A meta-evaluation of schools' methods for selecting site-managed projects. Studies in Educational Evaluation, 24, 213-228.[CrossRef]
• Burt, R., & Celotto, N. (1992). The network structure of management roles in a large matrix firm. Evaluation and Program Planning, 15, 303-326.[CrossRef][Web of Science]
• Cooksy, L. (1999). The metaevaluand: The evaluation of project TEAMS. American Journal of Evaluation, 20(1), 123-136.[Abstract]
• Cousins, J.B., & Shulha, L.M. (2006). A comparative analysis of evaluation and its cognate fields of inquiry: Current issues and trends. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The SAGE handbook of evaluation (pp. 233-254). Thousand Oaks, CA: Sage.
• Dunet, D., Sparling, P., Hersey, J., Williams-Piehota, P., Hill, M., Hanssen, C., et al. (in press). A new evaluation tool to obtain practice-based evidence of worksite health promotion programs. Preventing Chronic Disease.
• Fetterman, D., & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today and tomorrow. American Journal of Evaluation, 28, 179-198.[Abstract/Free Full Text]
• Fitzpatrick, J., Sanders, J., & Worthen, B. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). New York: Pearson.
• Grasso, P. (1999). Metaevaluation of an evaluation of reader focused writing for the veterans benefits administration. American Journal of Evaluation, 20, 355-370.[Web of Science]
• Gullickson, A., Wingate, L., Lawrenz, F.A., & Coryn, C. (2006). The NSF advanced technology education program: Final evaluation report. Kalamazoo: Western Michigan University: The Evaluation Center.
• Hersey, J., Williams-Piehota, P., Dunet, D., Sparling, P., Alexander, J., Hill, M., et al. (in press). Promising practices in promotion of healthy weight at small and medium-sized U.S. work sites. Preventing Chronic Disease.
• Ives, W. (1992). Evaluating new multimedia technologies for self-paced instruction. Evaluation and Program Planning, 15, 287-296.[CrossRef][Web of Science]
• Jinkerson, D., Cummings, O., Neisendorf, B., & Schwandt, T. (1992). A case study of methodological issues in cross-cultural evaluation. Evaluation and Program Planning, 15, 273-285.[CrossRef][Web of Science]
• The Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.
• Lawrenz, F., Keiser, N., & LaVoie, B. (2004). Site visits: A methodological review. American Journal of Evaluation, 24, 341-352.[Web of Science]
• Mark, M., & Henry, G. (2004). The mechanisms and outcomes of evaluation influence Evaluation, 10(1), 35-57.[Abstract]
• Niemiec, R., Sikorski, M., Clark, G., & Walberg, H. (1992). Effects of management education: A quantitative synthesis. Evaluation and Program Planning, 15, 297-302.[CrossRef][Web of Science]
• Niemiec, R., & Walberg, H. (1992). Evaluation of corporate education. Evaluation and Program Planning, 15, 271-272.[CrossRef][Web of Science]
• Nilsson, N., & Hogben, D. (1983). Meta-evaluation. In E. R. House (Ed.), Philosophy of evaluation: New Directions for Program Evaluation (No. 19, pp. 83-97). San Francisco: Jossey-Bass.
• Patton, M. (1997). Utilization-focused evaluation (3rd ed.). New York: Sage.
• Scott-Little, C., Hamann, M., & Jurs, S. (2002). Evaluations of after school programs: A metaevaluation of methodologies and narrative synthesis of findings. American Journal of Evaluation, 23, 387-419.[Abstract]
• Scriven, M. (1991). Evaluation thesaurus (4th ed.). New York: Sage.
• Scriven, M. (2004). Key evaluation checklist. Retrieved December 28, 2006, from http://www.wmich.edu/evalctr/checklists/kec.htm
• Society for Prevention Research. (2004). Standards of evidence: Criteria for efficacy, effectiveness, and dissemination. Retrieved December 31, 2006, from http://www.preventionresearch.org/commlmon.php#SofE
• Stufflebeam, D. (2001). The metaevaluation imperative. American Journal of Evaluation, 22, 183-209.[Abstract]
• Stufflebeam, D., & Shinkfield, A. (2007). Evaluation theory, models, & applications. San Francisco: John Wiley.
• Uusikyla, P., & Virtanen, P. (2000). Meta-evaluation as a tool for learning: A case study of the European structural fund evaluations in Finland. Evaluation, 6(1), 50-65.[Abstract/Free Full Text]
• Woodside, A., & Sakai, M. (2001). Metaevaluation of performance audits of government tourism marketing programs. Journal of Travel Research, 39, 369-379.[Abstract/Free Full Text]

This version was published on December 1, 2008

American Journal of Evaluation, Vol. 29, No. 4, 572-582 (2008)
DOI: 10.1177/1098214008320462


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				L. J. Cooksy Evaluators' Reflections on the Ethical Implications of Their Early Experiences American Journal of Evaluation, December 1, 2009; 30(4): 572 - 574. [Abstract] [PDF]

This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Hanssen, C. E. Articles by Dunet, D. O. Search for Related Content Social Bookmarking                         What's this?