Evaluation Standards

Below you will find the Evaluation Standards set up by the Evaluation Institute and the American Evaluation Association. They are guidelines that every evaluation should adhere to. I realize it might be a bit "boring" to read, but these are the beliefs that support all evaluations. As we work to flesh out the design of the Project IMPACT Evaluation Advisory Group, we need to keep these in mind.

THE PROGRAM EVALUATION STANDARDS Summary of the Standards

Utility Standards The utility standards are intended to ensure that an evaluation will serve the information needs of intended users.

U1  Stakeholder Identification  Persons involved in or affected by the evaluation should be identified, so that their needs can be addressed.

U2  Evaluator Credibility  The persons conducting the evaluation should be both trustworthy and competent to perform the evaluation, so that the evaluation findings achieve maximum credibility and acceptance.

U3  Information Scope and Selection  Information collected should be broadly selected to address pertinent questions about the program and be responsive to the needs and interests of clients and other specified stakeholders

U4  Values Identification  The perspectives, procedures, and rationale used to interpret the findings should be carefully described, so that the bases for value judgments are clear.

U5  Report Clarity  Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation, so that essential information is provided and easily understood.

U6  Report Timeliness and Dissemination  Significant interim findings and evaluation reports should be disseminated to intended users, so that they can be used in a timely fashion.

U7  Evaluation Impact  Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the likelihood that the evaluation will be used is increased.

Feasibility Standards The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.

F1  Practical Procedures  The evaluation procedures should be practical, to keep disruption to a minimum while needed information is obtained.

F2  Political Viability  The evaluation should be planned and conducted with anticipation of the different positions of various interest groups, so that their cooperation may be obtained, and so that possible attempts by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted or counteracted.

F3  Cost Effectiveness  The evaluation should be efficient and produce information of sufficient value, so that the resources expended can be justified

Propriety Standards The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.

P1  Service Orientation  Evaluations should be designed to assist organizations to address and effectively serve the needs of the full range of targeted participants.

P2  Formal Agreements  Obligations of the formal parties to an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that these parties are obligated to adhere to all conditions of the agreement or formally to renegotiate it.

P3  Rights of Human Subjects  Evaluations should be designed and conducted to respect and protect the rights and welfare of human subjects.

P4  Human Interactions  Evaluators should respect human dignity and worth in their interactions with other persons associated with an evaluation, so that participants are not threatened or harmed.

P5  Complete and Fair Assessment  The evaluation should be complete and fair in its examination and recording of strengths and weaknesses of the program being evaluated, so that strengths can be built upon and problem areas addressed.

P6  Disclosure of Findings  The formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation and any others with expressed legal rights to receive the results.

P7  Conflict of Interest  Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results.

P8  Fiscal Responsibility  The evaluator's allocation and expenditure of resources should reflect sound accountability procedures and otherwise be prudent and ethically responsible, so that expenditures are accounted for and appropriate

Accuracy Standards The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.

A1  Program Documentation  The program being evaluated should be described and documented clearly and accurately, so that the program is clearly identified.

A2  Context Analysis  The context in which the program exists should be examined in enough detail, so that its likely influences on the program can be identified.

A3  Described Purposes and Procedures  The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed.

A4  Defensible Information Sources  The sources of information used in a program evaluation should be described in enough detail, so that the adequacy of the information can be assessed.

A5  Valid Information   The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the interpretation arrived at is valid for the intended use.

A6  Reliable Information  The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable for the intended use.

A7  Systematic Information  The information collected, processed, and reported in an evaluation should be systematically reviewed, and any errors found should be corrected.

A8  Analysis of Quantitative Information  Quantitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A9  Analysis of Qualitative Information  Qualitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A10  Justified Conclusions  The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can assess them.

A11  Impartial Reporting  Reporting procedures should guard against distortion caused by personal feelings and biases of any party to the evaluation, so that evaluation reports fairly reflect the evaluation findings.

A12  Metaevaluation  The evaluation itself should be formatively and summatively evaluated against these and other pertinent standards, so that its conduct is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.

The Program Evaluation Standards. The Joint Committee on Standards for Educational Evaluation, James R. Sanders, Chair. Sage Publications: Thousand Oaks, CA. 1994.

Evaluation Continued. . . .

Getting answers to the evaluation questions.

There are many methods, each with their own uses, advantages and difficulties. Methods include:

·      Surveys

·      Analysis of Administrative Data

·      Key Informant Interviews

·      Observation

·      Focus Groups

Evaluations could use any, not necessarily all, of these methods, depending on the question and goal of the evaluation.

Surveys are a set of questions that are asked of everyone in the same way. Surveys can answer question about how many and how often. For example:

·      How many clients are satisfied with services?

·      How often do people have difficulties using the services?

Typical questions might be like this:

How satisfied are you with the program?

Very                Satisfied          Neither            Dissatisfied     Very

Satisfied                                                                                  Dissatisfied

Surveys might be used to describe the entire client population, if respondents were chosen randomly or systematically and if the sample is sufficiently large.

Randomly or systematically choosing people to respond to surveys means using some defined method to select people. For example:

·      Randomly choosing – generate a list of random numbers, assign each person a random number, sort the people by the random number and take the people listed first. They were put on top of the list randomly.

·      Systematic selection – a typical method is to start with the 5^th person and then select every 7th person after that. The numbers, the 5 th and the 7 th are also chosen randomly.

Randomly select locations to be in the sample, and then survey everyone in that location.

Random or systematic selection means that the group of people you select are more likely to be similar to your clients, in general. You aren't excluding any particular groups, or including only certain groups. You are avoiding bias, in sampling terms. If you do use random or systematic selection, then most likely you can use the results of your survey to make conclusions about your

clients. If you don't use random or systematic selection, you can NOT use the results of your survey to make conclusions about your clients population. That is, you cannot generalize from your study to your client population. You can only say “The people who took this

survey said ...”

Analysis of administrative data is just using statistical analysis on program data that is already collected.

Administrative data has advantages:

·      No new data collection is required

·      Many databases are relatively large

·      Data may be available electronically

Disadvantages:

·      Data were gathered for another purpose, so may not have necessary variables.

·      In all administrative data sets, some fields are likely to be more accurate than others.

Focus groups are structured discussions among small groups of people. Generally, a facilitator leads a group of 8-10 people in a discussion about selected topics with planned questions, while allowing for interesting, new or unplanned follow up questions.

Typical focus group questions are like these:

·      What is your overall impression of the program?

·      What are the things you like or dislike about the program?

·      What have you gained in this program?

·      If you have not noticed any changes in yourself, what do you think are the reasons?

Key informant interviews are qualitative, in-depth interviews of 15 to 35 people selected for their first-hand knowledge about a topic of interest.

Key informant interviews also include a planned set of questions on the topics of interest.

Key informant interviews are useful to when candid information about sensitive topics are needed. Group discussions may inhibit people from giving candid feedback.

Interviews should include a very diverse range of people.

Observations are methods that yield a systematic description of events or behaviors in the social setting chosen for study. Observation methods can be highly structured, for example:

Systematic Social Observation - a field research method in which teams of researchers observe the object of study in its natural setting. Researchers record events as they see and hear them and do not rely upon others to describe or interpret events. The researchers follow well-specified procedures that can be duplicated.

Observations can also be unstructured, for example, participant observation, or taking an active part in group activities. The premise underlying participant observation is that the researcher becomes a more effective observer by taking an active role in the performance of regular activities. In other words, knowledge gained through doing is of a higher quality than what is obtained only through observation. In many cases, involvement with ordinary chores will not only enhance the researcher's understanding of the processes, techniques, and words associated with these activities, but will also result in better rapport with informants.

Focus groups, interviews and observation are qualitative research methods, that is, methods that are less likely to rely on statistical analysis.

Advantages

·      Useful to help figure out major program problems that cannot be explained by more formal methods of analysis.

·      The evaluator may see things that participants and staff may not see.

·      The evaluator can learn about things which participants or staff may be unwilling to reveal in more formal methods

·      Useful when it's not clear what the program problems might be.

·      Useful to give good ideas of what topics program participants and staff think are important.

·      Useful in developing surveys, in determining what questions or issues are important to include.

·      Useful when a main purpose is to generate recommendations

·      Useful when quantitative data collected through other methods need to be interpreted.

Disadvantages:

·      The evaluator's subjective views can introduce error.

·      The focus of the evaluator is only on what is observed at one time in one place.

·      Information from observations/ interviews/ groups can be time consuming and difficult to interpret.

·      Focus groups could be dominated by one individual and his/her point of view.

·      Generally, information from focus groups, interviews, and observations CANNOT be used to describe the client population.