Sustainability and Scaling Formats

As you think about sustainability and scaling, it’s essential to think about embedding evaluation processes into the day-to-day functioning of your organization. Previous entries have talked about these issues; the next several entries will provide you with various formats to utilize. The question is: when the grant monies end, how will you determine what’s working; how will you collect and analyze data: how will you use the data to make decisions on sustainability and scaling of effective services, activities and resources; how will you “market” your results to the financial authorities at your college? Results-Based Accountability is one approach.

Trying Hard Is Not Good Enough by Mark Friedman

 What is Results-Based Accountability™?

Results-Based Accountability™ (RBA), also known as Outcomes-Based Accountability™ (OBA), is a disciplined way of thinking and taking action that communities and educational systems can use to improve the lives of children, youth, families, adults and students. RBA is also used by organizations to improve the performance of their programs or services. Developed by Mark Friedman and described in his book Trying Hard is Not Good Enough, (www.resultsaccountability.com). RBA is being used throughout the United States, and in countries around the world, to produce measurable change in people’s lives.

What’s different about RBA?

RBA uses a data-driven, decision-making process to help schools and organizations get beyond talking about problems to taking action to solve problems. It is a simple, common sense framework that everyone can understand. RBA starts with ends and works backward, towards means. The “end” or difference you are trying to make looks slightly different if you are working on a broad community level or are focusing on your specific program or organization.

Creating Organizational Impact with RBA

Organizational impact focuses on conditions of well-being for children, young adults and the organization as a whole that a group of leaders is working collectively to improve. For example: “Young adults ready for the workforce,” or “A safe and productive manufacturing site”. In RBA, these conditions of well-being are referred to as results or outcomes.

It is critical to identify powerful measures to determine the progress a community college is making towards achieving its outcomes. For organizations, the measurements are known as indicators and are usually collected by governmental agencies. A community wanting to have residents with good jobs may look at “turning the curve” on the unemployment rate.

Performance Accountability

Organizations and programs can only be held accountable for the customers they serve.  RBA helps organizations identify the role they play in community-wide impact by identifying specific customers who benefit from the services the organization provides.
For programs and organizations, the performance measures focus on whether customers are better off as a result of your services.  These performance measures also look at the quality and efficiency of these services.  RBA asks three simple questions to get at the most important performance measures:

• How much did we do?
• How well did we do it?
• Is anyone better off?

In answering these questions, a job training program might measure:

• The number of trainees in its program
• The ability of its trainers to explain concepts
• The percentage of its trainees who obtain and keep a job.

Turn the Curve Thinking

Once you identify the most powerful measure(s) to improve, RBA provides a step-by-step process to get from ends to means. This process is called “Turn the Curve” thinking.

Why use RBA?

RBA improves the lives of students, employers and communities and the performance of programs because RBA:

• Gets from talk to action quickly;
• Is a simple, common sense process that everyone can understand;
• Helps groups to surface and challenge assumptions that can be barriers to innovation;
• Builds collaboration and consensus;
• Uses data and transparency to ensure accountability for both the well-being of people and the performance of programs.

TAACCCT Research Questions

In addition to the quantitative data required by the Department of Labor for Project IMPACT, they are also wanting a response to several research questions:

•What service delivery and/or system reform innovations resulted in improved impacts for participants?

•Under what conditions can these innovations most effectively be replicated?

•What are the types of emerging ideas for service delivery change and/or system reform that seem the most promising for further research? Under what conditions are these ideas most effective?

•What directions for future research on the country’s public workforce system, and workforce development in general, were learned?

As you know, I conduct quarterly site visits to gather ongoing information about the status of Project IMPACT at each of the individual community colleges. I will continue to do that through the duration of the grant. In addition, I will ask for responses on the four research questions and add that information to the case studies. 

As always, I have appreciated your willingness to share your many successes and your limited concerns. I look forward to visiting with all you in the new year.   

Coordinating Internal and External Evaluation

The following article is transcribed from the EvaluATE Newsletter (Lori Wingate, Fall, 2014).

All ATE (like all TAACCCT) projects are required to allocate funds for external evaluation services. So, when it comes to internal and external evaluation, the one certain thing is that you must have an external evaluator. . . . There are many options for coordinating internal and external evaluation functions. Over the years, I have noted four basic approaches:

(1) External Evaluator as Coach: The external evaluator provides guidance and feedback to the internal project team through the life of the grant. This is a good approach when there is already some evaluation competence among team members. The external evaluator's involvement enhances the credibility of the evaluation and helps the team continue to build their evaluation knowledge and skills.

(2) External Evaluator as Heavy-Lifter: The external evaluator takes the lead in planning the evaluation, designing instruments, analyzing results, and writing reports. The internal team mainly gathers data and provides it to the external evaluator for processing. In this approach, the external evaluator should provide clear-cut data collection protocols to ensure systematic collection and handling of data by the internal team before they turn the information over to the external evaluator.

(3) External Evaluator as Architect; The external evaluator designs the overall evaluation and develops data collection instruments. The project team executes the plan, with technical assistance from the eternal evaluator as needed - particularly at critical junctures in the evaluation such as analysis and reporting. With this approach, it is important to front-load the evaluation budget in the first year of the project to allow for intensive involvement by the external evaluator.

(4) Divide-and-Conquer: The internal team is responsible for evaluating project implementation and immediate results. The external evaluator handles the evaluation of longer-term outcomes. . . . The external evaluator is responsible for determining and assessing the impact of the work in terms of application of the content and changes in evaluation practice.

Taking on part of an evaluation internally is often seen as a means of conserving project resources, and it can have that effect. But do not make the mistake of thinking internal evaluation is cost-free. At minimum, it takes time, which is sometimes a rarer commodity than money. In short, there is no one best way to coordinate internal and external evaluation. Your approach should make sense for your project in light of available resources (including staff time and expertise) and what you need your evaluation to do for your project.

Evaluator as a "Critical Friend"

The following is a reprint from the EvaluATE Newsletter for Fall, 2014:

The term critical friend describes a stance an evaluator can take in his or her relationship with the program of project they evaluate. Costa and Kallick (1993) provide this seminal definition: "A trusted person who asks provocative questions, provides data to be examined through another lens, and offers critique of a person's work as a friend." (p. 50)

The relationship between a project and an evaluator who is a critical friend is one where the evaluator has the best interests of the program at heart and the project staff trusts that this is the case. The evaluator may see their role as being both a trusted advisor and a staunch critic. He or she pushes the program to achieve its goals in the most effective way possible while maintaining independence. The evaluator helps the project staff to view information in different ways, while still being sensitive to the projects staff''s own views and priorities. The evaluator will call attention to negative or less effective aspects of a project, but will do so in a constructive way. By pointing out potential pitfalls and flaws in the project, the critical friend evaluator can help the project to grow and improve.

To learn more. . . .

Costa, A.L. & Kallick, b. (1993). Through the lens of a critical friend. Educational Leadership, 51(2) 49-51. http://bit.ly/crit-friend

Rallis, S.F., & Rossman, G.B. (2000). Dialogue for learning: Evaluator as critical friend. New Directions for Evaluation, 86, 81-92.

Evaluation versus Research

Several times during our conversations about data collection, the issue of whether what we're doing is in line with "research methodologies." Such issues as validity, reliability, "control groups," correlations, etc. have come up. Below is a short comparison between research and evaluation. I admit I "borrowed" it from an online source, but it does a good job of showing, in a brief way, the differences. Our goal with the TAACCCT grants is to provide information on what's working, how various activities are successful, which ones might need some revision, what is sustainable and scalable and providing data to the US Department of Labor. Please let me know what you think.

Research vs Evaluation
 

Research and evaluation are important tools in the hands of researchers and educators to gain insight into new domains and to assess the efficacy and efficiency of a specific program or methodology. There are many similarities and overlapping between research and evaluation, to suggest they are almost interchangeable. However, there are many differences also in their form, purpose, and content that is made use of by experts to achieve different goals. Let us take a closer look.

Research

Research is a systematic, logical, and rational activity that is undertaken by scientists and experts in humanities to gain knowledge and insight in various fields of studies. This body of knowledge is later used to develop applications and tools that make our life better and richer.

Research can be undertaken to prove hypothesis, theorems, works of earlier experts, or it can be undertaken to establish new theories and facts. Whether basic or applied, research is always helpful in expanding human knowledge. It is not that research can be done only in science subjects. On the contrary, much of the research and development work in the world is today being carried out in humanities and behavioral sciences to enrich and better human lives. The basic purpose behind all research is to expand human knowledge.

Evaluation

Evaluation is the procedure that aims at improving the performance or efficiency of individuals, groups, programs, policies and even governments around the world. Evaluation means a judgment or assessment. Any evaluation tool is so designed so as to answer questions pertaining to efficacy and efficiency of a system or an individual. It is only through unbiased evaluation that we come to know if a program is effective or ineffective.

Evaluation as a tool serves the purpose of knowing about how well a person or program is doing and what needs to be done to improve efficacy and efficiency. Evaluation of a program or policy can help the management to come up with solutions to the problems so that the performance levels can be improved.

What is the difference between Research and Evaluation?
• Evaluation is done to judge or assess the performance of a person, machine, program or a policy while research is done to gain knowledge in a particular field
• Evaluation makes judgment and assessment that is helpful for decision makers so that they can implement changes to improve efficacy and efficiency
• Research and evaluation both enhance our knowledge, but evaluation leads to changes that cause improvement whereas research is mostly undertaken to prove something
• Research is undertaken to generalize the findings from a small sample to a large section of the population. On the other hand, evaluation is done in particular situations and circumstances, and its findings are applicable for that situation only.

Read more: http://www.differencebetween.com/difference-between-research-and-vs-evaluation/#ixzz3ELF0ddRZ

Designing for Scalable Educational Improvement

The following concepts are gleaned from an article in Scaling Up Success (2005, John Wiley & Sons, Inc) by Chris Dede, James P. Honan and Laurence C. Peters. Please think about them as you progress through the last years of your grant.

1. Educational improvement must be an ongoing systems-level activity, especially if it is to be scaled up.

2. Educational improvement must occur at multiple levels, with alignment across levels.

3. Collaborations and partnerships are critical to educational improvement, but they are hard work.

4. Educational improvement efforts need to be studied and documented, so that they can contribute to the development of a systematic knowledge base about efforts to innovate, scale up innovations, and then sustain them.

5. The core of educational improvement is building human capacity for effective performance at all levels of the educational system, but especially at those levels most proximal to students.

6. Change is initiated, sustained, and carried through systems by people.

7. Social structures such as learning communities, practitioners' networks, and study groups can facilitate change.

8. Educational practitioners need opportunities to learn with understanding, so that they grasp the principles of educational improvement efforts and not just the practices and procedures.

9. Inquiry-based approaches to professional development build knowledge that makes sense.

10. Changes in thinking and practice come about through hard work in a context that provides opportunities to try out changes in the classroom and to receive feedback and coaching.

How does these concepts and ideas apply to your project? What do you need to be doing in order to implement these thoughts? These challenges can provide the framework for ongoing dialogue among your staff and with partners.

Sustainability, Scaling and Evaluation Capacity Building

Over the next several months, I will be addressing three issues in this blog:

• Sustainability of successful services, resources & activities
• Bringing those successful elements to scale to meet the needs of the individual community college and their students
• Embedding evaluation capacity building in the the culture of the community college related to elements of Project IMPACT

In the past posts, I have added information about evaluation capacity building and sustainability. This post will discuss scaling, defined as "adapting a locally successful innovation to a wide variety of settings while retaining it effectiveness" (C. Dede et al, Scaling Up Success). Bloom defines it as "the process of closing the gap between the real and ideal conditions as they pertain to particular needs or problems" (P. Bloom, Scaling Your Social Venture). Sutton has a more poetic way of looking at scaling: "Scaling requires the wherewithal to hound yourself and others with questions about what it takes to link the never-ending now - the perpetual present tense that every person is trapped in - to the sweet dreams you hope to realize later" (R. Sutton, Scaling Up Excellence).

Bloom suggests that seven questions about organizational capabilities should be considered carefully (SCALERS):

• Staffing: Can you become effective at recruiting, training, organizing, managing, and retaining the additional employees, managers, volunteers and board members need to support scaling?
• Communicating: Can you become effective at configuring and delivering messages that persuade potential donors, employees, volunteers, partners, regulators and beneficiaries about the value of what you do?
• Alliance-Building: Can you become effective at forming partnerships and alliances with other parties and organizations that will allow you to leverage their resources and capabilities in addition to your own?
• Lobbying: Can you become effective at advocating with influential people and public policy makers for the changes you seek?
• Earnings-Generation: Can you develop a business model that will allow you to be financially sustainable?
• Replicating: Can you become effective at developing systems and procedures that facilitate replicating or repeating your successful interventions?
• Stimulating Market Forces: Can you become effective at using market incentives to encourage others to donate, buy, invest, volunteer, or otherwise behave in ways that benefit your venture?

As Project IMPACT moves into its third year, please consider the above seven questions - keep them in the back (or the front) of your mind. The grant money ends in a little over two years. What do you want to bring to scale and how will you accomplish it? 

Project IMPACT Evaluation Training/Update Sept 9th

August 21^st, 2014

Good morning – as you might be aware, we have scheduled a Project IMPACT Evaluation Training/Update meeting for September 9^th from 9:00 am to 3:30 pm at The Central Community College Board Room. I know several members of the Project IMPACT Evaluation Advisory Group are planning to attend. I would encourage ALL of you to please show your support for the project by either attending in person or sitting in via phone conferencing. Your input continues to be a vital part of the evaluation process. We need both internal and external viewpoints on the progress of Project IMPACT.

I would encourage you to view the webpage - http://www.impactnebraska.org/ and send me your feedback. If you have access to the OneDrive account, please review the wealth of information pertaining to curriculum, instruction, marketing, etc. Again your feedback is important.

In addition, I would like your views on two other issues:

- How to raise the level of your involvement

- Explore the possibility of adding new members to the Evaluation Advisory Group

I feel I have not done an adequate job of utilizing this group to its potential. I know all of you are very busy with work and family; however you bring such a high level of expertise that I want to find better ways of tapping your advice so I might send that information on to the leadership at Project IMPACT. If you could recommend other potential members, particularly from the world of manufacturing, I would be open to expanding the group. But it needs to be worth it – I’ll work to make it worthy of their time.

Let me know – let’s get this blog going as a way of communicating. I know it’s hard to get into the habit of going to the blog (as I’m aware in my too infrequent additions!), but let’s give it the ol’ Nebraska Effort!!

Greetings Project IMPACT Evaluation Advisory Group - I apologize for the long lapse in any entries. No excuses! We've all been busy.

I've added some ideas about Evaluation Capacity Building. Let me know what you think. Embedding evaluation into the way we do business is essential if our organizations are to learn and grow.

Evaluation Capacity Building

ECB involves the design and implementation of teaching and learning strategies to help individuals, groups, and organizations, learn about what constitutes effective, useful, and professional evaluation practice.

The ultimate goal of ECB is sustainable evaluation practice—where members continuously ask questions that matter, collect, analyze, and interpret data, and use evaluation findings for decision-making and action.

For evaluation practice to be sustained, participants must be provided with leadership support, incentives, resources, and opportunities to transfer their learning about evaluation to their everyday work.

Sustainable evaluation practice also requires the development of systems, processes, policies, and plans that help embed evaluation work into the way the organization accomplishes its mission and strategic goals.

Strategies to promote learning about evaluation practice.

Individual, group, organizational learning about….

·      Evaluation design, implementation, management

·      Evaluative thinking

·      Evaluation as performance improvement strategy

·      Organizational learning culture

Capacity building approaches

Top-down: change organizational policies and practices.

Bottom-up: skill-building (continuous learning and improvement, ‘reflective practitioners’).

Partnerships: strengthening relationships between organizations (two-way flow of knowledge).

ECB goal

Sustainable evaluation practice

·      Continuously ask questions that matter

·      Collect, analyze, and interpret data

·      Use evaluation findings for decision-making and action

Sustainable evaluation practice

Supported by….

·      Leadership support

·      Incentives

·      Resources

·      Opportunities for learning transfer

·      Systems, processes, policies, and plans to embed evaluation work into organizational operations

Role of internal evaluator

Evaluation practitioner –ECB practitioner

·      Teacher (curriculum = evaluation process & findings)

·      Collaborative team member

·      Facilitator of evaluation champions

·      Evaluation infrastructure architect

·      Evaluation study guide

·      Evaluator-directed study in consultation with stakeholders

·      Evaluator as coach with stakeholders leading study

·      Collaboration between evaluator and stakeholders

Useful attributes of internal evaluator

·      Respectful communicator

·      Group facilitation & interpersonal skills

·      Open-minded

·      Trustworthy

·      Ensures all voices are heard

·      Has ability to manage project details

·      Identifies teachable moments

Think about how Project IMPACT might begin the process of building an internal network of evaluators and how the Evaluation Advisory Group might assist. 

Draft Agenda for Project IMPACT Retreat Evaluation Sessions

Here is the draft agenda for the Evaluation Advisory Group break-out sessions for Thursday, January 30th, 2014 at the Project IMPACT Retreat. I'm hoping all of you can and will attend, but I understand that you all lead busy lives outside of this project. I will audio tape the sessions and take notes. I will post both on this blog if it will allow me to add an audio attachment. I will also post them on the Project IMPACT SkyDrive.

Take care - I hope to see you next week. If you need information on the registration, please email Sue Baer at sbaer@cccneb.edu.

Mike

 Project IMPACT Retreat

Evaluation Sessions

January 30th, 2014

Session #1 (9:45 – 10:45) – Data Collection, Comparison Group

·      Intake form – Descriptions, Process

·      Comparison Groups – Potential Groups (Propensity Score Matching)

·      Data Sharing Agreements

·      Logistics (Mfg. Survey, other data collection options)

Session #2 (1:15 – 2:15) –Curriculum Evaluation

·      Overview of four (4) courses (view from SkyDrive?)

·      Tooling U and other resources

·      Contextual Learning/Basic Skills

Session #3 (2:30 – 3:30) – Coaching Evaluation

·      Overview of coaching

·      Potential resources

·      Use of coaches in data collection