Results from Sept 24-25 Trainings - Feedback

I don't know if you have gone on the SkyDrive website to look lately, but Dr. Grandgenett posted the results of the flip charts regarding the nine (9) outcomes of Project IMPACT. I would appreciate any feedback on the comments. 

I'm scheduling site visits to the five community colleges in December to solidify the data collection processes, including data sharing agreements, types of data and the format, collection protocol, and other logistical issues (data storing, coding, and transmission). If you have any suggestions for any of the processes, please let me know. We will be scheduling a phone conference for January 2014 to update all of you on the status of the project. 

Results from Sept 24-25 Evaluation Trainings

  

1. Total Unique Participants Served

Cumulative total number of individuals entering any of the grant-funded programs offered

Year 1:                       0

Year 2:                       365

Year 3:                       473

Total:                         838

Comments:

-       Seems small, is entire regions for the 5 community colleges represented?

-       A library of definitions for acronyms be available and understood by the consortium.

-       How does this number compare to state need?

-       We will track all our participants but what if only a subset or our participants receive coaching?  Do we report these groups together or separately?

-       Will employers have a path for employers to be part of it?

-       Why can’t existing curriculum by used rather than redeveloping what already exists?

-       How will students be selected?

-       Do we count the comparison group as participants if they are taking the courses we create, but not receiving other services?

-       Will this also be broken down into specific programs?

-       Need to define “participant”.  Is this someone who takes any class offered?  (assume yes)

-       Can a institution count  training such as safety 10 hour, although the current employers are not enrolled in a certificate or degree?

-       I’m discussing ROI the assumption is a pay increase.  What is it going to take to convince industry the KSA’s are worth the investment?

2. Total Number of Participants Completing a TAACCCT-Funded Program of Study

Number of unique participants having earned all of the credit hours needed for the award of a degree or certificate in any grant-funded program

Year 1:           0

Year 2:           280

Year 3:           310

Total:             590

-       Question:  Earned all credit hours or actually applied for/received an award?  Within a specific period of time?

-       From a numbers perspective, defining TAACCCT funded program of study is key to capturing appropriate data.

-       We can count certificate level completers, correct?  X2

-       Any effort to promote life long learning from certificate to diploma to AAS.

-     Are we taking participants or completers?  (Perkins Terminology)

3. Total Number of Participants Still Retained in Their Program of Study or Other TAACCCT-Funded Program

Number of unique participants enrolled who did not complete and are still enrolled in a grant-funded program of study

Year 1:                       0

Year 2:                       90

Year 3:                       77

Total:                        167

-       How long does the retention period last?  When are the participants no longer in the program?

-       Do we count students who completed certificate and are still working toward associates?

-       Is there a time frame for these participants?  What if a funding for program isn’t continued?

-       Can we track to determine why and encourage reinvolvement ?  Soft skill help.

4. Total Number of Participants Completing Credit Hours

Total number of students enrolled that have completed any number of credit hours to date.

Year 1:                       0

Year 2:                       308

Year 3:                       420

Total:                         728

-       How does the KSAO testing take into consideration disabilities (ADA), results, disclosure of information to students, coaches / instructors?

-       Who, what, when, where, and why on workshops.

-       Is number of completed credit hours important?

-       Big jump between year 1 and 2 – maybe slower start up.

-       Why is DOL asking for this?  What is the big question being addressed here?

5. Total Number of Participants Earning Credentials

Aggregate number of degrees and certificates completed by participants in grant-funded programs of study

Year 1:                       0

Year 2:                       337

Year 3:                       452

Total:                         789

-       Differentiate between degrees and certificate.

-       Does this have to be unique users?

-       Is the CSHA 10 hour a credential or simply a participant?

-       We presumed we would count multiple credentials earned per student.  Ok?

-       At Topeka TAACCCT conference, one grant recipient said they could only count each participant once.

-       How will credentials be defined?  How will industry certifications be tracked?

-       Defining credentials is key – keeping a database of base information might be important in case the definition changes.

-       At Topeka TAACCCT conference, one grant recipient said they could only count each participant once.

6. Total Number of Participants Enrolled in Further Education After TAACCCT-funded Program of Study Completion

Total number of students who complete a grant-funded program of study and enter another program of study

Year 1:                       0

Year 2:                       46

Year 3:                       46

Year 4 (follow-up only): 51

Total:                       143

-       Colleges can request data through the national clearing house.

-       Of those tracked, how many choose higher level education.

-       Does this apply only to students transferring for a BS, or can it be another program such as welding once they’ve achieved a certificate?

-       Does the student have to declare their degree or does it count if the student completes a certificate, then continues on for a diploma?

7. Total Participants Employed After TAACCCT-funded Program of Study Completion

Total number of students (non-incumbent workers) who completed a grant-funded program of study entering employment in the quarter after the program exit

Year 1:                       0

Year 2:                       159

Year 3:                       177

Year 4 (follow-up only): 111

Total:                           447

-       What effort can be carried out to assess future promotibility?

-       How does low Nebraska unemployment impact this?

-       Can we include individuals employed following non-credit study leading to MSSC credentials?

8. Total Number of Participants Retained in Employment After Program of Study Completion

Total number of students (non-incumbent) who completed a grant-funded program of study and who entered employment in the quarter after the quarter of program exit who retain employment in the second and third quarters after program exit

Year 1:                       0

Year 2:                       145

Year 3:                       158

Year 4 (follow-up only): 99 

Total:                          402

-       DOL track for employees still employed?

-       How will program be sustainable once put into a program that already exists?

-       Is this employment with the same employer or is it the same job with multiple employers?

-       How would you schedule students at different institutions to get the use of the physical machine that will be purchased?

-       How will employment / wage information be tracked?  Access to data?

-       Does DOL have a tracking program after the first quarter?  They have in the past.

-       DOL MOU’s

9. Total Number of Those Participants Employed at Enrollment Who Received a Wage Increase Post-Enrollment

Total number of students who are incumbent workers and enrolled in a grant funded program who received an increase in wages after enrollment

Year 1:                       0

Year 2:                       107

Year 3:                       121

Year 4 (follow-up only): 52

Total:                        280

-       Does this information absolutely need to be validated?

-       Important to consult employers.

-       Can the wage increase be simply an annual increase or does it have to be specific to the skills learned thru the grant or certifications?

"Comparison Groups"

I realize the phrase "control groups" raised some ethical and pragmatic issues. It implied that some individuals who potentially could benefit from any of the programs offered by Project IMPACT would not be admitted, even through they qualified. That practice runs contrary to the mission of the community colleges. In fact, it was never a viable option, but in the brainstorming process, it was included.

The real issue is finding a "comparison group," to demonstrate that as a result of completing the manufacturing generalist certification and possibly continuing through the laddered options, participants land employment in high wage, high skill, and high need organizations. And that the coursework did indeed make a difference in acquiring a position, being retained and increasing their wages.

In order to find that potential comparison group and collect data, not only on Project IMPACT participants, but also on the comparison group, we need to complete data sharing agreements between the community colleges and Shain Evaluation & Consulting (for educational data), and the colleges and the Nebraska Department of Labor (for employment data). Those discussions are in process.

In addition, we need to "lock down" the comparison group; one method would be to examine the results of previous years (2011-2013) with students in coursework similar to those being offered by Project IMPACT. Another would be to work with several employers who would be willing to share data on new employees not affiliated with Project IMPACT and those new employees who have completed any offerings by the project. The latter would take considerable effort by the employers but could yield vital qualitative data, in addition to quantitative data (and would be the most interesting!). A third method might be to compare employment results from the various "layers" of completion of Project IMPACT participants, to compare those who leave after one course or certification with those who complete a diploma or AAS. If you have other ideas, please let me know. I want us to have a process in place by the end of the calendar year.

Determining Control Groups for Project IMPACT

I've brainstormed some quasi-experimental design options for selecting a control group to match to the participants in Project IMPACT. Please take a look and let me know what you think. I'm also open to other ideas.

 Definitions of "control group":

The control group is composed of participants who do not receive the experimental treatment. When conducting an experiment, these people are randomly selected to be in this group. They also closely resemble the participants who are in the experimental group, or the individuals who receive the treatment. While they do not receive the treatment, they do play a vital role in the research process. Experimenters compare the experimental group to the control group to determine if the treatment had an effect. By serving as a comparison group, researchers are able to isolate the independent variable and look at the impact it had.

Options:

1.     Compare the general population of all community college students who register for courses, certificates, diplomas and/or associate degree programs in technical fields to participants in Project IMPACT(comparing population data to target data)

2.     A sample of new employees in manufacturing who have not interacted with community colleges compared to students who are part of Project IMPACT (comparing convenience sample data to target data)

3.     A comparison of the range of involvement of participants, using those who take the fewest courses with those who complete the MG certification, a diploma, an AAS degree and/or a BS at UNL (in essence, those who are least involved might constitute a “control group”

4.     Select a random sample of participants who register for Project IMPACT courses to receive no support (ie. remediation, coaching, etc.) and compare the progress to those participants who receive the full complement of services provided by the program

5.     Select a community college in the Midwest that has courses similar to Project IMPACT and compare educational and employment outcomes

6.     Utilize meta-analysis to determine a “pseudo-sample” that could function as a control group and compare to participants in Project IMPACT

7.     Select a small sample of participants in Project IMPACT activities and a matched small sample of participants in technical courses not affiliated with the project and monitor over time using the case study method

Eight Tips to Creating an Effective Advisory Board

You don’t need to navigate unfamiliar waters alone. Put together a good board of advisers, and you’ll create a powerful asset that can make a huge difference when you need to get objective advice, scout the marketplace, gauge future trends, seek new strategic positions, have introductions made or build repeat customers.

Unlike corporate boards, advisory boards have no fiduciary responsibility and their advice is non-binding. Some are hands-on, meeting monthly or more, even getting involved in the daily grind. Others meet quarterly, with an eye to the big picture. Many consist solely of interested outsiders, but a good number include investors as well. What all such boards share is this: They advise, evaluate and play devil’s advocate.  Here are eight (8) rules of thumb to follow when building an effective advisory board.

1.Determine the Objective of Your Advisory Board: Advisory boards can be general in scope or targeted to specific markets, industries or issues such as adopting new technology or going global. They provide timely knowledge about trends and competitors, as well as identifying upcoming political, legislative and regulatory developments. They can help you enter new businesses and look at your own operations with an open mind. Advisory boards can also be made up of customers and prospects who provide insights into product development and marketing issues.

2.Choose the Right People:  Of course, when forming a board you need to understand its purpose, but you also need to know what specific skills to seek. In general, look for diverse skills, expertise and experience. You want members to be problem solvers who are quick studies, have strong communications skills and are open minded.

Big names can be a bonus ... but not always: Getting a heavyweight on your board of advisers can give you credibility, but it’s also important to have members who are going to spend the time to give you thoughtful advice or are well connected and willing to make introductions. 

3. Set Expectations: When inviting a prospective member to join your advisory board, you should lay down the ground rules about what is expected in terms of time, responsibilities and term of office. Specify the areas in which you’re seeking help. If the advisory board is going to discuss issues that include private information, members should be notified that they will be asked to sign a confidentiality agreement. 

4. Get the Most Out of Advisory Board Meetings: Prepare for meetings well in advance. Choose a site that is comfortable and free of distractions. Careful thought should be given to developing the agenda and managing the meeting. Solicit input for the agenda, and distribute important information ahead of time. Run the session as you would any professional meeting, and follow it with an action plan. The facilitator should know which experts to draw out and how to stimulate a dialogue. He or she should be result-oriented, as ideas without action aren't worth much. The minutes should be written up and circulated to top management. The notes should include recommendations on key issues. 

5. Ask for Honesty: An advisory board must be open and frank, so don’t be offended if you hear things you don’t like. Your board will also suggest ways of correcting the problems they identify.

If appropriate, encourage members to tell you about their mistakes so you can avoid making the same ones. You can learn a lot by finding out what other people did wrong. 

6. Consider Alternative Feedback Methods: Getting the entire board together on a regular basis may not be possible. Instead, meet or have conference calls with specific members about topics relevant to their expertise as needed. E-mail is a great way to reach everyone and have them respond to you at their convenience. 

7. Respect your Board's Contributions:  Don't abuse or waste their time. Listen to what the board says. Sometimes, a business executive is so close to an issue, you can’t see the forest for the trees. But remember: This isn’t a corporate board, so you don't have to do everything they suggest. Ask yourself, “Does this work for my company? Am I comfortable with that?” Then make a decision. 

8. Keep Board Members Informed:  Once they’re on the board, keep members excited about your business by giving them updates at times when you aren't soliciting their advice. The fact that they've agreed to be on your board means they care about your company, so keeping up-to-date will help them be of greater value to you. Remember that these people are evangelists for the company.

Evaluation Standards

Below you will find the Evaluation Standards set up by the Evaluation Institute and the American Evaluation Association. They are guidelines that every evaluation should adhere to. I realize it might be a bit "boring" to read, but these are the beliefs that support all evaluations. As we work to flesh out the design of the Project IMPACT Evaluation Advisory Group, we need to keep these in mind.

THE PROGRAM EVALUATION STANDARDS Summary of the Standards

Utility Standards The utility standards are intended to ensure that an evaluation will serve the information needs of intended users.

U1  Stakeholder Identification  Persons involved in or affected by the evaluation should be identified, so that their needs can be addressed.

U2  Evaluator Credibility  The persons conducting the evaluation should be both trustworthy and competent to perform the evaluation, so that the evaluation findings achieve maximum credibility and acceptance.

U3  Information Scope and Selection  Information collected should be broadly selected to address pertinent questions about the program and be responsive to the needs and interests of clients and other specified stakeholders

U4  Values Identification  The perspectives, procedures, and rationale used to interpret the findings should be carefully described, so that the bases for value judgments are clear.

U5  Report Clarity  Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation, so that essential information is provided and easily understood.

U6  Report Timeliness and Dissemination  Significant interim findings and evaluation reports should be disseminated to intended users, so that they can be used in a timely fashion.

U7  Evaluation Impact  Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the likelihood that the evaluation will be used is increased.

Feasibility Standards The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.

F1  Practical Procedures  The evaluation procedures should be practical, to keep disruption to a minimum while needed information is obtained.

F2  Political Viability  The evaluation should be planned and conducted with anticipation of the different positions of various interest groups, so that their cooperation may be obtained, and so that possible attempts by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted or counteracted.

F3  Cost Effectiveness  The evaluation should be efficient and produce information of sufficient value, so that the resources expended can be justified

Propriety Standards The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.

P1  Service Orientation  Evaluations should be designed to assist organizations to address and effectively serve the needs of the full range of targeted participants.

P2  Formal Agreements  Obligations of the formal parties to an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that these parties are obligated to adhere to all conditions of the agreement or formally to renegotiate it.

P3  Rights of Human Subjects  Evaluations should be designed and conducted to respect and protect the rights and welfare of human subjects.

P4  Human Interactions  Evaluators should respect human dignity and worth in their interactions with other persons associated with an evaluation, so that participants are not threatened or harmed.

P5  Complete and Fair Assessment  The evaluation should be complete and fair in its examination and recording of strengths and weaknesses of the program being evaluated, so that strengths can be built upon and problem areas addressed.

P6  Disclosure of Findings  The formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation and any others with expressed legal rights to receive the results.

P7  Conflict of Interest  Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results.

P8  Fiscal Responsibility  The evaluator's allocation and expenditure of resources should reflect sound accountability procedures and otherwise be prudent and ethically responsible, so that expenditures are accounted for and appropriate

Accuracy Standards The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.

A1  Program Documentation  The program being evaluated should be described and documented clearly and accurately, so that the program is clearly identified.

A2  Context Analysis  The context in which the program exists should be examined in enough detail, so that its likely influences on the program can be identified.

A3  Described Purposes and Procedures  The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed.

A4  Defensible Information Sources  The sources of information used in a program evaluation should be described in enough detail, so that the adequacy of the information can be assessed.

A5  Valid Information   The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the interpretation arrived at is valid for the intended use.

A6  Reliable Information  The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable for the intended use.

A7  Systematic Information  The information collected, processed, and reported in an evaluation should be systematically reviewed, and any errors found should be corrected.

A8  Analysis of Quantitative Information  Quantitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A9  Analysis of Qualitative Information  Qualitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A10  Justified Conclusions  The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can assess them.

A11  Impartial Reporting  Reporting procedures should guard against distortion caused by personal feelings and biases of any party to the evaluation, so that evaluation reports fairly reflect the evaluation findings.

A12  Metaevaluation  The evaluation itself should be formatively and summatively evaluated against these and other pertinent standards, so that its conduct is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.

The Program Evaluation Standards. The Joint Committee on Standards for Educational Evaluation, James R. Sanders, Chair. Sage Publications: Thousand Oaks, CA. 1994.

Evaluation Continued. . . .

Getting answers to the evaluation questions.

There are many methods, each with their own uses, advantages and difficulties. Methods include:

·      Surveys

·      Analysis of Administrative Data

·      Key Informant Interviews

·      Observation

·      Focus Groups

Evaluations could use any, not necessarily all, of these methods, depending on the question and goal of the evaluation.

Surveys are a set of questions that are asked of everyone in the same way. Surveys can answer question about how many and how often. For example:

·      How many clients are satisfied with services?

·      How often do people have difficulties using the services?

Typical questions might be like this:

How satisfied are you with the program?

Very                Satisfied          Neither            Dissatisfied     Very

Satisfied                                                                                  Dissatisfied

Surveys might be used to describe the entire client population, if respondents were chosen randomly or systematically and if the sample is sufficiently large.

Randomly or systematically choosing people to respond to surveys means using some defined method to select people. For example:

·      Randomly choosing – generate a list of random numbers, assign each person a random number, sort the people by the random number and take the people listed first. They were put on top of the list randomly.

·      Systematic selection – a typical method is to start with the 5^th person and then select every 7th person after that. The numbers, the 5 th and the 7 th are also chosen randomly.

Randomly select locations to be in the sample, and then survey everyone in that location.

Random or systematic selection means that the group of people you select are more likely to be similar to your clients, in general. You aren't excluding any particular groups, or including only certain groups. You are avoiding bias, in sampling terms. If you do use random or systematic selection, then most likely you can use the results of your survey to make conclusions about your

clients. If you don't use random or systematic selection, you can NOT use the results of your survey to make conclusions about your clients population. That is, you cannot generalize from your study to your client population. You can only say “The people who took this

survey said ...”

Analysis of administrative data is just using statistical analysis on program data that is already collected.

Administrative data has advantages:

·      No new data collection is required

·      Many databases are relatively large

·      Data may be available electronically

Disadvantages:

·      Data were gathered for another purpose, so may not have necessary variables.

·      In all administrative data sets, some fields are likely to be more accurate than others.

Focus groups are structured discussions among small groups of people. Generally, a facilitator leads a group of 8-10 people in a discussion about selected topics with planned questions, while allowing for interesting, new or unplanned follow up questions.

Typical focus group questions are like these:

·      What is your overall impression of the program?

·      What are the things you like or dislike about the program?

·      What have you gained in this program?

·      If you have not noticed any changes in yourself, what do you think are the reasons?

Key informant interviews are qualitative, in-depth interviews of 15 to 35 people selected for their first-hand knowledge about a topic of interest.

Key informant interviews also include a planned set of questions on the topics of interest.

Key informant interviews are useful to when candid information about sensitive topics are needed. Group discussions may inhibit people from giving candid feedback.

Interviews should include a very diverse range of people.

Observations are methods that yield a systematic description of events or behaviors in the social setting chosen for study. Observation methods can be highly structured, for example:

Systematic Social Observation - a field research method in which teams of researchers observe the object of study in its natural setting. Researchers record events as they see and hear them and do not rely upon others to describe or interpret events. The researchers follow well-specified procedures that can be duplicated.

Observations can also be unstructured, for example, participant observation, or taking an active part in group activities. The premise underlying participant observation is that the researcher becomes a more effective observer by taking an active role in the performance of regular activities. In other words, knowledge gained through doing is of a higher quality than what is obtained only through observation. In many cases, involvement with ordinary chores will not only enhance the researcher's understanding of the processes, techniques, and words associated with these activities, but will also result in better rapport with informants.

Focus groups, interviews and observation are qualitative research methods, that is, methods that are less likely to rely on statistical analysis.

Advantages

·      Useful to help figure out major program problems that cannot be explained by more formal methods of analysis.

·      The evaluator may see things that participants and staff may not see.

·      The evaluator can learn about things which participants or staff may be unwilling to reveal in more formal methods

·      Useful when it's not clear what the program problems might be.

·      Useful to give good ideas of what topics program participants and staff think are important.

·      Useful in developing surveys, in determining what questions or issues are important to include.

·      Useful when a main purpose is to generate recommendations

·      Useful when quantitative data collected through other methods need to be interpreted.

Disadvantages:

·      The evaluator's subjective views can introduce error.

·      The focus of the evaluator is only on what is observed at one time in one place.

·      Information from observations/ interviews/ groups can be time consuming and difficult to interpret.

·      Focus groups could be dominated by one individual and his/her point of view.

·      Generally, information from focus groups, interviews, and observations CANNOT be used to describe the client population.