Monday, November 25, 2013

Results from Sept 24-25 Trainings - Feedback

I don't know if you have gone on the SkyDrive website to look lately, but Dr. Grandgenett posted the results of the flip charts regarding the nine (9) outcomes of Project IMPACT. I would appreciate any feedback on the comments. 

I'm scheduling site visits to the five community colleges in December to solidify the data collection processes, including data sharing agreements, types of data and the format, collection protocol, and other logistical issues (data storing, coding, and transmission). If you have any suggestions for any of the processes, please let me know. We will be scheduling a phone conference for January 2014 to update all of you on the status of the project. 

Results from Sept 24-25 Evaluation Trainings
  
1. Total Unique Participants Served

Cumulative total number of individuals entering any of the grant-funded programs offered

Year 1:                       0
Year 2:                       365
Year 3:                       473

Total:                         838

Comments:
-       Seems small, is entire regions for the 5 community colleges represented?
-       A library of definitions for acronyms be available and understood by the consortium.
-       How does this number compare to state need?
-       We will track all our participants but what if only a subset or our participants receive coaching?  Do we report these groups together or separately?
-       Will employers have a path for employers to be part of it?
-       Why can’t existing curriculum by used rather than redeveloping what already exists?
-       How will students be selected?
-       Do we count the comparison group as participants if they are taking the courses we create, but not receiving other services?
-       Will this also be broken down into specific programs?
-       Need to define “participant”.  Is this someone who takes any class offered?  (assume yes)
-       Can a institution count  training such as safety 10 hour, although the current employers are not enrolled in a certificate or degree?
-       I’m discussing ROI the assumption is a pay increase.  What is it going to take to convince industry the KSA’s are worth the investment?

2. Total Number of Participants Completing a TAACCCT-Funded Program of Study
Number of unique participants having earned all of the credit hours needed for the award of a degree or certificate in any grant-funded program

Year 1:           0
Year 2:           280
Year 3:           310

Total:             590

-       Question:  Earned all credit hours or actually applied for/received an award?  Within a specific period of time?
-       From a numbers perspective, defining TAACCCT funded program of study is key to capturing appropriate data.
-       We can count certificate level completers, correct?  X2
-       Any effort to promote life long learning from certificate to diploma to AAS.
-     Are we taking participants or completers?  (Perkins Terminology)

3. Total Number of Participants Still Retained in Their Program of Study or Other TAACCCT-Funded Program

Number of unique participants enrolled who did not complete and are still enrolled in a grant-funded program of study
Year 1:                       0
Year 2:                       90
Year 3:                       77

Total:                        167


-       How long does the retention period last?  When are the participants no longer in the program?
-       Do we count students who completed certificate and are still working toward associates?
-       Is there a time frame for these participants?  What if a funding for program isn’t continued?
-       Can we track to determine why and encourage reinvolvement ?  Soft skill help.

4. Total Number of Participants Completing Credit Hours

Total number of students enrolled that have completed any number of credit hours to date.

Year 1:                       0
Year 2:                       308
Year 3:                       420

Total:                         728

-       How does the KSAO testing take into consideration disabilities (ADA), results, disclosure of information to students, coaches / instructors?
-       Who, what, when, where, and why on workshops.
-       Is number of completed credit hours important?
-       Big jump between year 1 and 2 – maybe slower start up.
-       Why is DOL asking for this?  What is the big question being addressed here?

5. Total Number of Participants Earning Credentials

Aggregate number of degrees and certificates completed by participants in grant-funded programs of study

Year 1:                       0
Year 2:                       337
Year 3:                       452

Total:                         789

-       Differentiate between degrees and certificate.
-       Does this have to be unique users?
-       Is the CSHA 10 hour a credential or simply a participant?
-       We presumed we would count multiple credentials earned per student.  Ok?
-       At Topeka TAACCCT conference, one grant recipient said they could only count each participant once.
-       How will credentials be defined?  How will industry certifications be tracked?
-       Defining credentials is key – keeping a database of base information might be important in case the definition changes.
-       At Topeka TAACCCT conference, one grant recipient said they could only count each participant once.

6. Total Number of Participants Enrolled in Further Education After TAACCCT-funded Program of Study Completion

Total number of students who complete a grant-funded program of study and enter another program of study

Year 1:                       0
Year 2:                       46
Year 3:                       46
Year 4 (follow-up only): 51

Total:                       143

-       Colleges can request data through the national clearing house.
-       Of those tracked, how many choose higher level education.
-       Does this apply only to students transferring for a BS, or can it be another program such as welding once they’ve achieved a certificate?
-       Does the student have to declare their degree or does it count if the student completes a certificate, then continues on for a diploma?


7. Total Participants Employed After TAACCCT-funded Program of Study Completion

Total number of students (non-incumbent workers) who completed a grant-funded program of study entering employment in the quarter after the program exit

Year 1:                       0
Year 2:                       159
Year 3:                       177
Year 4 (follow-up only): 111

Total:                           447


-       What effort can be carried out to assess future promotibility?
-       How does low Nebraska unemployment impact this?
-       Can we include individuals employed following non-credit study leading to MSSC credentials?

8. Total Number of Participants Retained in Employment After Program of Study Completion

Total number of students (non-incumbent) who completed a grant-funded program of study and who entered employment in the quarter after the quarter of program exit who retain employment in the second and third quarters after program exit

Year 1:                       0
Year 2:                       145
Year 3:                       158
Year 4 (follow-up only): 99 

Total:                          402

-       DOL track for employees still employed?
-       How will program be sustainable once put into a program that already exists?
-       Is this employment with the same employer or is it the same job with multiple employers?
-       How would you schedule students at different institutions to get the use of the physical machine that will be purchased?
-       How will employment / wage information be tracked?  Access to data?
-       Does DOL have a tracking program after the first quarter?  They have in the past.
-       DOL MOU’s

9. Total Number of Those Participants Employed at Enrollment Who Received a Wage Increase Post-Enrollment

Total number of students who are incumbent workers and enrolled in a grant funded program who received an increase in wages after enrollment

Year 1:                       0
Year 2:                       107
Year 3:                       121
Year 4 (follow-up only): 52

Total:                        280

-       Does this information absolutely need to be validated?
-       Important to consult employers.
-       Can the wage increase be simply an annual increase or does it have to be specific to the skills learned thru the grant or certifications?

Wednesday, November 6, 2013

"Comparison Groups"

I realize the phrase "control groups" raised some ethical and pragmatic issues. It implied that some individuals who potentially could benefit from any of the programs offered by Project IMPACT would not be admitted, even through they qualified. That practice runs contrary to the mission of the community colleges. In fact, it was never a viable option, but in the brainstorming process, it was included.

The real issue is finding a "comparison group," to demonstrate that as a result of completing the manufacturing generalist certification and possibly continuing through the laddered options, participants land employment in high wage, high skill, and high need organizations. And that the coursework did indeed make a difference in acquiring a position, being retained and increasing their wages.

In order to find that potential comparison group and collect data, not only on Project IMPACT participants, but also on the comparison group, we need to complete data sharing agreements between the community colleges and Shain Evaluation & Consulting (for educational data), and the colleges and the Nebraska Department of Labor (for employment data). Those discussions are in process.

In addition, we need to "lock down" the comparison group; one method would be to examine the results of previous years (2011-2013) with students in coursework similar to those being offered by Project IMPACT. Another would be to work with several employers who would be willing to share data on new employees not affiliated with Project IMPACT and those new employees who have completed any offerings by the project. The latter would take considerable effort by the employers but could yield vital qualitative data, in addition to quantitative data (and would be the most interesting!). A third method might be to compare employment results from the various "layers" of completion of Project IMPACT participants, to compare those who leave after one course or certification with those who complete a diploma or AAS. If you have other ideas, please let me know. I want us to have a process in place by the end of the calendar year.