Part II: Evaluation Plan

Reader's Questions: To what extent are the data elements and data collection procedures clearly described and appropriate to measure the attainment of activity objectives and to measure the success of the project in achieving the goals of the comprehensive development plan? To what extent are the data analysis procedures clearly described and to what extent are they likely to produce formative and summative results on attaining activity objectives and measuring the success of the project on achieving the goals of the comprehensive development plan? 

The evaluation will include monitoring both the outcomes from the activity, as well as the processes for achieving those outcomes. The University will engage an external evaluator with extensive experience in the development and measurement of student learning programs. Although much of the information collection will be carried out by Millikin's Institutional Research Office, the Evaluator will assist University staff and faculty in the analysis process. 

Summative evaluation

Objective Indicators
By 2006-2007 increase freshman-sophomore retention from 79% to 83%.
Annual data collected by Institutional Research Office.
By 2006-2007 increase the six-year graduation rate from 61% to 70%.
Annual data collected by Institutional Research Office.
By September 2007 increase the freshman to sophomore retention rate for Exploratory Students and other at-risk students will increase to 83%
Annual data collected by Institutional Research Office.
By 2006-2007 increase to 100% the number of students who use self-guided tools for development of an Individual Plan of Study.
Use of software to measure level of use by students of web-enhanced process.
By 2007 reduce the gap between importance and satisfaction on the Noel-Levitz inventory from 1.07 to .50.
Noel-Levitz data from student satisfaction survey.
By September 2007 increase to 90% the number of faculty members who either 1) have an advising load that falls within a range defined in the Policies and Procedures manual, or 2) receive compensation or release time for advising loads exceeding that range.
Documentation collected by Department Chairs and provided to Project Coordinator in years following initiation of balanced advising loads.
By September 2007 increase to 100%, the number of first-year faculty who are fully conversant with the MPSL and student service resources by the end of their first year of service.*
Survey of all faculty in May of each year.

Formative Evaluation
The Title III Project involves a number of distinct processes that involve not only the accomplishment of specific tasks, but also the agreement by major stakeholder groups to participate in those processes. Advising will not change without the agreement of faculty to formalize the advising process through inclusion of standards in the Policies and Procedures Manual. Students and faculty must see the value of the Plan of Study if they are to begin using the process in a meaningful way. The processes in the Title III Project must be monitored to determine not only whether the work plan is being carried out, but also the barriers encountered and how those barriers are addressed.
During the Millikin Centennial Planning Process, the Provost's Office maintained a constant stream of communication to the University population at large. That level of communication benefited the planning process. The Title III Process will do the same, but use not only paper documents but also web-based processes for keeping stakeholders informed. Each of the major processes (such as Advising, MPSL Assessment) will involve a record-keeping process through which progress will be documented and posted to a web site. Members of the University community will be invited to comment on the process in progress and the comments will be reviewed to ensure that the staff responsible for each part of the process are aware of and respond to concerns expressed by members of the community.

The Evaluator will also be present on campus for five days during each year of the project. While on campus the Evaluator will work with the project's task forces in designing and assessing the formative processes for their specific responsible, as well as interview key personnel involved in implementation of the Title III Project. Individual and group interviews will be held with key staff and with task force members responsible for specific tasks. The Evaluator will also meet with small groups to students to monitor their response to the grant activities and document their perceptions of changes in advising, registration, support for at-risk students and the MPSL process. Interviews with key staff will assist in determining the degree to which the implementation process has gone as expected, what barriers were encountered and how College personnel overcame those barriers. 

The Evaluator will compile an annual report, as well as a final evaluation report. The Project Coordinator annually documents the modification of project activities in response to process observations made by the Evaluator. 

Brief Vitae: Karen Maitland Schilling, Ph.D.
Karen Maitland Schilling is Chair and Professor of Psychology at Miami University. She recently completed a multi-year, FIPSE- supported project involving seven mid-western universities on increasing expectations for student academic effort. She also recently conducted an evaluation of the multi-million dollar portfolio of grant projects on reform of the accreditation process supported by the Pew Charitable Trusts. 

Karen served from 1990-1995 as Miami University's first University Director of Liberal Education, overseeing implementation of a very broadly based curriculum reform project, and developing an approach to program assessment based in comprehensive student portfolios. She also directed the Women's Studies Program at Miami University. She has consulted at dozens of universities in the U.S. and abroad on assessment and curriculum development and has served for four years as a program faculty member for the Asheville Institute on General Education. 

Karen played a significant role in the development of the monograph, Strong Foundations for General Education from the Association of American Colleges and Universities. She also authored the book, Proclaiming and sustaining excellence: Assessment as a faculty role and several other influential articles and chapters on assessment in higher education, including "Looking back/moving ahead: Assessment in the senior year." In J. Gardner & G. Van der Veer (eds.) The Senior Year Experience: Facilitating integration, reflection, closure, and transition from Jossey-Bass in 1997.
Schilling is on the Editorial Board of the Journal of General Education, the Accreditation Review Council of the North Central Association, and she just completed a term on the Executive Board of the Association for General and Liberal Studies, Karen completed undergraduate work at Tufts University and her Ph.D. in clinical psychology at the University of Florida.