Procrastinate and Perish

As an experienced assessment professional, I often get calls to help institutions and/or programs with their student outcomes assessment plans as they are preparing for an accreditation visit.  Without exception (well, almost), my phone rings a year before the report is due.  Unfortunately this is an indication that this process is treated as an event instead of a process. 

A well-developed assessment plan is a continuous process.  This does not mean that you assess every outcome continuously.   What it does mean is that you have a systematic plan that enables you to smooth out of the workload over time.  There are well-defined cycles of data collection with defined timelines and areas of responsibility.  For example, if you have six learning outcomes and you have a three-year cycle, you would assess two outcomes every year.  This does not mean that there is no activity related to the outcomes for four more years.  To see an example of what a systematic cycle might look like, see an example table below:



Notice that data are collected in '10-11, evaluated in '11-12 and action taken with implementation of recommendations taken in '12-13 and a second cycle of data collection in '13-14.  The cycle is: outcomes defined into measurable statements (performance indicators), curriculum mapped, decision made about where to collect the data, data collected, evaluation of results and processes, recommendations for improvements, improvements designed and implemented, data collection. 

When last minute processes are developed, it becomes chaotic and puts an undue burden on faculty.  Also, it generally results in massive data collection which is difficult, if not impossible, to interpret.  Plan wisely!

Using Sampling for Assessment of Programs

One of the most important questions around program assessment is the one of data collection: how many, from whom, how often.  It is critical to remember that the process of assessing the program is about the program, NOT individual students.  We are trying to answer the question:  "How is the program doing in promoting student learning."  Whether we are talking about general education curriculum or a specific program, the issues are the same.

If the number of students in any program are sufficient, sampling is acceptable.  Good sampling techniques should be used and the sample being assessed should mirror the population of students on those characteristics which are identified as critical to justify generalizing the findings.  For example, if there are seven sections of a particular course where students are given opportunities to demonstrate their learning it may not be necessary to assess every section.  If the number of students in the population are not sufficient to justify sampling then you would be need to assess the performance of all students in the cohort.

Refer to my post  of May 22, 2010, "Data Do Not Always Information Make." for a discussion of how many data need to be collected.  In my next post, I will address the issue of "how often" to collect data.

Using Techology in Program Assessment

I often get asked about the use of technology in program assessment.  In the mid-90's I was involved in the development of an electronic portfolio system that was used for students to document their work related to the performance indicators (see previous posting) and for faculty to rate student performance using scoring rubrics.  It was also used to map the curriculum and generate reports.  It was a huge investment in human and capital resources as this was before there were commercial products available.  However, I remain very proud of the effort and the results.

Since that time there have been numerous commercial products developed to manage the outcomes assessment process.  Is it necessary to buy commercial tools to have a robust outcomes assessment process?  Absolutely not!  Can technology solutions be helpful in managing the process and tracking the results?  Absolutely.

There are lots of considerations when considering a technology solution to the outcomes assessment process.  The first thing is to be very clear about what a system can and cannot do.  It CANNOT do your program assessment and evaluation for you!  The institution or program must first define the intended outcomes and performance indicators.  Without a doubt, that is the most difficult part of the process.  Once the indicators have been defined you need to be clear about the role of students and faculty in the use of the techology.  Also, who is the technology "owner"--who will maintain it, keep the outcomes/indicators current, generate reports, etc. etc.

Get references!  Talk to other institutions/programs to see what their experience has been using the technology.  Don't just listen to the company sales people.  Ask yourself, "What are we currently doing or not doing that the use of technology would make it more efficient or effective?"  Use that as a starting point. 

For your convenience, here is a list of SOME of the commercial tools available as a way to get started in your search. I have included two commonly used survey tools as well.
Technology for Program Assessment

What is a "performance indicator" anyway?

In the discussion of the difference between data and information in my previous post, I indicated that in order to have meaningful data collection around student outcomes it needed to be focused on student achievement of specific performance indicators that define the program outcome.  What is a performance indicator?  The best way to think about performance indicators is to relate them to the concept of leading indicators used in economics.  Through experience, it has been found that there are certain characteristics that are the best indicators of the overall health of the economy.  Taken together, these indicators can provide information about the current state of the economy and also serve to predict future economic trends.  Although one could argue that there are lots of economic factors that one could look at to describe the state of the economy, experience has found that there are certain leading indicators that serve as the best describe the state of the economy.

So it is with outcomes of student learning.  A program does not need to look at every possible skill or knowledge element related to a given student outcome to know how well the program is doing on student attainment of the outcome.  Faculty need to have a good understand of their students, their faculty, their program educational objectives (what students should be able to do AFTER graduation), and the needs of their constituents and ask themselves, "How will we know when our students have achieved the desired outcome(s) which will prepare them for early career success?" 

 Every program in any discipline (not just technical disciplines) has as one of its student outcomes that they would like their students to be "effective communicators."  What does that mean?  Does being an "effective communicator" mean the same thing for a communications major as it does for a European history major?  Does it mean the same thing for a civil engineering major as it does for a chemical engineering major?  Does "writing skills" mean the same thing for a civil engineering program at X University as it does for the civil engineering program at "Y" college.  Probably not.  In order to develop meaningful performance indicators we need to think about how students will be using their communication skills related to the profession.  Performance indicators should be developed which identify the focus of instruction (content referent) and the level at which students should demonstrate their performance (cognitive/affective level).   Here is ONE example of performance indicators for "effective writing skills" (your indicators will undoubtedly look different).

• Students consistently use the rules of standard English (application level)
• Word choices are appropriate to the audience (evaluation level)
• Supporting details utilize appropriate graphical representation (application level)
• Organizational pattern is logical (application level)

Are there other performances we could use as indicators?  Absolutely.  However, we are only looking for those indicators which the faculty believe are the leading indicators of student performance related to the writing skills.  Measures are taken on the indicators indicate the degree to which the outcome is being met and provide information that can be used to improve student performance.

Data Do Not Always Information Make

In my last post, I indicated that it is not necessary to collect data in every course on every student in order to understand how well the program is doing in meeting its student outcomes. However, I continued to be amazed at the number of programs who have developed very elaborate data collection process which usually involve homegrown databases, spreadsheets, web applications, course reports and sometimes commercial software, etc.  Faculty members dutifully enter data related to the extent to which they believe the students in their courses have met the program student outcome(s).  Data are collected in multiple courses (sometimes all courses) and then aggregated, averaged and reported as the level to which students have demonstrated the outcome.  Targets of expectation are set (usually in the 70-75% range) and victory is declared.  Of course, all of this generally happens the year before the accreditation visit (so much for CONTINUOUS quality improvement).

It is important to remember that having data and targets for performance does not necessarily translate into information that can be used to improve the teaching/learning process.  Information means that you are able to look at the data and understand what the students strengths and weaknesses are related to the outcome--that you are able to discern from the data how well the program is meeting the outcome and what can be done to improve.   In order to accomplish this, the program needs to define the outcome into a few performance indicators and understand how the results inform the achievement of the outcome.  Data are collected related to the performance indicators and improvements are focused on improving student performance on the indicators (I will discuss performance indicators more fully in my next post).  This enables the program to focus the data collection (perhaps different courses for different indicators) and to clearly understand what student strengths and weaknesses are related to the outcome.  This will also enable the program to target improvement efforts and will set the stage for the next cycle of data collection and evaluation.  We need to stop the data dump approach to continuous quality improvement as it only promotes continuous faculty frustration.

Why all the course-level assessment?

When students are asked an open-ended question on an exam and they don't know the answer, they tend to write everything they do know that they weren't asked...and they write and write and write. Why do they do that? Probably for one of two reasons: they hope the faculty member will find the right answer in all their ramblings or they will be given extra "points" for their effort. Why do we approach program assessment the same way? I can hear it now, "We don't know what the accreditors want, so we will just give them everything."

This approach always ends up in creating undue work on the part of faculty and, while producing massive amounts of data, produces absolutely no information that can be effectively used to improve the student learning experience. We have lost sight of the fact that we are assessing the "program" not the student. Yes, we gather evidence from students but it is not for the purpose of assessing them--we are already doing that in their courses. It is for the purpose of assessing ourselves. Can we provide evidence that by the end of the academic program students have attained the ability to (fill in the blank)? If we believe that student learning is cumulative over time and that what is learned in one course is applied in another course, built upon in another course, etc., throughout the curriculum, then by the time students are ready to graduate, their learning should be more than the sum of all their courses. Why do we collect data in lower level courses and average them with the data taken in upper level courses and pretend like we know what they mean? Are we really saying that all courses are equal in how they contribute to cumulative learning and that the complexity and depth/breadth at which students are to perform is the same in all courses for any given outcome? Why not only collect "evidence" of student learning in the course where students have a culminating experience related to the outcome. Yes, collecting evidence in a lower-level course would be helpful in understanding student strengths and weaknesses related to any given outcome. This would enable faculty to reinforce and emphasize those concepts where students were weak prior to the culminating experience. However, these data should not be aggregated with the data collected in the culminating experience. Why do you need more than one data point for each student from whom you have collected data?

We need to bring some sanity back into what we are doing or the entire "outcomes assessment" process will collapse under its own weight--as well it should.

Accountability not going away!

In the April 13, 2010, Inside Higher Education article, "No let up from Washington" (http://www.insidehighered.com/news/2010/04/13/hlc), Molly Corbett Broad, president of the American Council on Education, is quoted as saying, "...I believe it’s wise for us to assume they (federal policy makers) will have little reservation about regulating higher education now that they know it is too important to fail." This was in the context of holding higher education accountable for learning outcomes.

Are institutions really prepared or have they been dragging their feet hoping that the change in adminstrations would make it all go away? It is clear that institutions and educational programs must get serious about demonstrating student learning and do it in a way that honors faculty time and produces information and not just tons of data that remain a mystery as to what they mean. This also means that accrediting agencies must get serious about preparing their peer evaluators to "know it when they see it." As the feds put pressure on the accrediting agencies to demand accountability for student learning, the accrediting agencies need to become more intentional in the preparation of their peer evaluators who, for the most part, are faculty and adminstrators of higher education.