I often get asked about the use of technology in program assessment. In the mid-90's I was involved in the development of an electronic portfolio system that was used for students to document their work related to the performance indicators (see previous posting) and for faculty to rate student performance using scoring rubrics. It was also used to map the curriculum and generate reports. It was a huge investment in human and capital resources as this was before there were commercial products available. However, I remain very proud of the effort and the results.
Since that time there have been numerous commercial products developed to manage the outcomes assessment process. Is it necessary to buy commercial tools to have a robust outcomes assessment process? Absolutely not! Can technology solutions be helpful in managing the process and tracking the results? Absolutely.
There are lots of considerations when considering a technology solution to the outcomes assessment process. The first thing is to be very clear about what a system can and cannot do. It CANNOT do your program assessment and evaluation for you! The institution or program must first define the intended outcomes and performance indicators. Without a doubt, that is the most difficult part of the process. Once the indicators have been defined you need to be clear about the role of students and faculty in the use of the techology. Also, who is the technology "owner"--who will maintain it, keep the outcomes/indicators current, generate reports, etc. etc.
Get references! Talk to other institutions/programs to see what their experience has been using the technology. Don't just listen to the company sales people. Ask yourself, "What are we currently doing or not doing that the use of technology would make it more efficient or effective?" Use that as a starting point.
For your convenience, here is a list of SOME of the commercial tools available as a way to get started in your search. I have included two commonly used survey tools as well.
Technology for Program Assessment
July 28, 2010
Subscribe to:
Post Comments (Atom)
7 comments:
Effective assessment can be done without "technology" (to some extent), just as institutional book keeping, student enrollment management, accounting, inventory management, even basic communication. But who would advocate doing all the latter without technology? We recognize in those instances that we'd be in a real mess were we to go backward and try to do any of those key functions without reliance on the sophisticated, systemically-supported processes now afforded us through information management systems alone. Higher education tends to be in the Dark Ages of information management, even as we tell our students repeatedly, "Shift Happens"...
Brian is absolutely right. The biggest concern that I have is that programs/institutions will jump to a technological solution before they define the problem that they believe the technical solution will solve. The commercial products and well- marketed and lots of promises are made often without a realistic picture of costs/benefits. Buyer beware.
Several years ago in an online discussion with colleagues, I suggested that with the right information-management approach we could in fact gather and assess the evidence of student learning where and when it happens; with this assessed evidence we could immediately and simultaneously have deep insight into individual students' learning, as well as learning within courses, disciplines, programs, and across institutions. In almost these words I was told I must be a product of an hallucinating, '60's generation experience!
Besides being just a few years too young for that charge, I also have a fairly decent grasp of information management tools and processes, to know quite well the practical possibilities of what I am suggesting. The unfortunate part of the whole picture is that companies have an interest to sell their products, and their marketing often drives institutions to purchase technology that is appealing-- far ahead of institutions being able to understand the depth of what they (the institutions) might achieve, or what the currently marketed technology might deliver. And I would suggest that much of the technology falls far short of what we should expect of our learning-assessment potential.
Driven by accreditation anxiety, or external demands, or fear of the next wave of Federal expectations, or the lure of grant money for achieving some type of dreams etc., institutions are tempted to grasp at timely-marketed e-"solutions" (portfolios or otherwise) before understanding what kinds of possibilities each tool opens up, or forecloses, by the very things on which it focuses and which it promises to deliver. We need both a better understanding of what today's information technology is fully capable of delivering, and informed by this, a more up-to-date imagination of what we should be able to expect of learning assessment through the use of such technology. Otherwise--as I am afraid so many institutions are now experiencing-- we are stumbling through someone else's confusing marketplace of poorly understood "solutions."
Gloria, the use of an in-site developed IT solution for us, simplified things but also made us change our students outcomes, because we realized that they were not well defined. Do I have to inform ABET about these changes for our accredited programs?
Thank you Rebecca. No, this is a process improvement and as long as you can demonstrate that your outcomes are consistent with the ABET outcomes and that your students are attaining the outcomes, you are okay. You will want to report this process improvement under Criterion 4: Continuous Improvement. Let me know if you have other questions.
Gloria,
What do you think about assessing outcomes using team projects or team assignments?
Thank you
Remember, team projects or team assignments provide the context for the assessment process. Individuals demonstrate effective teaming skills. These can be best assessed in the context of the group/team experience. Having clear expectations (performance indicators) will enable students to focus their efforts and provide the assessor with clear guidelines for scoring.
Post a Comment