Interesting article on assessment

Our Assistant Vice President (library director) forwarded me this article recently: What Are They Learning? Pre- and Post-Assessment Surveys for Libr 1100, Introduction to Library Research" by Jon Robert Hufford. It's from College and Research Libraries preprint articles page, which incidentally has a feed.

It's an article about the experience of a group of Texas Tech librarians who administered a pre- and post-test to their library research course. Although our situation is a bit different in several significant ways, I found it very helpful, and very much wish I might have had a chance to read it before we got started in our assessment efforts.

• Elapsed time--from what I understand, they tested their students at the beginning and at the end of a semester. In our assessment, we pre- and post-test one shot classes at the beginning and the end of class. Although ours is different, its very illuminating to compare results of a semester-long class as opposed to a 50 minute session.


• There is a useful literature review in the beginning, though elsewhere the author laments the fact that there are not many articles written in the literature on this particular subject, and near the end of the article encourages other people to contribute with their experiences.
  

They included an analysis of their questions and the responses, which was also highly useful.

Assessment fun with clickers; and mobile phone clickers too

And welcome back from the hiatus--I don't know where the semester went! Please forgive the retro-post, getting back into the saddle and finishing up some of the posts I started and didn't quite publish!

We are continuing to focus on our assessment efforts. We have our big assessment project involving first year English composition students. We crunched the numbers with that and also going forward with revising our tests again. I will save that for a later post, but we did do our first pilot assessment using our Senteo/Synchroneyes clicker software and equipment as part of our continuing efforts to extend assessment into other classes. We are targeting next our Freshman Seminar classes.

I am looking forward to trying this out. I did not personally do the pilot clicker assessment but my colleague indicated that he felt it went very well. It was a simple post-test consisting of three questions, given at the end of the instruction session. I can say that I handled the statistics for the class. Synchroneyes/Senteo has a very decent system for results--they come out in a neat graph with basic statistics like class average and percentage answering correctly in an Excel file.

The only drawback is that the statistics do not appear to be cumulative, and so to extend this to all our other Freshman Seminar classes we'd have to evolve a system to name each Excel file with the name of the librarian, class, and possibly date or time, and then save in a shared folder. And then later if I wanted to add up the statistics for all the students taking that post-test, I would have to combine the Excel files....

If anyone out there has used Synchroneyes/Senteo and has a better approach, please please do let me know!

Another intriguing development is the use of web-based software so mobile phones/smartphones can be used as clickers. (an example would be Polleverywhere) I did a very desultuory search on Google and there were a few links. One of the things that came to mind immediately as a possible use for this technology would be our massively large Freshman Orientation sessions.

The fall orientations usually last 3 days, working about 3000 students in batches throughout the day. We generally show them several videos, and the last few years we have given them a paper "quiz" which we collect at the end of the session. It's quite rough to tabulate so many "quizzes" and so it seems having the students vote on the answers to various simple questions tied to the outcomes for this orientation, with a chart displayed on the big screen of the teaching theater where the orientation sessions happen would bring a welcome sense of interactivity to the library portion.

I am not quite sure given the scale of this orientation, that it might work well, but it's definitely an interesting idea..........:) and one I hope might come to pass eventually.

ACRL 2009 - the assessment sessions

Hello. I am back with more about some of the sessions I attended at ACRL. The last post was about some useful technology for researching and creative instruction ideas. This post will cover assessment topics that were presented at ACRL 2009 in Seattle.

I attended the Fishing for Information: Using Focus Group Research to Discover Student Perceptions of Library Services and Resources The presenters, Rebecca Byrum and William Wearefrom from Valparaiso University in Valparaiso, Indiana had 3 volunteers from the audience pretend to be their example focus group to demonstrate the methods used for a focus group study. The presentation was active rather than passive since they were using the same type of questions for the volunteers as they did for their original focus group. The original focus group was conducted to find out what the students needs were in the library.

They pointed out some good things to know when you are planning a focus group study:

• Need to get and oral and written consent from the participants
• Names of participants should not be revealed
• Let participants know that results will be used for presentation
• Make participants feel comfortable, so they will be honest
• Use clear simple questions
  

Here they types of questions they suggested to use:

1. Intro Question
2. Transition Question
3. Key Question - grade certain characteristics of topic (report card)
   
4. Discussion of report card - Allow participants to drive questions, e.g. "Are there any other categories that you would like to assign a grade to?"
   
5. Open question - "Do you have anything else you want to say or tell us?"
   
6. Ending Questions - Summary, "Have we missed anything?"

Summarize the results and you have done a qualitative study that you can use to improve your services with. Thanks for the good practical assessment ideas!

More on assessment in the next post.

Back from Hiatus

And I am terribly sorry for the delay. It's been a bit of an interesting summer. I went to Hungary and the Czech Republic for vacation most of July, and then it was pretty much straight to Denver at the beginning of August for the Reference Renaissance conference. (I think i will save some of the presentations and our presentation for a later post).

Assessment

But there are lots of interesting and cool things to catch up on. Over the summer, we continued to work on our Assessment efforts for our second-semester Freshman English course. Since we only had two classes it gave us a chance to test the Synchroneyes quiz/test tool a little more comfortably. In the contest of Synchroneyes vs. Survey Monkey, my admittedly non-scientific preference at this time is for Survey Monkey, only because when I used Synchroneyes I had to print out every single survey.

I wasn't able to make it to the LOEX of the West conference, but my colleagues did, and they told me about a great presentation from University of Texas Libraries(which is about 30 minutes away up I-35) called "Assessing out Assessment: Failures and Successes at UT-Austin." I think it is proving to be helpful for understanding and planning our assessment effort, and I wanted to give a shout-out to AJ Johnson, who let me ask questions about their presentation/project and generally let me pick his brain.

And, along other lines, we are going to experiment with using "clickers" to do assessment. We have used them in some of our classes as part of our active learning activities. With that in mind I have been reading this article: Clicking your way to library instruction assessment.

Embedding Librarians in TRACS (SAKAI)

I know this is already something that has been happening in other places, but this semester we are going to make an effort to "embed" ourselves in the course software. We have done some things in the past, with a few classes, mostly building wiki pages using the course management software wiki function (as distinct from our wiki) But we'd like to try more classes and do more, like embed a meebo widget in the class site. So stay tuned for that.

I think I will stop for now. :)

Of classroom software and assessment

Although it is the May mini-mester and there are relatively few students we are keeping busy with various projects. Next week will be a bit more busy with several workshops for faculty and also for IL librarians as well. I will be conducting a short training (something of a refresher course) on our classroom equipment. We have a SMARTBoard, along with Synchroneyes classroom control software and Senteo clickers. We are all at different levels of comfort with these tools.

I would very much like to hear from anyone who uses any or all of these tools, and any interesting tidbits about using them in the classroom. One of the things we noticed about the SMARTBoard is that because our classroom is good-sized, it can be difficult for visibility with large classes. I myself love to use it with smaller classes, in particular with ESL classes. I have done a desultory search on google for more (Synchoneyes and assessment in particular) but haven't found much in the literature as of yet.

We will be testing the Synchroneyes quiz function this summer hopefully to see if it is better or easier to use for our Freshmen English assessment. We have gotten through two semesters with Surveymonkey, which works pretty well, but is a bit awkward in terms of delivery (the teacher has to send the quiz link to the students). The other consideration though, is the ease of availability of the raw scores/results. Surveymonkey has a download feature that lets you get the raw scores and results along with other information in an excel file. It has also been helpful to be able to print out the surveys in case of technical malfunction, and then enter in the paper responses manually, which is another feature of Surveymonkey.

At any rate, I foresee a future blog post once I know more, comparing one and the other.

An idea for Google notebook, and catching up

I was noodling around and rediscovered Google Notebook again. I had noticed it last year (judging from the date of the little snippet I had saved) and most probably forgot about it. It seems to have been around for a while, and probably people are already using it. It just occured to me as I looked at it again that maybe it might be a good tool to use for an assignment for evaluating websites. Students could be assigned several websites, and they could evaluate each assigned site and make notes in their notebook, and then share them with the class. The only hitch is that the students would need a Google account. I did a quick search to see if anybody else is doing something like this--I found this blog post written by Laura Pressley of Wake Forest University referencing a presentation by Jennie Hunt. I am curious how that class/assignment worked out. I am sure to find more if I look deeper, but in the meantime I am thinking it might be an interesting activity to develop, possibly.

I am considering to test it as an activity with an ESL undergraduate class, that is normally a two part class, and only about 10 students.

Interim/Summer is always a very busy time, because although there are less students, we are preparing projects for upcoming semesters and finishing up old projects. Our team has gone through two semesters of assessing freshman year English composition classes, and so currently we are revising the survey and looking to investigate using either our class management software TRACS or Synchroneyes as possible alternatives to using Survey Monkey. I know there have been gallons of ink spilled on assessment already, and really all I venture to add to this vast knowledge is the observation that reading about how other people do assessment is helpful but there is no better learning experience than putting a project into practice. I think i will save a detailed description for a separate post.

We are also setting up several workshops/trainings for staff which should be interesting. One will be on using our our SMARTBoard, Synchroneyes (classroom control software), and Senteo (clickers). Another will be a Web 2.0 presentation for staff. So, looking up to be a busy May.