Observations on the 2012-2013 Annual Assessment Report
FY 2012-2013 was the library’s second year of collecting and reporting data as part of the SACA-mandated process. It was also the year in which the library underwent its first Periodic Review. While the library has always valued good assessment processes, and always will, the workload of doing quality assessment while serving a growing campus population with a library staff that is not growing is becoming a serious problem.
The UCM Library’s 2012-2013 Annual Assessment (AA) Plan included 6 goals, 19 outcomes, and 46 measures. The library collected useful data for 36 of those measures (78%). The measures and data are listed below. As is always the case, data that answer the principle question of, “What is the impact of the UC Merced Library on research and student success?” is always the most elusive and labor-intensive to collect. In reviewing the results of our 2012-2013 assessment efforts, we realized that we will be better off putting concentrated effort into fewer measures, and as a result have adjusted our 2013-2014 AA Plan to be more focused. Of course measures that require little effort on the part of library staff will continue to be reported.
Nothing in the data gathered this year sounds any alarm bells in terms of the quality of the information resources and services the library provides; however, there is data to suggest the library is reaching the limits of it capacity to provide these desiderata. When faced with capacity issues, the options are to 1) find more resources to do provide more of what is wanted, 2) find more efficient ways to provide what is wanted, 3) stop providing something that is wanted less and apply those resources to provide what is wanted more. Assessment data is, of course, one of the guideposts the library uses when making decisions about how to best apply the resources at its command.
Some highlights from the Report:
A
Measure A1
The UC Merced Library will annually compare its opening hours with those of other UC Libraries with the goal of being at or above the UC Systemwide median for total hours open.
UC Merced Library's opening hours fell right in the middle of opening hours for all UC libraries. UC Merced Library's opening hours were exactly the same as UC Davis Library's hours.
Responsibility: Eric Scott
Measure A2
The UC Merced Library will measure building occupancy rates, tying measured occupancies to day of the week, time of day, and library locations. The Library Access Services Department will conduct daily occupancy counts on nights and weekends; it will conduct weekday counts from 2 to 4 times per semester. The library will analyze this data and consider it in planning the scheduling of operating hours, staffing levels, and furniture acquisition.
The library collected data for October 2012 and April 2013.
| April Averages 2012 and 2013 | October Averages 2012 and 2013 | |||||||||||
| Days Included | Time | 4/2012 Averages | 4/2013 Averages | Days Included | Time | 10/2011 Averages | 10/2012 Averages | |||||
| M-F | 12:00 PM | 361 | 478 | M-F | 10:00 AM | 315 | 298 | |||||
| M-Sun | 2:00 PM | 335 | 365 | M-Sun | 12:00 PM | 327 | 475 | |||||
| M-Sun | 4:00 PM | 333 | 392 | M-Sun | 2:00 PM | 344 | 433 | |||||
| M-Th & Sun | 6:00 PM | 297 | 319 | M-Th & Sun | 4:00 PM | 323 | 412 | |||||
| M-Th & Sun | 8:00 PM | 229 | 190 | M-Th & Sun | 6:00 PM | 246 | 278 | |||||
| M-Th & Sun | 10:00 PM | 171 | 104 | M-Th & Sun | 8:00 PM | 176 | 198 | |||||
| M-Th & Sun | 11:00 PM | 79 | 67 | M-Th & Sun | 10:00 PM | 146 | 127 | |||||
| M-Th & Sun | 11:00 PM | 70 | 59 | |||||||||
Measure A3
Seventy-five percent of students in focus groups will respond that they are "satisfied" or "very satisfied" with the library's physical resources. Findings inconsistent with this goal will be examined for space planning implications.
Responsibility: Eric Scott
Although the library was not able to pose this exact question to focus groups, we did collect the following relevant data:Graduating Student Survey Results for 2012-2013
The individual study spaces in the library supported my academic progress.
75 + 175 / 321 = 77.9%
The collaborative study spaces in the library supported my academic progress.
124 + 174 / 334 = 89.2%
Measure A4
The UC Merced Library will collect data on the number and type of events hosted in the library with the goal of creating a longitudinal record of the use of the library.
Responsibility: Eric Scott
table { }td { padding-top: 1px; padding-right: 1px; padding-left: 1px; color: black; font-size: 11pt; font-weight: 400; font-style: normal; text-decoration: none; font-family: Calibri,sans-serif; vertical-align: bottom; border: medium none; white-space: nowrap; }.xl63 { border: 0.5pt solid windowtext; }
| KL 355 (Event Space) | 36 |
| KL 155 (Event Space) | 24 |
| KL 155 Tabling (inside) | 47 |
| KL 155 Tabling (outside) | 70 |
| KL 474 Meeting Room | 1251 |
| KL 397 Meeting Room | 826 |
| KL 362 Conference Room | 578 |
| KL 361 AV Room | 313 |
| KL 360 Meeting Room | 914 |
| KL 470 | 1245 |
| KL 464 | 1271 |
| KL 376 | 1139 |
| KL 374 | 1175 |
| KL 372 | 1170 |
| KL 368 | 1152 |
| KL 276 | 1198 |
| KL 274 | 1244 |
| KL 272 | 1244 |
| KL 268 | 1240 |
| KL 266 | 1249 |
| KL 264 | 1276 |
| KL 262 | 1307 |
B
Measure B1:
Provide information-literacy instruction to over 50% of WRI 10 sections.
Responsibility: Sara Davidson
Fall 2012: 60%
Spring 2013: 32.7%
Combined: 43%
Measure B2:
Fulfill 100% of requests for instruction.
Responsibility: Sara Davidson
Measure B3:
At least 70% of incoming freshmen students will have taken the iPod Touch Tour between the start of fall semester and the middle of spring semester.
Responsibility: Sara Davidson
Actual: 24.99% (37% of freshman participating in fall 2012 and then 3.5% in spring 2013) (did not meet measure)
Details at Source: Picon > User Comm and Instruction > Assessment > iPodTouchTour > Cumulative > ipod_stats_score_participate.xlsx > [Tab]: WRI 1 & 10 Data
Measure B4:
Eighty percent of students in WRI 10 who receive library instruction will indicate through their portfolios that library instruction helped them successfully find information for their research papers.
Responsibility: Sara Davidson
In summer 2013, librarians evaluated the "Research Ethics" section of 30 Writing 10 online portfolios using a newly revised rubric. The rubric was designed to assess the impact of library instruction on student use of the library's resources. Though students are not required to refer to the library in the cover letter, it is included as a possible topic for inclusion in the "Research Ethics" cover letter. However since it is not required, students who do not write about the library may have found its service helpful even if they did not express it.
In this evaluation round, five librarians read and scored the cover letters. Every cover letter was scored by two librarians; those scores were then averaged. For each item evaluated, a student could receive a 4 (highly evident) to 1 (minimal/no evidence). For this measure, we used the item of "Library Use & Impact" from the rubric and took a score of 3 or higher to indicate that a student found that the library did assist them in completing their academic work (whether or not they specifically mentioned instruction). This year we did not meet this measure. Only 43.3% of those evaluated referred to or mentioned the library (information resources, space, services, instruction etc.) as helping them to complete academic work. This could be influenced by a number of factors, and we plan to define and refine this process of evaluation.
Measure B5:
Eighty percent of students who take the iPod Touch Tour will indicate on the assessment that their knowledge of the library's space, services, and resources is "good" or "excellent."
Responsibility: Sara Davidson
Actual: 91.99% (measure met)
Details at Source: Picon > User Comm and Instruction > Assessment > iPodTouchTour > 2012-2013 > StatisticalReports > stats_survey_12-13.xlsx > [Tab]: Combined Data
Measure B6:
Seventy-five percent of the students who complete the iPod Touch Tour follow up assignment will receive scores of "high" or "perfect."
Responsibility: Sara Davidson
Actual: 88.2% (measure met)
Source: Picon > User Comm and Instruction > Assessment > iPodTouchTour > Cumulative > ipod_stats_score_participate.xlsx > [Tab]: WRI 1 & 10 Data
Measure B7:
Students in focus groups who received instruction from the library or used library guides in their course work will report that these were helpful.
Responsibility: Sara Davidson
The library was unable to collect this data.Measure B8:
Eighty percent of students who respond to questions on library research skill will report that their current ability is "good," or "very good," or "excellent."
Responsibility: IPA/Sara Davidson
UCCUES is distributed biannually in the spring of even years. In the 2012 results for UC Merced, 80% of respondents rated their level of proficiency in Library research skills as “good”, “very good”, or “excellent.
Measure B9:
Eighty percent of the students responding to the graduating senior campus survey who received library instruction will "agree" or "strongly agree" with the statement "The library research instruction I received helped me find suitable information resources for my papers and projects."
Responsibility: IPA/Sara Davidson
Graduating Student Survey Results for 2012-2013
When I asked for help finding information for my papers and projects, I received satisfactory assistance from library staff.
233/253 = 92.1%
Measure B10:
Develop a method to automatically gather and report data on use of the UC Libraries Systemwide Tutorial by UC Merced students.
Responsibility: Sara Davidson
In spring 2013, four sections of WRI 10 for a total of 66 students completed one or more sections of The Research Process Tutorial series. Library staff was able to see which quizzes were completed and individual scores. This information was made available in a downloadable spreadsheet. Individual students scores were also made available to Writing faculty who wanted to view participation rates and student scores for classroom grading purposes.
Measure B11:
The library will collect baseline statistics on the number of users visiting the library's website, LibGuides, and Facebook pages.
Responsibility: Sara Davidson
Facebook "likes" grew by 152 over the course of the year.
Library Website Visits for Fiscal Year 2012-2013
Unique Visits (Number of people who’ve visited the site): 103,123
Total Visits: 194,833
% New Visits: 52.90%
Pageviews: 425,229
Pages/Visit: 2.18
Average Visit Duration: 3:09
Summary: The library website is the virtual door to library resources and services. The most popular pages on the site are those that connect library users with databases and other search tools, and those that provide information about library services such as hours and room reservations. The data suggesting that users visit an average of two pages on the site, and stay on the site just over three minutes are actually an indication that users are successful at finding the information they need, or are using the site for short periods of time to connect to databases hosted elsewhere.
Libguide Visits for Fiscal Year 2012-2013
11,924
44 of these visits were via a mobile device
Measure B12:
The library will collect baseline data from students in focus groups indicating how they have learned about the library's resources and services.
Responsibility: Sara Davidson
The library was unable to collect this data.C
Measure C1:
The per-capita use of 24/7 digital reference service by UC Merced students will be above the mean for all UC campuses.
Responsibility: Elizabeth McMunn-Tetangco.
The UC Merced Library is 4.8% above the systemwide average.
Measure C2:
Eighty percent of UC Merced users who complete the 24/7 digital reference service feedback form will report that they will use the service again.
Responsibility: Elizabeth McMunn-Tetangco.
We did not quite meet our target of 80%.
76.47% report they will very likely use this service again. 17.65% report they maybe will use this service again.
Measure C3:
The library will analyze verbatim transcripts of 24/7 sessions initiated by UC Merced students. The goal is to use this data for continuous improvement of the library web site and user interfaces based on actual questions from library users.
Responsibility: Elizabeth McMunn-Tetangco.
The library was unable to collect this data.Measure C4:
The library will conduct usability testing of the library website with new library student assistants looking for any patterns that indicate either the extent of success or failure on the part of information seekers. The goal is to use this data for continuous improvement of the web site and user interfaces based on actual use.
Responsibility: Susan Mikkelson
The library was unable to collect this data.The library will analyze data gathered by the LibAnswers software to identify repeat questions or any pattern of problems that student encounter when search for information or using the library. The goal is continuous improvement of the UC Merced Library web site, user interfaces, services, and instruction.
Responsibility: Susan Mikkelson
In April 2013, library staff looked at SpyQuery data in LibAnswers to determine what types of questions were asked that did not retrieve a user a FAQ response. A librarian analyzed the SpyQuery information and based on this data made changes to existing questions, tagged those questions for greater accessibility, and added new FAQs as needed.
Measure C6:
The library will gather a random-sample of data concerning the number and complexity of questions asked at the Library's Helpdesk and Service desk. The library will analyze this data to insure that staffing levels are appropriate based on day of week and time of day.
Responsibility: Joe Ameen
Measure C7:
The Library's Roving Reference student employees collect data on the number of questions they are asked, the time and day on which the questions are asked, and the complexity of the questions they are asked. The library will analyze this data to ensure that levels of Roving Reference staff are adequate.
Responsibility: Robin Milford
During the 2012-2013 school year not all Roving Reference student assistants collected data on the number of questions they were asked, the time and day on which the questions were asked, and the complexity of the questions they are asked. As such, the transactions that were recorded are not an accurate reflection of the overall Roving Reference transactions during the year.
The Roving Reference program was under transition this year as Robin took over management of the program from Sara in November 2013. Robin observed Sara training in January 2013 and completed her first round of student training in May 2013.
Beginning in August 2013, Access Services will begin using Gimlet to track all interactions at the Help Desk (1st Floor) and 2nd Floor (2nd floor). Gimlet is a web-based tool licensed to record statistics and information about encounters at library service points, similar to Ref Analytics, but with the option of single sign-on and simpler user interface which makes entering content easier for student assistants. Roving Reference students will also use Gimlet to record reference transactions. Once all student assistants are required to record all transactions (both directional and reference) at all service points and while roving, it will hopefully become a more integrated/inherent aspect of the Roving Reference student assistant workflow.
Measure C8:
The Library will have at least one career staff member on duty at all times the library is open including nights and weekends.
Responsibility: Eric Scott
The library was unable to meet this goal due to budget limitations.Measure C9:
Seventy-five percent of individuals who respond to the Graduating Student Survey will "agree" or "strongly agree" that they received satisfactory assistance from library staff at library service desks.
Responsibility: IPA/Sara Davidson
Graduating Student Survey Results for 2012-2013
When I asked for help at a library service desk, I received satisfactory assistance from library staff.
303/318 = 95.3%
Measure C10:
Seventy-five percent of individuals who respond to the Graduating Student Survey will "agree" or "strongly agree" that they received satisfactory assistance from library staff when they asked for help finding information for their papers and projects.
Responsibility: IPA/Sara Davidson
Graduating Student Survey Results for 2012-2013
When I asked for help finding information for my papers and projects, I received satisfactory assistance from library staff.
233/253 = 92.1%
Measure C11:
Library staff will provide a minimum of three events to faculty promoting economically sustainable publishing.
Responsibility: Susan Mikkelsen
The library sponsored three events for faculty during Open Access Week 2012.
Monday Oct. 22, 2012
The library hosted a SPARC webcast focused on the World Bank Open Knowledge Repository. Speakers for the webcast included representatives from Creative Commons, PubMed Central, National Institutes of Health and World Bank.
Wednesday October 24, 2012
The library hosted a workshop “Resources for Open Data” presented by UC Merced Digital Curation Librarian, Susan Borda. The presentation was focused on Open Data, Open Data Repositories such as FigShare and Dryad, Open Source Science Tools, and Open Notbook Science.
Friday, October 26, 2012
The library hosted a faculty panel discussion/luncheon focused on open access publishing trends and initiatives here at UC Merced. The panel included faculty members Peggy O’Day (academic senate chair), Michael Dawson, Ignacio Lopezcalvo, Miriam Barlow and Interim University Librarian Donald Barclay.
D
Measure D1:
Seventy-five percent of students in focus groups will respond that they are "satisfied" or "very satisfied" with the library's information resources.
Responsibility: Sara Davidson.
The library was unable to collect this data, but similar data from the Graduating Student Survey proved relevant:
Graduating Student Survey Results for 2012-2013
The information resources provided by the library supported my academic progress.
107 + 179 / 323 = 88.5%
Measure D2:
The University Librarian will work with the Academic Senate Graduate Research Council Subcommittee on the Library and Information in order to get additional faculty input on information resources.
Responsibility: Donald Barclay
The GRC Subcommittee on the library was abandoned. The Academic Senate-Administration Library Working Group was formed to set up an formal advisory structure for the library. The Interim UL is co-chairing this group.
Measure D3:
Review 2012 systemwide ILL Customer Satisfaction Survey to establish a baseline for future assessment.
Responsibility: Eric Scott
The Library will continue to participate in future ILL Customer Satisfaction Surveys. Future survey data will be used in decision-making for setting programmatic priorities and responding to user needs.Measure D4:
The library will achieve a 95% fill rate for direct requests from faculty to purchase books or DVDs.
Responsibility: Jim Dooley
The library achieved a 99% fill rate (336 out of 338 requests) for direct requests from faculty to purchase books or DVDs. The two requests that could not be filled were for non-English language small press publications.
Measure D5:
UC Merced Library will be at or above the median for percentage of total budget expended on information resources as compared the other UC Libraries.
Responsibility: Jim Dooley
UCD: 47%
UCM: 42.7%
UCI: 42.7%
UCB: 38.4%
UCR: 37.5%
UCSD: 33.5%
UCSB: 33%
UCLA: 27.5%
UCSC: N/A
UCSF: N/A
Measure D6:
Technical service staff will put all physical materials (books, DVDs, etc.) on the shelf and load records into OCLC WorldCat within five working days of receipt.
Responsibility: Jim Dooley
Technical services staff put all physical materials (books, DVDs, etc.) on the shelf and loaded records into WorldCat within five working days of receipt for all weeks except one. The goal was missed for one week during intersession because staff were fully involved shifting books in the stacks.
Measure D7:
For the library's Supplemental Course Resources (SCR), library staff will prioritize and fulfill all requests in order to meet instructional needs in a timely manner.
Responsibility: Emily Lin
In 2012-2013 the library uploaded 1,632 non-duplicate documents (fall and spring semesters, totaled) to fulfill faculty requests, all prior to the start of the semester or prior to the date required.
Measure D8:
Co-invest with the California Digital Library in support of open-access publishing by UC Merced faculty and/or graduate students.
Responsibility: Susan Mikkelsen
In January 2013, the UC Merced Library launched a pilot fund to support open access publishing costs. Along with all other UC campus libraries, the library partnered with the California Digital Library (CDL), to assist scholars who do not have other grant funds available to pay for open access publishing fees. CDL contributed $10,000 to the pilot fund along with $5,000 contributed by the library for a total of $15,000. The primary goals of the program are to support UC researchers interested in reshaping the current models of scholarly publishing and to foster greater dissemination of the scholarly output of the university. Educating and encouraging scholars to understand the importance of controlling their own copyright are additional aims of the pilot.
As of June 30, 2013, five authors have been awarded funds totaling $6478.50. One additional award for $1215 was approved in early July although the funds have not yet been awarded.
Measure D9
The library will analyze the need and cost/benefits for all major gifts of materials and maintain records for all decisions for accepted and rejected offers.
Responsibility: Jim Dooley.
Completed.E
Measure E1:
UC Merced Library annual participation in staff training/professional development will be maintained at or above the level of previous year.
Responsibility: Eric Scott
The library achieved this goal.Measure E2:
The library will host at least three in-service activities for library staff during the fiscal year.
Responsibility: Eric Scott
The library achieved this goal.Measure E3:
When surveyed about their satisfaction with the training and professional development opportunities they receive, at least 75% of library staff will report that they are "satisfied" or "highly satisfied."
Responsibility: Eric Scott
The library was unable to collect this data.Measure E4:
100% of student employees will receive the following minimum amount of training, which will be tracked.
- 4 hours of classroom training
- 4 hours of additional individual training
- 4 hours of coaching per year
- complete a post-training assessment
A 90% attendance and completion rate will be considered successful.
In addition, public service student employees will complete all Question of the Week assignments.
Responsibility: Robin Milford and Joe Ameen
100% for library student employees working at the time training was held. Each library student employee in Access Services received over 4 hours of coaching and individual training. The library did not do QOW last year, but we will do it again starting in Fall 2013.
Measure E5:
The UC Merced Library will achieve a retention rates among student employees of 85% (excluding students who separate due to graduation, transfer, or by otherwise separating from the university).
Responsibility: Robin Milford and Joe Ameen
This goal was achieved.
F
Measure F1:
The library will examine and compare collection policies and guidelines for research data from peer institutions in order to develop baselines for future assessment.
Responsibility: Emily Lin
The library was unable to collect this data. The information was not avaialble from the other UC campus libraries.Measure F2:
Of the electronic theses and dissertations (ETDs) submitted by the Graduate Division, 100% of those submitted with signed author agreements will be archived and made accessible according to rights agreement terms.
Responsibility: Emily Lin
The Graduate Division did not submit any ETDs to the library between July 2012 and June 2013. UC Merced has been piloting submission through ProQuest ETD administrator, and the Library only recently (Aug 2013) obtained information from ProQuest regarding access to ETD files. Currently, 44 ETDs have been archived in the UC Merritt repository and 40 are available on eScholarship.Measure F3:
Conduct follow-up with respondents from 2011 faculty data curation survey who requested more information on data curation services to evaluate current services and propose initiatives.
Responsibility: Emily Lin
Of the 19 people on the list, 3 are no longer here, and 6 have been contacted. Only 1 has responded.Measure F4:
Assess the use of EZID service by UC Merced researchers.
Responsibility: Emily Lin
There was no use of the EZID service by UC Merced researchers during FY12-13.
Copyright @ The Regents of the University of California. All rights reserved.
