Testing the online catalogue: results of user testing

Back in the Spring of this year, we carried out two phases of user testing on our online catalogue, Borthcat.   The key results of the first phase of testing are described over on our Digital Archivist, Jen Mitcham's, blog, as well as some of the actions we were able to take prior to the public launch of Borthcat in April 2016. While the basic phase of testing allowed us to make some really practical and in some cases speedy changes to the catalogue interface, the second phase of testing really gave us insight into individual user interactions with the catalogue  to examine how users search for and retrieve information from our holdings. Whilst results of similar testing have been carried out with Access to Memory (AtoM: our web-based, open-source archival description software) in one or two other institutions in North America, at the time of testing the Borthwick was the first UK AtoM user to carry out such detailed work. The findings of both the first and second phase of testing formed the backbone of my Masters dissertation in Archives and Records Management and, as I'm graduating from the University of Dundee next week (!), it seemed like the perfect time to give you a report into the results from the more detailed tests we carried out.

The Borthcat homepage

Why did we do it?

Across the archives sector, online access to information is now pretty much routine and users often expect digital versions of finding aids and, ideally, digitised version of the documents that they can search and examine. In developing our own online interface, Borthcat, we wanted to make sure that not only was information available to users but (more crucially, in my opinion) that users were able to successfully find that information using the tools we had provided. Looking at our own user base from the statistics we capture, between March and April 2016 - just prior to the launch of Borthcat - there were over double the amount of remote enquiries (1614) to physical visits (731) to the searchroom. In the same period there were over 13,300 unique hits our online digital document repositories: Find My Past, the Cause Papers and the Archbishops' Registers. We wanted to make our catalogue as informative and accessible as possible, not least because a large proportion of both our current and future users are researchers who may never be able to physically visit us at all.

How did we do it?

Jen's blog explores the results of the first phase of testing that we carried out - recruiting our users through a mixture of social media channels and onsite advertising in the searchroom and asking them to complete a brief online questionnaire.

The second stage of testing asked participants to work through a series of set exercises using Borthcat while being observed. During these sessions, participants' screens were recorded using Screencastify to capture their mouse movements and the number of clicks they made.

We wanted to capture some qualitative data on user interactions to enrich the statistical information we'd obtained in the first round of testing. We used the Archival Metrics Toolkit (a fantastic resource!) to help design the exercises and tried to ameliorate the effects of such a controlled environment. Of course, we couldn't hope to fully replicate a researcher's independent enquiry but the results we obtained were interesting and gave us an insight into our users that we hadn't had before.

What did we find out?

Here are just a few of the main findings from the testing.

Limited use of hierarchical menu

Users heavily relied on using the free text search bar at the top of the Borthcat homepage to identify records. Only one participant in the second phase of testing used the hierarchical information available on the left-hand side of the screen, and another used free text searching as their sole retrieval technique throughout the test period. This could be for a mixture of reasons. The majority of our entries in Borthcat are at collection level, and so there are fewer hierarchical descriptions available currently (although the test exercises focused on those archives with full catalogues). Further, and as reflected in the basic phase of testing, many users have become familiar with a free-text search when using search engines like Google.

Overwhelming 'wall of text'

Users found the level of information available on each entry, and the amount of results returned for some searches, to be overwhelming in some cases. There was an overall idea, again in common with the basic testing, that users wanted Borthcat to be able to tailor information more specifically to their queries. I think that this is where the presence of an archivist or the staff in the searchroom who understand our holdings are the most valuable asset we can have; this situation would be more easily resolved for a researcher who was onsite and able to consult a staff member for advice. Where the researcher is remote and is searching for unfamiliar (or unknown!) material then it is vital that the catalogue presents information clear enough for them to make an informed choice.  Some users in the observed tests used keyboard techniques like CTRL+F to narrow down occurrences of specific terms within an archival description, although the majority didn't.

Typeahead search suggestions 

Understanding icons

A further usability issue to come out of the detailed testing was that the icons used in Borthcat's 'Typeahead' search - where potential results are generated as you type - are not defined in our customised iteration of AtoM. In an exercise designed to look at how users interacted with our subject-term listings, participants were asked to find out how many of our holdings contain diaries - a subject terms that has been linked across several separate archives. The majority of users did this by searching for the term ‘diaries’ in the free-text search box. When they did so they were confronted by several entries, all called 'Diaries', at item, file and sub-series level, all from separate archives, as well as a subject term entry for diaries in general. The archival entries are all marked with a 'description' icon and the subject entry is marked with an icon showing a label. For those of us working with Borthcat on a daily basis, it was simple to select the relevant entry and to continue our work but most participants in the study repeated the search several times in order to work through all the options before finding the entry they required. This allowed us to identify a way to improve the usability of Borthcat as well as giving us food for thought in how we construct the titles of our records. The linking of records through the use of different subject terms is one of the most interesting capabilities of AtoM - it allows connections to be made across archives in a way that would be very difficult to do using paper finding aids and can draw out unexpected links. Being able to see how users interact with this capability over the longer term will be very important in understanding how researchers can make the most of the information we hold.

A personal connection

It really came across during the testing that users value a personal connection to their research, either through searching for personal names or through bringing their own research contexts and knowledge to the way they search for information. Several users commented that they would really appreciate a feature that would allow them to collect all the records they found interesting in order to look at them again or to send them through to the searchroom for retrieval. This wasn't something I had expected, but is something that other archives do. A great example is the pinboard feature at the Marks and Spencer archive, Marks in Time.

Carrying out this exercise has been really helpful in understanding that what our users want from our records and what we think they want isn’t always the same thing. Involving our users directly in the development of Borthcat was also a fantastic opportunity to engage more with our audience on a project that will be of practical benefit. I must thank all of the participants who took part in each phase of testing; without their invaluable contribution of time in completing both the survey and the observed exercises, we would not have been able to gain the insights we have done into how our users retrieve information, and how they'd like to be able to retrieve information in the future. It is of vital importance for us and for other archive repositories to keep our users’ needs at the heart of their considerations when making archival information accessible online.

Lydia Dean

Archivist

The results of this user testing have been discussed in more detail in my MLitt dissertation ‘Access to Memory: Understanding how users of the Borthwick Institute search for online archival information’ through the University of Dundee. You can find out more about our work with AtoM through our blog and Jen's blog 'Digital Archiving at the Borthwick' .