Re: [ACAT] cataloger / cataloger studies, Inter-rater reliability

Posting to Autocat

On Fri, Nov 1, 2013 at 5:59 PM, Nathan Rinne wrote:

I am wondering about the topic of “inter-rater reliability” (see here: ) within the cataloging community.

Obviously, catalogers do not always interpret a resource the same way and this will affect their assignment of subject headings.

That said, I think it is responsible to say that even a relatively inexperienced cataloger – given a good knowledge of the available LCSH headings – would probably not mess up a book that needed a heading like “United States — History — Civil War, 1861-1865 — Personal narratives”

This would be even more true for a person who is an expert in the subject area.

I seem to recall there being a study done on this years ago – the words “only 30% overlap [for something or other]” stick in my mind. Does anyone know about this study or other studies that have been done on this topic?

There are lots of studies under the term “inter-indexer consistency” and you can find many of them through Google Scholar.”inter+indexer+consistency”

This problem has been around a long, long time, but I suspect it has become much worse for the user since the introduction of OPACs because the cross-references have never been implemented in a useful way. While a cataloger may not assign precisely the same subject heading as another, they very well may assign a related term. Without the syndetic structure of cross references fully incorporated into the catalog, the searcher could never have the slightest idea what is available.
So far as I am aware, the cataloging profession has never taken this up as a serious problem to solve, but it is serious. I have always thought that tools could be created to help catalogers find the consistency that is already in the catalog, so that they could add their records to it more efficiently.