Article on Ability of Voice Assistants and Generative AI to Respond to Reference Questions

The September 2024 issue of The Journal of Academic Librarianship has published an article by 2 McGill University librarians on Comparing generative artificial intelligence tools to voice assistants using reference interactions:

"This study investigates the ability of voice assistants and generative AI tools to respond to reference questions traditionally received by academic librarians. The authors created a sample of 25 questions based on queries received on the virtual reference service at their institution. They then created a rubric to evaluate the quality of the answers that the AI powered tools provided. The authors determined that the tools understand reference questions well and provide relevant answers but that the quality of the references provided, and the accuracy of the answers can be lacking. They suggest that more research needs to be done to understand the place of AI powered tools in reference services (...)"

"Conclusion:
 Given the increasing presence of voice assistants and the near constant improvements to generative AI tools like ChatGPT, librarians may express a fear that users would employ them instead of using more traditional reference services. While Google Assistant performed well in responding to reference inquiries, it is interesting to note that most of the references that it recommended were academic library guides. It could be of interest to investigate how often voice assistants rely on library resources to provide answers. Even though voice assistants can provide an interesting starting point for research questions, this study shows that they should be used in conjunction with traditional reference services. As for ChatGPT, it can understand prompts and provide relevant answers but its inability to cite authoritative sources creates issues with reliability and authority. Perhaps other uses should be explored for these tools such as brainstorming or explaining concepts. Future studies could also explore how AI tools can be used in the provision of reference services in partnership with reference librarians."