Publications

Abstract
Advancements in large language models (LLMs) have changed information retrieval, offering users a more personalised and natural search experience with technologies like OpenAI ChatGPT, Google Bard (Gemini), or Microsoft Copilot. Despite these advancements, research into user tasks and information needs remains scarce. This preliminary work analyses a Google Bard prompt log with 15,023 interactions called the Bard Intelligence and Dialogue Dataset (BIDD), providing an understanding akin to query log analyses. We show that Google Bard prompts are often verbose and structured, encapsulating a broader range of information needs and imperative (e.g., directive) tasks distinct from traditional search queries. We show that LLMs can support users in tasks beyond the three main types based on user intent: informational, navigational, and transactional. Our findings emphasise the versatile application of LLMs across content creation, LLM writing style preferences, and information extraction. We document diverse user interaction styles, showcasing the adaptability of users to LLM capabilities.

Cite

@inproceedings{trippas2024what,
 author = {Trippas, Johanne R and Al Lawati, Sara Fahad Dawood and Mackenzie, Joel and Gallagher, Luke},
 booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval},
 series = {SIGIR '24},
 title = {What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild},
 year = {2024},
 doi = {10.1145/3626772.3657914},
 address = {New York, NY, USA},
 location = {Washington, DC, USA},
 publisher = {ACM}
}

Abstract

Conversational assistants are becoming prevalent among the wider population due to their simplicity and increasing utility. However, the shortcomings of these tools are as renowned as their benefits. In this work, we present a “first look” at an extensive collection of conversational queries, aiming to identify limitations and improvement opportunities specifically related to information access (i.e., search interactions). We explore over 600,000 Google Assistant interactions from 173 unique users, examining usage trends and the resulting deficiencies and strengths of these assistants. We aim to provide a balanced assessment, highlighting the assistant’s shortcomings in supporting users and delivering relevant information to user needs and areas where it demonstrates a reasonable response to user inputs. Our analysis shows that, although most users conduct information-seeking tasks, there is little evidence of complex information-seeking behaviour, with most interactions consisting of simple, imperative instructions. Finally, we find that conversational devices allow users to benefit from increased naturalistic interactions and the ability to apply acquired information in situ, a novel observation for conversational information seeking.

Cite

@inproceedings{trippas2024reevaluating,
 author = {Trippas, Johanne R and Gallagher, Luke and Mackenzie, Joel},
 booktitle = {Proceedings of the 33rd ACM International Conference on Information and Knowledge Management},
 series = {CIKM '24},
 title = {Re-evaluating the Command-and-Control Paradigm in Conversational Search Interactions},
 year = {2024},
 doi = {doi.org/10.1145/3627673.3679588},
 address = {New York, NY, USA},
 location = {Boise, ID, USA},
 publisher = {ACM}
}

Abstract
The increasing reliance on digital information necessitates advancements in conversational search systems, particularly in terms of information transparency. While prior research in conversational information-seeking has concentrated on improving retrieval techniques, the challenge remains in generating responses useful from a user perspective. This study explores different methods of explaining the responses, hypothesizing that transparency about the source of the information, system confidence, and limitations can enhance users’ ability to objectively assess the response. By exploring transparency across explanation type, quality, and presentation mode, this research aims to bridge the gap between system-generated responses and responses verifiable by the user. We design a user study to answer questions concerning the impact of (1) the quality of explanations enhancing the response on its usefulness and (2) ways of presenting explanations to users. The analysis of the collected data reveals lower user ratings for noisy explanations, although these scores seem insensitive to the quality of the response. Inconclusive results on the explanations presentation format suggest that it may not be a critical factor in this setting.

Cite

@inproceedings{lajewska2024explainability,
 author = {Łajewska, Weronika and Spina, Damiano and Trippas, Johanne R. and Balog, Krisztian},
 booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval},
 series = {SIGIR '24},
 title = {Explainability for Transparent Conversational Information-Seeking},
 year = {2024},
 doi = {10.1145/3626772.3657768},
 address = {New York, NY, USA},
 location = {Washington, DC, USA},
 publisher = {ACM}
}

Abstract

Searching through conversational interactions has been emphasized as the next frontier. Nowadays, conversational agents can generate natural language responses, transforming how we search for information. A key challenge in conversational information-seeking is how these agents present information: should they only reflect facts, cater to human cognitive preferences, or strike a balance between them? These challenges raise questions about aligning conversational agents with human cognitive processes. Our position paper emphasizes the role of narrative in addressing these questions. We explore how narratives influence human comprehension and propose a framework for optimal conversational narratives. These narratives aim to enhance interaction between humans and conversational agents in explanatory information-seeking scenarios.

Cite

@inproceedings{sadiri2024unveiling,
 title ={Unveiling Information Through Narrative In Conversational Information Seeking},
 Author = {Sadiri Javadi, Vahid and Trippas, Johanne R. and Lucie Flek},
 Booktitle = {Proceedings of the ACM International Conference on Conversational User Interfaces (CUI 2024)},
 Year = {2024},
 numpages = {6},
 pages = {1--6},
 doi={10.1145/3640794.3665884},
 address = {New York, NY, USA},
 location = {Washington, DC, USA},
 publisher = {ACM}
}

Abstract

The effectiveness of clarification question models in engaging users within search systems is currently constrained, casting doubt on their overall usefulness. To improve the performance of these models, it is crucial to employ assessment approaches that encompass both real-time feedback from users (online evaluation) and the characteristics of clarification questions evaluated through human assessment (offline evaluation). However, the relationship between online and offline evaluations has been debated in information retrieval. This study aims to investigate how this discordance holds in search clarification. We use user engagement as ground truth and employ several offline labels to investigate to what extent the offline ranked lists of clarification resemble the ideal ranked lists based on online user engagement. Contrary to the current understanding that offline evaluations fall short of supporting online evaluations, we indicate that when identifying the most engaging clarification questions from the user’s perspective, online and offline evaluations correspond with each other. We show that the query length does not influence the relationship between online and offline evaluations, and reducing uncertainty in online evaluation strengthens this relationship. We illustrate that an engaging clarification needs to excel from multiple perspectives, and SERP quality and characteristics of the clarification are equally important. We also investigate if human labels can enhance the performance of Large Language Models (LLMs) and Learning-to-Rank (LTR) models in identifying the most engaging clarification questions from the user’s perspective by incorporating offline evaluations as input features. Our results indicate that Learning-to-Rank models do not perform better than individual offline labels. However, GPT, an LLM, emerges as the standout performer, surpassing all Learning-to-Rank models and offline labels.

Cite

@inproceedings{tavakoli2024online,
 title ={Online and Offline Evaluation in Search Clarification},
 author = {Tavakoli, Leila and Trippas, Johanne R. and Zamani, Hamed and Scholer, Falk and Sanderson, Mark},
 journal = {ACM Transactions on Information Systems (TOIS)},
 year = {2024},
 numpages = {28},
 doi = {10.1145/3681786},
 address = {New York, NY, USA},
 location = {Washington, DC, USA},
 publisher = {ACM}
}

Abstract

Generative Information Retrieval (GenIR) signifies an advancement in Information Retrieval (IR). GenIR encourages more sophisticated, conversational responses to search queries by integrating generative models and chat-like interfaces. However, this approach retains core principles of traditional IR and conversational information seeking, illustrating its capacity to augment current IR frameworks. In this chapter, we propose that introducing GenIR enhances traditional information retrieval tasks and expands their scope. This allows systems to manage more complex queries, including generative, critiquing, and extractive tasks. These advancements surpass traditional systems, handling queries with greater depth and flexibility. This sometimes speculative chapter suggests Generative Information Access (GenIA), a term that more accurately encapsulates the widened scope and enhanced functionalities of GenIR, particularly in how this relates to tasks. By investigating the impact of GenIR, this discussion aims to reiterate that generative research should not abandon traditional interactive information retrieval research but rather incorporate it into future research and development efforts.

Cite

@incollection{trippas2024adapting,
 address = {Cham, Switzerland},
 author = {Trippas, Johanne R. and Spina, Damiano and Scholer, Falk},
 booktitle = {Information Access in the Era of Generative AI},
 editor = {White, Ryen W. and Shah, Chirag},
 publisher = {Springer Nature Switzerland AG},
 title = {Adapting Generative Information Retrieval Systems to Users, Tasks, and Scenarios},
 year = {2024}
}

Abstract
Creating and deploying customized applications is crucial for operational success and enriching user experiences in the rapidly evolving modern business world. A prominent facet of modern user experiences is the integration of chatbots or voice assistants. The rapid evolution of Large Language Models (LLMs) has provided a powerful tool to build conversational applications. We present Walert, a customized LLM-based conversational agent able to answer frequently asked questions about computer science degrees and programs at RMIT University. Our demo aims to showcase how conversational information-seeking researchers can effectively communicate the benefits of using best practices to stakeholders interested in developing and deploying LLM-based chatbots. These practices are well-known in our community but often overlooked by practitioners who may not have access to this knowledge. The methodology and resources used in this demo serve as a bridge to facilitate knowledge transfer from experts, address industry professionals’ practical needs, and foster a collaborative environment. The data and code of the demo are available at https://github.com/rmit-ir/walert.

Cite

@inproceedings{pathiyan2024walert,
 author = {Sachin Pathiyan Cherumanal and Lin Tian and Futoon M.~Abushaqra and Angel F.~ Magnossao de Paula and Kaixin Ji and Halil Ali and Danula Hettiachchi and Johanne~R. Trippas and Falk Scholer and Damiano Spina},
 booktitle = {Proceedings of the ACM Conference on Information Interaction and Retrieval},
 series = {CHIIR '24},
 title = {{Walert: Putting Conversational Search Knowledge into Action by Building and Evaluating a Large Language Model-Powered	Chatbot}},
 year = {2024},
 doi = {10.1145/3627508.3638309},
 address = {New York, NY, USA},
 location = {Washington, DC, USA},
 publisher = {ACM}
}

Abstract

With the rise and growing prevalence of generative models, particularly multi-modal ones, it is an opportune time to explore beyond existing interactive information retrieval research trends. Indeed, it is essential to determine new avenues to explore how users interact with these models as well as revisit existing avenues that can be embellished with new technology. In this session, we aim to create a venue to workshop ideas that explore the future of search experiences and user interactions with information in a collaborative, low-pressure environment. This UnExplored FrontCHIIRs workshop enables participants to form a sub-community within CHIIR to facilitate further development of the proposed ideas and allow deeper collaborative problem-solving than just presenting late-breaking work.

Cite

@inproceedings{roegiest2024unexplored,
  title={UnExplored FrontCHIIRs: A Workshop Exploring Future Directions for Information Access},
  author={Roegiest, Adam and Trippas, Johanne},
  booktitle={Proceedings of the 2024 Conference on Human Information Interaction and Retrieval},
  pages={436--437},
  year={2024},
  doi = {10.1145/3627508.3638302},
 address = {New York, NY, USA},
 location = {Washington, DC, USA},
 publisher = {ACM}
}

Abstract

The First Search Futures Workshop, in conjunction with the Fourty-sixth European Conference on Information Retrieval (ECIR) 2024, looked into the future of search to ask questions such as:
-How can we harness the power of generative AI to enhance, improve and re-imagine Information Retrieval (IR)?
-What are the principles and fundamental rights that the field of Information Retrieval should strive to uphold?
-How can we build trustworthy IR systems in light of Large Language Models and their ability to generate content at super human speeds?
-What new applications and affordances does generative AI offer and enable, and can we go back to the future, and do what we only dreamed of previously?

The workshop started with seventeen lightning talks from a diverse set speakers. Instead of conventional paper presentations, the lightning talks provided a rapid and concise overview of ideas, allowing speakers to share critical points or novel concepts quickly. This format was designed to encourage discussion and introduce a wide range of topics within a short period, thereby maximising the exchange of ideas and ensuring that participants could gain insights into various future search areas without the deep dive typically required in longer presentations. This report, co-authored by the workshop’s organisers and its participants, summarises the talks and discussions. This report aims to provide the broader IR community with the insights and ideas discussed and debated during the workshop – and to provide a platform for future discussion.

Cite

@article{azzopardi2024report,
 title={Report on The Search Futures Workshop at ECIR 2024},
 author={Leif Azzopardi and Charles L. A. Clarke and Paul Kantor and Bhaskar Mitra and Johanne R. Trippas and Zhaochun Ren},
 year = {2024},
 issue_date = {June 2024},
 publisher = {Association for Computing Machinery},
 address = {New York, NY, USA},
 volume = {58},
 number = {1},
 journal = {SIGIR Forum},
 numpages = {41}
}

Abstract

Spoken Conversational Search (SCS) poses unique challenges in understanding user-system interactions due to the absence of visual cues, and the complexity of less structured dialogue. Tackling the impacts of cognitive bias in today’s information-rich online environment, especially when SCS becomes more prevalent, this paper integrates insights from information science, psychology, cognitive science, and wearable sensor technology to explore potential opportunities and challenges in studying cognitive biases in SCS. It then outlines a framework for experimental designs with various experiment setups to multimodal instruments. It also analyzes data from an existing dataset as a preliminary example to demonstrate the potential of this framework and discuss its implications for future research. In the end, it discusses the challenges and ethical considerations associated with implementing this approach. This work aims to provoke new directions and discussion in the community and enhance understanding of cognitive biases in Spoken Conversational Search.

Cite

@inproceedings{ji2024towards,
author = {Ji, Kaixin and Pathiyan Cherumanal, Sachin and Johanne R. Trippas and Hettiachchi, Danula
and Salim, Flora D. and Scholer, Falk and Spina, Damiano},
booktitle = {Adjunct Publication of the 26th International Conference on Human-Computer Interaction with Mobile Devices and Services},
doi = {10.1145/3640471.3680245},
series = {MobileHCI '24},
title = {Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational Search},
year = {2024}
}

Abstract

Voice-based systems like Amazon Alexa, Google Assistant, and Apple Siri, along with the growing popularity of OpenAI’s Chat-GPT and Microsoft’s Copilot, serve diverse populations, including visually impaired and low-literacy communities. This reflects a shift in user expectations from traditional search to more interactive question-answering models. However, presenting information effectively in voice-only channels remains challenging due to their linear nature. This limitation can impact the presentation of complex queries involving controversial topics with multiple perspectives. Failing to present diverse viewpoints may perpetuate or introduce biases and affect user attitudes. Balancing information load and addressing biases is crucial in designing a fair and effective voice-based system. To address this, we (i) review how biases and user attitude changes have been studied in screen-based web search, (ii) address challenges in studying these changes in voice-based settings like SCS, (iii) outline research questions, and (iv) propose an experimental setup with variables, data, and instruments to explore biases in a voice-based setting like Spoken Conversational Search.

Cite

@inproceedings{pathiyan2024towards,
author = {Pathiyan Cherumanal, Sachin and Scholer, Falk and Johanne R. Trippas and Spina, Damiano},
booktitle = {ACM International Conference on Multimodal Interaction},
doi = {10.1145/3686215.3690156},
series = {ICMI '24},
title = {{Towards Investigating Biases in Spoken Conversational Search}},
numpages = {6},
year = {2024}
}

Abstract

Cite

@inproceedings{lajewska2024explainability,
 author = {Łajewska, Weronika and Balog, Krisztian and Spina, Damiano and Trippas, Johanne R.},
 booktitle = {Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region},
 series = {SIGIR-AP '24},
 title = {{Can Users Detect Biases or Factual Errors in Generated Responses in Conversational Information-Seeking?}},
 year = {2024},
 doi = {},
 address = {New York, NY, USA},
 location = {Tokyo, Japan},
 publisher = {ACM}
}

Abstract

Cite

@inproceedings{javadi2024can,
title={Can Stories Help {LLM}s Reason? Curating Information Space Through Narrative},
author={Vahid Sadiri Javadi and Johanne Trippas and Lucie Flek},
booktitle={The First Workshop on System-2 Reasoning at Scale, NeurIPS'24},
year={2024}
}

Abstract

This study is the first attempt to explore the impact of clarification question modality on user preference in search engines. We introduce the multi-modal search clarification dataset, MIMICS-MM, containing clarification questions with associated expert-collected and model-generated images. We analyse user preferences over different clarification modes of text, image, and combination of both through crowdsourcing by taking into account image and text quality, clarity, and relevance. Our findings demonstrate that users generally prefer multi-modal clarification over uni-modal approaches. We explore the use of automated image generation techniques and compare the quality, relevance, and user preference of model-generated images with human-collected ones. The study reveals that text-to-image generation models, such as Stable Diffusion, can effectively generate multi-modal clarification questions. By investigating multi-modal clarification, this research establishes a foundation for future advancements in search systems.

Cite

@inproceedings{tavakoli2024understanding,
author = {Leila Tavakoli and Giovanni Castiglia and Federica Cal{\`o} and Yashar Deldjoo and Hamed Zamani and Johanne R. Trippas},
booktitle = {1st Workshop on Multimodal Search and Recommendations},
series = {CIKM MMSR '24},
title = {{Understanding Modality Preferences in Search Clarification}},
numpages = {9},
year = {2024}
}

Abstract
Topic models such as Latent Dirichlet Allocation (LDA) have been extensively used for characterizing text collections according to the topics discussed in documents. Organizing documents according to topic can be applied to different information access tasks such as document clustering, content-based recommendation or summarization. Spoken documents such as podcasts typically involve more than one speaker (e.g., meetings, interviews, chat shows or news with reporters). This paper presents a work-in-progress based on a variation of LDA that includes in the model the different speakers participating in conversational audio transcripts. Intuitively, each speaker has her own background knowledge which generates different topic and word distributions. We believe that informing a topic model with speaker segmentation (e.g., using existing speaker diarization techniques) may enhance discovery of topics in multi-speaker audio content.

Cite

@inproceedings{spina2015speakerlda,
  title={SpeakerLDA: Discovering topics in transcribed multi-speaker audio contents},
  author={Spina, Damiano and Trippas, Johanne R and Cavedon, Lawrence and Sanderson, Mark},
  booktitle={Proceedings of the Third Edition Workshop on Speech, Language \& Audio in Multimedia},
  pages={7--10},
  year={2015}
}

Abstract
We propose research to investigate a new paradigm for Interactive Information Retrieval (IIR) where all input and output is mediated via speech. Our aim is to develop a new framework for effective and efficient IIR over a speech-only channel: a Spoken Conversational Search System (SCSS). This SCSS will provide an interactive conversational approach to determine user information needs, presenting results and enabling search reformulations. We have thus far investigated the format of results summaries for both audio and text, features such as summary length and summaries documents (noisy document or clean document) generated from (noisy) speech-recognition output from spoken document. In this paper we discuss future directions regarding a novel spoken interface targeted at search result presentation, query intent detection, and interaction patterns for audio search.

Cite

@inproceedings{trippas2015results,
author = {Trippas, Johanne R. and Spina, Damiano and Sanderson, Mark and Cavedon, Lawrence},
title = {Results Presentation Methods for a Spoken Conversational Search System},
year = {2015},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/2810355.2810356},
booktitle = {Proceedings of the First International Workshop on Novel Web Search Interfaces and Systems},
pages = {13--15},
numpages = {3},
location = {Melbourne, Australia}
}