Actively engaging learners with learning materials has been shown to be very important in the Search as Learning (SAL) setting. One active reading strategy relies on asking so-called adjunct questions, i.e., manually curated questions geared towards essential concepts of the target material. However, manual question creation is impractical given the vast online content. Recent research has explored the effects of Automatic Question Generation (AQG) on aiding human learning. These studies have primarily focused on user studies in controlled online reading scenarios with limited documents. However, the impacts of adjunct questions on learning in the SAL setting, which involves learning through web searching, are not yet well understood. This paper addresses this gap by conducting a user study with automatically generated adjunct questions integrated into the reading interface built on top of a search system. We conducted a between-subjects user study (N = 144) to investigate the incorporation of automatically generated adjunct questions on participants' learning. We employed three different question generation strategies as well as a control condition: (i) synthesis questions; (ii) factoid questions targeting random text spans; and (iii) factoid questions targeting terms and phrases relevant to the information need at hand. We present four major findings: (i) participants who received adjunct questions exhibited significantly more fine-grained reading behaviour, such as longer document dwell time and more scrolls, than those without adjunct questions. However, adjunct questions' influence on learning outcomes depends on the AQG strategy. (ii) Question types significantly influence participants' reading behaviour. (iii) The adjunct questions' target spans significantly influence learning outcomes. Lastly, (iv) participants' prior knowledge levels affect adjunct questions' effects on their learning outcomes and their reaction to different AQG strategies. Our findings have significant design implications for learning-oriented search systems. The data and code is available at https://github.com/zpeide/AQG-AdjunctQuestions.

@en

The creation of relevance assessments by human assessors (often nowadays crowdworkers) is a vital step when building IR test collections. Prior works have investigated assessor quality & behaviour, and tooling to support assessors in their task. We have few insights though into the impact of a document's presentation modality on assessor efficiency and effectiveness. Given the rise of voice-based interfaces, we investigate whether it is feasible for assessors to judge the relevance of text documents via a voice-based interface. We ran a user study (n = 49) on a crowdsourcing platform where participants judged the relevance of short and long documents-sampled from the TREC Deep Learning corpus-presented to them either in the text or voice modality. We found that: (i) participants are equally accurate in their judgements across both the text and voice modality; (ii) with increased document length it takes participants significantly longer (for documents of length > 120 words it takes almost twice as much time) to make relevance judgements in the voice condition; and (iii) the ability of assessors to ignore stimuli that are not relevant (i.e., inhibition) impacts the assessment quality in the voice modality-assessors with higher inhibition are significantly more accurate than those with lower inhibition. Our results indicate that we can reliably leverage the voice modality as a means to effectively collect relevance labels from crowdworkers.

@en

Advertisements, sponsored links, clickbait, in-house recommendations and similar elements pervasively shroud featured content. Such elements vie for people's attention, potentially distracting people from their task at hand. The effects of such "distractors"is likely to increase people's cognitive workload and reduce their performance as they need to work harder to discern the relevant from non-relevant. In this paper, we investigate how people of varying cognitive abilities (measured using Perceptual Speed and Cognitive Failure instruments) are affected by these different types of distractions when completing search tasks. We performed a crowdsourced within-subjects user study, where 102 participants completed four search tasks using our news search engine over four different interface conditions: (i) one with no additional distractors; (ii) one with advertisements; (iii) one with sponsored links; and (iv) one with in-house recommendations. Our results highlight a number of important trends and findings. Participants perceived the interface condition without distractors as significantly better across numerous dimensions. Participants reported higher satisfaction, lower workload, higher topic recall, and found it easier to concentrate. Behaviourally, participants issued queries faster and clicked results earlier when compared to the interfaces with distractors. When using the interfaces with distractors, one in ten participants clicked on a distractor - and despite engaging with a distractor for less than twenty seconds, their task time increased by approximately two minutes. We found that the effects were magnified depending on cognitive abilities - with a greater impact of distractors on participants with lower perceptual speed, and for those with a higher propensity of cognitive failures. Distractors - regardless of their type - have negative consequences on a user's search experience and performance. As a consequence, interfaces containing visually distracting elements are creating poorer search experiences due to the "distractor tax"being placed on people's limited attention.

@en

Time constraints are commonplace in our daily lives. While literature in recent years from the Information Retrieval (IR) community has increased our understanding of the effects of time constraints on search, practical effects on search outcomes have rarely been evaluated. Little is known about how different search interfaces influence search outcomes and experiences in time-constrained search. This constitutes a knowledge gap that we aim to address in our work. Through a pre-registered 4x4 between-subjects crowdsourced user study, we investigate the influence of four different interfaces (list view, grid-based view, absence of result snippets, and linear scanning pattern view) on search outcomes and experiences under imposed time constraints (no constraint and constraints at two, five, and eight minutes). Results from our study indicate that user task performance is considerably affected by time constraints. In addition, as time constraints are tightened, a trade-off between querying rates and click depths arises. While no interaction effects between SERP interfaces and time constraints were ultimately found, findings from this study form an essential foundation for future work on how search result presentation may assist those searchers under strict time constraints.@en

Complex search tasks—such as those from the Search as Learning (SAL) domain—often result in users developing an information need composed of several aspects. However, current models of searcher behaviour assume that individuals have an atomic need, regardless of the task. While these models generally work well for simpler informational needs, we argue that searcher models need to be developed further to allow for the decomposition of a complex search task into multiple aspects. As no searcher model yet exists that considers both aspects and the SAL domain, we propose, by augmenting the Complex Searcher Model (CSM), the Subtopic Aware Complex Searcher Model (SACSM)—modelling aspects as subtopics to the user’s need. We then instantiate several agents (i.e., simulated users), with different subtopic selection strategies, which can be considered as different prototypical learning strategies (e.g., should I deeply examine one subtopic at a time, or shallowly cover several subtopics?). Finally, we report on the first large-scale simulated analysis of user behaviours in the SAL domain. Results demonstrate that the SACSM, under certain conditions, simulates user behaviours accurately.

@en

The COVID-19 pandemic has changed the way we work, study, and conduct research. Ongoing stresses and uncertainties of the pandemic have impacted research activities and collaborations, especially for graduate researchers1 and Early Career Researchers (ECRs)2. It has also changed the way we connect with the broader research communities. For example, in the last year, conferences were either postponed or held online. Even though many conferences implemented social activities, connecting online with peers is hard. Thus, serendipity and forming new bonds or research connections at conferences have been more complex. Indeed, graduate researchers and ECRs have increased challenges connecting and establishing new research connections in online driven environments. This workshops aims to empower graduate and ECRs, make new research connections, and foster a sense of belonging. The First ECRs Roundtable on Information Access Research workshop at ACM CHIIR'22 looks into the future of research, collaborations, and self-development to ask the following. The workshop is hands-on and interactive-with two key talks to kick-start discussion. Rather than a series of technical talks, we solicit position statements from attendees on opportunities, problems, and solutions on (post-)pandemic research on information access within the wider CHIIR community. Building on work presented at ACM CHIIR 2021 [10], this workshop empowers attendees to share their do's and don'ts, review their practices for success, and refine which strategies work for them. The workshop provides a neutral platform for an open and honest discussion about the lessons learned from working in a pandemic. Outcomes include a technical report written by the attendees.

@en

The Search Engine Results Page (SERP) has evolved significantly over the last two decades, moving away from the simple ten blue links paradigm to considerably more complex presentations that contain results from multiple verticals and granularities of textual information. Prior works have investigated how user interactions on the SERP are influenced by the presence or absence of heterogeneous content (e.g., images, videos, or news content), the layout of the SERP (\emphlist vs. grid layout), and task complexity. In this paper, we reproduce the user studies conducted in prior works---specifically those of~\citetarguello2012task and~\citetsiu2014first ---to explore to what extent the findings from research conducted five to ten years ago still hold today as the average web user has become accustomed to SERPs with ever-increasing presentational complexity. To this end, we designed and ran a user study with four different SERP interfaces:(i) ~\empha heterogeneous grid ;(ii) ~\empha heterogeneous list ;(iii) ~\empha simple grid ; and(iv) ~\empha simple list. We collected the interactions of $41$ study participants over $12$ search tasks for our analyses. We observed that SERP types and task complexity affect user interactions with search results. We also find evidence to support most (6 out of 8) observations from~\citearguello2012task,siu2014first indicating that user interactions with different interfaces and to solve tasks of different complexity have remained mostly similar over time.@en

Prior work in education research has shown that various active reading strategies, notably highlighting and note-taking, benefit learning outcomes. Most of these findings are based on observational studies where learners learn from a single document. In a Search as Learning (SAL) context where learners have to iteratively scan and explore a large number of documents to address their learning objective, the effect of these active reading strategies is largely unexplored. To address this research gap, we carried out a crowd-sourced user study, and explored the effects of different highlighting and note-taking strategies on learning during a complex, learning-oriented search task. Out of five hypotheses derived from the education literature we could confirm three in the SAL context. Our findings have important design implications on aiding learning through search. Learners can benefit from search interfaces equipped with active reading tools—but some learning strategies employing these tools are more effective than others. (This research has been supported by DDS (Delft Data Science) and NWO projects SearchX (639.022.722) and Aspasia (015.013.027).)

@en

Models developed to simulate user interactions with search interfaces typically do not consider the visual layout and presentation of a Search Engine Results Page (SERP). In particular, the position and size of interfacewidgets ---such as entity cards and query suggestions---are usually considered a negligible constant. In contrast, in this work, we investigate the impact of widget positioning on user behaviour. To this end, we focus on one specific widget: the Query History Widget (QHW). It allows users to see (and thus reflect) on their recently issued queries. We build a novel simulation model based on Search Economic Theory (SET) that considers how users behave when faced with such a widget by incorporating its positioning on the SERP. We derive five hypotheses from our model and experimentally validate them based on user interaction data gathered for an ad-hoc search task, run across five different placements of the \qhw on the SERP. We find partial support for three of the five hypotheses, and indeed observe that a widget's location has a significant impact on search behaviour.@en

Undertaking a PhD is a challenging yet fulfilling experience. PhD candidates become deeply involved in developing a myriad of skills over many vital facets, including (but not limited to): (i) the development of their research ideas; (ii) learning how to conduct their research; (iii) engaging with others about their research - both locally and internationally; (iv) developing a profile asan independent researcher; and (v) developing their teaching portfolio. Of course, a candidate is likely to encounter many highs and lows during their candidature. Periods of turbulence can be overcome through the application of various techniques to adapt and learn from these experiences. This tutorial will partly aim to introduce attendees to several techniques to help them advance in the PhD process. It will be presented by two recent PhD graduates in the field of Interactive InformationRetrieval (IIR), who are both close enough to their respective times as PhD students to remember the highs and lows of PhD life, yet be far enough removed from the process that they can adequately reflect and provide insights into their own experiences - both good and bad. This tutorial will empower attendees to share their own do's and don'ts, review their practices for success, and refine what productivity strategies work for them. It will provide an impartial platform for an open and honest discussion about the journey of undertaking a PhD, led by the presenters without judgement.

@en

Studies involving user interfaces typically involve the capturing and recording (logging) of key user interactions between the user and the system being examined. However, anecdotal evidence suggests that researchers often implement their own logging infrastructure-sometimes in a piecemeal fashion-which can lead to numerous implementation mistakes (due to misunderstanding or ignoring differences between web browsers, for example). While efforts have been made to develop interaction logging solutions for experimentation and commercial use, many solutions either use obsolete technology, are prohibitively expensive, are complex to use (and require extensive programming knowledge), or have no source code available. To address these issues, we have developed LogUI, an easy-to-use yet powerful interaction logging framework that can capture virtually any user interaction within a web-based environment. LogUI has been successfully used in several user studies since its launch. This paper provides an in-depth discussion into how we have designed LogUI, and provides narrative on the key challenges that we are looking to address moving forward.

@en

Active reading strategies - -such as content annotations (through the use of highlighting and note-taking, for example) - -have been shown to yield improvements to a learner's knowledge and understanding of the topic being explored. This has been especially notable in long and complex learning endeavours. With web search engines nowadays used as the primary gateway for learners (or users) to find content that helps them realise their learning goals, they are often poorly equipped with the necessary tools to aid in sense-making, an important aspect of theSearch as Learning (SAL) process. Within theInformation Retrieval (IR) community, research efforts have explored ways to keep track of users' search context by providing a notepad-like interface for the collection of relevant articles, and aid them during the exploratory search process. However, these studies did not explicitly measure the effect that such tools have on knowledge and understanding during a complex, learning-oriented search task. In this paper, we address this research gap by carrying out an InteractiveIR experiment with highlighting and note-taking tools built into the search interface. We conducteda crowdsourced between-subjects study (N=115), where participants were assigned to one of four conditions: (i) control (a standard web search interface); (ii) high (highlighting enabled);(iii) note (note-taking enabled); and (iv) highnote (both highlighting and note-taking enabled). We assess participants' learning with a recall-oriented vocabulary learning task, and a cognitively more taxing essay writing task. We find that(i) active reading tools do not aid in the vocabulary learning task. However,(ii) participants in high covered 34% more subtopics, and participants in note covered 34% more facts in their essays when compared to control. Furthermore, (iii) we observed that incorporating active learning tools significantly changed the search behaviour of participants across a number of measures. This is the first work that sheds light on the effect of active reading tools on the SAL process, with important design implications for learning-oriented search systems.

@en

Logging user interactions is fundamental to capturing and subsequently analysing user behaviours in the context of web-based Interactive Information Retrieval (IIR). However, logging is often implemented within experimental apparatus in a piecemeal fashion, leading to incomplete or noisy data. To address these issues, we present the LogUI logging framework. We use (now ubiquitous) contemporary web technologies to provide an easy-to-use yet powerful framework that can capture virtually any user interaction on a webpage. LogUI removes many of the complexities that must be considered for effective interaction logging.@en

Web search engines are today considered to be the primary tool to assist and empower learners in finding information relevant to their learning goals- be it learning something new, improving their existing skills, or just fulfilling a curiosity. While several approaches for improving search engines for the learning scenario have been proposed (e.g. a specific ranking function), instructional scaffolding (or simply scaffolding)-a traditional learning support strategy-has not been studied in the context of search as learning, despite being shown to be effective for improving learning in both digital and traditional learning contexts. When scaffolding is employed, instructors provide learners with support throughout their autonomous learning process. We hypothesize that the usageof scaffolding techniques within a search system can be an effective way to help learners achieve their learning objectives whilst searching. As such, this paper investigates the incorporation of scaffolding into a search system employing three different strategies (as well as a control condition): (i) AQe, the automatic expansion of user queries with relevant subtopics; (ii) CURATEDsc, the presenting of a manually curated static list of relevant subtopics on the search engine result page; and (iii) FEEDBACKsc, which projects real-time feedback about a user's exploration of the topic space on top of the CURATEDsc visualization. To investigate the effectiveness of these approaches withrespect to human learning, we conduct a user study (N=126) where participants were tasked with searching and learning about topics such as genetically modified organisms. We find that (i) the introduction of the proposed scaffolding methods in the proposed topics does not significantly improve learning gains. However, (ii) it does significantly impact search behavior. Furthermore, (iii) immediate feedback of the participants' learning (FEEDBACKsc) leads to undesirable user behavior, with participants seemingly focusing on the feedback gauges instead of learning.

@en

Entity cards are a common occurrence in today's web Search Engine Results Pages (SERPs). SERPs provide information on a complex information object in a structured manner. Typically, they combine data from several search verticals. They have been shown to: (i) increase users' engagement with the SERP; and (ii) improve decision making for certain types of searches (such as health searches). In this paper, we investigate whether the benefits of showing entity cards also extend to the Search as Learning (SAL) domain. Do learners learn more when entity cards are present on the SERP? To answer this question, we designed a series of learning-oriented search tasks (with a minimum search time of 15 minutes), and conducted a crowdsourced Interactive Information Retrieval (IIR) user study (N=144) with four interface conditions: (i) a control with no entity cards; (ii) displaying relevant entity cards; (iii) displaying somewhat relevant entity cards; and (iv) displaying non-relevant entity cards. Our results show that (i) entity cards do not have an effect on participants' learning, but (ii) they do significantly impact participants' search behaviours across a range of dimensions (such as the dwell time and search session duration).@en

The use of simulation techniques is not foreign to information retrieval. In the past, simulation has been employed, for example, for constructing test collections and for model performance prediction and analysis in a broad array of information access scenarios. Nevertheless, a standardized methodology for performance evaluation via simulation has not yet been developed. The goal of this workshop is to create a forum for researchers and practitioners to promote methodology development and more widespread use of simulation for evaluation by: (1) identifying problem settings and application scenarios; (2) sharing tools, techniques, and experiences; (3) characterizing potentials and limitations; and (4) developing a research agenda.

@en