Publications

Training Wheels for Web Search: Multi-Perspective Learning to Rank to Support Children’s Information Seeking in the Classroom

Published in Boise State University Library, 2023

Bicycle design has not changed for a long time, as they are well-crafted for those that possess the skills to ride, i.e., adults. Those learning to ride, however, often need additional support in the form of training wheels. Searching for information on the Web is much like riding a bicycle, where modern search engines (the bicycle) are optimized for general use and adult users, but lack the functionality to support non-traditional audiences and environments. In this thesis, we introduce a set of training wheels in the form of a learning to rank model as augmentation for standard search engines to support classroom search activities for children (ages 6–11). This new model extends the known listwise learning to rank framework through the balancing of risk and reward. Doing so enables the model to prioritize Web resources of high educational alignment, appropriateness, and adequate readability by analyzing the URLs, snippets, and page titles of Web resources retrieved by a given mainstream search engine. Experiments including an ablation study and comparisons with existing baselines showcase the correctness of the proposed model. Outcomes of this work demonstrate the value of considering multiple perspectives inherent to the classroom setting, e.g., educational alignment, readability, and objectionability, when applied to the design of algorithms that can better support children’s information discovery.

Recommended citation: Garrett Allen. 2023. "Training Wheels for Web Search: Multi-Perspective Learning to Rank to Support Children's Information Seeking in the Classroom". Master's Thesis at Boise State University

Multi-Perspective Learning to Rank to Support Children’s Information Seeking in the Classroom

Published in 22nd IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology, 2023

We introduce a re-ranking model that augments the functionality of standard search engines to aid classroom search activities for children (ages 6–11). This model extends the known listwise learning-to-rank framework by balancing risk and reward. Doing so enables the model to prioritize Web resources of high educational alignment, appropriateness, and adequate readability by analyzing the URLs, snippets, and page titles of Web resources retrieved by a mainstream search engine. Experimental results demonstrate the value of considering multiple perspectives inherent to the classroom when designing algorithms that can better support children’s information discovery.

Recommended citation: Garrett Allen, Katherine Landau Wright, Jerry Alan Fails, Casey Kennington, and Maria Soledad Pera. 2023. "Multi-Perspective Learning to Rank to Support Children's Information Seeking in the Classroom". Proceedings of the 22nd IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology https://neelik.github.io/files/WI___REdORank.pdf

In a Hurry: How Time Constraints and the Presentation of Web Search Results Affect User Behaviour and Experience

Published in 23rd International Conference on Web Engineering, 2023

Time constraints are commonplace in our daily lives. While literature in recent years from the Information Retrieval (IR) community has increased our understanding of the effects of time constraints on search, practical effects on search outcomes have rarely been evaluated. Little is known about how different search interfaces influence search outcomes and experiences in time-constrained search. This constitutes a knowledge gap that we aim to address in our work. Through a pre-registered 4x4 between-subjects crowdsourced user study, we investigate the influence of four different interfaces (list view, grid-based view, absence of result snippets, and linear scanning pattern view) on search outcomes and experiences under imposed time constraints (no constraint and constraints at two, five, and eight minutes). Results from our study indicate that user task performance is considerably affected by time constraints. In addition, as time constraints are tightened, a trade-off between querying rates and click depths arises. While no interaction effects between SERP interfaces and time constraints were ultimately found, findings from this study form an essential foundation for future work on how search result presentation may assist those searchers under strict time constraints.

Recommended citation: Garrett Allen, Mike Beijen, David Maxwell, and Ujwal Gadiraju. 2023. "In a Hurry: How Time Constraints and the Presentation of Web Search Results Affect User Behaviour and Experience". Proceedings of the 23rd International Conference on Web Engineering https://neelik.github.io/files/ICWE_2023_In_A_Hurry.pdf

🍄 Power-up! What Can Generative Models Do for Human Computation Workflows?

Published in , 2023

We are amidst an explosion of artificial intelligence research, particularly around large language models (LLMs). These models have a range of applications across domains like medicine, finance, commonsense knowledge graphs, and crowdsourcing. Investigation into LLMs as part of crowdsourcing workflows remains an under-explored space. The crowdsourcing research community has produced a body of work investigating workflows and methods for managing complex tasks using hybrid human-AI methods. Within crowdsourcing, the role of LLMs can be envisioned as akin to a cog in a larger wheel of workflows. From an empirical standpoint, little is currently understood about how LLMs can improve the effectiveness of crowdsourcing workflows and how such workflows can be evaluated. In this work, we present a vision for exploring this gap from the perspectives of various stakeholders involved in the crowdsourcing paradigm — the task requesters, crowd workers, platforms, and end-users. We identify junctures in typical crowdsourcing workflows at which the introduction of LLMs can play a beneficial role and propose means to augment existing design patterns for crowd work.

Recommended citation: Garrett Allen, Gaole He, and Ujwal Gadiraju. 2023. "🍄 Power-up! What Can Generative Models Do for Human Computation Workflows?". Presented at the Generative AI and HCI Workshop at CHI'23: ACM CHI Conference on Human Factors in Computing Systems https://neelik.github.io/files/CHI_2023___Generative_AI_Workshop.pdf

What You Show is What You Get! Gestures For Microtask Crowdsourcing

Published in IUI, 2023

Crowdsourcing is a valuable tool to gather human input which enables the development of reliable artificial intelligence systems. Microtask platforms like Prolific and Amazon’s Mechanical Turk have flourished by creating environments where crowd workers can provide such human input in a diverse and representative manner. Such marketplaces have evolved to support several hundreds of workers in earning their primary livelihood through crowd work. Crowd workers, however, often perform these tasks in sub-optimal work environments with poor ergonomics. Additionally, many of the various microtasks require input via the standard method of a mouse and keyboard and are repetitive in nature. As such, crowd workers who primarily earn their livelihoods in microtask marketplaces are at risk of injuries such as carpal tunnel syndrome. By changing the input modality from a mouse and keyboard to gesture-driven input, crowd workers can complete their work while simultaneously improving or safeguarding their physical health. Through three distinct microtasks, we constructed a dataset that enables the exploration of the physical and mental health of crowd workers while using gestures. In this work, we present the process of constructing this dataset, how we applied it, and the future applications we foresee.

Recommended citation: Garrett Allen, Andrea Hu, and Ujwal Gadiraju. 2023. "What You Show is What You Get! Gestures For Microtask Crowdsourcing". Open Science Track of the 28th Annual Conference on Intelligent User Interfaces https://neelik.github.io/files/IUI2023___Open_Science.pdf

Gesticulate for Health’s Sake! Understanding the Use of Gestures as an Input Modality for Microtask Crowdsourcing

Published in HCOMP, 2022

Human input is pivotal in building reliable and robust artificial intelligence systems. By providing a means to gather diverse, high-quality, representative, and cost-effective human input on demand, microtask crowdsourcing marketplaces have thrived. Despite the unmistakable benefits available from online crowd work, the lack of health provisions and safeguards, along with existing work practices threatens the sustainability of this paradigm. Prior work has investigated worker engagement and mental health, yet no such investigations into the effects of crowd work on the physical health of workers have been undertaken. Crowd workers complete their work in various sub-optimal work environments, often using a conventional input modality of a mouse and keyboard. The repetitive nature of microtask crowdsourcing can lead to stress-related injuries, such as the well-documented carpal tunnel syndrome. It is known that stretching exercises can help reduce injuries and discomfort in office workers. Gestures, the act of using the body intentionally to affect the behavior of an intelligent system, can serve as both stretches and an alternative form of input for microtasks. To better understand the usefulness of the dual-purpose input modality of ergonomically-informed gestures across different crowdsourced microtasks, we carried out a controlled 2 x 3 between-subjects study (N=294). Considering the potential benefits of gestures as an input modality, our results suggest a real trade-off between worker accuracy in exchange for potential short to long-term health benefits.

Recommended citation: Garrett Allen, Andrea Hu, and Ujwal Gadiraju. 2022. "Gesticulate for Health's Sake! Understanding the Use of Gestures as an Input Modality for Microtask Crowdsourcing". In Proceedings of the AAAI Conference on Human Computation and Crowdsourcings https://doi.org/10.1609/hcomp.v10i1.21984

Supercalifragilisticexpialidocious: Why Using the “Right” Readability Formula in Children’s Web Search Matters

Published in ECIR, 2022

We investigate the effect of simple readability formulas on each step of the search process. We present findings indicating that careful consideration is required when assessing readability automatically, dependent on the step of the search process and the target domain.

Recommended citation: Garrett Allen, Ashlee Milton, Katherine Landau Wright, Jerry Alan Fails, Casey Kennington, and Maria Soledad Pera. 2022. "Supercalifragilisticexpialidocious: Why Using the "Right" Readability Formula in Children's Web Search Matters". In Proceedings of the 44th European Conference on Information Retrieval (ECIR '22). ACM, pp 3-18. https://link.springer.com/chapter/10.1007/978-3-030-99736-6_1

Using Conversational Artificial Intelligence to Support Children’s Search in the Classroom

Published in CUI@CSCW, 2021

We present pathways of investigation regarding conversational user interfaces (CUIs) for children in the classroom. We highlight anticipated challenges to be addressed in order to advance knowledge on CUIs for children. Further, we discuss preliminary ideas on strategies for evaluation.

Recommended citation: Garrett Allen, Jie Yang, Maria Soledad Pera and Ujwal Gadiraju. 2021. "Using Conversational Artificial Intelligence to Support Children's Search in the Classroom". Presented at CUI@CSCW '21 Workshop. https://arxiv.org/abs/2112.00076

Baby Shark to Barracuda: Analyzing Children’s Music Listening Behavior

Published in Fifteenth ACM Conference on Recommender Systems, 2021

Music is an important part of childhood development, with online music listening platforms being a significant channel by which children consume music. Children’s offline music listening behavior has been heavily researched, yet relatively few studies explore how their behavior manifests online. In this paper, we use data from LastFM 1 Billion and the Spotify API to explore online music listening behavior of children, ages 6–17, using education levels as lenses for our analysis. Understanding the music listening behavior of children can be used to inform the future design of recommender systems.

Recommended citation: Lawrence Spear, Ashlee Milton, Garrett Allen, Amifa Raj, Michael Green, Michael D. Ekstrand, and Maria Soledad Pera. 2021. "Baby Shark to Barracuda: Analyzing Children's Music Listening Behavior". In Proceedings of the 15th ACM Conference on Recommender Systems (RecSys 2021 Late-Breaking Results). https://dl.acm.org/doi/abs/10.1145/3460231.3478856

Engage!: Co-designing Search Engine Result Pages to Foster Interactions

Published in IDC, 2021

In this paper, we take a step towards understanding how to design search engine results pages (SERP) that encourage children’s engagement as they seek for online resources. For this, we conducted a participatory design session to enable us to elicit children’s preferences and determine what children (ages 6-12) find lacking in more traditional SERP. We learned that children want more dynamic means of navigating results and additional ways to interact with results via icons. We use these findings to inform the design of a new SERP interface, which we denoted CHIRP. To gauge the type of engagement that a SERP incorporating interactive elements–CHIRP–can foster among children, we conducted a user study at a public school. Analysis of children’s interactions with CHIRP, in addition to responses to a post-task survey, reveals that adding additional interaction points results in a SERP interface that children prefer, but one that does not necessarily change engagement levels through clicks or time spent on SERP.

Recommended citation: Garrett Allen, Benjamin L Peterson, Dhanush kumar Ratakonda, Mostofa Najmus Sakib, Jerry Alan Fails, Casey Kennington, Katherine Landau Wright, and Maria Soledad Pera. 2021. "Engage!: Co-designing Search Engine Result Pages to Foster Interactions". In Proceedings of Interaction Design and Children (IDC '21). ACM, pp 583-587. https://dl.acm.org/doi/abs/10.1145/3459990.3465183

To Infinity and Beyond! Accessibility is the Future for Kids’ Search Engines

Published in IR for Children 2000-2020: Where Are We Now? Workshop Co-located with ACM SIGIR, 2021

Research in the area of search engines for children remains in its infancy. Seminal works have studied how children use mainstream search engines, as well as how to design and evaluate custom search engines explicitly for children. These works, however, tend to take a one-size-fits-all view, treating children as a unit. Nevertheless, even at the same age, children are known to possess and exhibit different capabilities. These differences affect how children access and use search engines. To better serve children, in this vision paper, we spotlight accessibility and discuss why current research on children and search engines does not, but should, focus on this significant matter.

Recommended citation: Ashlee Milton, Garrett Allen, and Maria Soledad Pera. 2021. "To Infinity and Beyond! Accessibility is the Future for Kids' Search Engines". In Proceedings of the IR for Children 2000-2020: Where Are We Now? Workshop co-located with the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://arxiv.org/abs/2106.07813

CASTing a Net: Supporting Teachers with Search Technology

Published in KidRec Co-located with ACM IDC, 2021

Past and current research has typically focused on ensuring that search technology for the classroom serves children. In this paper, we argue for the need to broaden the research focus to include teachers and how search technology can aid them. In particular, we share how furnishing a behind-the-scenes portal for teachers can empower them by providing a window into the spelling, writing, and concept connection skills of their students.

Recommended citation: Garrett Allen, Katherine Landau Wright, Jerry Alan Fails, Casey Kennington, and Maria Soledad Pera. 2021. "CASTing a Net: Supporting Teachers with Search Technology". In Proceedings of the 5th International and Interdisciplinary Perspectives on Children and Recommender and Information Retrieval Systems: Search and Recommendation Technology through the Lens of a Teacher (KidRec 2021), co-located with ACM IDC 2021. https://arxiv.org/abs/2105.03456

BiGBERT: Classifying Educational Web Resources for Kindergarten-12th Grades

Published in 43rd European Conference on Information Retrieval, 2021

In this paper, we present BiGBERT, a deep learning model that simultaneously examines URLs and snippets from web resources to determine their alignment with children’s educational standards. Preliminary results inferred from ablation studies and comparison with baselines and state-of-the-art counterparts, reveal that leveraging domain knowledge to learn domain-aligned contextual nuances from limited input data leads to improved identification of educational web resources.

Recommended citation: Garrett Allen, Brody Downs, Aprajita Shukla, Casey Kennington, Jerry Alan Fails, Katherine Landau Wright, and Maria Soledad Pera. 2021. "BiG-BERT: Classifying Educational Web Resources for Kindergarten-12th Grades". In Proceedings of the 43rd European Conference on Information Retrieval (ECIR '21). ACM, pp 176-184. https://link.springer.com/chapter/10.1007/978-3-030-72240-1_13

Dont Judge a Book by its Cover: Exploring Book Traits Children Favor

Published in Fourteenth ACM Conference on Recommender Systems, 2020

We present the preliminary exploration we conducted to identify traits that can influence children’s preferences in books. Findings offer insights for the design of recommender algorithms that would look beyond patterns inferred from traditional user-system interactions (eg, ratings) for recommendation purposes, since when it comes to children such data is rarely, if at all, available.

Recommended citation: Ashlee Milton, Levesson Batista, Garrett Allen, Siqi Gao, Yiu-Kai D Ng, and Maria Soledad Pera. 2020. "Don't Judge a Book by it's Cover": Exploring Book Traits Children Favor. In Proceedings of the Fourteenth ACM Conference on Recommender Systems (RecSys '20). ACM, 6 pp. https://doi.org/10.1145/3383313.3418490