Publications
Journal Articles and Preprints (1 signals co-first authorship)
2024
Lim, S., Schmälzle, R., & Bente, G. (in press). Artificial social influence: Rapport-building, LLM-based embodied conversational agents for health coaching. Proceedings of CONNECT Workshop at the 24th ACM International conference on Intelligent virtual agents (IVA). [Link to Paper]
Abstract: Embodied conversational agents (ECAs) have been designed and implemented to provide support to humans, especially in the area of health. With the recent advancements in large language models (LLMs), ECAs can now be equipped with natural language capabilities, engaging in turn-taking dialogue with humans while exhibiting verbal rapport-building behavior. Our innovative study designed LLM-based ECAs that provide health coaching to people through immersive virtual reality (VR). Specifically, male and female avatars were integrated with ChatGPT, text-to-speech, and speech-to-text APIs on a VR platform. For our experiment, we manipulated human-ECA similarity via gender-matching. Participants were randomly assigned to either a gender-matched or unmatched embodied health coach and completed two interaction tasks (get-to-know-you and health consultation) in immersive VR. Our quantitative evaluations showed that those in the gender-unmatched conditions rated certain interaction metrics more favorably compared to those in the gender-matched condition. The qualitative evaluation showed that while the lack of nonverbals and other technology-related limitations could be improved, the LLM-based ECAs showed the potential to support people’s health-related decision-making.
Lim, S., Schmälzle, R, & Bente, G. (2024). Artificial social influence via human-embodied AI agent interaction in immersive virtual reality (VR): Effects of similarity-matching during health conversations. arXiv. https://doi.org/10.48550/arXiv.2406.05486
Abstract: Interactions with artificial intelligence (AI) based agents can positively influence human behavior and judgment. However, studies to date focus on text-based conversational agents (CA) with limited embodiment, restricting our understanding of how social influence principles, such as similarity, apply to AI agents (i.e., artificial social influence). We address this gap by leveraging the latest advances in AI (language models) and combining them with immersive virtual reality (VR). Specifically, we built VR-ECAs, or embodied conversational agents that can naturally converse with humans about health-related topics in a virtual environment. Then we manipulated interpersonal similarity via gender matching and examined its effects on biobehavioral (i.e., gaze), social (e.g., agent likeability), and behavioral outcomes (i.e., healthy snack selection). We found an interesting interaction effect between agent and participant gender on biobehavioral outcomes: discussing health with opposite-gender agents tended to enhance gaze duration, with the effect stronger for male participants compared to their female counterparts. A similar directional pattern was observed for healthy snack selection, though it was not statistically significant. In addition, female participants liked the VR-ECAs more than their male counterparts, regardless of the gender of the VR-ECAs. Finally, participants experienced greater presence while conversing with VR-embodied agents than chatting with text-only agents. Overall, our findings highlight embodiment as a crucial factor of influence of AI on human behavior, and our paradigm enables new experimental research at the intersection of social influence, human-AI communication, and immersive virtual reality (VR).
Peng, W., Lee, H., & Lim, S. (2024). Leverage chatbots to combat health misinformation for older adults: A participatory design approach. JMIR Formative Research. https://doi.org/10.2196/60712
Abstract: Background: Older adults, a population particularly susceptible to misinformation, may be victims of attempts of health-related scams or defrauding, and they may unknowingly spread misinformation. Prior research has investigated managing misinformation by media literacy education or supporting users with fact-checking information and cautioning for potential misinformation content, yet studies focusing on older adults are limited. Chatbots have the potential to educate and support older adults in misinformation management, but only limited research is available in this domain. Objective: This current study offers a perspective on how older adults may utilize chatbot’s capabilities for misinformation management. Methods: We conducted five participatory design workshops with a total of 17 older adult participants to ideate ways that chatbots can help them manage misinformation. Results: The results reveal the social roles that older adults expect chatbots to play as well as their concerns about using chatbots for misinformation management. This paper’s findings also demonstrate that issues from misinformation are interrelated with interpersonal relationships and power issues found within society at large, reaching beyond fact-checking. Conclusions: This paper discusses and proposes how chatbots can be designed as part of technological systems for educating older adults.
Lim, S., Cho, H. J., Jeon, M., Cui, X., & Schmälzle, R. (2024). Using VR and eye-tracking to study attention to and retention of AI-generated ads in outdoor advertising environments. bioRxiv. https://doi.org/10.1101/2024.08.15.607684 *Second/third authors contributed equally.
Abstract: In contemporary urban environments, advertisements are ubiquitous, capturing the attention of individuals navigating through cityscapes. This study simulates this situation by using VR as an advertising research tool and combining it with eye-tracking to rigorously assess attention to and retention of visual advertisements. Specifically, participants drove through a virtual city with 40 AI-generated, experimentally manipulated, and randomly assigned advertisements (20 commercial, 20 health) distributed throughout. Our results confirm theoretical predictions about the link between exposure, visual attention, and incidental memory. Specifically, we found that attended ads are likely to be recalled and recognized, and concurrent task demands (counting sales signs) decreased visual attention and subsequent recall and recognition of the ads. Finally, we identify the influence of ad placement in the city as a very important but previously hard-to-study factor influencing advertising effects. This paradigm offers great flexibility for biometric advertising research and can be adapted to varying contexts, including highways, airports, and subway stations, and theoretically manipulate other variables. Moreover, considering the metaverse as a next-generation advertising environment, our work demonstrates how causal mechanisms can be identified in ways that are of equally high interest to both theoretical as well as applied advertising research.
Lim, S., & Schmälzle, R. (2024). The effect of source disclosure on evaluation of AI-generated messages. Computers in Human Behavior: Artificial Human, 2(1), 100058. https://doi.org/10.1016/j.chbah.2024.100058
Abstract: Advancements in artificial intelligence (AI) over the last decade demonstrate that machines can exhibit communicative behavior and influence how humans think, feel, and behave. In fact, the recent development of ChatGPT has shown that large language models (LLMs) can be leveraged to generate high-quality communication content at scale and across domains, suggesting that they will be increasingly used in practice. However, many questions remain about how knowing the source of the messages influences recipients’ evaluation of and preference for AI-generated messages compared to human-generated messages. This paper investigated this topic in the context of vaping prevention messaging. In Study 1, which was pre-registered, we examined the influence of source disclosure on young adults’ evaluation of AI-generated health prevention messages compared to human-generated messages. We found that source disclosure (i.e., labeling the source of a message as AI vs. human) significantly impacted the evaluation of the messages but did not significantly alter message rankings. In a follow-up study (Study 2), we examined how the influence of source disclosure may vary by the adults’ negative attitudes towards AI. We found a significant moderating effect of negative attitudes towards AI on message evaluation, but not for message selection. However, source disclosure decreased the preference for AI-generated messages for those with moderate levels (statistically significant) and high levels (directional) of negative attitudes towards AI. Overall, the results of this series of studies showed a slight bias against AI-generated messages once the source was disclosed, adding to the emerging area of study that lies at the intersection of AI and communication.
Lim, S., & Schmälzle, R. (2024). Exploring the mechanisms of AI message generation: A chatbot development activity for students. Communication Teacher, 38(1), 21-27. https://doi.org/10.1080/17404622.2023.2269258 [Code]
Abstract: Courses: Health Communication, Public Communication Campaigns, Public Relations, Introduction to Communication. Objectives: By the end of this workshop, students will be able to: (1) understand how artificial intelligence–based large language learning models work and be able to explain core concepts such as word embeddings, neural networks, and prompting; and (2) apply what they have learned by building and improving an expert health chatbot. Overall, the workshop aims to empower students with the necessary knowledge to use rapidly advancing artificial intelligence responsibly for health communication.
Schmälzle, R., Lim, S., Wu, J., Bezbaruah, S., & Hussain, S. A. (2024). Two of a kind: Adolescent twins’ brains align more closely during a social movie. Journal of Media Psychology. https://doi.org/10.1027/1864-1105/a000422
Abstract: Recent work using neuroimaging has shown that brain responses to a movie are similar across viewers. These similar responses emerge because the movie recruits brain systems involved in sensory (e.g., responding to the flickering lights on screen), perceptual (e.g., identifying the characters’ faces), and social-cognitive processing (e.g., following and understanding the story, social, and affective responses) – separately in each individual brain, but collectively across the audience. Here we compare brain response similarities during an engaging, social, and nonverbal 5-minute Pixar movie across two levels: First, we show that at a macro-level, the movie-evoked brain responses among the current audience from Australia are correlated with the brain responses to the same movie watched by an audience from the USA. Second, we investigate whether twins, who maximize the preexisting similarity two individual audience members can have, exhibit more similar brain responses to the same movie. We find that shared responses measured in an audience from Australia were highly correlated with responses from an audience watching the same movie in the USA. Second, we find that twins (who are genetically more similar and usually raised in a similar environment) exhibit more strongly aligned brain responses compared to non-twin participants. These results support our predictions about the role of pre-existing similarities among audiences for brain-to-brain coupling during movie reception. Moreover, they suggest that brain-to-brain similarities in response to movies contain information about similarities at the social level.
Jeon, M., Lim, S., Lapinski, M. K., Bente, G., Spates, S., & Schmälzle, R. (2024). Attention and retention effects of culturally targeted billboard messages: An eye-tracking study in a realistic virtual reality environment. bioRxiv. https://doi.org/10.1101/2024.09.10.610975
Abstract: Targeting, the creation of a match between message content and receiver characteristics, is a key strategy in communication message design. Cultural targeting, or adapting message characteristics to be congruent with a group’s cultural knowledge, appearance, or beliefs of recipients, is used in practice and is a potentially effective strategy to boost the relevance of a message, affecting attention to messages and enhancing effects. However, many open questions remain regarding the mechanisms and consequences of targeting. This is partly due to methodological challenges in experimentally manipulating messages that match cultural recipient characteristics while simultaneously measuring effects and balancing experimental control and realism. Here, we used a novel VR-based paradigm in which participants drove along a virtual highway flanked by billboards with varying message designs. Specifically, we manipulated the message design to either match or mismatch peoples’ cultures of origin. We used unobtrusive eye tracking to assess participants’ attention (i.e., for how long and how often they look at matched vs. unmatched billboards). Results show a tendency of the participants to inspect culturally matched billboards more often and for longer. We further found that matched billboards produce better recall, indicating more efficient encoding and storage of the messages. Our results underscore the effectiveness of cultural targeting and demonstrate how researchers can rigorously manipulate relevant message factors using virtual environments. We discuss the implications of these findings regarding theories of cultural targeting and methodological perspectives for the objective measurement of exposure factors through eye tracking.
Cho, H. J., Lim, S., Turner, M., Bente, G., & Schmälzle, R. (2024). Eyes on VR: Unpacking the causal chain between exposure, reception, and retention for emotional billboard messages. bioRxiv. https://doi.org/10.1101/2024.07.19.604208
Abstract: The causal chain from message exposure to reception to effects is widely accepted as the basic explanatory model for communication outcomes. Problematically, the chain’s links are often studied in isolation, leaving measurement gaps that compromise the ecological validity and practical utility of experimental research. Here we introduce a VR-based paradigm that encompasses a realistic message reception context, i.e., a simulated car ride on a highway flanked by billboards. We varied attentional message factors (emotional content) as well as contextual task distractions (trash-counting). VR-integrated eye trackers were used to capture participants’ incidental message exposure dependent on their actual gaze behavior. Consistent with our predictions, results show that 1. exposure gates all subsequent effects; 2. distraction impacts likelihood of exposure; 3. both the manipulation of emotional content and distraction affect retention. This comprehensive analysis of the exposure-reception-retention chain can be broadly applied to a variety of message reception contexts that will be discussed.
Schmälzle, R.1, Wu, J.1, Lim, S., & Bente, G. (2024). The eyes have it: Inter-subject correlations of pupillary responses for audience response measurement in VR. bioRxiv. https://doi.org/10.1101/2024.01.22.576685
Abstract: The eye is the vanguard of the reception process, constituting the point where visual information arrives and is transformed into neural signals. While we view dynamic media contents, a fine-tuned interplay of mechanisms causes our pupils to dilate and constrict over time - and putatively similarly across audience members exposed to the same messages. Research that once pioneered pupillometry did actually use dynamic media as stimuli, but this trend then stalled, and pupillometry remained underdeveloped in the study of naturalistic media stimuli. Here, we introduce a VR-based approach to capture audience members’ pupillary responses during media consumption and suggest an innovative analytic framework. Specifically, we expose audiences to a set of 30 different video messages and compute the cross-receiver similarity of pupillometric responses. Based on this data, we identify the specific video an individual is watching. Our results show that this ‘pupil-pulse-tracking’ enables highly accurate decoding of video identity. Moreover, we demonstrate that the decoding is relatively robust to manipulations of video size and distractor presence. Finally, we examine the relationship between pupillary responses and subsequent memory. Theoretical implications for objectively quantifying exposure and states of audience engagement are discussed. Practically, we anticipate that this pupillary audience response measurement approach could find application in media measurement across contexts, ranging from traditional screen-based media (commercials, movies) to social media (e.g., TikTok and YouTube), and to next-generation virtual media environments (e.g., Metaverse, gaming).
Peng, W., Meng, J., Lim, S., & Liu, H. (2024). Online health [mis] information processing and evaluation among older adults: Comparing “Misinformation Detectives” and “Misinformation Victims”. https://dx.doi.org/10.2139/ssrn.4725205
Abstract: Older adults are vulnerable to online health misinformation. This study fills a research gap by exposing older adults to a mixture of health-related misinformation and factual information and employing the think-aloud protocol in a semi-structured interview to understand information processing and evaluation in the moment of information engagement. The participants were categorized into “misinformation detectives” and “misinformation victims” based on their abilities in discerning misinformation. Comparing and contrasting them, we illustrated distinctive information processing patterns in these two groups. We proposed guidelines for evaluating online misinformation based on the best practices of the “misinformation detectives” and pitfalls to avoid based on the “misinformation victims.”
2023
Lim, S., & Schmälzle, R. (2023). Artificial intelligence for health message generation: An empirical study using a large language model (LLM) and prompt engineering. Frontiers in Communication, 8, 1129082. https://doi.org/10.1371/journal.pone.0291924
Abstract: Introduction: This study introduces and examines the potential of an AI system to generate health awareness messages. The topic of folic acid, a vitamin that is critical during pregnancy, served as a test case. Method: We used prompt engineering to generate awareness messages about folic acid and compared them to the most retweeted human-generated messages via human evaluation with an university sample and another sample comprising of young adult women. We also conducted computational text analysis to examine the similarities between the AI-generated messages and human generated tweets in terms of content and semantic structure. Results: The results showed that AI-generated messages ranked higher in message quality and clarity across both samples. The computational analyses revealed that the AI generated messages were on par with human-generated ones in terms of sentiment, reading ease, and semantic content. Discussion: Overall, these results demonstrate the potential of large language models for message generation. Theoretical, practical, and ethical implications are discussed.
Schmälzle, R., Lim, S., Cho, H. J., Wu, J., & Bente, G. (2023). Examining the exposure-reception-retention link in realistic communication environments via VR and eye-tracking: The VR billboard paradigm. PLOS One, 18(11), e0291924. https://doi.org/10.1371/journal.pone.0291924
Abstract: Exposure is key to message effects. No effects can ensue if a health, political, or commercial message is not noticed. Yet, existing research in communication, advertising, and related disciplines often measures ‘opportunities for exposure’ at an aggregate level, whereas knowing whether recipients were ‘actually exposed’ to a message requires a micro-level approach. Micro-level research, on the other hand, focuses on message processing and retention, takes place under highly controlled laboratory conditions with forced message exposure, and largely ignores how recipients attend selectively to messages under more natural conditions. Eye-tracking enables us to assess actual exposure, but its previous applications were restricted to screen-based reading paradigms lacking ecological validity or field studies that suffer from limited experimental control. Our solution is to measure eye-tracking within an immersive VR environment that creates the message delivery and reception context. Specifically, we simulate a car ride down a highway alongside which billboards are placed. The VR headset (HP Omnicept Pro) provides an interactive 3D view of the environment and holds a seamlessly integrated binocular eye tracker that records the drivers’ gaze and detects all fixations on the billboards. This allows us to quantify the nexus between exposure and reception rigorously, and to link our measures to subsequent memory, i.e., whether messages were remembered, forgotten, or not even encoded. An empirical study shows that incidental memory for messages differs based on participants’ gaze behavior while passing the billboards. The study further shows how an experimental manipulation of attentional demands directly impacts drivers’ gaze behavior and memory. We discuss the large potential of this paradigm to quantify exposure and message reception in realistic communication environments and the equally promising applications in new media contexts (e.g., the Metaverse).
Wei, P., Lim, S., & Meng, J. (2023). Persuasive strategies in online health misinformation: A systematic review. Information, Communication, and Society, 26(11), 2131-2148.https://doi.org/10.1080/1369118X.2022.2085615
Abstract: A proliferation of a variety of health misinformation is present online, particularly during times of public health crisis. To combat online health misinformation, numerous studies have been conducted to taxonomize health misinformation or examine debunking strategies for various types of health misinformation. However, one of the root causes – strategies in such misinformation that may persuade the readers – is rarely studied. This systematic review aimed to fill this gap. We searched Web of Science, Scopus, PsycINFO, and Communication and Mass Media Complete for studies published between 2011 and 2021 on 29 May 2021. Peer-reviewed studies that discussed persuasive strategies in online misinformation messages were included. Of 1,700 articles identified, 58 were eligible and 258 persuasive strategies were extracted. Following the affinity diagraming process, 225 persuasive strategies in online health misinformation were categorized into 12 thematic groups, including: fabricating narrative with details, using anecdotes and personal experience as evidence, distrusting government or pharmaceutical companies, politicizing health issues, highlighting uncertainty and risk, inappropriate use of scientific evidence, rhetorical tricks, biased reasoning to make a conclusion, emotional appeals, distinctive linguistic features, and establishing legitimacy. Possible antecedents for why and how these persuasive strategies in online health misinformation may influence individuals were discussed. The findings suggest that media literacy education is essential for the public to combat health misinformation.
Book Chapters
Jones, C.M. & Lim, S. (2020). Technology in the workplace. In E. Mueller and M. Allen (Eds.), Business and Professional Communication (pp. 293-306). New York: McGraw Hill Education.