Redirecting to original paper in 30 seconds...
Click below to go immediately or wait for automatic redirect
📄 Abstract
Abstract: Making theory-of-mind inferences from human dialogue is a strong indicator of
a model's underlying social abilities, which are fundamental for adept AI
assistants. However, large language and reasoning models struggle to understand
sophisticated social phenomena in transcript data, such as sarcasm and irony.
To assess the weaknesses of current models and to identify their solutions, we
introduce SocialNLI (SoNLI) -- the first social dialogue inference dataset.
SoNLI consists of a collection of dialogue transcripts hand-picked to center
complex social nuances like irony and sarcasm, paired with inferences,
corresponding likelihood scores, and human-written explanations. We explore
social inference analysis as a facet of theory-of-mind, and evaluate LLM and
reasoning model theory-of-mind ability through multi-step counterfactual
reasoning.
Key Contributions
This paper introduces SocialNLI (SoNLI), the first dataset specifically designed for social dialogue inference, containing dialogue transcripts with complex social nuances (irony, sarcasm) paired with inferences, scores, and explanations. It explores social inference as a facet of theory-of-mind and evaluates LLMs' capabilities in this area through multi-step counterfactual reasoning, aiming to improve AI's social intelligence.
Business Value
Enables the development of more socially intelligent AI assistants and chatbots that can better understand and respond to human social cues, leading to more natural and effective human-AI interactions.