Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning

Published July 22, 2026 · Category: Robotics

Overview

arXiv:2604.10895v4 Announce Type: replace-cross Abstract: For a robot to be called socially intelligent, it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately. In this work, we investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user's internal states (latent) and actions (observable state). Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically. Drawing inspiration from theories in Cognitive Science, we propose a novel multi-task learning framework, termed as \textbf{SocialLDG} that explicitly models the dynamic relationship among the states represent as six distinct tasks. Our framework uses a language model to introduce lexical priors for each task and employs dynamic graph learning to model task affinity evolving with time. SocialLDG has three advantages: First, it achieves state-of-the-art performance on two challenging human-robot social interaction datasets available publicly. Second, it supports strong task scalability by learning new tasks seamlessly without catastrophic forgetting. Finally, benefiting from explicit modelling task affinity, it offers insights on how different interactions unfolds in time and how the internal states and observable actions influence each other in human decision making.

Source

Originally published at arxiv.org.

Related Articles

• RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation
• SKETCH: Semantic Key-Point Conditioning for Long-Horizon Vessel Trajectory Prediction
• STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models