Date of Award
5-1-2026
Document Type
Thesis
Degree Name
M.S. in Engineering Science
First Advisor
Bo Wang
School
University of Mississippi
Relational Format
dissertation/thesis
Abstract
Open-vocabulary human–object interaction (HOI) detection is a step towards building scal¬able systems that generalize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason about human–object relationships. However, stan¬dard evaluation metrics, such as mean Average Precision (mAP), treat HOI classes as dis¬crete categorical labels and fail to credit semantically valid but lexically different predictions (e.g., “lean on couch” vs. “sit on couch”), limiting their applicability for evaluating open-vocabulary predictions that go beyond any predefined set of HOI labels. I introduce SHOE (Semantic HOI Open-Vocabulary Evaluation), a new evaluation framework that incorporates semantic similarity between predicted and ground-truth HOI labels. SHOE decomposes each HOI prediction into its verb and object components, estimates their semantic similarity using the average of multiple large language models (LLMs), and combines them into a similarity score to evaluate alignment beyond exact string match. This enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models us¬ing standard benchmarks such as HICO-DET. Experimental results show that SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines, achieving an agreement of 85.73% with average human ratings. This work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions.
Recommended Citation
Noack, Maja, "Semantically Grounded Evaluation for Open-Vocabulary Human-Object Interaction Detection" (2026). Electronic Theses and Dissertations. 8857.
https://egrove.olemiss.edu/etd/8857