Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

• Supervised HOI methods perform strongly, but extensive annotations and dedicated training constrain scalability to compositional, long-tailed interactions.Source: §1 · p. 1
• MLLM-based adaptation methods still require task-specific training, whereas training-free paradigms exploit frozen pretrained models for interaction grounding and prediction.Source: §1 · pp. 1–2
• Existing training-free methods separate semantic reasoning from visual perception, leaving exploration passive and allowing early assumptions to bias grounding and relation judgments.Source: §1 · p. 2
• HarnessHOI projects interaction hypotheses into image space, reconciles localization candidates under persistent identities, and uses shared groundings to assess actions and semantic roles.Source: §1 · pp. 2–3
• On HICO-DET and V-COCO, HarnessHOI achieves state-of-the-art performance among training-free methods.Source: §1 · p. 3

2.1 TRADITIONAL HUMAN-OBJECT INTERACTION DETECTION

• Supervised HOI methods use two-stage proposal-and-classification pipelines or single-stage set prediction, but annotation and optimization demands limit open-world scalability.Source: §2.1 · p. 3

2.2 FOUNDATION MODELS FOR HOI DETECTION

• Foundation-model HOI methods include trained adaptation and training-free retrieval, candidate-conditioned generation, and agentic reasoning, but often isolate semantic reasoning from visual grounding.Source: §2.2 · p. 3

2.3 MLLM HARNESSING FOR VISUAL REASONING

• Generic active-perception and agentic workflows target visual question answering or coarse grounding, while HarnessHOI unifies interaction-guided perception with relational reasoning.Source: §2.3 · p. 3

3.1 INTERACTION-CENTRIC HOI HARNESS

• HarnessHOI uses interaction hypotheses to guide participant discovery and targeted observation, then adjudicates shared groundings before grounded reasoning evaluates actions and semantic roles.The same frozen MLLM performs all three stages.Source: §3.1 · p. 4

3.2 INTERACTION-GUIDED PERCEPTION

• Interaction-guided perception uses interaction semantics to recover overlooked participants and request local observations that refine ambiguous localizations.Initial discovery expands the participant set; high-resolution crops then refine localizations and ground additional participants under persistent identities.Source: §3.2 · pp. 4–5

3.3 RELATION-AGNOSTIC GEOMETRIC ADJUDICATION

• Geometric adjudication selects each participant’s localization by identity, category, and visible extent, independently of support for particular actions or roles.The selected box and category provide consistent spatial grounding for every relation involving that entity.Source: §3.3 · p. 5

3.4 GROUNDED INTERACTION REASONING

• Candidate pairs combine perception associations with expanded-scene evidence, then each pair is checked across category-compatible actions with per-action normalization to preserve multi-label predictions.The action domain extends beyond hypotheses used for participant discovery.Source: §3.4 · pp. 5–6
• Role prediction separates action occurrence, target visibility, and instance binding, including physical and null targets; missing, visible, and unknown states distinguish target status.Action, visibility, and binding questions use support labels for rejection, support, or unresolved evidence.Source: §3.4 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• The complete HICO-DET and V-COCO test sets contain 9,658 and 4,946 images, and inference uses one frozen Qwen3.5-9B with at most two local observations per image.HICO-DET reports Full, Rare, and Non-Rare mAP; V-COCO reports role AP under S1 and S2, with participant matching at IoU 0.5.Source: §4.1 · pp. 6–7

4.2 COMPARISON WITH STATE-OF-THE-ART METHODS

• 33.20% Default Full mAP on HICO-DET exceeds AgentHOI by 3.59 points, while V-COCO role AP reaches 44.65%/49.70% under S1/S2, above DYSCO by 1.85/1.88 points.HarnessHOI also reaches 36.85% under Known Object evaluation and 41.63% Rare mAP, 1.30 points above AgentHOI.Source: Table 1 · p. 7, §4.2 · p. 7
• 40.15%, 33.64%, 35.45%, and 33.87% Unseen mAP are reported across RF-UC, NF-UC, UO, and UV, respectively, with the best training-free results in all four partitions.The method also reports the strongest training-free Seen results in all four settings; gains include 4.02 points over AgentHOI on UV and 3.53 over ADA-CM on NF-UC.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 ABLATION STUDIES

• 31.45% HICO-DET mAP for the initial-instance baseline exceeds direct generation's 15.62%, and the complete harness reaches 33.20%.On V-COCO, the complete harness improves the initial-instance baseline from 42.90%/47.20% to 44.65%/49.70% AP; the component analysis includes discovered participants, targeted observation, geometric adjudication, and visual relation verification.Source: Table 3 · p. 8, §4.3 · pp. 8–9
• 0.41 HICO-DET mAP points and 0.47/0.54 V-COCO AP points favor shared geometric grounding over relation-conditioned adjudication.With relation candidates fixed, verification using selected participants and acquired visual evidence adds 0.70 HICO-DET mAP points and 0.73/0.86 V-COCO AP points over proposal-conditioned questions.Source: Table 4 · p. 9, §4.3 · p. 9
• 0.77 points of V-COCO S2 AP are gained through participant discovery, while interaction-guided discovery outperforms repeated localization by 0.75 HICO-DET mAP points and 0.78/0.82 V-COCO AP points.With region counts and sizes fixed, targeted observation improves over non-directed observation by 0.50 HICO-DET mAP points and 0.58/0.62 V-COCO AP points.Source: §4.3 · p. 9
• 44.65%/49.70% S1/S2 role AP follows compositional verification, compared with 41.10%/46.30% on identical role candidates.True-positive counts remain unchanged, so the reported improvement reflects more accurate ranking from composing action and target evidence.Source: §4.3 · p. 9
• 13.36 HICO-DET Full mAP points are gained over eight-response direct generation while measured cost falls from 19.95 to 13.26 GPU-worker seconds per image.On V-COCO, gains are 22.44/23.20 S1/S2 AP points, with costs of 22.07 seconds for HarnessHOI and 20.94 seconds for repeated generation.Source: §4.3 · p. 9

5 CONCLUSION

• HarnessHOI achieves state-of-the-art performance among training-free methods on HICO-DET and V-COCO.The authors attribute the result to coordinating visual exploration and relation assessment through shared instance groundings.Source: §5 · p. 10

AI USE STATEMENT; ETHICS STATEMENT

• The authors report that generative AI assisted only with language polishing and grammar, and recommend privacy, consent, bias awareness, and appropriate human oversight.They state that shared groundings make predictions examinable against image evidence and visual questions.Source: AI USE STATEMENT · p. 11, ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT; A.5 COMPLETE INFERENCE PROCEDURE

• The authors document methodology, inference procedures, prompts, implementation details, and evaluation protocols, and pledge to release code and evaluation materials upon publication.The implementation uses one frozen MLLM for generation and question-answering operations.Source: REPRODUCIBILITY STATEMENT · p. 11, Table 6 · p. 19

A.1 SHARED PARTICIPANTS AND INTERACTION EVIDENCE

• Persistent participant identities link localization candidates to interactions, and adjudication selects one shared box and category for each identity.Discovery adds participants and relations; observation refines groundings while retaining the designated participant’s identity.Source: §A.1 · p. 17

A.2 INTERACTION-GUIDED PERCEPTION

• Interaction hypotheses guide grouped discovery of additional participants and can request local observations when target localization is unresolved.Each person may participate in multiple actions and targets; newly discovered participants enter subsequent geometric and relational assessment.Source: §A.2 · pp. 17–18
• The harness executes at most two observation requests per image, prioritizing unlocalized targets and converting crop coordinates into full-image coordinates.A crop can refine eligible groundings and localize several participants; crop-contained candidates are eligible for refinement.Source: §A.2 · p. 18

A.3 RELATION-AGNOSTIC GEOMETRIC ADJUDICATION

• Relation-agnostic adjudication compares candidate category-and-box descriptions against the image and crops, selecting a grounding shared across actions and roles.With at most two observations, an identity can have up to three grounding candidates.Source: §A.3 · p. 18

A.4 GROUNDED INTERACTION REASONING

• Grounded reasoning assesses interactions and action predicates, while separate agent, role-state, and binding questions assess action occurrence, target visibility, and role identity.Normalized answer supports determine selected geometry and detection scores; HICO-DET evaluates compatible positive actions separately, while V-COCO combines agent, visibility, and binding supports for physical roles and agent and missing-target supports for NULL.Source: §A.4 · pp. 18–19

B.1 DETECTION AND SEMANTIC-ROLE EVALUATION

• HICO-DET evaluation covers 9,658 test images and 600 interaction classes, while V-COCO covers 4,946 images and reports agent AP and role AP across 25 action-role categories.HICO-DET detections require a matching class and both boxes at IoU 0.5; V-COCO physical-role matches require both person and target boxes at IoU 0.5.Source: §B.1 · p. 20
• HICO-DET uses VOC 2007 eleven-point AP, and Full mAP averages all 600 classes.Known Object evaluation uses the same predictions as Default but restricts images considered when assessing false positives.Source: §B.1 · p. 20

B.2 CLASS PARTITIONS AND DEVELOPMENT SETTINGS

• Open-vocabulary evaluation partitions classes into four Seen/Unseen settings; the frozen model and inference policy use the full public vocabulary across them.Prompts and policy are developed using fixed official-train subsets, and target-benchmark annotations are not supplied to the model at inference.Source: §B.2 · pp. 20–21

B.3 MODEL AND INFERENCE CONFIGURATION

• Frozen Qwen3.5-9B performs localization, discovery, observation, and question answering, with visual QA using raw first-token logits over its answer set.Predictions retain continuous scores, merge exact duplicates by maximum support, and use no score threshold or non-maximum suppression.Source: §B.3 · pp. 21–22, Table 7 · p. 21

B.4 BASELINE INFERENCE; B.5 EVALUATION COVERAGE

• The initial-instance baseline assesses initially localized participants, while HarnessHOI adds discovery, observation, and geometric adjudication beyond direct generation.Direct generation predicts participants and relations in one response and shares a person box across actions; evaluation includes all official test images, including images with no predictions.Source: §B.4 · p. 22, §B.5 · p. 22

B.6 EVALUATION IMPLEMENTATIONS

• The original HICO-DET code under Octave gives 32.97% Default and 36.61% Known Object Full mAP, versus 33.20% and 36.85% from the common evaluator.The passage attributes the difference to recall-threshold sequences sampling the precision envelope at slightly different boundaries; V-COCO role scores agree at reported precision.Source: §B.6 · p. 22, Table 8 · p. 23, Table 9 · p. 23

C.1 REGULAR AND OPEN-VOCABULARY COMPARISONS; C.2 MODELS AND INFERENCE RESOURCES

• The comparison tables cover regular-setting HICO-DET and V-COCO results and open-vocabulary Unseen, Seen, and Full mAP using predicted boxes.The cited comparison material includes additional model configurations and training-free baselines, with resource and inference-organization comparisons for related systems.Source: §C.1 · p. 22, Table 10 · p. 24, Table 11 · p. 25

C.2 MODELS AND INFERENCE RESOURCES

• The related-method comparisons distinguish learned interaction representations, detection-generation unification, distillation, and tool-augmented reasoning from HarnessHOI’s frozen-MLLM inference.The passage describes HarnessHOI as using the same frozen MLLM for visual exploration and relation verification, with geometric adjudication establishing groundings shared across actions and roles.Source: §C.2 · p. 26

C.3 COMMON-EVALUATOR COMPARISON

• HarnessHOI consistently outperforms AgentHOI on Rare and Non-Rare classes under Default and Known Object evaluation, with gains extending to Unseen classes in all four open-vocabulary settings.The common-evaluator comparisons are reported in Tables 13 and 14.Source: §C.3 · p. 27, Table 13 · p. 27, Table 14 · p. 27

C.4 HICO-DET INTERACTION CLASSES

• On HICO-DET, Default mAP is 36.84% for positive-interaction classes and 9.55% for no-interaction classes, yielding 33.20% class-weighted Full mAP.The corresponding Known Object results are 39.79% and 17.73%; positive actions and no-interaction labels use separate visual questions.Source: §C.4 · p. 27

C.5 V-COCO ACTIONS AND SEMANTIC ROLES

• Predictions match 9,728 of 12,423 physical-role annotations and 3,854 of 3,969 missing-role annotations.The table reports matching statistics for all 25 roles, separated by physical and missing targets.Source: §C.5 · p. 27, Table 16 · p. 28
• S2 exceeds S1 for every role; the 24-role means are 46.09% and 51.07%, respectively.S1 and S2 yield the same final true-positive counts, but their missing-role criteria change matching order and precision-recall curves.Source: §C.5 · p. 28, Table 16 · p. 28
• HarnessHOI obtains 75.58% mean agent AP across 26 actions by assessing each action on the localized person.For role-bearing actions, the same support is combined with target visibility and binding to produce role predictions.Source: §C.5 · p. 28, Table 17 · p. 29

C.6 RESULTS UNDER RELATED EVALUATION PROTOCOLS

• CL-HOI is reported under full supervision and MLLM distillation; HOI-MLLM uses different localization inputs; HOID-R1 evaluates instructions on reannotated HICO-DET.The protocols report HICO-DET Full mAP and V-COCO Role AP for CL-HOI, mean F1, precision, and recall for HOI-MLLM, and localization, action-accuracy, and mAP measures for HOID-R1.Source: §C.6 · p. 29, Table 18 · p. 29, Table 19 · p. 30

D HARNESS DESIGN ANALYSIS

• The harness design analysis evaluates interaction-guided perception, geometric adjudication, and grounded interaction reasoning using controlled experimental conditions.Table 21 summarizes the conditions used to evaluate each component.Source: D HARNESS DESIGN ANALYSIS · p. 30

D.1 EFFECTIVENESS OF THE COMPLETE HARNESS

• Grounded verification over initial participants substantially improves detection over direct generation, and perception guidance and geometric adjudication further improve both benchmarks.The controlled comparisons use the same common conditions described for the harness design.Source: §D.1 · p. 30

D.2 COMPLETE ABLATION RESULTS

• 40.15%, 33.64%, 35.45%, and 33.87% are HarnessHOI Unseen mAP across RF-UC, NF-UC, UO, and UV, respectively.Its advantage over direct generation extends across combinations, object categories, and actions.Source: §D.2 · p. 30
• Pair and label ablations examine candidate selection and compatible-action verification within the four class partitions.Each variant uses predictions from regular evaluation.Source: §D.2 · p. 30
• The HICO-DET harness comparisons report Default and Known Object mAP, with all variants using the same backbone and evaluator.The section also identifies complete HICO-DET and V-COCO ablation metrics in Tables 22 and 23.Source: §D.2 · p. 30, Table 22 · p. 31

D.3 INTERACTION-GUIDED PERCEPTION

• Interaction-guided discovery improves both benchmarks over repeated localization, and targeted observation gains over non-directed crops.The controls retain the same inputs and generation budget for discovery, and the same discovery output, observation count, and crop dimensions for observation.Source: §D.3 · pp. 32–33, Figure 3 · p. 33
• Discovery adds matching groundings for 571 HICO-DET interactions and 217 V-COCO roles; local observation adds 12 and 23 more, respectively.Final selected groundings cover 22,895 HICO-DET interactions and 14,473 V-COCO roles.Source: §D.3 · p. 33
• 878 additional HICO-DET interactions and 925 V-COCO roles are covered over the initial-instance baseline, including 172 and 447 small-target annotations and 534 multiple-action roles.The interaction groups can overlap; V-COCO small-target coverage rises from 3,407 after initial localization to 3,529 after observation.Source: §D.3 · p. 34, Table 26 · p. 34

D.4 RELATION-AGNOSTIC GEOMETRIC ADJUDICATION

• 479 HICO-DET and 424 V-COCO identities use observation-derived groundings shared across their relations, and relation-agnostic adjudication improves detection on both benchmarks.The criterion selects a candidate by participant identity, category, and visible extent, independently of the proposed relation.Source: §D.4 · p. 35

D.5 GROUNDED INTERACTION REASONING

• Combining visual pair assessment with discovery and observation associations achieves the best results on both benchmarks, while selected pairs retain 97.1% and 94.2% of grounded-interaction coverage.The combined sources improve HICO-DET mAP by 5.11 points over associations and V-COCO AP by 1.42/1.93 points over visual pair assessment; pair selection reduces the domains by 90.4% and 92.9%.Source: §D.5 · p. 37
• On identical proposed pairs, label expansion raises HICO-DET mAP from 18.35% to 29.85% and V-COCO AP from 36.43%/39.81% to 42.27%/47.14%.The expansion verifies multiple compatible relations over shared participants, beyond labels proposed during discovery.Source: §D.5 · p. 37
• Compositional verification raises V-COCO S1/S2 AP from 41.10%/46.30% to 44.65%/49.70%, while removing NULL predictions lowers S1 AP to 39.20%.Compositional verification leaves final true-positive counts unchanged; the reported gains include 15 of 25 roles in S1 and 20 of 25 in S2.Source: §D.5 · pp. 37–38

D.6 BACKBONE GENERALITY

• InternVL3-8B obtains 32.11% HICO-DET Full mAP and 43.60%/48.40% V-COCO S1/S2 AP; Qwen2.5-VL-7B obtains 31.61% and 43.10%/47.90%.Both backbones use the identical harness configuration, public vocabulary, and evaluator.Source: §D.6 · pp. 38–39, Table 30 · p. 39

E.1 COMPUTATIONAL STRUCTURE

• Verification workload depends on grounded participants and retained relation candidates; V-COCO also uses 26H agent-action, 25H role-state, and 25 requests per retained pair.The HICO-DET scoring requests include interaction and no-interaction pair assessment and positive-action verification.Source: §E.1 · p. 40

E.2 MEASURED COMPUTATION

• Perceptual generation averages 2.51 responses per HICO-DET image and 2.78 per V-COCO image; measured hardware costs are 13.26 and 22.07 GPU-worker seconds per image.Hardware measurements use frozen Qwen3.5-9B in BF16 on NVIDIA A6000 GPUs, with four GPUs for the harness.Source: §E.2 · p. 40

E.3 ALLOCATING INFERENCE COMPUTATION

• 33.20% HICO-DET mAP and 44.65%/49.70% V-COCO S1/S2 AP are HarnessHOI results; eight-response generation with pair-IoU suppression reaches 19.84% and 22.21%/26.50%, respectively.HarnessHOI costs 13.26 and 22.07 GPU-worker seconds per image on HICO-DET and V-COCO; the eight-response costs are 19.95 and 20.94 seconds, respectively.Source: §E.3 · p. 42, Table 33 · p. 42
• Pair-IoU suppression improves AP by removing overlapping detections, whereas exact-match fusion retains duplicates and AP decreases as more responses are combined.The comparison evaluates candidate quality using the best AP attainable by reordering predicted boxes and labels.Source: Table 34 · p. 42, §E.3 · p. 43

E.4 FIXED-CANDIDATE RANKING ANALYSIS

• HarnessHOI exceeds ground-truth-assisted ranking bounds for a single direct-generation response by 11.62 HICO-DET mAP and 13.96/14.69 V-COCO AP points.The bounds retain the generated boxes and labels and optimally order candidates using annotations.Source: §E.4 · p. 43, Table 35 · p. 43
• With eight responses, HarnessHOI exceeds exact-match fusion’s ranking upper bounds by 2.70 HICO-DET mAP and 2.75/3.30 V-COCO AP points, and also exceeds pair-IoU bounds.The comparison uses optimally ranked repeated-generation predictions with fixed boxes, labels, and candidate counts.Source: §E.4 · p. 43, Table 36 · p. 43

E.5 GENERATION STABILITY

• Across three seeds, HarnessHOI averages 32.95% Default Full mAP with 0.22-point sample standard deviation, versus direct generation at 15.66% ± 0.13.The backbone, inference policy, and complete test set are held fixed across runs.Source: §E.5 · p. 44, Table 37 · p. 44

F QUALITATIVE RESULTS

• Qualitative examples show recovered participants supporting multiple actions, shared bench groundings for two people, and distinct object and instrument roles in tennis.The examples include the book, bench, and racket/ball interactions.Source: F QUALITATIVE RESULTS · p. 45

F.1 PARTICIPANT DISCOVERY AND TARGETED OBSERVATION

• Discovery recovers an initially omitted book for holding and reading; targeted observation finds a shared bench and a small sandwich that supports both holding and eating.The two book actions are assessed independently on shared person-book groundings, and the two sitting relations use the selected bench grounding.Source: §F.1 · pp. 45–46

F.2 PAIR RETENTION IN A DISTANT GAZE RELATION

• Despite accurate localization, the distant gaze pair is excluded because its pair question favors false and neither discovery nor observation associates it.The pair therefore does not reach binding verification.Source: §F.2 · pp. 46–47, Table 40 · p. 48

F.3 SHARED GEOMETRY AND DISTINCT SEMANTIC ROLES

• Hitting and cutting examples distinguish instrument from object targets while sharing action evidence and using target-specific visibility and binding assessments.The racket and knife serve as instruments; the ball and cake serve as objects, and the racket and knife also participate in holding.Source: §F.3 · pp. 47–48, Table 41 · p. 48, Figure 11 · p. 49

G.1 PARTICIPANT LOCALIZATION / G.2 INTERACTION DISCOVERY AND OBSERVATION PLANNING

• Localization prompts prioritize interaction participants and group each person with associated targets; discovery prompts request omitted participants and targeted observation regions for unresolved targets.The prompts also request role distinctions for role-based actions.Source: §G.1 · p. 50, §G.2 · p. 50

G.3 TARGETED OBSERVATION

• Targeted observation uses a crop and target description to update eligible instances or add participants, preserving identities and mapping returned boxes back to the full image.It avoids extrapolating occluded boundaries and returns no changes when localization is unreliable.Source: §G.3 · pp. 50–51

G.4 GEOMETRIC ADJUDICATION

• Geometric adjudication selects the candidate matching an instance’s identity, category, and complete visible boundary, independently of action or role support.In the suitcase example, it selects the original box as the shared grounding over an alternative observation box.Source: §F.1 · p. 46, §G.4 · p. 51

G.5 GROUNDED RELATION QUESTIONS

• Grounded relation questions require visual evidence beyond proximity or typical use; pair assessment determines whether any legal positive interaction is supported before role binding.Action questions separately assess predicates, while role questions assess target visibility and whether an instance fills the object or instrument role.Source: §G.5 · pp. 51–52

G.6 PUBLIC ROLE DEFINITIONS

• V-COCO role meanings specify each target, while action definitions distinguish activities such as holding a phone from talking on one.HICO-DET definitions distinguish predicates including cut from cut_with and sit_at from sit_on.Source: §G.6 · p. 52, Table 43 · p. 52