PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanSkill-4D: Training-Free Language Querying of Animatable Gaussian Avatars in Space and Time

Junsu Kim, Naeun Kim, Dongyoon Han, Seungryul Baek

arXiv:2610.24452v2cs.CVcs.AI

TL;DR

Language queries over Gaussian avatars must identify adjacent parts and garments, including indirect references, and locate fine-grained motions that can look alike in RGB. HumanSkill-4D uses per-avatar memories and shared geometric operators with a frozen MLLM; it reports best performance across part segmentation and 3D and 4D referring segmentation. The authors also note that its selectable regions and checked motions are bounded by the parser categories and motion templates.

  • Problem

    Language querying must resolve adjacent or overlapping parts and garments, including indirect references, while fine-grained motions can look alike in RGB despite differing dynamics.

    Source: §1 · p. 2

  • Method

    HumanSkill-4D gives a frozen MLLM per-avatar semantic and motion memories plus shared geometric operators, which it composes into programs evaluated on joint trajectories.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    HumanSkill-4D performs best in part segmentation and 3D and 4D referring segmentation across three human-annotated benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Category choices and operator outputs make selections traceable, and new motion conditions can be handled by composing existing operators or adding new ones.

    Source: §1 · p. 3, §6 · p. 9

    Selectable regions are limited to parser categories and checked motions to registry templates, so details such as logos or watches and actions such as dancing or carrying may not be covered.

    Source: E LIMITATIONS AND FUTURE DIRECTIONS · p. 23

Abstract

from arXiv · show

We introduce avatar referring segmentation, which selects regions of animatable Gaussian avatars from language queries. A query can also specify a pose or motion to select matching intervals. This lets users interact with avatars for local editing and motion inspection. However, making these selections with existing language Gaussian splatting methods poses two challenges: adjacent parts and garments are hard to separate and may be referred to indirectly (region ambiguity), while fine-grained motions can look alike in RGB (motion ambiguity). We address these challenges with HumanSkill-4D, a training-free framework that shares a multimodal large language model (MLLM) and geometric operators across avatars. For each avatar, a semantic memory links body-part and garment categories to persistent Gaussian regions and appearance descriptions. The MLLM uses category names and descriptions to select categories by appearance, relation, or function. To find matching intervals, it composes operators into programs that check query conditions on driving joint trajectories stored in a motion memory. These category choices and operator outputs make selections traceable, while composing or adding operators supports new motion conditions. To evaluate this task, we introduce three human-annotated benchmarks and show that HumanSkill-4D outperforms the baselines in part segmentation and 3D and 4D referring segmentation, with compact memories and only minutes of setup per avatar.

1 INTRODUCTION

HumanSkill-4D addresses region and motion ambiguity in language queries over Gaussian avatars with a training-free framework that uses avatar-specific memories and shared geometric procedures. It introduces avatar referring segmentation and three benchmarks, and reports best performance across part segmentation and 3D and 4D referring segmentation.

1 INTRODUCTION

  • Avatar referring segmentation selects a region from a 3D language query and, for a 4D query, also selects intervals matching a pose or motion.These selections support avatar editing and motion inspection; prior language Gaussian methods address scene regions and time intervals, while human-focused methods segment or track parts.Source: §1 · p. 2
  • Region ambiguity makes adjacent or overlapping parts and garments difficult to distinguish, especially when queries refer to them through relations or functions.Motion ambiguity arises because fine-grained motions can look alike in RGB, although their dynamics differ.Source: §1 · p. 2
  • Semantic memory links body-part and garment categories to persistent Gaussians, while an MLLM selects categories using names and appearance descriptions.For motion queries, the MLLM composes geometric operators into programs executed on driving joint trajectories, making selections traceable and extensible.Source: §1 · p. 2
  • HumanSkill-4D performs best in part segmentation and 3D and 4D referring segmentation across three human-annotated benchmarks.The framework uses compact memories and requires only minutes of setup per avatar.Source: §1 · p. 3

2 RELATED WORK

HumanSkill-4D differs from language-feature matching methods by interpreting region queries through avatar-specific semantic memories and evaluating motion conditions on driving joint trajectories. It combines per-avatar knowledge with reusable procedures for new avatars and motion conditions.

2 RELATED WORK

  • Unlike methods that match queries to language-aligned features, HumanSkill-4D uses an MLLM and semantic memory of avatar parts and garments for referring and part segmentation.Prior methods include per-scene language fields, registered features, learned referring features, and open-vocabulary part segmentation.Source: §2 · p. 3
  • HumanSkill-4D evaluates pose and motion conditions on driving joint trajectories rather than relying mainly on RGB evidence, which can make different motions look alike.It uses compact memory to return the queried region with matching intervals and distinguish differences such as movement direction and extent.Source: §2 · p. 3
  • The framework follows the agent-skill paradigm by pairing per-avatar memories with shared geometric operators and temporal relations.A new avatar needs its memories, while a new motion condition can use composed procedures or added ones.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe Gaussian avatars as persistent sets of Gaussians animated by driving poses, and explain how posed Gaussians contribute to rendered pixels. The overview depicts how avatar memories and shared operators combine to produce spatial and temporal selections.

3 PRELIMINARIES

  • A Gaussian avatar uses a fixed set of Gaussians defined in canonical space, then deforms them with the body into each target pose while preserving each Gaussian’s identity across frames.The driving sequence specifies joint rotations and root translation, which determine 3D joint positions.Source: §3 · p. 3
  • Rendering projects posed Gaussians into a camera view and blends their colors front to back; each Gaussian’s color coefficient depends on its opacity and transmittance through Gaussians in front.Source: §3 · pp. 3–4
  • The overview shows offline construction of per-avatar semantic and motion memories plus a shared operator registry, followed online by spatial and temporal mask generation.Its example composes outward-arm movement checks with a final-frame horizontal-state check.Source: Figure 2 · p. 4

4 METHOD

The method prepares per-avatar semantic and motion memories and a shared operator registry, then uses a frozen MLLM to interpret queries and produce spatial, temporal, or combined selections. Its geometric operators and temporal relations support reusable motion-condition checks.

4.1 OVERVIEW

  • The overview separates offline memory construction from online query interpretation, using a frozen MLLM to select regions and, for 4D queries, matching intervals.The spatial and temporal masks combine into a spatiotemporal selection.Source: §4.1 · p. 4

4.2 BUILDING THE SEMANTIC MEMORY

  • Semantic memory aggregates frozen human-parser labels across views and poses into Gaussian category scores, then associates each Gaussian with categories whose score meets a threshold.It adds MLLM-generated category descriptions of visible color, texture, graphics, and garment coverage to support appearance-based references.Source: §4.2 · p. 5

4.3 BUILDING THE MOTION MEMORY AND OPERATOR REGISTRY

  • The motion memory stores joint positions and root orientation and translation, while the shared registry provides geometric templates and temporal relations for checking pose and motion conditions.The registry defines 46 templates—20 state and 26 change templates—across seven measurement types.Source: §4.3 · p. 5
  • Applying a template to a compatible body part creates an operator that reads the corresponding joint trajectories; state operators select valid frames and change operators identify qualifying intervals.The registry can be extended by composing existing operators and relations or adding new entries.Source: §4.3 · pp. 5–6

4.4 ANSWERING 3D AND 4D QUERIES

  • For a query, the MLLM selects target categories and composes motion operators with temporal relations; an executor runs the program on motion memory to produce matching intervals.The spatial mask selects Gaussians whose stored category scores meet the threshold, and combining it with the temporal mask yields the same region in each matching interval.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments evaluate part segmentation and 3D/4D referring segmentation on three human-annotated benchmarks. HumanSkill-4D achieves the strongest reported results across these tasks, while ablations and cost comparisons examine its choices and setup.

5.1 EVALUATION PROTOCOL

  • The protocol introduces ZJU-Refer-3D, HuMMan-Refer-4D, and ZJU-Refer-4D, with query-based region and interval annotations.Evaluation reports mIoU for part and 3D referring segmentation; 4D evaluation uses vIoU for joint region and timing, plus separate temporal metrics.Source: §5.1 · p. 6

5.2 RESULTS ON THE 3D AND 4D EVALUATIONS

  • HumanSkill-4D achieves the highest mIoU on all nine avatars, across part-name, appearance, and relation/function queries, and the highest vIoU, tIoU, and temporal precision on 4D queries.The 4D results average queries from ZJU-Refer-4D and HuMMan-Refer-4D.Source: Table 1 · p. 7, §5.2 · p. 7, Table 2 · p. 7
  • On ZJU-Refer-3D, HumanSkill-4D achieves the highest mIoU for part, appearance, and relation/function queries, with its largest margin on relation queries, outperforming feature matching even with identical Sapiens2 masks.Source: §5.2 · p. 7, Table 2 · p. 7
  • On 4D queries, HumanSkill-4D addresses spatial and temporal errors by resolving target categories through semantic memory and executing motion programs on driving joint trajectories.OpenHuman4D has 100% temporal recall but lowers tIoU and joint vIoU by keeping predictions active throughout the clip.Source: §5.2 · pp. 7–8, Table 3 · p. 7

5.3 ABLATIONS AND COMPUTATIONAL ANALYSIS

  • Ablations favor MLLM-based category selection over text-embedding retrieval, while performance remains stable for score thresholds τ = 0.2–0.5.Qwen3.8 ties for the highest score in both 3D and 4D; at τ = 0.1, mIoU and vIoU drop slightly, consistent with weakly supported Gaussians leaking across boundaries.Source: Table 4 · p. 8, §5.3 · p. 8
  • HumanSkill-4D reaches the first query in about 4 minutes for 3D and 8 minutes for 4D, with memories of about 5 MB and 10 MB, respectively.The 4D memory is nearly 10× smaller than the most compact 4D baseline; the method has the shortest setup time among compared methods and no per-avatar training.Source: §5.3 · pp. 8–9, Table 5 · p. 9

6 CONCLUSION

HumanSkill-4D selects avatar regions and intervals matching queried pose or motion conditions using a training-free skill that gives a frozen MLLM Gaussian-avatar structure.

6 CONCLUSION

  • HumanSkill-4D links body-part and garment categories and appearance descriptions to persistent Gaussian regions, while composing geometric operators over driving joint trajectories to locate motion intervals.The MLLM selects categories and operators, making selections traceable; new motion conditions can use existing or added operators and relations.Source: §6 · p. 9
  • Across three human-annotated benchmarks, HumanSkill-4D selects regions and intervals most accurately, using compact memories and only minutes of setup per avatar.Source: §6 · p. 9

AI USE STATEMENT

AI assistants supported development and presentation tasks under the authors’ guidance and supervision; the MLLMs ΦMLLM were used only in the method and ablations.

AI USE STATEMENT

  • AI assistants were used under the authors’ guidance and supervision to debug code, search for papers, check grammar, and improve readability and concision.Source: AI USE STATEMENT · p. 10
  • AI assistants also helped format LaTeX tables and adjust figure layouts and output formats.Source: AI USE STATEMENT · p. 10
  • The MLLMs ΦMLLM were used only in the method and ablations.Source: AI USE STATEMENT · p. 10

SUPPLEMENTARY MATERIAL

The supplement details semantic categories, motion operators and temporal relations, MLLM inputs and outputs, and benchmark and evaluation materials.

SUPPLEMENTARY MATERIAL

  • The supplement’s contents list method details, benchmarks, evaluation protocols and metrics, additional results, limitations and future directions, query lists, and MLLM instructions.Source: SUPPLEMENTARY MATERIAL · p. 14, §B.2 · p. 14

A.1 SEMANTIC CATEGORIES

  • The semantic memory uses 16 categories mapped from the parser’s labels, including body parts, clothing, shoes, socks, and eyeglasses.Source: §A.1 · p. 15

A.2 OPERATOR TEMPLATES

  • Operator templates specify a measurement and condition, distinguish frame-level states from changes over time, and can be composed or extended with new registry entries.The registry adapts geometric pose and motion codes to conditions over SMPL trajectories.Source: §A.2 · p. 15, Table S1 · p. 16

A.3 STATE FRAMES AND CHANGE OCCURRENCES

  • Change operators split measurements into directional runs, then retain occurrences passing minimum total-change and duration thresholds; magnitudes encode the size of selected angle and root-rotation changes.The minimum duration is four frames, and thresholds are set from the 99.5th percentile of short-run changes in HuMMan sequences not used for evaluation.Source: §A.3 · pp. 16–17
  • The supplement defines directional measurements in a fixed frame based on the initial root position, gravity, and facing direction, while using default SMPL shape joints for comparable thresholds across subjects.The fixed frame does not rotate when the body later turns.Source: §A.3 · p. 17

A.4 COMBINING OPERATORS WITH TEMPORAL RELATIONS

  • The running query illustrates interval selection: operators identify outward arm movements and horizontal states, then temporal relations retain the lift ending in the shared horizontal state.For the example, the returned interval is frames 37–56; the execution records allow errors to be traced to individual steps.Source: §A.4 · p. 18, Figure S3 · p. 18

A.5 MLLM INPUTS AND OUTPUTS

  • The MLLM’s 3D selector chooses categories from names and descriptions, while its 4D compiler also receives operator and relation cards and returns categories plus a motion program or an unsupported-query declaration.The running query uses CoOccurrence, AllStates, and StateAtEnd to require both arms to move together and end horizontal.Source: §A.5 · pp. 18–19, Table S3 · p. 19

B.1 ZJU-REFER-3D

  • The supplement describes ZJU-Refer-3D annotations: nine subjects, eight frames, and four held-out cameras yield 288 ground-truth masks across 16 categories.Each 3D query is evaluated on 32 held-out images of its subject.Source: §B.1 · p. 19

B.2 4D REFERRING EVALUATION

The 4D evaluation uses human-annotated queries with per-frame target masks and evaluates region overlap alongside temporal localization.

B.2 4D REFERRING EVALUATION

  • HuMMan-Refer-4D and ZJU-Refer-4D each contain 20 queries, with target masks in two held-out cameras for every active frame.Annotators manually determine intervals satisfying each query’s pose or motion conditions.Source: §B.2 · p. 19

C.1 METRICS

  • Temporal precision decreases when predicted intervals include non-ground-truth frames, while recall decreases when ground-truth active frames are omitted.Source: §C.1 · p. 20
  • vIoU scores mask overlap on predicted and ground-truth active frames, assigns zero to extra or missed frames, and averages across cameras and queries.Because frame mask IoU is at most one, vIoU cannot exceed tIoU and reaches it only when overlapping-frame masks match exactly.Source: §C.1 · p. 20

C.2 IMPLEMENTATION AND COMPARISON PROTOCOLS

The supplementary section specifies shared evaluation protocols, implementation and cost accounting, and how performance changes with smaller MLLMs. In 3D a smaller model comes close to the largest, while all tested smaller models fall far behind larger models in 4D.

C.2 IMPLEMENTATION AND COMPARISON PROTOCOLS

  • For part segmentation, methods share reconstructed avatars, observations, and 16 categories; text-based methods receive all category names together.Find3D uses canonical Gaussian centers, averaged RGB, and estimated normals; 4D-DRESS labels rendered pixels via their visible mesh vertices.Source: §C.2 · p. 21
  • For 3D referring segmentation, methods use the same Gaussian avatars, queries, and annotations, with specified feature, mask, and training setups for each baseline.LangSplat and Dr. Splat also have Sapiens2-mask variants; ReferSplat’s training masks come from Grounded-SAM or Sapiens2 category masks.Source: §C.2 · p. 21
  • For 4D referring segmentation, 4-LEGS scores video-text relevance over time, while 4D LangSplat matches queries to spatial and temporal language fields.The 4-LEGS comparison renders high-relevance Gaussians and uses high-relevance frames as intervals.Source: §C.2 · p. 21

C.3 COMPUTATIONAL COST ACCOUNTING

  • Cost accounting starts from completed avatars and inputs, includes preparation through saving the first query’s masks, and counts model loading and inference on an A100 80GB.Stored per-avatar information includes method-specific features, networks, masks, or HumanSkill-4D’s scores, text, and motion; shared inputs and pretrained weights are excluded.Source: §C.3 · p. 22

D.1 SMALLER MLLMS

  • In 4D, all four smaller MLLMs fall far below larger models, while Qwen3-VL-8B comes close to Qwen3.8-27B in 3D and exceeds Gemma 4-31B and Mistral 3.2-24B.The authors suggest that composing motion programs from motion descriptions demands more capacity than selecting categories.Source: §D.1 · p. 22

D.2 QUALITATIVE EXAMPLES

Qualitative examples show stable part labels across viewpoints and compare referring masks across methods. For 4D queries, HumanSkill-4D’s intervals generally align with the annotated occurrences, with some extra or split intervals.

Part segmentation

  • Part labels remain on the same body parts across held-out camera views because each label belongs to a canonical Gaussian rather than a single image.Figure S4 shows stored category assignments for two avatars.Source: §D.2 · p. 22, Figure S4 · p. 23

3D referring segmentation

  • For “Select both hands,” several methods find the hands, but ReferSplat covers most of the body and other methods add unrelated regions such as a shoe or forearm.LangSplat also selects unrelated body regions, while CLIP adds ankle regions.Source: §D.2 · p. 22
  • Across two 3D queries, HumanSkill-4D selects the target alone in both, whereas neither CLIP nor JINA does so in both.For the multicolored-patch query, JINA and HumanSkill-4D select the patched lower garment alone.Source: §D.2 · p. 23

4D referring segmentation

  • For two 4D queries, HumanSkill-4D closely matches ground-truth intervals; other examples include a split first occurrence and a short extra interval before a lift.4-LEGS and 4D LangSplat often activate outside ground-truth intervals, while OpenHuman4D remains active throughout.Source: §D.2 · p. 23

E LIMITATIONS AND FUTURE DIRECTIONS

The stated scope boundary is the parser’s category set for regions and the motion-template set for actions. Supplementary materials also provide benchmark query lists and MLLM instructions with in-context examples.

E LIMITATIONS AND FUTURE DIRECTIONS

  • HumanSkill-4D selects only parser categories and checks motion only with templates, so details such as logos or watches and actions such as dancing or carrying objects may not be selectable or checked.The authors suggest finer segmentation, part-level action conditions, or object-relative templates as extensions.Source: E LIMITATIONS AND FUTURE DIRECTIONS · p. 23

F COMPLETE QUERY LIST

  • Supplementary Table S6 lists ZJU-Refer-3D queries from main-paper Table 2, while Tables S7 and S8 list HuMMan-Refer-4D and ZJU-Refer-4D query subsets from Table 3.Source: F COMPLETE QUERY LIST · p. 27

G MLLM INSTRUCTIONS

Related papers

Loading 2610.24452v2…