PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Being Accurate: Implications for Using Natural Language Descriptions for Accessible 3D Modeling with~Blind and Low Vision Makers

Gene S-H Kim, Nicholas Oskiper, Sean Follmer, Megan Hofmann

arXiv:2610.35562v1cs.HC

TL;DR

Because 3D modeling tools rely on visual geometry, it remains unclear whether natural-language descriptions give BLV people the spatial understanding needed to edit models. The authors developed strategies through a pilot and tested them with BLV participants; they found a use-to-editing gap and propose a two-phase evaluation framework.

  • Problem

    Visual 3D modeling tools exclude BLV people, and it is unclear whether language descriptions can provide the spatial understanding needed for design tasks.

    Source: §1 · p. 1

  • Method

    An autoethnographic pilot with 12 sighted participants developed descriptive strategies, which were applied as scripts in a Wizard-of-Oz study with 10 BLV participants.

    Source: §1 · p. 2

  • Results

    Descriptions supported general model impressions but not modification specification, revealing a use-to-editing gap and four failure types that sighted-expert accuracy assessment alone cannot detect.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    The proposed two-phase framework assesses model comprehension and modification specification to diagnose description failures and guide targeted interventions.

    Source: §1 · p. 2, §6 · p. 9

    Only two BLV participants had prior 3D modeling experience, and pilot strategies shaped by the first author’s engineering background may not generalize to other expertise or experience.

    Source: §7 · p. 11

Abstract

from arXiv · show

People who are blind or have low vision (BLV) are largely excluded from 3D modeling tools that rely on visual representations of geometry. Natural language descriptions offer a potential path to accessibility, but it is unclear whether language alone can support the deep spatial understanding that editing, not just browsing, requires. Advances in generative AI offer the promise of automatic model description, yet descriptions suited to BLV modelers are structurally distinct from alt-text and from existing 3D model captioning datasets, and the domain-specific requirements for generating them are not well understood. We address this through two coordinated studies: an autoethnographic pilot in which the blind first author worked with 12 sighted participants to develop a taxonomy of descriptive strategies, and a Wizard-of-Oz main study that applied those strategies as idealized scripts with 10 BLV participants. We identify a systematic use-to-editing gap: descriptions that support accurate but incomplete mental models are insufficient for specifying modifications, because editing requires knowing the state of features the user did not know to ask about. We show that this gap produces four failure types (accuracy, completeness, expression, and navigation) that accuracy-focused evaluation frameworks cannot detect. We propose a two-phase evaluation framework that diagnoses the failure type a description system exhibits and guides developers toward targeted design interventions, thereby providing a methodological standard for assessing whether description-based systems genuinely support BLV modelers as creators.

1 Introduction

The paper asks whether natural-language descriptions can make 3D modeling accessible to BLV people, given the visual demands of current tools and the distinct information editing requires. Two coordinated studies develop and test description strategies, finding an understanding-to-editing gap and proposing an evaluation framework for diagnosing it.

1 Introduction

  • BLV people are excluded from visual CAD tools, and tactile alternatives can limit their ability to lead their own modeling tasks.The paper asks whether text descriptions can support the 3D comprehension needed for design.Source: §1 · p. 1
  • Descriptions for BLV modelers need domain-specific requirements because they differ from alt-text, maker documentation, and existing 3D caption datasets.The paper notes that generative AI may help produce descriptions, but suitable training data and requirements remain poorly understood.Source: §1 · p. 1
  • Rare, individualized BLV modeling practices, unreliable early AI behaviors, and uncertainty about model representations complicate study of description-based 3D modeling.Source: §1 · p. 1
  • An autoethnographic pilot with 12 sighted participants developed descriptive strategies, which were tested as scripts in a Wizard-of-Oz study with 10 BLV participants.Source: §1 · p. 2
  • Descriptions supported general impressions but were inadequate for specifying modifications, exposing a use-to-editing gap that accuracy-focused evaluation misses.The paper identifies completeness, expressiveness, and navigability as requirements for editing.Source: §1 · p. 2
  • The proposed two-phase framework diagnoses description failures and guides targeted interventions; the paper argues descriptions alone are unlikely to suffice, though generative AI may still play a meaningful role.Source: §1 · p. 2

2 Related Work

Prior work offers tactile, audio, and language-based approaches to accessible visual content, but the paper describes continuing challenges in conveying complex 3D geometry and supporting BLV users’ needs. It situates its work among accessible CAD, context-sensitive descriptions, and AI verification research.

2 Related Work

  • 3D printing supports several accessibility uses, but accessible 3D modeling remains an open challenge despite BLV people’s sophisticated customization practices.Source: §2 · p. 2

2.1 Accessible CAD

  • Haptic and audio approaches can miss complex features such as concavities, small segments, and precise contours, and a widely available domain-agnostic nonvisual modeling solution remains unrealized.ShapeCAD is effective but limited by comparatively low resolution.Source: §2.1 · p. 2

2.2 Natural Language for Accessibility

  • Existing 3D captions and alt-text work do not establish standards for 3D descriptions tailored to the geometric information BLV modelers need.The paper also notes that description requirements differ between model selection and model editing.Source: §2.2 · p. 2
  • Spatial descriptions often shift among reference frames, while route-based egocentric framing helped BLV participants form mental models from verbal descriptions.Source: §2.2 · p. 3

2.3 Generative AI for Accessible Image Modification and Verification

  • Generative AI could scale access to complex visual descriptions, but scale does not remove the need for high-quality descriptions.Source: §2.3 · p. 3
  • BLV users may struggle to verify AI-generated visual content, motivating work on accessible uncertainty and limitation communication.Prior research reports compensatory verification strategies, persistent difficulty detecting errors, and the value of tactile exploration and revealing system limitations.Source: §2.3 · p. 3

3 Methods

The methods combine an autoethnographic pilot with a Wizard-of-Oz study: pilot strategies informed prepared descriptions that BLV participants explored through conversation and clay replicas. The pilot findings shaped the scripts, which were deliberately idealized to isolate whether language descriptions support BLV modelers.

3 Methods

  • The pilot paired a blind modeler with sighted assistants to surface descriptive strategies, then a Wizard-of-Oz study had 10 BLV participants explore models using scripts read by sighted researchers.Source: §3 · p. 3

3.1 Data Collection and Analysis

  • Researchers analyzed transcripts, screen recordings, clay replicas, and interviews using reflexive thematic analysis, with two authors independently coding half the sessions before developing themes.Source: §3.1 · p. 4

3.2 Participants

  • The study recruited 10 BLV participants, but only two had prior 3D modeling experience, despite prioritizing experienced modelers.Source: §3.2 · p. 4

3.3 3D Model Sampling

  • Models were stratified from Thingiverse’s 1,000 most popular models in June 2024 across four categories, which diversified the set but were neither comprehensive nor independent.Source: §3.3 · p. 4

3.4 Autoethnographic Pilot Study

  • The pilot found a shared overview-then-components structure, while participants’ expressive strategies varied across participants and model types and informed the main-study scripts.Source: §3.4 · p. 4, §3.4.1 · p. 5

3.4.1 Segmentation and Shared Language.

  • Assistants established labeled component anchors and then related components spatially, using shared vocabulary rather than repeatedly redescribing geometry.Source: §3.4.1 · p. 5

3.4.2 A Taxonomy of Expressive Strategies.

  • Expressive strategies included geometric terms and transformations, object or symbol comparisons, embodied metaphors, and shifts among reference frames.Participants used no single consistent reference frame; scripts therefore included multiple framings for each component.Source: §3.4.2 · p. 5

3.5 Sighted-Assistant Script Development

  • Prepared scripts covered identified components with each expressive strategy, and were designed as idealized agents complete in coverage, consistent in expression, and free of hallucination.This choice isolated whether the description medium supports BLV modelers from whether current AI can reliably produce such descriptions.Source: §3.5 · p. 5

4 Findings

Across the findings, participants needed both manageable descriptions and enough relevant detail to understand models. A persistent gap emerged between understanding a model and specifying edits, while varied descriptions and navigation strategies helped participants make sense of geometry.

4.1 Overwhelming Description Depth

  • Descriptions were simultaneously too long to process and too incomplete for users’ needs.Participants interrupted or requested repeats of overviews, yet nearly all also wanted additional details.Source: §4.1 · p. 6, §4.1.1 · p. 6, §4.1.2 · p. 6

4.1.1 Overwhelming Description Length.

  • All participants found overview descriptions overwhelming, while irrelevant details—not detail itself—were especially irritating.One participant contrasted three pages of description with the 15 seconds a sighted person might need.Source: §4.1.1 · p. 6

4.1.2 The Value of More Details.

  • Participants requested missing fine-grained details, and requests were not confined to technically experienced modelers; preferred detail and tolerance varied by context.Examples included polygon counts, dimensions, contours, and textures.Source: §4.1.2 · p. 6

4.3 Challenges Specific to Model Editing

  • Participants often understood models well yet struggled to specify edits, because editing surfaced unknown existing features and required more detail than general comprehension.Requested changes ranged from moving, transforming, reskinning, or adding features to changes that affected geometry or functionality.Source: §4.1.2 · p. 6, §4.3 · p. 7

4.2 Approaches to Navigating Descriptions

  • Multiple description styles, alongside a consistent overview, helped participants clarify geometry and use reference points to navigate desired edits.Participants valued transitions among styles, including embodied metaphors and step-by-step construction with simple shapes.Source: §4.2 · pp. 6–7
  • When users struggled to ask for details, they often offered hypothetical feature descriptions; scripted assistants repeated script language or could not answer directly.This difficulty also affected participants with significant technical backgrounds.Source: §4.2 · p. 6

5 Discussion

Descriptions can support a general impression of a model yet leave BLV makers without the understanding needed to specify edits. Completeness, expression, and navigation shape whether information is accurate, usable, and reachable.

5.1 Complete Descriptions for Use and Editing

  • Editing requires knowing the precise state of any feature a proposed change might affect, including features the user did not know to ask about.An accurate but incomplete mental model may therefore support use without supporting modification.Source: §5.1 · p. 7
  • Accuracy and completeness failures differ: errors misrepresent features, while omissions leave users unaware of relevant features even when stated content is correct.BLV users cannot independently verify description errors, which can create misconceptions that persist into downstream reasoning.Source: §5.1 · p. 7

5.2 Expression: Aligning Frames of Reference

  • No single expressive strategy supports spatial understanding across BLV participants or model types; multiple consistent framings can help users test their inferences.Different reference frames make different spatial relationships salient, so descriptions must balance initial expressivity with terminology that remains consistent through editing.Source: §5.2 · p. 8
  • Expression failures occur when accurate, complete information is framed in a way that does not map onto the user’s spatial knowledge, producing a misconception.Sighted experts may miss this failure because the information is present and factually correct; assessment must directly measure BLV comprehension.Source: §5.2 · p. 8

5.3 Navigation: Scaffolding Model Exploration

  • On-demand snippets reduce description length but require users to know what to ask, and participants often struggled to formulate useful questions despite sensing gaps.Hypocognition can lead users to assume omitted features do not exist, while early descriptions disproportionately shape later exploration.Source: §5.3 · p. 8
  • Navigation failures arise when relevant information exists but users cannot reach it, for example because they cannot formulate the needed inquiry or lack an initial scaffold.Description utility varies with whether a user needs only a general impression or enough understanding to specify a modification.Source: §5.3 · p. 8

6 A Framework for Evaluating Visual-Description Systems

The framework assesses whether descriptions build spatial comprehension and whether that comprehension supports editing. It combines a mental-model assessment with methods suited to the study setting.

6.1 Phase 1: Comprehension Assessment

  • The first phase assesses whether participants’ mental models accurately represent model segments and their structural relationships.Table 2 presents comprehension evaluations for all BLV participants; Figure 1 contextualizes criteria with original models and clay replicas.Source: §6.1 · p. 9, Table 2 · p. 9, Figure 1 · p. 9

6.1.1 Segment Comprehension.

  • Segment accuracy evaluates misrepresented or invented features, while segment completeness checks whether salient features of the original appear in the participant’s mental model.The examples include a pelvis shape mismatch and omitted texture, as well as a replica containing only two portico columns.Source: §6.1.1 · p. 9

6.1.2 Structural Comprehension.

  • Structural accuracy checks whether segments connect correctly and retain their proportions, while structural completeness checks for missing segments or spatial relationships.Examples include an inaccurate pelvis-to-femur proportion, merged building sections, and omitted stairwell walls.Source: §6.1.2 · pp. 9–10

6.1.3 Conducting Comprehension Assessments.

  • Clay replicas made participants’ spatial mental models observable, while post-hoc interviews helped diagnose expression and navigation failures.Other settings may call for structured verbal protocols or surveys, depending on deployment and participant population.Source: §6.1.3 · p. 10

6.2 Phase 2: Editing Specification Assessment

The second phase tests whether demonstrated model comprehension supports a participant’s specification of a requested modification.

6.2 Phase 2: Editing Specification Assessment

  • Researchers give participants a modification goal and an editing method suited to the deployment context, then assess their editing specifications.Possible methods include structured prompting, a programming-based tool, or a prototype CAD interface.Source: §6.2 · p. 10
  • Specifications are judged on correct, precise segment identification and awareness of how the edit interacts with adjacent segments.Researchers trace failures to Phase 1: poor segment specificity after comprehension may indicate navigation failure, while missed structural dependencies may reflect accuracy or completeness errors.Source: §6.2 · p. 10

6.3 Designing Systems after Failure Diagnosis

The framework uses failure diagnosis to direct system improvements, since a generic goal of improving description quality is unlikely to close the use-to-editing gap.

6.3 Designing Systems after Failure Diagnosis

  • The intervention should match the diagnosed failure type rather than treating description quality as an undifferentiated target.Different failure diagnoses direct developers toward different approaches to system improvement.Source: §6.3 · p. 10

6.3.1 Addressing Accuracy and Completeness Failures.

  • Accuracy and completeness failures reflect limits in the model representation, which can hide internal structure, occluded features, or precise contours.Richer mesh or feature representations can expose more geometry but may impose computational costs; disclose limits that cannot be remedied.Source: §6.3.1 · p. 10

6.3.2 Addressing Expression Failures.

  • Expression failures can be addressed by making reference-frame switching accessible and offering multiple framings of the same feature.Anchoring descriptions across semantic, textual, and code representations can reduce the cost of switching frames, but mismatches with users’ prior spatial knowledge also require system support.Source: §6.3.2 · p. 10

6.3.3 Addressing Navigation Failures.

  • Navigation failures call for task-responsive descriptions and contextual scaffolding beyond a fixed sequential overview.Model purpose, creator descriptions, adjustable parameters, and the user’s posed questions can help surface information relevant to a specific edit.Source: §6.3.3 · p. 10

7 Limitations

The study’s findings are bounded by participant experience, the Wizard-of-Oz simulation, and imperfect comprehension measures.

7 Limitations

  • Only two BLV participants had prior 3D modeling experience, and strategies drawn from the first author’s engineering background may not generalize to other expertise levels.Alternative approaches from sighted assistants working with modelers of different backgrounds did not inform the scripts.Source: §7 · p. 11
  • Scripted responses let researchers test language descriptions independently of AI reliability, but could not represent AI hallucinations or richer model embeddings.The authors note that including all seemingly relevant detail in a 3D model representation remains an open challenge [2].Source: §7 · p. 11
  • Clay replicas are an imperfect comprehension proxy because quality also depends on motor skills and clay experience, while assessment lacked multiple raters and comparable models.The authors propose a structured two-phase protocol with validated criteria, multiple raters, and larger participant pools for future work.Source: §7 · p. 11

8 Conclusion

Two coordinated studies identify a gap between comprehension supported by natural-language model descriptions and the spatial understanding needed for editing. The authors propose a two-phase framework to assess that gap and guide targeted interventions.

8 Conclusion

  • Descriptions supported general model comprehension but not necessarily editing; four failure types—accuracy, completeness, expression, and navigation—extend beyond what sighted-expert-only evaluation can detect.Expression and navigation failures require observing BLV users navigating descriptions under task pressure; artifact-based clay replicas can reveal accuracy and completeness failures.Source: §8 · p. 11
  • The proposed framework assesses BLV users’ spatial mental models, then tests whether that comprehension supports editing specification to diagnose failure types and guide interventions.The authors offer it as a methodology for future refinement and evaluation of whether description-based systems support BLV modelers as creators.Source: §8 · p. 11

A Sighted-Assistant Pilot Study Participants

Table 3 summarizes demographics for sighted participants identified by the letter A and a number.

A Sighted-Assistant Pilot Study Participants

  • Table 3 summarizes sighted-participant demographics, with participants referenced by A and an identifying number.Source: Table 3 · p. 14

B Likert Scale Assessments from Sighted-Assistant Study

Table 4 reports Likert-scale assessments of the blind first author’s model comprehension.

B Likert Scale Assessments from Sighted-Assistant Study

  • Table 4 reports the blind first author’s self-rated confidence and a sighted researcher’s assessment of clay replicas as comprehension measures.Source: Table 4 · p. 14

D 3D Model Descriptions

The section presents a varied collection of 3D models, from functional parts and anatomical forms to animals and buildings. Table 5 lists the models in study order and provides Thingiverse IDs for locating their sources.

D 3D Model Descriptions

Related papers

Loading 2610.35562v1…