Source-linked AI summary
LLM Agents Can See Code Repositories
Dongjian Ma, Silin Chen, Yufei Yang, Yuling Shi, Yanfu Yan, Xiaodong Gu
TL;DR
Coding agents mostly treat repositories as text, leaving open whether visual structure can improve repository-level issue resolution. This paper evaluates multimodal repository representations and finds that hybrid text-and-vision access reduces token costs while maintaining or improving accuracy, unlike vision-only interaction.
Problem
Existing coding agents largely linearize repository artifacts into text, leaving limited evidence about whether visual structure is a more effective modality for repository context.
Method
The paper evaluates four multimodal models using SeeRepo, which supplements standard text-based code access with visual graph renderings of repository structure.
Results
Adding SeeRepo reduces token cost by up to 46% while maintaining or improving issue-resolution accuracy across models.
Takeaways & Limitations
Hybrid text-and-vision representations offer a better effectiveness–efficiency trade-off, with visualization most effective during fault localization.
Takeaways & Limitations
Evaluation uses only Python repositories from SWE-bench Verified, so generalization to other programming languages and architectural conventions remains unclear.
Abstract
from arXiv · showhide
Coding agents powered by large language models have demonstrated strong performance on software engineering tasks. Yet most agents consume repositories almost entirely as text, which differs from how human developers use visual structure such as folder hierarchies and dependency relationships to orient themselves in large codebases. With multimodal large language models (MLLMs), it is an open question whether agents can effectively benefit from visual representations of repositories. This paper presents the first systematic empirical study of visual repository representations for LLM-based agents on repository-level issue resolution. We evaluate four recent multimodal models. Our results show that a strictly vision-only setup degrades accuracy and increases token cost, because agents lack sufficient symbolic detail and compensate with repeated visual queries. In contrast, integrating visual graphs of repository structure as a supplementary modality alongside standard text interfaces helps agents understand structure more efficiently: input token consumption decreases by up to 26% while issue-resolution accuracy is maintained or improved. Visualization is most useful during fault localization and when the agent autonomously controls exploration depth. These findings point to a practical hybrid text-and-vision design for next-generation coding agents.
1 Introduction
This paper presents the first systematic empirical study of visual repository representations for coding agents on repository-level issue resolution. It finds that vision-only interaction hurts accuracy and raises token cost, whereas supplementary visual structure alongside text improves efficiency while maintaining or improving accuracy.
- Study scope: The study evaluates how modality, image–text balance, layout, and invocation stage affect repository-level issue resolution across four multimodal models.The models are GPT-5-mini, GPT-5.1, Doubao-Seed-2.0-Lite, and Kimi K2.5.
- Vision-only representation: Vision-only interaction degrades resolution accuracy across all three evaluated models.GPT-5-mini drops from 55.0% to 41.4% (−13.6), Doubao-Seed-2.0-Lite from 51.0% to 16.9% (−34.1), and Kimi K2.5 from 70.3% to 55.0% (−15.3).
- Hybrid text-and-vision representation: Supplementary visual repository context reduces token cost while maintaining or improving resolution accuracy.GPT-5-mini reaches Pass@1 of 55.4% (+0.4) with input tokens reduced by 25% and cost by 26%; Kimi K2.5 improves from 68.8% to 70.6% while reducing cost by 3%.
- Visual layout: Graph-based layout achieves the greatest token reduction among the tested visual layouts.It delivers −25% input tokens and −26% cost with Pass@1 of 55.4% (+0.4), while nested and tabular layouts yield marginal accuracy gains of +0.8 and +1.2 respectively.
- Invocation stage: Visualization is most effective when invoked during fault localization.At this stage, SeeRepo achieves Pass@1 of 55.4% (+0.4), reduces input tokens by 25%, and lowers cost by 26%.
2 Background
MLLMs jointly process visual and textual tokens, preserving spatial information that helps align visual regions with textual semantics and capture global or relational structure. Software repositories contain rich typed topological structures, motivating visual representations as an alternative or complement to text serialization, though systematic study remains absent.
- Multimodal Language Models: MLLMs transform image patches into visual tokens that a unified Transformer processes jointly with text tokens.A vision encoder produces dense embeddings, and a learned projection maps them into the language model’s token space.
- Multimodal Language Models: Preserved 2D patch arrangement and positional embeddings enable cross-modal attention to align visual regions with textual semantics.This lets MLLMs capture global context and relational structure that linear text sequences represent less directly, including in software engineering.
- Repository Structure: Repositories exhibit dependency graphs, call relations, and modular hierarchies that encode global program organization.A repository can be modeled as a directed heterogeneous graph whose nodes represent files, classes, and functions, with typed relationships such as contains, imports, inherits, and invokes.
- Visual Repository Representations: Prior work mostly serializes repository structure into text, which can obscure higher-order relationships and create substantial token overhead.Although recent work explores visual modalities for software engineering, a systematic study of visual repository representations remains absent; rendering the graph as an image preserves spatial locality and connectivity patterns.
3 Experimental Setup
The experiments evaluate multimodal coding agents on SWE-bench Verified and repository-level QA, measuring task performance and interaction efficiency under a graph-augmented Mini-SWE-Agent framework with fixed run budgets.
- Benchmarks: Experiments use SWE-bench Verified, a manually inspected benchmark of 500 real-world software engineering instances from widely used Python projects.The benchmark is selected to provide more reliable and reproducible evaluations of autonomous coding agents.
- Benchmarks: The evaluation tests whether visual structural grounding generalizes from issue resolution to evidence-heavy repository question answering.Repository-level QA is included alongside new issue-resolution tasks.
- Metrics: The study reports Pass@1, Overall Score, and Number of API Calls to measure issue resolution, repository QA quality, and interaction efficiency.Pass@1 is reported for SWE-bench Verified and SWE-Rebench, while Overall Score uses a 0–100 scale for SWE-QA.
- Implementation: The framework extends Mini-SWE-Agent with a repository graph module representing contains, imports, invokes, and inherits relations and rendering queried subgraphs as PNG images.Graphs are pre-constructed and serialized for reuse during inference, with agents querying them through an external tool.
- Implementation: Each agent run is capped at 250 interaction steps and a cost budget of $3.0 per instance, with results averaged over three independent runs unless specified otherwise.These limits define the default experimental setting.
4 RQ1: Effectiveness of Current MLLMs at Issue Resolution Tasks
Vision-only repository interaction substantially reduces issue-resolution accuracy across evaluated MLLMs while increasing token and API costs. Agents compensate for missing textual detail through repeated graph queries, but graph images alone provide insufficient symbolic information for accurate patches.
- Compensatory exploration: API calls increase by up to 95% as agents repeatedly query visual graphs to compensate for missing precise symbolic information.Kimi K2.5 nearly doubles its API calls (+95%), while repeated graph queries accumulate token overhead without improving accuracy.
- Model-specific strategies: Doubao-Seed-2.0-Lite shows a 379% input token surge alongside the steepest accuracy drop (−34.1).Its extensive visual-graph engagement reflects compensation for missing textual information.
- Interpretation: Graph images alone provide insufficient symbolic information for accurate issue resolution.Agents process visual repository structure to varying degrees, but some abandon visual exploration earlier and generate patches with incomplete context.
- Vision-only effectiveness: Accuracy drops by 13.6 to 34.1 points across all evaluated models under vision-only repository interaction.GPT-5-mini falls from 55.0% to 41.4% (−13.6), Doubao-Seed-2.0-Lite from 51.0% to 16.9% (−34.1), and Kimi K2.5 from 70.3% to 55.0% (−15.3).
- Vision-only effectiveness: Cost increases by up to 268% when agents receive visual graph outputs instead of textual command results.GPT-5-mini incurs a 42% cost increase, Doubao 268%, and Kimi K2.5 27%.
5 RQ2: Effect of Multimodal Context Integration
SeeRepo integrates visual dependency-graph representations with standard text interfaces to guide repository exploration during issue resolution. This multimodal design reduces interaction cost while preserving or improving effectiveness across models and additional benchmarks.
- Method: SeeRepo augments coding agents with visual renderings of repository dependency graphs alongside their standard text-based interfaces.The graph includes contains, imports, inherits, and invokes relationships and supports runtime queries by node identifier, edge type, and traversal depth.
- Evaluation: The evaluation applies SeeRepo to bug localization using imports for file hunting, invokes for logic hunting, and inherits or contains for hierarchy and path verification.Agents then read code, implement fixes with standard Bash commands, and execute tests for verification.
- Results: Kimi K2.5 increases input tokens by +5% while improving accuracy and decreasing overall cost by 3%.Its API call count changes only from 41 to 40, indicating that graph queries supplement rather than replace textual exploration.
- Cross-benchmark transfer: On SWE-Rebench Leaderboard, SeeRepo raises Pass@1 from 25.45% to 26.36% while cutting input tokens by 34.89% and cost by 9.6%.On SWE-QA, it improves the official Overall Score from 66.8, while the broader results preserve or improve effectiveness and reduce interaction cost.
6 RQ3: Effect of Visual Layout
The study compares graph, nested, and tabular repository visualizations and examines hierarchy depth. All visual layouts outperform the text-only baseline, while graph and agent-decided depth provide the strongest efficiency gains and deeper fixed depths improve accuracy.
- Layout comparison: All three visual layouts improve over the text-only baseline, while text representation reduces input tokens by 17% but decreases accuracy by −1.2.The comparison uses GPT-5-mini on 500 SWE-bench Verified instances.
- Layout comparison: Graph layout achieves the best token efficiency, reducing input tokens by −25% and cost by −26% while increasing accuracy by +0.4.Graph preserves repository dependency structure through directed edges, node-type icons, and dependency-direction arrows.
- Layout comparison: Tabular layout achieves the highest Pass@1 at 56.2%, with a +1.2 gain and a −16% cost reduction.Its efficiency gain is lower than graph layout’s.
- Hierarchy depth: Depth 4 achieves the highest Pass@1 at 57.2%, with a +2.2 gain, while increasing fixed-depth input-token usage as hierarchy depth grows.Depth 1 slightly hurts accuracy by −0.6 because shallow neighborhoods may omit key dependencies.
- Hierarchy depth: Agent-decided hop depth delivers a +0.4 Pass@1 gain with the lowest input tokens at −25% and cost at −26% across depth configurations.This configuration corresponds to the default SeeRepo setup evaluated in RQ2.
7 RQ4: Effectiveness of Visualization in Different Stages
Visualization is most beneficial during bug localization, where it improves issue-resolution performance while reducing token and cost overhead; invoking it during repair or validation instead underperforms the baseline.
- Stage-specific invocation: Issue resolution is divided into bug localization, patch repair, and patch validation, with visualization invocation isolated to one stage in each variant.Stage-specific access was controlled through system prompts and phase-specific tool definitions.
- Patch repair: 50.0% Pass@1 (−5.0) results from enabling visualization during repair, with only a marginal 5% cost reduction.After localization, broader dependency context can distract from precise textual reasoning during code editing.
- Patch validation: 51.6% Pass@1 (−3.4) results from deferring visualization to validation, still below baseline.Late structural context may prompt unnecessary follow-up modifications, expanding patch scope and increasing regression risk.
- Bug localization: 55.4% Pass@1 (+0.4) with 25% fewer input tokens and 26% lower cost makes localization the most effective invocation stage.Visual context narrows the candidate search space and identifies relevant code entities before repair, reducing redundant exploration.
8 Case Study
In the astropy__astropy-13398 case, SeeRepo replaced broad, iterative shell exploration with topology-aware graph queries that reached equivalent structural understanding in two steps. Both agents produced correct patches, while SeeRepo reduced token consumption by 32.6% and interaction steps by 29%.
- Case setup: The task required implementing direct ITRS↔AltAz and ITRS↔HADec transformations and registering the new module in astropy.The case study contrasts baseline and SeeRepo localization trajectories on this SWE-bench Verified instance.
- Baseline exploration: Without structural context, the baseline used iterative grep-and-read exploration, producing broad low-relevance output before inferring the relevant frame structure.Its process included repository listing, repository-wide keyword search, and subsequent file and class-level searches.
- SeeRepo exploration: SeeRepo reached the same structural understanding in two steps through graph queries that surfaced builtin_frames/ and its complete file roster.The roster exposed the *_observed_transforms.py naming pattern and provided an implementation template.
- Efficiency outcome: 32.6% lower total token consumption (143,558 →96,816) and 29% fewer interaction steps (17 →12) were achieved by SeeRepo, with both agents producing correct patches.The efficiency difference arose during localization, where baseline grep outputs and file reads accumulated approximately 25K low-information-density tokens.
- Interpretation: The case illustrates that topology-aware queries let agents spend more context on reasoning and code modification instead of trial-and-error navigation.SeeRepo supplied equivalent, more structured information at a fraction of the token cost.
9 Discussion
The study’s conclusions are limited by benchmark scope, efficiency proxies, and possible model-specific interactions with visualization. Future work should improve visualization scalability and control while incorporating dynamic repository signals.
- Limitations: The evaluation uses only Python repositories from SWE-bench Verified, so visualization benefits may depend on Python-specific organization and dependency structures.Whether the same efficiency and reasoning improvements generalize beyond Python remains unclear.
- Limitations: Token consumption and reasoning trajectory length serve as efficiency proxies but may miss interpretability and developer-aligned debugging strategies.Human evaluation and finer-grained behavioral analyses could provide a more comprehensive assessment.
- Limitations: Visualization effects may interact with model architecture, planning, and tool-use tendencies, potentially changing token reduction ratios across models.The experiments otherwise hold settings constant relative to the text-only baseline, apart from structural context.
- Future Work: Static Graphviz layouts may become visually dense for large repositories with complex dependency structures.Adaptive strategies could emphasize query-relevant subgraphs or progressively reveal structural information to improve interpretability and scalability.
- Future Work: More principled visualization-scope controls could use learning-based invocation policies or uncertainty-aware triggers to request context selectively.The current framework lets agents choose structural exploration depth, but principled scope mechanisms remain unexplored.
- Future Work: Dynamic signals such as execution traces and runtime dependencies could extend structural grounding beyond static analysis.Hybrid representations may distinguish frequently executed paths from rarely triggered branches, enabling more targeted localization and repair.
10 Related Work
Prior work has advanced repository-level software engineering agents through interfaces, memories, and search, while multimodal coding benchmarks have begun incorporating visual evidence. This work introduces visual repository structure as a new design dimension and finds efficiency benefits without sacrificing resolution accuracy.
- Software Engineering Agents: LLM-based agents have rapidly progressed on repository-level issue resolution through improved scaffolds, memories, and inference-time search.SWE-agent is cited as demonstrating that repository navigation, editing, and execution can substantially improve agent performance.
- Software Engineering Agents: This work is presented as the first study of multimodal code representations as a design dimension for software engineering agents.The study contrasts visual repository representations with prior improvements based on scaffolds, memories, or inference-time search.
- Software Engineering Agents: Visual graph images of repository structure consistently reduce token cost while maintaining resolution accuracy, especially when invoked during fault localization.The benefit is most pronounced at the localization stage.
- Multimodal Coding Agents: SWE-bench Multimodal extends SWE-bench with visual, user-facing JavaScript repository tasks containing images in problem statements or tests.The benchmark supports evaluation of agents in visual software domains.
11 Conclusion
The paper introduces SeeRepo, a multimodal framework that represents repository structure and dependencies visually while retaining code as text. Experiments show that vision-only access is insufficient, whereas appropriately designed multimodal integration can reduce cost while maintaining or improving issue-resolution accuracy.
- Contributions: SeeRepo renders structural and dependency relationships as visual graph images while retaining code content as text.The framework matches repository information types to modalities and exploits complementary MLLM capabilities.
- Key findings: Up to 34.1 points of accuracy degradation occurs when text access is replaced with graph images in the vision-only setup.The vision-only modality is insufficient and also inflates token cost.
- Key findings: Up to 46% lower cost results from adding SeeRepo alongside standard text tools while maintaining or improving resolution accuracy across all models.The experiments were conducted on SWE-bench Verified with four models.
- Key findings: Graph layout provides the best token efficiency, while agent-decided hop depth achieves the best cost reduction with competitive accuracy.Visualization is most effective during fault localization rather than repair.
- Implications: Visual repository representations are practical complements to text interaction when designed and invoked appropriately, because modality shapes agent behavior and cost.The conclusion motivates further exploration of multimodal representations for future coding agents.