PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Interactive Understanding of Code APIs

Dhananjay Ashok, Jesse Thomason, Jonathan May

arXiv:2609.32081v1cs.AI

TL;DR

External APIs obscure their source code, leaving interactive understanding less studied than code reading. The paper introduces PAU to evaluate black-box API exploration and description; frontier models struggle, while AAC tuning improves exploration, with Qwen3-8B matching GPT-5-mini.

  • Problem

    Many code-understanding evaluations provide source access, while external APIs obscure their internals and interaction-based understanding remains less understood.

    Source: §1 · pp. 1–2

  • Method

    PAU gives models black-box access to Python methods for exploratory querying and description, while AAC tuning uses a critic with privileged true descriptions to reward exploration.

    Source: §1 · p. 2

  • Results

    Over 45% of PAU methods stump every evaluated model, while AAC-tuned Qwen3-8B matches GPT-5-mini.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors argue that AAC’s successful adaptation for interactive code understanding opens the way for applying the paradigm in other domains.

    Source: §1 · p. 3

    PAU evaluates Python methods and uses only stateless methods, excluding some API functionality such as memory management.

    Source: §7 · p. 10

Abstract

from arXiv · show

Empowered by advances in Language Model agents, systems have made substantial strides in code generation and understanding. However, these approaches often rely on read access to the relevant code, an assumption which does not hold when dealing with external APIs. In this work, we introduce the PAU (Python API Understanding) benchmark, where we provide models with black-box, API-level access to code snippets. Models must query the API with exploratory inputs and draw insights from the resulting outputs, with the goal of describing the snippet's true functionality. By treating the code snippets as external tools that must be understood via interaction alone, PAU studies the more general problem of unsupervised tool understanding, specifically for tools implemented as Python methods. Despite recent progress in coding agents, even frontier models struggle to achieve high performance on PAU, with the best model (Claude-4-Opus) failing to understand over 45% of the PAU test set. An investigation into the common error modes reveals that models are overconfident; they often overrate the quality of their current hypothesis, leading to insufficient exploration and premature termination. Finally, we take inspiration from the Asymmetric Actor Critic (AAC) paradigm, frequently used in robot learning, to post-train models for interactive code understanding. Models trained with AAC conduct more active exploration of the APIs, with an AAC-tuned Qwen3-8B model matching the performance of GPT-5-mini.

1 Introduction

The paper introduces PAU to test whether models can infer Python API functionality through black-box interaction, finding that frontier systems struggle and exploring AAC tuning as an alternative.

1 Introduction

  • Existing code-understanding evaluations often give models source access, unlike external or proprietary APIs whose internals are obscured.The paper motivates studying functionality through behavior rather than source inspection.Source: §1 · pp. 1–2
  • PAU gives models black-box access to Python methods and asks them to infer and describe functionality by querying inputs and examining outputs.Strong performance requires meaningfully different inputs and insights from the resulting outputs.Source: §1 · p. 2
  • Every evaluated model, including frontier systems, fails to understand over 45% of PAU methods; static examples often perform similarly to interactive access.The authors suggest this limited improvement indicates models fail to progressively improve their understanding over multiple interaction rounds.Source: §1 · p. 2
  • Models often overrate their hypotheses, explore shallowly, and terminate before thoroughly investigating an API’s functionality.The authors hypothesize a link to sycophancy and inflated self-judgment.Source: §1 · p. 2
  • An AAC-tuned Qwen3-8B matches GPT-5-mini, while its critic uses privileged information to provide step-level rewards for exploration.The critic receives the method’s true description, which would not be accessible during deployment.Source: §1 · pp. 2–3

2 Python API Understanding Benchmark

PAU is a benchmark of Python methods that models must understand through black-box interaction. It combines description ratings with behavioral code-match evaluation and uses curated methods and test inputs.

2 Python API Understanding Benchmark

  • PAU contains 3.9K training and 741 test instances, each pairing a Python method with a gold description, context examples, and test inputs.Methods are stateless, and each call is processed independently.Source: §2 · p. 3
  • Models receive a method header, two contextual input-output examples, and black-box API access, then write a description of the method’s functionality.They can query arbitrary Python tuples and observe returned outputs.Source: §2 · p. 3
  • The benchmark rates descriptions on a 1–5 LM-as-a-judge scale and measures code match by testing a coding model’s reconstruction against the method’s outputs.The judge is validated against human judgments and more powerful models; code match uses a dedicated coding model.Source: §2 · p. 3
  • Neither evaluation approach guarantees a perfect metric because semantic equivalence of methods and descriptions is underspecified.The authors say their combination nevertheless provides clear signals of differences between models and methods.Source: §2 · p. 4
  • Benchmark construction filters out methods with fewer than five distinct test inputs or identical outputs, then checks statement and branch coverage.Average statement and branch coverage across test methods exceeds 99%.Source: §2 · p. 4

3 Evaluating Models for Interactive Function Discovery

PAU evaluates whether models can infer Python methods through black-box API interaction. Results show that current models often fail to use interactive access effectively.

3 Evaluating Models for Interactive Function Discovery

  • The evaluation spans instruction-tuned, coding, reasoning, and frontier models, comparing interaction-free in-context prediction with interactive API access.The in-context mode uses two examples, while the interactive mode provides API access.Source: §3 · p. 4
  • Interactive evaluation uses a fixed workflow in which models conceptualize and generate inputs, observe outputs, revise a running hypothesis, and may terminate when confident.The workflow follows a step-wise ReACT paradigm; fixed orchestration was chosen after open-ended agents struggled to query APIs properly.Source: §3 · pp. 4–5

3.1 Results

  • Even the per-instance Max aggregation fails to fully understand 25% of the test set, despite combining the best score from any model for each instance.These unsolved cases include hypotheses contradicted by the models’ own queries, although graduate-level annotators could reverse-engineer them through interaction.Source: Figure 3 · p. 5, §3.1 · p. 5
  • Claude-4-Opus scores 2 or lower on over 45% of methods, and its predicted descriptions yield code match scores below 80% on over 45%.The figure’s colored segments show the percentage of the test set within each score range.Source: Figure 3 · p. 5

3 Evaluating Models for Interactive Function Discovery

  • Alternative prompts, interaction structures, and a training-set RAG baseline yield similar performance, while software-testing hints do not improve results.The tested hints included boundary-value analysis, query diversification, and counterexample search strategies.Source: §3 · p. 5

3.1 Results

  • The per-instance Max score exceeds any individual model’s score, and Gemini-3-Flash and Claude-4-Opus have a Spearman rank correlation of 0.54.Different models struggle with different methods, leaving scope for improvement.Source: §3.1 · p. 6

4 Error Analysis

Failures among the strongest models are concentrated in insufficient exploration and reasoning rather than formatting or hallucination. Models often stop without testing their hypotheses adequately.

4 Error Analysis

  • Format-following errors arise when an agent supplies an invalid Python expression as API input, and are more common among the smallest models.The majority of larger models generate valid input tuples.Source: §4 · p. 6
  • Insufficient exploration and reasoning errors together account for over 90% of failures; format-following errors and hallucinations are nearly absent at this scale.The figure omits format following (0.0%) and hallucination (0.7%).Source: Figure 5 · p. 7
  • Hallucination means reasoning from fabricated input-output examples; it is largely absent from the strongest models’ failures.The reported share is 0.7% of failures for the three strongest models.Source: §4 · p. 7
  • Reasoning errors account for 43.3% of failures by the three strongest models, with agents sometimes accepting hypotheses contradicted by examples already observed.The authors describe this pattern as overconfidence and report that it persists even in powerful models.Source: §4 · p. 7
  • Insufficient exploration is the largest failure source for the three strongest models, accounting for 48.7% of their failures.Models often avoid inputs that could falsify a hypothesis or test edge cases, producing partial accounts of API functionality.Source: §4 · p. 7

5 Asymmetric Actor Critic for Interactive Code API Understanding

The section adapts Asymmetric Actor Critic to interactive API understanding, using a gold-description-aware critic to reward exploration. AAC tuning improves API-understanding scores and encourages more active exploration.

5 Asymmetric Actor Critic for Interactive Code API Understanding

  • Supervised fine-tuning did not improve results, while useful step-by-step exploration examples were unavailable and verifiable rewards were difficult to define.The training set uses different Python methods but matches the test set’s structure.Source: §5 · pp. 7–8
  • The critic uses the gold API description as privileged information to reward the agent’s actions during interactive training.The paradigm adapts asymmetric actor-critic methods from robot learning, where a critic can use training-only information unavailable to the agent.Source: §5 · p. 8, Figure 6 · p. 8
  • The reward schedule combines parsing-based and critic-based signals to encourage valid inputs, effective tests of hypotheses, and appropriate decisions to continue or stop exploration.The critic rewards continued exploration when the current hypothesis is inaccurate and penalizes it when the hypothesis is accurate.Source: §5 · p. 8

5.1 Results

  • AAC tuning raises average score by +0.13 and average code match score by +6.27%; Qwen3-8B reaches a 2.591 description score, near GPT-5-mini’s 2.595.The reported results are significant at p<0.05; the tuned model’s in-context performance remains comparable to its base model.Source: §5.1 · p. 9, Figure 7 · p. 9
  • AAC-tuned models make fewer format-following errors and query APIs more often with diverse inputs.The environmental reward discourages invalid Python tuple inputs, while the critic’s reward schedule penalizes premature termination.Source: §5.1 · p. 9, Figure 7 · p. 9
  • The results suggest AAC may be promising for unverifiable tasks where final solutions are known but intermediate reasoning is difficult to annotate.The proposed role for the critic is to use knowledge of the final solution to provide step-level reward signals.Source: §5.1 · p. 9

6 Related Work

Prior work studies tool use and black-box API discovery, but often assumes specifications, delegates exploration decisions to an oracle, or tests limited function sets. PAU evaluates a broader interactive API-understanding task and proposes AAC tuning.

6 Related Work

  • Formal black-box testing can provide guarantees but depends on API specifications, while learned specifications may still rely on API-specific interface descriptions to generate exploratory inputs.Other approaches enumerate possible specifications and test whether invariants hold; concurrent work applies language models and graph algorithms to web API traffic logs.Source: §6 · p. 9
  • Wei et al. [61] use an oracle that knows the true function to provide counterexamples and decide when to stop, delegating two challenges PAU studies to the oracle.Fine-tuning on those answer-aware trajectories is reported as harmful in the PAU setting.Source: §6 · p. 10
  • Geng et al.’s [19] test set has 100 list-mapping programs, whereas PAU covers varied arguments, types, and more complex logic; Gemini-3-Pro scores 4.13/5 on 80 of those programs in PAU’s interactive setting.Gemini-3-Pro achieves a perfect score on 87.5% of those programs; the cited prior works do not propose a solution to their task setups, while this work presents an AAC adaptation.Source: §6 · p. 10

7 Limitations and Ethical Considerations

The benchmark covers stateless Python methods rather than the full range of black-box APIs, and its test methods may have appeared in language-model pretraining. The authors also note a potential misuse risk if stronger models enable private-code extraction.

7 Limitations and Ethical Considerations

  • The benchmark omits some code-related functionality, including memory management, and simplifies the task by using only stateless methods.The authors say test methods sourced from popular benchmarks may have been exposed during language-model pretraining.Source: §7 · p. 10
  • The authors warn that stronger models may enable illicit extraction of private code through deep exploration and interactive understanding.They describe reverse engineering sophisticated APIs through interaction alone as likely beyond existing models’ capabilities.Source: §7 · p. 10

8 Conclusion

The conclusion presents PAU as a testbed for black-box Python API understanding and reports that frontier models struggle with the task. Its source datasets include existing Python-code benchmarks, with CodeAlpaca used for training data generation.

8 Conclusion

  • PAU tests interactive understanding of Python APIs, and the conclusion reports that no evaluated model understands more than 45% of its test APIs.The conclusion identifies overconfidence and premature exploration termination as recurring failure modes, and reports that AAC-tuned models explore more actively and understand APIs better.Source: §8 · p. 10

A.1 Source datasets

  • PAU contains 3.9K training and 741 test instances sourced from existing benchmarks, while CodeAlpaca is used to generate training data alone.The listed source datasets include HumanEval, MBPP, and CruxEval; CodeAlpaca comprises instruction-following samples for code-related tasks.Source: §A.1 · p. 16

A.2 Data Cleaning

The data-cleaning pipeline filters and anonymizes Python functions, validates that they can run safely, and creates diverse, usable examples for evaluation.

A.2 Data Cleaning

  • Functions are filtered to retain well-formed Python definitions, isolating the target function and necessary helpers while discarding unsupported constructs.Multi-function files preserve helper definitions as preamble code.Source: §A.2 · p. 16
  • Anonymization removes docstrings and return annotations, renames functions and arguments, and discards functions importing unsafe system-level modules.Arguments are rebound inside the function body, and a validator stub is prepended.Source: §A.2 · p. 16
  • A generated input validator checks argument types and structural constraints; functions that fail compilation, instantiation, or isolated loading are dropped.The pipeline also removes functions whose initialization creates files or directories.Source: §A.2 · p. 16
  • Generated examples must satisfy validator constraints and execute successfully; functions without valid examples are discarded.The pipeline retains two training examples and requires at least five held-out test examples per function.Source: §A.2 · p. 18
  • Test examples achieve 99% branch coverage on average across retained functions, and coverage below full branch coverage is flagged.The prompts for validation-function and example creation are shown in Figures 8 and 9.Source: §A.2 · p. 18

A.3 Final Benchmark

The appendix reports the final PAU benchmark's size and discusses contamination risk, task recoverability, and evaluation prompts. It argues that model failures reflect exploration limits, while acknowledging that not every instance is guaranteed recoverable.

A.3 Final Benchmark

  • 741 Python methods remain after cleaning, each with two contextual examples and an average test-suite size of 7.45.Each method can be queried with inputs and either return a value or raise TypeError for invalid parameters.Source: §A.3 · p. 19

A.4 Data Contamination

  • Descriptions may have leaked from popular source benchmarks, but anonymized names and generated contextual examples limit the direct impact of contamination.The paper also points to frontier models' poor PAU performance as evidence that the benchmark remains challenging.Source: §A.4 · p. 19

A.5 Addressing Impossibility of API Discovery:

  • Different models fail on different instances, and human annotators recovered two universally unsolved examples through interaction, though the authors cannot guarantee every instance is recoverable.The authors treat the MAX setting as evidence of headroom beyond individual-model performance.Source: §A.5 · p. 21

B Evaluation Metrics

B.1 Validating Robustness of Evaluation Metrics

Evaluation checks find that model rankings are robust to prompt and judge changes. Alternative interaction structures and software-testing guidance do not reliably improve performance, and some variants reduce it.

B.1 Validating Robustness of Evaluation Metrics

  • 0.08 description-score variation and 0.02 code-match-score variation across prompts were below the average-score difference between any two models.Changing prompts did not alter relative model rankings.Source: §B.1 · pp. 21–22
  • Manual annotations and stronger judge or coding models left model rankings unchanged, supporting the robustness of the evaluation rankings.The manual check covered 50 predicted descriptions per model and code scores for a subset of 25 descriptions.Source: §B.1 · p. 22

C Interactive Mode Baseline:

  • The interactive baseline permits up to 25 API calls, while prompt and memory-engineering alternatives did not substantially improve performance.Models typically alternate between proposing inputs, observing outputs, and revising their understanding.Source: C Interactive Mode Baseline · pp. 22–23

C.1 Prompt and Structure Variation

  • 0.0061 average description-score variance across prompt variants was far smaller than average score differences between models, while more complex variants degraded some models.GPT-4o-mini scored 2.419 vs 2.384 with batch interaction and 2.12 with structured hypothesis tracking.Source: §C.1 · pp. 23, 25

C.2 Injecting Software Testing Specific Knowledge

  • Software-testing guidance had an overall negative effect across evaluated models, with smaller models burdened by longer contexts and stronger models narrowing their exploration.Source: §C.2 · p. 26

C.3 RAG-based approach

The memory-based approach did not significantly improve performance, and most models performed similarly in in-context and interactive settings. The reward schedule describes a three-step exploration cycle with separate rewards for each step.

C.3 RAG-based approach

  • The retrieve-then-reflect baseline explores the test function, retrieves the five most similar training functions, then uses critiques informed by their known descriptions to guide re-exploration.The critiques identify exploration gaps by comparing predictions and chosen queries with true behavior and more revealing examples.Source: §C.3 · pp. 26–27
  • Additional memory produced no significant performance increase; Qwen3-8B improved by +0.015, a result not significant at p < 0.2.GPT-4o-mini and Qwen3-1.7B decreased marginally after incorrectly revising accurate hypotheses.Source: §C.3 · p. 27

D.1 Training Hyperparameters

  • Training uses GRPO via SkyRL, with hyperparameters listed in Table 2.Table 2 is identified as the SkyRL training-hyperparameters table.Source: §D.1 · p. 27, Table 2 · p. 29

D.2 Reward Schedule (Supervised Setting)

  • The supervised reward schedule uses a reasoning → input proposal → reflection cycle, with separate scalar rewards and an informativeness judge for proposed inputs.The actor proposes an exact Python argument tuple, observes the result, then states whether it has identified the function and gives its hypothesis.Source: §D.2 · pp. 27–28

C.4 Incontext Baseline

  • For most models, in-context performance was not significantly lower than interactive performance, and even frontier models struggled to fully use interactive API access.The comparison uses two contextual examples and a direct description prediction in the in-context setting.Source: §C.4 · p. 27, Figure 18 · p. 28

D.2 Reward Schedule (Supervised Setting)

  • Termination is rewarded or penalized according to the hypothesis score, while reaching 40 steps incurs a −1.0 penalty and all three steps receive length penalties.Verbose inputs are penalized using a tight length threshold of τ=30.Source: §D.2 · pp. 27–29

D.3 What Does the Asymmetric Critic Add?

AAC outperforms the alternative training paradigms, while answer-aware teacher trajectories and a critic without privileged information can harm performance. The appendix also reports the computational hardware and run-time range.

D.3 What Does the Asymmetric Critic Add?

  • AAC is the only approach that substantially improves over the base model and outperforms all alternatives on description score.The comparison uses an LM-as-a-judge score on a 1–5 scale.Source: Table 4 · p. 30, §D.3 · p. 30
  • SYM-INFO harms performance, FINAL's sparse reward fails to converge, DISTILL yields smaller gains, and CodeARC-SFT degrades below the base model.The authors attribute CodeARC-SFT's degradation to answer-aware teacher trajectories that are too optimal and bias models toward fewer queries; AAC uses privileged information for reward assignment while optimizing agent-generated sequences.Source: §D.3 · p. 30

E Computational Hardware

  • Inference used 8 NVIDIA A40 GPUs for most experiments, while AAC training of 8B models used 2 NVIDIA H100 GPUs; benchmark runs ranged from 3 to 120 hours.Most experiments could be conducted with less than 16GB of GPU RAM.Source: E Computational Hardware · pp. 30–31

Related papers

Loading 2609.32081v1…