DataSense-Bench: The First Step Toward an AI Scientist

• Training-data selection is difficult because observable qualities such as low loss or a successful trajectory do not establish how much a particular model will learn from an example.Reliable judgment requires connecting data properties to learning under a fixed training procedure.Source: §1 · p. 2

Our Benchmark

• DataSense-Bench tests both whether an agent’s top-ranked subset yields a useful model and whether its predicted subset ordering matches observed performance.A strong top choice need not imply an accurate ordering, and correctly ordering weak subsets need not yield a useful recommendation.Source: §1 · p. 2

Our Findings

• Across TBLite and BFCL, selection gains over random are limited, and agents do not reliably rank groups; Astra finds the best group in all three BFCL runs but only one of three TBLite runs.The introduction also reports that no agent evaluated on both tasks finds the best group in all three runs on each task.Source: §1 · p. 2
• The benchmark instantiates data selection in terminal problem solving and multi-turn tool use, using OpenThoughts-Agent and EnvScaler data with TBLite and BFCL evaluation, respectively.Source: §1 · p. 2

Benchmarks for CLI Agents

• CLI-agent benchmarks span realistic terminal tasks, repository issue resolution, scientific programming, and machine-learning experimentation; PostTrainBench also evaluates broader post-training choices under a compute budget.DataSense-Bench instead isolates data-selection judgment before training feedback, for a fixed base model and protocol.Source: §2 · pp. 2–3

Training Data Selection

• Training-data selection methods use quality, difficulty, importance, or coverage signals, while RHO-LOSS prioritizes examples expected to reduce generalization loss.These approaches motivate the quality, loss, and coverage signals examined in the paper’s procedure analysis.Source: §2 · p. 3

Performance Forecasting

• Performance forecasting ranges from example-level influence and contribution estimates to subset- or mixture-level datamodels and performance predictors.These approaches differ in their use of target examples, gradients, and training feedback; DataSense-Bench evaluates subset selection and relative downstream-performance forecasts.Source: §2 · p. 3

3.1 TASK FOR CLI AGENTS: SELECT USEFUL DATA AND PREDICT ITS VALUE

• Agents select five equal-sized, disjoint subsets and rank them by expected post-training performance, using inference-only access without training models or seeing evaluation tasks.They may inspect data, execute analysis code, and run base-model forward passes.Source: §3.1 · p. 4
• The benchmark studies terminal problem solving and multi-turn tool use, fine-tuning a fresh Qwen3-4B copy on each selected group of complete trajectories.Tool-use groups contain 50 trajectories and terminal groups contain 1,000; the tasks are evaluated on BFCL and TBLite, respectively.Source: §3.1 · p. 4

3.2 POST-CHECK AND POST-TRAINING EVALUATION

• After submission validation, each group is used to fine-tune a fresh base-model copy, and outcomes are evaluated with the task’s shared scoring protocol without revising the submitted predictions.The agents submit group assignments and a method report before observing training or evaluation outcomes.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 EVALUATION METRICS

• Selection quality measures the top group’s score and gain over same-size random selection, while ranking accuracy tests whether it is best among selected groups and whether the full ordering agrees.Best-group accuracy counts a tied highest score as correct; Spearman correlation uses average ranks for ties and is undefined when all five outcomes are equal.Source: §3.3 · pp. 5–6

4.1 DOWNSTREAM EVALUATION

• Models are evaluated on TBLite terminal tasks and BFCL tool-use cases after fine-tuning, while selection agents cannot access the evaluation tasks.TBLite reports average reward over 100 tasks with three attempts per task; BFCL covers 800 cases across four categories and uses three evaluation seeds per checkpoint.Source: §4.1 · p. 6

4.2 SELECTION QUALITY AND RANKING ACCURACY

• BFCL g1 gains over random range from 0.35 to 1.78 percentage points, while TBLite g1 rewards range from 11.00% to 12.54% versus 11.55% for random; agents also rank groups unreliably.On BFCL, average Spearman correlations range from −0.27 to 0.83; on TBLite, they range from −0.50 to 0.30. Astra identifies the best group in all three BFCL runs but only one of three TBLite runs.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 HOW AGENTS MAKE SELECTION DECISIONS

• Across 30 TBLite and 18 BFCL procedures, agents often use similar data signals but interpret their training value differently and vary their recipes across runs.TBLite procedures commonly use completion cues, length, and format checks; agents also construct predicted group orderings through different scoring, partitioning, and selection rules.Source: §4.3 · pp. 7–8

4.4 CASE STUDY

• Fable 5.1’s terminal-task recipe combines example-level scores with group-level coverage choices, and its best first group coexists with a mistaken remaining order.g1 achieves 14.83% reward, while g2 reaches 11.08% and g3 reaches 12.12%.Source: Figure 4 · p. 8, §4.4 · p. 9

4.5 RESOURCE REQUIREMENTS

• Selection cost is distinct from SFT and downstream-evaluation cost, and evaluation is the larger time component on both tasks.Each selection requires five trained checkpoints; the TBLite evaluation time is a scheduling estimate rather than a measured end-to-end runtime.Source: §4.5 · p. 9, Table 2 · p. 9

5 CONCLUSION

• Across terminal problem solving and multi-turn tool use, selection gains over random selection are limited, and agents do not reliably rank selected groups.Ranking ability also does not hold consistently across tasks.Source: §5 · p. 9
• DataSense-Bench evaluates data sense by testing subset selection and prediction of relative post-training performance.The benchmark frames data sense as connecting training-data properties to what a model will learn.Source: §5 · p. 9
• The benchmark covers selecting and ranking existing SFT data; future work should extend it to reinforcement-learning post-training and generating new training data.Those extensions would test whether agents can anticipate which experiences to collect or create and how those choices will improve learning.Source: §5 · p. 9

AI USE STATEMENT

• OpenAI Codex assisted with drafting data-selection prompts, preparing code snapshots, monitoring experiments, and polishing the manuscript.Source: AI USE STATEMENT · p. 10
• Frontier models acted as experimental participants, as described in Section 3.1.Source: AI USE STATEMENT · p. 10
• Numerical results were computed from recorded selections and downstream evaluations, and the authors were responsible for the final text, methods, results, and claims.Source: AI USE STATEMENT · p. 10

BROADER IMPACT

• DataSense-Bench makes AI systems’ data choices testable, supporting closer scrutiny of claims about autonomous AI research and recursive self-improvement.Source: BROADER IMPACT · p. 10
• Optimizing benchmark performance alone may favor narrow capabilities or amplify biases in candidate data, while repeated aggregate-performance selection could exclude rare but important cases.Source: BROADER IMPACT · p. 10
• The paper calls for considering data provenance, privacy, and representativeness alongside strong results, and says human review remains necessary when model errors affect people.Source: BROADER IMPACT · p. 10

A.1 TRAINING CONFIGURATION

• Both tasks use full-parameter SFT with assistant-token supervision; TBLite includes thinking in supervision and uses a gradient-clipping threshold of 10−3.Selected BFCL trajectories are converted into assistant-supervised examples, and agent-selected groups within each task share the realized training configuration.Source: §A.1 · p. 15
• Table 3 describes the configurations of the two selection-and-ranking tasks, with group sizes counting complete trajectories.Source: Table 3 · p. 15

A.2 SELECTION MODELS AND INFERENCE CONFIGURATION

• The appendix maps result names to selection-agent model identifiers and notes that CLI wrappers leave selection temperature, sampling seed, and reasoning-effort override to provider defaults.Summary-display options affect recorded traces but do not specify a reasoning-token budget.Source: §A.2 · p. 15, Table 4 · p. 15

A.3 DOWNSTREAM EVALUATION AND RESULT AGGREGATION

• The TBLite harness executes model-proposed commands through Terminus-2, while BFCL evaluates each seed on the full 800-case multi-turn suite.TBLite task time limits range from 300 to 3,600 seconds; BFCL uses a 65,536-token context and temperature 0.7.Source: §A.3 · pp. 15–16
• Checkpoint scores are averaged before run-level metrics, ranking compares all five groups within valid runs, and failed submissions remain in run-level averages.TBLite checkpoint scores include all 300 task attempts; random selection repeatedly evaluates one fixed subset per task.Source: §A.3 · p. 16

A.4 SELECTION-TIME CHECKS AND RECIPE INFORMATION

• Selection checks disable common training entry points and retain traces, scripts, assignments, reports, and validation outputs for review; all groups use a fixed recipe.The checks do not certify every access restriction: metadata are not weight-content hashes, and keyword matches can arise from reading recipes.Source: §A.4 · p. 16

A.5 COST AND RUNTIME MEASUREMENT

• Cost estimates exclude GPU inference, fine-tuning, and evaluation; Astra’s incomplete usage and duration records make its cost a lower bound and omit its duration.Training-time samples exclude queueing and model loading, with separate reported samples for TBLite jobs, BFCL runs, and BFCL evaluation jobs.Source: §A.5 · p. 16

B.1 SELECTION PROCEDURES

• Terminal selection summaries distinguish valuation signals from grouping rules, with runs using varied quality, execution, loss, structural, and completion features and different grouping schemes.Examples include deduplication and redundancy penalties, similarity-penalized ordering, and consecutive score-ranked blocks.Source: §B.1 · pp. 17–18

B.2 BFCL SELECTION PROCEDURES

• The BFCL procedure review covers 18 runs with 50-trajectory groups; quality, correctness, and difficulty describe agents’ selection signals, not independent utility labels.The appendix summarizes submitted group-formation rules using final method reports and selection scripts or command traces.Source: §B.2 · p. 18

• Evaluation scores are averaged within each checkpoint, while ranking metrics are computed within each selection before aggregation.Source: §B.3 · p. 18

Terminal group-level results

• Terminal group results report rewards on a 0–1 scale, with within-run shading and ρ comparing predicted group order with observed performance.Each reward averages 100 tasks with three attempts per task; g1 rank counts groups scoring strictly higher.Source: Table 7 · p. 20

BFCL group-level results

• BFCL results report accuracy for 50-trajectory groups, with within-run shading, full-order agreement ρ, and g1 rank.Scores average the 800-case evaluations across evaluation seeds.Source: Table 6 · p. 19, Table 8 · p. 21

C.1 SELECTION CONFIGURATION EXAMPLES

• Selection configuration examples specify local settings, and the BFCL candidate pool contains 8,962 trajectories after validation-split exclusion.The max-len field gives the selection-time length limit; realized training settings are documented elsewhere in the appendix.Source: §C.1 · p. 22

C.2 TERMINAL DATA SELECTION

• Terminal selections are evaluated as agents solving unseen tasks in fresh Unix environments, with success determined by automated tests.The prompt describes a fixed fine-tuning recipe and specifies the group-assignment deliverable.Source: §C.2 · p. 22
• Terminal agents are asked to value examples for downstream performance, prioritize the strongest single group, then order groups from highest to lowest expected value.The prompt allows single or combined signals, including loss, trajectory quality, difficulty, efficiency, diversity, coverage, and length effects.Source: §C.2 · p. 22
• Terminal selection uses five disjoint groups of 1,000 rows, permits forward passes but bars training and benchmark access, and uses a fixed recipe.The dataset’s message column is formatted with the chat template, truncated at max length, and masks non-assistant tokens from loss.Source: §C.2 · pp. 22–23

C.3 MULTI-TURN TOOL-USE SELECTION

• Tool-use selection also uses five disjoint groups and inference-only access; its fixed recipe is the only training variable across selected subsets.The prompt describes about 9,000 trajectories averaging roughly 18 turns and 6,500 rendered tokens each, within a four-hour selection budget.Source: §C.3 · pp. 23–25
• Tool-use agents must predict trajectory value for downstream performance, make group 1 as strong as possible, and order the remaining groups by predicted value.Suggested signals include trajectory quality, tool-call correctness, efficiency, environment and tool coverage, and length effects.Source: §C.3 · p. 24
• Tool-use submissions must pair each selected row position with its separate metadata identifier, and include only the selected rows in correctly sized groups.The prompt warns that row position and metadata index differ for most records, so confusing them can select different trajectories.Source: §C.3 · p. 24
• Tool-use trajectories contain alternating user and assistant turns, and training expands each trajectory into one supervised sample per assistant turn.The recipe removes prior assistant reasoning from context while supervising the current assistant turn, using the chat template and truncation limit.Source: §C.3 · p. 25