PaperlayerView exact revision on arXiv

Source-linked AI summary

RunningTab: Direct Workspace Interaction with Environment-Side Tabs

Jinheon Baek, Soyeong Jeong, Yumin Choi, Dongsu Han, Sung Ju Hwang

arXiv:2610.10444v1cs.AIcs.CL

TL;DR

Workspace deliverables combine information scattered across files, while task requirements and encountered content can be displaced from the context window. RunningTab adds an environment-maintained tab that records requirements, reads, and unopened candidates. Across three benchmarks and three LLMs, it improves over plain DWI and model-kept-record baselines.

  • Problem

    Workspace tasks assemble content scattered across files, but task requirements and what has been read or listed can be displaced from the context window without a lasting record.

    Source: §1 · p. 1

  • Method

    RunningTab keeps an environment-side task tab with agent-stated requirements, environment-captured read excerpts and unopened-file candidates, and a review and finish check.

    Source: §1 · p. 1, §3.2 · p. 5, §6 · p. 9

  • Results

    RunningTab improves over direct workspace interaction alone and model-kept-record baselines on three benchmarks with three LLMs.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings position an environment-side tab as a complement to terminal access, keeping track of the task until the deliverable is handed in.

    Source: §6 · p. 10

    In real-world settings, RunningTab may retain model biases and errors and store sensitive excerpts, motivating safeguards such as access controls and deliverable review.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

Much knowledge work produces new deliverables from files a workspace already holds, and LLM agents are beginning to take such work over. Through direct corpus interaction, an agent can search and read any of those files from a terminal with no indexing, and producing a deliverable from many of them in this way is what we call direct workspace interaction (DWI). Reaching the files, however, is only half the task: nothing keeps track of what the task asks for, what has been read, and what was listed but never opened, all of which slip through the context window without leaving a trace, so an agent may extract a figure and still deliver a report without it. To address this, we present RunningTab, a framework that equips direct workspace interaction with an environment-side tab: a per-task record of what the task still owes, kept by the environment alongside the agent. Specifically, the agent adds its requirements, while the environment records every file read as an excerpt with its provenance and every listed but unopened file as a candidate; the agent can then see each requirement beside its best-matching excerpts and top unopened candidates, resolve it against matching content or set it aside with a reason, and, should it try to finish with requirements still open, receive them in a finish check. We validate RunningTab on three benchmarks with three LLMs, where it consistently outperforms plain DWI and baselines that keep the record in the model, while its tab usually holds the values a deliverable needs once seen.

1. Introduction

Workspace deliverables combine information scattered across files over a long horizon, but task requirements and encountered content can be displaced from the context window. RunningTab adds an environment-maintained record to direct workspace interaction, and the paper reports improvements over plain DWI and model-kept-record baselines.

1. Introduction

  • Direct workspace interaction (DWI) has an agent use terminal tools to search and read raw workspace files while assembling a deliverable over many steps.Unlike retrieval over a prebuilt index, this approach requires no indexing or preprocessing.Source: §1 · p. 1
  • DWI leaves task requirements, read files, and listed-but-unopened files untracked, so observations can be displaced and relevant content omitted from the deliverable.The introduction also notes that an agent may list a relevant file without opening it.Source: §1 · p. 1
  • RunningTab gives DWI an environment-side tab that records task requirements, read excerpts with provenance, and listed-but-unopened files as ranked candidates.The agent reviews open requirements beside matching excerpts and candidates, resolves them against matching content or with a reason, and receives a finish check if any remain open.Source: §1 · pp. 1–2
  • Across three benchmarks and three LLMs, RunningTab improves over plain DWI and baselines in which the model keeps track of the task.The tab usually holds the values a deliverable needs once seen.Source: §1 · p. 2

2. Related Work

Prior work studies workspace agents, direct corpus search, and agent memory. RunningTab addresses the record of task requirements and reached content that these lines of work leave to the model or do not track.

Workspace Agents

  • Workspace benchmarks find deliverables often omit content held in the workspace, with prior diagnoses emphasizing document understanding, heterogeneous file understanding, and lineage tracing.The paper distinguishes these issues from content already opened and observed but later missing from the deliverable.Source: §2 · p. 2

Direct Corpus Interaction

  • Direct corpus interaction gives agents the corpus to search with terminal tools, while prior work typically targets short answers; RunningTab instead records task requirements and which opened files satisfy them.It changes what happens to content after it is reached, not how the agent reaches files.Source: §2 · p. 3

Agent Memory

  • Unlike agent-memory and task-record systems where models write and mark items complete, RunningTab's environment captures reads and listings and keeps the record outside the context window.Items are marked done only against a captured excerpt or set aside with a reason.Source: §2 · p. 3

3. Method

The method frames multi-file deliverables as direct workspace interaction and adds an environment-maintained tab to track requirements, reads, and unopened files. The tab supports evidence-linked resolution and a final check before delivery.

3.1. Direct Workspace Interaction

  • DWI lets agents search raw workspace files through terminal tools, but leaves task progress and previously read content to the context window.Workspace tasks combine elements scattered across files, and a deliverable is complete only when every requested element is carried into it.Source: §3.1 · pp. 3–4
  • 51.1% of DCI’s lost rubric points came from extraction, versus 17.6% from discovery; 19.8% of observed numeric values were absent from deliverables.Extraction comprised 12.3% Shown but Missing and 38.9% Dropped or Distorted.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2. Environment-Side Tab

  • The environment-side tab comprises requirements, a read log of excerpts with provenance, and candidates for listed but unopened files.The environment captures reads and listings outside the context window, while the agent states the requirements.Source: §3.2 · p. 5

3.3. Keeping the Tab during the Task

  • At task start, the agent states requirements; the environment logs file content as read and paths alone as listed, ranks candidate paths with BM25, and reports tab status with observations.Read-log entries preserve observations as shown to the agent, without parsing or summarization.Source: §3.3 · pp. 5–6

3.4. Settling the Tab before Delivery

  • Before resolution, review pairs each open requirement with BM25-ranked read-log matches and unopened files; resolution requires a matching citation or a reason for unavailability.At the stop, a finish check reports open requirements and unopened files and asks the agent to add what is missing before finishing again.Source: §3.4 · p. 6

4. Experimental Setup

The experiments compare RunningTab with direct corpus interaction and model-kept tracking baselines across workspace tasks and document QA. They use three benchmarks, three LLMs, and an average over three runs.

Benchmarks

  • Three benchmarks cover workspace deliverables, company tasks, and enterprise-document question answering.Workspace-Bench evaluates rubric pass rate and task completion; TheAgentCompany reports success and score; OfficeQA Pro reports accuracy at 0% and 1% allowable absolute relative error.Source: §4 · pp. 6–7

Baselines

  • RunningTab is compared with direct corpus interaction and three baselines that keep task tracking in the model.The listed model-side approaches include TODO, Self-Refine, and Self-Tracking; RunningTab keeps the task record in an environment-side tab.Source: §4 · p. 7

Implementation

  • All methods use DCI-Agent-Lite with a 300-turn budget, a one-hour task limit, and three LLMs.The implementation follows runtime context management level 3 and reports averages over three runs.Source: §4 · pp. 7–8

5. Experimental Results

RunningTab outperforms plain DCI and model-kept tracking baselines across every benchmark, metric, and LLM. The results indicate that environment-captured reads and listings, surfaced beside requirements, contribute to its gains, especially on reading-heavy tasks.

Main Results

  • RunningTab achieves the best performance on every benchmark, metric, and LLM, while model-kept tracking baselines are less effective than the environment-side record.The comparison spans assembling office-file deliverables, company tasks, and answering questions from archival documents.Source: §5 · p. 8

What the Tab Holds

  • The tab stores a median of four to five requirements, 6 to 18 read-log entries, and 147 to 294 listed-but-unopened files per Workspace-Bench attempt.Read-log excerpts span 12K to 40K characters from 5 to 11 distinct files; 69.9% to 88.0% of open requirements are shown with a matching read-log entry.Source: §5 · p. 8

Recovering Shown but Missing Content

  • 94.8% of shown-but-missing rubric values are held in the tab; masked-value recovery is 8.4% without it, 45.8% from review, and 78.1% from the whole read log.The figure reports held values on the left and recovery by the part of the tab provided on the right; the read log stores file content, not computed values or values seen only in listings.Source: §5 · p. 8, Figure 4 · p. 8
  • 27.0% of Shown but Missing rubrics are solved with RunningTab, versus 20.7% with DCI.These are rubric success rates for the Shown but Missing category.Source: §5 · p. 8

Ablation Studies

  • Removing any tab component lowers Pass Rate and TCR@70; removing environmental capture causes the largest Pass Rate drop, to 41.36 against DCI's 41.63.Removing the review causes the next largest drop; the evidence check and finish check complement capture and review at requirement resolution and task completion.Source: §5 · p. 9, Table 4 · p. 9

Gains on Reading-Heavy Tasks

  • RunningTab's gain over DCI generally grows with reading volume, averaging twice as much in the heaviest-reading third as in the lightest.DCI's Pass Rate falls by 16 to 27 points between the lightest- and heaviest-reading thirds.Source: §5 · p. 9, Figure 5 · p. 9

6. Conclusion

RunningTab adds an environment-maintained record to direct workspace interaction, where agents search and read workspace files to produce deliverables. Across three benchmarks and three LLMs, it improves over direct workspace interaction alone and model-maintained tracking baselines.

6. Conclusion

  • RunningTab equips direct workspace interaction with a per-task record maintained by the environment outside the context window.The agent states requirements; the environment records each read file as an excerpt with its source and each listed but unopened file as a candidate.Source: §6 · p. 9
  • Across three benchmarks and three LLMs, RunningTab improves over direct workspace interaction alone and baselines in which the model itself keeps track of the task.Ablations attribute most of the gain to what the environment captures on its own, and the tab usually holds the values a deliverable needs once seen.Source: §6 · p. 10
  • Agents review open requirements beside matching excerpts and candidates, resolve them by citing a supporting excerpt or giving a reason, and receive a finish check when stopping.The check identifies any requirements still open before delivery.Source: §6 · p. 10

Ethics Statement

The work aims to improve LLM agents’ production of workspace-file deliverables using existing models and public benchmarks, while noting risks from model errors and stored sensitive content.

Ethics Statement

  • RunningTab agents may retain their models’ biases and errors, and the tab may hold sensitive content from files read in real workspaces.The authors encourage keeping the tab under the workspace’s access controls and reviewing deliverables in real-world deployments.Source: Ethics Statement · p. 10
  • The work aims to improve LLM agents’ production of deliverables from workspace files using existing models and public benchmarks.Source: Ethics Statement · p. 10
  • For real-world deployment, the authors encourage access controls for the tab and review of deliverables.Source: Ethics Statement · p. 10

A. Qualitative Examples

The appendix presents one recorded attempt showing RunningTab’s environment-side tab, then compares direct workspace interaction and RunningTab on two other Workspace-Bench tasks using the same LLM.

A. Qualitative Examples

A.1. Example: An Environment-Side Tab of RunningTab

A recorded attempt on a business-travel guide task illustrates how RunningTab tracks requirements, read policy files, and unresolved candidates through completion. The agent writes requirements and resolutions; the environment maintains the rest of the tab.

A.1. Example: An Environment-Side Tab of RunningTab

  • The example task requests a business-travel and reimbursement guide covering five topics drawn from four policy files.Source: §A.1 · p. 15
  • Before opening files, the agent adds six requirements; the environment logs 98 listed files as candidates and records four opened policy files as read-log entries with text, path, and command.As files move from candidates to the read log, the count changes from 98 candidates and no entries at turn 3 to 94 candidates and four entries at turn 8.Source: §A.1 · p. 15
  • The tab links the reimbursement-standard requirement to read-log entry L2, which contains the policy’s Article 7 text.At review, the requirement appears beside the matching line and an unopened file on the same subject; the agent later resolves it by citing L2.Source: §A.1 · p. 15
  • At turn 22, the finish check confirms all six requirements are resolved before delivery.Source: §A.1 · p. 15
  • The agent writes only the initial requirements and their resolutions; the environment keeps and returns the read log, candidates, status line, review, and finish check.This division distinguishes the environment-side tab from a record the model keeps in its context.Source: §A.1 · p. 15

A.2. Case Study 1: Tracking Every Source of a Deliverable

In a water-and-electricity manual task using four policy files, RunningTab retained the policy’s chapter and article structure, while DCI did not.

A.2. Case Study 1: Tracking Every Source of a Deliverable

  • Both methods read the same four policy files within a few turns to prepare a water-and-electricity-saving manual.The comparison used the same LLM.Source: §A.2 · p. 15
  • RunningTab tracked one requirement per policy file, logged read files, showed matches in a review, resolved requirements, and returned a finish check.Source: §A.2 · p. 15
  • RunningTab preserved all five chapters and twenty articles in their original order; DCI merged them into an outline and lost the chapter structure.Source: §A.2 · p. 15

A.3. Case Study 2: Backing Every Row with Its Source

In a task to summarize ten purchase orders by supplier, RunningTab produced a row for every supplier and tied each row to its source order; DCI left out orders it had read.

A.3. Case Study 2: Backing Every Row with Its Source

  • The task asks for a table summarizing ten purchase orders by supplier, with each order expected in its own row.Source: §A.3 · p. 15
  • RunningTab’s table held a row for every supplier, each tied in the tab to its source order.DCI’s table held only one row despite reading the orders, including Supplier 15’s largest order of ¥55,000.Source: §A.3 · p. 15
  • After a finish check noted that no requirement was recorded, the agent stated one requirement per row, naming supplier, status, amount, and source order, then resolved each against its order entry.Source: §A.3 · p. 15
  • Table 7 presents a value DCI read but omitted alongside the same value in the tab, tied to the read-log entry for its purchase order.Source: Table 7 · p. 17

B. Prompts

The prompt figures show how RunningTab is added to the system prompt, exposed as tab operations, and used through environment messages and a finish check.

B. Prompts

  • Figure 6 locates RunningTab’s additions within the harness system prompt.Source: Figure 6 · p. 18
  • Figure 7 presents tab operations as LLM tools, with their parameters and allowed values or descriptions.Source: Figure 7 · p. 19
  • Figure 8 shows the status line, requirements review, and resolve replies; review entries and files are ranked by BM25 against the requirement.Each entry displays its first line sharing a term with the requirement.Source: Figure 8 · p. 19
  • Figure 9 shows the finish check sent at the first stop; a listed file is marked above the relevance threshold when its path shares at least two distinct task terms.Source: Figure 9 · p. 20

Related papers

Loading 2610.10444v1…