Source-linked AI summary
Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesis
Aryan Luthra, Kshitij Jain, Siddharth Arya, Bobby Filar, Anna Bertiger
TL;DR
Operational security lacks reliable labeled benchmarks for judging whether continual learning harnesses improve an LLM or which harness is better. The paper evaluates harnesses by measuring a smaller student's convergence toward a stronger teacher, finding teacher-relative improvement correlates with gold-standard improvement across tasks, models, and harnesses.
Problem
Operational security lacks reliable labeled benchmarks for determining whether a continual learning harness helps or which harness performs better.
Method
The framework uses scaling-based student-teacher pairs and sparse teacher corrections to evaluate harnesses through the student's convergence toward the stronger teacher.
Results
Teacher-relative improvement strongly correlates with gold-standard improvement across security tasks, model families, and harnesses, while similarly powered judges provide no usable signal.
Takeaways & Limitations
The framework lets security operators assess whether a harness helps and compare harnesses before production deployment without a labeled benchmark.
Takeaways & Limitations
Proxy correlation drops significantly when the student-teacher capability gap is small or both models perform extremely poorly.
Abstract
from arXiv · showhide
Agentic "Continual Learning Harnesses", systems that pair an LLM with retrieval or memory to improve from feedback without retraining, have shown growing value in cybersecurity. But their value is conventionally measured by gains against labeled benchmarks, an approach that often fails in operational security settings. Benchmark labels are scarce, stale, and unrepresentative, so a practitioner often cannot tell whether a given harness helps at all or which of two is better for their task. Traditional LLM-as-a-judge offers little signal because it is no stronger than the agent it evaluates, and distillation is unreliable on scarce, sporadic, and biased labels. We propose a framework for evaluating learning harnesses end-to-end without a labeled benchmark, grounded in the scaling hypothesis. A stronger teacher model provides sparsely sampled corrections to a smaller student with a continual learning harness. We score a harness by how much its student converges toward the teacher over time. Across security tasks, model families, and harness designs, we show that improvement relative to the teacher correlates with improvement relative to a held-out gold standard, validating teacher-relative lift as a proxy for true harness uplift when labels are absent. We further show that LLM-as-a-judge between similarly powered models yields no usable signal. These results suggest that a teacher-sized model can be improved through the same harness when humans provide the same kind of sparse, high-precision corrections.
1. Introduction
Continual learning harnesses adapt LLMs through feedback, but operational security lacks reliable labeled benchmarks and stronger judges for measuring their value. The paper proposes teacher-relative lift from sparse corrections and shows it tracks gold-standard improvement, unlike same-strength LLM judging.
- Motivation: Continual learning harnesses wrap LLMs and absorb corrections over time to improve performance or personalization, making rapid adaptation appealing for shifting security threats.The paper defines a continual learning harness as an LLM system that takes in corrections over time.
- Problem: Operational security evaluation is difficult because systems may already use the strongest available model, leaving no stronger LLM judge and making same-strength judging ineffective.The paper states that traditional optimization and measurement tools fail in these settings, including LLM-as-a-judge.
- Method: The proposed label-free framework uses larger and smaller same-family models as teacher and student, leveraging the scaling hypothesis to identify a teacher that outperforms the student.Both models label a large pool of unlabeled samples, enabling teacher-relative evaluation without directly identifying performance from human labels.
- Method: Sparse teacher corrections train the harness-equipped student*, whose held-out pseudo-label performance is compared with the student to estimate harness improvement.An uptick under this purely pseudo-labeled regime is treated as evidence that a teacher-capability model would improve similarly.
- Results: Across security tasks, model families, and harnesses, teacher-relative performance improvements correlate with oracle-label improvements, while same-strength judges cannot recognize gains over baseline.These findings support teacher-relative lift as a proxy for true harness uplift when labels are absent.
2. Related Work
Prior work shows that cybersecurity evaluation is constrained by scarce, stale labels and by the dynamic, interactive behavior of agentic harnesses. Existing approaches use evolving benchmarks, proxy reference signals, and retrieval- or memory-based adaptation, but security evaluation must also account for feedback persistence and abstention under uncertainty.
- Evaluation constraints: Cybersecurity evaluations often face proprietary data, scarce labels, and labels that become stale rapidly, making learning-harness assessment especially difficult.These constraints are particularly acute in AI SOC scenarios, where labels and data evolve quickly while testing learning remains important.
- Benchmark development: Cybersecurity benchmarks increasingly generate realistic or emerging samples, including phishing messages and continuously refreshed threat-intelligence-based vulnerabilities.Examples include Phishfuzzer and AthenaBench, an extension of CTIBench using sources such as MITRE ATT&CK and the NVD API.
- Agent evaluation: Agentic harnesses are harder to evaluate than base models because dynamic, interactive environments cannot be represented by a single held-out test set.Prior work decomposes the challenge into perspectives including core agentic capabilities and application-specific benchmarks.
- Proxy evaluation: Methods such as expert reasoning chains and LLM judges reduce dependence on annotated data but assume their reference signals are trustworthy substitutes for gold labels.This assumption is made explicit in the proposed framework rather than left implicit.
- Retrieval and memory: Retrieval-augmented and memory-based systems adapt behavior by retrieving external context or reusable lessons, while security applications additionally require retrieval quality, feedback persistence, and abstention under uncertainty.Unlike conventional top-k retrieval, security classification may lack a relevant lesson, making indiscriminate retrieval harmful and requiring a decision about whether to act on a lesson.
3. Experiment Design
The experiments proxy sparse, high-precision human corrections by training a weaker student with corrections from a stronger teacher, then testing whether teacher-relative harness uplift tracks gold-label uplift. The design spans multiple security tasks, model families, harnesses, supervision conditions, and cross-family teacher–student pairs.
- Experimental proxy: The study replaces unavailable human corrections with sparse, high-precision corrections from a stronger teacher model delivered over successive rounds.The procedure uses up to five sampled high-impact misclassifications per round, with no more than 20 corrections across four rounds.
- Evaluation procedure: Teacher-relative uplift is tested as a proxy by comparing harness uplift against teacher pseudo-labels with uplift against gold labels on the same held-out items.Gold labels are withheld until final evaluation, so the relationship between ∆T and ∆G tests whether pseudo-labels rank harnesses like gold labels.
- Tasks: The design evaluates three security tasks spanning classification, root-cause attribution, and fine-grained ATT&CK tactic labeling across a difficulty and label-granularity gradient.The tasks are PhishFuzzer, CTIBench, and ATT&CK tactic classification, selected because they provide gold labels for validating the proxy where ASA does not.
- Robustness checks: Cross-family teacher–student pairs test whether the proxy depends on shared architecture or instead reflects the capability gap between stronger teachers and weaker students.Pairs are selected only when the teacher outperforms the student on the base task, reducing concern that correlated in-family errors explain the result.
- Experimental factors: Each task compares a no-learning baseline with Memory, Lesson, and Few-shot harnesses across Gemini, GPT, and Qwen model families under proper and sabotaged supervision.Sabotaged supervision injects incorrect labels as a negative control, testing whether faithful harnesses lose accuracy under corrupted feedback.
4. Experiment Results
Experiments validate teacher-relative alignment as a label-free proxy for harness improvement: it tracks gold-label uplift within and across model families when the teacher is stronger. In contrast, similarly powered LLM judges are unreliable because of positional and self-preference biases.
- Teacher alignment: Teacher-relative learning strongly correlates with additional performance on the gold-labeled dataset, and uplift signs usually agree across teacher and gold labels.This indicates stronger models can provide proxy labels for identifying whether a harness improves task performance.
- Scaling hypothesis: Larger teacher models outperform smaller students across all three tasks within every model family, supporting the scaling hypothesis.The larger teacher also exceeds students from other families on these tasks.
- Teacher alignment: Cross-family teacher-student pairs preserve the ∆T −∆G correlation when the teacher outperforms the student, with strength comparable to within-family pairs.Comparable cross-family and within-family correlations suggest the capabilities gap, rather than shared architecture or training data, governs proxy validity.
- LLM-as-a-judge: Same-strength judges are unreliable: two of three judges reverse verdicts when answer positions swap, revealing strong positional bias.The comparisons used student copies as judges, evaluated harness-versus-baseline labels twice with A/B positions switched.
- LLM-as-a-judge: Stable same-strength judges favor the un-harnessed baseline a majority of the time, contradicting harness gains observed through teacher alignment and ground truth.This anti-harness preference reflects model preference rather than reliable assessment of output quality.
- LLM-as-a-judge: 48% reliability against gold is achieved by same-strength judges, while student-judges reach 25%, showing below-chance self-preference over stronger teachers.The same-strength result is no better than a coin flip, and student judges systematically prefer their own answers.
5. Conclusions
The conclusions argue that teacher-relative proxy validity depends on the student–teacher capability gap and correction precision, supporting label-free harness evaluation in operational security. They also identify failure modes and call for validation against gold-labeled operational data.
- Conclusions: Proxy validity depends on the student–teacher capability gap and the precision of supplied corrections, regardless of whether corrections come from a model or human.The harness interacts only with the high-precision corrections, not with the teacher itself.
- Conclusions: The Scaling Hypothesis supports selecting a stronger student–teacher pair because substantially larger same-family models are likely stronger on the task.Human review of escalated security cases is presented as an extreme version of this relationship.
- Conclusions: High proxy uplift (∆T) for a harness–task pair across multiple model families is proposed as evidence that the harness adds value beyond the piloting model.The passage presents transfer to the human-frontier regime as a conjecture for operational security settings.
- Conclusions: Proxy correlation drops significantly when the capability gap is small or both student and teacher are extremely bad, as observed for Qwen on RCM.These dependencies are difficult to measure without labels, so the authors propose mitigating them with dependency proxies.
- Conclusions: The framework should next be tested against a real gold-labeled operational security dataset, while its apparent similarity to operational distillation motivates further exploration.The authors suggest that even a small dataset might provide evidence for the human-frontier conjecture.
- Conclusions: Security operators can use the framework to determine whether a harness helps and compare two harnesses before production commitment, with a basis for human corrections to frontier models.The passage characterizes this evaluation as more rigorous than committing directly to production.