PaperlayerView exact revision on arXiv

Source-linked AI summary

Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox

Federico Lucca

arXiv:2610.32955v1cs.CLcs.HCcs.LG

TL;DR

Personal agents’ stored labels are their own outputs, leaving their decision quality unmeasured in this setting. The paper measures a production inbox agent against classical and language-model alternatives using human labels and thread-grouped evaluation. Its reported conclusion favors a trained logistic-regression baseline for accuracy, while showing that action costs can change which system is cheapest.

  • Problem

    Personal agents’ stored labels are their own outputs, so their decision quality cannot be established by counting those labels.

    Source: §1 · p. 1

  • Method

    The paper measures a production email agent using human-reviewed labels, frozen inputs, thread-grouped evaluation, and comparisons among rules, a trained baseline, and zero-shot models.

    Source: §1 · pp. 1–2, §4 · p. 3

  • Results

    Two hundred labels and logistic regression give the highest accuracy on every field, while the baseline is cheapest only when the cost of a missed email is below 3.

    Source: §6 · pp. 4–5

  • Takeaways & Limitations

    The paper concludes that accuracy alone is not the product metric: choosing by explicit action costs can favor Gemma 4 over the accuracy-leading baseline.

    Source: §6 · p. 5, §9 · p. 6

    The evaluation is a single-user, single-inbox study, and 231 labels yield accuracy intervals of roughly ±6 points, making system differences below eight points nonsignificant.

    Source: §8 · p. 6

Abstract

from arXiv · show

Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop) that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner (231 unbiased, 500 adversarially sampled), we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B). Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship (client, supplier, service) is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action (18% vs 4% for the best LLM); a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.

1 Introduction

The paper asks whether typed decision models can replace generative steps in personal agents, a question existing agent-written labels and prior evaluations do not answer for personal multilingual correspondence. It introduces a measurement protocol, compares nine systems, and separates message-level decisions from sender relationships.

1 Introduction

  • Prior evidence does not establish whether typed decision models can replace generative steps for personal, multilingual correspondence, where stored labels are the system’s own output.Earlier comparisons concern network traffic, triage benchmarks, or vendor benchmarks rather than this setting.Source: §1 · p. 1
  • The protocol yields human ground truth for a production email agent through stratified sampling, frozen inputs, deterministic splits, and labels collected in the agent’s interface.The system is measured without changing it.Source: §1 · p. 1
  • The study compares nine systems across message type, urgency, agent switches, and project assignment, and distinguishes message-text decisions from sender relationships measured through contact-graph inheritance.It includes calibration, latency, and cost.Source: §1 · pp. 1–2

2 Related Work

Prior work introduced typed decision interfaces, compared decision models with classifiers and general-purpose LLMs, and developed per-user email classifiers. This paper tests zero-shot decisions against a user-trained classifier for personal correspondence.

2 Related Work

  • Typed decision models take a state and typed questions—choice, yes/no, or score—and return a distribution per question; evaluations compare models on identical examples under the same procedure.A broader benchmark compares Jev with nineteen general-purpose LLMs, and an evaluation checklist was proposed [5] [6].Source: §2 · p. 2
  • On network-flow classification, Random Forest and Extra Trees trained on 8,000 records beat Jev in every test week [7].A community test also found a plain Qwen3.5-9B ahead of 14 decision models on 56 triage tasks.Source: §2 · p. 2
  • Earlier email-classification work covered filing, folder prediction, reply prediction, and attachment prediction, while per-user classifiers used a few hundred labelled messages.The paper’s baseline uses a similar label scale.Source: §2 · p. 2
  • The paper tests whether zero-shot generative or typed decisions can outperform a classifier trained on one user’s labels, in a privacy-motivated local setting [14].Prior work also notes that users need a handful of broad categories rather than many folders.Source: §2 · p. 2

3 The System

Foyer’s production pipeline combines rules, grouping, and two LLM passes, with distinct stages for type and urgency versus project assignment. The study evaluates the system as it actually ran, not as its documentation described it.

3 The System

  • Foyer ingests Gmail and IMAP, then applies noise and self-mail rules, groups messages by sender, and assigns type and urgency using keyword heuristics.The heuristics cannot produce cliente, spam, or newsletter; some messages later inherit spam or newsletter from a project.Source: §3 · p. 2
  • The first LLM pass processes message groups and existing projects to return project details, contacts, tasks, and timeline information; a second pass merges groups.Project assignment, not per-email type or urgency, is the LLM’s role in this pipeline.Source: §3 · p. 2
  • The study measures the hosted-gateway production configuration: code inspection showed the local model was only a fallback and type and urgency were never model-produced.A local 3B model had been demoted after timing out on CPU.Source: §3 · p. 2

4 Measurement Protocol

The measurement protocol samples and labels real messages while preserving the exact model inputs and preventing thread leakage across evaluation splits. It records production accuracy before training and uses pre-registered hypotheses.

4 Measurement Protocol

  • Stratification by predicted type addresses skewed stored labels, including 39% altro and 94% bassa urgency, so sampling does not simply teach the prior.The dataset command samples across seven predicted type classes.Source: §4 · p. 2
  • Frozen snapshots preserve the sender, subject, truncated body, date, channel, and prediction; secrets are redacted before storage.Human labels cover type, area, project, urgency, needs-reply, action-within-48h, and is-noise.Source: §4 · p. 3
  • The protocol reports production correctness before training and pre-registers each labelling round’s hypotheses, thresholds, and verification script.For the third round, eleven hypotheses were committed before the labelled file was opened.Source: §4 · p. 3
  • Thread-grouped cross-validation is primary because earlier ID-keyed splits put 11 threads across splits and gave 4 test emails near-duplicates in training.Splits are stratified by type, and the baseline is trained out-of-fold.Source: §4 · p. 3

5 Experimental Setup

The setup separates message labels from sender relationships and evaluates multiple system types using owner-reviewed labels, thread-grouped data, and several performance and deployment metrics.

Data

  • 231 emails in 207 threads form the unbiased v2 reference, while the 722-email v3 corpus is an adversarial stress test used for hypotheses, contact-graph results, and action analysis.The v3 sample includes 491 kept emails from 500 adversarially selected emails; system comparisons are reported on v2.Source: §5 · p. 3, Table 1 · p. 3
  • The ground truth is based on a single human reviewer who labelled with model proposals, then jointly revised labels with an assistant.The first-round interface hid production predictions; the owner confirmed or corrected model-proposed type and urgency labels.Source: §5 · p. 3

Labels

  • Labels separate six message types from eight sender relationships assigned per contact, three urgency levels, three agent switches, and project assignment.The switches are needs-reply, action-48h, and is-noise; 26 project labels are empty because the project-manager agent had not proposed one.Source: §5 · p. 3

Systems

  • The comparison includes production rules, a TF-IDF/logistic-regression baseline, local and hosted typed decision models, and hosted generative LLMs.The logistic-regression baseline is the only trained system; hosted Gemma 4 26B results are single-run estimates despite temperature 0.Source: §5 · p. 3

Metrics

  • Evaluation reports accuracy, macro-F1, confidence on correct versus incorrect decisions, median latency, and provider cost per 1,000 emails.Because offerta and spam have only 5 and 3 examples, respectively, type macro-F1 is unstable, so accuracy and F1 on the four large classes are emphasized.Source: §5 · p. 3

6 Results

The trained TF-IDF + logistic-regression baseline leads on accuracy across fields, though it misses the rarest message types and is statistically tied with Gemma 4 on type. Results also show that action costs, contact history, and project descriptions change how system performance should be interpreted.

Classification performance

  • The production heuristic reaches 0.28 type accuracy against a 0.57 majority class, and 0.68 on urgency against 0.70.The heuristic’s type field was below the majority class on a sample stratified by its own predictions.Source: §6 · pp. 3–4
  • B reaches 0.79 on type, 0.74 on urgency, 0.83 on needs-reply, and 0.90 on is-noise; its type accuracy is statistically tied with Gemma 4 26B and 31B.B never predicts the two rarest classes, so its type macro-F1 is 0.47 versus Gemma 4’s 0.54.Source: §6 · p. 4, Table 2 · p. 4
  • Among zero-shot systems, Gemma 4 26B leads on type at 0.74, while Nimble leads on urgency and is-noise at 0.71 and 0.86.Few-shot prompting does not close the gap: Qwen gains one type-accuracy point, not significant, and Gemma 4 26B is unchanged.Source: §6 · p. 4

Agent switches

  • For action-48h, B reaches 0.73 while zero-shot models do not beat the 0.91 majority-class accuracy.There were 20 positive examples among 231 emails.Source: §6 · p. 5

Contact relationship

  • Where a contact has another labelled email, relationship inheritance is correct 98.6% of the time; a majority fallback gives 0.857 overall.Text-only inference by Tev1 4B and Nimble 9B reaches 0.52 and 0.55, respectively, against a 0.50 majority class.Source: Table 5 · p. 4, §6 · p. 5

Project assignment

  • With only project names, zero-shot models score 0.26–0.47, while the baseline reaches 0.70 on project assignment.Zero-shot models over-use “none”; a fair comparison requires owner-written project descriptions.Source: §6 · p. 5, Table 7 · p. 5

Actions and costs

  • On v2, the baseline ignores 12 of 68 emails needing action (18%), versus 3 (4%) for Gemma 4 26B; cost-chosen thresholding reduces the baseline to 7 of 68.With Platt scaling, false noise falls to 4 of 68; at missed-email cost 10, the baseline’s total cost is 79 versus Nimble’s 99 on v2.Source: §6 · p. 5, Table 6 · p. 5

Calibration

  • Nimble v4-max is best calibrated across fields, with ECE 0.04–0.11 on v2 and 0.04–0.09 on v3.Recalibration cuts the baseline’s type ECE from 0.47 to 0.11 on v2, at a five-point type-accuracy cost on 231 rows.Source: §6 · p. 5

7 Error Analysis

The error analysis groups shared system mistakes into six causes, including relational labels absent from message text, label ambiguity, and a taxonomy that mixes message type with sender relationship.

7 Error Analysis

  • Relational classes are not inferable from one message: all five initial cliente emails were misread by at least three systems, and no zero-shot system reaches F1 0.3 on altro.The sender’s relationship belongs in the contact graph.Source: §7 · p. 6
  • At 8–12% of the corpus, debatable labels bound achievable accuracy: four or more systems disagree with the reviewer on 9 type and 5 urgency labels.These cases are candidates for a second annotator rather than evidence of model error.Source: §7 · p. 6
  • The stored taxonomy mixes message kind with sender relationship, which the analysis identifies as the root of relational-class errors.Automated senders were not misclassified in the reviewed labels; the upstream robot filter removes the hard cases.Source: §7 · p. 6
  • Sampling 118 presumed offerta or spam cases yielded only 3 and 7 true cases, respectively, while models over-predicted commercial mail.The inbox, already filtered upstream by Gmail, contains little commercial mail.Source: §7 · p. 6
  • Every system failed on bodies under 20 characters in the first round; those thin-text cases are now excluded as ignore.Source: §7 · p. 6

8 Limitations

The evaluation centers on one owner’s two-language inbox, and its labels and sampled comparisons carry limits that constrain interpretation and transfer to other users.

8 Limitations

  • The study covers one user, one inbox, and two languages; the second inbox is identified as the next study needed to test transfer.Baseline weights, contact graph, class priors, and confidence thresholds are user-specific.Source: §8 · p. 6
  • The owner reviewed model-proposed labels, and final definitions were jointly revised with an assistant, without an independent second annotator.This process may reflect anchoring as much as agreement.Source: §8 · p. 6
  • With 231 labels, accuracy intervals are roughly ±6 points, so system differences below eight points are not significant.The production-heuristic sample is stratified on its predictions and does not estimate production accuracy.Source: §8 · p. 6
  • Round 3 was sampled on presumed LLM errors and shown with the baseline proposal first, so its comparisons inherit both biases.The data cannot be released, though the protocol, harness, and aggregate statistics can.Source: §8 · p. 6

9 Conclusion

The conclusion favors measuring before choosing an inbox system: a small labeled set changes the model ranking, while action costs can reverse it again. Foyer’s planned changes assign type and urgency away from rules and inherit relationships from contacts.

9 Conclusion

  • On 231 emails, logistic regression beat every zero-shot model on each agent decision, while the most accurate system most often ignored emails needing action.A cost-chosen noise threshold made the classifier cheapest again.Source: §9 · p. 6
  • The production changes will remove rule-based type and urgency, inherit relationship from contacts, and let the three switches drive actions directly.The change was in progress at the time of writing.Source: §9 · pp. 6–7
  • The reproducibility materials include code, commands, environment variables, result hashes, run logs, result JSONs, and a single-cell T4 notebook.The listed local models ran on Ollama 0.40, with scikit-learn 1.9.Source: §9 · p. 7

A Fixed-split results

The fixed thread-grouped split is retained in the labeled data and result files, but its sparse rare classes make it unsuitable for figures in this paper.

A Fixed-split results

  • The fixed split contains 159 training, 39 validation, and 33 test emails, and is recorded in the labeled JSONL and result files.Source: A Fixed-split results · p. 7
  • The split is not used for figures because its test portion contains no alta and only one offerta.Source: A Fixed-split results · p. 7
  • The split is retained so future labeling rounds can report a held-out number once the small classes have enough examples.Source: A Fixed-split results · p. 7

B Questions, definitions and prompts (verbatim)

The appendix specifies shared label definitions and the procedures used to prompt decision models, prompt generative models, and classify messages with production heuristics.

B Questions, definitions and prompts (verbatim)

B.1 Type

  • Message type has seven labels: richiesta, aggiornamento, offerta, cliente, newsletter, spam, and altro.The definitions distinguish requests or needed actions from updates, targeted commercial offers, client correspondence, mass mailings, unsolicited commercial mail or phishing, and other messages.Source: §B.1 · p. 7

B.2 Urgency

  • Urgency is alta for action within about 48 hours or an appointment today, media for action or careful reading without a near deadline, and bassa when no action is required.Source: §B.1 · p. 7

B.3 Decision-model

  • Typed decision models received one choice question per call, with the sender, subject, and body as state; email text was cleaned and truncated to 1,500 characters.Cleaning removed style/script blocks, tags, and URLs, then collapsed whitespace.Source: §B.3 · p. 7

B.4 Generative prompt (OpenRouter)

  • Generative models received a cleaned email as JSON and were instructed to return exactly one defined label as JSON.The prompt used temperature 0, a 60-token limit, and JSON response format; answers outside the vocabulary counted as errors.Source: §B.3 · p. 8

B.5 Production heuristic

  • Production type rules matched keywords in the lower-cased subject and snippet, assigning offerta, richiesta, aggiornamento, or otherwise altro.The heuristic assigned urgency alta or media through keyword matches, and bassa otherwise; its docstring described it as a simple heuristic to replace with Ollama when available.Source: §B.3 · p. 8

C Labelling procedure and guidelines

The owner reviewed email rows in a spreadsheet, prioritizing disagreements and low-confidence proposals, and applied the supplied type and urgency guidelines with a narrower client criterion.

C Labelling procedure and guidelines

  • Rows were ordered with disagreements and low-confidence proposals first; production predictions were hidden, and four ambiguous skipped rows were excluded.A row counted as reviewed when confirmed or corrected; optional fields were filled on every row, but project labels were not retained in the imported label table.Source: C Labelling procedure and guidelines · p. 8
  • The reviewer applied cliente more narrowly than the written guideline, counting an actual paying client rather than a prospective one.Source: C Labelling procedure and guidelines · p. 8
  • The guidelines define richiesta as a request or needed action, aggiornamento as information without a request, and offerta as a commercial proposal aimed at the recipient.They also distinguish subscribed mass mailings, unsolicited commercial mail, and other messages; urgency spans action within about 48 hours, action without a near deadline, and no action.Source: C Labelling procedure and guidelines · p. 8
  • The reviewer saw sender, subject, and the first 650 body characters, without production predictions; empty or markup-only bodies were judged from sender and subject.Source: C Labelling procedure and guidelines · p. 8
Loading 2610.32955v1…