Source-linked AI summary
Learning the Lake: Reliable Experience for Adaptive Data Product Discovery
Yixi Zhou, Fan Zhang, Sikun Wang, Yingfan Xu, Haipeng Zhang
TL;DR
Repeated workloads make it possible to contract data-product search, but similarity alone cannot justify a route for a conjunctive product. SafeLake separates familiarity-based scope from calibrated product steering; steering gains are task-dependent, and clean confirmed-feedback streams reduce cumulative asset exposure.
Problem
Data-product discovery starts with full-lake search even when workloads revisit related products, while similarity alone cannot establish that a conjunctive product is complete.
Source: §1 · p. 1Method
SafeLake stores source-labelled query–product–region evidence above a fixed regional index and separates familiarity-based scope selection from independently calibrated product routing.
Source: §1 · p. 2Results
Takeaways & Limitations
Product steering should be enabled only after paired calibration shows a map gain, and deployment requires a centering mean estimated from earlier workload data.
Source: §1 · p. 2, §8 · p. 10Evidence covers three question-answering-derived lakes with fixed membership and one pinned retriever; asset exposure approximates work, and audits cannot certify completeness.
Source: §7 · pp. 9–10
Abstract
from arXiv · showhide
Data-product discovery searches a full lake even when workloads revisit related products and regions. Repetition permits contracted search, but similarity cannot justify a route because one omitted asset invalidates a conjunctive product. We study when serving experience can safely reduce this work. Evolving Discovery Memory records source-labelled query--product--region evidence above a fixed regional index. SafeLake separates operational familiarity, which determines how much to search, from independently calibrated product evidence, which determines where to search. The fixed-probe comparison holds the adaptive budget constant between SafeLake and Familiarity-only. On TAT-QA, product steering raises Product Recall by 0.072; ConvFinQA shows no resolved map gain, while the HybridQA sensitivity favors Familiarity-only in Full R@100. Trace-only, missing, and false feedback expose boundaries on map steering, while scope-audit agreement cannot certify the source. Across clean confirmed-feedback streams under the frozen transductive protocol, the formal controller saves 49.5--82.7% of cumulative asset exposure. Experience determines when to contract; reliable evidence determines where to contract.
1 Introduction
SafeLake asks when retained serving experience can reduce repeated full-lake discovery without treating similarity as sufficient evidence for complete products. It separates search-scope decisions from product-based routing.
1 Introduction
- Exact replay is limited to identical requests under the same corpus revision, while semantic similarity cannot establish multi-product completeness.Request-local shard or region selection does not accumulate source-calibrated evidence across served requests.Source: §1 · p. 1
- SafeLake separates operational familiarity, which determines how much to search, from independently calibrated product evidence, which determines where to search.Evolving Discovery Memory stores source-labelled query–product–region evidence above a fixed regional index.Source: §1 · p. 2
- The pipeline freezes source-labelled observations before the next request, and a regional ranking cannot confirm its own products.A scheduled audit or fallback can use the memory-independent global path before serving and updating memory.Source: §1 · p. 2
- On TAT-QA, confirmed evidence raises Full R@100 by 0.057 and Product Recall by 0.072; ConvFinQA resolves no map gain, while HybridQA sensitivity favors Familiarity-only in Full R@100.The fixed-budget comparison retains SafeLake's controller and budget decisions while removing the map prior, isolating the routing contribution.Source: §1 · p. 2
- Under the frozen transductive protocol, clean confirmed-feedback streams save 49.5–82.7% of cumulative asset exposure.The reported measure is asset exposure, an approximation of physical work.Source: §1 · p. 2
- Earlier positions in serve-then-update streams cover 65.9–73.5% of later gold-asset mentions, motivating reuse of serving experience.The benchmark formalizes discovery as conjunctive retrieval of tables and text passages, and its released hybrid retriever starts with a full-lake search.Source: §1 · pp. 1–2
2 Related Work
Related work improves discovery representations, caching, request-local adaptation, and indexes, but leaves cross-request product evidence outside its state. SafeLake studies that longitudinal setting above a fixed physical substrate.
2 Related Work
- Data-discovery systems improve representations or structure for one request, while DPDisc targets heterogeneous conjunctive products over tables and text [26].Examples include Starmie for table-union search, DeepJoin for joins [8], and GRAFT for retrieving and fusing a connected table subgraph [12].Source: §2 · p. 2
- Semantic caches reuse query regions or generated outputs, whereas Evolving Discovery Memory retains evidence about where fresh retrieval should execute.Each non-replay request still runs retrieval over a selected physical scope; cached answers or model states cannot establish all assets for a fresh multi-product request.Source: §2 · p. 2
- SafeLake’s pipeline separates memory learning, budget and scope selection, and guarded execution, including audit or fallback to full-lake results.The global path provides an independent route alongside regional execution.Source: Figure 2 · p. 3
- Query-performance prediction, shard selection, and adaptive retrieval methods make request-local choices; SafeLake tests whether evidence accumulated across requests licenses smaller physical search.Baikal allocates a finite budget across semantic lake regions within one complex request [1].Source: §2 · pp. 2–3
- Index-adaptation systems change physical structures, while SafeLake fixes the corpus, retriever, and regional index to isolate serving experience.The boundary is modular: an adaptive index or dynamic-corpus retriever could replace those components without changing the source-labelled evidence contract.Source: §2 · p. 3
3 Reliable reuse needs completeness
Conjunctive discovery must retrieve a complete multi-asset product, not merely plausible partial results. Serving history can reveal reuse, but safe reuse requires controlling physical search scope while preserving independent checks.
3.1 Conjunctive discovery needs completeness
- Product Recall measures partial gold-product coverage, while Full R@100 measures whether the entire product is available among the top 100 results.Because one omitted asset can leave a product incomplete, partial coverage cannot substitute for complete retrieval.Source: §3.1 · p. 3
3.2 Serving history exposes reuse
- Serve-then-update ordering permits reuse only from earlier requests, and the streams show substantial overlap between earlier requests and later gold-asset mentions.HybridQA retains high train-to-evaluation familiarity, whereas TAT-QA and ConvFinQA use disjoint official asset identifiers across splits.Source: §3.2 · p. 4
3.3 Safe reuse needs physical scope
- Safe reuse must label evidence sources, convert operational familiarity into physical exposure, and retain query-only routing, audits, or global fallback.The controller charges for every searched region and asset, including audit work, rather than treating top-K output size as cost.Source: §3.3 · p. 4
4 SafeLake separates map and index
SafeLake keeps a fixed regional index beneath workload state that records source-labelled serving evidence. Its pipeline uses prior evidence to choose search scope while retaining independent routes and calibration checks.
4 SafeLake separates map and index
- The fixed regional index executes retrieval, while Evolving Discovery Memory stores source-labelled request–product–region associations that can be reset without rebuilding the index.This separates static locality from experience accumulated during serving.Source: §4 · p. 4
4.1 Serve-then-update blocks self-use
- Each request uses a frozen prior snapshot, and the service records new observations only after fixing the returned output.The lifecycle separates request embeddings for familiarity, product observations for region support, and external confirmation for product utility.Source: §4.1 · p. 4
4.2 Learn Where labels evidence
- Learn Where records confirmed, global-path, and weak-anchor evidence, but source labels alone cannot establish that application feedback identifies the correct product.Regional outputs cannot confirm their own products, and current-request retrieval does not read post-hoc confirmations.Source: §4.2 · pp. 4–5, Table 1 · p. 4
4.3 Decide How Much turns evidence into scope
- Decide How Much combines query-based region scores with a map prior, then preserves a two-region query-only anchor before adding map-ranked regions.The formal controller maps operational familiarity to Full, Broad, Mid, or Narrow budgets; lower familiarity can only expand scope.Source: §4.3 · p. 5
4.4 Guard and Expand preserves independence
- Guard and Expand retains query-only routing, scope audits, and global fallback, while trace-only service enforces a 5% minimum cumulative audit rate.Audits measure agreement with the global path but do not certify correctness or detect every false application confirmation.Source: §4.4 · p. 5
4.5 Calibration separates scope from steering
- Calibration holds the familiarity controller constant when map steering is removed, and a positive map weight is admitted only after a paired calibration test.A corpus or retriever revision triggers fresh budget and map-weight checks; until then, the global path and Familiarity-only are conservative operating points.Source: §4.5 · p. 5
4.6 The regional index executes the scope
- The fixed regional index restricts dense and lexical candidate generation to selected regions before the original reciprocal-rank fusion retriever runs.Regional budgets count regions, while cost counts their assets because region sizes vary.Source: §4.3 · p. 5, §4.6 · p. 6
5 Evaluation
The evaluation separates adaptive search scope from product-based routing through fixed-probe controls, then tests how workload reuse and feedback reliability affect quality and cumulative work. Confirmed product evidence improves TAT-QA routing at a matched budget, while other diagnostics bound when map steering and recovery are reliable.
5.1 Experimental setup
- The evaluation pairs fixed probes with a 400-request memory stream, label-blind holdouts, matched controls, and separate boundary diagnostics.The formal multi-level comparison carries the method claim; clean confirmed-feedback streams carry the cumulative-cost claim. A released-backend discrepancy remains unresolved, so comparisons use a stateless result from the same backend.Source: §5 · p. 6, §5.1 · pp. 6–7
5.2 Serving history creates reuse opportunities
- Earlier requests cover 73.45% of later gold-asset mentions on HybridQA, 65.93% on TAT-QA, and 71.79% on ConvFinQA.These are within-stream reuse opportunities; cross-split transfer remains untested.Source: §5.2 · p. 7
5.3 Reliable evidence improves TAT-QA routing
- 0.072 higher Product Recall and 0.057 higher Full R@100 distinguish SafeLake from Familiarity-only on TAT-QA fixed probes.The methods share every budget decision; both paired intervals exclude zero, isolating the map’s routing contribution. ConvFinQA has no resolved quality difference, while the HybridQA sensitivity favors Familiarity-only in Full R@100.Source: §5.3 · p. 7, Table 3 · p. 8
- 65 of 72 one-level diagnostic cells pass all three gates, but seven TAT-QA or ConvFinQA cells fail Product Recall.Static-Calibrated comparisons show SafeLake raises TAT-QA Full R@100 by 0.053 while searching 35.2 percentage points fewer assets; ConvFinQA quality intervals include zero while search falls by 27.6 points. These diagnostics do not replace the primary fixed-probe comparison.Source: §5.3 · pp. 7–8
5.4 Feedback reliability bounds map authority
- Five feedback settings preserve quality tolerances at search ratios of 0.133–0.306, but TAT-QA trace-only fails both quality gates despite budget contraction.Confirmed TAT-QA improves Full R@100 by 0.043 with Product Recall drop 0.009; retaining only 75% of true confirmations passes the Product Recall gate, whereas 50% does not. At 20% false confirmations, one schedule fails.Source: §5.4 · p. 8
- Neither audit-agreement gate meets its promotion rule, and shuffling HybridQA associations reduces confirmed gold-region prior mass from 0.372 to 0.016.The cumulative gate can reject clean feedback, while the rolling gate keeps the map active under 20% corruption; the study retains source calibration as admission control.Source: §5.4 · p. 8
5.5 Action headroom bounds recovery
- In the one-level workload-shift test, observable recovery appears only where the action has room to expand.On HybridQA, all three seeds recover after 10–13 requests; TAT-QA and ConvFinQA begin with actions exposing 62–63% of assets and show inconsistent contraction.Source: §5.5 · p. 8
5.6 Formal control amortizes by request six
- After 400 requests, the formal controller saves 82.7% on HybridQA, 61.6% on TAT-QA, and 49.5–50.5% on ConvFinQA cumulative asset exposure.Cumulative exposure falls below repeated stateless search after four requests on TAT-QA and ConvFinQA and six on HybridQA; break-even assumes an existing regional index.Source: §5.6 · p. 8, Table 6 · p. 9
6 Discussion
The discussion separates static locality, adaptive scope, and product evidence, while emphasizing that safe steering depends on evidence quality and available action headroom.
6 Discussion
- Evidence quality sets dataset-specific operating boundaries, and route agreement cannot establish feedback truth.When feedback quality is unavailable, the authors recommend retaining λ = 0 and pairing weaker sources with broader budget floors or more frequent audits.Source: §6 · p. 9
- Only confirmed TAT-QA resolves a benefit from product evidence; ConvFinQA and the HybridQA sensitivity do not.Familiarity-only retains adaptive scope and accumulated workload history when paired calibration finds no map benefit.Source: §6 · p. 9
- Recovery depends on action headroom: the HybridQA increase misses the pre-specified amplitude gate and offers no general recovery guarantee.TAT-QA and ConvFinQA start from broader regional actions, suppressing measurable expansion.Source: §6 · p. 9
7 Limitations
The evaluation is bounded by fixed benchmark lakes and a pinned retriever, while deployment assumptions and measurement limits constrain what its results establish.
7 Limitations
- Evidence covers three question-answering-derived DPDisc lakes with fixed membership and one pinned retriever; corpus updates, other stacks, and cross-lake transfer require recalibration.Centered familiarity uses a label-blind split-level query-embedding mean, which deployment must estimate from an earlier workload.Source: §7 · p. 9
- Asset exposure approximates work, and the latency study excludes end-to-end serving.The paired intervals condition on frozen streams, probes, and budgets.Source: §7 · pp. 9–10
- Feedback controls omit delayed, correlated, and population-dependent errors; global-route audits cannot certify completeness.The controller guarantees monotonic scope selection, not recall preservation or shift detection.Source: §7 · pp. 9–10
8 Conclusion
Under the frozen transductive protocol, the paper separates familiarity-based scope from calibrated product steering and identifies conditions for deployment.
8 Conclusion
- Under the frozen transductive protocol, fixed probes resolve steering value only on confirmed TAT-QA.Deployment requires paired calibration and a centering mean estimated from earlier workload data.Source: §8 · p. 10
- Repeated discovery reuses serving experience by separating familiarity-based scope from calibrated product steering.Experience determines when to contract; reliable evidence determines where to contract.Source: §8 · p. 10
- Deployment requires paired calibration and a centering mean estimated from earlier workload data.Source: §8 · p. 10
Ethical Considerations
The ethical discussion addresses omission risks, safeguards, and privacy concerns associated with workload memory and data-product retrieval.
Ethical Considerations
- Omissions can affect financial or enterprise decisions, and quality gates, audits, replay, and fallback reduce risk without guaranteeing completeness.Operators should set harm-based targets and audit rates, retain fallbacks, and require human review.Source: Ethical Considerations · p. 11
- Workload memory can disclose sensitive queries and proprietary links.Recommended safeguards include minimizing retained fields, separating logs from embeddings, tenant-aware access and deletion, and recording provenance.Source: Ethical Considerations · p. 11
- The evaluation uses public benchmarks without human subjects, and retrieval evaluation is not financial advice.A large language model was used only to polish the paper's writing.Source: Ethical Considerations · p. 11