PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Elicitation via Sequential Probabilistic Inference for High-Dimensional Prediction

Pedram Daee, Tomi Peltola, Marta Soare, Samuel Kaski

arXiv:1612.03328v2cs.AIcs.HCcs.LGstat.ML

TL;DR

Small-sample, high-dimensional prediction is challenging, and domain experts’ knowledge is not efficiently incorporated. This paper formulates elicitation as sequential probabilistic inference and selects informative features to query in sparse linear regression. Experiments with simulated and real users report improved prediction accuracy with a small number of interactions.

  • Problem

    In small-n, large-p prediction, expert knowledge is a valuable but insufficiently exploited source of information, and incorporating tacit knowledge can burden experts.

    Source: §1 · pp. 1–2

  • Method

    The method sequentially models and incorporates expert feedback, selecting informative features to query for sparse linear regression.

    Source: §1 · p. 2

  • Results

    Simulated and real-user evaluations show improved prediction accuracy with a small number of user interactions.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Sequential elicitation can support richer interaction and knowledge elicitation for high-dimensional parameters without overwhelming the user.

    Source: §6 · p. 12

    The reported results use simple feedback types as a proof of concept; extending the types of interaction remains future work.

    Source: §6 · p. 12

Abstract

from arXiv · show

Prediction in a small-sized sample with a large number of covariates, the "small n, large p" problem, is challenging. This setting is encountered in multiple applications, such as precision medicine, where obtaining additional samples can be extremely costly or even impossible, and extensive research effort has recently been dedicated to finding principled solutions for accurate prediction. However, a valuable source of additional information, domain experts, has not yet been efficiently exploited. We formulate knowledge elicitation generally as a probabilistic inference process, where expert knowledge is sequentially queried to improve predictions. In the specific case of sparse linear regression, where we assume the expert has knowledge about the values of the regression coefficients or about the relevance of the features, we propose an algorithm and computational approximation for fast and efficient interaction, which sequentially identifies the most informative features on which to query expert knowledge. Evaluations of our method in experiments with simulated and real users show improved prediction accuracy already with a small effort from the expert.

1 INTRODUCTION

Small-sample, high-dimensional prediction is difficult, while expert knowledge is often tacit and burdensome to incorporate. The paper proposes sequential probabilistic elicitation and studies an efficient approach for sparse linear regression.

1 INTRODUCTION

  • Small-sample, high-dimensional prediction is ill-posed, and statistical methods have limits on how low the sample size can be [1].Prior approaches include assumptions about model structure, transfer from related data, active data collection, and prior elicitation.Source: §1 · p. 1
  • Expert knowledge is often unstructured and tacit, while eliciting it in a suitable format can burden both domain experts and statistical experts.As a result, experts often apply knowledge post hoc, and collaboration feedback loops can be slow.Source: §1 · pp. 1–2
  • The paper formulates knowledge elicitation as probabilistic inference, sequentially incorporating expert feedback before selecting the next query.The approach targets prediction with insufficient training data and prioritizes informative queries to reduce the number of interactions.Source: §1 · p. 2

Contributions and Outline

  • For sparse linear regression, the paper develops sequential elicitation for coefficient values or feature relevance, efficient computation, and evaluations with simulated and real users.The paper also identifies user-interface design as future work.Source: Contributions and Outline · p. 2

2 RELATED WORK

The paper relates its approach to interactive learning, active learning and experimental design, prior elicitation, and privileged information, while distinguishing the source and integration of expert input.

Interactive Learning

  • Unlike interactive-learning work aimed at identifying user preferences, this method uses expert knowledge with small-n data to improve prediction.It also relates probabilistically to expert-feedback approaches for Bayesian networks and visual data exploration.Source: §2 · pp. 2–3

Active Learning

  • Like active learning and experimental design, the method selects informative queries, but it obtains added information from experts rather than new data samples.The paper says it is most useful when obtaining additional samples would be too expensive.Source: §2 · p. 3

Prior Elicitation and Privileged Information

  • Unlike pure prior elicitation or privileged-information approaches, this method interactively integrates expert knowledge with training data and models feedback reliability.Prior elicitation uses expert knowledge to construct a prior; privileged-information approaches typically do not interactively integrate feedback or model its reliability.Source: §2 · p. 3

3 KNOWLEDGE ELICITATION AS INTERACTIVE PROBABILISTIC MODELLING

The paper frames elicitation as probabilistic inference, with user feedback modeled alongside observations and acquired iteratively. Its general interaction scheme is illustrated through prior Bayesian-network and visual-exploration approaches.

3.1 Key Components

  • Knowledge elicitation is formulated as probabilistic inference in which user feedback supplies information about model parameters through a feedback model.A query algorithm and model-update process support iterative interaction.Source: §3 · p. 3, §3.1 · p. 3
  • Feedback models can represent uncertainty or user interests, while query algorithms guide attention toward useful feedback and can use expected information gain.Because feedback is treated as additional data, Bayes’ theorem updates the model sequentially; efficient computation matters for real-time interaction.Source: §3.1 · pp. 3–4

3.2 Overall interaction scheme

  • The interaction scheme first computes a posterior from observed data, then iteratively queries the user and updates that posterior with feedback.The query algorithm uses current parameter beliefs and a posterior predictive distribution of user feedback to formulate queries.Source: §3.2 · p. 4

3.3 Examples

  • Cano et al. integrate expert knowledge into Bayesian-network learning through edge-presence queries, while House et al. support interactive visual exploration by interpreting movements of plotted points as feedback.Cano selects the next edge query by maximizing information gain and assumes answers are correct with some probability; House et al. have no query algorithm and afford closed-form updates.Source: §3.3 · pp. 4–5

4 FEEDBACK MODELS AND QUERY ALGORITHM FOR SPARSE LINEAR REGRESSION

For sparse linear regression, the paper models coefficient values and feature inclusion, elicits expert feedback, and selects queries by expected information gain. Deterministic posterior approximations make the computation faster.

4.1 Sparse Regression Model

  • The sparse regression model uses Gaussian observations, a spike-and-slab prior on coefficients, and binary inclusion indicators for covariates.The prior inclusion probability controls expected sparsity, while model hyperparameters are assumed fixed.Source: §4.1 · p. 5

4.2 Feedback Models

  • The feedback models represent expert knowledge of either a coefficient’s value or its relevance, with ω² and π controlling feedback uncertainty or strength.ω² is uncertainty in a coefficient estimate; π is the probability that relevance feedback correctly reflects the inclusion indicator.Source: §4.2 · p. 5

4.3 Query Algorithm

  • The query algorithm selects the feature with the greatest expected information gain, measured by the KL divergence between predictive distributions with and without its feedback.It averages over the posterior predictive distribution of possible feedback.Source: §4.3 · p. 6

4.4 Computation

  • Because the model lacks closed-form posteriors and an information-gain solution, the method uses deterministic approximations for fast computation.Expectation propagation approximates the spike-and-slab prior and feedback models, variational Bayes approximates residual variance, and one update iteration per candidate feature avoids full convergence.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments evaluate sequential expert-feedback queries in small-sample, high-dimensional regression, using simulated data, review-rating tasks, and a proof-of-concept study with real users. Across these settings, sequential querying improves prediction more quickly than random query selection, while comparisons also examine feedback against acquiring additional samples.

Results.

  • With n = 10 and m* = 10, sequential queries improve prediction faster than random queries from the first feedbacks across dimensions and both feedback types.As dimensionality grows, random-query performance degrades; coefficient-value feedback performs better than relevance feedback for both strategies.Source: Figure 2 · p. 7, Results · p. 7

5.1 Simulated Data

  • The simulated study generates sparse linear-regression data and compares one-feature-at-a-time query algorithms using mean squared error.It tests feedback about coefficient values and feature relevance.Source: §5.1 · p. 7

Results.

  • At fixed dimension m = 100, sequential querying also outperforms random selection as training sample size varies, especially for small samples, and performs better than its non-sequential version.The comparison indicates that earlier user feedback affects the next query.Source: Results · p. 8

5.2 Review Rating Prediction

  • For review-rating prediction, Amazon and Yelp reviews are represented by word features, with infrequent features removed before analysis.The Amazon subset contains kitchen-appliance reviews; the Yelp subset uses reviews from 2004.Source: §5.2 · p. 8, §5.2.1 · p. 8

5.3.1 Results

  • On Amazon and Yelp, simulated expert feedback reduces prediction error for all query algorithms, while sequential selection finds informative words faster than random selection.On Yelp, the method approaches the oracle strategy early and gets close to ground truth after 200 interactions; on Amazon, it exceeds the information from the 26 relevant words after 140 feedbacks.Source: §5.3.1 · pp. 9–10

5.3.2 Expert Knowledge Elicitation vs. Collecting More Samples

  • For Yelp, MSE = 1.18 is reached with 25 relevance feedbacks or 12 actively selected additional samples; random sampling requires 94 additional samples.The comparison is specific to the Yelp data and assumed feedback model; the authors report similar observations for Amazon.Source: Table 2 · p. 10, §5.3.2 · pp. 10–11

5.4 User Study

  • The user study collected relevance judgments on 824 Amazon words from ten participants, then tested query algorithms using 100 training reviews and 40 random runs.The simulated user knowledge strength was set to π = 0.7.Source: §5.4 · p. 11
  • With real feedback from ten participants, sequential querying outperforms random ordering from the first feedbacks, with a significant paired-test difference after the first.The tests use a Bonferroni-corrected threshold of α = 0.05/200.Source: Figure 4 · p. 11, §5.4 · p. 11

6 CONCLUSION

The conclusion reports improved prediction with few expert interactions and outlines the model and deterministic posterior-approximation computation used for sequential elicitation.

6 CONCLUSION

  • Simulated and real data with simulated and real users showed improved prediction accuracy with few interactions, for feedback on coefficient values or feature relevance.The approach sequentially acquires and integrates expert knowledge with training data; the authors also note richer interaction without overwhelming users compared with pure prior elicitation.Source: §6 · p. 12
  • The authors observed that weak feature-relevance feedback and randomly acquired samples required numbers of additions of the same order to reach a given MSE reduction.They qualify this observation as coming from a noisy dataset and a simplifying interaction setting.Source: §6 · p. 12
  • The method is presented as general, but the reported results use simple feedback types; extending interaction types is left for future work.The probabilistic formulation is described as allowing analysis and tailoring to other elicitation settings.Source: §6 · p. 12

A.1 Model

  • Relevance and coefficient-value feedback are represented by separate feature-index sets, with the model's listed hyperparameters treated as fixed.Fγ and Fw index features receiving relevance and weight feedback, respectively.Source: §A.1 · p. 15

A.2 Posterior approximation

  • The posterior approximation combines exponential-family site terms, with unobserved feedback terms assigned zero parameters.The approximation's covariance includes contributions from the observation and weight-feedback terms.Source: §A.2 · p. 16

A.3 Computation of the posterior approximation

  • The computation uses variational Bayes for the observation-model terms and expectation propagation for the other terms, with damped updates iterated to convergence.The updates have closed-form solutions.Source: §A.3 · pp. 16–17

B Bayesian Experimental Design

The design selects feedback by expected information gain, approximating the posterior after candidate feedback with partial EP updates rather than running EP to full convergence for every candidate.

B Bayesian Experimental Design

  • The query objective is to choose feedback that maximizes expected information gain over the unqueried features.The stated criterion averages over possible feedback and the training data.Source: B Bayesian Experimental Design · p. 17
  • The score compares approximate predictive distributions, with Gaussian predictive distributions for the output and feedbacks.The feedback predictive distributions are approximated as Gaussian or Bernoulli, depending on feedback type.Source: B Bayesian Experimental Design · p. 17
  • Partial EP updates approximate the posterior after a candidate feedback, reducing the cost of evaluating many candidates.The procedure treats weight and relevance feedback separately.Source: B Bayesian Experimental Design · pp. 17–18
  • For weight feedback, the method adds its Gaussian site term and uses a matrix-inversion lemma to evaluate the updated covariance efficiently.The update leaves other site-term approximations unchanged.Source: B Bayesian Experimental Design · p. 18
  • For relevance feedback, a scalar EP update adds the feedback site and corresponding prior term, avoiding costly matrix operations.The method also uses the matrix-inversion lemma when computing the updated posterior covariance.Source: B Bayesian Experimental Design · p. 18

C.1 Synthetic data

The synthetic experiments extend the study of algorithm behavior while retaining the Sect. 5.1 setting except for specified changes.

C.1 Synthetic data

  • The synthetic experiments retain the Sect. 5.1 setting except for the specifications introduced in this section.Source: §C.1 · p. 18

C.1.1 Heatmaps with varying number of training data

The section varies the number of training observations in a fixed 100-dimensional synthetic setting and compares the method with random feature selection.

C.1.1 Heatmaps with varying number of training data

  • The experiment varies training data from 1 to 50 with dimensionality fixed at 100 and 10 relevant features.Source: §C.1.1 · p. 19
  • For n < 10, a performance difference becomes visible after 20–30 feedbacks; for 10 < n < 30, the method's MSE reduction is more visible from the first feedbacks, while for n > 30 both strategies have much smaller MSE.Figure 5 reports mean squared errors averaged over 100 runs, with 10 relevant coefficients among 100 dimensions.Source: Figure 5 · p. 19, §C.1.1 · p. 19

C.1.2 Sequential vs Non-sequential Experimental Design

In simulated data, experimental-design query methods reduce prediction loss more quickly when user interactions are limited. Sequential selection improves on a preselected query sequence, especially when feedback concerns coefficient values.

C.1.2 Sequential vs Non-sequential Experimental Design

  • The comparison uses simulated data with n = 10, m = 100, and m* = 10, averaging MSE over 500 runs.The non-sequential method fixes its feature-query sequence before receiving expert feedback.Source: §C.1.2 · p. 19
  • Both experimental-design methods reduce prediction loss more in early iterations than the other methods when user interactions are limited.They prioritize the most informative coefficients; for relevance feedback, their performance is close to suggesting relevant features first.Source: §C.1.2 · p. 19
  • Sequential experimental design reduces prediction error faster than non-sequential selection, especially with coefficient-value feedback.Figure 6 compares MSE across query algorithms for simulated data and coefficient-value or relevance feedback.Source: Figure 6 · p. 20, §C.1.2 · p. 20

C.2 Comparison of Training and Test Set Errors and the Average Accumulated Suggestion Behaviour

With coefficient-value feedback, training error can rise as feedback shifts the model away from its training-data optimum toward better generalization. Training and test MSE converge as feedback accumulates, faster for query algorithms that begin with nonzero-effect features.

C.2 Comparison of Training and Test Set Errors and the Average Accumulated Suggestion Behaviour

  • Training error rises with expert feedback as the model moves from its training-data optimum toward better generalization.Without feedback, the model has already fit the training data well, subject to its regularizing priors.Source: §C.2 · p. 20
  • Training and test MSE curves converge as feedback increases.Convergence is faster for query algorithms that first suggest features with nonzero effects, which are more informative.Source: §C.2 · p. 20

C.4 User Study

In the Amazon user study, the proposed method captured most participants’ knowledge with a limited query budget, and initially focused queries on a small set of relevant words.

C.4 User Study

  • The proposed method captured most participants’ knowledge after about 200 queries out of the 824-feature subset.Convergence speed was measured by normalizing each iteration’s MSE improvement by that user’s total improvement; Figure 8 reports MSE for 10 Amazon-data participants.Source: §C.4 · pp. 20–21, Figure 8 · p. 22
  • Table 3 reports the number of samples or feedbacks needed to reach particular MSE levels, averaged over 100 independent runs.The supplied caption does not state the table’s numerical results.Source: Table 3 · p. 21
  • The algorithm initially asked mostly about relevant words, with 39 of 824 words classified as relevant using E[γj] > 0.7.Relevance was determined by training a spike-and-slab model on all Amazon data; Figure 9 names the relevant words and shows their coefficients.Source: §C.4 · p. 21, Figure 9 · p. 22
Loading 1612.03328v2…