Source-linked AI summary
HintMiner: Automatic Question Hints Mining From Q&A Web Posts with Language Model via Self-Supervised Learning
Zhenyu Zhang, JiuDong Yang
TL;DR
Q&A forums may not provide timely, satisfactory answers, and finding useful material across their posts can take time. HintMiner retrieves relevant posts and uses MiningNet to generate hints from them. The authors report that it outperforms several Q&A methods, with average BLEU of 36.17% and average ROUGE-2 of 36.29%.
Problem
Q&A forums cannot guarantee timely, satisfactory answers, while searching their posts takes users time.
Source: §1 · p. 1Method
HintMiner retrieves relevant Q&A posts and uses MiningNet to select and combine useful text spans into generated hints.
Source: §1 · p. 1, §3.2.1 · p. 4Results
Takeaways & Limitations
The authors make the HintMiner tool and experimental data publicly available.
Source: §6 · p. 8The authors leave evaluation of practical usefulness to future user studies and plan further work on code and numerical-expression comprehension.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Users often need ask questions and seek answers online. The Question - Answering (QA) forums such as Stack Overflow cannot always respond to the questions timely and properly. In this paper, we propose HintMiner, a novel automatic question hints mining tool for users to help them find answers. HintMiner leverages the machine comprehension and sequence generation techniques to automatically generate hints for users' questions. It firstly retrieve many web Q\&A posts and then extract some hints from the posts using MiningNet that is built via a language model. Using the huge amount of online Q\&A posts, we design a self-supervised objective to train the MiningNet that is a neural encoder-decoder model based on the transformer and copying mechanisms. We have evaluated HintMiner on 60,000 Stack Overflow questions. The experiment results show that the proposed approach is effective. For example, HintMiner achieves an average BLEU score of 36.17\% and an average ROUGE-2 score of 36.29\%. Our tool and experimental data are publicly available.
1 Introduction
Questions on Q&A forums may lack timely, satisfactory answers, while searching those resources can take users considerable time. HintMiner retrieves relevant posts and combines useful text spans into generated hints, with experiments reporting gains over baseline methods.
1 Introduction
- Q&A forums can lack timely, satisfactory answers, and searching their accumulated resources takes users time.The paper notes that many Stack Exchange questions lack accepted answers.Source: §1 · p. 1
- HintMiner retrieves relevant Q&A posts, selects useful text spans, and merges them into hints for users’ questions.MiningNet is trained with a self-supervised objective on question, relevant-post, and hint/answer triplets.Source: §1 · pp. 1–2
- 36.17% average BLEU and 36.29% average ROUGE-2; HintMiner outperforms several information-retrieval methods.The authors also report that MiningNet outperforms retrieval and generation language-model baselines.Source: §1 · p. 2
- The authors provide HintMiner’s code and data publicly.The paper also describes extracting paragraphs from online Q&A forums for a useful-posts dataset.Source: §1 · p. 2
2 The Q&A Forums and Dataset
The paper draws on Stack Exchange posts and community-marked links to construct training examples, then retrieves and filters posts to form inference context. The dataset uses link distance to distinguish relevant from less relevant posts.
2.1 Question Answering Web Resources
- Stack Overflow posts include question details, tags, answers, community-linked posts, and system-retrieved related posts.The paper describes Stack Exchange as a resource where questions may take a long time to receive responses or may never be answered.Source: §2.1 · p. 2
2.2 The Construction of the Q&A Dataset
- About 17 million posts from four Stack Exchange sites support a post-link graph whose shorter link distances indicate more useful post content for a question.The graph represents posts as nodes and weighted links as edges; distances are grouped as 0, 1, 2, and ≥3.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
2.2.1 Selecting Relevant Posts For Training
- Training uses about 3.6 million question-passage-hints triplets, with accepted-answer passages as hints and some less relevant paragraphs added as noise.Questions without a neighbor at distance 1 are removed; paragraphs are shuffled, and the triplets draw top 2, 1, 1, and 1 paragraphs at distances 0, 1, 2, and ≥3.Source: §2.2.1 · p. 3
2.2.2 Selecting Relevant Posts For Inference
- For inference, HintMiner retrieves five posts, replaces code and mathematical expressions with placeholders, and filters nonessential sentences using three algorithms.The filtering methods are LexRank, KL greedy search, and latent semantic analysis; their selected sentences are merged and duplicates removed to form MiningNet’s context passage.Source: §2.2.2 · p. 3
3 HintMiner: Generating Hints to User’s Questions
HintMiner takes a question and relevant Q&A posts, then uses MiningNet to select and generate hint text. MiningNet combines contextual encoding, attention-guided copying, and self-supervised training from Q&A posts.
3.1 System Overview
- HintMiner forms a context from relevant posts and feeds it with the question to MiningNet, which generates hints by copying and generating tokens.The task is framed as selecting useful text spans from existing posts and combining them into hints.Source: §3.1 · pp. 3–4
3.2.1 The Structure of the Model
- MiningNet encodes question and context semantics, then uses a transformer decoder and CopyNet to produce answer text from selected input spans.The decoder attends to the question and context; CopyNet selectively copies text spans, generating tokens iteratively until an end token or length limit.Source: §3.2.1 · p. 4
3.2.2 Transformer Decoder and CopyNet
- Encoder-decoder attention supplies CopyNet with token-level signals for selecting context spans that can clarify and explain the question.The decoder computes attention over the question and context, and CopyNet uses the attention vectors to copy selected input tokens.Source: §3.2.2 · pp. 4–5
3.3 Self-supervised Learning Objective
- MiningNet is trained to identify useful hints in noisy context using over 1,900,000 question-context-hints triplets constructed from Q&A posts.For each post, the three top-rated answers are shuffled and concatenated as context; the best answer serves as the hint target.Source: §3.3 · p. 5
3.4 The Implementation and Training Details
- Training uses BERT-base with 12 encoder-decoder attention heads, followed by teacher forcing and policy-gradient fine-tuning with BLEU4 reward.The model is trained with Adam for 200,000 steps at an initial learning rate of 1e-5 and batch size 32.Source: §3.4 · p. 5
4 Experiments
The experiments compare HintMiner with retrieval and generation baselines, test different sources of relevant posts, and inspect generated hints for solved and unsolved questions. Results favor HintMiner, while the examples expose limits in code and numerical content.
RQ1: How effective is HintMiner in generating hints?
- HintMiner is compared with retrieval and generation methods, with baseline methods given the same data processing for fair comparison.Retrieval baselines include AnswerBot, SimCSE, and PageRank++; generation baselines include GPT2, BART, and UniLM.Source: RQ1: How effective is HintMiner in… · pp. 5–6
4.3 Experimental Results for RQ1
- 36.17% BLEU and 36.29% ROUGE-2 at 100 words; MiningNet significantly beats generation baselines, while span-level HintMiner significantly beats retrieval baselines.The experiments report HintMiner outperforming retrieval baselines across answer lengths from 50 to 200 words.Source: Table 1 · p. 6, §4.3 · p. 7
4.4 Experimental Result for RQ2
- Linked posts yield the best retrieval performance, but only around 19% of posts have links; Google, Stack Exchange, and Elastic Search retrieval cause only small score drops.The authors describe performance with these alternative sources as acceptable.Source: Table 2 · p. 6, RQ2: How effective is HintMiner when… · p. 6, §4.4 · p. 7
4.1 Experimental Design
- The evaluation samples 60,000 posts with accepted answers and uses the first accepted-answer paragraph longer than 10 words as the gold hint.Hint quality is measured with 4-gram BLEU and 2-gram ROUGE-2.Source: RQ2: How effective is HintMiner when… · p. 6
4.2 Evaluation Metrics
- Generated hints are scored against gold hints using 4-gram BLEU and 2-gram ROUGE-2.Both metrics compare generated hints with the gold hints.Source: RQ2: How effective is HintMiner when… · p. 6
4.3 Experimental Results for RQ1
- Manual checks suggest question-similar sentences may merely repeat the question, while sentence-level selection and general-domain pretraining are poorly suited to extracting useful hints.The authors attribute weaker baseline performance on noisy related posts to these limitations.Source: §4.3 · p. 7
4.5 Examples of the Generated Hints
- Generated hints are generally readable and useful, including for some unsolved questions, but may contain grammatical errors and omit code or mathematical expressions.Code-bearing content can appear as a [CODE] marker.Source: §4.5 · pp. 7–8
5 Related Work
Related work covers Q&A approaches and the limited use of forum resources, then explains copying mechanisms and machine reading comprehension. These concepts provide context for HintMiner’s selection and generation approach.
5 Related Work
- Prior Q&A research has focused mainly on general open-domain systems, with limited support for using Q&A forum resources.The cited review found 28.57% of surveyed papers used information retrieval and 34.9% used knowledge bases.Source: §5 · p. 8
5.1 Copying Mechanism
- Copying mechanisms select input tokens for output; CopyNet uses encoder-decoder attention to form an output distribution over source tokens.The mechanism is described as inspired by pointer networks and used in sequence-to-sequence models.Source: §5.1 · p. 8
5.2 Machine Reading Comprehension
- Machine reading comprehension answers a question by selecting a passage span, represented by the start and end token indices.The cited examples include DrQA, multi-granularity attention fusion, and R-Net [Chen et al., 2017, Wang et al., 2018b, Wang et al., 2017].Source: §5.2 · p. 8
6 Conclusion
HintMiner retrieves relevant posts and uses MiningNet to generate question hints, with experiments reporting advantages over several Q&A methods. The authors identify code and numerical comprehension, retrieval, metrics, and user studies as future work.
6 Conclusion
- HintMiner combines relevant-post retrieval with MiningNet, a self-supervised model that selects suitable content to generate hints, and outperforms several Q&A methods in evaluation.The tool and experimental data are publicly available.Source: §6 · p. 8
- Future work includes better relevant-post discovery, code and numerical-expression comprehension, improved evaluation metrics, and user studies of practical usefulness.The paper specifically names [CODE] and [NUM] representations as areas for further model development.Source: §6 · p. 8