Query Focused Multi-document Summarisation of Biomedical Texts

• Query-focused multi-document summarisation combines information from multiple documents to answer a question, with applications in biomedical and clinical evidence retrieval.Source: §1 · pp. 1–2
• The authors frame BioASQ8b Task B Phase B as extractive summarisation by scoring candidate sentences and selecting the highest-scoring ones.Source: §1 · p. 2
• The paper incorporates BERT and BioBERT into the prior architecture, including Siamese variants and PPO-based reinforcement learning.Source: §1 · p. 2
• The paper evaluates these approaches through cross-validation on BioASQ8b training data and submitted BioASQ runs.Source: §1 · p. 2

• The baseline word embeddings are 100-dimensional word2vec vectors trained on PubMed documents, followed by bidirectional LSTM sentence reductors with unshared weights.Source: §2 · p. 2
• The architecture generates word embeddings, reduces them to sentence embeddings, combines sentence position and sentence–question similarity, and predicts each sentence’s label.Source: §2 · p. 2, Figure 1 · p. 3
• The framework uses an element-wise product as the similarity between candidate-sentence and question embeddings.Source: §2 · p. 3
• Regression trains on each candidate’s ROUGE-SU4 F1 value with mean squared error, whereas classification labels the five highest-scoring candidates as 1 and optimises binary cross-entropy.Source: §2 · p. 3

• The untrained BERT variant freezes pretrained BERT and averages its contextual word embeddings to represent each sentence and question.Source: §3 · p. 4
• The trained BERT variant fine-tunes the BERT embedding generator and reductors during training.Source: §3 · p. 4
• The BERT LSTM variant uses BERT-generated contextual word embeddings followed by a bidirectional LSTM reductor, differing from the NNC baseline only in its embedding generator.Source: §3 · p. 4
• BioBERT variants reuse the untrained and LSTM configurations with a biomedical-domain pretrained model, while all experiments use sigmoid classification and binary cross-entropy.Source: §3 · p. 4

4.1 Siamese LSTM

• Siamese networks share the encoding weights used for the candidate sentence and question so both sentence embeddings are generated by the same process.Source: §4 · p. 4, §4.1 · p. 4
• The Siamese LSTM implementation applies shared weights to the candidate-sentence and question embedding reductors within the base architecture.Source: §4.1 · p. 4

4.2 Sentence-BERT

• The SBERT implementation uses a Siamese BERT setup to determine whether a candidate sentence is similar to the question.Source: §4.2 · p. 5, Figure 2 · p. 5
• SBERT variants process embeddings through regression, classification, or jointly optimised multitask prediction heads.Source: §4.2 · p. 5
• Classification concatenates the sentence and question embeddings with their element-wise absolute difference before a softmax layer, while regression optimises cosine-similarity prediction with MSE.Source: §4.2 · p. 5
• The experiments also include BERT and BioBERT Sentence Transformer variants intended to produce biomedical-space sentence embeddings.Source: §4.2 · p. 5, Table 2 · p. 6

5 Experiments with Reinforcement Learning

• PPO is selected because prior work reports that it penalises policy changes, while the authors observe more consistent learning curves than with REINFORCE.Source: §5 · p. 6

5.1 Approach

• PPO trains a policy to classify each candidate sentence as included or excluded, using ROUGE-SU4 F1 directly as the reward.The approach is implemented in the BioASQ summarisation environment using stable baselines.Source: §5.1 · p. 6
• Training uses a 1000-step horizon, four minibatches, and 500,000 timesteps; actions are sampled during training and selected from 100 samples during testing.Source: §5.1 · p. 6
• The PPO setup uses candidate sentence, question, summary-so-far, following-sentences, and document inputs to make inclusion decisions.Source: §5.1 · p. 6

6. Length of summary generated so far

• The network represents the first five inputs by mean word2vec embeddings and appends the current summary length before prediction.Source: §6 · p. 7
• The PPO architecture combines six inputs into a 501-unit layer, passes them through two 200-unit hidden layers, and outputs policy and value functions.Source: Figure 3 · p. 7, §6 · p. 7

5.2 With BERT

• The PPO BERT variant keeps the same architecture but replaces word2vec with BERT embeddings for BioASQ8b batches 4 and 5.The authors report a minor ROUGE-SU4 F1 improvement for PPO BERT.Source: §5.2 · p. 7

6.1 Cross-Validation Results

• Cross-validation uses 10 folds for non-reinforcement-learning experiments, whereas reinforcement learning uses a train/test partition because training takes several days.Source: §6.1 · p. 8, §6.2 · p. 9
• The classification setup outperforms the regression setup; untrained BERT does not improve classification, and trained BERT performs worse.Source: §6.1 · p. 8
• Changing BERT training from 1 to 20 epochs produces no evaluation change, despite a very small loss improvement.The authors suggest gradual unfreezing as a possible route to better results.Source: §6.1 · p. 8
• BERT and BioBERT with an LSTM sentence reductor improve over mean-of-embeddings versions, although these experiments missed the BioASQ8b submission deadline.Source: §6.1 · p. 9
• The Siamese LSTM variant does not improve over classification, while BioBERT variants perform worse than BERT variants.Source: §6.1 · p. 9

6.2 Reinforcement Learning Results

• PPO with BERT features improves over the corresponding comparison, whereas word2vec evaluations show no clear difference between REINFORCE and PPO.Source: §6.2 · p. 10
• Reinforcement-learning results do not outperform the experiments in Table 3.Source: §6.2 · p. 10, Table 5 · p. 10

7 Submissions to BioASQ8b

• Table 6 reports Macquarie University submissions using ROUGE-SU4 F1, selected because F1 correlates better with human judgments than recall and similarly to precision.The table covers the MQ runs submitted to BioASQ8b.Source: §7 · p. 10, Table 6 · p. 11
• The classifier system (NNC) produced the best results in most BioASQ8b runs, consistent with cross-validation, while regression was usually second-best.All systems were retrained with the same hyperparameters on the full training data; the authors suggest the BERT system may not have been trained to its best.Source: §7 · p. 12
• Reinforcement learning experiments showed the lowest evaluation results, although prior BioASQ7b human evaluations ranked those runs higher than other submissions.The BioASQ8b reinforcement-learning systems were not retrained on the entire training set, and PPO showed a more consistent learning curve than REINFORCE.Source: §7 · p. 12
• The Australian National University submissions are reported as SBERT runs in Table 7.These runs provide the submission context for the SBERT comparison reported in the text.Source: §7 · p. 12, Table 7 · p. 12
• SBERT runs performed worse or on par with the untrained BERT model, while BioBERT models also appeared to produce lower results.The authors suggest identical processing may be disadvantageous because questions and candidate sentences have different styles; multitask learning performed better than classification or regression alone.Source: Table 7 · p. 12, §7 · p. 13

8 Conclusions

• The paper presents BERT, BioBERT, Siamese architectures, SBERT, and PPO-based reinforcement learning for BioASQ8b extractive summarisation.These experiments target query-focused multi-document extractive summarisation.Source: §8 · p. 13
• BERT word embeddings followed by an LSTM to obtain sentence embeddings had the most promising results.This was the strongest architecture identified in the reported experiments.Source: §8 · p. 13
• BioBERT variants did not improve results, conflicting with reported BioBERT improvements for biomedical question answering.The conclusion describes this as an observed discrepancy rather than resolving its cause.Source: §8 · p. 13
• Siamese architectures did not improve over the base versions, possibly because questions and candidate sentences have different sentence styles.The paper proposes fine-tuning BERT and BioBERT variants as further research.Source: §8 · p. 13