Source-linked AI summary
A deep learning model for estimating story points
Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, Trang Pham, Aditya Ghose, Tim Menzies
TL;DR
Agile story-point estimation has limited issue-level research despite the importance of estimating individual user stories and the inaccuracy of many expert estimates. The paper introduces a 23,313-issue dataset and an end-to-end LD-RNN combining LSTM with recurrent highway networks. LD-RNN consistently outperforms three common baselines and two alternatives on Mean Absolute Error and Standardized Accuracy, though the dataset is not representative of all software projects, especially commercial ones.
Problem
Research has focused substantially on traditional project-level effort estimation, while agile issue- or user-story-level estimation remains limited and expert estimates can be inaccurate or inconsistent.
Method
The paper builds a 23,313-issue dataset and proposes an end-to-end Long-Deep Recurrent Neural Network combining LSTM and recurrent highway network architectures to predict story points from raw issue text.
Results
LD-RNN consistently outperforms Mean, Median, and Random baselines and two alternatives in Mean Absolute Error and Standardized Accuracy.
Takeaways & Limitations
The dataset supports issue-level story-point research, while the model is intended to complement team estimation practices as a decision-support system.
Takeaways & Limitations
The dataset may not represent all software projects, especially commercial settings, because contributor and stakeholder characteristics can differ from open source projects.
Abstract
from arXiv · showhide
Although there has been substantial research in software analytics for effort estimation in traditional software projects, little work has been done for estimation in agile projects, especially estimating user stories or issues. Story points are the most common unit of measure used for estimating the effort involved in implementing a user story or resolving an issue. In this paper, we offer for the \emph{first} time a comprehensive dataset for story points-based estimation that contains 23,313 issues from 16 open source projects. We also propose a prediction model for estimating story points based on a novel combination of two powerful deep learning architectures: long short-term memory and recurrent highway network. Our prediction system is \emph{end-to-end} trainable from raw input data to prediction outcomes without any manual feature engineering. An empirical evaluation demonstrates that our approach consistently outperforms three common effort estimation baselines and two alternatives in both Mean Absolute Error and the Standardized Accuracy.
I. INTRODUCTION
Agile development shifts effort estimation toward individual user stories, where subjective story-point estimates can be inaccurate and inconsistent. The paper introduces an issue-level dataset and an end-to-end LD-RNN system to support team estimation.
- Motivation: Agile projects deliver functionality incrementally through iterations containing multiple user stories.This creates a need to estimate individual stories rather than only entire projects.
- Motivation: Around half of agile teams report estimates inaccurate by 25% or more, while expert-based practices can also produce inconsistent estimates.Teams commonly use planning poker, analogy, and expert judgment to assign story points.
- Contributions: 23,313 issues from 16 open source projects form the paper’s first dataset focused on issue- or user-story-level story-point estimation.Traditional effort-estimation datasets generally operate at the project level.
- Contributions: LD-RNN combines LSTM and recurrent highway network architectures to model issue text and predict story points.LSTM captures long-term textual context, while the recurrent highway network provides a deep representation.
- Contributions: The end-to-end system learns representations directly from raw words without manual feature engineering and consistently outperforms three baselines and two alternatives on MAE and Standardized Accuracy.The reported claims were additionally tested with Wilcoxon and Vargha–Delaney analyses.
- Practical significance: Story points support velocity-based planning, so consistent team estimates help preserve predictability for project and release management.Velocity sums resolved story-point estimates and is used to forecast completion time.
III. STORY POINT DATASETS
The paper constructs and publicly releases an issue-level story-point dataset because existing public effort-estimation datasets target complete projects. It gathers filtered, story-point-labeled issues from diverse open source repositories.
- Dataset motivation: Existing public effort-estimation datasets are designed for project-level estimation rather than estimating individual issues.The authors therefore needed to build an issue-level dataset.
- Availability: The dataset was made publicly available to support result verification and further research.
- Dataset construction: The dataset contains 23,313 story-point-labeled issues from 16 projects across nine major open source repositories.The repositories include Apache, Appcelerator, DuraSpace, Atlassian, Moodle, Lsstcorp, MuleSoft, Spring, and Talendforge.
- Dataset construction: Issues were collected through JIRA’s REST API, retaining reports with assigned story points and extracting their story point, title, and description.Collection covered issues available up to August 8, 2016.
- Dataset construction: The final dataset excludes zero, negative, and unrealistically large story-point values, filtering out about 2.66% of collected issues.Projects were included only when they had more than 300 issues with story points.
IV. APPROACH
The proposed LD-RNN takes an issue’s title and description as raw text and predicts its story-point estimate through a sequential deep-learning pipeline. The complete system is trainable end to end.
- Input: The prediction system uses only an issue’s title and description as input to estimate its story points.The design assumes additional issue metadata may not be available when an issue is created.
- Architecture: LD-RNN sequentially applies word embedding, LSTM document representation, recurrent highway deep representation, and differentiable regression.
- Document representation: The LSTM processes the embedded word sequence, after which its outputs are pooled into a document-level vector.
- Prediction: The recurrent highway network repeatedly transforms the document vector before a regressor produces the story-point estimate.The implementation uses simple linear regression as the final regressor.
- Training: Training propagates raw word signals forward to the output and prediction errors backward to the word layer.The automatically learned text features remove the need for manual feature engineering.
A. Document representation
The document-representation component converts issue words into embeddings and uses LSTM to accumulate sequence information into a fixed document vector. Recurrent highway transformations then deepen this representation while preserving trainability through parameter sharing and gated information flow.
- Word embedding: Words are represented as low-dimensional continuous vectors using an embedding matrix initialized from issue-report corpora.
- LSTM representation: LSTM is used instead of a standard RNN because issue reports can contain long sequences that are difficult to train.Its memory updates by partially forgetting old information and accepting new input.
- LSTM representation: Pooling the LSTM output states produces a length-invariant vector representing the whole document.The implementation uses averaging as the pooling method.
- Recurrent highway representation: The recurrent highway network applies multiple nonlinear transformations to make the LSTM representation deeper before regression.Highway-style transformations address difficulties associated with training deep feedforward networks.
- Recurrent highway representation: The highway gate allows information to pass linearly between layers, with α_l →1 enabling simple copying.This design is reported to support networks with up to a thousand layers.
- Recurrent highway representation: Parameter sharing makes the recurrent highway representation compact and can help avoid overfitting.The same parameter set is reused across repeated transformations.
V. MODEL TRAINING
The LD-RNN model is trained as a differentiable parameterized function optimized by stochastic gradient descent, with dropout and pre-training used to support learning and reduce overfitting.
- LD-RNN maps raw issue-report words to story-point predictions through a parameterized function with learnable parameters θ.
- Stochastic gradient descent updates θ opposite the gradient of the loss function L(θ), using RMSprop as an adaptive method.
- Dropout randomly zeros input and output state elements during training, while parameter averaging is used during testing to prevent overfitting.
- Pre-training provides a favorable parameter initialization and can help learning converge faster to good solutions.
B. Pre-training
The model pre-trains its word-level layers as a language model, then evaluates story-point estimation against baseline and component alternatives across several research questions.
- B. Pre-training: Pre-training uses unlabeled text and natural-language predictiveness to initialize the embedding and LSTM layers without ground-truth story points.
- B. Pre-training: The word-level model predicts the next word from the preceding sequence state using a softmax function.
- B. Pre-training: The language model represents sentence probability through the chain rule and is trained by optimizing negative log probability.
- B. Pre-training: Noise-Contrastive Estimation reduces evaluation cost from vocabulary-size time to M ≪ |V|, with M potentially as small as 100.
- Evaluation questions: The evaluation compares LD-RNN with Random Guessing, Mean Effort, and Median Effort, and tests alternatives involving Random Forests and Bag-of-Words representations.
- Evaluation questions: Cross-project estimation tests whether the approach can transfer from a source project to a new target project using issue titles and descriptions.
A. Experimental setting
The experiments simulate deployment by using temporally ordered issue splits and evaluate estimation accuracy with MAE, SA, significance testing, and effect-size analysis.
- Each project’s issues are split chronologically into 60% training, 20% validation, and 20% test data.
- Mean Absolute Error and Standardized Accuracy are used because prior measures can bias toward underestimation or be unstable for model comparisons.
- For issue i, MAE averages the absolute difference between its actual and estimated story points across the N test issues.
- Standardized Accuracy compares a model’s MAE against the MAE of many random guesses, with higher SA or lower MAE indicating better predictive performance.
- The Wilcoxon Signed Rank Test assesses statistical significance without assuming an underlying data distribution.
- Vargha and Delaney’s Â12 statistic measures effect size, equaling 0.5 for equivalent models and exceeding 0.5 when model M performs better than N.
C. Hyper-parameter settings for training a LD-RNN model
Training hyper-parameters are tuned by varying embedding size and recurrent-highway hidden-layer count on validation data, alongside fixed optimization settings.
- The tuning procedure varies word-embedding dimensions and recurrent-highway hidden-layer counts while monitoring validation-set MAE.
- Embedding sizes of 10, 50, 100, and 200 and twelve hidden-layer settings from 2 to 200 are tested.
- The embedding size of 50 and hidden-layer count of 10 produce the lowest MAE in the Apache Mesos experiments and are selected.
- Pre-training uses 100 epochs with batch size 50, while the main LD-RNN model uses 1,000 epochs with batch size 100.
- Dropout rates are set to 0.5 for RHW layers and 0.2 for LSTM layers.
D. Pre-training
Pre-training learns semantic word representations that support later story-point estimation, while LD-RNN improves over baseline estimators and alternative architectures across the evaluated projects.
- Pre-training: 50,000 unlabeled issues per repository were used for pre-training, with Apache word embeddings clustered into nine semantically related groups.Examples include networking-related words such as soap, configuration, tcp, and load.
- RQ1: Sanity check: LD-RNN achieves 2.09 MAE and 52.66 SA across projects, compared with 2.84 MAE and 36.36 SA for the best baselines.MAE is lower-is-better, while SA is higher-is-better.
- RQ1: Sanity check: LD-RNN significantly outperforms Mean Effort, Median Effort, and Random Guessing in 45/48 comparisons, with effect sizes greater than 0.5.The Wilcoxon test and Vargha–Delaney A12 statistic assess significance and effect size.
- RQ2: Benefits of deep representation: Recurrent Highway Networks significantly improve over LSTM+RF in 13/16 projects, with the three nonsignificant projects having very few issues.The comparison uses p < 0.05 and effect sizes greater than 0.5 as the reported thresholds.
- RQ2: Benefits of deep representation: LD-RNN consistently outperforms LSTM+RF across all sixteen projects, averaging 2.09 versus 2.61 MAE and 52.66 versus 44.05 SA.These averages compare the full LD-RNN model with the LSTM document representation followed by Random Forest regression.
- RQ3: Benefits of LSTM document representation: LSTM-based representations significantly improve over BoW in 13/16 cases, with effect sizes greater than 0.5 and better MAE and SA in those cases.The comparison holds Random Forest fixed as the regressor in both settings.
F. Verifiability
The paper provides a replication package containing the dataset, model source code, benchmark models, and instructions for independently verifying the reported experiments.
- The replication package includes the full dataset and source code for LD-RNN and the benchmark models.The benchmarks include the baselines, LSTM+RF, and BoW+RF.
- Detailed instructions explain how to run the code and replicate all reported experiments.
G. Threats to validity
The study addresses validity threats through diverse real-world data and statistical testing, but its validation and generalizability remain constrained. Cross-fold validation was omitted, team composition is assumed static, and the dataset may not represent commercial projects.
- Construct validity was supported with real-world issue data and assigned story points, while conclusion validity was addressed through selected error measures and statistical tests.
- Cross-fold validation was omitted because training the deep neural network required substantial time.The authors leave cross-fold validation for future work.
- The approach assumes that the team remains static over time, although team changes may alter available skills and story-point estimation.Modeling team dynamics is identified as future work.
- 23,313 issues from 16 open source projects were used to mitigate threats to external validity.The projects differ in size, complexity, developer teams, and communities.
- The dataset may not represent all software projects, especially commercial settings, where contributor and stakeholder differences may affect story-point estimation.The authors call for further investigation using commercial agile projects.
VII. RELATED WORK
Prior effort-estimation research emphasizes expert-based and model-based approaches and often estimates whole projects. This paper focuses specifically on agile issue-level story points, introducing an end-to-end deep learning system and identifying broader evaluation needs.
- Existing effort-estimation methods are commonly classified as expert-based, model-based, or hybrid approaches.Expert-based methods rely on human expertise and remain the most popular technique in practice.
- Model-based approaches use past-project data but may rely on fixed models suited only to the kinds of projects used to build them.
- Most prior work estimates whole projects, whereas this paper targets agile issues using story points.The authors identify issue-level story-point estimation as the key difference from previous agile effort-prediction work.
- The paper contributes a first story-point dataset from 16 large and diverse projects and a fully end-to-end system combining LSTM with RHN.The system avoids manually hand-crafted features.
- Future work includes evaluating commercial projects, modeling team changes, testing incremental learning and metadata, comparing language models, and assessing practical use.