PaperlayerView exact revision on arXiv

Source-linked AI summary

Software development in startup companies: A systematic mapping study

Nicolò Paternoster, Carmine Giardino, Michael Unterkalmsteiner, Tony Gorschek, Pekka Abrahamsson

arXiv:2307.13104v1cs.SE

TL;DR

Software development in startups is underrepresented in structured software-engineering research despite the distinctive uncertainty and resource constraints of these companies. This study maps the literature, evaluates study rigor and relevance, and analyzes reported work practices, finding limited high-quality evidence and context-adapted practice selection.

  • Problem

    Research on software development activities in newly created companies is scarce, and no structured software-engineering review had investigated startups.

  • Method

    The study conducts a systematic mapping study using a classification schema, rigor and relevance assessment, and analysis of reported startup software-development practices.

  • Results

    43 primary studies were mapped; 19 (44%) focused on managerial and organizational factors, while only 16 (37%) were entirely dedicated to startup software development.

  • Takeaways & Limitations

    Software engineering work practices in startups are chosen opportunistically, adapted, and configured to provide value under startup constraints.

  • Takeaways & Limitations

    20 papers could not be retrieved, although their titles, keywords, and abstracts suggested fewer than 1 potentially relevant primary study given the 4% precision rate.

Abstract

from arXiv · show

Context: Software startups are newly created companies with no operating history and fast in producing cutting-edge technologies. These companies develop software under highly uncertain conditions, tackling fast-growing markets under severe lack of resources. Therefore, software startups present an unique combination of characteristics which pose several challenges to software development activities. Objective: This study aims to structure and analyze the literature on software development in startup companies, determining thereby the potential for technology transfer and identifying software development work practices reported by practitioners and researchers. Method: We conducted a systematic mapping study, developing a classification schema, ranking the selected primary studies according their rigor and relevance, and analyzing reported software development work practices in startups. Results: A total of 43 primary studies were identified and mapped, synthesizing the available evidence on software development in startups. Only 16 studies are entirely dedicated to software development in startups, of which 10 result in a weak contribution (advice and implications (6); lesson learned (3); tool (1)). Nineteen studies focus on managerial and organizational factors. Moreover, only 9 studies exhibit high scientific rigor and relevance. From the reviewed primary studies, 213 software engineering work practices were extracted, categorized and analyzed. Conclusion: This mapping study provides the first systematic exploration of the state-of-art on software startup research. The existing body of knowledge is limited to a few high quality studies. Furthermore, the results indicate that software engineering work practices are chosen opportunistically, adapted and configured to provide value under the constrains imposed by the startup context.

1. Introduction

Software startup research addresses software development in newly created companies facing uncertainty, time pressure, competition, and limited resources. This mapping study organizes the literature, assesses its transfer potential, and extracts reported engineering practices.

  • Research on software development activities in newly created companies is scarce, with no prior structured investigation focused on software startups.
  • Startups face intense market time pressure, tough competition, chaotic and uncertain conditions, and limited resources.
  • Software development in startups differs from established-company contexts because prescriptive methodologies are difficult to follow.
  • The study uses a systematic mapping study to characterize startup research, understand its context, assess technology-transfer potential, and analyze software development practices.
  • 43 primary studies were identified from an initial set of 1053 papers, and 213 software engineering work practices were identified for practitioner use.

2. Background and Related Work

Prior work characterizes startups through limited history and resources, multiple influences, and dynamic technologies and markets, while emphasizing that their engineering solutions must fit this context. The background also frames startup development as market-driven and difficult to structure with standard processes.

  • Startup research includes early studies of young software firms and seeks to determine how the startup concept has been adopted or broadened.
  • Startups are characterized by little or no operating history, limited resources, multiple influences, and dynamic technologies and markets.
  • Startup development progresses through startup, stabilization, growth, and maturity phases, with increasingly robust and predictable product-development processes.
  • Implementing methodologies to structure and control startup development is challenging, and several proposed models have not achieved significant benefits.
  • Product-oriented practices preserve flexibility and let teams change direction quickly, while giving employees more control than rigid guidelines.
  • Market-driven startups prioritize time-to-market while handling invented, rarely documented requirements in fast-moving and uncertain markets.
  • Reviews of related small-firm contexts similarly indicate that solutions and technologies should be adapted to the characteristics of smaller and younger companies.

3. Research methodology

The study uses a systematic mapping study because the research question covers a broad, poorly defined area. It follows established mapping guidelines, adds rigor and relevance assessment, and searches multiple scientific databases using structured keywords.

  • A systematic mapping study was selected over a systematic literature review because the overall question spans a broad and poorly defined research area.
  • The review follows Kitchenham and Charters’ guidelines and Petersen et al.’s mapping process, extending it with rigor and relevance assessment and constant-comparison synthesis.
  • The mapping asks about startup development context, reliability and transferability of research results, and reported software engineering practices.
  • Primary studies were identified through database searches using population, intervention, and comparison concepts, while omitting outcome and context restrictions.
  • Search terms were expanded with synonyms, related concepts, alternative spellings, and parts of speech, without startup-definition keywords to avoid search bias.
  • The search covered selected scientific databases and Google Scholar, with database-specific syntax customization to increase coverage.

3.3. Screening of Relevant Papers

Paper screening progressively removed duplicates and excluded ineligible studies through metadata and abstract review, followed by full-text review for disagreements or incomplete abstracts. The process ended with 43 primary studies and recorded exclusion rationales.

  • Studies were included when they contributed knowledge about software development in startups through experience reports, engineering practices, development models, or lessons learned.
  • Screening excluded non-peer-reviewed, non-English, obsolete, non-software, established-company, and purely technical startup studies.
  • Duplicate removal combined automatic metadata matching with manual deletion, leaving 1053 papers for screening.
  • Metadata screening reduced the set to 722 papers, and abstract review reduced it to 64 papers while search strings were iteratively improved.
  • Full-paper review resolved disagreements or incomplete abstracts and produced the final set of 43 primary studies.

3.4. Keywording

The keywording process created and refined a classification schema for organizing primary studies and supporting systematic data extraction, synthesis, and rigor/relevance assessment.

  • Keywording assigned abstracts sets of keywords identifying each primary study’s main contribution area.
  • High-level categories were formed by combining keywords, then progressively refined as papers were fitted into the schema and new data emerged.
  • The resulting schema supported systematic extraction of study metadata and synthesis across primary studies.
  • Snowballing screened each paper’s bibliography for additional relevant studies, but identified none.
  • The mapping extended the traditional SMS framework by evaluating primary studies’ scientific rigor and industrial relevance.
  • Rigor was scored across context, study design, and validity, while relevance covered subjects, context, scale, and research method.

3.7. Synthesis

The synthesis organized concepts from primary studies into comparable categories and working practices, documenting their advantages, disadvantages, and applicability gaps in startup contexts.

  • Concepts from each primary study were summarized using original author terms and organized in tables for cross-study comparison.
  • The synthesis identified working practices and classification categories, documented reported advantages and disadvantages, and exposed applicability gaps.
  • The analysis examined how practices apply in startups, including their benefits, liabilities, and relationship to findings from other studies.

3.8. Threats to validity

The study addressed validity threats involving publication bias, search strategy, inaccessible papers, inconsistent startup definitions, omitted context, cross-disciplinary literature, researcher judgment, and ranking design.

  • Publication bias: Publication bias favored positive outcomes and reduced the study’s ability to assess work-practice performance.
  • Identification of primary studies: The search string excluded stand-alone “startup” and “start-up” terms, potentially omitting relevant primary studies despite added qualifiers and synonyms.
  • Identification of primary studies: 4% precision resulted from 43 relevant papers among 1053 retrieved papers, while multiple databases were used to mitigate exclusion risk.
  • Identification of primary studies: 20 papers were inaccessible, although titles, keywords, and abstracts suggested that fewer than one potentially relevant study was missed.
  • Identification of primary studies: Researchers used inconsistent definitions of “startup,” ranging from severely resource-constrained new firms to companies with more than 150 employees.
  • Identification of primary studies: The mapping prioritized a general overview of practices over detailed analysis of application-domain, market, and other contextual factors.
  • Identification of primary studies: Relevant information from business innovation and marketing research may have been omitted because those areas were outside the study’s scope.
  • Study selection and data extraction: Rigor and relevance scores depended on reporting quality and researcher judgment, so they ranked rather than excluded studies and require reader qualification.

4. Classification schema

The classification schema organized studies by research type, contribution type, research focus, and pertinence to engineering activities in startups, forming the basis for the systematic maps.

  • The schema contains four facets: research type, contribution type, focus, and pertinence.
  • Research type: Research type distinguishes study types independently of their specific underlying research methodology.
  • Contribution type: Contribution type distinguishes weak contributions from strong contributions such as theories, frameworks or methods, and models.
  • Focus: Focus categories separate software development practices, higher-level process management, tools and technologies, and managerial or organizational aspects.
  • Pertinence: Pertinence distinguishes whether a study’s research focus is fully, partially, or marginally directed toward engineering activities in startups.
  • The schema provided the basis for the systematic maps presented in the study.

5. Results

The mapping study identified 43 primary studies from an initial sample of 1053 papers and classified their themes, contributions, research types, and relevance. The results show substantial emphasis on managerial and organizational factors, with varied contributions and limited pertinence to software engineering in startups.

  • 43 primary studies were identified from an initial sample of 1053 papers.
  • 346 extracted keywords, including 125 unique terms, formed the basis for classifying study focus and pertinence.
  • 11 studies (26% of the total) focused on managerial and organizational factors through evaluation research, while 8 contributed models.
  • 6 of the 10 models had only marginal pertinence to engineering activities in software startups.
  • 20 studies (47%) were journal articles, 16 (37%) appeared in conference proceedings, and 7 (16%) in magazines.
  • 9 studies (21%) combined high rigor and relevance, while 21 (49%) showed moderate industry relevance but low scientific rigor.

6. Analysis of the state-of-art

The state-of-the-art is recent but fragmented: studies use inconsistent startup definitions, many have limited pertinence or rigor, and only a small subset offers strong evidence. The ranking prioritizes pertinence to engineering activities while incorporating rigor, relevance, publication age and type, contribution, research type, and focus.

  • More than 65% of the 43 primary studies were published between 2004 and 2013, indicating that startup research remains relatively new.
  • 16 studies (37%) were entirely dedicated to software development in startups, but 10 produced weak contributions.
  • 19 studies (44%) focused on managerial and organizational factors, and none had full pertinence to engineering activities in startups.
  • The strongest contributions were concentrated in a few publications, although three relied on the same dataset from 21 companies.
  • Authors used inconsistent startup definitions across company size, age, organizational stage, and innovation criteria.
  • Recurring startup themes included resource scarcity, flexibility, innovation, uncertainty, time pressure, and fast growth, requiring explicit study-boundary definitions.
  • 7 studies (16%) combined average industrial relevance (2) with zero scientific rigor, creating a stated threat to industry transferability.
  • The ranking score ranges from 0 to 10, with pertinence weighted at 25%, followed by rigor and relevance at 17.5% each.

7. RQ3 - Work Practices in startups

The review extracted and categorized 213 reported work practices from 43 studies. Startups commonly use lightweight, adaptive processes, fast releases, prototyping, and tailored combinations of practices rather than following a single prescribed methodology.

  • 213 work practices were extracted from the 43 reviewed primary studies and categorized by development focus.
  • Process management practices: Agile methodologies were considered viable because they accommodate change and support iterative, incremental releases with rapid deployment.
  • Process management practices: Lean Startup practices identify business risks and use minimum viable products to test and plan subsequent iterations.
  • Process management practices: XP was reported as the most used methodology because of reduced process costs and low documentation requirements.
  • Process management practices: Startups opportunistically select and tailor practices such as pair-programming and backlogs to their specific development contexts.
  • Process management practices: Reportedly useful practices include lightweight methodologies for flexibility and fast releases for evolutionary prototyping and rapid user feedback.

Discussion

Software startups require flexible, context-tailored development practices because uncertainty makes rigorous prediction and prescriptive processes ineffective. The reviewed practices emphasize rapid learning, scalable architecture, lightweight implementation, and customer-focused testing.

  • Process adaptation: Flexible, reactive methods are preferred because startups cannot accurately identify all risks or predict required practices through extensive analysis.Such methods stimulate customer feedback and broaden the perspectives and solutions available to decision makers.
  • Process adaptation: Startups should enable developers to change direction quickly, learn from failures, and later balance flexibility with repeatable, scalable processes.The proposed approach adapts common Agile practices to startup cultures and needs while recognizing preparation for growth.
  • Requirements: Requirements are market-driven, difficult to specify, and rapidly changing, so startups emphasize basic elicitation, customer involvement, scenarios, and minimal functional requirements.These practices support testing problem/solution fit before investing in unsolicited requirements.
  • Design and Architecture practices: Architecture practices favor limited up-front analysis, reusable frameworks and components, modular design, and decisions that preserve scalability and later refactoring.Framework selection and early analysis of risky decisions can reduce difficulties as product complexity and user-base grow.
  • Implementation, Maintenance and Deployment practices: Implementation often begins informally, adding coding standards, pair programming, metrics, and refactoring when project complexity or growth requires them.Early-stage teams keep the codebase small and focused on core functionality, while business goals drive later engineering effort.
  • Quality Assurance practices: Testing is frequently compromised, but customer acceptance, early-adopter groups, and outsourced testing provide practical quality assurance for uncertain markets.Acceptance testing validates product effectiveness, while time-efficient automation and usable interface testing remain important research needs.

8. Conclusions and future work

The mapping study finds a limited and uneven evidence base for software development in startups, while identifying opportunistically adapted practices suited to resource constraints, uncertainty, and time pressure. It calls for clearer startup definitions, stronger contextualization, and more rigorous research to improve transferability.

  • 43 primary studies provide an inadequate basis for understanding software development in startups, despite the importance of software to startup activities.The study characterizes the existing scientific body of knowledge as insufficient to explain the underlying phenomenon.
  • 19 studies (44%) focus on managerial and organizational factors, while only 16 (37%) are entirely dedicated to software development in startups.Only four contributions are both fully dedicated to engineering activities and strong, evidence-based contributions; three rely on the same data.
  • Startups commonly face resource shortages, rapid reactivity, intense time pressure, uncertainty, and fast growth, but their contextual boundaries remain blurred.The authors recommend that researchers explicitly define the features of the startups they study.
  • 9 studies (21%) provide results judged transferable and reliable, whereas 23 (53%) combine moderate industry relevance with low scientific rigor.The study warns that this evidence profile makes transferring results to practitioners unlikely or potentially dangerous.
  • Lightweight, tailorable processes are preferred because conventional agile and traditional methodologies struggle under startup uncertainty and time pressure.Reported practices include partial and late adoption, market-driven and lightly documented requirements, framework-based design, refactoring, pair programming, and customer-acceptance testing.
  • Future research should establish common startup terminology, explicitly describe study context and validity threats, and build a more consistent evidence-based knowledge base.The study links clearer definitions and stronger reporting to improved support for startup decisions and greater potential for transferring results to industry.
Loading 2307.13104v1…