PaperlayerView exact revision on arXiv

Source-linked AI summary

FirmCORe: A Benchmark for Structured Reasoning about Inter-Firm Collaboration Opportunities

Tian Du, Tiantong Wu, Yafei Wang, Mengyu Liu, Xingyan Chen, Mu Wang

arXiv:2609.17128v1cs.AI

TL;DR

Structured data on inter-firm relationships are scarce, and business similarity alone does not establish collaboration potential. FirmCORe provides a human-annotated benchmark for structured reasoning over firm profiles; results indicate that LLMs detect broad opportunities more reliably than they identify specific relationship fields.

  • Problem

    Structured inter-firm relationship data are scarce, and profile-based inference is difficult because similarity may signal competition rather than capability complementarity.

    Source: §1 · p. 1

  • Method

    FirmCORe is a human-annotated benchmark of 2,805 firm pairs for predicting opportunity existence and, for positive pairs, strength, primary type, and role direction.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    LLMs detect broad collaboration opportunities substantially more easily than they correctly identify collaboration types, role directions, and complete four-field relation structures.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Firm profiles must be assessed for evidence of capability complementarity rather than treated as sufficient evidence based on business similarity alone.

    Source: §1 · p. 1

    The benchmark tests whether profiles support potential collaboration, not whether firms have collaborated in practice.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Comprehensive structured data on inter-firm relationships is often scarce or inaccessible because many relationships are privately negotiated, selectively disclosed, and fragmented across proprietary databases. This scarcity hinders the discovery of collaboration opportunities, particularly for startups and small and medium-sized enterprises. Firm profiles are readily available, but collaboration potential cannot be inferred from business similarity alone, since similar firms may be competitors, whereas dissimilar firms may offer complementary products, technologies, channels, capabilities, or capital. We present FirmCORe (Inter-Firm Collaboration Opportunity Reasoning), a human-annotated benchmark for pairwise reasoning over weakly structured firm profiles, comprising 2,805 labeled firm pairs. Given two firm profiles, a model must determine whether the available evidence supports a collaboration opportunity and, for positive pairs, jointly predict its strength, primary collaboration type, and role direction. FirmCORe also provides parallel Chinese- and English-language evaluation sets containing identical instances and gold labels, enabling controlled analysis of input-language sensitivity. Experiments with representative locally deployed and hosted large language models (LLMs) show that the strongest model achieves a macro-F1 score of 74.51 for opportunity detection but only 61.57% exact match across all four output fields. Language effects vary across models, and high cross-language agreement can mask errors shared across languages. These results indicate that current LLMs are substantially more reliable at detecting broad collaboration opportunities than at identifying their specific types and role directions.

1 Introduction

Structured inter-firm relationship data are scarce, while firm profiles are more widely available but difficult to interpret as evidence of collaboration. FirmCORe targets the unresolved task of assessing profile-grounded collaboration opportunities and their specific relationship fields.

1 Introduction

  • Structured relationship data are scarce, particularly for startups and small and medium-sized enterprises, while firm profiles offer a more widely available evidence source.Existing transaction networks and supply–demand databases capture only some realized relationships and provide even less information about opportunities that have not materialized.Source: §1 · p. 1
  • Collaboration inference requires capability complementarity rather than business similarity: similar firms may compete, while dissimilar firms may complement one another.The task includes determining opportunity existence, strength, primary type, and role direction from incomplete, weakly structured profiles.Source: §1 · p. 1
  • Existing approaches do not jointly test whether two firm profiles support a potential collaboration, leaving LLMs’ multidimensional collaboration reasoning largely unknown.Similarity methods capture relatedness, graph-based prediction relies on observed relationships, and partner recommendation often targets retrieval or ranking in specific settings.Source: §1 · p. 1
  • FirmCORe introduces a human-annotated benchmark of 2,805 gold-labeled firm pairs and frames profile-based collaboration reasoning as jointly predicting four relation fields.The benchmark also provides parallel Chinese- and English-language evaluation sets.Source: §1 · p. 2

2 Related Work

  • Prior work addresses firm similarity and graph-based link prediction, while FirmCORe evaluates structured, profile-grounded collaboration reasoning without networks or platform interactions.Other LLM benchmarks primarily evaluate individual questions, documents, simulated scenarios, or predefined workflows.Source: §2 · p. 2

3 The FirmCORe Benchmark

FirmCORe defines pairwise prediction of evidence-supported collaboration opportunities from firm profiles, with labels for existence and, for positive pairs, strength, type, and role direction. Its benchmark is built through profile preparation, stratified pair mining, screening, and blind human annotation.

3.1 Task Definition

  • Each firm pair receives an existence label and, when positive, labels for strength, primary collaboration type, and role direction.Strength distinguishes direct, well-supported interfaces from plausible but indirect or less clearly supported ones; direction is relative to the displayed A/B order.Source: §3.1 · p. 3
  • The benchmark measures whether the provided profiles support a potential collaboration, not whether the firms have collaborated in practice.Models must rely solely on the provided profiles.Source: §3.1 · p. 3

3.2 FirmCORe Framework

A. Firm Profile Construction

  • Unified firm profiles contain firm name, industry, core business, products and services, and a textual summary; missing fields are omitted without imputation.Profiles are deduplicated, and records that are too sparse for meaningful pairwise assessment are excluded.Source: A. Firm Profile Construction · p. 3

B. Stratified Candidate Pair Mining

  • Candidate pairs are mined from four strata, including hard same-industry pairs with business relatedness but no explicit supply–demand complementarity.Upstream relationship scores guide candidate selection only, not ground-truth labels; stratum information is hidden from annotators.Source: B. Stratified Candidate Pair Mining · p. 3

C. Human Annotation

  • Two annotators independently label each pair using only the firm profiles, requiring an evidence-supported collaboration interface rather than industry similarity or lexical overlap alone.For negative pairs, strength, type, and direction are all set to None.Source: C. Human Annotation · p. 3

D. Gold Label Aggregation

Gold labels were resolved at the complete four-field tuple level, and the evaluation distinguishes end-to-end structured prediction from attribute reasoning after successful detection. The parallel Chinese–English sets hold pairs and gold labels constant while changing profile language.

D. Gold Label Aggregation

  • 2,805 gold-standard instances resulted from tuple-level adjudication; unresolved disagreements excluded 195 cases.Third-annotator adjudication was required for 30.55% of samples; initial annotators averaged 84.26% field-level and 69.45% full-tuple agreement.Source: D. Gold Label Aggregation · p. 3

E. Structured Model Evaluation

  • End-to-end attribute scores count No, invalid, or missing predictions as incorrect, while oracle detection isolates attribute reasoning after successful opportunity detection.Opportunity detection is evaluated on all instances; strength, type, and role direction are evaluated on gold-positive instances.Source: E. Structured Model Evaluation · p. 4
  • Four-field exact match evaluates full structured prediction, and paired exact agreement measures consistency between the parallel Chinese and English sets.The parallel design enables instance-level comparison while holding sample composition and gold labels constant.Source: E. Structured Model Evaluation · p. 4, §3.3 · p. 4

3.3 Chinese–English Parallel Evaluation Set

  • The English parallel set preserves the same 2,805 firm pairs and gold labels as Chinese, differing only in profile language.English profiles were translated field by field, with human review required for 94 profiles appearing in 144 pairs flagged for potential ambiguities.Source: §3.3 · p. 4

4 Experiments

Experiments evaluate 17 hosted and locally deployed LLMs under a shared zero-shot setup. The protocol assesses opportunity detection, relation attributes, complete structured predictions, and Chinese–English agreement.

4.1 Experimental Setups

  • The evaluation covers 17 LLMs, from 1.5B-parameter models to large mixture-of-experts systems, including open-weight local and hosted models.The listed models include DeepSeek, Llama, Qwen, Gemma, GLM, and Kimi variants.Source: §4.1 · p. 4
  • All models received the same standardized prompt and were evaluated zero-shot, without examples or task-specific fine-tuning.Each sample was evaluated once at temperature 0 with a maximum output length of 2,048 tokens.Source: §4.1 · p. 4
  • Evaluation measures opportunity-detection Macro-F1, end-to-end and oracle relation attributes, four-field exact match, paired language agreement, validity, and confidence reliability.The metrics follow the structured evaluation protocol defined in Section 3.2.Source: §4.1 · pp. 4–5

5 Results and Analysis

LLMs detect broad collaboration opportunities more reliably than they recover complete relation structures, and performance varies with candidate stratum, input language, and reliability measure. Across models, valid or matching predictions do not necessarily indicate correct structured reasoning.

5.1 RQ1: Distinguishing Collaboration Opportunities from Relatedness

  • 74.51 Macro-F1 is Qwen3.6-27B’s best opportunity-detection result, but its 59.99% precision and 84.71% recall accompany a broader tendency to overpredict plausible industrial or commercial connections.Several models approach saturated recall while achieving substantially lower precision.Source: §5.1 · p. 5, Table 2 · p. 5
  • 58.77 Macro-F1 on same-industry hard pairs is below Qwen3.6-27B’s 66.42 on randomly sampled non-edge pairs; the best zero-shot LLM nevertheless outperforms all evaluated baselines.Because stratum prevalence differs substantially, cross-stratum scores are descriptive rather than controlled estimates of intrinsic difficulty.Source: §5.1 · p. 5, Figure 2 · p. 5

5.2 RQ2: Recovering Structured Collaboration Relations

  • 61.98% is the highest positive four-field exact match, achieved by Kimi-K2.7; Qwen3-2B’s 4.65% on gold-positive pairs contrasts with its high overall exact match, which mainly reflects rejecting negative pairs.The paper concludes that LLMs identify broad collaboration potential more reliably than they reconstruct complete strength, mechanism, and role direction.Source: §5.2 · p. 6
  • 1.11 points is Kimi-K2.7’s mean oracle–end-to-end gap across three attributes despite detecting 97.42% of gold-positive pairs; Qwen3-2B’s 19.21-point gap accompanies only 26.96% coverage.For high-recall systems, remaining errors mainly arise after detecting an opportunity; missed detection is the dominant bottleneck for Qwen3-2B.Source: §5.2 · p. 6, Figure 3 · p. 6

5.3 RQ3: Sensitivity to Input Language

  • 11.34 points higher for Qwen3.6-35B and 4.96 points lower for Qwen3-8B, English-input four-field exact match varies by model; Qwen3.6-Max’s 79.57% paired agreement includes only 54.62% correct in both languages.Qwen3.6-27B changes by −1.43 points, and Figure 4’s diagonal marks equal Chinese and English performance.Source: Figure 4 · p. 6, Figure 5 · p. 6, §5.3 · p. 7
  • No collaboration category has a consistent language advantage across systems: English improves some categories for particular models and degrades the same categories for others.Language sensitivity is model-, instance-, and category-dependent rather than a uniform performance shift.Source: Figure 6 · p. 6, §5.3 · p. 7

5.4 Reliability Analysis

  • 0.21% is Qwen3-Coder’s output-violation rate, while high-confidence subsets for three models still contain 31.86–39.64% errors.Most hosted models produce syntactically valid JSON, but cross-field consistency varies substantially and self-reported confidence is only partially informative.Source: §5.4 · p. 7

6 Conclusion

FirmCORe evaluates whether weakly structured firm profiles support collaboration opportunities and whether models can recover their relational structure. Its scope is profile-inferred potential rather than realized or commercially viable partnerships, and its experiments leave several prompting and run-variability questions unexplored.

6 Conclusion

  • Current LLMs identify broad collaboration potential more reliably than they reconstruct opportunity strength, type, and role direction.They may overpredict from relatedness, conflate mechanisms, and misidentify resource-flow direction; language effects vary, and cross-language consistency can repeat errors.Source: §6 · p. 7

7 Limitations

  • FirmCORe evaluates opportunities inferred from provided profiles, not realized partnerships or their commercial viability.Its pairwise setup also does not cover large-scale partner retrieval or ranking.Source: §7 · p. 7
  • Experiments use zero-shot prompting and single deterministic runs, leaving fine-tuning, prompt sensitivity, tool use, and multi-run variability unexplored.These choices limit what the experiments establish about performance beyond the tested setup.Source: §7 · p. 7

A Candidate-Mining Details

Candidate mining uses supply–demand and similarity signals to select firm pairs, while Chinese–English translation aims to preserve profile meaning, labels, and evidential boundaries across parallel examples.

A Candidate-Mining Details

  • Supply–demand and similarity signals are computed from firm label sets and used for candidate mining, not as gold labels.The candidate pool is generated and capped by stratum, with duplicates removed and pairs meeting multiple criteria assigned to one stratum.Source: A Candidate-Mining Details · p. 9
  • At most 750 pairs per stratum proceed to human annotation after screening for potential leakage, privacy risks, and data-quality issues.Annotators do not see the candidate strata, upstream scores, or source-record identifiers.Source: A Candidate-Mining Details · p. 9

B Chinese–English Parallel Translation

  • The English set contains the same 2,805 pairs and unchanged gold labels as the Chinese set, with translation designed to preserve meaning and evidential boundaries.The translation does not silently repair source-level inconsistencies.Source: B Chinese–English Parallel Translation · pp. 9–10
  • Each of 2,256 unique profiles received one English representation reused across pairs, preventing inconsistent translations of repeated profiles.Profiles were decomposed into entity name, industry, core business, products or services, and firm summary.Source: B Chinese–English Parallel Translation · p. 9
  • Business descriptions were rendered in professional English without adding unsupported capabilities, products, qualifications, or relationships.Long expressions could be reorganized for readability while preserving semantic content and enumeration boundaries.Source: B Chinese–English Parallel Translation · p. 9
  • Entity names use official English forms when verifiable; otherwise, standardized renderings and review levels record translation provenance and confidence.Distinctive Chinese brand terms may be transliterated, and uncertain or ambiguous names can be marked for review.Source: B Chinese–English Parallel Translation · p. 9
  • A controlled glossary and fixed collaboration labels preserve functional terminology and explicit role direction across translations.A2B means Firm A provides resources to Firm B; B2A denotes the reverse, and no direction is inferred or altered during translation.Source: B Chinese–English Parallel Translation · p. 9
  • Automatic and manual checks align English records with Chinese sources and verify that labels, scores, and role directions remain unchanged.Checks also cover missing fields, residual Chinese characters, entity-name consistency, and malformed profiles.Source: B Chinese–English Parallel Translation · p. 10

C Data Statement

The data statement documents how FirmCORe is sourced, filtered, labeled, and bounded; the prompts specify profile-grounded judgments and structured outputs for evaluating those judgments.

C Data Statement

  • FirmCORe combines platform organization and business records with derived profile fields and upstream AI- or platform-generated intermediate fields.Commercial access and licensing constraints mean only the final gold-labeled benchmark is released.Source: C Data Statement · p. 10
  • Organization profiles are matched mainly by names and deterministic rules, then filtered to remove invalid, duplicate, self, and signal-revealing pairs.The process is not a comprehensive entity-resolution system.Source: C Data Statement · p. 10
  • The 2,805 gold-labeled pairs come from four deliberately mined strata spanning strong signals, borderline signals, same-industry hard pairs, and random non-edges.Up to 750 pairs per stratum entered annotation, limiting domination by trivial random negatives.Source: C Data Statement · p. 10
  • Released profiles omit direct contact details and sensitive platform fields, but these measures do not replace independent legal, privacy, and data-governance review.Any public release should be limited to the minimum information needed to reproduce the task.Source: C Data Statement · p. 10
  • A positive label means the profiles provide sufficient evidence for a plausible opportunity under the guidelines, not that the firms have collaborated.A negative label means evidence is insufficient in the profiles, not that collaboration is impossible in practice.Source: C Data Statement · p. 10
  • The data primarily reflects Chinese commercial and institutional contexts; its English set translates the same profiles, and label imbalance and time-sensitive profiles may affect evaluation.Positive instances concentrate in strong opportunities, Supply and Production, and one displayed role direction; the paper reports Macro-F1 and per-class metrics as well as accuracy.Source: C Data Statement · pp. 10–11

D Prompts

  • Chinese and English prompts share the task definition, label space, consistency constraints, and JSON schema, while requesting rationales in their respective languages.The instructions require judgments to rely on the provided profiles rather than unsupported external facts.Source: D Prompts · p. 11
  • The structured output links opportunity existence to strength, collaboration type, and role direction, with field constraints and profile-evidence criteria governing valid labels.A strong opportunity requires a relatively direct relationship; similarity, shared industry, or lexical overlap alone is insufficient.Source: D Prompts · p. 11

E Additional Results

Among the reported baselines, Qwen3.6-27B is the strongest opportunity detector, while similarity and rule-based methods underperform. Bootstrap intervals quantify instance-level uncertainty, but not run-to-run or prompt variation.

E.1 Baselines and Detection Uncertainty

  • Below 48 Macro-F1, unsupervised similarity and rule-based methods lag; supply–demand complementarity has 67.29% precision but 7.44% recall.The passage attributes the low recall to direct label overlap missing implicit capability complementarity.Source: §E.1 · pp. 11–12
  • 74.51 Macro-F1 is Qwen3.6-27B’s best overall result, above the lightweight supervised classifier at 71.69 and upstream score at 67.29 Macro-F1.Its positive precision and recall are 59.99% and 84.71%, respectively.Source: §E.1 · p. 12, Table 5 · p. 12
  • 74.51 Macro-F1 for Qwen3.6-27B has a 95% bootstrap CI of 72.89–76.07, with Qwen3.6-Max and Qwen3.7-Plus next.The intervals quantify instance-level uncertainty, not prompt sensitivity or run-to-run variation, because each model was evaluated once under deterministic decoding.Source: §E.1 · p. 12

E.2 Joint and Oracle Structured Prediction

Joint prediction remains harder than detecting positive pairs: the strongest positive four-field exact match is 61.98%. Cross-language agreement can conceal shared errors, and reversing firm order reveals additional prediction instability.

E.2 Joint and Oracle Structured Prediction

  • For most high-recall systems, the small oracle–end-to-end gap indicates that errors mainly arise in assigning relation attributes after detecting an opportunity; missed detection dominates for Qwen3-2B.Conditional attribute scores must be read with positive coverage because high oracle scores at low coverage describe only the detected subset.Source: §E.2 · p. 12

E.3 Complete Cross-Lingual Results

  • Cross-language effects vary by model, and similar aggregate exact-match scores can mask substantial asymmetry between Chinese-only and English-only subsets.Paired agreement exceeds the proportion correct in both languages, partly because models repeat the same incorrect tuple across languages.Source: §E.3 · pp. 12–13

E.2 Joint and Oracle Structured Prediction

  • 61.98% positive four-field exact match is Kimi-K2.7’s highest score, followed by Qwen3-Coder at 56.10%.Qwen3-2B’s 51.98% overall exact match contrasts with 4.65% on gold-positive pairs, alongside only 26.96% positive coverage.Source: §E.2 · p. 12, Table 6 · p. 13

E.4 Robustness to A/B Firm-Order Permutation

  • 76.47% average exact equivariance and 85.90% direction equivariance show that predictions do not always preserve expected structure after firm-order reversal.18.91% of predictions change opportunity existence, strength, or collaboration type, not just direction.Source: §E.4 · p. 13
  • 2.15 points is the average net correction after permutation: 4.61% of instances change from incorrect to correct versus 2.46% from correct to incorrect.The balanced variant is a stress test for order sensitivity, not evidence against the original benchmark’s main conclusions.Source: Figure 8 · p. 13, §E.4 · p. 13

E.5 Output Validity and Confidence Reliability

The error analyses distinguish missed opportunities from mistakes in relation attributes after a positive prediction. They also show structural-output and confidence caveats, alongside recurring false positives, false negatives, and type and direction errors.

E.5 Output Validity and Confidence Reliability

  • Qwen3-Coder has the lowest cross-field violation rate at 0.21%, but syntactic validity and parsing success do not ensure structural consistency.Common violations include retaining relation attributes after predicting No and omitting required fields after predicting Yes.Source: §E.5 · p. 14
  • High–Low confidence exact-match gaps reach 67.05 points for Qwen3.6-27B, yet its High-confidence predictions still contain 31.86% errors.Confidence interpretation depends on coverage, and some models’ Low-confidence estimates are unstable because nearly all instances fall in the High subset.Source: §E.5 · p. 14

F.1 Error Decomposition

  • 600/968 (62.0%) is Kimi-K2.7’s positive four-field exact match; 343 pairs (35.4%) have an attribute error after a positive prediction.Qwen3.6-27B achieves 437/968 (45.1%) positive exact match, with 148 (15.3%) false negatives and 383 (39.6%) attribute errors.Source: Figure 9 · p. 14, §F.1 · pp. 14–15

F.2 Systematic Failure Modes

  • 18,551 of 23,621 false-positive decisions (78.5%) are High confidence, and 1,773 of 1,836 misclassified gold-negative pairs are misclassified by at least three models.The recurring predictions often rely on industry proximity, shared terminology, or market adjacency without a sufficiently specific collaboration interface.Source: §F.2 · p. 15
  • At least one model predicts No for 299/358 borderline pairs (83.5%) and 347/447 strong-signal pairs (77.6%), indicating false negatives are not confined to low-signal cases.The pattern suggests difficulty combining evidence distributed across products, services, capabilities, channels, and business roles.Source: §F.2 · p. 15
  • Only 6.5% of valid type predictions for Technology instances are correct; 48.9% are labeled Supply and Production and 23.4% R&D and Co-development.Distribution and R&D are also frequently mapped to Supply and Production, at 27.6% and 25.6%, respectively.Source: §F.2 · p. 15
  • 72.31% accuracy is attainable from an always-B2A baseline; Kimi-K2.7 reaches direction Macro-F1 67.71 and 77.38% accuracy, only 5.07 points above that baseline.Model direction biases vary, including excess B2A predictions, excess Bidirectional predictions, and a strong A2B bias.Source: §F.2 · p. 15

F Error Analyses

  • The permutation analysis treats changes to opportunity existence, strength, or collaboration type as severe, distinguishing them from direction-only changes.Exact equivariance requires preserving the complete prediction for unswapped pairs and only reversing A2B/B2A for swapped pairs.Source: §F.2 · p. 16

F.3 Cross-Lingual and Confidence-Related Failures

Cross-language agreement does not guarantee correctness, and self-reported confidence remains a model-specific diagnostic rather than a calibrated probability. Representative cases illustrate how errors persist from opportunity detection through recovery of the full relational structure.

F.3 Cross-Lingual and Confidence-Related Failures

  • Identical incorrect tuples occur in both languages for 21.5% of Qwen3.6-27B pairs, 25.0% of Qwen3.6-Max pairs, and 26.2% of Qwen3.7-Plus pairs.For Gemma3-12B, identical wrong tuples account for 38.8% of pairs, while another 43.3% receive different incorrect tuples.Source: §F.3 · p. 16, Table 12 · p. 18
  • 31.86% of Qwen3.6-27B’s High-confidence Chinese predictions are wrong under four-field exact match, despite 90.20% of predictions being labeled High confidence.High-confidence error rates are 34.60% for Qwen3.6-Max and 39.64% for Qwen3.7-Plus.Source: §F.3 · p. 16

F.4 Representative Failure Cases

  • Representative cases expose three boundaries: plausible relatedness without a specific interface, implicit complementarity requiring combined evidence, and errors between semantically adjacent mechanisms.One case changes both collaboration mechanism and resource-flow direction; another retains the correct direction but substitutes Marketing and Distribution for Supply and Production despite High confidence.Source: §F.4 · p. 16, Table 13 · p. 18
Loading 2609.17128v1…