Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth

• Heterogeneous subset-specific scorers can use different evidence representations and incomparable scales, so their outputs cannot be directly compared or meaningfully fused without calibration.Source: §1 · p. 1
• The framework aligns subset-specific scorers on a synthetic ordinal reference space before fusing them into a supervision score.Source: §1 · p. 1
• The evaluation spans a residential energy-retrofit case with text-derived and structured scorers, plus housing and medical benchmarks.The stated contributions also include formal properties and empirical evidence on ordinal preservation and supervision quality.Source: §1 · p. 2

• Programmatic weak supervision assumes a shared label space and estimates latent labels, whereas this work aligns heterogeneous scoring functions on incomparable semantic scales.Source: §2 · pp. 2–3
• Multimodal learning typically fuses heterogeneous representations for prediction with observable targets; this work instead constructs supervision without a ground-truth target.Source: §2 · pp. 2–3
• Prior research offers foundations, but the reviewed areas do not directly address constructing supervision without ground-truth labels or a common target space.Source: §2 · p. 2
• Psychometrics places indicators on a common measurement scale, while this approach uses an external synthetic reference structure without ground-truth labels or a pre-existing instrument.Source: §2 · pp. 2–3

3.1.1 Notation and Assumptions

• The framework assumes each calibration feature is monotonically related to the latent concept; if this fails, the resulting score has no semantic-validity guarantee.Source: §3.1.1 · p. 4

3.1.2 Subset-Specific Scorers

• Each scorer operates on a predefined coherent feature subset and may be learned or heuristic, but its real-valued output must preserve a common direction relative to the latent concept.Source: §3.1.2 · p. 4

3 Framework

• The framework has calibration and inference phases: scorers are evaluated over a synthetic reference space, then standardized and fused on new unlabeled instances.Source: §3 · p. 3, §3.2 · p. 5

3.2 Operational Phases

• Calibration evaluates each scorer on its corresponding evidence generated from reference points, estimating scorer-specific statistics for later use on unseen instances.Source: §3.2 · p. 5, §3.2.1 · p. 5

3.1.1 Notation and Assumptions

• Calibration features form an ambient configuration space, and the synthetic reference space uses ordered levels whose combinations span the latent concept.Source: §3.1.1 · p. 4, §3.2.1 · p. 5

3.2.1 Calibration via a Synthetic Reference Space

• The reference space is built from ordered calibration levels and their Cartesian product; its independence from observed samples supports comparable, stable mappings reusable across datasets.Source: §3.2.1 · pp. 5–6
• Using k levels across n features can require evaluating k^n reference points with each scorer; larger k raises computational cost, so resolution trades off detail against tractability.Source: §3.2.1 · pp. 5–6

3.2.2 Supervision Score Fusion & Normalization

• Inference standardizes each scorer using its calibration statistics, preserving scorer ordering, then equally averages the standardized outputs into one supervision score.Equal weighting is a neutral default because the framework has no empirical basis for estimating scorer reliability or assigning differential confidence.Source: §3.2.2 · pp. 6–7

4.1 Cross-Domain Validation

• Across all three domains, calibrated fusion improves over uncalibrated averaging with Bonferroni-adjusted intervals excluding zero, and has higher primary-metric point estimates than the best individual scorer.Primary-metric comparisons use paired bootstrap resampling with B=50,000 and Bonferroni-adjusted 98.33% confidence intervals.Source: §4.1 · p. 7, Table 1 · p. 8
• Against empirical z-scoring, Borda rank averaging, and PCA, adjusted differences are domain-dependent: calibrated fusion is less than 0.02 lower than all three on Ames Housing, with all differences distinguishable from zero.On Breast Cancer Wisconsin, calibrated fusion is higher than empirical z-score averaging by 0.0017 after correction, with no adjusted difference from Borda or PCA; Wine Quality shows no adjusted differences from the three methods.Source: §4.1 · p. 7

4.2 Empirical Illustration of Framework Properties

• Across four scorer-level sensitivity tests, greater efficiency did not increase estimated retrofit need, and calibration preserved each raw-output ordering across 195,000 pairwise comparisons per scorer.The tests varied one scorer’s inputs across five ordered efficiency levels while holding the others at the neutral third level.Source: §4.2 · pp. 8–9
• Jointly inefficient evidence produced the highest retrofit-priority signal and jointly efficient evidence the lowest for both concepts; mixed conditions fell between, with asymmetric contributions.The asymmetric intermediate results reflect differences in the magnitudes of calibrated text-derived and structured contributions.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 Calibration Resolution Sensitivity Analysis

• k=9 yields 0.9987 correlation and 96.7% top-10% overlap with k=15 using 12.9% of its calibration points; k=15 adds 0.0013 correlation but takes approximately 2.6 to nearly 20 hours.Figure 2 plots Spearman ρ against EnergyPlus compute time across tested k values.Source: §4.3 · pp. 9–10, Figure 2 · p. 10

5 Conclusion

• In the residential retrofit case study, the implementation showed predicted monotonic behavior, and lower-resolution grids closely approximated higher-resolution results at substantially lower computational cost.The conclusion presents the framework as a way to construct supervision scores when ground-truth labels and a shared annotation space are unavailable.Source: §5 · p. 10
• Across three benchmarks, calibrated fusion outperformed uncalibrated averaging and had higher primary-metric point estimates than the best individual scorer.Differences from uncalibrated averaging remained distinguishable from zero after Bonferroni correction.Source: §5 · p. 10
• Against empirical z-scoring, Borda rank averaging, and PCA, performance varied by domain; calibrated fusion was within 0.02 of each on Ames Housing.After correction, all three Ames differences remained distinguishable from zero; one adjusted difference was detected on Breast Cancer Wisconsin, and none on Wine Quality.Source: §5 · p. 10
• Calibration uses an externally defined synthetic reference space rather than the observed sample, supporting score comparability across cohorts and transferability across datasets.The framework also supports stability under changes in sample composition and separation of scorer behavior from the observed-sample distribution.Source: §5 · pp. 10–11
• The framework relies on domain knowledge to select and order calibration features, and future work should examine sensitivity to those construction assumptions.The authors also identify incorporating scorer reliability and principled calibration-resolution selection as future directions.Source: §5 · p. 11

A Propositions & Proofs

• Calibration preserves a scorer's latent-consistent ordinal structure because standardization with positive scale preserves its ordering.The result is conditional on subset-specific scorers being monotone observations of the latent concept.Source: A Propositions & Proofs · p. 13
• Fusion ranks one instance above another exactly when its aggregate calibrated support exceeds the other's.The proof partitions calibrated scorer differences into positive and negative contributions.Source: A Propositions & Proofs · pp. 13–14

B Calibration Space Construction & Resolution Analysis

• The calibration space is a shared reference formed from combinations of ordered feature levels, with its point count growing exponentially in feature count and level count.For the Synthetic Homes case study, the calibration features are HVAC heating and cooling COP, roof R-value, and wall R-value.Source: B Calibration Space Construction &… · p. 14, §B.1 · p. 14, §B.2 · pp. 14–15

B.1 Selection of Calibration Levels

• Calibration levels span each feature's feasible range while preserving ordinal relationships; the framework allows application-specific procedures for selecting them.Main-experiment levels are approximately evenly spaced, while lower-resolution sensitivity grids are nested subsets of the k=15 reference that preserve endpoints and similar coverage.Source: §B.1 · p. 14

B.2 Calibration Resolution and Computational Cost

• With four independently varied calibration features and k ordinal levels per feature, calibration points equal k^4; finer reference-space coverage requires additional scorer evaluations and, in the Synthetic Homes case study, additional EnergyPlus simulations.Source: §B.2 · pp. 14–15
• Figure 3 illustrates that calibration points grow as a function of k; increasing resolution improves agreement with the highest-resolution reference space, but computational cost grows rapidly.Source: §B.2 · p. 15

B.3 Resolution Agreement and Diminishing Returns

• Agreement with the k=15 reference improves rapidly at lower resolutions before approaching saturation at higher resolutions.Table 6 reports results across resolutions, including calibration-space size, computation time, Spearman correlation, and top-10% overlap.Source: §B.3 · p. 15, Figure 4 · p. 16
• Increasing k improves agreement but requires increasingly larger calibration spaces, with higher resolutions bringing only marginal rank-preservation gains at substantially greater computational expense.The calibration space grows as k^4 when four calibration features are varied independently.Source: §B.3 · pp. 15–16, Figure 3 · p. 16

B.4 Implications for Calibration Resolution Selection

• The appropriate k depends on available computational resources and the precision required for an application.Automated, elbow-based selection of calibration resolution is suggested as future work.Source: §B.4 · p. 16

C Cross-Domain Validation Details

• Across the three validation datasets, target labels are withheld during supervision construction, and feature groups and calibration endpoints are selected a priori based on semantic correspondence and expected monotonicity.The feature groupings and endpoint values are not selected using held-out target labels.Source: C Cross-Domain Validation Details · p. 17

C.1 Ames Housing

• For Ames Housing, the latent concept is property value, represented by four heuristic scorers using ordered calibration dimensions and five levels per dimension.The SalePrice target is removed before scorer construction; the four dimensions produce 625 synthetic calibration points.Source: §C.1 · p. 17, Table 7 · p. 17

C.2 Breast Cancer Wisconsin

• For Breast Cancer Wisconsin, the latent concept is malignancy risk, and its target is withheld during supervision construction.Four subset-specific mean scorers use feature sets varied across five levels, yielding 625 synthetic reference points.Source: §C.2 · pp. 17–18, Table 8 · p. 18

C.3 Wine Quality

• For Wine Quality, four heuristic scorers represent the latent concept of wine quality, with target labels removed before supervision construction.The four calibration dimensions are varied across five levels to yield 625 synthetic calibration points.Source: §C.3 · p. 18, Table 9 · p. 19

C.4 Calibration, Fusion, and Baselines

• Each scorer’s observed output is standardized using its reference mean and population standard deviation, and calibrated outputs are averaged with equal weight to produce κ.Source: §C.4 · p. 19
• The uncalibrated baseline is the arithmetic mean of four raw scorer outputs; other comparisons include empirical z-score averaging, Borda rank averaging, unsupervised PCA’s first principal component, and each scorer individually.Source: §C.4 · p. 19
• Ames Housing and Wine Quality use Spearman correlation as the primary metric, while Breast Cancer Wisconsin uses AUROC; paired primary comparisons are reported in Table 1.Kendall correlation and top-10% overlap are secondary and additional metrics for Ames Housing and Wine Quality; Breast Cancer Wisconsin uses average precision and Spearman correlation with malignancy status.Source: §C.4 · p. 19

D Synthetic Homes Implementation Details

• The Synthetic Homes implementation addresses residential retrofit priority using four calibration features spanning HVAC efficiency and insulation.Source: D Synthetic Homes Implementation Details · p. 19, §D.1 · p. 19

D.1 Calibration Features and Subset-Specific Scorers

• Two LLM text scorers estimate HVAC and insulation retrofit need from inspection notes, while two structured scorers evaluate the corresponding numerical feature subsets.All four outputs are oriented so that larger values indicate greater retrofit need.Source: §D.1 · pp. 19–20

D.2 Synthetic Ordinal Reference Space

• Five ordered efficiency levels for each of four calibration features form 625 synthetic reference points, with domain-selected values held fixed across experiments.The values span inefficient to highly efficient conditions but are not building-code thresholds, recommendations, or housing-stock quantiles.Source: §D.2 · p. 20

D.3 Whole-Home Inspection Notes

• Heating and cooling levels, and roof and wall levels, are each collapsed into a note level; one HVAC and one insulation text component are then combined into a home note.The LLM evaluates each complete note once per request and returns separate HVAC and insulation scores in [0, 1]; duplicate note outputs are cached while retaining their multiplicity in the calibration distribution.Source: §D.3 · pp. 21–22

D.4 Structured Scorers

• Structured scorers interpolate efficiency levels between calibration anchors and map them to retrofit-need values, with higher efficiency corresponding to lower need.At the five anchors, feature-specific retrofit-need values are 5, 4, 3, 2, and 1; these scores use neither EnergyPlus outputs nor held-out retrofit labels.Source: §D.4 · pp. 22–23

D.5 Text Scorer

• The text scorer uses GPT-4.1-mini at temperature zero to return HVAC and insulation retrofit-need scores between 0 and 1.Zero indicates definitely unnecessary retrofit, one definitely necessary retrofit, and intermediate values partial need or uncertainty.Source: §D.5 · p. 23

D.6 Calibration and Fusion

• Each scorer is standardized using statistics computed over the reference points; calibrated outputs are fused within building-system categories and averaged across scorers into the overall supervision signal κ.The same note-construction and scoring procedures are applied at inference, using the corresponding scorer’s reference distribution.Source: §D.6 · pp. 23–24

E Exploratory Psychometric Analysis

• The psychometric datasets contain related subscales, not naturally heterogeneous sources operating on incompatible scales, so these experiments are exploratory rather than direct validation.Source: E Exploratory Psychometric Analysis · p. 24
• The analysis tests whether fusion of domain or subscale scores can recover latent structure by comparing κ with the first principal component of the full item-response matrix.Source: E Exploratory Psychometric Analysis · p. 24

• Domain scores from the 50-item BIG5 inventory’s five personality domains were fused into κ.The domains are Extraversion, Neuroticism, Agreeableness, Conscientiousness, and Openness.Source: §E.1 · p. 24
• The 240-item HEXACO inventory’s six domain scores were fused into κ.Its domains are Honesty–Humility, Emotionality, Extraversion, Agreeableness, Conscientiousness, and Openness.Source: §E.1 · p. 24
• MGKT domain-level scores were treated as evidence sources and fused into κ.The test contains objective general-knowledge questions spanning multiple domains.Source: §E.1 · p. 24
• For each dataset, κ was compared with the first principal component extracted from the item-level response matrix.Source: §E.1 · p. 24

E.2 Results

• 0.987 Spearman correlation: MGKT’s fused score κ strongly corresponded to the dominant latent factor from item responses.The authors say calibrated fusion almost perfectly reproduces the ranking implied by item-level latent structure.Source: §E.2 · pp. 24–25
• BIG5 showed a moderate relationship between κ and PC1, consistent with its design to represent several personality dimensions.Source: §E.2 · p. 25
• HEXACO showed substantially weaker agreement, consistent with its multidimensional design and conceptually distinct domains.The authors do not expect a single fused score to align strongly with a one-dimensional latent representation.Source: §E.2 · p. 25

E.3 Discussion

• Calibrated fusion and uncalibrated averaging produced nearly identical results because these psychometric sources had comparable scales and response formats.With little scale mismatch, calibration had relatively little work to perform.Source: §E.3 · p. 25
• The authors say calibration behaves sensibly with aligned sources and matters most when heterogeneous sources operate on substantially different scales.Source: §E.3 · p. 25
• These experiments are supplementary evidence, not a direct evaluation of calibration across heterogeneous information sources, because the psychometric sources were already on comparable scales.The MGKT results suggest domain-level fusion can recover latent structure similar to item-level analysis.Source: §E.3 · p. 25