Data Analysis for Precision 21 cm Cosmology

Motivation

• The 21 cm line can fill the observational gap between roughly 400,000 and 1.5 billion years after the Big Bang, corresponding to z ∼1,100–3.Existing probes primarily examine the early and late Universe, while intermediate redshifts remain incompletely surveyed.Source: §1 · p. 1

Scientific opportunity

• Redshifted 21 cm emission or absorption traces neutral hydrogen, enabling fully three-dimensional mapping with unprecedented survey volume.The line can also probe matter, ionization state, and temperature because redshift supplies line-of-sight distance information.Source: §1 · pp. 1–2
• 21 cm observations can access extremely high redshifts, integrated faint emission, and finer cosmological scales through readily obtained spectral resolution.Measurements average over many unresolved objects while retaining information from sources too dim for individual detection.Source: §1 · p. 2

Field status

• Recent cross-correlation detections, a tentative z ∼17 global-signal detection, and new upper limits have shifted emphasis toward precision data analysis.Analysis pipelines must meet stringent requirements in calibration, data volume, and systematic-error control, making instruments as much software telescopes as hardware telescopes.Source: §1 · p. 2

Paper scope

• The paper provides a pedagogical introduction to 21 cm analysis, using theory and instrumentation as background for discussions of calibration, mapmaking, power spectra, foreground mitigation, and parameter extraction.It also surveys systematic effects, global-signal measurements, and other measurement types before summarizing conclusions.Source: §1 · p. 2

2.1. 21cm line basics

• 21 cm brightness temperature traces spin-temperature contrast with the CMB and depends on neutral fraction, gas density, and line-of-sight velocities, making it sensitive to astrophysics and cosmology.Emission occurs when Ts exceeds Tγ, while absorption occurs when Ts is lower; observations target either the global signal or spatial fluctuations.Source: §2.1 · p. 4

2.2. Ultra high redshift: the Dark Ages

• The Dark Ages provide a clean probe of linear matter fluctuations before luminous sources form, with vast small-scale mode access for inflation, neutrino-mass, dark-matter, and primordial-physics tests.This opportunity is limited by extremely long wavelengths, bright foregrounds, and ionospheric distortions.Source: §2.2 · pp. 5–6

2.3. High redshift: Cosmic Dawn and the Epoch of Reionization

• Cosmic Dawn and reionization make 21 cm fluctuations sensitive to Lyα coupling, X-ray heating, and clustered ionization, while uncertain astrophysics leaves broad theoretical possibilities.Each process can produce rise-and-fall fluctuation signatures, and measurements can constrain ionized-bubble sizes, reionization timing, and properties of early galaxies.Source: §2.3 · pp. 6–8

2.4. Low redshift: Post-Reionization

• ≲20 meV constraints on the sum of neutrino masses are possible by combining intensity mapping with broadband power-spectrum, BAO, and other-probe measurements.Post-reionization neutral hydrogen traces large-scale structure, enabling wide-volume surveys, BAO expansion-history measurements, and astrophysical constraints on ΩHI(z).Source: §2.4 · pp. 8–9

3. OBSERVATIONAL 21CM COSMOLOGY

• 21 cm instruments generally require high sensitivity, broadband frequency coverage, access to scientifically relevant spatial scales, and temporal stability to measure faint signals amid strong contaminants.High sensitivity motivates large collecting areas and 1000 hrs or more of integration; broadband coverage also improves calibration and systematic diagnosis.Source: §3 · p. 9

3.1. Single dish telescopes or interferometers?

• The single-dish versus interferometer choice depends on the application: interferometers offer high angular resolution and configurable scale sensitivity, whereas single dishes provide more complete uv sampling and avoid interferometric gaps.Single-dish measurements incur a positive noise bias from squaring data, while independent-antenna correlations in interferometers remain mean zero except for the zero-length baseline.Source: §3.1 · pp. 11–12

3.2. From imaging to statistical measurements

• When individual Fourier modes lack signal-to-noise sufficient for imaging, statistical measurements combine modes—typically by isotropic wavenumber—to estimate power as fluctuation variance across lengthscales.Imaging requires reconstructing each mode’s amplitude and phase with roughly unit signal-to-noise or better; power spectra sacrifice image-level information but retain substantial physical information.Source: §3.2 · pp. 12–13
• Power spectra capture much of the relevant physics: during reionization, power shifts from high k to low k as ionized bubbles grow, then vanishes as the neutral fraction approaches zero.Power-spectrum measurements can constrain model parameters and select between models, although the bubble-growth interpretation is not fully rigorous.Source: §3.2 · p. 14
• Cylindrical power spectra separate line-of-sight from perpendicular fluctuations, revealing mode accessibility, foreground structure, and the Fourier-space window relevant to 21 cm measurements.Spherical Fourier-Bessel spectra similarly separate angular scales, while cylindrical spectra use k⊥ and k∥ to isolate transverse and radial modes; interferometer chromaticity produces a foreground wedge.Source: §3.2 · p. 15, §3.3 · p. 16, Figure 5 · p. 17

3.3. Designing an interferometer: Achieving sensitivity to the right modes

• Interferometer design should match the signal-to-noise regime: compact redundant arrays improve low-signal-to-noise sensitivity, whereas less compact arrays provide more independent modes against cosmic variance.Redundant baselines reduce power-spectrum errors as 1/N_bl t through coherent averaging, while independent modes average more slowly as 1/√N_bl t but become advantageous once individual modes reach signal-to-noise of order unity.Source: §3.3 · pp. 17–18

3.4. Costs and systematics in instrument design

• Practical instrument choices trade sensitivity against cost and stability: drift scans can reduce moving parts and systematics, while regular arrays reduce correlator cost from O(N_ant^2) to O(N_ant log N_ant) and enable redundant calibration.Regular arrays also provide repeated baselines whose consistency can be used to solve for calibration parameters, although FFT acceleration for irregular layouts may incur zero-padding overhead.Source: §3.4 · pp. 18–19

3.5. A recipe for estimating sensitivity

• A sensitivity forecast maps baselines into uvη cells, accumulates integration time, combines instrumental and cosmic-variance errors, and inverse-variance averages cells into power-spectrum bins.For drift scans, independent sky patches approximately reduce errors by 1/√N_patch; rigorous curved-sky treatments require spherical Fourier-Bessel or m-mode methods, and realistic noise forecasting must include uvη covariances.Source: §3.5 · pp. 19–20
• The forecasting recipe is approximate because drift-scan patch scaling and assumed sharp uvη coherence boundaries neglect the full curved sky and smoothly varying noise correlations.A rigorous calculation should over-resolve uvη space and propagate covariances rather than treating cells as simply correlated or uncorrelated; the approximate approach is implemented by 21cmSense (Pober et al. 2013b,a).Source: §3.5 · p. 20

Published observations

• Existing experiments have produced EoR and Cosmic Dawn upper limits, while GBT and Parkes have detected post-reionization 21 cm fluctuations through cross-correlations with galaxy surveys.A tentative sky-averaged signal was also reported at z ∼17, and GBT additionally constrained the combination of cross- and autocorrelation measurements.Source: §4 · pp. 20–22

Instrument-specific pipelines

• Instrument design shapes analysis strategy: MWA uses sky-based calibration, hybrid foreground mitigation, and map-based power-spectrum estimation, whereas PAPER uses redundant calibration and direct visibility-based delay-spectrum estimation.LOFAR applies direction-dependent calibration with GMCA or Gaussian Process Regression, while OVRO-LWA uses m-mode map-making, Karhunen-Loève mode projection, and quadratic estimators.Source: §4 · pp. 21–22

Current results

• No 21 cm auto-power-spectrum detection has yet been achieved, although upper limits have become increasingly stringent and new instruments aim for high-significance detections.Figure 7 summarizes current EoR upper limits, but comparisons are imperfect because each experiment’s best limit can occur at a different k value.Source: §4 · p. 22, Figure 7 · p. 23

Next-generation experiments

• HERA is designed for high-significance reionization and Cosmic Dawn power-spectrum detections using a compact, redundantly calibratable 350-dish array with long-baseline outriggers.Its exploded-hexagon layout reduces grating lobes, while the drift-scan design and broadband feed target 50–250 MHz observations.Source: §4 · p. 22

Upcoming experiments

• Upcoming post-reionization surveys span interferometers, dishes, cylinders, and single-dish systems, including CHIME, HIRAX, Tianlai, BINGO, and SKA, with complementary redshift coverage and cross-correlation opportunities.The review’s upcoming-experiment list emphasizes funded or likely-to-be-funded projects whose main science objectives include 21 cm cosmology.Source: §4 · pp. 23–24

5.1. Antenna Response

• Antenna i maps sky emission into a voltage through its frequency- and direction-dependent complex vector beam A_i, encoding sensitivity, polarization response, and relative delay.The beam is operationally defined, but measuring or simulating it is a major challenge for 21 cm cosmology.Source: §5.1 · p. 25

5.2. Sky Emission

• Sky radiation is generally Gaussian, spatially and spectrally incoherent, and described by the coherency matrix, whose Stokes parameters represent total, linear, and circular polarization.Stokes V is usually assumed zero because few astrophysical sources produce circular polarization, while the statistical coherency can remain correlated across frequencies and angles.Source: §5.2 · pp. 25–26

5.3. Interferometry in theory. . .

• Equation (51) exactly relates sky emission, antenna beams, curved-sky geometry, polarization, and measured visibilities, the fundamental observable of interferometry.For Gaussian sky emission and linear antenna measurements, visibilities contain all information in the signals’ covariance matrix.Source: §5.3 · p. 26

5.4. . . .and in practice

• Interferometers separate nanosecond, roughly 10 µs, and roughly 1 s timescales, enabling frequency channelization by the F-engine and covariance estimation by the X-engine.The F-engine transforms time samples into frequency channels, while the X-engine averages statistically equivalent samples to form visibilities.Source: §5.4 · p. 27

5.4.1. Analog Chain

• The analog chain amplifies and filters antenna voltages while minimizing added noise, but signal transport can introduce systematics.A low-noise amplifier raises the signal above later thermal noise, and unwanted frequencies are removed before digitization.Source: §5.4.1 · p. 27

5.4.2. F-engine

• The F-engine digitizes and channelizes time streams, using Nyquist, mixing, or alias sampling and Fourier-based methods to trade spectral resolution, response localization, sensitivity, and computation.Modern systems commonly use Polyphase Filter Banks implemented on FPGAs, improving spectral localization while introducing computational costs and slight temporal correlations.Source: §5.4.2 · pp. 27–28

5.4.3. X-engine

• The X-engine transposes frequency-ordered spectra into input-ordered data before cross-correlating all inputs to estimate visibilities.This transpose is a major large-array bottleneck requiring fast networks and memory, while correlation scales as ∼ΔνN^2 operations per second and is parallelizable.Source: §5.4.3 · p. 28
• For typical interferometers, averaging M ≳ 10^5 spectra produces visibility estimators whose noise is approximately Gaussian, with distinct visibilities uncorrelated in the standard noise-dominated regime.The estimator’s exact noise description follows from its covariance, while the large number of averaged spectra justifies the Gaussian approximation.Source: §5.4.3 · pp. 28–29

6. INSTRUMENTAL SYSTEMATICS

• Instrumental systematics must be constrained below the 21 cm signal-to-foreground ratio, often around ≲10^-4, because undetermined effects proliferate foreground modes and bias results.Known effects can be modeled or removed, but doing so may be computationally challenging and can reduce sensitivity.Source: §6 · p. 29

6.1. Gain Variations

• Time-dependent, partially independent gain variations limit combining baselines and frequencies, while cross talk requires complex-matrix rather than scalar corrections.Thermal environments drive difficult-to-predict gain changes, and unknown gains can bias maps and foreground removal.Source: §6.1 · pp. 29–30

6.2. Primary Beams

• Frequency-dependent, spatially complex, polarization-sensitive primary beams vary across feeds and array positions, imposing technique-dependent requirements on beam knowledge, smoothness, and polarization purity.Feed-to-feed variations can introduce non-redundancy and contaminate redundant-calibration gain solutions.Source: §6.2 · p. 30

6.3. Polarization leakage

• Polarization leakage can misestimate total intensity and create spurious spectral structure, with Q leakage caused by beam-amplitude mismatch and U leakage by real beam non-orthogonality.Stokes V can leak into I through polarization delay, but is often used as a noise proxy when calibration errors and intrinsic circular polarization are limited.Source: §6.3 · pp. 30–31

6.4. Noise Systematics

• Correlated noise between closely packed antennas biases visibilities, especially on short baselines, and average-subtraction methods require stable noise while accounting for removed average sky signal.Noise coupling decreases with feed separation, but environmental changes can make the subtraction imperfect.Source: §6.4 · p. 31

6.5. A summary of interferometry

• A visibility is approximately a Fourier mode of the sky, but this interpretation depends on interferometric approximations and is modified by instrumental systematic challenges.These assumptions and systematics inform the analysis methods developed later in the paper.Source: §6.5 · p. 32

Foreground severity

• Foregrounds are ∼10s to 100s of Kelvin at 408 MHz and stronger at lower frequencies, vastly exceeding the ∼10s of mK EoR and ∼0.1 mK low-redshift signals.Galactic synchrotron dominates, while free-free emission and radio point sources remain important contaminants.Source: §7 · p. 32, Figure 8 · p. 33

Why 21 cm foregrounds are difficult

• Unlike CMB observations, 21 cm frequency channels each contain unique cosmological information, so foreground separation is underdetermined and requires sequestering contamination into limited modes.With N channels, there are N cosmological degrees of freedom and up to N foreground degrees of freedom, leaving too few measurements for self-consistent separation.Source: §7 · pp. 33–34

Spectral-mode mitigation

• Spectral smoothness allows slowly varying foreground modes to be discarded while rapidly varying cosmological modes are retained, but this assumption may fail at the precision required.Empirical and physical models generally support spectrally simple foregrounds, while radio recombination lines are unlikely to be bright enough to matter.Source: §7 · p. 34

Polarization complications

• At frequencies around 300 MHz, a rotation measure up to 100 radm−2 corresponds to a full polarization-vector rotation within ∼10 MHz, producing strong polarized-intensity spectral variation absent from total intensity.Instrumental polarization leakage may irreversibly contaminate observed total intensity with spectral structure from polarized emission, limiting foreground removal.Source: §7 · p. 34

Instrumental complications

• Instrumental effects can cause foreground mode proliferation, making the interaction between foregrounds and systematics the chief unresolved obstacle to 21 cm cosmology.The required calibration precision is driven by the large foreground-to-signal dynamic range, and near-term progress depends on addressing this problem.Source: §7 · p. 34

• The pipeline seeks to undo the transformation and corruption that convert cosmic radiation into contaminated interferometric visibilities, recovering scientific information encoded in the 21 cm line.Source: §8 · pp. 34–35
• Subsequent sections cover interferometer calibration, sky mapping, and power spectrum estimation, with power spectra obtainable with or without preceding mapmaking.Source: §8 · p. 35
• The roadmap then addresses foreground mitigation, constraints on theoretical models from power spectra, and statistics beyond the power spectrum.Source: §8 · p. 35

9. CALIBRATION

• Calibration solves for unknown complex antenna gains to recover true visibilities, but the unconstrained measurement system has more unknowns than measurements.With Nant antennas, the number of true visibilities plus gain factors exceeds the available baseline measurements, making the fully general problem mathematically impossible without additional assumptions.Source: §9 · p. 35

9.1. Calibration by fitting to a sky model

• A restricted sky model makes calibration solvable, although incomplete catalogs and diffuse emission can imprint artificial spectral structure and motivate long-baseline or pulsar-gated strategies.Single-source calibration is viable when the source is isolated and unresolved; more realistic models improve solutions, but missing faint sources and diffuse Galactic emission remain important limitations.Source: §9.1 · pp. 35–36

9.2. Calibration using self consistency

• Redundant calibration exploits repeated baselines to solve gains and unique visibilities with limited sky assumptions, but leaves four single-polarization degeneracies that require minimal sky modeling.The unconstrained modes are global amplitude, global phase, and two linear phase gradients; additional degeneracies arise when polarizations are treated independently.Source: §9.2 · p. 37

9.3. Calibration using hybrid techniques

• Hybrid methods combine sky information with self-consistency and can model partial baseline correlations, while CorrCal includes redundant calibration as a limiting case and relaxes perfect redundancy.CorrCal encodes redundancy, sky information, and partial correlations in a covariance matrix; taking α →∞ recovers perfect redundant calibration, whereas reduced off-diagonal correlations represent imperfectly redundant baselines (Sievers 2017).Source: §9.3 · pp. 37–38

9.4. Calibration solutions as a function of frequency and time

• Frequency- and time-dependent calibration must avoid overfitting noise or sky-model errors, because unsmooth solutions can contaminate foreground mitigation; smoothness can be imposed jointly, distributed, or through post-calibration filtering.Strategies include downweighting chromatic long baselines, simultaneous multi-frequency CorrCal, distributed SAGECal-CO optimization (Yatawatta et al. 2008), and Wiener-filter smoothing.Source: §9.4 · pp. 39–40
• Smooth calibration solutions should match the instrument’s actual frequency dependence and temporal stability rather than impose smoothness unconditionally.If the instrumental response is complicated in frequency, calibration should inherit that complexity.Source: §9.4 · p. 40
• For experiments setting upper limits, imposing calibration smoothness is conservative because residual spectral structure generally raises the measured power spectrum.The resulting power increase is conservative only when interpreted as an upper limit, not as a 21 cm detection.Source: §9.4 · p. 40
• A 21 cm detection requires extensive end-to-end simulations to establish that calibration errors have not biased the power spectrum.Ideally, simulations should capture covariance between calibration and power-spectrum errors so calibration uncertainty can be marginalized consistently by inflating the latter.Source: §9.4 · p. 40

10. MAP-MAKING

• Map-making is essential for interpreting visibility data but is not required for further analysis, and proceeding directly from visibilities can sometimes avoid map-related problems.The predominant radio-astronomy approach uses CLEAN variants, although the section first develops linear map-makers because CLEAN has problematic properties for 21 cm analysis.Source: §10 · pp. 40–41

10.1. Linear Map-makers

• Linear map-making statistically inverts the noisy, generally non-invertible forward model, with Wiener filtering suppressing low-signal-to-noise modes that otherwise amplify noise.Maximum-likelihood inversion is unbiased and optimal but can severely upweight noise-dominated modes below the telescope’s resolution limit; Wiener filtering regularizes this tradeoff using a sky covariance prior.Source: §10.1 · pp. 41–42

10.2. CLEAN

• CLEAN iteratively removes point-source components from dirty images, making it effective for point sources but less effective for diffuse Galactic synchrotron and 21 cm emission.Its nonlinear iteration makes output covariance difficult to derive, often requiring Monte Carlo simulations.Source: §10.2 · p. 43

10.3. Noise Covariance

• Map-based analyses require covariance matrices, but CLEAN’s nonlinear processing makes noise covariance difficult to assess, while visibility-space noise is typically more compactly represented.Map-making correlates noise among pixels, whereas visibility-space correlations are often diagonal or short-ranged across frequencies, times, and baselines.Source: §10.3 · p. 43

10.4. Practicalities of map-making

• Practical map-making must address billion-scale pixel counts, telescope observing strategy, and mixed skies containing both point sources and diffuse emission.FFT-based operations and conjugate gradients can mitigate linear-map computational costs, while mosaicking or m-mode methods combine overlapping pointings.Source: §10.4 · p. 43

10.5. Map-making with m-modes

• The m-mode formalism enables computationally efficient wide-field polarimetric imaging for drift-scan telescopes, trading generality for efficiency.It applies only to drift scans, which are predominant among upcoming 21 cm experiments.Source: §10.5 · p. 43

10.5.1. Introduction

• For transit telescopes, each visibility Fourier mode maps exactly to the same spherical-harmonic m order, yielding an exact polarized sky-to-visibility relation without summing over m.The formalism represents polarization using T, E, B, and V components and assumes approximate sidereal periodicity after averaging over days.Source: §10.5.1 · p. 44

10.5.2. Application to map-making

• In m-mode map-making, finite spherical-harmonic ranges determined by array dimensions make the sky and visibility degrees of freedom finite vectors and matrices, enabling Wiener filtering.The relevant limits scale as l ≲ 2πDmax/λ and m ≲ 2πDEW/λ.Source: §10.5.2 · pp. 44–45
• For drift-scan telescopes, m-mode map-making is essentially exact, naturally wide-field, and fully polarized, while keeping large-matrix operations tractable.These advantages contrast with linear map-makers whose large matrices may be impractical to invert or decompose.Source: §10.5.2 · p. 45
• When noise and sky covariances are diagonal in m, the map-making equations can be evaluated independently for each m.Noise becomes diagonal in m for stationary correlations, while Gaussian-random-field sky statistics provide a useful approximation for the 21 cm field but may fail for Galactic foregrounds.Source: §10.5.2 · p. 45
• For many instruments with mmax ≳ 10^3, m-mode decomposition saves ≳10^6 in computation for matrix-matrix operations and reduces matrix-vector costs by O(mmax).The savings arise from applying operations m-by-m, with matrix-vector multiplication reduced by O(mmax) and matrix-matrix operations by O(mmax^2).Source: §10.5.2 · p. 45
• At low frequencies, nonstationary sky-dominated noise makes the estimator non-optimal rather than biased, and computational savings may outweigh its increased noise.Neglecting stationarity can enable more thorough analyses despite noise exceeding the theoretical optimum.Source: §10.5.2 · p. 45

11. POWER SPECTRUM ESTIMATION

• Quadratic estimators represent discretized power-spectrum bandpowers through data vector x and analyst-chosen matrices E_α, with covariance C linearly related to the signal bandpowers.The framework generalizes beyond P(k) to angular, cross-angular, and spherical Fourier-Bessel two-point statistics.Source: §11 · pp. 46–47
• Cross-multiplying two datasets with identical sky contributions but independent noise removes the additive noise bias that otherwise appears because power-spectrum estimation squares the data.The method can split repeated observations, such as odd- and even-numbered days, provided the noise is uncorrelated between datasets.Source: §11 · p. 47
• The minimum-variance quadratic estimator is guaranteed to have the smallest possible error bars when the covariance statistics are known, using inverse-covariance weighting of the data.Different invertible normalization matrices preserve information while trading off window-function width, error correlations, and error-bar size.Source: §11 · pp. 50–51
• Under the unbiased-estimator constraint, choosing M = F^-1 makes the quadratic power-spectrum estimator optimal, with error covariance V = F^-1 and no estimator having smaller error bars.This choice gives W = I, producing spike-like window functions concentrated at one k bin.Source: §11 · p. 50
• Narrower window functions generally produce larger errors, while broader windows reduce errors but mix neighboring k bins; normalization choices also determine whether bandpower errors are correlated.An F^-1/2-based normalization provides intermediate window widths and error bars with uncorrelated errors.Source: Figure 10 · p. 49, §11 · pp. 50–51
• Because 21 cm fluctuations are non-Gaussian, Gaussian-based covariance formulas may misstate errors and the nominally optimal estimator may lose strict optimality, although correct normalization remains possible.The power spectrum also omits non-Gaussian information, including features visible in the brightness-temperature distribution during Cosmic Dawn and the EoR.Source: §11 · pp. 45, 51, Figure 9 · p. 46
• Power-spectrum estimation is computationally prohibitive for raw interferometric data, motivating data-volume reductions and specialized estimators while empirical covariance modeling risks signal loss.A 300-element interferometer can produce roughly 50,000,000 data values per one-second integration, while the covariance matrix is correspondingly much larger and must be inverted.Source: §11 · pp. 51–52

11.1. Power spectrum estimation via map-making

• Map-making provides lossless compression into configuration, harmonic, or hybrid spaces, but power-spectrum estimation still requires covariance propagation and approximations that can neglect strong mode correlations.Noise is commonly assumed diagonal but non-white in frequency, while nearby angular modes and sky facets can remain correlated, especially for regular antenna layouts.Source: §11.1 · pp. 52–53

11.2. Power spectrum estimation via delay spectra

• Delay spectra square frequency-transformed single-baseline visibilities to estimate power, offering foreground diagnostics and reduced calibration requirements but losing cross-baseline and coherent time-averaging sensitivity.The estimator is approximate because baselines drift across k⊥ with frequency; “square then average” errors scale roughly as 1/√Nt, versus roughly 1/Nt for coherent “average then square” processing.Source: §11.2 · pp. 54–56

11.3. Power spectrum estimation via m-mode analysis

• m-mode analysis makes drift-scan data computationally tractable because independently solvable m modes decouple, simplifying direct visibility-based foreground subtraction while requiring matching block-diagonal covariance structures.The independence follows from time Fourier transformation: each Vm depends only on spherical-harmonic coefficients with the same m index, although power-spectrum estimation requires additional assumptions for C and Qα.Source: §11.3 · pp. 56–57
• Under statistical isotropy and stationary noise, m-mode covariance matrices decouple, preserving computational savings by estimating the power spectrum separately for each m mode and averaging.Foreground anisotropy couples m modes, but applying the resulting non-optimal weighting still produces unbiased estimates, ⟨p̂⟩ = p, provided covariance modelling is not empirical.Source: §11.3 · pp. 57–58, §11.4 · p. 58

11.4. Bayesian power spectrum estimation

• Bayesian inference applies Bayes’ theorem to combine priors and likelihoods, with MCMC sampling the posterior when brute-force evaluation becomes infeasible in high dimensions.Marginalization over nuisance parameters is implemented by integrating them out, or equivalently ignoring them when constructing MCMC histograms.Source: §11.4 · pp. 58–59
• The interferometric Bayesian formulation jointly infers Fourier coefficients and bandpowers from visibility data, assuming Gaussian visibility noise and a linear transformation from coefficients to visibilities.The final power-spectrum posterior is obtained by marginalizing over the Fourier coefficients; the simplified Sims et al. (2019) formulation omits additional parameters for fluctuations beyond the survey volume.Source: §11.4 · p. 59
• Bayesian estimation yields full probability distributions for power-spectrum bandpowers and can incorporate foreground models for statistically disciplined uncertainty treatment.Sims et al. (2019) use Hamiltonian Monte Carlo or analytic marginalization over Fourier coefficients, while Sims & Pober (2019) unify power-spectrum estimation with foreground mitigation.Source: §11.4 · p. 59

12.1.1. Parametrized fits

• Parametrized smooth-spectrum fits can fail on fine angular scales because sparse baseline coverage and interferometric chromaticity imprint unsmooth spectral features.Fitting in uv space can partially alleviate this by assigning zero weight where measurements are absent, but does not remove the underlying chromaticity; log-log fits are also complicated by potentially non-positive interferometric spectra.Source: §12.1.1 · p. 60

12.1.2. Non-parametric fits

• Non-parametric methods fit foregrounds using smoothness criteria, coherence-length differences, or Gaussian-process covariances rather than fixed spectral functions.These approaches include curvature-penalized Wp smoothing, frequency-averaged uv templates, and Gaussian processes whose foreground kernels are assigned longer coherence lengths than the 21 cm signal and can be learned from data.Source: §12.1.2 · pp. 60–61

12.1.3. Mode projection

• Mode projection removes selected foreground-dominated components, including bright point sources and data-derived PCA modes, but risks cosmological-signal loss and depends on regularization or covariance models.KL modes improve protection against signal loss by ordering components by signal-to-contaminant ratio, although the number of modes and covariance models remain difficult to choose, especially for reionization experiments.Source: §12.1.3 · pp. 62–63

12.1.4. Mode weighting

• Mode weighting preserves information by downweighting contaminated modes through inverse-covariance weighting rather than projecting them out completely.It can improve constraints when foregrounds and signal are comparable, but inaccurate covariance models can destroy cosmological signal, making clean projection preferable for initial detection-era experiments unless specific modes must be recovered.Source: §12.1.4 · pp. 63–64

12.1.5. Avoiding foregrounds in the foreground wedge

• Foreground avoidance excludes contaminated Fourier-space regions, but interferometric chromaticity causes intrinsically smooth foregrounds to leak beyond low k∥ modes through mode mixing.The resulting contamination is especially important at high k⊥ and low k∥, where foreground power is inadvertently mixed into measured power spectra.Source: §12.1.5 · pp. 64–65
• Foregrounds form a robust wedge with contamination decreasing toward higher k∥, enabling measurements outside it but sacrificing modes and potentially biasing redshift-space analyses.Leakage beyond the wedge can arise from non-flat spectra, calibration errors, and instrumental effects; hybrid subtraction-and-avoidance strategies remain an open question.Source: §12.1.5 · pp. 65–67

12.1.6. Decorrelating the foreground wedge

• Wedge decorrelation can reduce high-k foreground contamination, but numerical instability, nonstationary foregrounds, and sparse-array uv coverage limit complete removal.Matrix methods can isolate the wedge through triangular window functions, while map-level inversion requires sufficiently dense baseline coverage or a suitable instrument.Source: §12.1.6 · p. 68

12.1.7. Cross correlations

• Cross-correlating independent tracers removes foreground bias in expectation, but finite survey areas and foreground contributions to variance still affect errors.Such measurements have produced the only detections of cosmological 21 cm fluctuations to date, while future [CII]-HI cross-correlations could complement HI auto-power spectra.Source: §12.1.7 · pp. 68–69

12.2. Interloper lines

• Interloper lines can mimic target emission at different redshifts; mitigation includes masking sources, matching companion lines, external cross-correlations, and fitting predictable anisotropies.They are generally negligible for HI 21 cm mapping but can seriously affect other intensity-mapping lines, such as [CII] contaminated by CO transitions.Source: §12.2 · p. 69

12.3. Ionosphere

• Ionospheric absorption, scintillation, refractive shifts, and Faraday rotation can bias 21 cm power spectra, especially during strong ionospheric activity.Direction-dependent calibration, polarized calibration, suitable array configurations, wedge-based mitigation, and averaging multiple observing days can suppress these effects.Source: §12.3 · pp. 70–71

12.4. Radio Frequency Interference

• Radio-frequency interference from broadcasts, communications, transmitters, meteor reflections, and telescope electronics can be orders of magnitude brighter than astrophysical signals and must be mitigated.Remote or radio-quiet sites reduce exposure, but protection is frequency-dependent and may not suit all 21 cm observing epochs.Source: §12.4 · p. 71
• RFI mitigation progresses from threshold flagging of time or frequency outliers to neighborhood-aware, morphological, PCA-based, and machine-learning methods.Morphological approaches reduce false positives from overly aggressive thresholds, while PCA can identify persistent RFI modes and source locations.Source: §12.4 · pp. 71–72
• Neural-network RFI flaggers are generally competitive with state-of-the-art algorithms and much faster after training, but often miss single-pixel time-frequency blips.They use waterfall plots of visibility amplitude and phase and remain an emerging approach likely to improve with further development.Source: §12.4 · p. 73

12.5. Signal loss

• Signal loss can bias 21 cm power-spectrum estimates low when foreground or calibration models absorb cosmological-signal modes.Oversubtraction has caused revised upper limits, and frequency-dependent calibration can fit out spectrally fluctuating cosmological signals.Source: §12.5 · pp. 73–74
• Signal-loss corrections require transfer-function simulations that include correlations between cosmological signals and data-derived foreground models, although inaccurate low-frequency models complicate interpretation.Signal injection into real data is an alternative, while algorithms intrinsically less susceptible to signal loss are preferred when simulations are unreliable.Source: §12.5 · p. 74
• Bayesian joint foreground and power-spectrum estimation represents foreground–signal degeneracies through broader, potentially asymmetric uncertainties rather than treating the effect as lost signal.Marginalizing over foreground parameters can produce an unbiased power spectrum in principle, provided posterior summaries do not misrepresent asymmetric distributions.Source: §12.5 · p. 74
• End-to-end simulations and comparisons among analysis pipelines are needed to validate calibration, foreground suppression, mapmaking, and power-spectrum estimation against signal loss.Careful validation is crucial because calibration errors, as well as foreground mitigation, can suppress the cosmological signal.Source: §12.5 · p. 74

13. POWER SPECTRA TO PARAMETERS/SCIENCE

• Bayesian inference compares measured bandpowers with theoretical predictions to sample posterior distributions of physical-model parameters, typically using MCMC with a Gaussian likelihood.Non-Gaussian power-spectrum errors can instead be incorporated through the full power-spectrum posterior.Source: §13 · p. 75, §13.1 · p. 75

13.1. Light cone effects

• Light-cone evolution breaks translation invariance, making the power spectrum incomplete; cross-frequency spectra, wavelets, or narrower redshift subbands provide possible treatments.Cross-angular spectra Cℓ(ν, ν′) retain frequency-direction correlations, while small bandwidths can justify evolution-averaged power spectra despite possible effects on upper-limit interpretation.Source: §13.1 · pp. 75–76

13.2. Computationally expensive model predictions

• Emulators replace slow theoretical evaluations with fast interpolations, enabling MCMC forecasts and analyses spanning astrophysical and cosmological parameters.This is important because reionization and Cosmic Dawn simulations or semi-analytic predictions can require hours per parameter-space evaluation, and accurate constraints may require cosmological parameters alongside astrophysical ones.Source: §13.2 · p. 76, Figure 14 · p. 77

13.3. Uncertainties in the underlying theoretical models

• Uncertain theoretical frameworks motivate flexible models or Bayesian model selection, with evidence ratios favoring models according to their evidences and prior plausibilities.Bayesian evidence has forecast sensitivity to distinguish reionization-model classes, select foreground parameterizations, and evaluate signal, foreground, and systematic models in real global-signal data.Source: §13.3 · pp. 76, 78

14.1. Cross correlations

• Cross-correlations with external tracers require overlap in sky, redshift, and Fourier-space scales, while foreground mitigation can remove shared modes and reduce signal-to-noise.Candidate tracers span galaxy surveys, CO, [CII], Lyman-α, Hα, [OII], and [OIII] maps across different redshifts.Source: §14.1 · pp. 78–79

14.2. Higher order statistics

• Higher-order statistics access information absent from Gaussian power spectra when nonlinear evolution, lensing, or intrinsically non-Gaussian 21 cm fields couple Fourier modes.Tidal reconstruction can recover large-scale modes, while lensing estimators require accounting for correlated source planes and intrinsic mode-coupling biases.Source: §14.2 · pp. 79–80
• Bispectrum studies predict favorable integration-time scaling, possible parameter-degeneracy breaking, and reionization sign changes, but measurements remain sensitive to foregrounds and calibration.Bispectrum phases, or closure phases, are immune to per-antenna gain calibration and may permit foreground separation through delay-spectrum analysis.Source: §14.2 · pp. 80–81

14.3. Probability distribution functions

• Voxel PDFs capture non-Gaussianity compactly, but foreground convolution makes them difficult to interpret because foreground variations can reach hundreds of kelvin versus millikelvin cosmological fluctuations.Two-point and difference PDFs recover some spatial correlation information, while multiple tracers may enable foreground deconvolution; skewness and kurtosis remain interpretable with forward models after filtering.Source: §14.3 · p. 81

14.4. Imaging

• Images preserve spatial information but generally have lower signal-to-noise than binned statistics; they can contextualize galaxies, identify extreme structures, and predict ionization probabilities despite lost wedge modes.CNNs have recovered simulated theoretical and phenomenological parameters and performed model selection, but foregrounds and systematics must be addressed before robust real-data application.Source: §14.4 · pp. 81–83

15. GLOBAL SIGNAL MEASUREMENTS

• The global 21 cm signal is the mean brightness temperature versus redshift averaged over all sky directions, representing the independent (ℓ, m) = (0, 0) monopole mode.An example theoretical prediction appears in the middle panel of Figure 1.Source: §15 · p. 83

The low redshift (z ≲6) global signal

• At low redshift, the unknown global-signal normalization perfectly degenerates with fσ8, but mildly nonlinear information or multi-tracer surveys can break this degeneracy and enable competitive constraints.The degeneracy arises because fσ8 and T_b^2 b_HI enter multiplicatively; higher-order terms depend on higher powers of fσ8 while the global signal remains an overall normalization.Source: The low redshift (z ≲6) global signal · p. 83

15.1.2. The high redshift (z ≳6) global signal

• At high redshift, the global signal probes Cosmic Dawn phenomena: reionization tracks the ionization history, while earlier evolution probes X-ray heating and Lyα coupling.During reionization, the signal approaches zero with the neutral fraction; at higher redshifts, it traces energy injection into the intergalactic medium.Source: §15.1.2 · p. 83

15.2. Measuring the global 21 cm signal in practice

• Global-signal experiments use wide-beam single antennas and coarse channels because they measure gradual spectral evolution rather than spatial fluctuations, reaching low noise in less than a day.Typical frequency resolution is ∆ν ∼1 MHz for global experiments versus ∆ν ∼50 kHz for Cosmic Dawn fluctuation experiments.Source: §15.2 · pp. 83–84
• Global-signal measurements require stringent control of foregrounds, RFI, ionospheric effects, beam chromaticity, cable reflections, environmental conditions, and single-element noise bias.Calibration combines laboratory and in situ measurements with principal-component modeling, while experiment design must avoid proliferating nuisance calibration parameters.Source: §15.2 · pp. 84–85
• Foreground separation is especially difficult during reionization because the smoothly decreasing cosmological signal resembles spectrally smooth foregrounds, motivating Cosmic Dawn targets with non-monotonic absorption features.Spatial avoidance, correlation modeling, polarization modulation, and velocity-induced dipoles provide additional foreground-rejection strategies, though spectral methods remain necessary.Source: §15.2 · pp. 84–85

15.3. The current status of global signal experiments

• SARAS ruled out rapid reionization for 6 < z < 10 when dT_b/dz < 114 mK, while EDGES constraints excluded ∆z ≲1 at z_mid ∼8.5 and ∆z ≲0.4 for 6.5 < z_mid < 14.8.EDGES also reported a 78 MHz absorption feature with potentially major implications for Cosmic Dawn, but instrumental and methodological controversies remain and independent experiments are testing the claim.Source: §15.3 · pp. 85–86

16. CONCLUSION

• 21 cm cosmology has shifted from a data-starved field to experimental reality, with many experiments possessing sufficient sensitivity to detect and characterize the cosmological signal.Greater signal-to-noise also enables more incisive diagnosis and mitigation of systematics.Source: §16 · p. 86
• The central challenges are extreme sensitivity, foreground mitigation, and systematic control, whose interactions can irretrievably contaminate additional Fourier modes through the foreground wedge.Interferometric observations make foregrounds less spectrally smooth than their intrinsic sky emission, worsening contamination beyond intrinsic foreground expectations.Source: §16 · p. 86
• Calibration, map-making, power spectrum estimation, foreground mitigation, and parameter estimation are inextricably linked to one another and to telescope hardware.This interdependence applies to both spatial-fluctuation experiments and global-signal measurements.Source: §16 · pp. 86–87
• 21 cm experiments are effectively software telescopes, requiring careful hardware-aware analysis pipelines for successful observations.The conclusion applies the software-telescope perspective to both instrumental design and data analysis.Source: §16 · p. 87
• Data analysis remains an unsolved problem because the community has no consensus on the optimal path from raw radio-telescope data to the science enabled by redshifted 21 cm measurements.Multiple highly sensitive experiments nevertheless provide testbeds for the field’s broad landscape of analysis ideas.Source: §16 · p. 87
• Future progress is expected to test tentative global-signal detections, lower upper limits on 21 cm fluctuations across redshifts, and ultimately deliver robust detections and high-significance measurements.These outcomes would enable cosmological measurements probing the Universe across space and time with unprecedented precision and accuracy.Source: §16 · p. 87
• The authors acknowledge contributors, editorial and referee assistance, institutional hospitality, and support from multiple research grants and programs.Acknowledged support includes NSF Grant No. 1836019, the New Frontiers in Research Fund, NSERC, CIFAR, and Berkeley research institutions.Source: §16 · p. 87

A. DRIFT SCAN OBSERVATIONS

• Drift scans measure a fixed set of Fourier modes, directly distinguishing u modes within the primary-beam footprint, while tracking observations trace an elliptical uv-plane path through Earth rotation.Drift scans gain access to individual azimuthal m-modes over a full sidereal day but do not expand their overall Fourier-mode area; tracking expands mode coverage but not resolution beyond the primary beam.Source: A. DRIFT SCAN OBSERVATIONS · pp. 89–90, Figure 16 · p. 90

B. EXPLICIT QUADRATIC ESTIMATOR EXAMPLES

• The quadratic-estimator examples connect data covariance and bandpower responses to inverse-covariance-weighted Fourier transforms, squared and integrated over the target Fourier-space bin.This formalism accommodates map voxelization, complicated pixelization, and contaminant or noise contributions alongside the cosmological signal.Source: §B.1 · pp. 91–92

B.1. Measuring P(k) from a three-dimensional configuration-space map

• For a three-dimensional configuration-space map, the optimal estimator is obtained by inverse-covariance weighting the data, Fourier transforming it, and summing squared modes within each k-space voxel.The derivation assumes a piecewise-constant power spectrum within each Fourier-space voxel and uses fine voxel resolution when applying the sinc approximation.Source: §B.1 · pp. 91–92

B.2. Measuring P(k⊥, k∥) or P(k)

• Power spectra can be binned as P(k⊥, k∥) for diagnosing systematics or as scalar P(k) on spherical Fourier-space shells for final cosmological results.Both choices follow directly by redefining the Fourier-space bandpower regions in the quadratic-estimator expressions.Source: §B.2 · p. 92

B.3. Measuring P(k) from a Fourier space map

• For Fourier-space maps, power estimation is approximately the inverse-covariance-weighted square of the component at the desired wavevector when the survey volume is large.The full quadratic-estimator matrix is required to treat finite survey volumes exactly.Source: §B.3 · p. 92

B.4. Measuring Cℓ(ν) from an angular sky map

• The same quadratic-estimator framework measures angular Cℓ by inverse-covariance weighting a sky map, taking spherical-harmonic transforms, squaring coefficients, and summing over m.Cross-frequency Cℓ(ν, ν′) replaces the single-frequency square with multiplication of spherical-harmonic coefficients from different frequencies; incomplete sky coverage can couple ℓ modes.Source: §B.4 · p. 93

B.5. Measuring a power spectrum from visibilities

• A single-baseline visibility measurement can be recast in comoving coordinates as a generalized pixelization of the sky, enabling a direct quadratic power-spectrum estimator.This construction uses the narrow-field, flat-sky approximation and represents the visibility data as a vector of frequency-channel measurements.Source: §B.5 · p. 93
• Neglecting the frequency dependence of all terms except the channel profile γ_i(ν) yields the delay-spectrum estimator.Source: §B.5 · p. 94
• Retaining frequency dependence in the complex exponential incorporates foreground-wedge phenomenology into the visibility-based estimator, as worked out in Liu et al. (2014a).Source: §B.5 · p. 94