Source-linked AI summary
Intercepting the Kangaroo: Experimental Astrolinguistics with Constructed Lexicons, Active Probing, and Large Language Models as Informants and Hypothesis Proposers
Francesco Cordella, Mauro Cappelli
TL;DR
Astrolinguistics lacks an experimental protocol for identifying referents across fundamentally different category systems, leaving Quinean translation indeterminacy untested. This paper evaluates an active, falsification-based protocol with constructed language-model informants and finds zero undetected mistranslations in the tested regime, while coverage varies with the instruments.
Problem
Astrolinguistics remains a design exercise without an experimental arm, and no protocol has been tested against the problem of silently attaching words to wrong referents.
Method
The study uses incompatible constructed lexicons and an active protocol that probes discriminating scenes, performs recovery, quarantines uncertainty, and verifies proposed rules.
Results
Zero undetected mistranslations persisted across the tested protocol conditions, while coverage varied with recovery, proposer capability, and informant noise.
Takeaways & Limitations
Within the tested regime, correctness is a property of the protocol, whereas coverage is a property of the instruments.
Takeaways & Limitations
The study uses a small 32-scene micro-world and leaves larger ontologies, synonymy, polysemy, graded referents, and hierarchical categories untested.
Abstract
from arXiv · showhide
Astrolinguistics -- communication with minds that categorize reality differently from ours -- has been purely speculative since Freudenthal's Lincos (1960). We make it experimental. Two language models with deliberately incompatible constructed lexicons (one encoding shape, color, and motion; the other fusing color with motion, encoding parity, and lacking shape) serve as informants with complete ground truth, while a fully scripted orchestrator translates between the two category systems. The central failure mode is the kangaroo effect: the silent attachment of a word to the wrong referent -- Quine's indeterminacy of translation, operationalized. Across 400+ simulated and live runs, a protocol combining cross-situational elimination, pre-registered predictive probes, active scene selection, a stricter recovery round, and quarantine produced no undetected mistranslations under the tested conditions and exceeded a passive baseline's coverage (d = 0.62). Injected kangaroo traps defeated naive ostension and pure statistical learning in 100% of runs, while the full protocol intercepted every decoy and, where discriminating evidence is ontologically unavailable, declared Quinean equivalence classes instead of guessing. Under informant noise it degrades gracefully: zero kangaroos persist up to 2% per-word noise; at 10% the protocol predominantly abstains rather than errs. Finally, words outside the scripted hypothesis space (a history-dependent relational term and an XOR contextual homonym) are recovered by a generate-and-test loop in which an LLM proposes rules and the script verifies them: coverage scales with proposer capability (0% -> 18% -> 72% -> 100%) while undetected mistranslations stayed at zero throughout. In the tested conditions, correctness is a property of the protocol; coverage is a property of the instruments.
1. Introduction
This paper turns astrolinguistics from a design exercise into an experimentally testable protocol by simulating incompatible category systems with LLM informants and complete ground truth. Its active, scripted procedure intercepts kangaroo effects under tested conditions, quantifies underdetermination, and preserves zero undetected mistranslations while coverage varies with probing and proposer capability.
- Experimental design: The protocol operationalized incompatible categories by assigning deliberately different constructed lexicons to LLM informants while keeping translation, testing, and verification deterministic.It combines cross-situational elimination, contrastive minimal pairs, preregistered predictive probes, active scene selection, recovery, and quarantine/revocation; the 32-scene world has complete ground truth.
- Empirical contributions: Active probing and recovery together drove undetected kangaroo translations to zero across 50 random worlds while exceeding the passive baseline’s coverage.The mechanisms were synergistic rather than additive: recovery without active selection recovered little and could promote a false translation.
- Failure modes and underdetermination: Injected kangaroo traps defeated naive ostension and pure statistical learning in 100% of runs, whereas the full protocol intercepted every decoy and declared Quinean underdetermination when evidence could not discriminate.This treats equivalence classes as a measurable outcome distinct from both successful translation and failure.
- Open-ended hypothesis generation: Coverage for words outside the scripted hypothesis space rose from 18% to 72% and then 100% across increasingly capable oracles, while undetected mistranslation remained zero.An LLM generate-and-test loop proposes rules, but the deterministic script verifies them.
- Robustness: The zero-kangaroo guarantee survived up to 2% per-word informant noise, after which degradation occurred predominantly through abstention rather than error.The study therefore measures the validity domain and degradation profile instead of assuming robustness.
2. Related work and positioning
The paper builds on cross-situational learning, emergent communication, latent-space alignment, experimental semiotics, and astrolinguistics while reversing the usual referential-game direction. Its novelty is the configuration of fixed, deliberately incompatible lexicons, LLM informants, and evaluated anti-kangaroo machinery.
- Cross-situational learning: The paper’s core learning mechanism is cross-situational learning by hypothesis elimination, grounded in Siskind’s computational formulation and later human-learning confirmation by Blythe (2011).Siskind (1996) formalized intersecting candidate meanings across situations with covering and exclusivity constraints; the paper cites this tradition as foundation, not contribution.
- Emergent communication and referential games: Unlike emergent referential-game communication, the study translates between two pre-existing fixed lexicons and treats the LLM as an informant rather than a learner.This positions the work against emergent-communication studies by Lazaridou et al. (2016), Steels (2015), and Kouwenhoven et al. (2024).
- Latent-space alignment and concept alignment: The broader positioning connects the programme to future representation-space alignment, experimental semiotics, post-Lincos astrolinguistics, and AI-safety work on concept alignment.The cited alignment tools include relative representations by Moschella et al. (2023) and Maiorca et al. (2023), anchor-based cross-lingual mapping by Conneau et al. (2017), and concept-alignment work by Sucholutsky et al. (2023).
- Novelty and positioning: The claimed novelty is the previously unassembled configuration of transversal constructed categories, LLM informants that interpret rather than learn, and anti-kangaroo machinery as the evaluated object.The setup shifts the question from learning a language to translating between different ways of categorizing the world, while making interpretive phenomena experimentally observable.
3. The micro-world and the constructed lexicons
The experimental micro-world contains 32 scenes defined by shape, colour, motion, and number parity, while two deliberately incompatible lexicons reorganize these attributes. The learner must reconstruct the alien lexicon from interaction using a finite hypothesis space and reproducible informant behaviour.
- The world comprises 32 scenes formed by shape, colour, motion, and number, with learners observing number only through parity.Scenes are presented to informants as short natural-language descriptions.
- The learner’s task is to reconstruct each L2 word as a conjunction of attribute–value atoms from interaction alone.
- The two lexicons are categorially incompatible: L1 independently encodes shape, colour, and motion, whereas L2 fuses colour with motion, encodes parity, and omits shape.Table 1 characterizes L2’s categories as transversal to L1’s.
- Nonce word forms are empirically screened for neutrality, and all informant calls use temperature 0 with a fixed lexicon prompt to ensure reproducible rule application.Perfect neutrality is unattainable for models trained on human text, so residual associations are verified rather than assumed away.
- The base hypothesis space contains all conjunctions of one or two atoms over distinct attributes, comprising 32 hypotheses.
4. The anti-kangaroo protocol
The anti-kangaroo protocol combines elimination, contrastive active probing, predictive checks, recovery, and quarantine to prevent premature word–referent commitments. It distinguishes genuine mistranslations from unresolved equivalence classes when evidence cannot discriminate among surviving hypotheses.
- Elimination and minimal pairs: Cross-situational elimination retains hypotheses whose predicted word presence matches each observed utterance, while minimal pairs expose attributes on which surviving hypotheses disagree.With a perfectly faithful informant, elimination cannot remove the true referent; the remaining risk is choosing incorrectly among survivors.
- Probing, promotion, and recovery: Predictive probes require registered forecasts; failures block promotion and feed corrective scenes back into elimination, while recovery re-probes non-promoted anchors and quarantines later failures.Promotion additionally requires a clean round, contrastive evidence, and, from round two, positive evidence rather than absences alone; no reported run left a false translation promoted.
- Active scene selection: Active scene selection targets the largest hypothesis split, expected word presence, and attribute novelty, directing probes toward scenes where rival interpretations disagree.This makes discriminating evidence a deliberate target rather than an accidental consequence of passive observation.
- Outcome classification: The protocol labels promoted anchors CORRECT only when the hypothesis set uniquely contains the true referent, and otherwise reports unresolved status or a KANGAROO when the truth is excluded.When the true referent survives alongside extensionally indistinguishable rivals, the output is their Quinean equivalence class rather than a guess.
- Determinism and reproducibility: Canonical sorting makes set and dictionary iteration deterministic, including the Occam tie-break, while random-baseline comparisons remain valid only under controlled RNG streams.These engineering choices address reproducibility and a previously latent dependence on PYTHONHASHSEED.
5. Experiments
Across simulated and live campaigns, active probing plus recovery increased coverage without undetected mistranslations, while injected traps showed that passive learning can select decoys when evidence is restricted. The protocol instead intercepted decoys or reported underdetermination/abstention, and its fail-safe guarantee persisted under limited noise and expanded hypothesis generation.
- Recovery campaign: 4.72 correct entries per run exceeded the random baseline’s 4.18 (Δ = +0.54, 95% bootstrap CI [0.20, 0.86], Cohen’s d = 0.62) with zero kangaroos in 50 Active + R2 runs.All 33 round-2 promotions were correct, at a cost of ≈6 extra exchanges per run; the live GPT-4o-mini campaign reproduced the simulated outcomes with 100% fidelity.
- Recovery campaign: 12/50 random-arm runs contained at least one kangaroo versus 0/50 in Active + R2 (Fisher exact p = 2.3 × 10−4).The safety of recovery depended on pairing it with active selection: random recovery produced a kangaroo by promoting ZUMU as red ∧triangle instead of red ∧rapid.
- Injected traps: 100% of passive trap-arm runs fell into injected decoys, whereas the full protocol promoted no kangaroos in 200 trap runs.In weak traps it intercepted the decoy in every run; when discriminating scenes did not exist, it declared an equivalence class or abstained rather than guessing.
- Noise robustness: Zero-kangaroo performance survived 2% per-word noise, while at 10% noise the protocol predominantly abstained rather than erred.At 5–10% noise, a small leak appeared because corrupted observations could delete the true referent and let a rival pass probes by chance.
6. Discussion
Across the tested experiments, the protocol’s correctness remained zero surviving promoted-and-wrong translations, while coverage depended on the instruments and proposer capability. The discussion frames the same script-controlled interception architecture as applicable both to synthetic informants and to unreliable LLM-generated hypotheses.
- Limits and underdetermination: Within finite constructed worlds, the kangaroo effect operationalizes extensional Quinean underdetermination, requiring equivalence-class reporting when evidence cannot discriminate.The class-aware metric distinguishes declared underdetermination from guessing.
- LLM roles: The LLM serves first as a synthetic native informant and later as a hypothesis generator, while scripted judgment remains the common control mechanism.Its interpretive behavior under imposed lexicons is data in Sections 3–5.2; in Section 5.3 it proposes rules for spaces not hand-coded by researchers.
- Transferable architecture: The protocol intercepts both wrong referents and wrong proposed rules rather than preventing kangaroos, supporting its transfer to safely harnessing unreliable generative models.The paper characterizes this architecture as a behavioral concept-identity check based on survival under active falsification.
- Relation to XSL: Relative to Siskind’s (1996) eliminative XSL, the base learner adds verification before commitment, active evidence construction, and first-class revocation.These additions address commitment on convergence, passive learning, and corrupted entries that cannot otherwise be repaired.
7. Limitations and future work
The study’s 32-scene micro-world leaves broader referential uncertainty and ontology scaling untested. Its single-shot recovery round also remains limited by residual trap-T1-type cases.
- Scope limitations: The 32-scene micro-world does not test Siskind-scale referential uncertainty, synonymy, polysemy, graded or probabilistic referents, or hierarchical category systems.Channel noise was tested up to 10% per word, but these broader forms of uncertainty remain outside the tested scope.
- Future work: Scaling the ontology is the most direct way to test whether the small world flatters eliminative learners.The current micro-world is small enough that its results may not generalize to larger category systems.
- Recovery limitations: The recovery round is single-shot, and residual trap-T1-type cases indicate that this recovery design remains incomplete.The passage identifies these residuals as a limitation of the current recovery procedure.
Data availability
The study’s code, results, interaction logs, and proposer responses are archived and available from the authors, with SIM results reproducible from seeds at minimal reported API cost.
- Data availability: All code, per-seed CSV results, JSONL interaction logs, and raw proposer responses are archived with Section 5.3 version labels and available from the authors.SIM results are bit-reproducible from seeds.
- Data availability: The reported LLM campaigns cost less than EUR 5 in total, while the noise experiment was SIM-only and free.