Source-linked AI summary
High-Value Synthetic Supervision for Parameter-Efficient Adaptation of a Compact Japanese Speech Model
Sidi Chang, Peiying Zhu
TL;DR
Scarce redistributable data makes specialization of compact models for private speech difficult, and Japanese care handoffs require mapping spoken language into structured notes. The paper adapts a compact Japanese audio model using provenance-linked synthetic examples; full tuning and LoRA achieve large paired gains over the same base. The evidence supports a narrow synthetic speech-to-structure capability, not clinical validity or measured device performance.
Problem
The paper asks whether a compact audio-native model can learn Japanese care-handoff speech-to-structure from a few hundred synthetic examples despite scarce redistributable private-domain speech data.
Source: §1 · p. 1Method
Results
Full tuning reaches 0.8664 and LoRA 0.8461, with paired gains over the same base of +0.8164151 and +0.7961004, respectively.
Source: §4 · p. 2Takeaways & Limitations
A compact model can acquire a narrow speech-to-structure capability from synthetic clips, while the evidence does not establish clinical validity or measured on-device performance.
Source: §6 · p. 3All training and evaluation speech is synthetic, references are model-proposed, the judge is uncalibrated, and device execution metrics were not measured.
Source: §5 · p. 3
Abstract
from arXiv · showhide
Private domain speech is difficult to collect and redistribute, while compact models need task-specific supervision. We study an auditable synthetic pipeline that maps Japanese care handoffs directly to six-field structured notes. Using 182 synthetic training and development clips, we adapt a 1.47B audio model by full fine-tuning and rank-16 LoRA. On a 39-clip scenario-seed-disjoint synthetic test, an unadapted model obtains a model-judged factuality-recall score of 0.0500, full tuning 0.8664, and LoRA 0.8461. LoRA reaches 97.7% of the full-tuning aggregate as a descriptive ratio while project telemetry reports 12.4M trainable parameters, about 0.85% of the backbone. Both adaptations show large paired gains over the same base; the full-versus-LoRA interval crosses zero, and differing optimization settings preclude an equivalence claim. This is a parameter-efficient capability-acquisition result, not a device-performance result: latency, memory, energy, and real-time factor were not measured. All evaluation speech and targets are synthetic, references are model-proposed, and the judge is uncalibrated. The evidence shows that a compact model can acquire a narrow audio-to-structure transformation from a few hundred provenance-linked synthetic examples; it does not establish clinical validity, real-speech transfer, or superiority to a clean cloud system.
1 Introduction
Sensitive private speech is difficult to redistribute, and Japanese care handoffs require mapping compressed spoken language into distinct note fields. The paper asks whether a compact audio-native model can learn this transformation from a few hundred designed synthetic examples.
1 Introduction
- The central question is whether a compact audio-native model can acquire speech-to-structure capability from a few hundred designed synthetic examples.Japanese care handoffs combine compressed spoken input with outputs separating observations, assessment, actions, and plans.Source: §1 · p. 1
- Each supervision item links scenario seeds, fact requirements, spoken realization, rendered audio, structured target, and quality metadata.The authors emphasize investing in the specification and audit trail around examples rather than assuming parameter scale alone solves the task.Source: §1 · p. 1
- The study compares one 1.47B backbone unadapted, fully tuned, and adapted with rank-16 LoRA, while bounding its parameter-efficiency claim by unmeasured device execution.The contributions include provenance-linked supervision and a same-backbone, seed-disjoint capability-acquisition result with paired uncertainty.Source: §1 · p. 1
2 Synthetic supervision
The benchmark pairs Japanese spoken handoffs with six-field JSON notes and fact checklists, retaining provenance and quality metadata. Its synthetic split is seed-disjoint, but held-out coverage is incomplete and unbalanced.
2 Synthetic supervision
- The task maps a Japanese spoken handoff to six JSON strings—Focus, Subjective, Objective, Assessment, Intervention, and Plan—with a scenario-specific fact checklist.The checklist and reference note are generated benchmark artifacts, not certified clinical gold; records retain provenance and other metadata.Source: §2 · p. 1
- Planted omissions and fabrications are excluded from fine-tuning manifests, while the uncalibrated ASR consistency alert remains in the data and is not an exclusion gate.Source: §2 · p. 1
- Of 221 faithful clips from 32 seeds, 182 train-plus-development clips and 39 held-out clips are partitioned without any seed crossing splits.The held-out IDs exactly match the manifest.Source: §2 · pp. 1–2
- The 39-clip holdout is seed-disjoint but not family-disjoint or balanced, covers four of six families, and its distributions bound generalization claims.It includes fall risk (13), meal intake (8), night wandering (14), and vital change (4).Source: §2 · p. 2
- In a separate 75-item review of selected synthetic inputs, 58 were usable as-is, 17 with minor issues, none unusable, and 72 were marked natural Japanese.Each item had one independent label, with no inter-annotator agreement estimate; this review does not establish reference correctness or model-output quality.Source: §2 · p. 2
3 Adaptation and evaluation
The evaluation compares the unadapted backbone, a transcription-plus-cloud cascade, full tuning, and LoRA on the same 39 clips using model-judged metrics. The section specifies aggregate scoring and paired uncertainty while noting judge and baseline-integrity caveats.
3 Adaptation and evaluation
- Full tuning uses a 2 × 10^-5 learning rate for 200 steps, while rank-16 LoRA uses 10^-4 for 150 steps and targets attention projections and w1–w3.Both use batch size 4; LoRA uses α = 16 and dropout 0.2.Source: §3 · p. 2
- Project telemetry reports 12.4M LoRA trainable parameters, approximately 0.85% of 1.47B, but the count was not independently reconstructed from the checkpoint.It is treated as project telemetry rather than a verified measurement.Source: §3 · p. 2
- The four systems are scored on the same 39 clips using a separate instruction model for FC, CR, hallucinations, and missed facts.FC and CR are separately elicited per-clip scalars and macro-averaged; Hall. and Missed sum separately returned error-list lengths.Source: §3 · p. 2
- The judge was not calibrated against a sufficiently large independently adjudicated clinical set, and reported hallucination and missed-fact counts are model-judge detections.Source: §3 · p. 2
- FCR is the harmonic mean of macro FC and CR, and uncertainty is estimated with a paired bootstrap of 20,000 resamples of the 39 clips.Each resample retains paired system outputs and recomputes the aggregate harmonic score; percentile 95% intervals use seed 20260806.Source: §3 · p. 2
- Twenty-one of 39 stored cascade outputs contain 143 Unicode replacement characters, so the registered aggregate is retained for traceability but not used to claim clean cloud parity.The affected-versus-unaffected score difference is a post-hoc association, not a causal estimate.Source: §3 · p. 2
4 Results
Both adaptation methods substantially outperformed the same unadapted backbone on the synthetic benchmark. Full tuning’s numerical advantage over LoRA was uncertain, and comparisons with the cascade were inconclusive.
4 Results
- Full tuning scored 0.8664 and LoRA 0.8461, with paired gains over the same base of +0.8164151 and +0.7961004, respectively.The paired-gain intervals support capability acquisition on the frozen synthetic benchmark.Source: §4 · p. 2
- Full tuning exceeded LoRA by 0.0203147, but the paired 95% CI [−0.0293397, 0.0729566] crossed zero.The study was not a non-inferiority test, and the adaptation settings differed in learning rate and step count.Source: §4 · p. 2
- Full-minus-cascade and LoRA-minus-cascade differences were +0.0209988 and +0.0006841, with both comparisons inconclusive before accounting for cascade corruption.The respective intervals were [−0.0271980, 0.0698786] and [−0.0648183, 0.0655097].Source: §4 · pp. 2–3
5 Scope and limitations
The results support a narrow combined transformation, but the synthetic benchmark and study design limit attribution and deployment claims.
5 Scope and limitations
- Without a format-only ablation, the measured gain cannot be partitioned between learning the six-field JSON contract and semantic fact transfer.The supported claim is that a compact model learned the combined narrow transformation from examples that explicitly represented it.Source: §5 · p. 3
- All training and evaluation speech is synthetic; no real-room, speaker, accent, or clinical transfer is tested.References are model-proposed, and the judge is uncalibrated.Source: §5 · p. 3
- The held-out set is small and incomplete, runs do not establish seed robustness, and the cascade baseline is corrupted.Source: §5 · p. 3
- No device latency, peak memory, energy, power, or real-time factor was measured, so efficient execution on a named device is not demonstrated.A local-inference path is not itself a device-performance measurement.Source: §5 · p. 3
- Deployment would still require consent, security controls, retention limits, and professional verification of every draft.Synthetic scenarios can encode implausible practice and demographic omissions.Source: §5 · p. 3
6 Conclusion
A compact Japanese audio model acquired a narrow speech-to-structure capability from provenance-linked synthetic clips. The conclusion is limited to capability acquisition and adaptation efficiency, not clinical validity or measured on-device performance.
6 Conclusion
- A 1.47B Japanese audio model acquired a narrow speech-to-structure capability from 182 provenance-linked synthetic clips.Source: §6 · p. 3
- Full and LoRA adaptation produced large paired gains over the same base, while LoRA retained 97.7% of the full-tuning aggregate with a small reported trainable fraction.Source: §6 · p. 3
- The result does not establish clinical validity or measured on-device performance; those require real-speech, human-centered, and device-specific studies outside the present claim.Source: §6 · p. 3