Source-linked AI summary
Real-Time Localisation of Dorsal Anatomical Regions on Dairy Cows in Commercial Rotary Parlours from RGB-D
Vineet Prasad, Shiron Melvin, Sheikh Azid, Hai Wang
TL;DR
Automated dairy monitoring often needs measurements from specific body regions, but those regions are usually placed by hand and existing systems do not provide reusable named masks for crowded rotary parlours. The paper presents a whole-frame RGB-D network that detects the target cow, landmarks, and dorsal regions, with reported accuracy, real-time performance, and farm-rig transfer.
Problem
Method
A whole-frame RGB-D network jointly finds the target cow, ten dorsal landmarks, and four named region masks, whose depth can be transferred to a co-calibrated sensor.
Source: §1 · p. 1Results
ConvNeXt-T reaches PCK@10 0.905, while RGB-D models with a region head reach rule-region IoU 0.87 to 0.91 for back and rump, 0.73 to 0.77 for tail-head, and 0.67 to 0.70 for withers.
Source: §6.1 · p. 4Takeaways & Limitations
On the authors’ commercial-dairy rig, ten labelled frames adapt the model, and projected regions yield one temperature per region and cow that is stable within a pass.
Source: §8 · p. 9The viewing geometry constrains performance: the oblique dataset view weakens withers landmarks, while the narrower thermal field of view leaves many rump and tail-head pixels outside the image.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Automated monitoring of dairy cows increasingly measures one part of the body, such as the rump for temperature or the back for posture, yet the region itself is usually placed by hand. Existing systems either score the cow as a whole or detect sparse keypoints, often in controlled walkways, and neither yields the named region masks that such measurements need on a crowded commercial rotary platform. We present a perception front end that finds the target cow, ten bony landmarks and four named dorsal regions (withers, back, rump and tail-head) in one forward pass over a whole RGB-D frame. Colour and depth encoders are fused late and share one decoder with three heads. On an anatomical region dataset built on public rotary-parlour RGB-D images (538 annotated frames, cow-disjoint test set), the network (ConvNeXt-T backbone) reaches PCK@10 0.905 for landmarks and region IoU 0.70 to 0.91, and runs end to end in 26.4 ms per frame on a T4 GPU, against an 11 fps camera. Removing the region head costs 0.15 PCK@10 (ResNet-18), but a control shows that it mainly adds training signal: wider landmark heatmap targets recover the gain without it. Agreement between the two heads flags unreliable frames without labels. On our own rig at Western Australia's largest dairy farm, the ConvNeXt-T model transfers with ten labelled frames (PCK@10 0.85), and its regions are carried through depth into a radiometric thermal camera, giving one temperature per region and cow that is stable within a pass.
1 Introduction
Automated dairy monitoring often measures a specific body part, but the region is generally placed by hand rather than measured or validated as an object in its own right. This work targets named dorsal-region masks in crowded rotary parlours with a whole-frame RGB-D network and an annotated dataset.
1 Introduction
- Existing measurements use hand-placed or task-specific regions, which are not measured, reused, or validated in their own right.A new measurement therefore needs a new region-finding method.Source: §1 · p. 1
- Rotary-parlour vision must isolate the target cow and localise anatomy among neighbours, rails, and hoses from changing oblique views, while keeping pace with the platform.Colour alone is unreliable on black-and-white coats, and depth alone loses fine landmarks.Source: §1 · p. 1
- The proposed whole-frame RGB-D network uses late fusion and shared decoding for target-cow masks, ten visible or occluded landmarks, and four named region masks.The regions retain the depth behind each pixel for transfer to a co-calibrated sensor.Source: §1 · p. 1
- The anatomical-region dataset adds 538 landmark frames, 18 hand-drawn region outlines, 36 within-pass repeat pairs, reviewed cow masks, and depth-to-colour calibration to public rotary-parlour images.The underlying recording is cited as.Source: §1 · p. 1
3. A controlled study on cow-disjoint and session splits,
The evaluation covers modality and fusion, head ablations, backbone comparisons, and a label-free quality flag. It also reports inference timing and a farm-rig transfer proof of concept.
3. A controlled study on cow-disjoint and session splits,
- The study compares input modality and fusion, head ablations, four backbones with a backbone-specific recipe, and a label-free quality flag from head agreement.Each experiment uses three seeds.Source: §3 · p. 2
- 13.8 to 28.6 ms per frame is the reported end-to-end timing on a T4 GPU at batch size one, three to six times faster than the camera.Source: §3 · p. 2
- The farm-rig proof of concept transfers the model to a second RGB-D camera and projects regions into radiometric thermal images.The rig is at Peninsula Downs, Western Australia’s largest dairy farm.Source: §3 · p. 2
2 Related Work
Prior dorsal-measurement systems largely derive features from cropped body surfaces, predict keypoints or whole-cow scores, or place thermal regions by hand. This paper’s related work also motivates testing RGB-D fusion and joint dense-task learning.
2 Related Work
- Top-view depth systems derive hand-designed features from cropped dorsal surfaces; Spoliansky et al. [2016] report the next cow’s head hiding the tail-head.Source: §2 · p. 2
- Duan et al. [2025] predict back-curvature features from keypoints, while Ruchay et al. [2026a] output one body-condition score per frame; thermal studies place regions by hand [Perez Marquez et al., 2019; Salles et al., 2016].The cited thermal studies report temperature changes before ovulation and a relationship between flank temperature and temperature-humidity index, respectively.Source: §2 · p. 2
- Prior work reports improved semantic segmentation from separate depth encoders versus channel stacking and benefits from jointly learning dense tasks [Kendall et al., 2018].Source: §2 · p. 2
3 Data and Annotation
The dataset adds anatomical annotations and calibration to a public rotary-parlour recording, then defines cow-disjoint and session-based evaluation splits. Its region rules use bony landmarks, with explicit handling for an off-frame withers landmark.
3 Data and Annotation
- The dataset annotates a public rotary recording of 1,285 passes from 1,025 cows across four sessions, using colour and unregistered depth frames from an oblique Kinect view.The recording provides body-condition scores, but the anatomical labels and calibration are newly annotated for this dataset.Source: §3 · p. 2
- One frame from each of 538 passes received ten landmarks with visibility flags and a target-cow box; a second frame from 36 passes was relabelled for repeatability.Source: §3 · p. 2
- The four regions are defined from landmark boundaries, with K1 placed 0.38 hook widths ahead of K2 when it is outside the image.Source: §3 · p. 2
- Landmark-bounded regions avoid steep body edges where thermal readings are least reliable, and freehand outlines on 18 test frames checked the rules against anatomy.Source: §3 · p. 3
- Reference target-cow masks were prompted with SAM 2.1 [Ravi et al., 2025] and reviewed; PnP calibration reduced median depth-to-colour reprojection error from 27 px to about 7 px.The calibration used 96 clicked point pairs on 12 frames.Source: §3 · p. 3
- Cows were split 70/10/20 into 406, 58, and 74 frames with no cow shared across sets; a separate session split trains on afternoons and tests on 141 morning frames.The morning test set has 31% of passes at body condition 4.0 or above, versus about 1% in the afternoons.Source: §3 · p. 3
- Test scoring excludes frames where the labelled target cow is over 300 px from the image centre, leaving 69 cow-split and 138 session-split test frames.The target is defined as the cow in the stall nearest the image centre.Source: §3 · p. 3
4 Method
The method processes an uncropped whole RGB-D frame with late-fused colour and depth features, then predicts the target cow, landmarks, and named dorsal regions. It also compares region predictions with landmark-derived regions for a label-free quality flag and projects regions into a co-calibrated sensor.
4 Method
- The whole frame is not cropped around the labelled cow, so locating the target among neighbours is part of the task.Colour and reprojected depth are input together.Source: §4 · p. 3
- Colour and depth use separate encoders whose feature maps are summed at strides 4–32 before a feature-pyramid decoder produces stride-4 features.Early fusion instead stacks depth as a fourth input channel to one encoder.Source: §4 · p. 3
- Three heads predict the target-cow mask, ten landmark heatmaps and visibility scores, and four region masks.The region masks are trained from rule regions built from labelled landmarks, adding dense supervision without additional annotation.Source: §4 · p. 3
- The total loss combines heatmap, visibility, region, cow-mask, and consistency losses, with consistency training applied from epoch 10.The fixed weights place the sparse heatmap error on a scale comparable to the mask losses; visibility and consistency terms are halved.Source: §4 · p. 3
- Mean IoU agreement between the region head and landmark-derived regions flags potentially unreliable frames without labels.Shared-encoder failures can affect both heads, so high agreement does not guarantee correctness.Source: §4 · p. 3
- Depth gives each region pixel a 3D position, allowing projection into a second camera with known pose and intrinsics.The paper uses this projection to read region temperatures from a thermal camera.Source: §4 · p. 3
- ConvNeXt-T was unstable under the shared training recipe, so it used a separate recipe with a lower encoder learning rate and additional optimization controls.Three of six runs failed over two learning rates under the shared recipe.Source: §4 · p. 3
5 Experimental Set-up
The setup defines landmark, region, and cow-mask metrics and evaluates learned models on cow-disjoint test frames. It also specifies the pass-based stability analysis and end-to-end timing procedure.
5 Experimental Set-up
- Landmark accuracy is the percentage of keypoints within 5% or 10% of labelled hook width, while region masks are scored against rule regions and hand-drawn outlines.The cow mask is scored against the reviewed reference mask.Source: §5 · pp. 3–4
- Figure 2 depicts a whole-frame RGB-D input, three prediction heads, a label-free agreement flag, and region projection into a thermal camera.Its final panel shows temperatures within projected regions on one farm-rig frame, using the 95th percentile as the region temperature.Source: Figure 2 · p. 4
- Learned results are means and standard deviations over three seeds on 69 cow-disjoint test frames with 661 labelled landmarks, unless stated otherwise.Confidence intervals resample test cows, and paired configuration differences use the same resampled cows.Source: §5 · p. 4
- Within-pass stability is measured without labels on all 20 frames of every pass.Farm results use four-fold cross-validation by pass.Source: §5 · p. 4
- Timing is measured end to end on an NVIDIA T4 at batch size one in FP16, from host-memory frame upload through decoded masks and landmarks.The procedure includes preprocessing and network execution and reports the median of three sessions after at least 500 frames each.Source: §5 · p. 4
6 Results
On the cow-disjoint test set, the RGB-D system localizes landmarks and named regions, with late fusion and joint region supervision benefiting landmark performance. It also transfers to a farm rig and projects regions into thermal imagery, though absolute temperature accuracy was not tested.
6.1 Accuracy against the human
- ConvNeXt-T reaches PCK@10 0.905, while ResNet-18 reaches 0.753 ± 0.007; network-predicted regions also agree with rule regions at IoU 0.67 to 0.91.On 17 outline frames, a median 88% to 97% of predicted regions lay inside the annotator’s freehand outlines; these regions were narrower by design.Source: §6.1 · p. 4
6.2 Inputs and fusion
- Separate colour and depth encoders with late fusion improve landmark PCK@10 by 0.05 to 0.09 over every single-encoder variant; region IoU is nearly unchanged.Depth alone reaches PCK@10 0.701 against 0.669 for colour alone.Source: §6.2 · p. 5
6.3 Multi-task design and quality flag
- Removing the region head drops PCK@10 from 0.753 to 0.600; wider heatmaps reach 0.819 ± 0.019, while head agreement flags errors most clearly for weaker models.On the final ConvNeXt-T, the lowest-agreement 20% of frames reach PCK@10 0.84 against 0.92 for the rest.Source: §6.3 · p. 5
6.4 The right cow
- The cow head reaches IoU 0.96 with ResNet-18 and 0.98 with ConvNeXt-T, outperforming depth-only and SAM 2 cuts that merge touching neighbours.Trained on whole frames, the network learns the target cow from stall layout rather than a crop.Source: §6.4 · p. 5
6.6 Backbones and speed
- On a T4 at batch size 1 in FP16, ConvNeXt-T achieves PCK@10 0.905 ± 0.008; end-to-end latency is 26.4 ms (38 fps), versus ResNet-18’s 13.8 ms (73 fps), below the 90 ms camera interval.ConvNeXt-T’s accuracy depended on its own recipe; the shared recipe failed in three of six runs over two learning rates. K2 was hardest, at PCK@10 0.66.Source: Table 2 · p. 6, §6.6 · pp. 6–7
6.5 Sessions and stability
- PCK@10 is 0.760 ± 0.017 on 111 cows unseen in training; relative descriptors repeat consistently, but absolute Kinect-derived widths do not (ICC 0.02 to 0.12).For four of six descriptors, the model is steadier than the annotator’s repeat labels.Source: §6.5 · p. 6
6.7 Farm proof of concept at Peninsula Downs
- Ten labelled farm frames raise PCK@10 to 0.846; 39 reach 0.913, and within-pass region temperatures vary by 0.11 to 0.29 °C SD, without absolute accuracy validation.The tail-head lies partly outside the thermal camera’s field of view, so its temperature describes the visible portion only.Source: §6.7 · pp. 7–8, Figure 6 · p. 8
7 Discussion and Limitations
The pipeline runs faster than the camera, while late fusion and dense landmark supervision are the key design choices. Its principal scope limitation is camera viewing geometry.
Fusion and supervision
- Separate encoders and sufficient landmark training signal matter more than modality; dense region supervision or wider heatmaps can supply that signal.The region head adds one convolutional head to the shared decoder.Source: §7 · pp. 8–9
Real-time margin
- 35 to 73 fps on a 70 W inference GPU exceeds the 11 fps camera rate, leaving real-time processing margin.The margin also leaves room for an embedded GPU, which the paper has not yet measured.Source: §7 · p. 9
Limitation: camera placement
- Camera placement limits the viewing geometry: oblique views foreshorten the cow, while the narrower thermal view can exclude rump and tail-head regions.The proposed strictly overhead mount with matched colour, depth and thermal fields of view addresses both constraints.Source: §7 · p. 9
8 Conclusion
A single RGB-D network detects the target cow, ten dorsal landmarks and four named regions, and transfers to a farm rig for thermal-region measurements.
8 Conclusion
- The network segments four depth-associated regions and transfers with ten labelled frames to produce stable within-pass temperatures per region and cow.The conclusion identifies a matched overhead rig and multi-session farm recordings as the next steps for testing physiological signal.Source: §8 · p. 9