4D Panoptic Lidar Segmentation

Mehmet Aygun1
Aljosa Osep1
Mark Weber1
Maxim Maximov1
Cyrill Stachniss2
Jens Behley2
Laura Leal-Taixe1

1Technical University of Munich      2University of Bonn
In CVPR, 2021

 [Paper]  [Code]  [Benchmark]


In this paper, we propose 4D panoptic LiDAR segmentation to assign a semantic class and a temporally consistent instance ID to a sequence of 3D points. To this end, we present an approach and a point centric evaluation metric. Our approach determines a semantic class for every point while modeling object instances as probability distributions in the 4D spatio-temporal domain. We process multiple point clouds in parallel and resolve point-to-instance associations, effectively alleviating the need for explicit temporal data association. Inspired by recent advances in benchmarking of multi-object tracking, we propose to adopt a new evaluation metric that separates the semantic and point-to-instance association aspects of the task. With this work, we aim at paving the road for future developments of temporal LiDAR panoptic perception.

Key Ideas

Our paper introduce two main ideas for tackling 4D Lidar Panpoptic Segmentation.

i) Forming 4D volumes using consecutive lidar frames using ego-motion and obtain instance and semantic segmentation predictions on both space and time.

ii) A point-based encoder-decoder network which works on 4D volumes and outputs necessary ingredients to cluster points via gaussian distribution function, as well as semantic interpretation for all points.


Aygun et.al.

4D Panoptic Lidar Segmentation

CVPR, 2021. [Paper] [Bibtex]


Semantic and Instance Predictions for Sequence 8 from SemanticKITTI. Our method segment all points, find instances and track them through the sequence.


This project was funded by the Humboldt Foundation through the Sofja Kovalevskaja Award and the EU Horizon 2020 research and innovation programme under grant agreement No. 101017008 (Harmony). We thank Ismail Elezi and the whole DVL group for helpful discussions.. This webpage template was borrowed from colorful folks.