3D semantic understanding is essential for a wide range of robotics applications. Availability of datasets is a strong driver for research, and whilst obtaining unlabeled data is straightforward, manually annotating this data with semantic labels is time-consuming and costly. Recently, foundation models have facilitated open-set semantic segmentation, potentially aiding automatic labeling. However, these models have largely been limited to 2D images. This work introduces Label Any Pointcloud (LeAP), which leverages 2D Vision Foundation Models (VFMs) to automatically label 3D data with any set of classes in any kind of application. VFMs are used to create image labels for the desired classes which are then projected to 3D points. Using the Bayesian update, point-wise labels are combined into voxels to improve label consistency, and label points outside the camera fustrum. A novel Cross-Modal Self-Training (CM-ST) approach further enhances label quality. Through extensive experiments, we demonstrate that our method can generate high-quality 3D semantic labels across diverse fields without any manual 3D labeling. Models adapted to new application domains using our labels show up to 3.7× (12.9 → 47.1) mIoU improvement compared to the unadapted baselines. This ability to provide labels for any domain can help accelerate 3D perception research.

Automotive radars are increasingly used in automated driving systems due to their cost effectiveness, ease of integration, and ability to withstand adverse weather conditions. Semantic segmentation for radar point clouds is a crucial step in radar pre-processing, which can be used on almost all downstream tasks of radars, such as detection and tracking. Radar point clouds are noisier compared to LiDAR point clouds due to sensor noise and the multi-path propagation, which makes the segmentation task for radar more challenging. In this paper, we address the problem of segmentation in noisy radar point clouds in terms of ghost target vs. real detection, moving vs. static objects as well as semantic segmentation of moving road users. We demonstrate how these three tasks can be performed in a single, unified pipeline using an auto-labeled radar dataset. Our approach, called Real, Moving, and Semantic Segmentation Network (RMSNet), is able to output point-wise labels for all three tasks simultaneously. On our dataset, RMSNet attains an IoU of 82.5% for real detection segmentation, 66.9% IoU for moving object segmentation, and 64.9% mIoU for semantic segmentation. We also did a live demonstration, using the model on two different radars. The result shows that the model is capable of running inference in real-time and demonstrates good ability in model generalization.