abstract | bibtex | paper | project page

Photorealistic 3D scene reconstruction plays an important role in autonomous driving, enabling the generation of novel data from existing datasets to simulate safety-critical scenarios and expand training data without additional acquisition costs. Gaussian Splatting (GS) facilitates real-time, photorealistic rendering with an explicit 3D Gaussian representation of the scene, providing faster processing and more intuitive scene editing than the implicit Neural Radiance Fields (NeRFs). While extensive GS research has yielded promising advancements in autonomous driving applications, they overlook two critical aspects: First, existing methods mainly focus on low-speed and feature-rich urban scenes and ignore the fact that highway scenarios play a significant role in autonomous driving. Second, while LiDARs are commonplace in autonomous driving platforms, existing methods learn primarily from images and use LiDAR only for initial estimates or without precise sensor modeling, thus missing out on leveraging the rich depth information LiDAR offers and limiting the ability to synthesize LiDAR data. In this paper, we propose a novel GS method for dynamic scene synthesis and editing with improved scene reconstruction through LiDAR supervision and support for LiDAR rendering. Unlike prior works that are tested mostly on urban datasets, to the best of our knowledge, we are the first to focus on the more challenging and highly relevant highway scenes for autonomous driving, with sparse sensor views and monotone backgrounds.

@inproceedings{kung2024lihi,
          title={LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction},
          author={Kung, Pou-Chun and Zhang, Xianling and Skinner, Katherine A and Jaipuria, Nikita},
          booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
          journal={arXiv preprint arXiv:2412.15447},
          year={2024}
        }
        

abstract | bibtex | paper | project page

This paper proposes SAD-GS, a depth-supervised Gaussian Splatting (GS) method that provides accurate 3D geometry reconstruction by introducing a shape-aligned depth supervision strategy. Depth information is widely used in various GS applications, such as dynamic scene reconstruction, real-time simultaneous localization and mapping, and few-shot reconstruction. However, existing depth-supervised methods for GS all focus on the center and neglect the shape of Gaussians during training. This oversight can result in inaccurate surface geometry in the reconstruction and can harm downstream tasks like novel view synthesis, mesh reconstruction, and robot path planning. To address this, this paper proposes a shape-aligned loss, which aims to produce a smooth and precise reconstruction by adding extra constraints to the Gaussian shape. The proposed method is evaluated qualitatively and quantitatively on two publicly available datasets. The evaluation demonstrates that the proposed method provides state-of-the-art novel view rendering quality and mesh accuracy compared to existing depth-supervised GS methods.

@inproceedings{kung2024sad,
          title={SAD-GS: Shape-aligned Depth-supervised Gaussian Splatting},
          author={Kung, Pou-Chun and Isaacson, Seth and Vasudevan, Ram and Skinner, Katherine A},
          booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
          pages={2842--2851},
          year={2024}
        }
        

abstract | bibtex | arXiv | project page

This paper proposes LONER, the first real-time LiDAR SLAM algorithm that uses a neural implicit scene representation. Existing implicit mapping methods for LiDAR show promising results in large-scale reconstruction, but either require groundtruth poses or run slower than real-time. In contrast, LONER uses LiDAR data to train an MLP to estimate a dense map in real-time, while simultaneously estimating the trajectory of the sensor. To achieve real-time performance, this paper proposes a novel information-theoretic loss function that accounts for the fact that different regions of the map may be learned to varying degrees throughout online training. The proposed method is evaluated qualitatively and quantitatively on two open-source datasets. This evaluation illustrates that the proposed loss function converges faster and leads to more accurate geometry reconstruction than other loss functions used in depthsupervised neural implicit frameworks. Finally, this paper shows that LONER estimates trajectories competitively with state-ofthe-art LiDAR SLAM methods, while also producing dense maps competitive with existing real-time implicit mapping methods that use groundtruth poses.

@article{isaacson2023loner,
          title={Loner: Lidar only neural representations for real-time slam},
          author={Isaacson, Seth and Kung, Pou-Chun and Ramanagopal, Mani and Vasudevan, Ram and Skinner, Katherine A},
          journal={IEEE Robotics and Automation Letters},
          year={2023},
          publisher={IEEE}
        }
        

abstract | bibtex | arXiv | video

Radar shows great potential for autonomous driving by accomplishing long-range sensing under diverse weather conditions. But radar is also a particularly challenging sensing modality due to the radar noises. Recent works have made enormous progress in classifying free and occupied spaces in radar images by leveraging lidar label supervision. However, there are still several unsolved issues. Firstly, the sensing distance of the results is limited by the sensing range of lidar. Secondly, the performance of the results is degenerated by lidar due to the physical sensing discrepancies between the two sensors. For example, some objects visible to lidar are invisible to radar, and some objects occluded in lidar scans are visible in radar images because of the radar's penetrating capability. These sensing differences cause false positive and penetrating capability degeneration, respectively.
  In this paper, we propose training data preprocessing and polar sliding window inference to solve the issues. The data preprocessing aims to reduce the effect caused by radar-invisible measurements in lidar scans. The polar sliding window inference aims to solve the limited sensing range issue by applying a near-range trained network to the long-range region. Instead of using common Cartesian representation, we propose to use polar representation to reduce the shape dissimilarity between long-range and near-range data. We find that extending a near-range trained network to long-range region inference in the polar space has 4.2 times better IoU than in Cartesian space. Besides, the polar sliding window inference can preserve the radar penetrating capability by changing the viewpoint of the inference region, which makes some occluded measurements seem non-occluded for a pretrained network.

@inproceedings{kung2021radar,
          Author = {Pou-Chun Kung and
                    Chieh-Chih Wang and
                    Wen-Chieh Lin},
          Title = {Radar Occupancy Prediction with Lidar Supervision while Preserving Long-Range Sensing and Penetrating Capabilities},
          Booktitle = {arXiv:2112.04282}
        }
        

abstract | bibtex | arXiv | video | presentation

Existing radar sensors can be classified into automotive and scanning radars. While most radar odometry (RO) methods are only designed for a specific type of radar, our RO method adapts to both scanning and automotive radars. Our RO is simple yet effective, where the pipeline consists of thresholding, probabilistic submap building, and an NDT-based radar scan matching. The proposed RO has been tested on two public radar datasets: the Oxford Radar RobotCar dataset and the nuScenes dataset, which provide scanning and automotive radar data respectively. The results show that our approach surpasses state-of-the-art RO using either automotive or scanning radar by reducing translational error by 51% and 30%, respectively, and rotational error by 17% and 29%, respectively. Besides, we show that our RO achieves centimeter-level accuracy as lidar odometry, and automotive and scanning RO have similar accuracy.

@inproceedings{kung2021normal,
              Author = {Pou-Chun Kung and
                        Chieh-Chih Wang and
                        Wen-Chieh Lin},
              Title = {A Normal Distribution Transform-Based Radar Odometry Designed For Scanning and Automotive Radars},
              Booktitle = {ICRA},
              Year = {2021}
            }
          

abstract | video

Radar presents a promising alternative to LiDAR in autonomous driving applications by showing several advantages, such as the ability to work under diverse weather conditions, direct doppler velocity estimation, and relatively low price. However, 4D radar outputs a 3D heatmap instead of a precise 3D shape of the surrounded environment. Moreover, the current state-of-the-art radar feature detection method, constant false alarm rate, only output sparse radar features.
  In this paper, we show the potential to convert radar into the 3D shape measurement sensor by demonstrating the preliminary results of particular targets. We introduce the 3D-Unet model to the radar data and takes 3D shape measurements of LiDAR as the training ground truth. However, since the low-cost 4D radar only provides timestamp at second resolution and do not provides time interval between two consecutive data, it is particularly challenging to synchronize radar and LiDAR data. As a result, we only collect static data, including few classes of target objects and human postures, in our experiment. We demonstrate the preliminary experiment that estimates the 3D shape of particular targets from the radar module. The result shows the great potential of converting radar into the 3D shape measurement sensor if the data collection is more effective in the near future.

abstract

Setup sensors on the vehicle and collect data in crowded urban.
Sensors:
- Lucid Camera x4
- Velodyne VLS-128 Lidar
- Ouster OS1-128 Lidar
- Baraja Lidar x2
- Navtech CIR504-X Radar
- Automotive Radar (Continental ARS 408-21) x2

abstract | webpage | video | slide | code

In this project, we used Hokuyo Laser Scanner with scan matching (ICP) and factor graph optimization (GTSAM) to achieve Lidar odometry. We ran our algorithm onboard on Nvidia TX2 in Ubuntu18.04. Instead of doing consecutive Lidar scan matching only, we added the estimated transformation between the current and previous scans as constraints to optimize the robot pose.

user's manual | ros2 slide

abstract | product webpage | video | code

Extended Duckiebot with 2D Lidar to achieve 2D SLAM and navigation. Implemented Lidar particle filter SLAM, EKF SLAM, AMCL localization, and A* path planning on the robot. More than 200 KnightCars were saled as the teaching material in 5 colleges in a year.

abstract | webpage | video | code

Implemented LSD-SLAM with MultiSensor-Fusion EKF (fused with imu/sonar) on quadrotor to achieveautonomous indoor navigation. With this project, I won first place out of 26 teams in NSYSU College of Engineering Project Competition.

abstract

Implemented face recognition (Haar Cascades), classification (InceptionV3), tracking (KLT tracker) and PID controller on quadrotor to achieve people tracking.