Keyframe-based Thermal-Inertial Odometry 
Shehryar Khattak            Christos Papachristos                  Kostas Alexis
Autonomous Robots Lab, University of Nevada, Reno
Picture
Abstract: ​Autonomous navigation of Micro Aerial Vehicles in environments that are simultaneously GPS-denied and visually-degraded, and especially in dark, texture-less and dust- or smoke-filled settings is rendered particularly hard. However, a potential solution arises if such aerial robots are equipped with Long Wave Infrared thermal vision systems which are unaffected by darkness and can penetrate most types of obscurants. In response to this fact, this work proposes a keyframe-based thermal--inertial odometry estimation framework tailored to the exact data and concepts of operation of thermal cameras. The front-end component of the proposed solution utilizes full radiometric data to establish reliable correspondences between thermal images, as opposed to operating on rescaled images as previous efforts have presented. In parallel, taking advantage of a keyframe-based optimization back-end the proposed method is suitable for handling periods of data interruption which are commonly present in thermal cameras, while it also ensures the joint optimization of re-projection errors of 3D landmarks and inertial measurement errors. The developed framework was verified with respect to its resilience, performance and ability to enable autonomous navigation in an extensive set of experimental studies including multiple field deployments in severely degraded, dark and obscurants-filled underground mines.

Publication

Under Review

Video Results

Inactive Underground Mine & Vicon Room
Active Underground Mine & Urban Parking Lot

Code

Coming soon! 

Highlights from the Underground Thermal Data

To emphasize the importance of operating on full radiometric (14-bit or 16-bit) thermal data, below we present two versions of the rescaled thermal imagey from an underground field deployment in the Comstock Exploratory Mine in Virginia City, Nevada, US. The first version (left) is based on histogram equalization per frame (analogous to AGC typically introduced by thermal camera manufacturers). The second version is based on fixed-range normalization.​
Indicative time instance: 
  • 1:55 - 2:15 An example of particularly low contrast thermal imagery as robot turns a corner in the mine while facing towards the wall, presenting a uniform temperature surface.
Indicative time instance: 
  • ​0:40 - 0:55 An example of rapid and significant appearance change between successive images.

Dataset

This dataset contains thermal images either a FLIR Tau2 or a FLIR Boson (16-bit format), VN100 IMU measurements and in some cases pointclouds from a Velodyne PUCK Lite LiDAR or an Ouster OS1-64. The provided datasets were collected during the autonomous flight experiments of our robot using an on-board computer running Ubuntu 16.04 and ROS Kinetic at a) the Comstock underground mine in Northern Nevada, b) an active gold mine in Northern Nevada, c) an Urban Parking Lot, as well as d) our Autonomous Robots Arena. All datasets contain both sensor data, as well as intrinsic and extrinsic sensor calibrations (YAML files). 
​
 
Relevant Downloads