Haresh Karnan

I am a PhD candidate at The University of Texas at Austin, specializing in Reinforcement Learning, Computer Vision and Artificial Intelligence for Robotics. I'm advised by Peter Stone, and affiliated with the Learning Agents Research Group (LARG) at UT Austin Computer Science department. I'm also a member of the UT Austin RoboCup@Home team.

I was previously an Applied Science Intern at Amazon Scout where I worked on Deep Learning and Computer Vision for their package delivery robot.

In my past life, I worked with Dr. Robert Skelton at Texas A&M University, College Station, on state estimation and control of Tensegrity robots.

Email  /  CV  /  Biography  /  Google Scholar  /  LinkedIn

profile photo
Pre-Prints
3DSP Learning to Extrapolate Human Preferences for Preference-Aligned Path Planning
Haresh Karnan, Elvin Yang, Garrett Warnell, Joydeep Biswas, Peter Stone
Accepted for Publication @ ICRA 2024!
Paper / Video / Poster / PATERN @ Lake Austin

Robots navigating off-road environments typically require additional human feedback to learn operator preferences for visually novel terrains. In this work, we introduce a framework to extrapolate operator preferences from known terrains by leveraging multi-modality.

3DSP Targeted Learning: A Hybrid Approach to Social Robot Navigation
Amir Hossain Raj, Zichao Hu, Haresh Karnan, Rohan Chandra, Amirezza Payandeh, Luisa Mao, Peter Stone, Joydeep Biswas, Xuesu Xiao
Accepted for Publication @ ICRA 2024!
Paper / Video

In this work, we challenge the prevailing shift towards solely utilizing learning-based approaches for robot navigation in social contexts, advocating instead for a hybrid approach that toggles between classical and learning-based motion planners.

3DSP Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience
Haresh Karnan, Elvin Yang, Daniel Farkash, Garrett Warnell, Joydeep Biswas, Peter Stone
Conference on Robot Learning (CoRL), 2023
Project Page / Paper / Video / STERLING @ Lake Austin / Poster / Code

Visual terrain awareness is crucial for autonomous off-road navigation. We propose a novel way to learn relevant terrain representations from unconstrained, multi-modal robot experience collected with any policy. We show how such learned representations enable aligning off-road navigation behaviors with human operator preferences.

Publications
3DSP Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation Haresh Karnan, Anirudh Nair, Xuesu Xiao, Garrett Warnell, Soeren Pirk, Alexander Toshev, Justin Hart, Joydeep Biswas, Peter Stone
IEEE Robotics and Automation Letters (IEEE RA-L) , 2022
Paper / Short Video / Long Video / Poster / Dataset

We introduce a large-scale, first-person-view dataset of socially compliant robot navigation demonstrations. SCAND consists of 138 trajectories, 25 miles of socially compliant navigation demonstrations collected on 2 robots by 4 human demonstrators within the UT Austin campus.

3DSP High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamics
Haresh Karnan, Kavan Singh Sikand, Pranav Atreya, Sadegh Rabiee, Xuesu Xiao, Garrett Warnell, Joydeep Biswas, Peter Stone
International Conference on Intelligent Robots and Systems (IROS) , 2022
Paper / Video

We introduce VI-IKD, a visual-inertial IKD model that learns to anticipate the kinodynamic interactions of the vehicle with the terrain ahead, enabling accurate high-speed navigation.

3DSP Optim-FKD: High-Speed Accurate Robot Control using Learned Forward Kinodynamics and Non-linear Least Squares Optimization Pranav Atreya, Haresh Karnan, Kavan Singh Sikand, Sadegh Rabiee, Xuesu Xiao, Garrett Warnell, Joydeep Biswas, Peter Stone
International Conference on Intelligent Robots and Systems (IROS) , 2022
Paper / Video

In this work, we introduce Optim-FKD, an approach for high-speed navigation that uses a learned forward kinodynamics model (FKD) coupled with non-linear least squares optimization for multi-horizon control.

3DSP Autonomous Ground Navigation in Highly Constrained Spaces: Lessons Learned From the Benchmark Autonomous Robot Navigation Challenge at ICRA 2022 Xuesu Xiao, Zifan Xu, Zizhao Wang, Yunlong Song, Garrett Warnell, Peter Stone, Tingnan Zhang, Shravan Ravi, Gary Wang, Haresh Karnan, Joydeep Biswas, Nicholas Mohammad, Lauren Bramblett, Rahul Peddi, Nicola Bezzo, Zhanteng Xie, Philip Dames
Robotics and Automation Magazine (RA-M) , 2022
Paper

Won the 1st place in BARN Challenge @ ICRA 2022

3DSP VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation
Haresh Karnan, Garrett Warnell, Xuesu Xiao, Peter Stone
International Conference on Robotics and Automation (ICRA), 2022
Paper / Video / Poster

Imitating an expert's video-only demonstration can be challenging in the presence of egocentric viewpoint mismatch. In VOILA, we introduce a unique viewpoint-invariant reward function to effectively imitate demonstrations on a physically different agent.

3DSP Adversarial Imitation Learning from Video using a State Observer
Haresh Karnan, Garrett Warnell, Faraz Torabi, Peter Stone
International Conference on Robotics and Automation (ICRA), 2022
Paper / Video / Poster

SOTA approaches in Imitation from Video-only demonstrations exhibit poor sample efficiency when learning with access to proprioceptive features of the imitator. To tackle this, we introduce VGAIfO-SO, an IfO algorithm that improves sample efficiency through self-supervision.

3DSP Grounded Action Transformation for Sim-to-Real Reinforcement Learning
Josiah Hanna, Siddharth Desai, Haresh Karnan, Garrett Warnell, Peter Stone
Springer. Machine Learning, 2021
Paper

Sim-to-real is the problem of learning a control policy in an inaccurate simulated world such that the learned policy when transferred to the real-world, performs well. In this article, we explore the proposed black-box Sim-to-Real algorithm GAT, and its extension SGAT.

3DSP An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
Siddharth Desai, Ishan Durugkar, Haresh Karnan, Josiah Hanna, Garrett Warnell, Peter Stone
Neural Information Processing Systems (NeurIPS), 2020
NeurIPS site / Paper / Poster / arXiv

In this work, we propose the GARAT algorithm, which treats the Sim-to-Real problem as an Imitation from Observation (IfO) problem and uses advances in the IfO literature to transfer a control policy from a source domain to a target domain, using Adversarial Imitation Learning.

3DSP Stochastic Grounded Action Transformation for Robot Learning in Simulation
Haresh Karnan, Siddharth Desai, Josiah Hanna, Garrett Warnell, Peter Stone
International Conference on Intelligent Robots and Systems (IROS), 2020
Long Video / Short Video / Paper / Poster / arXiv

Real world dynamics are often stochastic and robot simulators have an inaccurate approximation of real world dynamics. In this work, we propose a Sim-to-Real algorithm called SGAT and transfer a Humanoid walk from Simulation to Real world.

3DSP Reinforced Grounded Action Transformation for Sim-to-Real Transfer
Haresh Karnan, Siddharth Desai, Josiah Hanna, Garrett Warnell, Peter Stone
International Conference on Intelligent Robots and Systems (IROS), 2020
Long Video / Short Video / Paper / Poster / arXiv

In this work, we propose a Sim-to-Real algorithm called RGAT to ground an inaccurate simulator with data from the real world, using Reinforcement Learning.

3DSP Visual Feedback Control of Tensegrity Robotic Systems
Haresh Karnan, Raman Goyal, Manoranjan Majji, Robert Skelton, Puneet Singla
International Conference on Intelligent Robots and Systems, 2017
Video / Paper

Tensegrity mechanisms are known for their minimal-mass and flexible properties. In this work, we propose using vision based sensing for shape control of such soft robotic manipulators.

Workshops, Symposiums, Extended Abstracts
3DSP Aligning Robot Navigation Behaviors with Human Intentions and Preferences
Haresh Karnan
IPPC: Integrated Planning, Perception and Control Workshop, IROS, 2023
Paper / Poster

The research explored in this paper helps address challenges of value alignment in machine learning for robot navigation. It presents new algorithms and a dataset designed to teach mobile robots navigation behaviors that align with human intentions and preferences. These innovations are applicable in diverse settings, including indoor, outdoor, and human-occupied environments.

3DSP Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience
Haresh Karnan, Elvin Yang, Daniel Farkash, Garrett Warnell, Joydeep Biswas, Peter Stone
Pretraining for Robotics Workshop, ICRA, 2023
Project Page / Paper / Video / STERLING @ Lake Austin / Poster

Visual terrain awareness is crucial for autonomous off-road navigation. We propose a novel way to learn relevant terrain representations from unconstrained, multi-modal robot experience collected with any policy. We show how such learned representations enable aligning off-road navigation behaviors with human operator preferences.

3DSP Learning to Extrapolate Human Preferences for Preference-Aligned Path Planning
Elvin Yang, Haresh Karnan, Garrett Warnell, Joydeep Biswas, Peter Stone
Pretraining for Robotics Workshop, ICRA, 2023
Paper / Poster

Robots navigating off-road environments typically require additional human feedback to learn operator preferences for visually novel terrains. In this work, we introduce a framework to extrapolate operator preferences from known terrains by leveraging multi-modality.

3DSP Principles and Guidelines for Evaluating Social Robot Navigation Algorithms
Anthony Francis, Claudia Perez-D'Arpino, Chengshu Li, Fei Xia,..., Haresh Karnan, Peter Stone
ArXiv, 2023
Paper

We introduce a standardized criteria and a metrics framework for evaluating social robot navigation in human occupied spaces. Key principles include safety, comfort, and social competency. This approach fosters fair algorithm comparison across diverse robots, simulators, and datasets, accelerating progress akin to computer vision and traditional robot navigation.

3DSP Benchmarking Social Robot Navigation Across Academia and Industry
Anthony Francis, Claudia Perez-D'Arpino, Chengshu Li, Fei Xia, Alexandre Alahi, Aniket Bera, Abhijat Biswas, Joydeep Biswas, Hao-Tien Lewis Chiang, Michael Everett, Sehoon Ha, Justin Hart, Haresh Karnan, Tsang-Wei Edward Lee, Luis J. Manso, Reuth Mirksy, Soren Pirk, Phani Teja Singamaneni, Peter Stone, Ada V. Taylor, Peter Trautman, Nathan Tsoi, Marynel Vazquez, Xuesu Xiao, Peng Xu, Naoki Yokoyama, Roberto Martın-Martın, Alexander Toshev
HRI in Academia and Industry, AAAI Spring Symposum, 2023 (Best Paper Award Nominee)
Paper
3DSP VOILA: Visual Observation-only Imitation Learning for Autonomous navigation
Haresh Karnan, Garrett Warnell, Xuesu Xiao, Peter Stone
AAAI ML4NAV Spring Symposium, 2021
Talk / Paper
3DSP Solving Service Robot Tasks: UT Austin Villa@Home 2019 Team Report
Rishi Shah, Yuqian Jiang, Haresh Karnan, Gilberto Briscoe-Martinez, Dominick Mulder, Ryan Gupta, Rachel Schlossman, Marika Murphy, Justin W. Hart, Luis Sentis, Peter Stone
AAAI AI-HRI Symposium, 2019
Video / Paper / arXiv
3DSP Extended Abstract: An Imitation from Observation Approach to Sim-to-Real Transfer
Siddharth Desai, Ishan Durugkar, Haresh Karnan, Josiah Hanna, Garrett Warnell, Peter Stone
Robotics Science and Systems, Sim-to-Real Workshop (RSS), 2020
Video / Paper / Poster

In this work, we propose the GARAT algorithm, which treats the Sim-to-Real problem as an Imitation from Observation (IfO) problem and uses advances in the IfO literature to transfer a control policy from a source domain to a target domain, using Adversarial Imitation Learning.

3DSP Visual Servoing of Unmanned Surface Vehicle from Small Tethered Unmanned Aerial Vehicle
Haresh Karnan, Aritra Biswas, Pranav Vaidik Dhulipala, Jan Dufek, Robin Murphy
arXiv preprint, 2017
Paper / arXiv
Patents
3DSP Systems and Methods for Utilizing Images to Determine the Position and Orientation of a Vehicle Alan Atherton, Umit Batur, Kishor Bhalerao, Haresh Karnan, Harshavardhan Shirolkar
USPTO, US11579622B2, 2021
Google Patents / News Coverage

Template Credits