Linfeng Zhao

CS Ph.D. Student

Northeastern University

Biography

I am Linfeng Zhao (赵林风), a CS Ph.D. student at Khoury College of Computer Sciences of Northeastern University. I’m advised by Prof. Lawson L.S. Wong and also working with Prof. Robin Walters. During PhD: I am visiting and collaborating with MIT Learning and Intelligent Systems group led by Prof. Leslie Kaelbling and Tomás Lozano-Pérez, was a research intern at Boston Dynamics AI Institute with Jennifer L. Barry (2023 Spring-Summer) and earlier at Amazon Science with Kari Torkkola and Dhruv Madeka (2021 Summer). During undergraduate: I interned at Microsoft Research Asia (2019) and earlier worked with Prof. Hao Su at UC San Diego (2018-19).

I focus on decision-making in robotics and work on the intersection of machine learning, robotics and artificial intelligence. My current research centers on improving open-world and long-horizon decision-making for mobile-manipulation robots, like Boston Dynamics Spot, which involves planning (in task and motion level) and learning (state/action representations, world models, skill policies). My recent work involves improving scalability, generalizability, and efficiency for robotic decision-making, including utilizing knowledge and capabilities in pre-trained multimodal models and leveraging symmetry and compositional structures.

News

[2024 Summer] I’m actively looking for internship in robotics and foundation models.
[2024/04] Give a guest lecture at Northeastern RL class on Learning for Long-horizon Planning (slides).
[2024/03] We will hold a workshop at RSS 2024 on Geometric and Algebraic Structure in Robot Learning.
[2024/01] A paper on equivariant graph planning accepted at RA-L with an option to present at IROS 2024.
[2024/01] Give in-person talks at Tsinghua University IIIS (host: Huazhe Xu) & ME (host: Rui Chen) (slides, recording to come).
[2023/11] Present a poster at Boston Symmetry Day at MIT on symmetry in control.
[2023/06] Continuing summer internship at Boston Dynamics AI Institute.
[2023/01] Starting internship at the new Boston Dynamics AI Institute locally in Boston.
[2023/01] Two papers (symmetric planning, implicit differentiable planning) accepted at ICLR 2023.
[2022/12] Physically attend NeurIPS in New Orleans.
[2022/07] Give in-person oral talk at ICML 2022 on compositional generalization (slides) and poster presentation for equivariant world models.
[2022/06] I’ll physically attend RLDM (Providence), RSS (New York City), and ICML (Baltimore) this summer. Let’s chat!
[2022/06] Spotlight talk on symmetric planning at RSS 2022 workshop Implicit Representation for Robotic Manipulation.
[2022/05] Oral talk on compositional generalization (virtually) at ICRA 2022 workshop Compositional Robotics.
[2022/05] Our paper on compositional generalization is accepted to ICML 2022 as Long Presentation (117/5630, 2.1%, oral).
[2022/05] Another paper on symmetric embedding for equivariant world models is accepted to ICML 2022 as Short Presentation (poster).
[2022/03] Two papers (compositional generalization, symmetric planning) to appear at RLDM 2022 (non-archival).
[2021/06] Summer internship at Amazon Science in Seattle.

Featured Publications

Nishanth Kumar, Tom Silver, Willie McClinton, Linfeng Zhao, Stephen Proul, Tomás Lozano-Pérez, Leslie Kaelbling, Jennifer Barry

February 2024 In Conference Submission

Practice Makes Perfect: Planning to Learn Skill Parameter Policies

We enable a robot to rapidly and autonomously specialize parameterized skills by planning to practice them. The robot decides what skills to practice and how to practice them. The robot is left alone for hours, repeatedly practicing and improving.

PDF Code Project Video

Linfeng Zhao, Hongyu Li, Taskin Padir, Huaizu Jiang^, Lawson L.S. Wong

June 2023 In RA-L 2024, Present at IROS 2024

E(2)-Equivariant Graph Planning for Navigation

We study E(2) Euclidean equivariance in navigation on geometric graphs and develop message passing network to solve it.

Project arXiv

Linfeng Zhao, Owen Howell, Jung Yeon Park, Xupeng Zhu, Robin Walters, Lawson L.S. Wong

May 2023 In TAGML Workshop @ ICML 2023

Can Euclidean Symmetry Help in Reinforcement Learning and Planning?

We study whether Euclidean symmetry can help in reinforcement learning and planning, which models the geometric transformations between reference frames of robots.

arXiv

Linfeng Zhao, Xupeng Zhu, Lingzhi Kong, Robin Walters, Lawson L.S. Wong

September 2022 In ICLR 2023, RLDM 2022

Integrating Symmetry into Differentiable Planning with Steerable Convolutions

We formulate how differentiable planning algorithms can exploit inherent symmetry in path planning problems, named SymPlan, and propose practical algorithms.

PDF Code Poster Slides ICLR page OpenReview arXiv Webpage (Available soon)

Linfeng Zhao, Huazhe Xu, Lawson L.S. Wong

September 2022 In ICLR 2023

Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation

We study how implicit differentiation helps scale up and improve convergence of differentiable planning algorithms.

PDF Code Poster Slides ICLR page OpenReview arXiv Webpage (Available soon)

Linfeng Zhao, Lingzhi Kong, Robin Walters, Lawson L.S. Wong

January 2022 In ICML 2022 (Long Presentation, 2.1%), RLDM 2022

Toward Compositional Generalization in Object‑Oriented World Modeling

We formulate compositional generalization in object-oriented world modeling, and propose a soft and efficient mechanism for practice.

PDF Code ICML Page Website (Available soon) Poster (ICML) Slides (ICML oral)

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan-Willem van de Meent, Robin Walters

January 2022 In ICML 2022

Learning Symmetric Embeddings for Equivariant World Models

We learn latent representations enforced with known transformation laws (group action), and apply this idea on equivariant latent world modeling.

PDF Code ICML Page

Publications

Publication List

Linfeng Zhao, Owen Howell, Xupeng Zhu, Jung Yeon Park, Zhewen Zhang, Robin Walters, Lawson L.S. Wong (2024). Equivariant Action Sampling for Reinforcement Learning and Planning. In Conference Submission.

Linfeng Zhao, Lawson L.S. Wong (2024). Learning to Navigate in Mazes with Novel Layouts Using Abstract Top-down Maps. Under Conference Review.

Extended version (with Appendix)

Nishanth Kumar, Tom Silver, Willie McClinton, Linfeng Zhao, Stephen Proul, Tomás Lozano-Pérez, Leslie Kaelbling, Jennifer Barry (2024). Practice Makes Perfect: Planning to Learn Skill Parameter Policies. In Conference Submission.

PDF Code Project Video

Neel Sortur, Linfeng Zhao, Robin Walters (2023). Sample Efficient Modeling of Drag Coefficients for Satellites with Symmetry. In Workshop on Symmetry and Geometry in Neural Representations @ NeurIPS 2023.

OpenReview

Linfeng Zhao, Hongyu Li, Taskin Padir, Huaizu Jiang^, Lawson L.S. Wong (2023). E(2)-Equivariant Graph Planning for Navigation. In RA-L 2024, Present at IROS 2024.

Project arXiv

Linfeng Zhao, Owen Howell, Jung Yeon Park, Xupeng Zhu, Robin Walters, Lawson L.S. Wong (2023). Can Euclidean Symmetry Help in Reinforcement Learning and Planning?. In TAGML Workshop @ ICML 2023.

arXiv

Owen Howell, David Klee, Ondrej Biza, Linfeng Zhao, Robin Walters (2023). Equivariant Single View Pose Prediction Via Induced and Restriction Representations. In NeurIPS 2023.

arXiv

Linfeng Zhao, Xupeng Zhu, Lingzhi Kong, Robin Walters, Lawson L.S. Wong (2022). Integrating Symmetry into Differentiable Planning with Steerable Convolutions. In ICLR 2023, RLDM 2022.

PDF Code Poster Slides ICLR page OpenReview arXiv Webpage (Available soon)

Linfeng Zhao, Huazhe Xu, Lawson L.S. Wong (2022). Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation. In ICLR 2023.

PDF Code Poster Slides ICLR page OpenReview arXiv Webpage (Available soon)

Linfeng Zhao, Lingzhi Kong, Robin Walters, Lawson L.S. Wong (2022). Toward Compositional Generalization in Object‑Oriented World Modeling. In ICML 2022 (Long Presentation, 2.1%), RLDM 2022.

PDF Code ICML Page Website (Available soon) Poster (ICML) Slides (ICML oral)

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan-Willem van de Meent, Robin Walters (2022). Learning Symmetric Embeddings for Equivariant World Models. In ICML 2022.

PDF Code ICML Page

Linfeng Zhao, Lawson L.S. Wong (2020). Model-based Navigation in Environments with Novel Layouts Using Abstract 2-D Maps. In Deep RL workshop @ NeurIPS 2020.

PDF Poster Slides Video

Fan Xie, Alexander Chowdhury, M. Clara De Paolis Kaluza, Linfeng Zhao, Lawson L.S. Wong, Rose Yu (2020). Deep Imitation Learning for Bimanual Robotic Manipulation. In NeurIPS 2020.

PDF Poster NeurIPS Page

Linfeng Zhao, Ziyan Luo, Wei Cheng, Sihao Chen, Qi Chen, Hui Xue, Haidong Wang, Chuanjie Liu, Mao Yang, Lintao Zhang (2020). Match Plan Generation in Web Search with Parameterized Action Reinforcement Learning. In WWW 2021.

PDF Video Source Document

Linfeng Zhao, Hao Su (2019). InterFact: Towards Interactive Factorization of Actionable Entities. In preparation.

See all publications

Projects

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. I worked with Kaichun and Shilin on preparing the annotation system, part hierarchy templates, objects, etc in Jan 2018 through July 2018 and was listed in acknowledgement.

PDF Code Dataset Project Poster Video Code (Kaichun's Annotator) Code (My Annotator Frontend) Code (My Annotator Backend)

Contact

zhao.linf@northeastern.edu