Head Image

Zhaoyang Lv  


Research Scientist, Reality Labs Research, Meta.


Previous Education:
Ph.D. in Robotics, School of Interactive Computing, Georgia Institute of Technology
M.Sc., Artificial Intelligence in Computing, Imperial College London
B.Sc., Electrical Engineering in Aeronauntics, Northwestern Polytechnical University


I have been a research scientist in Reality Labs Research, Meta since 2019. I finished my Ph.D. at Georgia Tech, jointly advised by Prof. James Rehg, and Prof. Frank Dellaert. During my Ph.D., I am also fortunate to intern at Nvidia Research in the group of Jan Kautz and at Max Planck Institute with Prof. Andreas Geiger. Before I started my Ph.D., I finished my Master thesis under the supervision of Prof. Andrew Davison at Imperial College London.

Thoughout my research, I am obessed about digitalizing the world in 3D and 4D as we human are able to see using commodity camera sensing. I am a believer that augmented reality with 3D sensing and content that can work on a small form factor devices (e.g. a pair of glasses) will become ubiquitous and fundamentally change the way we interact with world and record our memories. Quote Steve Jobs' comments on GUI when he visited Xerox PARC in 1979 :
Within 10 minutes, it was obvious to me that all computers would work like this someday. You could argue about how many number of years it would take, and you could argue about who the winners and the losers might be, but you couldn't argue about the inevitability.

I work on multidisciplinary field research and make my contributions towards it. Deep in my heart, I am a 3D computer vision guy, with some work strongly related to using computional photography and machine learning. I know a bit about graphics related to neural rendering and robotics (with a Phd degree in robotics).

Digital Twin Catalog

In ECCV 2024, we introduced a new 3D object dataset that contain super high quality object model. We will share more technical details with white paper soon!
Project Page

EgoLifter: Open-world 3D Segmentation for Egocentric Perception

Qiao Gu, Zhaoyang Lv, Duncan Frost, Simon Green, Julian Straub, Chris Sweeney,
European Conference on Computer Vision (ECCV) 2024, arXiv 2403.18118
Project Page
Video
Code

VideoLLM-online: Online Large Language Model for Streaming Video

Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, Mike Zheng Shou
Computer Vision and Pattern Recognition (CVPR) 2024, arXiv 2406.11816
Project Page
Code

LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing

Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, Raj Sodhi
ACM IUI 2024, arXiv 2402.10294
Project Page
Video

Aria Everyday Activities Dataset

Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, Kiran Somasundaram, Luis Pesqueira, Mark Schwesinger, Omkar Parkhi, Qiao Gu, Renzo De Nardi, Shangyi Cheng, Steve Saarinen, Vijay Baiyya, Yuyang Zou, Richard Newcombe, Jakob Julian Engel, Xiaqing Pan, Carl Ren
Project Page
Technical Report
Project Aria Tools

AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields

Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, Markus Steinberger
European Conference on Computer Vision (ECCV) 2022, arXiv 2207.10312
Project Page
Code

1st Project Aria Tutorial & Aria Pilot Dataset & Aria Data Tools

Project lead for Project Aria Tutorial Program Team ; Aria Pilot Dataset; Aria Data Tool Team
Computer Vision and Pattern Recognition (CVPR) 2022 Tutorial Program
CVPR 2022 Tutorial Program Page
Aria Pilot Dataset
Aria Data Tools
Project Aria Paper

Neural 3D Video Synthesis from Multi-view Video

Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, Zhaoyang Lv
Computer Vision and Pattern Recognition (CVPR) 2022, arXiv 2103.02597
Oral Presentation
Project Page
Dataset

STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Rendering

Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, Steven Lovegrove
Computer Vision and Pattern Recognition (CVPR) 2021, arXiv 2101.01602
Project Page

SENSE: A Shared Encoder Network for Scene-flow Estimation

Huaizu Jiang, Deqing Sun, Varun Jampani, Zhaoyang Lv, Erik Learned-Miller, Jan Kautz
International Conference in Computer Vision (ICCV) 2019 , Supplementary Materials
Oral Presentation
Code

Taking a Deeper Look at the Inverse Compositional Algorithm

Zhaoyang Lv, Frank Dellaert, James M. Rehg, Andreas Geiger
Computer Vision and Pattern Recognition (CVPR) 2019, Supplementary Materials, arXiv 1812.06861
Oral Presentation, Best Paper Finalist (<1%)
Video Slides (5 mins) , Live Recorded Video Presentation (5 mins)
Code , Poster

Multi-class Classification without Multi-class Labels

Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, Zsolt Kira
International Conference on Learning Representations (ICLR) 2019, openreview
Code

Learning to Cluster in Order to Transfer across Domains and Tasks

Yen-Chang Hsu, Zhaoyang Lv, Zsolt Kira
International Conference on Learning Representations (ICLR) 2018, arXiv:1711.10125
Code , A blog post on Machine Learning @ Gerogia Tech

Deep Image Category Discovery using a Transferred Similarity Function

Yen-Chang Hsu, Zhaoyang Lv, Zsolt Kira
arXiv:1612.01253


A Continuous Optimization Approach for Efficient and Accurate Scene Flow

Zhaoyang Lv, Chris Beall, Pablo F. Alcantarilla, Fuxin Li, Zsolt Kira, Frank Dellaert
European Conference on Computer Vision (ECCV) 2016 , arXiv 1607.07983
Project Page




miniSAM: A Flexible Factor Graph Non-linear Least Squares Optimization Framework

Jing Dong (main contributor), Zhaoyang Lv
Code , arXiv
Project Website

KinfuSeg System Image

KinfuSeg: A Dynamic SLAM Approach Based on KinectFusion

Zhaoyang Lv
Master Thesis , Imperial College London
Video Slides
Thesis Advisor: Prof. Andrew Davison
Distinguished Thesis in Department of Computing (3 among 71), Top 5%




Reality Labs Research, Redmond, Sept. 2019 - Present

Research Scientist

Nvidia Research, Santa Clara, Jan. 2019 - May 2019

Research Intern
Director: Dr. Jan Kautz, Mentors: Dr. Kihwan Kim, Dr. Deqing Sun, Dr. Alejandro Troccoli

Autonomous Vision Group, Max Planck Institute for Intelligent System, Tuebingen, June 2018 - Nov. 2018

Visiting student
Advisor: Prof. Andreas Geiger

Nvidia Research, Santa Clara, May 2017 - Aug. 2017

Research Intern
Director: Dr. Jan Kautz, Mentors: Dr. Kihwan Kim, Dr. Deqing Sun, Dr. Alejandro Troccoli

Qualcomm Research, Greater San Diego, May 2016 - Aug. 2016

Research Intern
Manager: Dr. Ali Agha

Zhejiang University, Hangzhou, Dec. 2013 - July 2014

Visiting student
Mentor: Prof. Guofeng Zhang



Instructor for CS 4476 Introduction to Computer Vision, Georgia Tech, Summer 2019


Teaching assistant for CS 7643 Deep Learning, Georgia Tech, Fall 2017

Instructor: Prof. Dhruv Batra

Teaching assistant for CS 4476 / 6476 Computer Vision, Georgia Tech, Fall 2016

Instructor: Prof. James Hays

Vice President in Public Relation for RoboGrads, Georgia Tech, Fall 2016 - Spring 2017


Multiple reviewer services for T-PAMI, IJCV, T-MM, CVPR, ICCV, ICRA, IROS

Outstanding Reviewer, CVPR 2019



Back to top

© Zhaoyang Lv. · Contact ·