Bo Liu

I am a Ph.D. student at National University of Singapore in Department of Computer Science. My research interest lies in the intersection of Reinforcement Learning, Reasoning and ML Systems with their applications in complex, real-world environments.

Before that, I worked as a Research Assistant with Prof. Jun Wang. I also had the privilege of working closely with Prof. Yaodong Yang. I received my B.S. in Machine Intelligence and B.A. in Economics from Peking University in 2020, where I was advised by Prof. Zongqing Lu.

I love playing soccer in my free time. I am also open to collaborating with people to explore the possibilities of reinforcement learning in various fields. Feel free to drop me an email for collaboration!

News

May 19, 2023	One open source project TorchOpt accepted as a PyTorch Ecosystem project. Check out the blog post!
Oct 21, 2022	One paper TorchOpt: An Efficient Library for Differentiable Optimization accepted at NeurIPS 2022 Workshop OPT.
Sep 27, 2022	Two papers A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning and EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine accepted at NeurIPS 2022.
Sep 29, 2021	One paper Neural Auto-Curricula in Two-Player Zero-Sum Games accepted at NeurIPS 2021.
Dec 18, 2020	One paper Learning Correlated Communication Topology in Multi-Agent Reinforcement learning accepted at AAMAS 2021.

Selected Publications [Full List]

NeurIPS

A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning

Bo Liu*, Xidong Feng*, Jie Ren, Luo Mai, Rui Zhu, Haifeng Zhang, Jun Wang, and Yaodong Yang

The 36th Conference on Neural Information Processing Systems, 2022.

Abs arXiv PDF Code TALK

Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of O(Kα^Kσ_In|τ|^-0.5) w.r.t. inner-loop update step K, learning rate α, estimate variance σ²_In and sample size |τ|, and 2) the multi-step Hessian estimation bias ∆_H due to the use of autodiff, which has a polynomial impact O((K-1)(∆_H)^K-1) on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner’s Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.
Workshop

TorchOpt: An Efficient Library for Differentiable Optimization

Jie Ren*, Xidong Feng*, Bo Liu*, Xuehai Pan*, Yao Fu, Luo Mai, and Yaodong Yang

The 36th Conference on Neural Information Processing Systems OPT Workshop, 2022.

PyTorch Ecosystem Project

Abs arXiv PDF Blog Code TALK MEDIA

Recent years have witnessed the booming of various differentiable optimization algorithms. These algorithms exhibit different execution patterns, and their execution needs massive computational resources that go beyond a single CPU and GPU. Existing differentiable optimization libraries, however, cannot support efficient algorithm development and multi-CPU/GPU execution, making the development of differentiable optimization algorithms often cumbersome and expensive. This paper introduces TorchOpt, a PyTorch-based efficient library for differentiable optimization. TorchOpt provides a unified and expressive differentiable optimization programming abstraction. This abstraction allows users to efficiently declare and analyze various differentiable optimization programs with explicit gradients, implicit gradients, and zero-order gradients. TorchOpt further provides a high-performance distributed execution runtime. This runtime can fully parallelize computation-intensive differentiation operations (e.g. tensor tree flattening) on CPUs / GPUs and automatically distribute computation to distributed devices. Experimental results show that TorchOpt achieves 5.2× training time speedup on an 8-GPU server. TorchOpt is available at: https://github.com/metaopt/torchopt.
NeurIPS

EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine

Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan

The 36th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.

Abs arXiv PDF Code

There has been significant progress in developing reinforcement learning (RL) training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and others, aim to improve the system’s overall throughput. In this paper, we aim to address a common bottleneck in the RL training system, i.e., parallel environment execution, which is often the slowest part of the whole system but receives little attention. With a curated design for paralleling RL environments, we have improved the RL environment simulation speed across different hardware setups, ranging from a laptop and a modest workstation, to a high-end machine such as NVIDIA DGX-A100. On a high-end machine, EnvPool achieves one million frames per second for the environment execution on Atari environments and three million frames per second on MuJoCo environments. When running EnvPool on a laptop, the speed is 2.8× that of the Python subprocess. Moreover, great compatibility with existing RL training libraries has been demonstrated in the open-sourced community, including CleanRL, rl_games, DeepMind Acme, etc. Finally, EnvPool allows researchers to iterate their ideas at a much faster pace and has great potential to become the de facto RL environment execution engine. Example runs show that it only takes five minutes to train agents to play Atari Pong and MuJoCo Ant on a laptop. EnvPool is open-sourced at https://github.com/sail-sg/envpool.
NeurIPS

Neural Auto-Curricula in Two-Player Zero-Sum Games

Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang

The 35th Conference on Neural Information Processing Systems, 2021.

Abs arXiv PDF Code

When solving two-player zero-sum games, multi-agent reinforcement learning (MARL) algorithms often create populations of agents where, at each iteration, a new agent is discovered as the best response to a mixture over the opponent population. Within such a process, the update rules of "who to compete with" (i.e., the opponent mixture) and "how to beat them" (i.e., finding best responses) are underpinned by manually developed game theoretical principles such as fictitious play and Double Oracle. In this paper1, we introduce a novel framework—Neural Auto-Curricula (NAC)—that leverages meta-gradient descent to automate the discovery of the learning update rule without explicit human design. Specifically, we parameterise the opponent selection module by neural networks and the bestresponse module by optimisation subroutines, and update their parameters solely via interaction with the game engine, where both players aim to minimise their exploitability. Surprisingly, even without human design, the discovered MARL algorithms achieve competitive or even better performance with the state-of-the-art population-based game solvers (e.g., PSRO) on Games of Skill, differentiable Lotto, non-transitive Mixture Games, Iterated Matching Pennies, and Kuhn Poker. Additionally, we show that NAC is able to generalise from small games to large games, for example training on Kuhn Poker and outperforming PSRO on Leduc Poker. Our work inspires a promising future direction to discover general MARL algorithms solely from data.
AAMASOral

Learning Correlated Communication Topology in Multi-Agent Reinforcement learning

Yali Du, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang

The 20th International Conference on Autonomous Agents and Multiagent Systems, 2021.

Oral Presentation

Abs PDF TALK

Communication improves the efficiency and convergence of multiagent learning. Existing study of agent communication has been limited on predefined fixed connections. While an attention mechanism exists and is useful for scheduling the communication between agents, it, however, largely ignores the dynamical nature of communication and thus the correlation between agents’ connections. In this work, we adopt a normalizing flow to encode correlation between agents interactions. The dynamical communication topology is directly learned by maximizing the agent rewards. In our end-to-end formulation, the communication structure is learned by considering it as a hidden dynamical variable. We realize centralized training of critics and graph reasoning policy, and decentralized execution from local observation and message that are received through the learned dynamical communication topology. Experiments on cooperative navigation in the particle world and adaptive traffic control tasks demonstrate the effectiveness of our method.