Distributional Reinforcement Learning

Draft (under submission)

Marc G. Bellemare and Will Dabney and Mark Rowland

This textbook aims to provide an introduction to the developing field of distributional reinforcement learning. The version provided below is a draft, currently under review at MIT Press.

We are grateful to all the people who helped make this book a reality – a full list will be provided in the final version of the book.

Distributional Reinforcement Learning

• Table of Contents
• Preface
• 1 Introduction
• 2 The Distribution of Returns
• 3 Learning the Return Distribution
• 4 Operators and Metrics
• 5 Distributional Dynamic Programming
• 6 Incremental Algorithms
• 7 Control
• 8 Statistical Functionals
• 9 Linear Function Approximation
• 10 Deep Reinforcement Learning
• 11 Two Applications and a Conclusion
• Notation
• Bibliography

FAQ and Caveat emptor

• Can I get a PDF of this book? Why this format for the web version of the book?

  Our agreement with the publisher allows us to make the draft available, but not as a PDF. This format gives access to the work to researchers who cannot readily purchase the published book.

• When will the final version be available?

  The book is still under submission and we are actively revising it based upon your feedback. It would be jinxing things to commit to a firm publication date.

• Why are some pages strangely formatted?

  We are aware of an excess of blank space on some pages – consider this part of enjoying reading a draft copy!

• How do I provide feedback?

  We welcome feedback and questions on all parts of the book (and in particular typographical issues and technical points). The preferred mode of communication is to email us at distributionalrl@gmail.com.

Citing the book

@book{bdr2022,
    title={Distributional Reinforcement Learning},
    author={Marc G. Bellemare and Will Dabney and Mark Rowland},
    publisher={MIT Press},
    note={\url{http://www.distributional-rl.org}},
    year={2023}
}