REINFORCEMENT LEARNING AND STOCHASTIC OPTIMIZATION
Clearing the jungle of stochastic optimization
Sequential decision problems, which consist of “decision, information, decision, information,” are ubiquitous, spanning virtually every human activity ranging from business applications, health (personal and public health, and medical decision making), energy, the sciences, all fields of engineering, finance, and e-commerce. The diversity of applications attracted the attention of at least 15 distinct fields of research, using eight distinct notational systems which produced a vast array of analytical tools. A byproduct is that powerful tools developed in one community may be unknown to other communities.
Reinforcement Learning and Stochastic Optimization offers a single canonical framework that can model any sequential decision problem using five core components: state variables, decision variables, exogenous information variables, transition function, and objective function. This book highlights twelve types of uncertainty that might enter any model and pulls together the diverse set of methods for making decisions, known as policies, into four fundamental classes that span every method suggested in the academic literature or used in practice.
Reinforcement Learning and Stochastic Optimization is the first book to provide a balanced treatment of the different methods for modeling and solving sequential decision problems, following the style used by most books on machine learning, optimization, and simulation. The presentation is designed for readers with a course in probability and statistics, and an interest in modeling and applications. Linear programming is occasionally used for specific problem classes. The book is designed for readers who are new to the field, as well as those with some background in optimization under uncertainty.
Throughout this book, readers will find references to over 100 different applications, spanning pure learning problems, dynamic resource allocation problems, general state-dependent problems, and hybrid learning/resource allocation problems such as those that arose in the COVID pandemic. There are 370 exercises, organized into seven groups, ranging from review questions, modeling, computation, problem solving, theory, programming exercises and a “diary problem” that a reader chooses at the beginning of the book, and which is used as a basis for questions throughout the rest of the book.
Warren B. Powell –
There are at least 15 distinct communities that work on sequential decision problems (dynamic programming, stochastic programming, stochastic search, optimal control, active learning, approximate dynamic programming and reinforcement learning are some of the most popular). Each of these fields deal with sequential decisions, using different notational systems and a wide variety of algorithmic strategies. This is the first book to provide a single modeling framework for any sequential decision problem. Decisions are made by methods known as policies, and while there are countless methods proposed in the literature, the book organizes them into four fundamental classes (plus hybrids). This book is suited for people looking to write software and solve real problems, ranging from optimal stopping problems, discrete choice problems, or complex resource allocation problems.
sambuddha chakrabarti –
I have been following Prof. Warren Powell’s work for almost 5 years now and needless to say, I am impressed by his approach to RL and Sequential decision making under uncertainty. This book is a nice compilation of this approach, which provides a through exposure to the reader. Very excited to get the book in hand.
Vincent Granville –
This book is for operations research professionals (decision makers in the field). My background is machine learning, and the content is too far away from my research. Though I use simulations and stochastic processes a lot, I found the book to be targeted to an audience I don’t really belong to.
I would have liked deeper dives into probabilistic models, and less focus on decision processes. That said, the book offers a nice summary of many modern machine learning methods. I have no doubt it is a great book for the right audience. The price is a bit high. On the plus side, it was delivered on time, and in perfect shape. Definitely very well written.
My 4 stars is because:
1) It is worth 3 stars to me since it is not relevant to my current research. But that’s my fault: I should have read the table of content more carefully before buying it.
2) I have no doubt that for the right audience (senior, decision maker, fully into operations research), it is worth 5 stars.
OptimAIzer –
This is a great reference book and a highly professional framework for sequential decisions under uncertainty.
Nathan Yang –
An incredibly comprehensive book about approximate dynamic programming and reinforcement learning. I’ll be sure to assign this book for the PhD courses I teach.
Darshan Chauhan –
Very well written book. Incredibly thorough yet easy to understand. It will be a book that I go over and over again through time :))
Anand houston –
I’m a self-taught programmer looking to create practical applications with deep reinforcement learning. I come from the business side of things and was looking for something that could me better understand how to apply sequential decision processes to real world applications. This book as more than overdelivered. I appreciate how thorough the author is while also making it more accessible to people outside of the academic world. Thank you for this spectacular guide!
Samuel Chen –
Very comprehensive book! All important concepts in sequential decision making is mentioned.
Math Customer –
Es un libro que puede parecer intimidante, pero creo que es apropiado dado que cubre, a gusto del autor, “una metodología unificada para resolver cualquier problema de optimización secuencial”.Divide los problemas de optimización en cuatro clases: PFA (policy function approximations), CFA (cost function approximations), VFA (value function approximations) y DLA (direct look ahead approximations).Los ejercicios son valiosos e ilustrativos, y recomendaría también su libro de “Sequential Analytics and Modeling”, que es mas sencillo y orientado a aplicaciones reales.