Design of Experiments for Reinforcement Learning (eBook)

Christopher Gatti (Autor)

eBook Download: PDF

2014 | 2015
XIII, 191 Seiten
Springer International Publishing (Verlag)
9783319121970 (ISBN)

Infos

This thesis takes an empirical approach to understanding of the behavior and interactions between the two main components of reinforcement learning: the learning algorithm and the functional representation of learned knowledge. The author approaches these entities using design of experiments not commonly employed to study machine learning methods. The results outlined in this work provide insight as to what enables and what has an effect on successful reinforcement learning implementations so that this learning method can be applied to more challenging problems.

Christopher Gatti received his PhD in Decision Sciences and Engineering Systems from Rensselaer Polytechnic Institute (RPI). During his time at RPI, his work focused on machine learning and statistics, with applications in reinforcement learning, graph search, stem cell RNA analysis, and neuro-electrophysiological signal analysis. Prior to beginning his graduate work at RPI, he received a BSE in mechanical engineering and an MSE in biomedical engineering, both from the University of Michigan. He then continued to work at the University of Michigan for three years doing computational biomechanics focusing on the shoulder and knee. He has been a gymnast since he was a child and is currently an acrobat for Cirque du Soleil.

Inhalt

Foreword 6
Acknowledgment 8
Book Note 9
Parts of this Thesis Have Been Published As 9
Contents 10
Chapter 1 Introduction 13
References 16
Chapter 2 Reinforcement Learning 18
2.1 Applications of Reinforcement Learning 22
2.1.1 Benchmark Problems 22
2.1.2 Games 25
2.1.3 Real-World Applications 25
2.1.4 Generalized Domains 27
2.2 Components of Reinforcement Learning 28
2.2.1 Domains 28
2.2.1.1 General Characteristics 29
2.2.1.2 State Space Dimensions 31
2.2.1.3 Action Space Dimensions 33
2.2.1.4 Reward Dimension 33
2.2.1.5 State Encoding-Dependent Characteristics 34
2.2.2 Representations 34
2.2.2.1 Look-up Tables 35
2.2.2.2 Linear Methods 36
2.2.2.3 Neural Networks 37
2.2.3 Learning Algorithms 40
2.2.3.1 Policy Evaluation Approaches 47
2.2.3.2 Learning Algorithm Convergence 48
2.2.3.3 Additional Reinforcement Learning Algorithms 48
2.3 Heuristics and Performance Effectors 49
2.3.1 Heuristics for Reinforcement Learning 49
2.3.1.1 Effectors of Reinforcement Learning Performance 50
References 53
Chapter 3 Design of Experiments 64
3.1 Classical Design of Experiments 66
3.2 Contemporary Design of Experiments 70
3.3 Design of Experiments for Empirical Algorithm Analysis 74
References 75
Chapter 4 Methodology 78
4.1 Sequential CART 78
4.1.1 CART Modeling 79
4.1.2 Sequential CART Modeling 80
4.1.3 Analysis of Sequential CART 86
4.1.4 Empirical Convergence Criteria 87
4.1.5 Example: 2-D 6-hump Camelback Function 89
4.2 Kriging Metamodeling 93
4.2.1 Kriging 94
4.2.2 Deterministic Kriging 95
4.2.3 Stochastic Kriging 96
4.2.4 Covariance Function 97
4.2.5 Implementation 99
4.2.6 Analysis of Kriging Metamodels 100
References 103
Chapter 5 The Mountain Car Problem 105
5.1 Reinforcement Learning Implementation 105
5.2 Sequential CART 107
5.2.1 Convergent Subregions 108
5.3 Response Surface Metamodeling 111
5.4 Discussion 117
References 119
Chapter 6 The Truck Backer-upper Problem 120
6.1 Reinforcement Learning Implementation 121
6.2 Sequential CART 123
6.2.1 Convergent Subregions 125
6.3 Response Surface Metamodeling 129
6.4 Discussion 131
References 135
Chapter 7 The Tandem Truck Backer-Upper Problem 137
7.1 Reinforcement Learning Implementation 139
7.2 Sequential CART 141
7.2.1 Convergent Subregions 142
7.3 Discussion 145
References 147
Chapter 8 Discussion 148
8.1 Reinforcement Learning 148
8.1.1 Parameter Effects 149
8.1.2 Neural Network 152
8.2 Experimentation 153
8.2.1 Sequential CART 155
8.2.2 Stochastic Kriging 156
8.3 Innovations 157
8.4 Future Work 159
References 161
Appendix A Parameter Effects in the Game of Chung Toi 164
A.1 Introduction 164
A.2 Methodology 165
A.2.1 Chung Toi 165
A.2.2 The Reinforcement Learning Method 165
A.2.3 The Environment Model 166
A.2.4 The Agent Model 167
A.2.5 Training and Performance Evaluation Methods 168
A.2.6 Experiments 169
A.3 Results 171
A.3.1 Individual Experiments 171
A.3.2 Optimal Experiments 174
A.4 Discussion 175
A.5 Conclusion 176
References 176
Appendix B Design of Experiments for the Mountain Car Problem 178
B.1 Introduction 178
B.2 Methodology 179
B.2.1 Mountain Car Domain 179
B.2.2 Agent Representation 179
B.2.3 Experimental Design and Analysis 181
B.3 Results 181
B.4 Discussion 183
References 184
Appendix C Supporting Tables 185
Glossary 194

Produkt-Details

• Erscheint lt. Verlag: 22.11.2014
• Reihe/Serie: Springer Theses
• Reihe/Serie: Springer Theses
• Zusatzinfo: XIII, 191 p. 46 illus., 25 illus. in color.
• Verlagsort: Cham
• Sprache: englisch
• Themenwelt: Informatik ► Theorie / Studium ► Künstliche Intelligenz / Robotik
• Themenwelt: Technik
• Schlagworte: Kriging Covariance Functions • Reinforcement Learning Algorithm • Response Surface Metamodeling • Sequential CART • Stochastic Kriging
• ISBN-13: 9783319121970 / 9783319121970