Deep Neural Networks in a Mathematical Framework (eBook)

Anthony L. Caterini, Dong Eui Chang (Autoren)

eBook Download: PDF

2018
84 Seiten
Springer International Publishing (Verlag)
978-3-319-75304-1 (ISBN)

Infos

This SpringerBrief describes how to build a rigorous end-to-end mathematical framework for deep neural networks. The authors provide tools to represent and describe neural networks, casting previous results in the field in a more natural light. In particular, the authors derive gradient descent algorithms in a unified way for several neural network structures, including multilayer perceptrons, convolutional neural networks, deep autoencoders and recurrent neural networks. Furthermore, the authors developed framework is both more concise and mathematically intuitive than previous representations of neural networks.

This SpringerBrief is one step towards unlocking the black box of Deep Learning. The authors believe that this framework will help catalyze further discoveries regarding the mathematical properties of neural networks.This SpringerBrief is accessible not only to researchers, professionals and students working and studying in the field of deep learning, but alsoto those outside of the neutral network community.

Inhalt

Preface 7
Contents 10
Acronyms 12
1 Introduction and Motivation 13
1.1 Introduction to Neural Networks 14
1.1.1 Brief History 14
1.1.2 Tasks Where Neural Networks Succeed 15
1.2 Theoretical Contributions to Neural Networks 16
1.2.1 Universal Approximation Properties 16
1.2.2 Vanishing and Exploding Gradients 17
1.2.3 Wasserstein GAN 18
1.3 Mathematical Representations 19
1.4 Book Layout 19
References 20
2 Mathematical Preliminaries 23
2.1 Linear Maps, Bilinear Maps, and Adjoints 24
2.2 Derivatives 25
2.2.1 First Derivatives 25
2.2.2 Second Derivatives 26
2.3 Parameter-Dependent Maps 27
2.3.1 First Derivatives 28
2.3.2 Higher-Order Derivatives 28
2.4 Elementwise Functions 29
2.4.1 Hadamard Product 30
2.4.2 Derivatives of Elementwise Functions 31
2.4.3 The Softmax and Elementwise Log Functions 32
2.5 Conclusion 34
References 34
3 Generic Representation of Neural Networks 35
3.1 Neural Network Formulation 36
3.2 Loss Functions and Gradient Descent 37
3.2.1 Regression 37
3.2.2 Classification 38
3.2.3 Backpropagation 39
3.2.4 Gradient Descent Step Algorithm 40
3.3 Higher-Order Loss Function 41
3.3.1 Gradient Descent Step Algorithm 44
3.4 Conclusion 45
References 46
4 Specific Network Descriptions 47
4.1 Multilayer Perceptron 48
4.1.1 Formulation 48
4.1.2 Single-Layer Derivatives 49
4.1.3 Loss Functions and Gradient Descent 50
4.2 Convolutional Neural Networks 52
4.2.1 Single Layer Formulation 52
Cropping and Embedding Operators 53
Convolution Operator 55
Max-Pooling Operator 58
The Layerwise Function 61
4.2.2 Multiple Layers 62
4.2.3 Single-Layer Derivatives 62
4.2.4 Gradient Descent Step Algorithm 63
4.3 Deep Auto-Encoder 64
4.3.1 Weight Sharing 64
4.3.2 Single-Layer Formulation 65
4.3.3 Single-Layer Derivatives 66
4.3.4 Loss Functions and Gradient Descent 67
4.4 Conclusion 69
References 70
5 Recurrent Neural Networks 71
5.1 Generic RNN Formulation 71
5.1.1 Sequence Data 72
5.1.2 Hidden States, Parameters, and Forward Propagation 72
5.1.3 Prediction and Loss Functions 74
5.1.4 Loss Function Gradients 74
Prediction Parameters 75
Real-Time Recurrent Learning 76
Backpropagation Through Time 77
5.2 Vanilla RNNs 82
5.2.1 Formulation 82
5.2.2 Single-Layer Derivatives 83
5.2.3 Backpropagation Through Time 84
5.2.4 Real-Time Recurrent Learning 86
Evolution Equation 86
Loss Function Derivatives 87
Gradient Descent Step Algorithm 88
5.3 RNN Variants 88
5.3.1 Gated RNNs 89
5.3.2 Bidirectional RNNs 90
5.3.3 Deep RNNs 90
5.4 Conclusion 90
References 91
6 Conclusion and Future Work 92
References 93
Glossary 94

Produkt-Details

• Erscheint lt. Verlag: 22.3.2018
• Reihe/Serie: SpringerBriefs in Computer Science
• Reihe/Serie: SpringerBriefs in Computer Science
• Zusatzinfo: XIII, 84 p.
• Verlagsort: Cham
• Sprache: englisch
• Themenwelt: Informatik ► Theorie / Studium ► Künstliche Intelligenz / Robotik
• Schlagworte: Artificial Intelligence • autoencoder • backpropagation • convolutional neural networks • Deep learning • Gradient descent • machine learning • Multilayer Perceptron • Neural networks • Recurrent Neural Network
• ISBN-10: 3-319-75304-5 / 3319753045
• ISBN-13: 978-3-319-75304-1 / 9783319753041