Smart Grid using Big Data Analytics (eBook)
John Wiley & Sons (Verlag)
978-1-118-71680-9 (ISBN)
Robert Caiming Qiu, Professor, Dept. of ECE, Tennessee Technological University, Cookeville, TN, USA. Professor Qiu was Founder-CEO and President of Wiscom Technologies, Inc., manufacturing and marketing WCDMA chipsets. Wiscom was acquired by Intel in 2003. Prior to Wiscom, he worked for GTE Labs, Inc. (now Verizon), Waltham, MA, and Bell Labs, Lucent, Whippany, NJ. He holds 5 U.S. patents (another two pending) in WCDMA. Professor Qiu has contributed to 3GPP and IEEE standards bodies, and delivered invited seminars to institutions including Princeton University and the U.S. Army Research Lab. Dr. Qiu was made an IEEE Fellow in 2014.
Dr. Paul Antonik, Chief Scientist, Information Directorate, Air Force Research Laboratory, Rome, N.Y., USA. Dr. Antonik serves as the directorate's principal scientific and technical adviser and primary authority for the technical content of the science and technology portfolio, providing principal technical oversight of a broad spectrum of information technologies.
This book is aimed at students in communications and signal processing who want to extend their skills in the energy area. It describes power systems and why these backgrounds are so useful to smart grid, wireless communications being very different to traditional wireline communications.
Robert Caiming Qiu, Professor, Dept. of ECE, Tennessee Technological University, Cookeville, TN, USA. Professor Qiu was Founder-CEO and President of Wiscom Technologies, Inc., manufacturing and marketing WCDMA chipsets. Wiscom was acquired by Intel in 2003. Prior to Wiscom, he worked for GTE Labs, Inc. (now Verizon), Waltham, MA, and Bell Labs, Lucent, Whippany, NJ. He holds 5 U.S. patents (another two pending) in WCDMA. Professor Qiu has contributed to 3GPP and IEEE standards bodies, and delivered invited seminars to institutions including Princeton University and the U.S. Army Research Lab. Dr. Qiu was made an IEEE Fellow in 2014. Dr. Paul Antonik, Chief Scientist, Information Directorate, Air Force Research Laboratory, Rome, N.Y., USA. Dr. Antonik serves as the directorate's principal scientific and technical adviser and primary authority for the technical content of the science and technology portfolio, providing principal technical oversight of a broad spectrum of information technologies.
Title Page 5
Copyright Page 6
Contents 9
Preface 17
Acknowledgments 21
Some Notation 23
Chapter 1 Introduction 25
1.1 Big Data: Basic Concepts 25
1.1.1 Big Data—Big Picture 25
1.1.2 DARPA’s XDATA Program 27
1.1.3 National Science Foundation 29
1.1.4 Challenges and Opportunities with Big Data 29
1.1.5 Signal Processing and Systems Engineering for Big Data 30
1.1.6 Large Random Matrices for Big Data 32
1.1.7 Big Data Across the US Federal Government 32
1.2 Data Mining with Big Data 33
1.3 AMathematical Introduction to Big Data 37
1.4 AMathematical Theory of Big Data 52
1.4.1 Boltzmann Entropy and H-Theorem 54
1.4.2 Shannon Entropy and Classical Information Theory 55
1.4.3 Dan-Virgil Voiculescu and Free Central Limit Theorem 55
1.4.4 Free Entropy 55
1.4.5 Jean Ginibre and his Ensemble of Non-Hermitian Random Matrices 56
1.4.6 Circular Law for the Complex Ginibre Ensemble 57
1.5 Smart Grid 58
1.6 Big Data and Smart Grid 60
1.7 Reading Guide 61
Bibliographical Remarks 63
Part I Fundamentals of Big Data 65
Chapter 2 The Mathematical Foundations of Big Data Systems 67
2.1 Big Data Analytics 68
2.2 Big Data: Sense, Collect, Store, and Analyze 69
2.2.1 Data Collection 70
2.2.2 Data Cleansing 70
2.2.3 Data Representation and Modeling 71
2.2.4 Data Analysis 71
2.2.5 Data Storage 72
2.3 Intelligent Algorithms 72
2.4 Signal Processing for Smart Grid 72
2.5 Monitoring and Optimization for Power Grids 72
2.6 Distributed Sensing and Measurement for Power Grids 73
2.7 Real-time Analysis of Streaming Data 74
2.8 Salient Features of Big Data 75
2.8.1 Singular Value Decomposition and Random Matrix Theory 75
2.8.2 Heterogeneity 76
2.8.3 Noise Accumulation 77
2.8.4 Spurious Correlation 77
2.8.5 Incidental Endogeneity 78
2.8.6 Impact on Computational Methods 78
2.9 BigData forQuantumSystems 78
2.10 Big Data for Financial Systems 79
2.10.1 Methodology 79
2.10.2 Marchenko–Pastur Law for Equal Time Correlations 82
2.10.3 Symmetrized Time-Lagged Correlation Matrices 83
2.10.4 Asymmetric Time-Lagged Correlation Matrices 85
2.10.5 Noise Reduction 86
2.10.6 Power-Law Tails 87
2.10.7 Free Random Variables 89
2.10.8 Cross-Correlations between Input and Output Variables 94
2.11 Big Data for Atmospheric Systems 97
2.12 Big Data for Sensing Networks 98
2.13 Big Data forWireless Networks 99
2.13.1 Marchenko–Pastur Law 99
2.13.2 The Single “Ring” Law 100
2.13.3 Experimental Results 100
2.14 Big Data for Transportation 102
Bibliographical Remarks 102
Chapter 3 Large Random Matrices: An Introduction 103
3.1 Modeling of Large Dimensional Data as RandomMatrices 103
3.2 A Brief of Random Matrix Theory 105
3.3 Change Point of Views: FromVectors to Measures 109
3.4 The Stieltjes Transform of Measures 110
3.5 A Fundamental Result: The Marchenko–Pastur Equation 112
3.6 Linear Eigenvalue Statistics and Limit Laws 113
3.7 Central Limit Theoremfor Linear Eigenvalue Statistics 123
3.8 Central Limit Theoremfor Random Matrix S?1T 125
3.9 Independence for Random Matrices 127
3.10 Matrix-Valued Gaussian Distribution 134
3.11 Matrix-ValuedWishart Distribution 136
3.12 Moment Method 136
3.13 Stieltjes Transform Method 137
3.14 Concentration of the Spectral Measure for Large Random Matrices 138
3.15 Future Directions 141
Bibliographical Remarks 141
Chapter 4 Linear Spectral Statistics of the Sample Covariance Matrix 145
4.1 Linear Spectral Statistics 145
4.2 GeneralizedMarchenko–Pastur Distributions 146
4.2.1 Central Limit Theorem 147
4.2.2 Spiked Population Models 150
4.2.3 Generalized Spiked Population Model 150
4.3 Estimation of Spectral Density Functions 151
4.3.1 Estimation Method 152
4.3.2 Kernel Estimator of the Limiting Spectral Distribution 154
4.3.3 Central Limit Theorems for Kernel Estimators 164
4.3.4 Estimation of Noise Variance 167
4.4 Limiting Spectral Distribution of Time Series 170
4.4.1 Vector Autoregressive Moving Average (VARMA) Models 170
4.4.2 General Linear Process 171
4.4.3 Large Sample Covariance Matrices for Linear Processes 173
4.4.4 Stationary Processes 173
4.4.5 Symmetrized Auto-cross Covariance Matrix 175
4.4.6 Large Sample Covariance Matrices with Heavy Tails 176
Bibliographical Remarks 178
Chapter 5 Large Hermitian Random Matrices and Free Random Variables 179
5.1 Large Economic/Financial Systems 180
5.2 Matrix-Valued Probability 181
5.2.1 Eigenvalue Spectra for the Covariance Matrix and its Estimator 183
5.3 Wishart-Levy Free Stable Random Matrices 190
5.4 Basic Concepts for Free Random Variables 192
5.5 The Analytical Spectrum of theWishart–Levy Random Matrix 196
5.6 Basic Properties of the Stieltjes Transform 200
5.7 Basic Theorems for the Stieltjes Transform 203
5.8 Free Probability for Hermitian Random Matrices 209
5.8.1 Random Matrix Theory 209
5.8.2 Free Probability Theory for Hermitian Random Matrices 211
5.8.3 Additive Free Convolution 212
5.8.4 Compression of Random Matrix 216
5.8.5 Multiplicative Free Convolution 217
5.9 Random Vandermonde Matrix 220
5.10 Non-Asymptotic Analysis of State Estimation 224
Bibliographical Remarks 225
Chapter 6 Large Non-Hermitian Random Matrices and Quatartenionic Free Probability Theory 227
6.1 Quatartenionic Free Probability Theory 228
6.1.1 Stieltjes Transform 229
6.1.2 Additive Free Convolution 230
6.1.3 Multiplicative Free Convolution 231
6.1.4 Quaternion-valued Functions for Hermitian Matrices 231
6.2 R-diagonal Matrices 233
6.2.1 Classes of R-diagonal Matrices 233
6.2.2 Additive Free Convolution 234
6.2.3 Multiplicative Free Convolution 235
6.2.4 Isotropic Random Matrices 239
6.3 The Sum of Non-Hermitian Random Matrices 240
6.4 The Product of Non-Hermitian Random Matrices 243
6.5 Singular Value Equivalent Models 250
6.6 The Power of the Non-Hermitian Random Matrix 258
6.6.1 The Matrix Power 258
6.6.2 Spectrum 258
6.6.3 The Product 260
6.7 Power Series of Large Non-Hermitian Random Matrices 263
6.7.1 The Geometric Series 264
6.7.2 Power Series 265
6.8 Products of Random Ginibre Matrices 270
6.9 Products of Rectangular Gaussian Random Matrices 273
6.10 Product of ComplexWishart Matrices 276
6.11 Spectral Relations between Products and Powers 278
6.12 Products of Finite-Size I.I.D. Gaussian Random Matrices 282
6.13 Lyapunov Exponents for Products of Complex Gaussian Random Matrices 284
6.14 Euclidean Random Matrices 288
6.15 Random Matrices with Independent Entries and the Circular Law 297
6.16 The Circular Law and Outliers 299
6.17 Random SVD, Single Ring Law, and Outliers 309
6.17.1 Outliers for Finite Rank Perturbation: Proof of Theorem6.17.3 316
6.17.2 Eigenvalues Inside the Inner Circle: Proof of Theorem6.17.4 318
6.18 The Elliptic Law and Outliers 319
Bibliographical Remarks 329
Chapter 7 The Mathematical Foundations of Data Collection 331
7.1 Architectures and Applications for Big Data 331
7.2 Covariance Matrix Estimation 332
7.3 Spectral Estimators for Large Random Matrices 336
7.3.1 Singular Value Thresholding 337
7.3.2 Stein’s Unbiased Risk Estimate (SURE) 337
7.3.3 Extensions to Spectral Functions 340
7.3.4 Regularized Principal Component Analysis 342
7.4 Asymptotic Framework for Matrix Reconstruction 343
7.4.1 Matrix Estimation with Loss Functions 343
7.4.2 Connection with Large Random Matrices 346
7.4.3 Asymptotic Matrix Reconstruction 348
7.4.4 Estimation of the Noise Variance 349
7.4.5 Optimal Hard Threshold for Matrix Denoising 351
7.5 Optimum Shrinkage 353
7.6 A Shrinkage Approach to Large-Scale Covariance Matrix Estimation 355
7.7 Eigenvectors of Large Sample Covariance MatrixEnsembles 362
7.7.1 Stieltjes Transform 362
7.7.2 Sample versus Population Eigenvectors 365
7.7.3 Asymptotically Optimal Bias Correction for the Sample Eigenvalues 367
7.7.4 Estimating Precision Matrices 370
7.8 A General Class of Random Matrices 375
7.8.1 Massive MIMO System 379
Bibliographical Remarks 383
Chapter 8 Matrix Hypothesis Testing using Large Random Matrices 385
8.1 Motivating Examples 386
8.2 Hypothesis Test of Two Alternative Random Matrices 387
8.3 Eigenvalue Bounds for Expectation and Variance 388
8.3.1 Theoretical Locations of Eigenvalues 390
8.3.2 Wasserstein Distance 390
8.3.3 Sample Covariance Matrices—Entries with Exponential Decay 391
8.3.4 Gaussian Covariance Matrices 392
8.4 Concentration of Empirical Distribution Functions 393
8.4.1 Poincare-Type Inequalities, Tensorization 396
8.4.2 Empirical Poincare-Type Inequalities 397
8.4.3 Concentration of Random Matrices 401
8.5 Random Quadratic Forms 405
8.6 Log-Determinant of Random Matrices 406
8.7 GeneralMANOVA Matrices 407
8.8 Finite Rank Perturbations of Large Random Matrices 410
8.8.1 Non-asymptoic, Finite-Sample Theory 414
8.9 Hypothesis Tests for High-Dimensional Datasets 415
8.9.1 Motivation for Likelihood Ratio Test (LRT) and Covariance Matrix Tests 416
8.9.2 Estimation of Covariance Matrices Using Loss Functions 418
8.9.3 Covariance Matrix Tests 423
8.9.4 Optimal Hypothesis Testing for High-Dimensional Covariance Matrices 428
8.9.5 Sphericity Test 432
8.9.6 Testing Equality ofMultiple Covariance Matrices of Normal Distributions 434
8.9.7 Testing Independence of Components of Normal Distribution 437
8.9.8 Test of Mutual Dependence 440
8.9.9 Test of Presence of Spike Eigenvalues 444
8.9.10 Large Dimension and Small Sample Size 446
8.10 Roy’s Largest Root Test 452
8.11 Optimal Tests of Hypotheses for Large Random Matrices 455
8.12 Matrix Elliptically Contoured Distributions 468
8.13 Hypothesis Testing for Matrix Elliptically Contoured Distributions 470
8.13.1 General Results 470
8.13.2 Two Models 472
8.13.3 Testing Criteria 474
Bibliographical Remarks 476
Part II Smart Grid 479
Chapter 9 Applications and Requirements of Smart Grid 481
9.1 History 481
9.2 Concepts and Vision 482
9.3 Today’s Electric Grid 483
9.4 Future Smart Electrical Energy System 488
Chapter 10 Technical Challenges for Smart Grid 495
10.1 The Conceptual Foundation of a Self-Healing Power System 495
10.2 How toMake an Electric Power Transmission System Smart 496
10.3 The Electric Power System as a Complex Adaptive System 497
10.4 Making the Power System a Self-Healing Network Using Distributed Computer Agents 498
10.5 Distribution Grid 498
10.6 Cyber Security 500
10.7 Smart Metering Network 501
10.8 Communication Infrastructure for Smart Grid 502
10.9 Wireless Sensor Networks 504
Bibliographical Remarks 507
Chapter 11 Big Data for Smart Grid 509
11.1 Power in Numbers: Big Data and Grid Infrastructure 509
11.2 Energy’s Internet: The Convergence of Big Data and the Cloud 510
11.3 Edge Analytics: Consumers, Electric Vehicles, and Distributed Generation 510
11.4 Crosscutting Themes: Big Data 510
11.5 Cloud Computing for Smart Grid 512
11.6 Data Storage, Data Access and Data Analysis 512
11.7 The State-of-the-Art Processing Techniques of Big Data 512
11.8 Big DataMeets the Smart Electrical Grid 512
11.9 4Vs of Big Data: Volume, Variety, Value and Velocity 513
11.10 Cloud Computing for Big Data 514
11.11 Big Data for Smart Grid 514
11.12 Information Platforms for Smart Grid 515
Bibliographical Remarks 515
Chapter 12 Grid Monitoring and State Estimation 517
12.1 Phase Measurement Unit 517
12.1.1 Classical Definition of a Phasor 518
12.1.2 Phasor Measurement Concepts 518
12.1.3 Synchrophasor Definition and Measurements 518
12.2 Optimal PMU Placement 519
12.3 State Estimation 519
12.4 Basics of State Estimation 519
12.5 Evolution of State Estimation 520
12.6 Static State Estimation 521
12.7 Forecasting-Aided State Estimation 524
12.8 Phasor Measurement Units 525
12.9 Distributed System State Estimation 526
12.10 Event-Triggered Approaches to State Estimation 526
12.11 Bad Data Detection 526
12.12 Improved Bad Data Detection 527
12.13 Cyber-Attacks 528
12.14 Line Outage Detection 528
Bibliographical Remarks 528
Chapter 13 False Data Injection Attacks against State Estimation 529
13.1 State Estimation 529
13.2 False Data Injection Attacks 531
13.2.1 Basic Principle 531
13.3 MMSE State Estimation and Generalized Likelihood Ratio Test 532
13.3.1 A Bayesian Framework and MMSE Estimation 533
13.3.2 Statistical Model and Attack Hypotheses 533
13.3.3 Generalized Likelihood Ratio Detector with ??????1-Norm Regularization 534
13.3.4 Classical Detectors with MMSE State Estimation 535
13.3.5 Optimal Attacks for the MMSE and the GLRT Detector 535
13.4 Sparse Recovery from Nonlinear Measurements 536
13.4.1 Bad Data Detection for Linear Systems 537
13.4.2 Bad Data Detection for Nonlinear Systems 538
13.5 Real-Time Intrusion Detection 539
Bibliographical Remarks 539
Chapter 14 Demand Response 541
14.1 Why Engage Demand? 541
14.2 Optimal Real-time Pricing Algorithms 544
14.3 Transportation Electrification and Vehicle-to-Grid Applications 546
14.4 Grid Storage 546
Bibliographical Remarks 547
Part III Communications and Sensing 549
Chapter 15 Big Data for Communications 551
15.1 5G and Big Data 551
15.2 5GWireless Communication Networks 551
15.3 Massive Multiple Input, Multiple Output 552
15.3.1 Multiuser-MIMO System Model 552
15.3.2 Very Long Random Vectors 554
15.3.3 Favorable Propagation 554
15.3.4 Precoding Techniques 556
15.3.5 Downlink System Model 557
15.3.6 Random Matrix Theory 558
15.4 Free Probability for the Capacity of the Massive MIMO Channel 561
15.4.1 Nonasymptotic Theory: Concentration Inequalities 561
15.5 Spectral Sensing for Cognitive Radio 563
Bibliographical Remarks 563
Chapter 16 Big Data for Sensing 565
16.1 Distributed Detection and Estimation 565
16.1.1 Computing while Communicating 565
16.1.2 Distributed Detection 566
16.1.3 Distributed Estimation 567
16.1.4 Consensus Algorithms 568
16.1.5 Random Geometric Graph with Euclidean Random Matrix (ERM) 570
16.2 Euclidean Random Matrix 571
16.3 Decentralized Computing 572
Appendix A: Some Basic Results on Free Probability 575
A.1 Non-Commutative Probability Spaces 575
A.2 Distributions 576
A.3 Asymptotic Freeness of Large Random Matrices 577
A.4 Limit Theorems 577
A.5 R-diagonal Random Variables 578
A.6 Brown Measure of R-diagonal Random Variables 578
Appendix B: Matrix-Valued Random Variables 581
B.1 Random Vectors and Random Matrices 581
B.2 Multivariate Normal Distribution 583
B.3 Wishart Distribution 584
B.3.1 CentralWishart Distribution 584
B.3.2 NoncentralWishart Distribution 585
B.4 Multivariate Linear Model 586
B.5 General Linear Hypothesis Testing 587
Bibliographical Remarks 588
References 591
Index 625
EULA 629
| Erscheint lt. Verlag | 23.1.2017 |
|---|---|
| Sprache | englisch |
| Themenwelt | Mathematik / Informatik ► Informatik ► Datenbanken |
| Technik ► Elektrotechnik / Energietechnik | |
| Technik ► Nachrichtentechnik | |
| Schlagworte | Big Data • Electrical & Electronics Engineering • Elektrotechnik u. Elektronik • Energie • Energy • Numerical Methods & Algorithms • Numerische Methoden u. Algorithmen • Smart Grid • smart grid, big data analytics, electrical engineering, algorithms • Systems Engineering & Management • Systemtechnik • Systemtechnik u. -management |
| ISBN-10 | 1-118-71680-9 / 1118716809 |
| ISBN-13 | 978-1-118-71680-9 / 9781118716809 |
| Informationen gemäß Produktsicherheitsverordnung (GPSR) | |
| Haben Sie eine Frage zum Produkt? |
PDF (Adobe DRM)Kopierschutz: Adobe-DRM
Adobe-DRM ist ein Kopierschutz, der das eBook vor Mißbrauch schützen soll. Dabei wird das eBook bereits beim Download auf Ihre persönliche Adobe-ID autorisiert. Lesen können Sie das eBook dann nur auf den Geräten, welche ebenfalls auf Ihre Adobe-ID registriert sind.
Details zum Adobe-DRM
Dateiformat: PDF (Portable Document Format)
Mit einem festen Seitenlayout eignet sich die PDF besonders für Fachbücher mit Spalten, Tabellen und Abbildungen. Eine PDF kann auf fast allen Geräten angezeigt werden, ist aber für kleine Displays (Smartphone, eReader) nur eingeschränkt geeignet.
Systemvoraussetzungen:
PC/Mac: Mit einem PC oder Mac können Sie dieses eBook lesen. Sie benötigen eine
eReader: Dieses eBook kann mit (fast) allen eBook-Readern gelesen werden. Mit dem amazon-Kindle ist es aber nicht kompatibel.
Smartphone/Tablet: Egal ob Apple oder Android, dieses eBook können Sie lesen. Sie benötigen eine
Geräteliste und zusätzliche Hinweise
Buying eBooks from abroad
For tax law reasons we can sell eBooks just within Germany and Switzerland. Regrettably we cannot fulfill eBook-orders from other countries.
aus dem Bereich
