Building a Data Integration Team (eBook)

Skills, Requirements, and Solutions for Designing Integrations

Jarrett Goldfedder (Autor)

eBook Download: PDF

2020 | First Edition
XXX, 237 Seiten
Apress (Verlag)
978-1-4842-5653-4 (ISBN)

Infos

Find the right people with the right skills. This book clarifies best practices for creating high-functioning data integration teams, enabling you to understand the skills and requirements, documents, and solutions for planning, designing, and monitoring both one-time migration and daily integration systems.

The growth of data is exploding. With multiple sources of information constantly arriving across enterprise systems, combining these systems into a single, cohesive, and documentable unit has become more important than ever. But the approach toward integration is much different than in other software disciplines, requiring the ability to code, collaborate, and disentangle complex business rules into a scalable model. 

Data migrations and integrations can be complicated. In many cases, project teams save the actual migration for the last weekend of the project, and any issues can lead to missed deadlines or, at worst, corrupted data that needs to be reconciled post-deployment. This book details how to plan strategically to avoid these last-minute risks as well as how to build the right solutions for future integration projects.

What You Will Learn

• Understand the 'language' of integrations and how they relate in terms of priority and ownership
• Create valuable documents that lead your team from discovery to deployment
• Research the most important integration tools in the market today
• Monitor your error logs and see how the output increases the cycle of continuous improvement
• Market across the enterprise to provide valuable integration solutions

Who This Book Is For

The executive and integration team leaders who are building the corresponding practice. It is also for integration architects, developers, and business analysts who need additional familiarity with ETL tools, integration processes, and associated project deliverables.

Jarrett Goldfedder is the founder of InfoThoughts Data, LLC, a company that specializes in data management, migration, and automation. He has significant experience in both cloud-based and on-premise technologies and holds various certificates in Salesforce Administration, Dell Boomi Architecture, and Informatica Cloud Data. He also served as a technical reviewer of the Apress book by David Masri titled Developing Data Migrations and Integrations with Salesforce: Patterns and Best Practices.

Inhalt

Table of Contents 5
About the Author 12
About the Technical Reviewer 13
Acknowledgments 14
Introduction 16
Chapter 1: Integration Background 28
The Current Landscape: Two Distinct Models 29
Waterfall Vs. Agile Models 30
The Waterfall Model 30
The Agile Model 32
Integration Process Vs. Integration Product Ownership 34
Integration Approaches 35
One-Time Migration 35
Nightly Integration 36
Scheduled Service 37
Web Service 37
Hybrid Approach 39
Chapter 2: Key Terms 40
The Simplicity of Migration 40
Source System 42
Connection Type 43
Source Endpoints 43
Metadata Model 44
Data Profiling 45
Structure Discovery 45
Content Discovery 45
Relationship Discovery 45
ETL Tool 47
Source-to-Target Mapping 47
Design Code 47
Business Rules 48
Exception Handling 48
Version Control 49
Scheduling 49
Logging 50
Target 50
Target Endpoint 51
Data Matching 51
Chapter 3: Team Qualifications 53
Team Personality 54
Types of Teams 55
Basic Team Flow 55
Functional 56
Cross-Functional 57
Virtual Team 59
Self-Directed Team 60
Team Sizes 61
Waterfall Vs. Agile Team Expectations 62
Waterfall 62
Agile 62
Best Approaches 62
The T-Shaped Person 63
Integration Team Roles 64
Project Sponsor 64
Stakeholders 64
Subject Matter Expert (SME) 64
Product Owner 65
Project Manager (PM) 65
Technical Lead 65
Data Integration Architect 65
Software Developers 66
Software Testers 66
User Acceptance Testers 66
Business Analyst 66
Release Manager 67
Recruiting Your Team 67
Determining the Roles 68
The Job Description 68
Finding Candidates 70
Internal Candidates 70
External Candidates 71
Referrals 71
Outsourcing 71
Posting to Job Boards 72
Hiring a Recruiter 72
Networking 72
Conducting Interviews 73
Summary 73
Chapter 4: Finding Your Purpose: Project Deliverables 75
Introduction 75
Project Management Phases 76
Initiation 77
Definition 77
Design Phase 77
Development 78
Implementation 78
Maintenance 80
Integration Milestones 80
Initiation 82
Definition 83
Data Migration Plan 84
The Discovery Process: Tips and Tricks 86
Learn the Architecture 86
Keep Your Session Small 86
Get System Access Early 87
Design 87
Data Profile Analysis 87
Issues Log 89
Source-to-Target Mapping 89
Purpose 89
Design 90
Usage 92
Data Migration Process Flow Design 92
Order of Operations 92
Mapping Logic 92
Deployment Steps 93
Log File Delivery 94
Migration Results 94
Final Signoff 94
ETL Scripts 95
Migration Results Report 95
Go-Live/Deployment Plan 96
Post-deployment: Maintenance and Updates 97
Summary 98
Chapter 5: Choosing an ETL Tool 100
Introduction 100
ETL Vs. ELT 100
ETL As Middleware 102
The Components of ETL: Choosing the Right Tool 103
Extraction 103
Establishing Connectors 104
Storing Temporary Data 104
Validating Inbound Data 105
Transformation 105
Modifying Data 106
Analyzing Data Integrity 106
Validating Result Data 109
Loading 110
Establishing Connectors 111
Determining Load Types 113
Rolling Back Errors 113
Tools on the Market 114
Jitterbit 115
Talend 116
Dell Boomi 117
Pentaho Data Integration (PDI) 119
Microsoft SQL Server Integration Services (SSIS) 121
MuleSoft 122
Informatica 123
Scripting Languages 125
Summary 126
Chapter 6: A Sample ETL Project 127
Introduction 127
Project Management Phases Revisited 128
Initiation 128
Definition 128
Design 133
Data Profiling 133
Source-to-Target Mapping 134
Data Migration Process Flow Design 140
Development and Implementation 141
Getting Ready 142
Clone the GitHub Directory 142
Download and Install PDI (Community Edition) 142
Opening the “create_target_table_buyer” Script 144
Opening the “Prototype” Script 146
Summary 149
Chapter 7: Platform Automation 151
Introduction 151
All About Environments 152
Why It Matters 152
Types of Environments 154
Development 154
Staging 155
Production 156
Environments for One-Time Migration 156
Environments for Repeatable Integrations 158
DevOps Practices: A Sample Deployment 159
Our Architecture 160
Preparing for Automation 162
Proposed Migration and Integration Approach 165
Demonstration 167
The Output Step 172
Rollback 174
Testing the Automation 175
Testing 176
Job Scheduling 179
Summary 185
Chapter 8: Monitoring Results 187
Introduction 187
Day One: Post-deployment 188
Level of Support 188
Common Integration Complaints and Solutions 190
The Daily Data Log 193
Updating Production 193
Week One: Post-deployment 195
Planning Your Next Steps 196
Continuous Improvement 196
Month One: Post-deployment 204
The Data Management Maturity (DMM) Model 204
Year One: Post-deployment 208
MDM 209
Summary 210
Chapter 9: Marketing Your Team 211
Introduction 211
Addressing Leadership 212
Educating Others 214
Internal Education 215
Formal Seminars 215
Workshops 216
Online Training 217
Lunch and Learns 218
External Education 220
Meetups 221
Networking Event 222
Conferences 223
The Data Integration Roadmap 224
Data Governance and the Data Governance Office 228
Summary 230
Appendix A:Data Migration Strategy Plan 231
Overview 231
Introduction 231
Purpose 231
References 232
Assumptions, Constraints, Risks, and Gaps 232
Assumptions 232
Constraints 232
Risks 233
Gaps 233
Architecture 233
Current State 233
Current State Data Sources 234
Current State Record Counts 234
Current State Data Model 234
Current State Integration Points 234
Future State 234
Future State Data Sources 234
Future State Data Model 234
Future State Integration Points 235
Development Tools 235
Environment Usage 235
Data Migration Approach 235
Scope 235
Approach 235
Team Roles and Responsibilities 236
Migration Process 236
Pre-migration Activities 236
Migration Activities 236
Post-Migration Activities 236
Contingency Plan 236
Testing/Validation 236
Testing Methods 237
Migration 237
Integration 237
Appendix B:Data Profiling Template 238
Appendix C:Issues Log Template 240
Appendix D:Source-to-Target Mapping Template 241
Appendix E:Data Migration Process Flow Design Template 244
Order of Operation 244
Mapping Logic 245
Deployment Steps 246
Description of Prerequisites 248
Contact 248
Account 248
Project Signoff/Document Delivery 248
Log File Delivery 249
Individual Delivery 249
Portal Storage 249
Migration Results 249
Contact 249
Inserts 249
Updates 250
Account 250
Final Signoff 250
Appendix F:Migration Results Report Template 251
Index 253

Produkt-Details

• Erscheint lt. Verlag: 27.2.2020
• Zusatzinfo: XXX, 237 p. 87 illus.
• Sprache: englisch
• Themenwelt: Mathematik / Informatik ► Informatik ► Theorie / Studium
• Themenwelt: Wirtschaft ► Betriebswirtschaft / Management ► Unternehmensführung / Management
• Schlagworte: Continous Improvement Model • data integration • data migration • Data Roadmaps • ETL • Extraction, Transformation, and Load • Integration Team • Platform Automation • Source to Target Mapping
• ISBN-10: 1-4842-5653-0 / 1484256530
• ISBN-13: 978-1-4842-5653-4 / 9781484256534