Mastering Data Integration With Ibm Datastage

Unlock the Power of Data Integration: Practical Training with IBM DataStage (ETL)

What you'll learn

Fundamentals of Data Integration: Understand the core concepts and types of data integration and explore real-world examples.

Navigating IBM Information Server: Get acquainted with the components of IBM Information Server and its role in data integration.

IBM Information Server Administration: Learn to navigate the IBM Information Server Administration Console and practice essential administrative tasks.

Exploring IBM DataStage: Dive into the architecture of IBM DataStage, its key components, and practical uses

Developing in IBM DataStage: Work hands-on in DataStage, create projects, explore job types, and utilize design elements for parallel processing.

DataStage Administration: Acquire practical skills in DataStage administration, including user management, permissions, and environment variables.

Metadata Management: Practice metadata management using DataStage Designer, importing, and exporting components.

Creating Parallel Jobs: Engage in practical sessions to create parallel jobs, define parameters, and document your jobs effectively.

Accessing Sequential Data: Hands-on experience in handling sequential data, utilizing the Sequential File stage, and managing reject links.

Implementing Partitioning and Collecting Algorithms: Gain practical insights into partition parallelism, partitioning algorithms, and collecting strategies.

Combining Data with Stages: Work with Lookup, Join, Merge, and Funnel stages, and practice their applications in real-world scenarios.

Group Processing Stages: Learn to sort data effectively, remove duplicates, and utilize Aggregator stages in practical exercises.

Transforming Data: Practice using the Transformer stage, constraints, and debugging techniques for data transformation.

Repository Functions: Explore practical aspects of using the repository, finding differences between jobs, and performing impact analyses.

Working with Relational Data: Engage in hands-on activities involving connector stages, reading and writing to database tables, and utilizing data connection ob

Job Sequence Control: Gain practical experience in creating job sequences, defining triggers, and managing job activities through various stages.

Real Practice: AWS Cloud Integration: Apply your skills to integrate data with AWS Cloud services in real-world scenarios.

Real Practice: Data Vault 1.0 & 2.0 Integration: Practical exercises in integrating Data Vault concepts into your data integration projects.

Requirements

Basic Understanding of Data Concepts: A fundamental grasp of data concepts is recommended. Students should understand terms like data sources, data transformation, and data loading.

SQL Knowledge (Optional): While not mandatory, having some familiarity with SQL (Structured Query Language) can be beneficial, especially when working with relational databases.

Access to IBM DataStage: Ideally, students should have access to IBM DataStage software to practice and follow along with the course.

IBM DataStage Software (Optional): If students want to practice the skills learned in the course, having access to IBM DataStage software is beneficial

Desire to Learn: A genuine interest in data integration and a willingness to learn and practice the concepts taught in the course are essential.

Description

Unlock the power of data integration with IBM DataStage, the industry-leading ETL (Extract, Transform, Load) tool. In this comprehensive course, you'll embark on a journey from data integration basics to advanced techniques, empowering you to harness the full potential of your data.What You'll Learn:Foundations of Data Integration: Begin by understanding the core concepts and types of data integration, laying a strong foundation for your journey.IBM Information Server: Explore the IBM Information Server ecosystem and its vital components to comprehend where DataStage fits in.Hands-On Administration: Get hands-on with DataStage administration tasks, managing users, roles, and permissions with ease.Mastering Metadata: Learn to work effectively with metadata, a crucial aspect of data integration, to streamline your processes.Parallel Jobs Creation: Dive into parallel job creation, understand its intricacies, and design efficient parallel jobs.Accessing Sequential Data: Master the art of accessing sequential data, a crucial skill in data integration.Advanced Algorithms: Explore partitioning and collecting algorithms, vital for efficient data processing.Combine Data Effectively: Get comfortable with stages like Lookup, Join, Merge, and Funnel to combine data seamlessly.Group Processing Stages: Learn to group process data, sort it, and aggregate it effectively.Transformer Stage: Dive deep into the Transformer stage and its capabilities for data transformation.Repository Functions: Understand repository functions, impact analysis, and how to compare different jobs.Relational Data Integration: Work with relational data using connector stages, read from and write to database tables.Job Sequence Control: Master job sequencing, control the flow of jobs, and create complex workflows.Real-world Practice: Apply your knowledge in real-world scenarios with practical AWS Cloud and Data Vault integration sessions.

Overview

Section 1: Introduction to Data Integration

Lecture 1 Introduction

Lecture 2 Outline of the course

Lecture 3 Get the matterials

Section 2: Data Integration in Data management

Lecture 4 The agenda of the session

Lecture 5 Data Integration in Data management

Lecture 6 Some concepts and kinds of Data Integration

Lecture 7 What does data integration look like?

Section 3: Introduction to IBM Information Server

Lecture 8 The agenda of this session

Lecture 9 Introduction to IBM Information Server

Lecture 10 Introduction to IBM Information Server (cont.)

Lecture 11 Key IBM Information Server Components

Lecture 12 IBM Information Server topology

Lecture 13 IBM Information Server topology (Cont.)

Section 4: IBM Information Server Administration Console

Lecture 14 The agenda of this session

Lecture 15 IBM Information Server Administration Console

Lecture 16 IBM Information Server Administration Console (Cont.)

Lecture 17 Real Practice 1 – IBM Information Server Administration Console

Section 5: Introduction to IBM DataStage

Lecture 18 The agenda of this session

Lecture 19 DataStage Architecture

Lecture 20 DataStage Administrator

Lecture 21 DataStage Designer

Lecture 22 DataStage Director

Section 6: Developing in DataStage and Features

Lecture 23 The agenda of this session

Lecture 24 Developing in DataStage

Lecture 25 DataStage project repository

Lecture 26 Types of DataStage Jobs

Lecture 27 Design Elements of Parallel Jobs

Lecture 28 Partition Parallelism

Lecture 29 Multi-Node Partitioning

Lecture 30 Job design versus execution

Lecture 31 Configuration File

Lecture 32 Configuration File (Cont.)

Lecture 33 Summary - Developing in IBM DataStage and Features

Section 7: DataStage Administration

Lecture 34 The agenda of this session

Lecture 35 Unit Objectives - DataStage Administration

Lecture 36 Information Server Web Console - Administration

Lecture 37 Web Console Login Window

Lecture 38 User and Group Management

Lecture 39 Creating a DataStage User ID

Lecture 40 Assign DataStage Roles

Lecture 41 DataStage Credentials

Lecture 42 DataStage Credentials Default Mapping

Lecture 43 Logging onto DataStage Administrator

Lecture 44 InforSphere DataStage Administration

Lecture 45 DataStage Administrator Tabs

Lecture 46 Environment variables

Lecture 47 Environment reporting variables

Lecture 48 DataStage Administrator Permissions tab

Lecture 49 Add users and groups

Lecture 50 Specify DataStage Role

Lecture 51 DataStage Administrator Logs Tab

Lecture 52 DataStage Administrator Parallel Tab

Lecture 53 Real Practice 2 - Task 1 - Information Server Administration - User & Group

Lecture 54 Real Practice 2 - Task 2 - Project Properties - General

Lecture 55 Real Practice 2 - Task 3 - Project Properties - Environment

Lecture 56 Real Practice 2 - Task 4 - Project Properties - User & Group

Lecture 57 Summary - DataStage Administration

Section 8: Work with metadata

Lecture 58 The agenda of this session

Lecture 59 Unit Objectives - Work with metadata

Lecture 60 Logging onto DataStage Designer

Lecture 61 DataStage Designer

Lecture 62 DataStage project repository

Lecture 63 Import and Export

Lecture 64 Exporting DataStage Components

Lecture 65 Export objects

Lecture 66 Import Procedure

Lecture 67 Import objects

Lecture 68 Real Practice 3 - Export & Import DataStage Components

Lecture 69 Summary - Work with metadata

Section 9: Create Parallel Job

Lecture 70 The agenda of this session

Lecture 71 Unit objectives - Create parallel jobs

Lecture 72 What is a parallel job?

Lecture 73 Tools Palette

Lecture 74 Create a new parallel job

Lecture 75 Drag stages and links from the Palette

Lecture 76 Rename links and stages

Lecture 77 Connection Properties

Lecture 78 Job parameters

Lecture 79 Define a job parameter

Lecture 80 Use a job parameter in a stage

Lecture 81 Add job documentation

Lecture 82 Job Properties window documentation

Lecture 83 Compile and run a job

Lecture 84 Compile, Errors or Successful message

Lecture 85 DataStage Director

Lecture 86 Run options

Lecture 87 Performance statistics

Lecture 88 Director Status View

Lecture 89 Job log, viewed from designer

Lecture 90 Other job log functions

Lecture 91 Director monitor

Lecture 92 Run jobs from the command line

Lecture 93 Parameter sets

Lecture 94 Create a parameter set

Lecture 95 Defining the parameters

Lecture 96 Defining values files

Lecture 97 Load a parameter set into a job

Lecture 98 Use parameter set parameters

Lecture 99 Run jobs with parameter set parameters

Lecture 100 Real Practice 4 - Task 1 - Database –> Transformer –> Database

Lecture 101 Real Practice 4 - Task 2 - Database –> Transformer –> Database with Parameter

Lecture 102 Summary - Create parallel jobs

Section 10: Create parallel jobs - Access sequential data

Lecture 103 The agenda of this session

Lecture 104 Unit objectives - Access sequential data

Lecture 105 How sequential data is handled

Lecture 106 Features of the Sequential File stage

Lecture 107 Job design with Sequential File Stages

Lecture 108 Sequential File stage properties

Lecture 109 Format Tab

Lecture 110 Multiple Readers

Lecture 111 Writing to a sequential file

Lecture 112 Reject links

Lecture 113 Source and target reject links

Lecture 114 Setting the Reject Mode property

Lecture 115 Real Practice 5 - Task 1 - File –> Transformer –> File

Lecture 116 Real Practice 5 - Task 2 - File –> Transformer –> DataSet with Parameter

Lecture 117 Real Practice 6 - File –> Transformer –> DataSet

Lecture 118 Summary - Access sequential data

Section 11: Partitioning and collecting algorithms

Lecture 119 The agenda of this session

Lecture 120 Unit objectives - Partitioning and collecting algorithms

Lecture 121 Partition parallelism

Lecture 122 Stage partitioning

Lecture 123 DataStage hardware environments

Lecture 124 Partitioning algorithms

Lecture 125 Collecting

Lecture 126 Collecting (Cont.)

Lecture 127 Collecting algorithms

Lecture 128 Keyless versus keyed partitioning algorithms

Lecture 129 Round Robin and Random partitioning

Lecture 130 Entire partitioning

Lecture 131 Hash partitioning

Lecture 132 Modulus partitioning

Lecture 133 Auto partitioning

Lecture 134 Partitioning / collecting link icons

Lecture 135 More partitioning icons

Lecture 136 Specify a partitioning algorithm

Lecture 137 Specify a collecting algorithm

Lecture 138 Configuration file

Lecture 139 Example configuration file

Lecture 140 Adding $APT_CONFIG_FILE as a job parameter

Lecture 141 Editing configuration files

Lecture 142 Summary - Partitioning and collecting algorithms

Section 12: Combine Data

Lecture 143 The agenda of this session

Lecture 144 Unit objectives - Combine Data

Lecture 145 Combine Data

Lecture 146 Lookup, Join, Merge stages

Lecture 147 Lookup Stage features

Lecture 148 Lookup types

Lecture 149 Match Lookup Stage Example

Lecture 150 Lookup Stage with an equality match

Lecture 151 Define the Lookup key

Lecture 152 Specify the output columns

Lecture 153 Lookup failure actions

Lecture 154 Specifying lookup failure actions

Lecture 155 Lookup stage with reject link

Lecture 156 Real Practice 7 - Task 1 - Combine data with Lookup Stage

Lecture 157 Join stage

Lecture 158 Job with Join Stage

Lecture 159 Join Stage Properties

Lecture 160 Output Mapping tab

Lecture 161 Merge stage

Lecture 162 Merge Stage Job

Lecture 163 Merge Stage Properties

Lecture 164 Funnel stage

Lecture 165 Funnel Stage Example

Lecture 166 Funnel Stage Properties

Lecture 167 Real Practice 7 - Task 2 - Combine data with Join Stage

Lecture 168 Real Practice 7 - Task 3 - Combine Data with Merge Stage

Lecture 169 Real Practice 7 - Task 4 - Combine Data with Funnel Stage

Lecture 170 Summary - Combine Data

Section 13: Group processing stages

Lecture 171 The agenda of this session

Lecture 172 Unit objectives - Group processing stages

Lecture 173 Group processing stages

Lecture 174 Sort data

Lecture 175 Sorting alternatives

Lecture 176 In-Stage sorting

Lecture 177 Sort Stage Properties tab

Lecture 178 Specify sort keys

Lecture 179 Partition sorts

Lecture 180 Aggregator stage

Lecture 181 Job with Aggregator Stage

Lecture 182 Aggregation types

Lecture 183 Aggregation types (Cont.)

Lecture 184 Output Mapping Tab

Lecture 185 Output Columns tab

Lecture 186 Calculation aggregation type

Lecture 187 Grouping methods

Lecture 188 Remove duplicates

Lecture 189 Remove Duplicates stage job

Lecture 190 Remove Duplicates stage properties

Lecture 191 Fork - Join Job Design

Lecture 192 Real Practice 8 - File –> Transformer –> DataSet –> Aggregator –> File

Lecture 193 Summary - Group processing stages

Section 14: Transfromer Stage

Lecture 194 The agenda of this session

Lecture 195 Unit Objectives - Transformer stage

Lecture 196 Transformer stage

Lecture 197 Job with a Transformer stage

Lecture 198 Inside the Transformer stage

Lecture 199 Transformer stage elements

Lecture 200 Transformer stage elements

Lecture 201 Constraints

Lecture 202 Constraints example

Lecture 203 Define a constraint

Lecture 204 Use the expression editor

Lecture 205 Otherwise links for data integrity

Lecture 206 Real Practice 9 - File –> Transformer –> File

Lecture 207 Parallel job debugger

Lecture 208 Set breakpoints

Lecture 209 Edit breakpoints

Lecture 210 Running a parallel job in the debugger

Lecture 211 Add columns to the watch list

Lecture 212 Checkpoint - Transformer stage

Section 15: Repository functions

Lecture 213 The agenda of this session

Lecture 214 Unit objectives - Repository funcions

Lecture 215 Quick find

Lecture 216 Found results

Lecture 217 Advanced Find window

Lecture 218 Advanced Find options

Lecture 219 Using the found results

Lecture 220 Performing an impact analysis

Lecture 221 Initiating an impact analysis

Lecture 222 Results in text format

Lecture 223 Results in graphical format

Lecture 224 Displaying the dependency graph

Lecture 225 Displaying the dependency path

Lecture 226 Viewing column-level data flow

Lecture 227 Finding where a column originates

Lecture 228 Displayed results

Lecture 229 Finding the difference between two jobs

Lecture 230 Initiating the comparison

Lecture 231 Comparison results

Lecture 232 Saving to an HTML file

Lecture 233 Summary - Repository functions

Section 16: Work with relational data

Lecture 234 The agenda of this session

Lecture 235 Unit Objectives - Work with relational data

Lecture 236 Connector stages

Lecture 237 Reading from database tables

Lecture 238 Connector stage GUI

Lecture 239 Navigation panel

Lecture 240 Connection properties

Lecture 241 Usage properties - Session and Before/After SQL

Lecture 242 Writing to database tables

Lecture 243 DB2 Connector GUI

Lecture 244 Connector write properties

Lecture 245 Data connection objects

Lecture 246 Data connection objects

Lecture 247 Creating a new data connection object

Lecture 248 Real Practice 11 - Relational Database –> Transformer –> File

Lecture 249 Summary - Work with relational data

Section 17: Job Sequence Control

Lecture 250 The agenda of this session

Lecture 251 Unit Objectives - Job Sequence control

Lecture 252 What is a job sequence?

Lecture 253 Basics for creating a job sequence

Lecture 254 Job sequence stages

Lecture 255 Job sequence example

Lecture 256 Job Activity stage properties

Lecture 257 Job Activity trigger

Lecture 258 Execute Command stage

Lecture 259 Notification Activity stage

Lecture 260 User Variables stage

Lecture 261 Wait for File stage

Lecture 262 Sequencer stage

Lecture 263 Nested Condition stage

Lecture 264 Loop stages

Lecture 265 Summary - Job control

Section 18: AWS Cloud Integration

Lecture 266 The agenda of this session

Lecture 267 Real Practice 12 - AWS Cloud Data Integration with IBM DataStage S3 Connector

Section 19: Data Vault 1.0 & 2.0 Integration

Lecture 268 Data Vault Introduction

Lecture 269 Data Vault Integration with IBM DataStage

Lecture 270 Real Practice 13 - Mini Project 1 - Data Vault 1.0 DWH with IBM DataStage

Lecture 271 Real Practice 13 - Mini Project 2 - Data Vault 2.0 DWH with IBM DataStage

Section 20: Summary

Lecture 272 What we have learned

Data Professionals: Data analysts, data engineers who want to enhance their data integration skills using IBM DataStage.,IT Professionals: IT specialists, software developers, and database administrators who need to work with data integration solutions.,Business Analysts: Business analysts who want to understand how data integration impacts their data-driven decision-making processes.,Students and Graduates: Students pursuing degrees or recent graduates looking to build a foundation in data integration and expand their job prospects.,IBM DataStage Users: Users of IBM DataStage looking to deepen their knowledge, explore advanced features, and improve their job performance.,Anyone Interested in Data Integration: If you have a general interest in data integration and want to learn how IBM DataStage can be used for these purposes, this course is suitable for you.