Subcribe and Access : 5200+ FREE Videos and 21+ Subjects Like CRT, SoftSkills, JAVA, Hadoop, Microsoft .NET, Testing Tools etc..

Batch Date: Nov 20th @6:30AM

Faculty: Mr. Sameer (10+ Yrs of Exp,.. & Real Time Expert)
(Leading Faculty in Twin Cities)

Duration : 3 Months

Venue :
DURGA SOFTWARE SOLUTIONS,
Flat No : 202, 2nd Floor,
HUDA Maitrivanam,
Ameerpet, Hyderabad - 500038

Ph.No: +91 - 9246212143, 80 96 96 96 96

Syllabus:

Azure Data Engineering

Module 1: Cloud Computing Concepts

• What is the "Cloud" ?
• Why Cloud Services
• Types of Cloud Models
  • Deployment Models
    • Private Cloud
    • Public Cloud
    • Hybrid cloud
• Types of Cloud Services
  • Infrastructure as a Service(IaaS)
  • Platform as a Service(PaaS)
  • Software as a Service(SaaS)
• Comparing Cloud Platforms
• Microsoft Azure,
• Amazon Web Services,
• Google Cloud Platform
• Characteristics of Cloud Computing
• On-demand self-service
• Broad network access
• Multi-tenancy and resource pooling
• Rapid elasticity and scalability
• Measured service
• Cloud Data Warehouse Architecture
• Shared Memory architecture
• Shared Disk architecture
• Shared Nothing architecture

Module 2: BigData Introduction

• What is BigData?
• BigData Sources
• Data vs Information
• Characteristics of BigData
  • Variety
  • Velocity
  • Volume
  • Veracity
  • Value
• Types Of BigData
  • Structured Data
  • Unstructured Data
  • Semi Structured Data

Module 3: Dimensional Modelling

• OLTP System
  • Relational Modelling
• Characteristics Features of OLTP
• Enterprise Data Warehouse
  • Dimensional Modelling
• Dimensional Modelling-Schemas
  • Star Schema
  • Snowflake Schema
  • Multi Star Schema
• Dimensional Tables
• Fact Tables
• Types of slowly Changing Dimensions
  • Type1 Dimension
  • Type2 Dimension
  • Type3 Dimension
• Types Facts
  • Additive Facts
  • Semi Additive Facts
  • Non-Additive Facts

Module 4: Azure SQL Database

• Introduction Azure SQL Database.
• Comparing Single Database
• Managed Instance
• Creating and Using SQL Server
• Creating SQL Database Services.
• Azure SQL Database Tools.
• Migrating on premise database to SQL Azure.
• Purchasing Models
• DTU service tiers
• vCore based Model
• Serverless compute tier
• Service Tiers
  • General purpose / Standard
  • Business Critical / Premium
  • Hyper scale
• Deployment of an Azure SQL Database
• Elastic Pools.
• What is SQL elastic pools
  • Choosing the correct pool size
• Creating a New Pool
• Manage Pools

Module 5: Azure Storage Service

• Azure Storage Account
• Features of Azure storage Service
• Introduction to Blob Storage Service
• Blob Storage Architecture
• Blob Storage Features
• Types of Blobs
  • Block Blobs
  • Append Blobs
  • Page Blobs
• Creating a Storage Account
• Azure Storage Performance Tiers
  • Standard
  • Premium Performance
• Understanding Data Replication
  • LRS ( Locally Redundant Storage)
  • ZRS (Zone Redundant Storage)
  • GRS (Geo Redundant Storage)
• Azure Storage-Access Tiers
  • Hot
  • Cold
  • Archive
• Working with Containers and Blobs
• Soft Delete
• Azure Storage Explorer
• Access blobs securely
• Access Key
• Account Shared Access Token
• Service Shared Access Token
• Azure Maximum Scalability Or Limits

Module 6: Azure Data Lake Storage Services

• Introduction to Azure Data Lake
• What is Data Lake?
• What is Azure Data Lake?
• Data Lake Architecture?
• Working with Azure Data Lake Storage Gen1
• Features of Data Lake Storage Gen1
• Understanding Azure Data Lake Gen2
• Features of Data Lake Storage Gen2
• Differences Between Gen1 & Gen2 Storage
• Explore Data Lake Storages
• Prevising Data Lake Storage Gen1 Service
• Provising Data Lake Storage Gen2 Service
• Uploading Sample File
• Using Azure Portal
• Using Storage Explorer

Azure Data Factory:

Module 7: Azure Data Factory Introduction

• What is Azure Data Factory (ADF)?
• Azure Data Factory Key Components
  • Pipeline
  • Activity
  • Linked Service
  • Data Set
  • Integration Runtime
  • Triggers
  • Data Flows
• Create Resource Group
• Create Storage Account
• Creation of Azure Data Factory Service

Module 8: Working with Copy Activity

• Understanding Azure Data Factory UI
• Copy Data from Blob Storage Service to Azure SQL Database
• Copy data from file storage account to file storage account
• Create Linked service for various data stores and compute
• Creation of Datasets that points to file and table
• Design Pipelines with various activities
• Create SQL Server on Virtual Machines( On-Premise)
• Define Copy activity and it features
• Copy Activity-Copy Behavior
• Copy Activity Data Integration Units
• Copy Activity- User Properties
• Copy Activity- Number of parallel copies
• Working with Lookup Activity
• Understanding of Each Activity
• Filter Activity
• Get Metadata Activity
• Lift and Shift
• Hosting Azure - SSIS Integration Runtime
• Execute SSIS Packages from ADF
• Monitoring Pipeline
• Debug Pipeline
• Trigger pipeline manually
• Monitor pipeline
• Trigger pipeline on schedule

Module 9: Practical Scenarios and Use Cases

• ADF_PracticeSession1_Blob_To_Blob
• ADF_PracticeSession2_CopyActivity_Prefix_Wildcard_FilePath_Blob_To_Blob
• ADF_PracticeSession3_Blob_To_Azure_SQLDB
• ADF_PracticeSession4_Blob_To_Azure_SQLDB
• ADF_PracticeSession5_Dataset_Parameters_Blob_To_Azure_SQLDB
• ADF_PracticeSession6_Blob_To_ADLS_Gen2
• ADF_PracticeSession7_ADLS_Gen1_To_ADLS_Gen2
• ADF_PracticeSession8_Pipeline_Dataset_LinkedService_Parameters
• ADF_PracticeSession9_FilteringFileFormats_Getmetadata_Filter_ForEach_Copy_Activity
• ADF_PracticeSession10_FilteringFileFormats_Getmetadata_Filter_ForEach_Copy_Activity
• ADF_PracticeSession11_BulkCopy_Tables_Files
• ADF_PracticeSession12_Container_Parameterization_Blob_To_Blob_Storage
• ADF_PracticeSession13_ExecuteCopyActivity_BasedOnFileCount
• ADF_PracticeSession14_StoredProcedures_Parameters
• ADF_PracticeSession15_CopyActivity_CustomSQL_Queries_StoredProcedures
• ADF_PracticeSession16_Pipeline_Audit_Log
• ADF_PracticeSession17_Copybehaviour
• ADF_PracticeSession18_CSV_To_JSON_Format
• ADF_PracticeSession19_Copy_JSON_File_To_AzureSQL
• ADF_PracticeSession20_Add_AdditionalColumns_WhileCopyingData
• ADF_PracticeSession21_CopyDataTool
• ADF_PracticeSession22_Custom_Email_Notification
• ADF_PracticeSession23_AzureKeyVault_Integration
• ADF_PracticeSession24_Incremental_Load
• ADF_PracticeSession25_Integration_Runtime
• ADF_PracticeSession26_On-Premise_SQLServer_ADLS_Gen2
• ADF_PracticeSession27_On-Premise_FileSystem_ADLS_Gen2
• ADF_PracticeSession28_REST_API_Integration
• ADF_PracticeSession29_CosmosDB_Introduction
• ADF_PracticeSession30_Eventbased_Trigger
• ADF_PracticeSession31_Scheduled_Trigger
• ADF_PracticeSession32_TumblingWindow_Trigger
• ADF_PracticeSession33_Blob_SQLDB_Executepipeline_Activity
• ADF_PracticeSession34_SQLDB_BLOB_Overwrite_Append_Mode
• ADF_PracticeSession36_Dataflows_Introduction
• ADF_PracticeSession37_Dataflows_Select_Filter_DerivedColumn_Transformation
• ADF_PracticeSession38_Dataflows_Select__DerivedColumn_Aggregator_Sort_Transformation
• ADF_PracticeSession39_Dataflows_ConditionalSplit_Transformation
• ADF_PracticeSession40_Dataflows_Join_Transformation
• ADF_PracticeSession41_Dataflows_Union_Transformation
• ADF_PracticeSession42_Dataflows_Lookup_Transformation
• ADF_PracticeSession43_Dataflows_Exists_Transformation
• ADF_PracticeSession44_Dataflows_Rank_Transformation
• ADF_PracticeSession45_Dataflows_Pivot_Transformation
• ADF_PracticeSession46_Dataflows_UnPivot_Transformation
• ADF_PracticeSession47_Dataflows_SurrogateKey_Transformation
• ADF_PracticeSession48_Dataflows_AlterRow_Transformation
• ADF_PracticeSession49_Remove Duplicate rows using data flows
• ADF_PracticeSession50_Slowly Changing Dimension Type1 (SCD1) with Hash Key Function

Module 10: Assignments & Case Studies

• ADF_Azure_HDInsight Integration
• ADF_Azure_HDInsight with Spark Cluster
• ADF_Azure_Databricks Integration

Azure Databricks:

Module 11: Introduction to Azure Databricks

• Introduction to Databricks
• Azure Databricks Architecture
• Azure Databricks Main Concepts

Module 12: Databricks Integration with Azure Blob Storage

• Read data from Blob Storage and Creating Blob mount point

Module 13: Databricks Integration with Azure Data Lake Storage Gen2

• Reading files from Azure Data Lake Storage Gen2

Azure Synapse Analytics:

Module 14: Introduction to Azure Synapse

• Technical requirements
• Interdiction the components of Azure synapse
• Creating synapse Workspace
• Understanding Azure Data Lake Exploring Synapse Studio

Module 15: Consideration for Your Compute Environment

Technical requirements Introducing SQL Pool

• Creating SQL Pool
• Understanding Synapse SQL Pool
• Architecture and component
• Examining DWUs
• Understanding distribution in Synapse SQL Pool
• Understanding portions in Synapse SQL Pool
• Using temporary table in Synapse SQL Pool
• Discovering the benefits of Synapse SQL Pool

Understanding Synapse SQL on demand

• SQL on-demand architecture and components
• Learning about the benefits of Synapse SQL on-demand

Module 16: Bringing Your Data to Azure Synapse

• Technical requirements
• Using Synapse pipelines to import data
• Bringing data to your Synapse SQL Pool using Copy Data tool
• Using Azure Data Factory to import data
• Using SQL Server integration Services to import data

Module 17: Using Synapse Pipelines to Orchestrate Your Data

• Technical requirements
• Introducing synapse pipe lines
  • Integration runtime
  • Activities
  • Pipelines
  • Triggers
• Creating linked services
• Defining source and target
• Using various activities in synapse pipelines
• Scheduling synapse pipelines
• Creating pipelines using samples