Subcribe and Access : 5200+ FREE Videos and 21+ Subjects Like CRT, SoftSkills, JAVA, Hadoop, Microsoft .NET, Testing Tools etc..
Batch
Date: Dec 3rd @8:30AM
Faculty: Mr. Sameer (10+ Yrs of Exp,.. & Real Time Expert)
(Leading Faculty in Twin Cities)
Duration : 3 Months
Venue
:
DURGA SOFTWARE SOLUTIONS,
Flat No : 202,
2nd Floor,
HUDA Maitrivanam,
Ameerpet, Hyderabad - 500038
Ph.No: +91 - 9246212143, 80 96 96 96 96
Syllabus:
Azure Data Engineering
Module 1: Cloud Computing Concepts
- What is the "Cloud" ?
- Why Cloud Services
- Types of Cloud Models
- Deployment Models
- Private Cloud
- Public Cloud
- Hybrid cloud
- Types of Cloud Services
- Infrastructure as a Service(IaaS)
- Platform as a Service(PaaS)
- Software as a Service(SaaS)
- Comparing Cloud Platforms
- Microsoft Azure,
- Amazon Web Services,
- Google Cloud Platform
- Characteristics of Cloud Computing
- On-demand self-service
- Broad network access
- Multi-tenancy and resource pooling
- Rapid elasticity and scalability
- Measured service
- Cloud Data Warehouse Architecture
- Shared Memory architecture
- Shared Disk architecture
- Shared Nothing architecture
Module 2: BigData Introduction
- What is BigData?
- BigData Sources
- Data vs Information
- Characteristics of BigData
- Variety
- Velocity
- Volume
- Veracity
- Value
- Types Of BigData
- Structured Data
- Unstructured Data
- Semi Structured Data
Module 3: Dimensional Modelling
- OLTP System
- Characteristics Features of OLTP
- Enterprise Data Warehouse
- Dimensional Modelling-Schemas
- Star Schema
- Snowflake Schema
- Multi Star Schema
- Dimensional Tables
- Fact Tables
- Types of slowly Changing Dimensions
- Type1 Dimension
- Type2 Dimension
- Type3 Dimension
- Types Facts
- Additive Facts
- Semi Additive Facts
- Non-Additive Facts
Module 4: Azure SQL Database
- Introduction Azure SQL Database.
- Comparing Single Database
- Managed Instance
- Creating and Using SQL Server
- Creating SQL Database Services.
- Azure SQL Database Tools.
- Migrating on premise database to SQL Azure.
- Purchasing Models
- DTU service tiers
- vCore based Model
- Serverless compute tier
- Service Tiers
- General purpose / Standard
- Business Critical / Premium
- Hyper scale
- Deployment of an Azure SQL Database
- Elastic Pools.
- What is SQL elastic pools
- Choosing the correct pool size
- Creating a New Pool
- Manage Pools
Module 5: Azure Storage Service
- Azure Storage Account
- Features of Azure storage Service
- Introduction to Blob Storage Service
- Blob Storage Architecture
- Blob Storage Features
- Types of Blobs
- Block Blobs
- Append Blobs
- Page Blobs
- Creating a Storage Account
- Azure Storage Performance Tiers
- Standard
- Premium Performance
- Understanding Data Replication
- LRS ( Locally Redundant Storage)
- ZRS (Zone Redundant Storage)
- GRS (Geo Redundant Storage)
- Azure Storage-Access Tiers
- Working with Containers and Blobs
- Soft Delete
- Azure Storage Explorer
- Access blobs securely
- Access Key
- Account Shared Access Token
- Service Shared Access Token
- Azure Maximum Scalability Or Limits
Module 6: Azure Data Lake Storage Services
- Introduction to Azure Data Lake
- What is Data Lake?
- What is Azure Data Lake?
- Data Lake Architecture?
- Working with Azure Data Lake Storage Gen1
- Features of Data Lake Storage Gen1
- Understanding Azure Data Lake Gen2
- Features of Data Lake Storage Gen2
- Differences Between Gen1 & Gen2 Storage
- Explore Data Lake Storages
- Prevising Data Lake Storage Gen1 Service
- Provising Data Lake Storage Gen2 Service
- Uploading Sample File
- Using Azure Portal
- Using Storage Explorer
Azure Data Factory:
Module 7: Azure Data Factory Introduction
- What is Azure Data Factory (ADF)?
- Azure Data Factory Key Components
- Pipeline
- Activity
- Linked Service
- Data Set
- Integration Runtime
- Triggers
- Data Flows
- Create Resource Group
- Create Storage Account
- Creation of Azure Data Factory Service
Module 8: Working with Copy Activity
- Understanding Azure Data Factory UI
- Copy Data from Blob Storage Service to Azure SQL Database
- Copy data from file storage account to file storage account
- Create Linked service for various data stores and compute
- Creation of Datasets that points to file and table
- Design Pipelines with various activities
- Create SQL Server on Virtual Machines( On-Premise)
- Define Copy activity and it features
- Copy Activity-Copy Behavior
- Copy Activity Data Integration Units
- Copy Activity- User Properties
- Copy Activity- Number of parallel copies
- Working with Lookup Activity
- Understanding of Each Activity
- Filter Activity
- Get Metadata Activity
- Lift and Shift
- Hosting Azure - SSIS Integration Runtime
- Execute SSIS Packages from ADF
- Monitoring Pipeline
- Debug Pipeline
- Trigger pipeline manually
- Monitor pipeline
- Trigger pipeline on schedule
Module 9: Practical Scenarios and Use Cases
- ADF_PracticeSession1_Blob_To_Blob
- ADF_PracticeSession2_CopyActivity_Prefix_Wildcard_FilePath_Blob_To_Blob
- ADF_PracticeSession3_Blob_To_Azure_SQLDB
- ADF_PracticeSession4_Blob_To_Azure_SQLDB
- ADF_PracticeSession5_Dataset_Parameters_Blob_To_Azure_SQLDB
- ADF_PracticeSession6_Blob_To_ADLS_Gen2
- ADF_PracticeSession7_ADLS_Gen1_To_ADLS_Gen2
- ADF_PracticeSession8_Pipeline_Dataset_LinkedService_Parameters
- ADF_PracticeSession9_FilteringFileFormats_Getmetadata_Filter_ForEach_Copy_Activity
- ADF_PracticeSession10_FilteringFileFormats_Getmetadata_Filter_ForEach_Copy_Activity
- ADF_PracticeSession11_BulkCopy_Tables_Files
- ADF_PracticeSession12_Container_Parameterization_Blob_To_Blob_Storage
- ADF_PracticeSession13_ExecuteCopyActivity_BasedOnFileCount
- ADF_PracticeSession14_StoredProcedures_Parameters
- ADF_PracticeSession15_CopyActivity_CustomSQL_Queries_StoredProcedures
- ADF_PracticeSession16_Pipeline_Audit_Log
- ADF_PracticeSession17_Copybehaviour
- ADF_PracticeSession18_CSV_To_JSON_Format
- ADF_PracticeSession19_Copy_JSON_File_To_AzureSQL
- ADF_PracticeSession20_Add_AdditionalColumns_WhileCopyingData
- ADF_PracticeSession21_CopyDataTool
- ADF_PracticeSession22_Custom_Email_Notification
- ADF_PracticeSession23_AzureKeyVault_Integration
- ADF_PracticeSession24_Incremental_Load
- ADF_PracticeSession25_Integration_Runtime
- ADF_PracticeSession26_On-Premise_SQLServer_ADLS_Gen2
- ADF_PracticeSession27_On-Premise_FileSystem_ADLS_Gen2
- ADF_PracticeSession28_REST_API_Integration
- ADF_PracticeSession29_CosmosDB_Introduction
- ADF_PracticeSession30_Eventbased_Trigger
- ADF_PracticeSession31_Scheduled_Trigger
- ADF_PracticeSession32_TumblingWindow_Trigger
- ADF_PracticeSession33_Blob_SQLDB_Executepipeline_Activity
- ADF_PracticeSession34_SQLDB_BLOB_Overwrite_Append_Mode
- ADF_PracticeSession36_Dataflows_Introduction
- ADF_PracticeSession37_Dataflows_Select_Filter_DerivedColumn_Transformation
- ADF_PracticeSession38_Dataflows_Select__DerivedColumn_Aggregator_Sort_Transformation
- ADF_PracticeSession39_Dataflows_ConditionalSplit_Transformation
- ADF_PracticeSession40_Dataflows_Join_Transformation
- ADF_PracticeSession41_Dataflows_Union_Transformation
- ADF_PracticeSession42_Dataflows_Lookup_Transformation
- ADF_PracticeSession43_Dataflows_Exists_Transformation
- ADF_PracticeSession44_Dataflows_Rank_Transformation
- ADF_PracticeSession45_Dataflows_Pivot_Transformation
- ADF_PracticeSession46_Dataflows_UnPivot_Transformation
- ADF_PracticeSession47_Dataflows_SurrogateKey_Transformation
- ADF_PracticeSession48_Dataflows_AlterRow_Transformation
- ADF_PracticeSession49_Remove Duplicate rows using data flows
- ADF_PracticeSession50_Slowly Changing Dimension Type1 (SCD1) with Hash Key Function
Module 10: Assignments & Case Studies
- ADF_Azure_HDInsight Integration
- ADF_Azure_HDInsight with Spark Cluster
- ADF_Azure_Databricks Integration
Azure Databricks:
Module 11: Introduction to Azure Databricks
- Introduction to Databricks
- Azure Databricks Architecture
- Azure Databricks Main Concepts
Module 12: Databricks Integration with Azure Blob Storage
- Read data from Blob Storage and Creating Blob mount point
Module 13: Databricks Integration with Azure Data Lake Storage Gen2
- Reading files from Azure Data Lake Storage Gen2
Azure Synapse Analytics:
Module 14: Introduction to Azure Synapse
- Technical requirements
- Interdiction the components of Azure synapse
- Creating synapse Workspace
- Understanding Azure Data Lake Exploring Synapse Studio
Module 15: Consideration for Your Compute Environment
Technical requirements Introducing SQL Pool
- Creating SQL Pool
- Understanding Synapse SQL Pool
- Architecture and component
- Examining DWUs
- Understanding distribution in Synapse SQL Pool
- Understanding portions in Synapse SQL Pool
- Using temporary table in Synapse SQL Pool
- Discovering the benefits of Synapse SQL Pool
Understanding Synapse SQL on demand
- SQL on-demand architecture and components
- Learning about the benefits of Synapse SQL on-demand
Module 16: Bringing Your Data to Azure Synapse
- Technical requirements
- Using Synapse pipelines to import data
- Bringing data to your Synapse SQL Pool using Copy Data tool
- Using Azure Data Factory to import data
- Using SQL Server integration Services to import data
Module 17: Using Synapse Pipelines to Orchestrate Your Data
- Technical requirements
- Introducing synapse pipe lines
- Integration runtime
- Activities
- Pipelines
- Triggers
- Creating linked services
- Defining source and target
- Using various activities in synapse pipelines
- Scheduling synapse pipelines
- Creating pipelines using samples