Subcribe and Access : 5200+ FREE Videos and 21+ Subjects Like CRT, SoftSkills, JAVA, Hadoop, Microsoft .NET, Testing Tools etc..
Batch
Date: July 19th @8:00AM
Faculty: Mr. Sameer (10+ Yrs of Exp,.. & Real Time Expert)
(Leading Faculty in Twin Cities)
Duration : 2 Months
Venue
:
DURGA SOFTWARE SOLUTIONS,
Flat No : 202,
2nd Floor,
HUDA Maitrivanam,
Ameerpet, Hyderabad - 500038
Ph.No: +91 - 9246212143, 80 96 96 96 96
Syllabus:
ADF
(Azure Data Factory)
Module 1: Cloud Computing Concepts
- What is the "Cloud" ?
- Why cloud services
- Types of cloud models
- Deployment Models
- private Cloud deployment model
- public Cloud deployment model
- hybrid cloud deployment model
- Types of cloud services
- Infrastructure as a Service,
- Platform as a Service,
- Software as a Service
- Comparing Cloud Platforms
- Microsoft Azure,
- Amazon Web Services,
- Google Cloud Platform
- characteristics of cloud computing
- On-demand self-service
- Broad network access
- Multi-tenancy and resource pooling
- Rapid elasticity and scalability
- Measured service
- Cloud Data Warehouse Architecture
- Shared Memory architecture
- Shared Disk architecture
- Shared Nothing architecture
Module 2: Core Azure services
- Core Azure Architectural components
- Core Azure Services and Products
- Azure solutions
- Azure management tools
Module 3: Security, Privacy, Compliance
- Securing network connectivity
- Core Azure identity services
- Security tools and features
- Azure Governance methodologies
- Monitoring and reportings
- Privacy, compliance, and data protection standards
Module 4: Azure Pricing and Support
- Azure subscriptions
- Planning and managing costs
- Azure support options
- Azure Service Level Agreements (SLAs)
- Service Lifecycle in Azure
Module 5: Azure SQL Database
- Introduction Azure SQL Database.
- Comparing Single Database
- Managed Instance
- Creating and Using SQL Server
- Creating SQL Database Services.
- Azure SQL Database Tools.
- Migrating on premise database to SQL Azure.
- Purchasing Models
- DTU service tiers
- vCore based Model
- Serverless compute tier
- Service Tiers
- General purpose / Standard
- Business Critical / Premium
- Hyperscale
- Deployment of an Azure SQL Database
- Elastic Pools.
- What is SQL elastic pools
- Choosing the correct pool size
- Creating a New Pool
- Manage Pools
- Monitoring and Tuning Azure SQL Database
- Configure SQL Database Auditing
- Export and Import of Database
- Automated Backup
- Point in Time Restore
- Restore deleted databases
- Long-term backup retention
- Active Geo Replication
- Auto Failover Group
Module 6: Azure Storage Service
- Storage Service and Account
- Creating a Storage Account
- Standard and Premium Performance
- Understanding Replication
- Hot, Cold and Archive Access Tiers
- Working with Containers and Blobs
- Types of Blobs
- Block Blobs,
- Append Blobs
- Page Blobs
- Blob Metadata
- Soft Delete
- Azure Storage Explorer
- Access blobs securely
- Access Key
- Account Shared Access Token
- Service Shared Access Token
- Shared Access Policy
- Storage Service Encryption
- Azure Key Vault
Module 7: Azure Data Lake
- Introduction to Azure Data Lake
- What is Data Lake ?
- What is Azure Data Lake ?
- Data Lake Architecture
- Working with Azure Data Lake
- Provisioning Azure Data Lake.
- Explore Data Lake Analytics
- Explore Data Lake Store
- Uploading Sample File
- Using Azure Portal
- Using Storage Explorer
- Using Azure CLI
Module 8: Azure Data Factory
- What is Data Factory ?
- Data Factory Key Components
- Pipeline and Activity
- Linked Service o Data Set
- Integration Runtime Provision Required Azure Resources
- Create Resource Group
- Create Storage Account
- Provision SQL Server and Create Database
- Provision Data Factory
Module 9: Working with Copy Activity
- Understanding Data Factory UI
- Copy Data from Blob Storage to SQL Database
- Copy data from storage account to storage account
- Create Linked service o Create Dataset
- Create Pipeline Integration Service
- Copy Data from on-premise SQL Server to Blob Storage Working with Activities
- Understanding Lookup Activity
- Understanding for Each Activity
- Filter Activity
- Get Metadata Activity Azure
- Lift and Shift
- Provisioning Azure - SSIS Integration Runtime
- Execute SSIS Packages from Azure
- Execute SSIS Packages from SSISDB Triggers,
- Monitoring Pipeline
- Debug Pipeline
- Trigger pipeline manually
- Monitor pipeline
- Trigger pipeline on schedule
Module 10: Practical Scenarios and Use Cases
- ADF Introduction
- Important Concepts in ADF
- Create Azure Free Account for ADF
- Integration Runtime and Types
- Integration runtime in ADF-Azure IR
- Create Your First ADF
- Create Your First Pipeline in ADF
- Azure Storage Account Integration with ADF
- Copy multiple files from blob to blob
- Filter activity__Dynamic Copy Activity
- Get File Names from Folder Dynamically
- Deep dive into Copy Activity in ADF
- Copy Activity Behavior in ADF
- Copy Activity Performance Tuning in ADF
- Validation in ADF
- Get Count of files from folder in ADF
- Validate copied data between source and sink in ADF
- Azure SQL Database integration with ADF
- Azure SQL Databases - Introduction Relational databases
- Creating Your First Azure SQL Database
- 1) Deployment Models
- 2) Purchasing Modes
- Overwrite and Append Modes in Copy Activity
- Full Load in ADF
- Copy Data from Azure SQL Database to BLOB in ADF
- Copy multiple tables in Bulk with Lookup & ForEach in Data Factory
- Logging and Notification Azure Logic Apps
- Log Pipeline Executions to SQL Table using ADF
- Custom Email Notifications Send Error notification with logic app
- Use Foreach loop activity to copy multiple Tables- Step by Step Explanation
- Incremental Load in ADF
- Incremental Load or Delta load from SQL to Blob Storage in ADF
- Multi Table Incremental Load or Delta load from SQL to Blob Storage
- Incrementally copy new and changed files based on Last Modified Date
- Azure Key Vault integration with ADF
- Azure Key Vault, Secure secrets, keys & certificates in Azure Data
- ADF Triggers:
- Event Based Trigger in ADF
- Tumbling window trigger dependency & parameters
- Schedule Trigger
- Self Hosted Integration Runtime
- Copying On Premise data using Azure Self Hosted integration Runtime
- Data Migration from On premise SQL Server to cloud using ADF
- Load data from on premise sql server to Azure SQL DB
- Data Migration with polybase and Bulk insert
- Copy Data from sql server to Azure SQL DW with polybase & Bulk Insert
- Data Migration from On premise File System to cloud using ADF
- Copy Data from on-premise File System to ADLS Gen2
- ToCopying data from REST API using ADF
- Loop through REST API copy data TO ADLS Gen2-Linked Service Parameters
- AWS S3 integration with ADF
- Migrate Data from AWS S3 Buckets to ADLS Gen2
- Activities in ADF
- Switch Activity-Move and delete data
- Until Activity-Parameters & Variables
- Copy Recent Files From Blob input to Blob Output folder without LPV
- Snowflake integration with ADF
- Copy data from Snowflake to ADLS Gen2
- Copy data from ADLS Gen2 to Snowflake
- Azure CosmosDB integration with ADF
- Copy data from Azure SQLDB to CosmosDB
- Copy data from blob to cosmosDB
- Advanced Concepts in ADF
- Nested ForEach -pass parameters from Master to child pipeline
- High Availability of Self Hosted IR &Sharing IR with other ADF
- Data Flows Introduction
- Azure Data Flows Introduction
- Setup Integration Runtime for Data Flows
- Basics of SQL Joins for Azure Data Flows
- Joins in Data Flows
- Aggregations and Derive Column Transformations
- Joins in Azure DataFlows
- Advanced Join Transformations with filter and Conditional Split
- Data Flows - Data processing use case1
- Restart data processing from failure
- Remove Duplicate Rows &Store Summary Credit Stats
- Difference Between Join vs.Lookup Transformation& Merge Functionality
- Dimensions in Data Flows
- Slowly Changing Dimension Type1 (SCD1) with HashKey Function
- Flatten Transformation
- Rank, Dense_Rank Transformatios
- Data Flows Performance Metrics and Data Flow Parameters
- How to use pivot and unpivot Transformations
- Data Quality Checks and Logging using Data Flows
- Batch Account Integration with ADF
- Custom Activity in ADF
- Azure Functions Integration with ADF
- Azure HDInsight Integration with ADF
- Azure HDInsight with Spark Cluster
- Azure Databricks Integration with ADF
- ADF Integration with Azure Databricks
- Azure Data Lake Analytics integration with ADF
Module 11: Spark Basics
- Spark Architecture
- Spark RDD
- Spark SQL
- Spark SQL Functions
- Spark SQL Advanced
Module 12: Azure Integration Services
- Azure Functions
- Azure service bus
- Azure logic apps
- Azure Event Grid
- Azure Event Hub
- Azure API Management