Subcribe and Access : 5200+ FREE Videos and 21+ Subjects Like CRT, SoftSkills, JAVA, Hadoop, Microsoft .NET, Testing Tools etc..

Batch Date: Dec 26th @6:00PM

Faculty: Mr. N. Vijay Sunder Sagar (17+ Yrs of Exp,..)

Duration : 45 Days

Venue :
DURGA SOFTWARE SOLUTIONS,
Flat No : 202, 2nd Floor,
HUDA Maitrivanam,
Ameerpet, Hyderabad - 500038

Ph. No: +91 - 9246212143, 80 96 96 96 96

Syllabus:

ADF (Azure Data Factory)

Overview of Azure Resources / Services

1. Azure Data Factory
2. Azure Data bricks
3. BLOB Storage, Data Lake Storage Gen1 and Gen2
4. Azure SQL Server, SQL Database
5. Key Vault
6. Azure Synapse
7. Azure CosmosDB
8. Azure Streaming

1. Azure Data Factory

1. What is Azure Data Factory?
2. Azure Data Factory Architecture
3. Azure Data Factory Portal UI
4. Top-level concepts

A. Pipelines
B. Activities
C. Linked services
D. Datasets
E. Triggers
F. Data Flows
G. Integration Runtimes

2. Pipeline

1. What is a Pipeline?
2. Create a new pipeline
3. Organize pipelines into folders
4. Debug pipeline
5. Publish pipeline
6. Parameters / Pipeline Parameters

3. Linked Service

1. What is a Linked Service?
2. Create a Linked Service for –

A. BLOB
B. SQL Database
C. SQL Server
D. Data Lake Storage Gen1
E. Azure Data Lake Storage Gen2 etc

3. Parameters / Linked Service Parameterization

4. DataSets

1. What is a Data Set?
2. Create a Data Set for –

A. Avro, Binary, CSV, Excel, JSON, ORC, Parquet, XML in BLOB/ADLS Gen1/ADLS Gen2.

B. Table in SQL Database, SQL Server, Oracle Database etc

3. Parameters / Data Set Parameterization

5. Activities

1. Wait
2. Variables

A. Create a variable
B. Set variable
C. Append variable

3. Copy Data

A. General
B. Source
C. Sink
D. Mapping
E. Settings
F. User Properties

4. Copy file(s) from one BLOB Container to another Container

A. One file from a folder
B. All files from a folder
C. All files and folders recursively from a folder

5. Copy data / file from BLOB to SQL Database / ADLS Gen2

A. As CSV, TSV, Parquet, Avro, ORC etc.

6. Databricks Notebook
7. Azure Function
8. Lookup, Stored Procedure
9. Get Metadata, Delete
10. Execute Pipeline
11. Validation, Fail
12. Iteration & Conditionals

A. Filter
B. ForEach
C. If Condition
D. Switch
E. Until

6. Triggers

1. What is a trigger?
2. Types

A. Schedule
B. Tumbling window
C. Storage Events

3. Triggers with Parameters

7. Integration Runtime (IR)

1. Azure AutoResolveIntegrationRuntime
2. Azure Managed Virtual Network
3. Self-Hosted
4. Linked Self-Hosted

8. Source control

1. Git configuration
2. ARM Template

A. Export / Import

3. Azure Devops Repos

9. Global parameters

10. Credentials

11. Managed private endpoints

12. Monitoring ADF Jobs

13. Alerts

14. Send Failure Notifications using Logic Apps

15. Data Flows

1. What is Data Flow?
2. Mapping Data Flow
3. Data Flow Debug
4. Transformations

A. Filter, Aggregate, Join
B. Conditional Split, Derived Column
C. Exists, Union, Lookup, Sort,
D. GroupBy, Pivot, Unpivot, Flatten etc.
E. Flatten, parase, stringify
F. Filter sort, alterrow,asset
G. flowlet

5. Validate Schema, Schema Drift
6. Remove Duplicate Rows using Mapping Data Flows in Azure Data Factory
7. Slowly Changing Dimension (SCD) Types Using Mapping Data Flow in Azure Data Factory

A. Type 1
B. Type 2

16. ADF Exercises

1. Create variables using set variable activity
2. How to use if condition using if condition activity
3. Iterating files using for loop activity
4. Creating linked services, Data sets
5. Copy activity - blob to blob
6. Copy activity - blob to azure SQL
7. Copy activity - pattern matching files copy
8. Copy activity - copy the filtered file formats
9. Copy activity - copy multiple files from blob to another blob
10. Copy activity - Delete source files after copy activity
11. Copy activity - using parameterized data sets
12. Copy activity - convert one file format to another file format
13. Copy activity - add additional columns to the source columns
14. Copy activity - filter files and copy from one blob to another
15. Delete the files from blob with more than 100KB
16. How to use getmetdata activity
17. Bulk copy tables and files
18. How to integrate keyvault in ADF
19. How to set up integration run time
20. Copy data from on premises to azure cloud
21. How to use databricks activity activity and pass paraemeters to it
22. How to use scheduling trigger
23. How to use tumbling window trigger
24. How to use event based trigger
25. How to use with Activity
26. How to use Until Activity
27. Dataflows - select the rows
28. Dataflows - Filter the rows
29. Dataflows - join Transformations
30. Dataflows - union Transformations
31. Dataflows - look up Transformations
32. Dataflows - window functions transformations
33. Dataflows - pivot, unpivot transformations
34. Dataflows - Alter rows transformations
35. Dataflows - Removing Duplicates transformations
36. Dataflows - SCD Types transformations
37. How to pass parameters to the pipeline
38. How to create alerts and rules
39. How to set global parameters
40. How to import and export ARM templates
41. How to integrate ADF with Devops
42. How to use Azure devops Repos
43. How to use Azure devops boards
44. How to implement pipelines in Azure devops
45. How to send mail notifications using logic apps
46. How to monitor the pipelines
47. How to debug the pipelines
48. How to schedule pipeline using triggers
49. How to create trigger dependency
50. How to add one pipeline in another pipeline