Courses Offered: SCJP SCWCD Design patterns EJB CORE JAVA AJAX Adv. Java XML STRUTS Web services SPRING HIBERNATE  

       

BIG DATA Analytics Course Details
 

Subcribe and Access : 5200+ FREE Videos and 21+ Subjects Like CRT, SoftSkills, JAVA, Hadoop, Microsoft .NET, Testing Tools etc..

Batch Date: Nov 30th & Dec 1st @6:00PM

Faculty: Mr. N. Vijay Sunder Sagar (20+ Yrs Of Exp,..)

Duration: 12 Weekends Batch

Venue :
DURGA SOFTWARE SOLUTIONS,
Flat No : 202, 2nd Floor,
HUDA Maitrivanam,
Ameerpet, Hyderabad - 500038

Ph. No: +91 - 9246212143, 80 96 96 96 96



Syllabus:

BIG DATA HADOOP

I: INTRODUCTION

  • What is Big Data?
  • What is Hadoop?
  • Need of Hadoop
  • Sources and Types of Data
  • Comparison with Other Technologies
  • Challenges with Big Data
    • i. Storage
    • ii. Processing
  • RDBMS vs Hadoop
  • Advantages of Hadoop
  • Hadoop Echo System components

II: HDFS (Hadoop Distributed File System)

  • Features of HDFS
  • Name node ,Data node ,Blocks
  • Configuring Block size,
  • HDFS Architecture ( 5 Daemons)
    • i. Name Node
    • ii. Data Node
    • iii. Secondary Name node
    • iv. Job Tracker
    • v. Task Tracker
  • Metadata management
  • Storage and processing
  • Replication in Hadoop
  • Configuring Custom Replication
  • Fault Tolerance in Hadoop
  • HDFS Commands

III: MAP REDUCE

  • Map Reduce Architecture
  • Processing Daemons of Hadoop
    • Job Tracker (Roles and Responsibilities)
    • Task Tracker(Roles and Responsibilities)
  • Phases of Map Reduce
    • i) Mapper phase
    • ii) Reducer phase
  • Input split
  • Input split vs Block size
  • Partitioner in Map Reduce
  • Groupings and Aggregations
  • Data Types in Map Reduce
  • Map Reduce Programming Model
    • Driver Code
    • Mapper Code
    • Reducer Code
  • Programming examples
  • File input formats
  • File output formats
  • Merging in Map Reduce
  • Speculative Execution Model
  • Speculative Job

IV: SQOOP (SQL + HADOOP)

  • Introduction to Sqoop
  • SQOOP Import
  • SQOOP Export
  • Importing Data From RDBMS to HDFS
  • Importing Data From RDBMS to HIVE
  • Importing Data From RDBMS to HBASE
  • Exporting From HASE to RDBMS
  • Exporting From HBASE to RDBMS
  • Exporting From HIVE to RDBMS
  • Exporting From HDFS to RDBMS
  • Transformations While Importing / Exporting
  • Filtering data while importing
  • Vertical and Horizontal merging while import
  • Working with delimiters while importing
  • Groupings and Aggregations while import
  • Incremental import
  • Examples and operations
  • Defining SQOOP Jobs

V: YARN

  • Introduction
  • Speculative Execution ,Speculative job and
  • Speculative Task.
  • Comparision of Hadoop1.xx with Hadoop2.xx
  • Comparision with previous versions
  • YARN Architecture Componets
    • i. Resource Manager
    • ii. Application Master
    • iii. Node Manager
    • iv. Application Manager
    • v. Resource Scheduler
    • vi. Job History Server
    • vii. Container

VI: NOSQL

  • What is “Not only SQL”
  • NOSQL Advantages
  • What is problem with RDBMS for Large
  • Data Scaling Systems
  • Types of NOSQL & Purposes
  • Key Value Store
  • Columer Store
  • Document Store
  • Graph Store
  • Introduction to cassandra – NOSQL Database
  • Introduction to MongoDB and CouchDB Database
  • Intergration of NOSQL Databases with Hadoop

VII: HBASE

  • Introduction to big table
  • What is NOSQL and colummer store Database
  • HBASE Introduction
  • Hbase use cases
  • Hbase basics
  • Column families
  • Scans
  • Hbase Architecture
  • Map Reduce Over Hbase
  • Hbase data Modeling
  • Hbase Schema design
  • Hbase CRUD operators
  • Hive & Hbaseinteragation
  • Hbase storage handlers

VIII: HIVE

  • Introduction
  • Hive Architecture
  • Hive Metastore
  • Hive Query Launguage
  • Difference between HQL and SQL
  • Hive Built in Functions
  • Loading Data From Local Files To Hive Tables
  • Loading Data From Hdfs Files To Hive Tables
  • Tables Types
  • Inner Tables
  • External Tables
  • Hive Working with unstructured data
  • Hive Working With Xml Data
  • Hive Working With Json Data
  • Hive Working With Urls And Weblog Data
  • Hive Unions
  • Hive Joins
  • Multi Table / File Inserts
  • Inserting Into Local Files
  • Inserting Into Hdfs Files
  • Hive UDF (user defined functions)
  • Hive UDAF (user defined Aggregated functions)
  • Hive UDTF (user defined table Generated functions
  • Partitioned Tables
  • Non – Partitioned Tables
  • Multi-column Partitioning
  • Dynamic Partitions In Hive
  • Performance Tuning mechanism
  • Bucketing in hive
  • Indexing in Hive
  • Hive Examples
  • Hive & Hbase Integration

 

PYSPARK

I ) PYSPARK INTRODUCTION

  • What is Apache Spark?
  • Why Pyspark?
  • Need for pyspark
  • spark Python Vs Scala
  • pyspark features
  • Real-life usage of PySpark
  • PySpark Web/Application
  • PySpark - SparkSession
  • PySpark – SparkContext
  • PySpark – RDD
  • PySpark – Parallelize
  • PySpark – repartition() vs coalesce()
  • PySpark – Broadcast Variables
  • PySpark – Accumulator

II) PYSPARK - RDD COMPUTATION

  • Operations on a RDD
  • Direct Acyclic Graph (DAG)
  • RDD Actions and Transformations
  • RDD computation
  • Steps in RDD computation
  • RDD persistence
  • Persistence features

II) PERSISTENCE Options:

  • 1) MEMORY_ONLY
  • 2) MEMORY_SER_ONLY
  • 3) DISK_ONLY
  • 4) DISK_SER_ONLY
  • 5) MEMORY_AND_DISK_ONLY

III) PYSPARK - CORE COMPUTING

  • Fault Tolerence model in spark
  • Different ways of creating a RDD
  • Word Count Example
  • Creating spark objects(RDDs) from Scala Objects(lists).
  • Increasing the no of partitons
  • Aggregations Over Structured Data:
  • reduceByKey()

IV) GROUPINGS AND AGGREGATIONS

  • i) Single Grouping and Single Aggregation
  • ii) Single Grouping and multiple Aggregation
  • iii) multi Grouping and Single Aggregation
  • iv) Multi Grouping and Multi Aggregation
  • Differences b/w reduceByKey() and groupByKey()
  • Process of groupByKey
  • Process of reduceByKey
  • Reduce() function
  • Various Transformations
  • Various Built-in Functions

V) Various Actions and Transformations:

  • countByKey()
  • countByValue()
  • sortByKey()
  • zip()
  • Union()
  • Distinct()
  • Various count aggregation
  • Joins
  • -inner join
  • -outer join
  • Cartesian()
  • Cogroup()
  • Other actions and transformations

VI) PySpark SQL - DataFrame

  • Introduction
  • Making data Structured
  • Case Classes
  • ways to extract case class objects
  • 1) using function
  • 2) using map with multiple exressions
  • 3) using map with single expression
  • Sql Context
  • Data Frames API
  • DataSet API
  • RDD vs DataFrame vs DataSet
  • PySpark – Create a DataFrame
  • PySpark – Create an empty DataFrame
  • PySpark – Convert RDD to DataFrame
  • PySpark – Convert DataFrame to Pandas
  • PySpark – show()
  • PySpark – StructType & StructField
  • PySpark – Row Class
  • PySpark – Column Class
  • PySpark – select()
  • PySpark – collect()
  • PySpark – withColumn()
  • PySpark – withColumnRenamed()
  • PySpark – where() & filter()
  • PySpark – drop() & dropDuplicates()
  • PySpark – orderBy() and sort()
  • PySpark – groupBy()
  • PySpark – join()
  • PySpark – union() & unionAll()
  • PySpark – unionByName()
  • PySpark – UDF (User Defined Function)
  • PySpark – map()
  • PySpark – flatMap()
  • pyspark – foreach()
  • PySpark – sample() vs sampleBy()
  • PySpark – fillna() & fill()
  • PySpark – pivot() (Row to Column)
  • PySpark – partitionBy()
  • PySpark – ArrayType Column (Array)
  • PySpark – MapType (Map/Dict)

VII) PySpark SQL Functions

  • PySpark – Aggregate Functions
  • PySpark – Window Functions
  • PySpark – Date and Timestamp Functions
  • PySpark – JSON Functions
  • PySpark – Read & Write JSON file

VIII) PySpark Built-In Functions

  • PySpark – when()
  • PySpark – expr()
  • PySpark – lit()
  • PySpark – split()
  • PySpark – concat_ws()
  • Pyspark – substring()
  • PySpark – translate()
  • PySpark – regexp_replace()
  • PySpark – overlay()
  • PySpark – to_timestamp()
  • PySpark – to_date()
  • PySpark – date_format()
  • PySpark – datediff()
  • PySpark – months_between()
  • PySpark – explode()
  • PySpark – array_contains()
  • PySpark – array()
  • PySpark – collect_list()
  • PySpark – collect_set()
  • PySpark – create_map()
  • PySpark – map_keys()
  • PySpark – map_values()
  • PySpark – struct()
  • PySpark – countDistinct()
  • PySpark – sum(), avg()
  • PySpark – row_number()
  • PySpark – rank()
  • PySpark – dense_rank()
  • PySpark – percent_rank()
  • PySpark – typedLit()
  • PySpark – from_json()
  • PySpark – to_json()
  • PySpark – json_tuple()
  • PySpark – get_json_object()
  • PySpark – schema_of_json()
  • Working Examples

IX) Pyspark External Sources

  • Working with sql statements
  • Spark and Hive Integration
  • Spark and mysql Integration
  • Working with CSV
  • Working with JSON
  • Transformations and actions on dataframes
  • Narrow, wide transformations
  • Addition of new columns, dropping of columns ,renaming columns
  • Addition of new rows, dropping rows
  • Handling nulls
  • Joins
  • Window function
  • Writing data back to External sources
  • Creation of tables fromDataframes (Internal tables, Temporary tables)

X) DEPLOYMENT MODES

  • Local Mode
  • Cluster Modes(Standalone , YARN

XI) PYSPARK APLLICATION

  • Stages and Tasks
  • Driver and Executor
  • Building spark applications/pipelines
  • Deploying spark apps to cluster and tuning
  • Performance tuning

PySpark Streaming Concepts

Integration with Kafka

PySpark-mllib