Master Big Data with Spark: Learn Scala, RDDs, SparkSQL, MLlib, and streaming. Build distributed systems using Hadoop, HDFS, and Kafka for real-time analytics and machine learning on large datasets.
| Date | Venue | Duration | |
|---|---|---|---|
| Available on Request | Sandton | 3 Days |
The analysis of large datasets involves using an equally large set of computers. Successfully using so many computers entails the use of distributed file systems, such as the Hadoop Distributed File System (HDFS), and parallel computational models, such as Hadoop, MapReduce, and Spark.
In this Big Data Analytics with Spark Training Course, you will learn the essential components of vast parallel computation projects and how to use Spark to minimize bottlenecks. This course will teach you how to conduct supervised and unsupervised machine learning on substantial datasets using the Machine Learning Library (MLlib) and gain hands-on experience using PySpark.
Skills Covered
This Big Data Analytics with Spark Training program will provide you with knowledge and expertise in:
Scala programming
Spark installation
Resilient Distributed Datasets (RDD)
SparkSQL
Spark Streaming
Spark ML Programming
GraphX programming
Upon successfully completing this Big Data Analytics with Spark Training course, participants will be able to:
Obtain an overview of Big Data & Hadoop, including HDFS and YARN (Yet Another Resource Negotiator)
Gain comprehensive knowledge of various tools in the Spark ecosystem
Understand how to ingest data into HDFS using Sqoop & Flume
Program Spark using PySpark
Identify the computational trade-offs in a Spark application
Model data using statistical and machine learning methods
Use real-time data feeds through a publish-subscribe messaging system like Kafka
Gain exposure to various real-life industry-based projects
Study projects in diverse domains, such as banking, telecommunications, social media, and government
Organisational Benefits
Companies that send employees to participate in this course can benefit by:
Adopting technology used successfully by multiple companies in various domains globally
Attracting more investors, as 56% of enterprises will increase their investment in big data over the next three years (according to Forbes)
Providing the workforce with flexible and cost-effective professional development opportunities
Analysing case studies in this domain and applying successful techniques in their organization
Comprehending the principles and practice of Big Data Analytics and its operational context
This Big Data Analytics with Spark Training course is suitable for:
Developers and Architects
BI / ETL / DW Professionals
Senior IT Professionals
Testing Professionals
Mainframe Professionals
Freshers
Big Data Enthusiasts
Software Architects, Engineers, and Developers
Data Scientists and Analytics Professionals
Our diverse instructional approaches ensure effective learning:
– Lectures & Presentations: Engage with expert-driven, stimulating content.
– Course Material: Access well-crafted supporting resources.
– Group Work: Collaborate on discussions and case studies for practical insights.
– Workshops & Role-Play: Participate in immersive, scenario-based activities.
– Practical Application: Focus on applying theoretical knowledge in real situations.
– Post-Training Support: Receive extensive support after training for skill implementation.
Module 1: Introduction to Big Data Hadoop and Spark
What is Big Data?
Big Data Customer Scenarios
Big Data and Hadoop
How Hadoop Solves the Big Data Problem
What is Hadoop?
Hadoop’s Key Characteristics
Hadoop Ecosystem and HDFS
Hadoop Core Components
Rack Awareness and Block Replication
YARN and its Advantage
Hadoop Cluster and its Architecture
Hadoop: Different Cluster Modes
Why Spark is needed?
What is Spark?
How Spark differs from other frameworks?
Spark at Yahoo!
Module 2: Introduction to Scala for Apache Spark
What is Scala?
Why Scala for Spark?
Scala in other Frameworks
Control Structures in Scala
Foreach loop, Functions, and Procedures
Collections in Scala: Array
Introduction to Scala REPL
Basic Scala Operations
Variable Types in Scala
ArrayBuffer, Map, Tuples, Lists, and more
Scala REPL Detailed Demo
Module 3: Functional Programming and OOP Concepts in Scala
Auxiliary Constructor and Primary Constructor
Singletons
Extending a Class
Overriding Methods
Traits as Interfaces and Layered Traits
OOP Concepts
Functional Programming
Higher-Order Functions
Anonymous Functions
Class in Scala
Getters and Setters
Custom Getters and Setters
Properties with only Getters
Module 4: Deep Dive into Apache Spark Framework
Submitting Spark Job
Spark Web UI
Data Ingestion using Sqoop
Building and Running Spark Application
Spark Application Web UI
Spark’s Place in the Hadoop Ecosystem
Spark Components & its Architecture
Spark Deployment Modes
Introduction to Spark Shell
Writing your first Spark Job Using SBT
Configuring Spark Properties
Data ingestion using Sqoop
Module 5: Playing with Spark RDDs
RDD Persistence
WordCount Program Using RDD Concepts
Passing Functions to Spark
Loading data in RDDs
Saving data through RDDs
RDD Transformations
Challenges in Existing Computing Methods
Probable Solution & How RDD Solves the Problem
What is RDD, Its Operations, Transformations & Actions
Data Loading and Saving Through RDDs
Key-Value Pair RDDs
Other Pair RDDs, Two Pair RDDs
RDD Lineage
RDD Actions and Functions
RDD Partitions
WordCount through RDDs
Module 6: DataFrames and Spark SQL
Need for Spark SQL
What is Spark SQL?
Spark SQL Architecture
Spark – Hive Integration
Spark SQL – Creating Data Frames
Loading and Transforming Data through Different Sources
Stock Market Analysis
Spark-Hive Integration
SQL Context in Spark SQL
User-Defined Functions
Data Frames & Datasets
Interoperating with RDDs
JSON and Parquet File Formats
Loading Data through Different Sources
Module 7: Machine Learning Using Spark MLlib
Why Machine Learning?
What is Machine Learning?
Where Machine Learning is Used?
Face Detection: USE CASE
Different Types of Machine Learning Techniques
Introduction to MLlib
Features of MLlib and MLlib Tools
Various ML algorithms supported by MLlib
Module 8: Deep Dive into Spark MLlib
K-Means Clustering
Linear Regression
Logistic Regression
Decision Tree
Random Forest
Machine Learning MLlib
Module 9: Understanding Apache Kafka and Apache Flume
What is Apache Flume?
Need of Apache Flume
Basic Flume Architecture
Flume Sources
Flume Sinks
Flume Channels
Flume Configuration
Need for Kafka
What is Kafka?
Core Concepts of Kafka
Kafka Architecture
Where is Kafka Used?
Understanding the Components of Kafka Cluster
Configuring Kafka Cluster
Kafka Producer and Consumer Java API
Integrating Apache Flume and Apache Kafka
Configuring Single Node Single Broker Cluster
Configuring Single Node Multi Broker Cluster
Producing and Consuming Messages
Flume Commands
Setting up Flume Agent
Streaming Twitter Data into HDFS
Module 10: Streaming – Multiple Batches
Why Streaming is Necessary?
Drawbacks in Existing Computing Methods
What is Spark Streaming?
Spark Streaming Features
Spark Streaming Workflow
How Uber Uses Streaming Data
Streaming Context & DStreams
Transformations on DStreams
Important Windowed Operators
Slice, Window, and ReduceByWindow Operators
Stateful Operators
Module 11: Apache Spark Streaming – Data Sources
Apache Spark Streaming: Data Sources
Apache Flume and Apache Kafka Data Sources
Example: Using a Kafka Direct Data Source
Perform Twitter Sentiment Analysis Using Spark Streaming
Streaming Data Source Overview
Different Streaming Data Sources
Module 12: Spark GraphX
Key Concepts of Spark GraphX
GraphX Algorithms and their Implementations
Discover how our courses enhance professionals’ effectiveness in their workplaces.
No courses found.
Fill in your details and we will send the brochure to your email.
Please check your email for confirmation.
Please check your email for confirmation.
Please check your email for confirmation.
We'll be in touch with you shortly.
