Big Data: week 8- Dataflow, HDFS, Spark

An intermediary between an unstructured data Lake and a structured database

• "Batch process"
• "streaming process"

• Object oriented databases
• NoSQL ('not just') databases
• Cloud databases
• Self-driving databases (uses machine learning to learn about the data)

A type of database. ETL is done as the data comes in, and then the data is stored

extract, transform, load

(Extract, Transform, Load)

structured schema, de normalized data ready for dashboards, analysis, etc

Store large amounts of data in a standard format in a central database

• Maintain historical records
• Keep data secure by storing it in a single locatoin
• Provides quick access to enable faster business decisions

• OnLine Transactional Processing
• Used by Databases

optimized to add, modify, delete records a lot

• OnLine Analytical Processing
• Used by Data Warehouses

optimized to execute a smaller number of complex queries

Normalized format

Reduces redundancy and increases consistency as data isn't stored in multiple locations

Denormalized format

More query efficient, but the data may exist in multiple places and become inconsistent

A focused version of data warehouse for special teams or departments

A central repository for raw data to go until it is needed

can handled structured, unstructured or semi-structured data

Usually includes raw data and data after ETL

streaming data

Batch data is updated in bulk

Streaming data is updated in 'real-time'

What happens when a data lake has poor data management

• Can happen if:
• has no metadata
• broken injestion procss
• broken metadata management
• no data governance

an intermediary between the unstructured data lake and very structured database/data warehouse

Storage technology that can power the lake house

Guarantees ACID transactions

Hadoop Distributed File System

Clusters data on multiple computers to analyze datasets in parallel

• Hadoop Distributed File System (HDFS)
• Amazon's Simple Storage Service (S3)
• Google's Cloud Storage (GCS)
• Azure's Blob Storage

works well for computations that can be split, run in parallel and combined

a signal is sent from a datanode to a namenode. If the namenode sees no signal, the datanode is dead

when datanodes fail, data may be under-replicated but namenodes will send signals to replicate and balance data replicates

Yet another resource negotiator

platform responsible for managing computing resources in clusters

Balancing- when datanodes fail, data may be under-replicated. Name node will send signals to replicate and balance data

programming paradigm that enables massive scalability across hundreds or thousands of servers in a Hadoop cluster

Leverages parallel computing

HDFS requires a decent amount of "on-premises" infrastructure

difficulties with scalability

adding more machines or larger disk spaces

adding additional computational power (CPU, RAM)

Object storage-includes the data, metadata, and a unique identifier

S3, GCS, Blob storage systems

Five major components to Spark:

Spark Core as its foundation

Spark SQL for SQL queries

Spark Streaming for real-time analytics

Spark MLlib for machine learning

Spark GraphX for graphic processing

Using pySpark!

allows for massively parallelized operations without worrying about worker distribution or fault tolerance.

• Does similar things as MapReduce, but is built to be faster by processing in memory (RAM)
• 

HDFS, Csv, json, S3, GCS (most data types!)

If your data is in an HDFS, spark's cluster manager will work with YARN

If you have data elsewhere, spark has its own cluster manager to determine how to split up data and run computations

Its what you start when you want to use Spark

• Here we define:
• What the cluster is and Workers to handle computation on that cluster/node.
• Central Spark coordinator (i.e. the Driver)
• The name of the app

sometimes you'll put a cluster url where 'local[*]' is in the code

Resilient Distributed Datasets

The core object that Spark works on

RDD- an immutable distributed collection of elements of your data, partitioned across nodes in your cluster that can be operated in parallel with a low-level API that offers transformations and actions

Transformation: something that creates a new RDD (say by filtering, grouping, or mapping)

Action: operation applied to an RDD that performs a computation and sends the result back

`.sparkContext.parallelize()` method

• map(): apply function to each RDD to return a new RDD
• filter(): returns a subsetted RDD

• elect()
• filterByRange()
• groupByKey()
• reduceByKey()
• distinct()
• sample()
• union()

• reduce()
• count()
• min()
• max()
• collect()
• take()
• first()
• foreach()
• aggregate()

If some data went down, backups were used (because data was re-replicated)

Turns transformations and actions into a directed acyclic graph (DAG) that allows computation to be picked back up if something fails

- they don't comute their resultes right away

• Transformations are built up through a DAG
• - computation only done when an action requires a result

Directed acyclic graph

allows computation to be picked back up if something else fails

Spark can share variables across clusters or machines

Broadcast variables give access to common (read-only) variables to all workers

Spark can share variables across clusters or machines

Accumulators are variables that each worker can do an operation on (usually things like sums)