SPARK CONTEXT
.What is Spark Context?
- SparkContext instance
- sets up internal services for a Spark Application and also
- establishes a connection to a Spark execution environment.
- Spark Context should be created by Spark Driver Application.
- SparkContext is the entry point to Spark.
- Using sparkContext you create RDDs which provided various ways of churning data.
- When a programmer creates a RDDs, SparkContext connect to the Spark cluster to create a new SparkContext object.
- SparkContext tell spark how to access the cluster.
- SparkConf is key factor to create programmer application.
Various ways to create contexts in spark ?
- Sparkconext
- Sqlcontext
- Sparksession
- Sqlcontext.sparkcontext
.How does Spark Context in Spark Application pick the value for Spark Master?
That can be done in two ways.
- Create a new SparkConf object and set the master using its setMaster() method.
- This Spark Configuration object is passed as an argument while creating the new Spark Context.
SparkConf conf = new SparkConf().setAppName(“JavaKMeansExample”)
.setMaster(“local[2]”)
.set(“spark.executor.memory”,”3g”)
.set(“spark.driver.memory”, “3g”);
- This Spark Configuration object is passed as an argument while creating the new Spark Context.
- JavaSparkContext jsc = new JavaSparkContext(conf);
<apache-installation-directory>/conf/spark-env.sh file, located locally on the machine, contains information regarding Spark Environment configuration. - Spark Master is one the parameters that could be provided in the configuration file