SPARK CONTEXT

SPARK CONTEXT

.What is Spark Context?

  • SparkContext instance
    • sets up internal services for a Spark Application and also
    • establishes a connection to a Spark execution environment.
  • Spark Context should be created by Spark Driver Application.
  • SparkContext is the entry point to Spark.
  • Using sparkContext you create RDDs which provided various ways of churning data.
  • When a programmer creates a RDDs, SparkContext connect to the Spark cluster to create a new SparkContext object.
  • SparkContext tell spark how to access the cluster.
  • SparkConf is key factor to create programmer application.

Various ways to create contexts in spark ?

  •  Sparkconext
  •  Sqlcontext
  •  Sparksession
  •  Sqlcontext.sparkcontext

.How does Spark Context in Spark Application pick the value for Spark Master?

That can be done in two ways.

  • Create a new SparkConf object and set the master using its setMaster() method.
    • This Spark Configuration object is passed as an argument while creating the new Spark Context.
      SparkConf conf = new SparkConf().setAppName(“JavaKMeansExample”)
      .setMaster(“local[2]”)
      .set(“spark.executor.memory”,”3g”)
      .set(“spark.driver.memory”, “3g”);
  • JavaSparkContext jsc = new JavaSparkContext(conf);
    <apache-installation-directory>/conf/spark-env.sh file, located locally on the machine, contains information regarding Spark Environment configuration.
  • Spark Master is one the parameters that could be provided in the configuration file