Skip to content

Instantly share code, notes, and snippets.

@wavescholar

wavescholar/gist:5589c5252fac487d69d8

Last active August 29, 2015 14:16
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save wavescholar/5589c5252fac487d69d8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save wavescholar/5589c5252fac487d69d8 to your computer and use it in GitHub Desktop.
Download ZIP
Spark Notes
Raw
gistfile1.txt
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[2] --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10
val lines = sc.textFile("README.md")
val rdd2 = sc.textFile("hdfs:///some/path.txt")
Spark UI
http://10.22.7.183:4040/jobs/
val rdd_er = sc.textFile("hdfs:///apps/hive/warehouse/meterdata.db/er")
val rdd_s = rdd_er.sample(false,1,20)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment