Spark Job SASL Authentication Error

Spark has an internal mechanism that authenticates executors with the driver controlling a given application. This can be controlled by setting “spark.authenticate” to “true”, as part of spark-submit’s parameters, like below: spark-submit –master yarn-cluster –conf spark.authenticate=true –conf spark.dynamicAllocation.enabled=true …. This setting is required if you have “spark.authenticate.enableSaslEncryption” enabled, or called …

Oozie Spark Action Not Loading Spark Configurations

Recently I was working on an issue that Oozie was not able to pick up Spark’s configuration and caused job to fail. The reason that I know it was not loading Spark’s configuration was because spark had “spark.authenticate=true” set in its configuration file under file /etc/spark/conf/spark-defaults.conf. $ head /etc/spark/conf/spark-defaults.conf spark.authenticate=true …