Spark jobs failed with delegation token renewal error

An Oozie Spark job failed with the following error: Job aborted due to stage failure: Task 103 in stage 194576.0 failed 4 times, most recent failure: Lost task 103.3 in stage 194576.0 (TID 119674041, ): org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for sparkpse: HDFS_DELEGATION_TOKEN owner=@HADOOP.CHARTER.COM, renewer=yarn, realUser=, issueDate=1482494610879, maxDate=1483099410879, sequenceNumber=274718, masterKeyId=166) can't be …

How to load different version of Spark into Oozie

This article explains the steps needed to load Spark2 into Oozie under CDH5.9.x which comes with Spark1.6. Although this was tested under CDH5.9.0, it should be similar for earlier releases. Please follow the steps below: Locate the current shared-lib directory by running: oozie admin -oozie http://<oozie-server-host>:11000/oozie -sharelibupdate you will get …