Oozie Spark Action Not Loading Spark Configurations

Recently I was working on an issue that Oozie was not able to pick up Spark’s configuration and caused job to fail. The reason that I know it was not loading Spark’s configuration was because spark had “spark.authenticate=true” set in its configuration file under file /etc/spark/conf/spark-defaults.conf. $ head /etc/spark/conf/spark-defaults.conf spark.authenticate=true …

Oozie Hive2 Action Failed with Error: “HiveSQLException: Failed to execute session hooks”

If you have an Oozie Hive2 job that fails with below error message randomly, which can be found in Oozie’s server log, located by default under /var/log/oozie: 2018-06-02 09:00:01,103 WARN org.apache.oozie.action.hadoop.Hive2Credentials: SERVER[hlp3058p.oocl.com] USER[dmsa_appln] GROUP[-] TOKEN[] APP[DMSA_CMTX_PCON_ETL_ONLY] JOB[0010548-180302135253124-oozie-oozi-W] ACTION[0010548-180302135253124-oozie-oozi-W@spark-6799] Exception in addtoJobConf org.apache.hive.service.cli.HiveSQLException: Failed to execute session hooks at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:241) at …

Oozie Spark Actions Fail with Error “Spark config without ‘=’: –conf”

Currently Oozie provides easy interface for Spark1 jobs via Spark1 action, so that user does not have to embed spark-submit into shell action. However, recently I have discovered an issue in Oozie that it has a bug to parse Spark configurations and incorrectly generated a spark-submit command to submit Spark …

Oozie SSH Action Failed With “externalId cannot be empty” Error

Last week I was working with an issue that when running a very simple SSH action through Oozie, the job kept failing with “externalId cannot be empty” error. The workflow only had one single SSH action, and nothing else. See the workflow example below: <workflow-app name="SSH Action Test" xmlns="uri:oozie:workflow:0.5"> <start …

Oozie Server failed to Start with error java.lang.NoSuchFieldError: EXTERNAL_PROPERTY

This issue happens in CDH distribution of Hadoop that is managed by Cloudera Manager (possibly in other distributions as well, due to known upstream JIRA, but I have not tested). Oozie will fail to start after enabling Oozie HA through Cloudera Manager user interface. The full error message from Oozie’s …