Oozie Server failed to Start with error java.lang.NoSuchFieldError: EXTERNAL_PROPERTY

This issue happens in CDH distribution of Hadoop that is managed by Cloudera Manager (possibly in other distributions as well, due to known upstream JIRA, but I have not tested). Oozie will fail to start after enabling Oozie HA through Cloudera Manager user interface.

The full error message from Oozie’s process stdout.log (can be found under /var/run/cloudera-scm-agent/process/XXX-oozie-OOZIE_SERVER/logs directory) file looks like below:

Wed Jan 25 11:07:41 GST 2017 
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera 
using 5 as CDH_VERSION 
using /var/lib/oozie/tomcat-deployment as CATALINA_BASE 
Copying JDBC jar from /usr/share/java/oracle-connector-java.jar to /var/lib/oozie 

ERROR: Oozie could not be started 

REASON: java.lang.NoSuchFieldError: EXTERNAL_PROPERTY 

Stacktrace: 
----------------------------------------------------------------- 
java.lang.NoSuchFieldError: EXTERNAL_PROPERTY 
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector._findTypeResolver(JacksonAnnotationIntrospector.java:777) 
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findPropertyTypeResolver(JacksonAnnotationIntrospector.java:214) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findPropertyTypeSerializer(BeanSerializerFactory.java:370) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory._constructWriter(BeanSerializerFactory.java:772) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(BeanSerializerFactory.java:586) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.constructBeanSerializer(BeanSerializerFactory.java:430) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanSerializer(BeanSerializerFactory.java:343) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:287) 
at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782) 
at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735) 
at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344) 
at org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420) 
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601) 
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) 
at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2566) 
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:2056) 
at org.apache.oozie.util.FixedJsonInstanceSerializer.serialize(FixedJsonInstanceSerializer.java:65) 
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:201) 
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:186) 
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217) 
at org.apache.oozie.util.ZKUtils.<init>(ZKUtils.java:141) 
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154) 
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70) 
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) 
at org.apache.oozie.service.Services.setService(Services.java:372) 
at org.apache.oozie.service.Services.loadServices(Services.java:305) 
at org.apache.oozie.service.Services.init(Services.java:213) 
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) 
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210) 
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709) 
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802) 
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) 
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) 
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) 
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) 
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) 
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) 
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) 
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) 
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068) 
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822) 
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060) 
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) 
at org.apache.catalina.core.StandardService.start(StandardService.java:525) 
at org.apache.catalina.core.StandardServer.start(StandardServer.java:759) 
at org.apache.catalina.startup.Catalina.start(Catalina.java:595) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) 
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) 

To fix the issue, please follow the steps below:

  1. Delete or move the following files under CDH’s parcel directory (most likely they are symlinks):
    /opt/cloudera/parcels/CDH/lib/oozie/libserver/hive-exec.jar
    /opt/cloudera/parcels/CDH/lib/oozie/libtools/hive-exec.jar
    
  2. Download hive-exec-{cdh version}-core.jar file from the Cloudera repo, for example, for CDH5.8.2, please go to:
    https://repository.cloudera.com/cloudera/cloudera-repos/org/apache/hive/hive-exec/1.1.0-cdh5.8.2/

    and put the file under the following directories on the Oozie server:

    /opt/cloudera/parcels/CDH/lib/oozie/libserver/
    /opt/cloudera/parcels/CDH/lib/oozie/libtools/
    
  3. Download kryo-2.22.jar from the maven repository:
    http://repo1.maven.org/maven2/com/esotericsoftware/kryo/kryo/2.22/kryo-2.22.jar

    and put it under directories on the Oozie server:

    /opt/cloudera/parcels/CDH/lib/oozie/libserver/
    /opt/cloudera/parcels/CDH/lib/oozie/libtools/
    
  4. Finally restart Oozie service

This is a known Oozie issue and reported in the upstream JIRA: OOZIE-2621, which has been resolved and targeted for 4.3.0 release.

Hope this helps.

How to setup multiple KDCs through Cloudera Manager

Currently Cloudera Manager does not support setting up multiple KDCs for the krb5.conf file natively, this article explains the workarounds we can have using the existing feature provided by Cloudera Manager.

This article also assumes that you have krb5.conf file managed by Cloudera Manager.

If you are using Cloudera Manager prior to 5.7, following the steps below:

  1. Go to CM > Administration > Settings > click on “Kerberos” on Filters on the left side > locate “KDC Server Host”, enter the KDC host in the text field:
    kdc-host1.com
    
  2. On the same page, locate “Advanced Configuration Snippet (Safety Valve) for the Default Realm in krb5.conf”, and enter the following into the text area:
    kdc = kdc-host2.com
    
  3. Save and then “Deploy Kerberos Client Configuration” (you might need to stop all service first before you can do this)
    The [realm] section in the krb5.conf will be updated like below:

    [realms]
    TEST.COM = {
    kdc = kdc-host1.com
    admin_server = kdc-host1.com
    kdc = kdc-host2.com
    }
    

If you are using CM5.7 and above, you can also do the following (above steps should still work):

  1. Go to CM > Administration > Settings > click on “Kerberos” on Filters on the left side > locate “KDC Server Host”, empty the KDC host in the text field, so that it contains no value
  2. On the same page, locate “Advanced Configuration Snippet (Safety Valve) for the Default Realm in krb5.conf”, and enter the following into the text area:
    kdc = kdc-host1.com
    kdc = kdc-host2.com
    admin_server = kdc-host1.com
    
  3. Save and then “Deploy Kerberos Client Configuration” (you might need to stop all service first before you can do this)
    The [realm] section in the krb5.conf will be updated like below:

    [realms]
    TEST.COM = {
    kdc = kdc-host1.com
    kdc = kdc-host2.com
    admin_server = kdc-host1.com
    }
    

The second option does not work prior to CM5.7 is because the older version of CM will generate the following line in krb5.conf if the KDC Server Host is empty:

kdc =

which will break the syntax in krb5.conf file.

How to use Cloudera Manager API to check a service role exited unexpectedly

This blog explains the steps to use Cloudera Manager API to check for a service role in CDH that was exited unexpectedly, so that proper action can be taken.

To check a service role’s status via Cloudera Manager API, please follow the steps below (I am taking Impala as an example):

  1. Determine the version of API you are using:
    curl -u username:password http://<cm-host>:7180/api/version
    

    in CDH5.7.x, it should return “v12”.

  2. get the cluster name from the output of:
    curl -u username:password http://<cm-host>:7180/api/v12/clusters
    

    sample output:

    {
      "items" : [ {
        "name" : "cluster",
        "displayName" : "Cluster 1",
        "version" : "CDH5",
        "fullVersion" : "5.7.0",
        "maintenanceMode" : false,
        "maintenanceOwners" : [ ],
        "clusterUrl" : "http://<cm-host>:7180/cmf/clusterRedirect/cluster",
        "hostsUrl" : "http://<cm-host>:7180/cmf/clusterRedirect/cluster/hosts",
        "entityStatus" : "GOOD_HEALTH"
      } ]
    }
    

    take note of the value for “name” attribute, in my case it is “cluster”.

  3. get the services in the cluster:
    curl -u username:password http://<cm-host>:7180/api/v12/clusters/<cluster-name>/services
    

    substitute with our cluster’s name:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/cluster/services
    

    please locate the impala service and get its “name”, in my case it is “impala”:

    {
        "name" : "impala",
        "type" : "IMPALA",
        "clusterRef" : {
          "clusterName" : "cluster"
        },
        ....
    }
    
  4. get the all the roles under impala service:
    curl -u username:password http://<cm-host>:7180/api/v12/clusters/<cluster-name>/services/<impala-name>/roles/
    

    in my case should be:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/cluster/services/impala/roles
    

    locate the role that you want to monitor, I picked Statestore:

    {
        "name" : "impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe",
        "type" : "STATESTORE",
        "serviceRef" : {
          "clusterName" : "cluster",
          "serviceName" : "impala"
        },
        ....
    }
    

    in my case it is “impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe”

  5. get the status for the role you want to monitor:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/<cluster-name>/services/<impala-name>/roles/<role-name>
    

    after substitution, it should be:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/cluster/services/impala/roles/impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe
    

    this will give you full status output for this particular role:

    {
      "name" : "impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe",
      "type" : "STATESTORE",
      "serviceRef" : {
        "clusterName" : "cluster",
        "serviceName" : "impala"
      },
      "hostRef" : {
        "hostId" : "eff96b49-739e-48d4-a19b-e5865a83b164"
      },
      .....
      "configStalenessStatus" : "FRESH",
      "maintenanceMode" : false,
      "maintenanceOwners" : [ ],
      "commissionState" : "COMMISSIONED",
      "roleConfigGroupRef" : {
        "roleConfigGroupName" : "impala-STATESTORE-BASE"
      },
      "entityStatus" : "GOOD_HEALTH"
    }
    

    look for the last attribute called “entityStatus”, it has the following possible values:

    UNKNOWN	
    NONE	
    STOPPED	
    DOWN	
    UNKNOWN_HEALTH	
    DISABLED_HEALTH	
    CONCERNING_HEALTH	
    BAD_HEALTH	
    GOOD_HEALTH	
    STARTING	
    STOPPING	
    HISTORY_NOT_AVAILABLE
    

    in the case that it is exited unexpectedly, the value would be “DOWN”, so that we can programmatically decide whether we can just restart it or not.

Please note that if you have enabled SSL for Cloudera Manager, the URL should be changed to: https://:7183 instead of http://:7180

More information about the apiRole entity can be found here: apiRole

Unable to generate keytab from within Cloudera Manager

When generating credentials through Cloudera Manager, sometimes Cloudera Manager will return you the following error:

/usr/share/cmf/bin/gen_credentials_ad.sh failed with exit code 53 and
output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf2781839247630884630.keytab
+ PRINC=sqoop2/<host>@REALM.COM
+ USER=kaupocSuFoZIOIDa
+ PASSWD=REDACTED
+ DIST_NAME=CN=kaupocSuFoZIOIDa,OU=Cloudera,OU=ServersUnix,OU=IT,OU=Basel,OU=AdminUnits,DC=emea,DC=XXXX,DC=com
+ '[' -z /etc/krb5-cdh.conf ']'
+ echo 'Using custom config path '\''/etc/krb5-cdh.conf'\'', contents below:'
+ cat /etc/krb5-cdh.conf
+ SIMPLE_PWD_STR=
+ '[' '' = '' ']'
+ kinit -k -t /var/run/cloudera-scm-server/cmf5575611164358256388.keytab
cdhad@REALM.COM
++ mktemp /tmp/cm_ldap.XXXXXXXX
+ LDAP_CONF=/tmp/cm_ldap.XRbR8Zco
+ echo 'TLS_REQCERT never'
+ echo 'sasl_secprops minssf=0,maxssf=0'
+ export LDAPCONF=/tmp/cm_ldap.XRbR8Zco
+ LDAPCONF=/tmp/cm_ldap.XRbR8Zco
++ ldapsearch -LLL -H ldaps://<ldap-host>:636 -b
OU=Cloudera,OU=ServersUnix,OU=IT,OU=Basel,OU=AdminUnits,DC=emea,DC=xxxx,DC=com
userPrincipalName=sqoop2/<host>@REALM.COM
SASL/GSSAPI authentication started
SASL username: cdhad@REALM
SASL SSF: 0
+ PRINC_SEARCH=
+ set +e
+ echo
+ grep -q userPrincipalName
+ '[' 1 -eq 0 ']'
+ set -e
+ ldapmodify -H ldaps://<ldap-host>:636
++ echo sqoop2/<host>@REALM.COM
++ sed -e 's/\@REALM.COM//g'
++ echo -n '"REDACTED"'
++ iconv -f UTF8 -t UTF16LE
++ base64 -w 0
SASL/GSSAPI authentication started
SASL username: cdhad@REALM.COMSASL SSF: 0
ldap_add: Server is unwilling to perform (53)
additional info: 0000052D: SvcErr: DSID-031A1248, problem 5003
(WILL_NOT_PERFORM), data 0
Generate credentials in Cloudera Manager failed with the following errors:

/usr/share/cmf/bin/gen_credentials_ad.sh failed with exit code 53 and
output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf2781839247630884630.keytab
+ PRINC=sqoop2/host@REALM.COM
+ USER=kaupocSuFoZIOIDa
+ PASSWD=REDACTED
+ DIST_NAME=CN=kaupocSuFoZIOIDa,OU=Cloudera,OU=ServersUnix,OU=IT,OU=Basel,OU=AdminUnits,DC=emea,DC=xxxx,DC=com
+ '&#91;' -z /etc/krb5-cdh.conf '&#93;'
+ echo 'Using custom config path '\''/etc/krb5-cdh.conf'\'', contents below:'
+ cat /etc/krb5-cdh.conf
+ SIMPLE_PWD_STR=
+ '&#91;' '' = '' '&#93;'
+ kinit -k -t /var/run/cloudera-scm-server/cmf5575611164358256388.keytab
cdhad@REALM.COM
++ mktemp /tmp/cm_ldap.XXXXXXXX
+ LDAP_CONF=/tmp/cm_ldap.XRbR8Zco
+ echo 'TLS_REQCERT never'
+ echo 'sasl_secprops minssf=0,maxssf=0'
+ export LDAPCONF=/tmp/cm_ldap.XRbR8Zco
+ LDAPCONF=/tmp/cm_ldap.XRbR8Zco
++ ldapsearch -LLL -H ldaps://host:636 -b
OU=Cloudera,OU=ServersUnix,OU=IT,OU=Basel,OU=AdminUnits,DC=emea,DC=xxxx,DC=com
userPrincipalName=sqoop2/<host>@REALM.COM
SASL/GSSAPI authentication started
SASL username: cdhad@REALM.COM
SASL SSF: 0
+ PRINC_SEARCH=
+ set +e
+ echo
+ grep -q userPrincipalName
+ '[' 1 -eq 0 ']'
+ set -e
+ ldapmodify -H ldaps://<ldap-host>:636
++ echo sqoop2/<ldap-host>@REALM.COM
++ sed -e 's/\@REALM.COM//g'
++ echo -n '"REDACTED"'
++ iconv -f UTF8 -t UTF16LE
++ base64 -w 0
SASL/GSSAPI authentication started
SASL username: cdhad@REALM.COMSASL SSF: 0
ldap_add: Server is unwilling to perform (53)
additional info: 0000052D: SvcErr: DSID-031A1248, problem 5003
(WILL_NOT_PERFORM), data 0

If you see the similar error and you know that you have AD enabled for your cluster, then you have landed on the right place. This is likely caused by a bug in Cloudera Manager that it does not allow users to change the complexity of the password generated if AD server has password complexity restrictions setup, and Cloudera Manager’s request will be rejected.

To fix this issue is simple, but requires changing some source code in Cloudera Manager, follow the steps below:

  1. Backup file /usr/share/cmf/bin/gen_credentials_ad.sh first on CM host
  2. Add this line to /usr/share/cmf/bin/gen_credentials_ad.sh on line number 15:
    PASSWD="$PASSWD-"
    

    after line:

    PASSWD=$4
    

    Basically this adds a hyphen to CM generated passwords.

  3. Run Generate Credentials again to see if this helps

If same error still happens, go back to step 2 and try different variations for the password:

PASSWD="ABC=$PASSWD" # prepends "ABC=" to generated password.

The idea is to meet the criteria of AD password requirement.

This issue is likely fixed already in Cloudera Manager’s source code to support more flexibility when generating passwords, but it won’t be release until CM5.8 at least.

“Update Hive Metastore NameNodes” Through Cloudera Manager Timed-Out

After enabling HA for HDFS, Cloudera Manager has a feature to allow you to perform an update to existing HMS database, so that all the references to the old HDFS URL is updated to use the new nameservice name:

update-hivemetastore-namenode

You can only do it after Hive service is stopped.

However, there is a hard limit of 150 seconds for this command to run in Cloudera Manager, once it is reached, an error will be returned.

"Command aborted because of exception: Command timed-out after 150 seconds"

There is already a JIRA to get this limit removed, however, it won’t happen until Cloudera Manager 5.5.

To solve this problem, we can manually run the Hive metatool from the command line, follow the steps below:

1) Test the metatool to verify that it is working from command line on the Hive gateway host:

hive --service metatool -listFSRoot

2) Run Hive Metatool to update the nameservice:

hive --service metatool -updateLocation hdfs://nameservice1 hdfs://oldnamenode.com -tablePropKey avro.schema.url

If you getting errors like SQL driver not found or not able to connect to HMS database, please add the following steps and try above again on the HMS host:

3) Locate the HMS run time configuration directory on the HMS host, /var/run/cloudera-scm-agent/process/-hive-HIVEMETASTORE, in my below example would be “521”

export HIVE_CONF_DIR=/var/run/cloudera-scm-agent/process/521-hive-HIVEMETASTORE

4) Determine which DB type you are using, MySQL, PostgeSQL or Orcale, in my below example I used PostgeSQL

export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH/jars/postgresql-9.1-901.jdbc4.jar

Then just wait for the command to finish and then restart Hive Services. So long as there is no other errors, you are good to go.