Simple Tool to Enable SSL/TLS for CM/CDH Cluster

Since earlier this year, Cloudera has started a new program that allows each Support Engineer to do a full week offline self-learning. Topics can be chosen by each individual engineer so long as the outcome has a value to the business, It can be either the engineer skilled up with a certification that helps with day to day work, or a presentation to share with the rest of the team what he/she had learnt from the week doing self-learning. Last week, from 27th of August to 31st of August was my turn.

After a careful consideration, I thought that my knowledge on SSL/TLS area needed to be skilled up, so I had decided to find some SSL/TLS related courses on either SafariOnline or Lynda, and then see if I could try to enable Cloudera Manager as well as most of the CDH services with SSL/TLS, ideally to put everything into a script so that this process can be automated. I discussed this with my manager and we agreed on my plan.

On the first two days, I found a couple of very useful video courses from Lynda.com, see below link:

SSL Certificates For Web Developers
Learning Secure Sockets Layer

They were very useful in helping me getting a better understanding of the fundamental of SSL/TLS and how to generate keys and sign the cerficate all by yourself.

After that I reviewed Cloudera’s official online documentation on how to enable SSL/TLS for Cloudera Manager as well as the rest of CDH services and built a little tool that is written in shell script to allow anyone to generate certificates on the fly and enable SSL/TLS for his/her cluster with a simple couple of commands.

The documentation links can be found below:

Configuring TLS Encryption for Cloudera Manager
Configuring TLS/SSL Encryption for CDH Services

I have published this little tool on github and is available here. Currently it supports enabling SSL/TLS for the following services:

Cloudera Manager (from Level 1 to Level 3 security)
HDFS
YARN
Hive
Impala
Oozie
HBase
Hue

With this tool, user can enable SSL/TLS for any of the above services with ease in a few minutes.

If you have any suggestions or comments, please leave them in the comment section below, thanks.

HiveMetaStore Failed to Start in Cloudera Manager: cmf.service.config.ConfigGenException: Unable to generate config file creds.localjceks

Recently I was dealing with an issues that HiveMetaStore failed to start in a Cloudera Manager managed environment. It failed with below errors:

Caused by: com.cloudera.cmf.service.config.ConfigGenException: Unable to generate config file creds.localjceks
        at com.cloudera.cmf.service.config.JceksConfigFileGenerator.generate(JceksConfigFileGenerator.java:63)
        at com.cloudera.cmf.service.HandlerUtil.emitConfigFiles(HandlerUtil.java:133)
        at com.cloudera.cmf.service.AbstractRoleHandler.generateConfiguration(AbstractRoleHandler.java:887)

This problem is very common if you have the following misconfiguration in your cluster:

1. Wrong version of Java being used. For a list of supported version of Java by Cloudera, please refer to below link:
CDH and Cloudera Manager Supported JDK Versions

2. Different version of Java used across the cluster hosts.

So run:

java -version

and check symlinks under /usr/java/jdk****-cloudera to confirm they are consistent across the whole cluster.

After all above were performed, try to restart failed service, most likely the issue should be resolved. If not, please let me know in the comments below.

Oozie Server failed to Start with error java.lang.NoSuchFieldError: EXTERNAL_PROPERTY

This issue happens in CDH distribution of Hadoop that is managed by Cloudera Manager (possibly in other distributions as well, due to known upstream JIRA, but I have not tested). Oozie will fail to start after enabling Oozie HA through Cloudera Manager user interface.

The full error message from Oozie’s process stdout.log (can be found under /var/run/cloudera-scm-agent/process/XXX-oozie-OOZIE_SERVER/logs directory) file looks like below:

 

Wed Jan 25 11:07:41 GST 2017 
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera 
using 5 as CDH_VERSION 
using /var/lib/oozie/tomcat-deployment as CATALINA_BASE 
Copying JDBC jar from /usr/share/java/oracle-connector-java.jar to /var/lib/oozie 

ERROR: Oozie could not be started 

REASON: java.lang.NoSuchFieldError: EXTERNAL_PROPERTY 

Stacktrace: 
----------------------------------------------------------------- 
java.lang.NoSuchFieldError: EXTERNAL_PROPERTY 
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector._findTypeResolver(JacksonAnnotationIntrospector.java:777) 
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findPropertyTypeResolver(JacksonAnnotationIntrospector.java:214) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findPropertyTypeSerializer(BeanSerializerFactory.java:370) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory._constructWriter(BeanSerializerFactory.java:772) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(BeanSerializerFactory.java:586) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.constructBeanSerializer(BeanSerializerFactory.java:430) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanSerializer(BeanSerializerFactory.java:343) 
at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:287) 
at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782) 
at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735) 
at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344) 
at org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420) 
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601) 
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) 
at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2566) 
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:2056) 
at org.apache.oozie.util.FixedJsonInstanceSerializer.serialize(FixedJsonInstanceSerializer.java:65) 
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:201) 
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:186) 
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217) 
at org.apache.oozie.util.ZKUtils.<init>(ZKUtils.java:141) 
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154) 
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70) 
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) 
at org.apache.oozie.service.Services.setService(Services.java:372) 
at org.apache.oozie.service.Services.loadServices(Services.java:305) 
at org.apache.oozie.service.Services.init(Services.java:213) 
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) 
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210) 
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709) 
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802) 
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) 
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) 
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) 
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) 
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) 
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) 
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) 
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) 
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068) 
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822) 
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060) 
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) 
at org.apache.catalina.core.StandardService.start(StandardService.java:525) 
at org.apache.catalina.core.StandardServer.start(StandardServer.java:759) 
at org.apache.catalina.startup.Catalina.start(Catalina.java:595) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) 
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) 

To fix the issue, please follow the steps below:

  1. Delete or move the following files under CDH’s parcel directory (most likely they are symlinks):

    /opt/cloudera/parcels/CDH/lib/oozie/libserver/hive-exec.jar
    /opt/cloudera/parcels/CDH/lib/oozie/libtools/hive-exec.jar
    

  2. Download hive-exec-{cdh version}-core.jar file from the Cloudera repo, for example, for CDH5.8.2, please go to:
    https://repository.cloudera.com/cloudera/cloudera-repos/org/apache/hive/hive-exec/1.1.0-cdh5.8.2/

    and put the file under the following directories on the Oozie server:

    /opt/cloudera/parcels/CDH/lib/oozie/libserver/
    /opt/cloudera/parcels/CDH/lib/oozie/libtools/
    

  3. Download kryo-2.22.jar from the maven repository:
    http://repo1.maven.org/maven2/com/esotericsoftware/kryo/kryo/2.22/kryo-2.22.jar

    and put it under directories on the Oozie server:

    /opt/cloudera/parcels/CDH/lib/oozie/libserver/
    /opt/cloudera/parcels/CDH/lib/oozie/libtools/
    

  4. Finally restart Oozie service

This is a known Oozie issue and reported in the upstream JIRA: OOZIE-2621, which has been resolved and targeted for 4.3.0 release.

Hope this helps.

How to setup multiple KDCs through Cloudera Manager

Currently Cloudera Manager does not support setting up multiple KDCs for the krb5.conf file natively, this article explains the workarounds we can have using the existing feature provided by Cloudera Manager.

This article also assumes that you have krb5.conf file managed by Cloudera Manager.

If you are using Cloudera Manager prior to 5.7, following the steps below:

  1. Go to CM > Administration > Settings > click on “Kerberos” on Filters on the left side > locate “KDC Server Host”, enter the KDC host in the text field:
    kdc-host1.com
    
  2. On the same page, locate “Advanced Configuration Snippet (Safety Valve) for the Default Realm in krb5.conf”, and enter the following into the text area:
    kdc = kdc-host2.com
    
  3. Save and then “Deploy Kerberos Client Configuration” (you might need to stop all service first before you can do this)
    The [realm] section in the krb5.conf will be updated like below:
    [realms]
    TEST.COM = {
    kdc = kdc-host1.com
    admin_server = kdc-host1.com
    kdc = kdc-host2.com
    }
    

If you are using CM5.7 and above, you can also do the following (above steps should still work):

  1. Go to CM > Administration > Settings > click on “Kerberos” on Filters on the left side > locate “KDC Server Host”, empty the KDC host in the text field, so that it contains no value
  2. On the same page, locate “Advanced Configuration Snippet (Safety Valve) for the Default Realm in krb5.conf”, and enter the following into the text area:
    kdc = kdc-host1.com
    kdc = kdc-host2.com
    admin_server = kdc-host1.com
    
  3. Save and then “Deploy Kerberos Client Configuration” (you might need to stop all service first before you can do this)
    The [realm] section in the krb5.conf will be updated like below:
    [realms]
    TEST.COM = {
    kdc = kdc-host1.com
    kdc = kdc-host2.com
    admin_server = kdc-host1.com
    }
    

The second option does not work prior to CM5.7 is because the older version of CM will generate the following line in krb5.conf if the KDC Server Host is empty:

kdc =

which will break the syntax in krb5.conf file.

How to use Cloudera Manager API to check a service role exited unexpectedly

This blog explains the steps to use Cloudera Manager API to check for a service role in CDH that was exited unexpectedly, so that proper action can be taken.

To check a service role’s status via Cloudera Manager API, please follow the steps below (I am taking Impala as an example):

  1. Determine the version of API you are using:

    curl -u username:password http://<cm-host>:7180/api/version
    

    in CDH5.7.x, it should return “v12”.

  2. get the cluster name from the output of:
    curl -u username:password http://<cm-host>:7180/api/v12/clusters
    

    sample output:

    {
      "items" : [ {
        "name" : "cluster",
        "displayName" : "Cluster 1",
        "version" : "CDH5",
        "fullVersion" : "5.7.0",
        "maintenanceMode" : false,
        "maintenanceOwners" : [ ],
        "clusterUrl" : "http://<cm-host>:7180/cmf/clusterRedirect/cluster",
        "hostsUrl" : "http://<cm-host>:7180/cmf/clusterRedirect/cluster/hosts",
        "entityStatus" : "GOOD_HEALTH"
      } ]
    }
    

    take note of the value for “name” attribute, in my case it is “cluster”.
  3. get the services in the cluster:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/<cluster-name>/services
    

    substitute with our cluster’s name:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/cluster/services
    

    please locate the impala service and get its “name”, in my case it is “impala”:

    {
        "name" : "impala",
        "type" : "IMPALA",
        "clusterRef" : {
          "clusterName" : "cluster"
        },
        ....
    }
    
  4. get the all the roles under impala service:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/<cluster-name>/services/<impala-name>/roles/
    

    in my case should be:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/cluster/services/impala/roles
    

    locate the role that you want to monitor, I picked Statestore:

    {
        "name" : "impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe",
        "type" : "STATESTORE",
        "serviceRef" : {
          "clusterName" : "cluster",
          "serviceName" : "impala"
        },
        ....
    }
    

    in my case it is “impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe”

  5. get the status for the role you want to monitor:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/<cluster-name>/services/<impala-name>/roles/<role-name>
    

    after substitution, it should be:

    curl -u username:password http://<cm-host>:7180/api/v12/clusters/cluster/services/impala/roles/impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe
    

    this will give you full status output for this particular role:

    {
      "name" : "impala-STATESTORE-52cc0fbf54f5cc038b2b0a67634034fe",
      "type" : "STATESTORE",
      "serviceRef" : {
        "clusterName" : "cluster",
        "serviceName" : "impala"
      },
      "hostRef" : {
        "hostId" : "eff96b49-739e-48d4-a19b-e5865a83b164"
      },
      .....
      "configStalenessStatus" : "FRESH",
      "maintenanceMode" : false,
      "maintenanceOwners" : [ ],
      "commissionState" : "COMMISSIONED",
      "roleConfigGroupRef" : {
        "roleConfigGroupName" : "impala-STATESTORE-BASE"
      },
      "entityStatus" : "GOOD_HEALTH"
    }
    

    look for the last attribute called “entityStatus”, it has the following possible values:

    UNKNOWN	
    NONE	
    STOPPED	
    DOWN	
    UNKNOWN_HEALTH	
    DISABLED_HEALTH	
    CONCERNING_HEALTH	
    BAD_HEALTH	
    GOOD_HEALTH	
    STARTING	
    STOPPING	
    HISTORY_NOT_AVAILABLE
    

    in the case that it is exited unexpectedly, the value would be “DOWN”, so that we can programmatically decide whether we can just restart it or not.

Please note that if you have enabled SSL for Cloudera Manager, the URL should be changed to: https://:7183 instead of http://:7180

More information about the apiRole entity can be found here: apiRole