When copying data across clusters using DistCp action through Oozie in a secured environment, due to a current Oozie bug, the job will fail with below error message:

java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; 
Host Details : local host is: "xxxxxxxxxx/xxx.xxx.xxx.xxx"; destination host is: "xxxxxxxxxx":8020;
...
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
 at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
 at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
 at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
 at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
 at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
 at org.apache.hadoop.ipc.Client.call(Client.java:1391)
 ... 43 more

This is a known issue and reported in the upstream JIRA of OOZIE-2900

This JIRA has been fixed in CDH6 and above. However, if you can not upgrade for the time being, you can add below configuration into Oozie workflow XML file to avoid the issue:

<configuration>
    <property>
        <name>oozie.launcher.mapreduce.job.hdfs-servers</name>
        <value>hdfs://nameservice-cluster1,hdfs://nameservice-cluster2</value>
    </property>
</configuration>

Then re-run the Oozie job again to confirm the fix.

Hope this can help.

2 Comments

    1. Eric Lin

      Hi Viktor,

      Thanks for posting comment on my blog. Can you please clarify on what issue on the Oozie Server side that you want to fix?

      As mentioned OOZIE-2900 has been fixed since CDH6.0 and above.

      Regards
      Eric

Leave a Reply

Your email address will not be published. Required fields are marked *