When copying data across clusters using DistCp action through Oozie in a secured environment, due to a current Oozie bug, the job will fail with below error message:

java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; 
Host Details : local host is: "xxxxxxxxxx/xxx.xxx.xxx.xxx"; destination host is: "xxxxxxxxxx":8020;
...
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
 at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
 at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
 at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
 at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
 at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
 at org.apache.hadoop.ipc.Client.call(Client.java:1391)
 ... 43 more

This is a known issue and reported in the upstream JIRA of OOZIE-2900

This JIRA has been fixed in CDH6 and above. However, if you can not upgrade for the time being, you can add below configuration into Oozie workflow XML file to avoid the issue:

<configuration>
    <property>
        <name>oozie.launcher.mapreduce.job.hdfs-servers</name>
        <value>hdfs://nameservice-cluster1,hdfs://nameservice-cluster2</value>
    </property>
</configuration>

Then re-run the Oozie job again to confirm the fix.

Hope this can help.

Leave a Reply

Your email address will not be published. Required fields are marked *