SPNEGO-enabled Hadoop DataNode misjudges kerberos "replay attack".
references https://docs.cloudera.com/cloudera-manager/7.5.5/security-troubleshooting/cm-security-troubleshooting.pdf https://search-guard.com/elasticsearch-kibana-kerberos/ I guess that this is caused by sharing the same kerberos keytab (/etc/security/keytabs/spnego.service.keytab) and principal(HTTP/_HOST@{REALM}) among Hadoop daemons (NameNode, DataNode, JournalNodes, ResourceManager, NodeManager …). I assume that DataNode misjudges it is a replay attack in certain circumstances. Adding the following jvm system properties to Hadoop daemons will fix this issue as a workaround. It means java process will not use replay cache. -Dsun.security.krb5.rcache=none