Exception in Yarn node manager with null JWT (not materialized) on multi-container job


I am trying to run a mapreduce job on Hopsworks installed on 5 nodes according to the following configuration: 2 resource manager and 3 node manager. When the application is submitted to yarn, this generates 2 containers which are run on 2 separate node manager (eg NM1 and NM3). On the first node manager I can see the jwt token materialized correctly while on the other node manager no, and the job is in exception as reported below:

Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): JWT is enabled but it either null or empty for application application_1587625393033_0004
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.materializeJWT(ContainerManagerImpl.java:954)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.materializeSecurityMaterial(ContainerManagerImpl.java:911)

On the first node manager, this exception does not occur because the jwt token is correctly materialized.

Is it a bug? Or I am doing some configuration wrong?

The jwt token is enabled because the following configuration is a true one: yarn.resourcemanager.rmappsecurity.jwt.enabled

Thanks so much,




Can you please make sure that your application’s fat-jar does NOT contain any hadoop jars? Your application should use the provided by our system libraries.


Hi Antonios,
thank you very much for your reply.
The exception is not obtained running a fat-jar but running a map reduce job (e.g. select count (*) from xxx) directly inside the hive cli ({hive-home}/bin/hive).
Am I wrong to use the hive cli?

Thanks so much,

Hi @Antony,

If you want to run Hive queries you can do that from a Jupyter notebook (Python Kernel) - You can see an example here: https://github.com/logicalclocks/hops-examples/blob/master/notebooks/hive/PyHive.ipynb

In Hopsworks we use Apache Tez as execution engine for Hive, not MapReduce.


Hi Fabio,
thank you for your reply.
I will do like you suggested.

Thanks so much,

1 Like