Exception in Yarn node manager with null JWT (not materialized) on multi-container job

Hopsworks
#1

Hi,

I am trying to run a mapreduce job on Hopsworks installed on 5 nodes according to the following configuration: 2 resource manager and 3 node manager. When the application is submitted to yarn, this generates 2 containers which are run on 2 separate node manager (eg NM1 and NM3). On the first node manager I can see the jwt token materialized correctly while on the other node manager no, and the job is in exception as reported below:

Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): JWT is enabled but it either null or empty for application application_1587625393033_0004
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.materializeJWT(ContainerManagerImpl.java:954)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.materializeSecurityMaterial(ContainerManagerImpl.java:911)

On the first node manager, this exception does not occur because the jwt token is correctly materialized.

Is it a bug? Or I am doing some configuration wrong?

The jwt token is enabled because the following configuration is a true one: yarn.resourcemanager.rmappsecurity.jwt.enabled

Thanks so much,

regards

Antony

#2

Hi,

Can you please make sure that your application’s fat-jar does NOT contain any hadoop jars? Your application should use the provided by our system libraries.

Thanks

#3

Hi Antonios,
thank you very much for your reply.
The exception is not obtained running a fat-jar but running a map reduce job (e.g. select count (*) from xxx) directly inside the hive cli ({hive-home}/bin/hive).
Am I wrong to use the hive cli?

Thanks so much,
Antony

#4

Hi @Antony,

If you want to run Hive queries you can do that from a Jupyter notebook (Python Kernel) - You can see an example here: https://github.com/logicalclocks/hops-examples/blob/master/notebooks/hive/PyHive.ipynb

In Hopsworks we use Apache Tez as execution engine for Hive, not MapReduce.

–
Fabio

#5

Hi Fabio,
thank you for your reply.
I will do like you suggested.

Thanks so much,
Antony

1 Like