Errors installing Hopsworks 3.0 on Premises

Cliff_Patterson · September 8, 2022, 3:22am

Failed to install Hopsworks on an on-premises VM running Ubuntu 18.04.6 with 32GB ram, 8 vCPUs, and a 100GB SSD. The user was a passwordless sudo user, as required. We used the official installation instructions here: Hopsworks Installer - Hopsworks Documentation

The installation fails at the hops::docker_registry and there are no logs (that we could find) either through the CLI or in Karamel to explain what went wrong.

Task | Status | Actions | Duration(ms)|

| find os-type | DONE | log| 131 |
| install chefdk | DONE | log| 14664 |
| make solo.rb | DONE | log| 50 |
| clone and vendor hopsworks-chef| DONE | log| 31792 |
| ndb::install | DONE | log| 82343 |
| ndb::mgmd | DONE | log| 5604 |
| ndb::ndbd | DONE | log| 28621 |
| ndb::mysqld | DONE | log| 23751 |
| hops::ndb | DONE | log| 99512 |
| hopsworks::default | DONE | log| 683683 |
| kkafka::default | READY | | |
| onlinefs::default | WAITING| | |
| kagent::default | DONE | log| 11285 |
| kzookeeper::default | DONE | log| 13824 |
| hops::rm | WAITING| | |
| hops::nm | WAITING| | |
| livy::default | WAITING| | |
| hopsmonitor::prometheus | DONE | log| 14151 |
| hops::dn | WAITING| | |
| tensorflow::default | WAITING| | |
| hops::fuse_mnt | WAITING| | |
| hadoop_spark::yarn | WAITING| | |
| hadoop_spark::historyserver | WAITING| | |
| hive2::default | WAITING| | |
| epipe::default | WAITING| | |
| hopslog::default | WAITING| | |
| hopslog::_filebeat-jupyter | WAITING| | |
| hopslog::_filebeat-services | WAITING| | |
| hopslog::_filebeat-spark | WAITING| | |
| hopslog::_filebeat-serving | WAITING| | |
| flink::yarn | WAITING| | |
| flink::historyserver | WAITING| | |
| hops::jhs | WAITING| | |
| consul::master | DONE | log| 13079 |
| hops::docker_registry | FAILED | retry skip log| 20503 |
| hops_airflow::default | WAITING| | |
| hops::nn | READY | | |
| elastic::default | DONE | log| 54281 |
| hopsmonitor::default | DONE | log| 17955 |
| kzookeeper::install | DONE | log| 4814 |
| hopsworks::install | DONE | log| 110792 |
| elastic::install | DONE | log| 5470 |
| tensorflow::install | DONE | log| 47123 |
| epipe::install | DONE | log| 9133 |
| kagent::install | DONE | log| 11372 |
| hadoop_spark::install | DONE | log| 65692 |
| onlinefs::install | DONE | log| 5500 |
| flink::install | DONE | log| 27968 |
| conda::install | DONE | log| 19659 |
| conda::default | DONE | log| 32356 |
| kkafka::install | DONE | log| 12025 |
| hopsmonitor::install | DONE | log| 6252 |
| hopsmonitor::alertmanager | DONE | log| 10364 |
| hopsmonitor::node_exporter | DONE | log| 8260 |
| livy::install | DONE | log| 9040 |
| hops_airflow::install | DONE | log| 5456 |
| consul::install | DONE | log| 9084 |
| hopslog::install | DONE | log| 47728 |
| hops::install | DONE | log| 52299 |
| hive2::install | DONE | log| 32464 |
| apt-get essentials | DONE | log| 8427 |

Consequently, the installation appears to be incomplete, no services run, and when trying to start a new project in Hopsworks, we get the following error: “An argument was not proviced or it was malformed java.net.UnknownHostException rpc.namenode.service.consul”

Are there updated installation instructions available or any pre-installation steps required?

Thanks!

Robin_Andersson · September 9, 2022, 11:50am

Hello @Cliff_Patterson !

You should be able to find the log file for the failing recipe at:

cat /home/${USER}/.karamel/install/hops__docker_registry.log

Where ${USER} is the user used for installation.

Do you find any errors at the end of the file?

Best,
Robin

Cliff_Patterson · September 12, 2022, 2:19pm

It is the exact same error that this user received. He solved it by downloading the image manually, but where does it go? Or is there a better solution?

Robin_Andersson · September 12, 2022, 3:40pm

I suspect what may be happening is the disk is running out of space during installation.

Can you verify that /srv/hops, /srv/hopsworks-data and /tmp/chef-solo does not get filled up while running the failing recipe?

If that’s the case, I recommend you create the following symlinks to the disk you want to place the installation on: Hopswork Docker_registry Error - #2 by Robin_Andersson

Cliff_Patterson · September 13, 2022, 7:31pm

Hi Robin,

Symlinks are not required in our configuration as all the directories mentioned are on the main data partition. However, if space could be a factor, I spun up a new VM with 8 vCPUs, 200GB SSD, 64GB of ram, and Ubuntu 18.04. I attempted the re-installation and it progressed further than the first time. This time it failed at hops::nn. After installation, the main partition was only 11% used, so space does not appear to be a factor.

 1. | find os-type                   | DONE   |   log| 127         |
 2. | install chefdk                 | DONE   |   log| 14067       |
 3. | make solo.rb                   | DONE   |   log| 51          |
 4. | clone and vendor hopsworks-chef| DONE   |   log| 17309       |
 5. | kzookeeper::install            | DONE   |   log| 6094        |
 6. | kzookeeper::default            | DONE   |   log| 14436       |
 7. | hops::rm                       | WAITING|                 |             |
 8. | hops::nm                       | WAITING|                 |             |
 9. | livy::default                  | WAITING|                 |             |
10. | kkafka::default                | READY  |                 |             |
11. | onlinefs::default              | WAITING|                 |             |
12. | conda::install                 | DONE   |   log| 20164       |
13. | conda::default                 | DONE   |   log| 34928       |
14. | hopsworks::default             | DONE   |   log| 706365      |
15. | kagent::default                | DONE   |   log| 11539       |
16. | hopsmonitor::prometheus        | DONE   |   log| 14357       |
17. | consul::master                 | DONE   |   log| 13552       |
18. | hopslog::default               | WAITING|                 |             |
19. | hopslog::_filebeat-jupyter     | WAITING|                 |             |
20. | hopslog::_filebeat-spark       | WAITING|                 |             |
21. | hopslog::_filebeat-services    | WAITING|                 |             |
22. | hopslog::_filebeat-serving     | WAITING|                 |             |
23. | hops::nn                       | FAILED | retry  skip  log| 1224744     |
24. | hops::fuse_mnt                 | WAITING|                 |             |
25. | hops::dn                       | WAITING|                 |             |
26. | flink::yarn                    | WAITING|                 |             |
27. | flink::historyserver           | WAITING|                 |             |
28. | hadoop_spark::yarn             | WAITING|                 |             |
29. | hadoop_spark::historyserver    | WAITING|                 |             |
30. | hive2::default                 | WAITING|                 |             |
31. | epipe::default                 | WAITING|                 |             |
32. | tensorflow::default            | WAITING|                 |             |
33. | hops::jhs                      | WAITING|                 |             |
34. | hops::docker_registry          | READY  |                 |             |
35. | hops_airflow::default          | WAITING|                 |             |
36. | hopsmonitor::default           | DONE   |   log| 17820       |
37. | elastic::default               | DONE   |   log| 54888       |
38. | ndb::install                   | DONE   |   log| 82160       |
39. | ndb::mgmd                      | DONE   |   log| 5964        |
40. | ndb::mysqld                    | DONE   |   log| 38578       |
41. | hops::ndb                      | DONE   |   log| 108748      |
42. | ndb::ndbd                      | DONE   |   log| 28505       |
43. | consul::install                | DONE   |   log| 11316       |
44. | elastic::install               | DONE   |   log| 5711        |
45. | hadoop_spark::install          | DONE   |   log| 88615       |
46. | kagent::install                | DONE   |   log| 26133       |
47. | hops_airflow::install          | DONE   |   log| 6080        |
48. | flink::install                 | DONE   |   log| 28564       |
49. | hops::install                  | DONE   |   log| 53779       |
50. | hopsworks::install             | DONE   |   log| 112724      |
51. | tensorflow::install            | DONE   |   log| 15768       |
52. | livy::install                  | DONE   |   log| 9788        |
53. | epipe::install                 | DONE   |   log| 9401        |
54. | kkafka::install                | DONE   |   log| 11779       |
55. | hive2::install                 | DONE   |   log| 32808       |
56. | onlinefs::install              | DONE   |   log| 5784        |
57. | hopslog::install               | DONE   |   log| 48709       |
58. | hopsmonitor::install           | DONE   |   log| 6728        |
59. | hopsmonitor::node_exporter     | DONE   |   log| 8736        |
60. | hopsmonitor::alertmanager      | DONE   |   log| 9412        |
61. | apt-get essentials             | DONE   |   log| 9064        |

I checked the hops__nn.log and it had a repeated error:

"https://hopsworks.glassfish.service.consul:8182/hopsworks-api/api/auth/service\": dial tcp: lookup hopsworks.glassfish.service.consul: no such host"
e[0m    time="2022-09-13T17:36:24Z" level=error msg="Failed to make HTTP request to login to https://hopsworks.glassfish.service.consul:8182/hopsworks-api/api/auth/service, retrying... Reason: Post \"https://hopsworks.glassfish.service.consul:8182/hopsworks-api/api/auth/service\": dial tcp: lookup hopsworks.glassfish.service.consul: no such host"
e[0m    time="2022-09-13T17:36:25Z" level=error msg="HTTP operation to login timed out!"
time="2022-09-13T18:04:24Z" level=error msg="HTTP operation to login timed out!"
time="2022-09-13T18:04:24Z" level=error msg="Could not create Hopsworks client: context deadline exceeded"
time="2022-09-13T18:04:24Z" level=fatal msg="Fatal error: context deadline exceeded"
STDERR: Error: context deadline exceeded
---- End output of "bash"  "/tmp/chef-script20220913-16329-yagj1k" ----
Ran "bash"  "/tmp/chef-script20220913-16329-yagj1k" returned 1

This error appears 100s of times then the process times out. Any idea why it now fails at this point?

antonios · September 14, 2022, 7:39am

Hi @Cliff_Patterson

It looks our internal service discovery system is having some trouble in your installation.

To begin with make sure Glassfish is up and running sudo systemctl restart glassfish-domain1, you can also check the logs at /srv/hops/domains/domain1/logs/server.log

If Glassfish is running, you can try dig hopsworks.glassfish.service.consul Normally the answer section should contain an IP address. If it doesn’t, it means service discovery is still not working.

Next step is to check the status and logs of the following two services:

dnsmasq
consul

Verify they are both running with sudo systemctl status SERVICE_NAME. You can see the latest logs with sudo journalctl -f -u SERVICE_NAME and finally restart them with sudo systemctl restart SERVICE_NAME

Let us know if you make any progress.

Kind regards,
Antonios

sam.lunageo · September 14, 2022, 2:55pm

Hi all, I am also working in this issue alongside with @Cliff_Patterson

@antonios
this is the result from dig hopsworks.glassfish.service.consul

; <<>> DiG 9.11.3-1ubuntu1.17-Ubuntu <<>> hopsworks.glassfish.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 30799
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;hopsworks.glassfish.service.consul. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Sep 14 14:49:58 UTC 2022
;; MSG SIZE  rcvd: 63

status of dnsmasq:

dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
   Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2022-09-13 17:14:35 UTC; 21h ago
 Main PID: 21739 (dnsmasq)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/dnsmasq.service
           └─21739 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service --trust-anchor=.,19036,8,2,49aac11d7b6f6446702e54a1607371607a1a41855

status of consul:

consul.service - "HashiCorp Consul - A service mesh solution"
   Loaded: loaded (/lib/systemd/system/consul.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2022-09-13 17:14:36 UTC; 21h ago
     Docs: https://www.consul.io/
  Process: 25930 ExecReload=/bin/bash -c CONSUL_TLS_SERVER_NAME=$(hostname -f | tr -d '[:space:]') /srv/hops/consul/bin/consul reload (code=exited, status=0/SUCCESS)
 Main PID: 21889 (consul)
    Tasks: 21 (limit: 4915)
   CGroup: /system.slice/consul.service
           └─21889 /srv/hops/consul/bin/consul agent -client {{ GetAllInterfaces | include "flags" "loopback" | include "type" "ipv4" | join "address" " " }} -bind 138.197.169.244 -config-dir=/srv/hops/consul/

I’ve checked the server.log file as sugested by @antonios here are the resutls:

the first error is this one

java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.invokeStatsProviderResetMethod(StatsProviderManagerDelegateImpl.java:618)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.resetStatistics(StatsProviderManagerDelegateImpl.java:575)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.enableStatsProvider(StatsProviderManagerDelegateImpl.java:422)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.tryToRegister(StatsProviderManagerDelegateImpl.java:198)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.register(StatsProviderManagerDelegateImpl.java:158)
        at org.glassfish.external.probe.provider.StatsProviderManager.registerStatsProvider(StatsProviderManager.java:91)
        at org.glassfish.external.probe.provider.StatsProviderManager.register(StatsProviderManager.java:66)
        at org.glassfish.external.probe.provider.StatsProviderManager.register(StatsProviderManager.java:57)
        at org.glassfish.web.admin.monitor.HttpServiceStatsProviderBootstrap.postConstruct(HttpServiceStatsProviderBootstrap.java:97)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:327)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710)
        at com.sun.enterprise.web.WebContainer.createStatsProviders(WebContainer.java:3458)
        at com.sun.enterprise.web.WebContainer.postConstruct(WebContainer.java:432)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:327)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710)
        at com.sun.enterprise.web.WebContainer.createStatsProviders(WebContainer.java:3458)
        at com.sun.enterprise.web.WebContainer.postConstruct(WebContainer.java:432)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:327)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
        at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:86)
        at org.glassfish.kernel.javaee.WebContainerStarter.startWebContainer(WebContainerStarter.java:241)
        at org.glassfish.kernel.javaee.WebContainerStarter.postConstruct(WebContainerStarter.java:180)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:327)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
 at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
        at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:86)
        at org.glassfish.kernel.javaee.WebContainerStarter.startWebContainer(WebContainerStarter.java:241)
        at org.glassfish.kernel.javaee.WebContainerStarter.postConstruct(WebContainerStarter.java:180)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:327)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
        at org.glassfish.web.admin.monitor.HttpServiceStatsProvider.getInitialOpenConnections(HttpServiceStatsProvider.java:217)
        at org.glassfish.web.admin.monitor.HttpServiceStatsProvider.reset(HttpServiceStatsProvider.java:558)
        ... 58 more

If i keep reading the file I get the following multiple times

[#|2022-09-13T17:05:38.966+0000|SEVERE|Payara 4.1|javax.enterprise.system.tools.deployment.common|_ThreadID=92;_ThreadName=deployment-jar-scanner;_TimeMillis=1663088738966;_LevelValue=1000;|Exception while vi$java.lang.IllegalArgumentException
        at org.glassfish.hk2.external.org.objectweb.asm.ClassReader.<init>(ClassReader.java:169)
        at org.glassfish.hk2.external.org.objectweb.asm.ClassReader.<init>(ClassReader.java:152)
        at org.glassfish.hk2.external.org.objectweb.asm.ClassReader.<init>(ClassReader.java:423)
        at org.glassfish.hk2.classmodel.reflect.Parser$5.on(Parser.java:359)
        at com.sun.enterprise.v3.server.ReadableArchiveScannerAdapter.handleEntry(ReadableArchiveScannerAdapter.java:165)
        at com.sun.enterprise.v3.server.ReadableArchiveScannerAdapter.onSelectedEntries(ReadableArchiveScannerAdapter.java:127)
        at org.glassfish.hk2.classmodel.reflect.Parser.doJob(Parser.java:345)
        at org.glassfish.hk2.classmodel.reflect.Parser.access$300(Parser.java:68)
        at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:304)
        at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

Then

[#|2022-09-13T17:07:50.705+0000|WARNING|Payara 4.1|io.hops.hopsworks.common.jobs.flink.FlinkController|_ThreadID=83;_ThreadName=admin-thread-pool::admin-listener(2);_TimeMillis=1663088870705;_LevelValue=900;|$java.io.FileNotFoundException: Path not found : /user/flink/completed-jobs
        at io.hops.hopsworks.common.hdfs.inode.InodeController.getChildren(InodeController.java:183)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
        at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy555.getChildren(Unknown Source)
        at io.hops.hopsworks.common.hdfs.inode.__EJB31_Generated__InodeController__Intf____Bean__.getChildren(Unknown Source)
        at io.hops.hopsworks.common.jobs.flink.FlinkController.getProjectsOfFlinkJobs(FlinkController.java:228)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
        at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy482.getProjectsOfFlinkJobs(Unknown Source)
        at io.hops.hopsworks.common.jobs.flink.__EJB31_Generated__FlinkController__Intf____Bean__.getProjectsOfFlinkJobs(Unknown Source)
        at io.hops.hopsworks.common.jobs.flink.FlinkCompletedJobsCache.init(FlinkCompletedJobsCache.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1047)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
 at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:125)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:417)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:380)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:2030)
        at com.sun.ejb.containers.AbstractSingletonContainer.createSingletonEJB(AbstractSingletonContainer.java:544)
        at com.sun.ejb.containers.AbstractSingletonContainer.access$000(AbstractSingletonContainer.java:82)
        at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:723)
        at com.sun.ejb.containers.AbstractSingletonContainer.instantiateSingletonInstance(AbstractSingletonContainer.java:397)
        at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:219)
        at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:180)
at org.glassfish.ejb.startup.SingletonLifeCycleManager.doStartup(SingletonLifeCycleManager.java:158)
        at org.glassfish.ejb.startup.EjbApplication.start(EjbApplication.java:169)
        at org.glassfish.internal.data.EngineRef.start(EngineRef.java:122)
        at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:294)
        at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:357)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:501)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:220)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:508)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:544)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:540)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:539)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:570)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:562)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:561)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1469)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:111)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1851)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1727)
        at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:407)
at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandMultInMultOut(CommandResource.java:256)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:173)
        at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:179)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:483)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:180)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:231)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:231)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
 at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573)
        at java.lang.Thread.run(Thread.java:750)

sam.lunageo · September 14, 2022, 2:55pm

More errors from serve.log

[#|2022-09-13T17:07:51.999+0000|SEVERE|Payara 4.1|io.hops.hopsworks.common.security.BaseHadoopClientsService|_ThreadID=118;_ThreadName=__ejb-thread-pool3;_TimeMillis=1663088871999;_LevelValue=1000;|Could not $java.io.FileNotFoundException: File containing crypto material password could not be found
        at io.hops.security.HopsUtil.readCryptoMaterialPassword(HopsUtil.java:85)
        at io.hops.hopsworks.common.security.BaseHadoopClientsService.init(BaseHadoopClientsService.java:91)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1047)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:125)
        at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:417)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:380)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:2030)
 at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:518)
        at com.sun.ejb.containers.StatelessSessionContainer.access$000(StatelessSessionContainer.java:99)
        at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:713)
        at com.sun.ejb.containers.util.pool.NonBlockingPool.getObject(NonBlockingPool.java:220)
        at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:442)
        at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2596)
        at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1987)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:210)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy565.getSuperTrustStorePath(Unknown Source)
        at io.hops.hopsworks.common.security.__EJB31_Generated__BaseHadoopClientsService__Intf____Bean__.getSuperTrustStorePath(Unknown Source)
        at io.hops.hopsworks.common.opensearch.OpenSearchClient.getClient(OpenSearchClient.java:103)
        at io.hops.hopsworks.common.opensearch.OpenSearchClient.init(OpenSearchClient.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1047)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:125)
        at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:417)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:380)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:2030)
        at com.sun.ejb.containers.AbstractSingletonContainer.createSingletonEJB(AbstractSingletonContainer.java:544)
        at com.sun.ejb.containers.AbstractSingletonContainer.access$000(AbstractSingletonContainer.java:82)
        at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:723)
        at com.sun.ejb.containers.AbstractSingletonContainer.instantiateSingletonInstance(AbstractSingletonContainer.java:397)
        at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:219)
        at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:180)
        at com.sun.ejb.containers.AbstractSingletonContainer.checkInit(AbstractSingletonContainer.java:369)
        at com.sun.ejb.containers.BMCSingletonContainer._getContext(BMCSingletonContainer.java:63)
 at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2596)
        at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1987)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:210)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy543.getClient(Unknown Source)
        at io.hops.hopsworks.common.opensearch.__EJB31_Generated__OpenSearchClient__Intf____Bean__.getClient(Unknown Source)
        at io.hops.hopsworks.common.opensearch.OpenSearchClientController.lambda$clusterHealthGet$15(OpenSearchClientController.java:479)
        at io.hops.hopsworks.common.opensearch.OpenSearchClientController.executeOpenSearchQuery(OpenSearchClientController.java:501)
        at io.hops.hopsworks.common.opensearch.OpenSearchClientController.clusterHealthGet(OpenSearchClientController.java:480)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
        at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy454.clusterHealthGet(Unknown Source)
        at io.hops.hopsworks.common.opensearch.__EJB31_Generated__OpenSearchClientController__Intf____Bean__.clusterHealthGet(Unknown Source)
        at io.hops.hopsworks.common.python.search.PyPiLibraryOpenSearchIndexer.execute(PyPiLibraryOpenSearchIndexer.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundTimeout(SystemInterceptorProxy.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
 at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)
        at com.sun.ejb.containers.BaseContainer.callEJBTimeout(BaseContainer.java:4074)
        at com.sun.ejb.containers.EJBTimerService.deliverTimeout(EJBTimerService.java:1228)
        at com.sun.ejb.containers.EJBTimerService.access$000(EJBTimerService.java:93)
        at com.sun.ejb.containers.EJBTimerService$TaskExpiredWork.run(EJBTimerService.java:1963)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

then we get the warnings

[#|2022-09-13T17:07:52.001+0000|WARNING|Payara 4.1|javax.enterprise.ejb.container|_ThreadID=118;_ThreadName=__ejb-thread-pool3;_TimeMillis=1663088872001;_LevelValue=900;|javax.ejb.EJBException: javax.ejb.EJBE$        at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:446)
        at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2596)
        at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1987)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:210)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy565.getSuperTrustStorePath(Unknown Source)
        at io.hops.hopsworks.common.security.__EJB31_Generated__BaseHadoopClientsService__Intf____Bean__.getSuperTrustStorePath(Unknown Source)
        at io.hops.hopsworks.common.opensearch.OpenSearchClient.getClient(OpenSearchClient.java:103)
        at io.hops.hopsworks.common.opensearch.OpenSearchClient.init(OpenSearchClient.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1047)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:125)
        at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:417)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:380)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:2030)
        at com.sun.ejb.containers.AbstractSingletonContainer.createSingletonEJB(AbstractSingletonContainer.java:544)
        at com.sun.ejb.containers.AbstractSingletonContainer.access$000(AbstractSingletonContainer.java:82)
        at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:723)
        at com.sun.ejb.containers.AbstractSingletonContainer.instantiateSingletonInstance(AbstractSingletonContainer.java:397)
        at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:219)
        at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:180)
        at com.sun.ejb.containers.AbstractSingletonContainer.checkInit(AbstractSingletonContainer.java:369)
        at com.sun.ejb.containers.BMCSingletonContainer._getContext(BMCSingletonContainer.java:63)
        at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2596)
        at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1987)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:210)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy543.getClient(Unknown Source)
        at io.hops.hopsworks.common.opensearch.__EJB31_Generated__OpenSearchClient__Intf____Bean__.getClient(Unknown Source)
        at io.hops.hopsworks.common.opensearch.OpenSearchClientController.lambda$clusterHealthGet$15(OpenSearchClientController.java:479)
        at io.hops.hopsworks.common.opensearch.OpenSearchClientController.executeOpenSearchQuery(OpenSearchClientController.java:501)
        at io.hops.hopsworks.common.opensearch.OpenSearchClientController.clusterHealthGet(OpenSearchClientController.java:480)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
        at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
 at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at com.sun.proxy.$Proxy454.clusterHealthGet(Unknown Source)
        at io.hops.hopsworks.common.opensearch.__EJB31_Generated__OpenSearchClientController__Intf____Bean__.clusterHealthGet(Unknown Source)
        at io.hops.hopsworks.common.python.search.PyPiLibraryOpenSearchIndexer.execute(PyPiLibraryOpenSearchIndexer.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundTimeout(SystemInterceptorProxy.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)
        at com.sun.ejb.containers.BaseContainer.callEJBTimeout(BaseContainer.java:4074)
        at com.sun.ejb.containers.EJBTimerService.deliverTimeout(EJBTimerService.java:1228)
        at com.sun.ejb.containers.EJBTimerService.access$000(EJBTimerService.java:93)
        at com.sun.ejb.containers.EJBTimerService$TaskExpiredWork.run(EJBTimerService.java:1963)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB
        at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:715)
        at com.sun.ejb.containers.util.pool.NonBlockingPool.getObject(NonBlockingPool.java:220)
        at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:442)
        ... 121 more
Caused by: javax.ejb.CreateException: Could not create stateless EJB
        at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:525)
        at com.sun.ejb.containers.StatelessSessionContainer.access$000(StatelessSessionContainer.java:99)
        at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:713)
        ... 123 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File containing crypto material password could not be found
        at io.hops.hopsworks.common.security.BaseHadoopClientsService.init(BaseHadoopClientsService.java:94)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1047)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
 at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:125)
        at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:998)
        at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:417)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:380)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:2030)
        at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:518)
        ... 125 more
Caused by: java.io.FileNotFoundException: File containing crypto material password could not be found
        at io.hops.security.HopsUtil.readCryptoMaterialPassword(HopsUtil.java:85)
        at io.hops.hopsworks.common.security.BaseHadoopClientsService.init(BaseHadoopClientsService.java:91)
        ... 151 more

and a really long list of errors for services that never installed or cant be reached.

sam.lunageo · September 14, 2022, 2:56pm

I’ve also restarted glassfish as sugested by @antonios

antonios · September 20, 2022, 8:21am

After you have restarted glassfish what is the result of dig hopsworks.glassfish.service.consul ?

Cliff_Patterson · September 21, 2022, 3:22pm

After restarting glassfish, it looks like it is running.

; <<>> DiG 9.11.3-1ubuntu1.17-Ubuntu <<>> hopsworks.glassfish.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 15225
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;hopsworks.glassfish.service.consul. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Sep 21 15:18:41 UTC 2022
;; MSG SIZE  rcvd: 63

Hopsworks still does not work.

Also, Samy has a reply limit of 3 so he can no longer contribute this post.

antonios · October 3, 2022, 1:55pm

Hi @Cliff_Patterson

It was a general limit on the forum, I increased the number of consecutive replies.

Something is off again. If you look closely you’ll see that you don’t get an IP address back. What’s the logs of consul and dnsmasq? You can get them with journalctl -r -u consul and journalctl -r -u dnsmasq