Error in installing hosworks on-prem

Hopsworks
#1

Our unix administrator tried to install hosworks but failed. here is the error message:

INFO [2021-05-13 16:20:48,978] se.kth.karamel.backend.ClusterManager: Forking groups ‘Hops’ …
INFO [2021-05-13 16:20:48,979] se.kth.karamel.backend.ClusterManager: \o/\o/\o/\o/\o/‘Hops’ GROUPS_FORKED \o/\o/\o/\o/\o/
INFO [2021-05-13 16:20:48,979] se.kth.karamel.backend.ClusterManager: Launching ‘Hops’ …
INFO [2021-05-13 16:20:48,979] se.kth.karamel.backend.ClusterManager: groups ‘[se.kth.karamel.backend.running.model.GroupRuntime@3e2e651a]’
INFO [2021-05-13 16:20:48,980] se.kth.karamel.backend.ClusterManager: Gogo
INFO [2021-05-13 16:20:48,980] se.kth.karamel.backend.ClusterManager: Using provider ‘se.kth.karamel.common.clusterdef.Baremetal@170df986’
INFO [2021-05-13 16:20:48,980] se.kth.karamel.backend.ClusterManager: Using launcher ‘se.kth.karamel.backend.launcher.baremetal.BaremetalLauncher@6558f4e2’
INFO [2021-05-13 16:20:48,980] se.kth.karamel.backend.ClusterManager: Using launcher ‘se.kth.karamel.backend.launcher.baremetal.BaremetalLauncher@6558f4e2’
INFO [2021-05-13 16:20:49,187] se.kth.karamel.backend.ClusterManager: \o/\o/\o/\o/\o/‘Hops’ MACHINES_FORKED \o/\o/\o/\o/\o/
INFO [2021-05-13 16:20:49,187] se.kth.karamel.backend.ClusterManager: Going to serve ‘SUBMIT_INSTALL_DAG’
INFO [2021-05-13 16:20:49,187] se.kth.karamel.backend.ClusterManager: Running the DAG for ‘Hops’ …
INFO [2021-05-13 16:20:49,241] se.kth.karamel.client.api.CookbookCacheIml: 0-level cookbooks for Hops is 22
INFO [2021-05-13 16:20:49,243] se.kth.karamel.client.api.CookbookCacheIml: 1-level cookbooks for Hops is 21
INFO [2021-05-13 16:20:59,011] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2021-05-13 16:20:59,128] se.kth.karamel.backend.machines.SshMachine: 10.66.64.36: connecting …
INFO [2021-05-13 16:20:59,130] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2021-05-13 16:20:59,145] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2021-05-13 16:20:59,342] se.kth.karamel.backend.machines.SshMachine: 10.66.64.36: Yey!! connected ^-^
ERROR [2021-05-13 16:20:59,391] se.kth.karamel.backend.machines.MachinesMonitor:
! net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
! at net.schmizz.sshj.SSHClient.auth(SSHClient.java:225) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:340) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:359) ~[sshj-0.20.0.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:479) ~[karamel-core-0.6.jar:na]
! … 7 common frames omitted
! Causing: se.kth.karamel.common.exception.KaramelException: 10.66.64.36: Authentication problem using ssh keys.
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:511) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.ping(SshMachine.java:537) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.MachinesMonitor.run(MachinesMonitor.java:101) ~[karamel-core-0.6.jar:na]
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_282]
! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_282]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_282]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_282]
! at java.lang.Thread.run(Thread.java:748) [na:1.8.0_282]
INFO [2021-05-13 16:21:04,152] se.kth.karamel.backend.machines.MachinesMonitor: Sending pause signal to all machines
ERROR [2021-05-13 16:22:59,146] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2021-05-13 16:22:59,146] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2021-05-13 16:23:09,394] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2021-05-13 16:23:09,404] se.kth.karamel.backend.machines.SshMachine: 10.66.64.36: connecting …
INFO [2021-05-13 16:23:09,404] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2021-05-13 16:23:09,419] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2021-05-13 16:23:09,444] se.kth.karamel.backend.machines.SshMachine: 10.66.64.36: Yey!! connected ^-^
ERROR [2021-05-13 16:23:09,491] se.kth.karamel.backend.machines.MachinesMonitor:
! net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
! at net.schmizz.sshj.SSHClient.auth(SSHClient.java:225) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:340) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:359) ~[sshj-0.20.0.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:479) ~[karamel-core-0.6.jar:na]
! … 7 common frames omitted
! Causing: se.kth.karamel.common.exception.KaramelException: 10.66.64.36: Authentication problem using ssh keys.
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:511) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.ping(SshMachine.java:537) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.MachinesMonitor.run(MachinesMonitor.java:101) ~[karamel-core-0.6.jar:na]

#2

We managed to get around the ssh problem, now we are currently stuck at ‘make solo.rb’ step…

#3

Is the cluster behind a proxy? If so, have a look at this post Error installing hopsworks (Ubuntu )

#4

Yes… we will try set some env variables according to the other thread.

http_proxy = [proxy host]
http_proxy_port = [proxy port]

#5

Still having problem… here is the error for ‘make solo.rb’:

INFO [2021-05-19 15:48:26,110] se.kth.karamel.backend.machines.SshMachine: 10.66.64.36: Running task: make solo.rb

INFO [2021-05-19 15:48:28,465] se.kth.karamel.backend.machines.MachinesMonitor: Sending resume signal to all machines

172.25.4.201 - - [19/May/2021:15:48:30 +0000] “GET /api/system/ping HTTP/1.1” 200 - “http://10.66.64.36:9090/index.html” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36” 1

INFO [2021-05-19 15:48:31,538] se.kth.karamel.backend.machines.SshMachine: 10.66.64.36: Running task: make solo.rb

172.25.4.201 - - [19/May/2021:15:48:31 +0000] “PUT /api/cluster/processCommand HTTP/1.1” 200 - “http://10.66.64.36:9090/index.html” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36” 3

172.25.4.201 - - [19/May/2021:15:48:37 +0000] “PUT /api/cluster/processCommand HTTP/1.1” 200 - “http://10.66.64.36:9090/index.html” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36” 3

ERROR [2021-05-19 15:48:38,438] se.kth.karamel.backend.dag.DagNode: Failed ‘make solo.rb on 10.66.64.36’ because '10.66.64.36: Command did not complete: mkdir -p /home/mweiner/.karamel/install ; cd /home/mweiner/.karamel/install; echo $$ > pid; echo '#!/bin/bash

set -eo pipefail

sudo touch solo.rb

sudo chmod 777 solo.rb

cat > solo.rb <<-‘END_OF_FILE’

file_cache_path “/tmp/chef-solo”

cookbook_path ["/home/mweiner/.karamel/cookbooks/hopsworks-chef_vendor"]

END_OF_FILE’ > make_solo_rb.sh ; chmod +x make_solo_rb.sh ; ./make_solo_rb.sh’, DAG is stuck here :frowning:

INFO [2021-05-19 15:48:38,469] se.kth.karamel.backend.machines.MachinesMonitor: Sending pause signal to all machines

#6

Hi jinjCC. The http_proxy support is, unfortunately, only experimental, so we provide no guarantees if it works or not. We recommend you install on a server with outbound Internet access without going through a HTTP Proxy.

The error above looks like the installation was performed by the user “mweiner”. An assumption of the installer is that there is a home directory for the user “mweiner” called “/home/mweinter”. For some reason that i do not know the answer to, the script fails at one of the commands that try to (1) create this directory: mkdir -p /home/mweiner/.karamel/install
or (2) create and run the script “make_solo_rb.sh”

We support a standard vanilla Ubuntu 18.04 or RedHat/Centos 7.x operating system. We do not support custom kernels with non-standard configurations.