Hopsworks 3.0 installation problem (karamel ssh)

Hopsworks
#1

Hello, I am installing Hopsworks 3.0 as on primes on Ubuntu Server 22.04 LTS via .sh file. Since I am new to this, I don’t know what to do so I wanted to ask here. What is the cause of the error below? And how do I solve this error?

ERROR [2023-05-28 11:24:46,667] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO  [2023-05-28 11:24:46,667] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
192.168.1.72 -  -  [28/May/2023:11:24:47 +0000] "PUT /api/cluster/processCommand HTTP/1.1" 200 - "http://192.168.1.61:9090/index.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 2
192.168.1.72 -  -  [28/May/2023:11:24:53 +0000] "PUT /api/cluster/processCommand HTTP/1.1" 200 - "http://192.168.1.61:9090/index.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 2
INFO  [2023-05-28 11:24:56,730] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO  [2023-05-28 11:24:56,734] se.kth.karamel.backend.machines.SshMachine: 192.168.1.61: connecting ...
INFO  [2023-05-28 11:24:56,735] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO  [2023-05-28 11:24:56,747] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
INFO  [2023-05-28 11:24:56,756] se.kth.karamel.backend.machines.SshMachine: 192.168.1.61: Yey!! connected ^-^
ERROR [2023-05-28 11:24:56,806] se.kth.karamel.backend.machines.MachinesMonitor:
! net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
! at net.schmizz.sshj.SSHClient.auth(SSHClient.java:225) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:340) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:359) ~[sshj-0.20.0.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:479) ~[karamel-core-0.6.jar:na]
! ... 7 common frames omitted
! Causing: se.kth.karamel.common.exception.KaramelException: 192.168.1.61: Authentication problem using ssh keys.
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:511) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.ping(SshMachine.java:537) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.MachinesMonitor.run(MachinesMonitor.java:101) ~[karamel-core-0.6.jar:na]
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_362]
! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_362]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_362]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_362]
! at java.lang.Thread.run(Thread.java:750) [na:1.8.0_362]
#2

Hi @cmlops,

Hopsworks 3.0 doesn’t support Ubuntu 22.04 LTS. You can try to install the newer version of Hopsworks (3.2) which supports Ubuntu 22.04 LTS.

This is the installer to use for 3.2: https://raw.githubusercontent.com/logicalclocks/karamel-chef/3.2/hopsworks-installer.sh

–
Fabio

#3

Hi @Fabio,

Thank you for your return. So, what are the prerequisites to install hopsworks as on primes? I don’t recall this being mentioned in the documentation, can you share a list?

Thank you.

#4

I downloaded the version you said and started to install it again and got the following error.

ERROR [2023-05-29 05:50:39,391] se.kth.karamel.backend.machines.SshMachine: 192.168.1.61: Terminal error while executing command
! se.kth.karamel.common.exception.KaramelException: 192.168.1.61: Command install chefdk failed with exit code 1
! at se.kth.karamel.backend.machines.SshMachine.runSshCmd2(SshMachine.java:377) [karamel-core-0.9-SNAPSHOT.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.runTask(SshMachine.java:286) [karamel-core-0.9-SNAPSHOT.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.run(SshMachine.java:163) [karamel-core-0.9-SNAPSHOT.jar:na]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_362]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_362]
! at java.lang.Thread.run(Thread.java:750) [na:1.8.0_362]
ERROR [2023-05-29 05:50:39,391] se.kth.karamel.backend.machines.SshMachine: 192.168.1.61: Exhausted all 2 retries, giving up!!!
ERROR [2023-05-29 05:50:39,391] se.kth.karamel.backend.dag.DagNode: Failed 'install chefdk on 192.168.1.61' because '192.168.1.61: Command did not complete: set -eo pipefail; mkdir -p /home/ggadmin/.karamel/install ; cd /home/ggadmin/.karamel/install; echo $$ > pid; echo '#!/bin/bash

RES=0
if [ ubuntu == "redhat" ] ; then

  yum list installed chefdk
  if [ $? -ne 0 ] ; then
    chefdkfile='chefdk-3.7.23-1.el7.x86_64.rpm'

    rm -f "$chefdkfile"
    wget "https://hopsworks-distribution.s3-eu-west-1.amazonaws.com/$chefdkfile"

    sudo yum install -y "$chefdkfile"
    RES=$?
    if [ $RES -ne 0 ] ; then
      sleep 10
      sudo yum install -y "$chefdkfile"
    fi
  fi

elif [ ubuntu == "ubuntu" ] ; then

  dpkg -s chefdk
  if [ $? -ne 0 ] ; then
    chefdkfile='chefdk_3.7.23-1_amd64.deb'
    rm -f "$chefdkfile"
    wget "https://hopsworks-distribution.s3-eu-west-1.amazonaws.com/$chefdkfile"

    sudo dpkg -i "$chefdkfile"
    RES=$?
    if [ $RES -ne 0 ] ; then
      sleep 10
      sudo dpkg -i "$chefdkfile"
    fi
  fi
else
 echo "Unrecognized version of linux. Not ubuntu or redhat family."
 exit 1
fi
if [ $RES -eq 0 ] ; then
  # Fix for expired Lets Encrypt CA
  sudo sed -ie "/DST Root CA X3/,+19d" /opt/chefdk/embedded/ssl/certs/cacert.pem
  echo 'install chefdk' >> succeed_list
fi
exit $RES
' > install-chefdk.sh ; chmod +x install-chefdk.sh ; ./install-chefdk.sh
', DAG is stuck here :(