Hopsworks 3.0 installation on EC2

AbhiD · August 19, 2022, 3:49am

I’m installing Hopsworks 3.0 on EC2 and getting error at task - “install chefdk”

However, I installed chefdk manually and restarted Hopsworks, but still getting the same. I’ve also installed Karamel via installation script separately.
EC2 has CentOS.

Below is from installation.log file

INFO [2022-08-18 16:41:49,147] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Running task: apt-get essentials
INFO [2022-08-18 17:27:08,391] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Running task: install chefdk
INFO [2022-08-18 17:27:34,563] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Running task: install chefdk
ERROR [2022-08-18 17:28:01,411] se.kth.karamel.backend.dag.DagNode: Failed ‘install chefdk on 10.208.57.174’ because '10.208.57.174: Command did not complete: set -eo pipefail; mkdir -p /home/user/.karamel/install ; cd /home/user/.karamel/install; echo $$ > pid; echo '#!/bin/bash

RES=0
if [ redhat == “redhat” ] ; then

yum list installed chefdk
if [ $? -ne 0 ] ; then
chefdkfile=‘chefdk-3.7.23-1.el7.x86_64.rpm’

rm -f "$chefdkfile"
wget "https://hopsworks-distribution.s3-eu-west-1.amazonaws.com/$chefdkfile"

sudo yum install -y "$chefdkfile"
RES=$?
if [ $RES -ne 0 ] ; then
  sleep 10
  sudo yum install -y "$chefdkfile"
fi

fi

elif [ redhat == “ubuntu” ] ; then

dpkg -s chefdk
if [ $? -ne 0 ] ; then
chefdkfile=‘chefdk_3.7.23-1_amd64.deb’
rm -f “$chefdkfile”
wget “https://hopsworks-distribution.s3-eu-west-1.amazonaws.com/$chefdkfile”

sudo dpkg -i "$chefdkfile"
RES=$?
if [ $RES -ne 0 ] ; then
  sleep 10
  sudo dpkg -i "$chefdkfile"
fi

fi
else
echo “Unrecognized version of linux. Not ubuntu or redhat family.”
exit 1
fi
if [ $RES -eq 0 ] ; then

Fix for expired Lets Encrypt CA

sudo sed -ie “/DST Root CA X3/,+19d” /opt/chefdk/embedded/ssl/certs/cacert.pem
echo ‘install chefdk’ >> succeed_list
fi
exit $RES
’ > install-chefdk.sh ; chmod +x install-chefdk.sh ; ./install-chefdk.sh
', DAG is stuck here
INFO [2022-08-18 17:28:03,859] se.kth.karamel.backend.machines.MachinesMonitor: Sending pause signal to all machines
ERROR [2022-08-18 17:42:57,019] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 17:42:57,019] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2022-08-18 17:43:08,943] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2022-08-18 17:43:08,954] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: connecting …
INFO [2022-08-18 17:43:08,955] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2022-08-18 17:43:08,964] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2022-08-18 17:43:08,991] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Yey!! connected ^-^
ERROR [2022-08-18 17:58:09,245] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 17:58:09,245] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2022-08-18 17:58:19,113] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2022-08-18 17:58:19,123] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: connecting …
INFO [2022-08-18 17:58:19,123] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2022-08-18 17:58:19,131] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2022-08-18 17:58:19,149] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Yey!! connected ^-^
ERROR [2022-08-18 18:13:19,409] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 18:13:19,409] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2022-08-18 18:13:39,277] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2022-08-18 18:13:39,288] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: connecting …
INFO [2022-08-18 18:13:39,288] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2022-08-18 18:13:39,298] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2022-08-18 18:13:39,311] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Yey!! connected ^-^
ERROR [2022-08-18 18:28:39,519] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 18:28:39,519] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2022-08-18 18:28:49,387] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2022-08-18 18:28:49,396] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: connecting …
INFO [2022-08-18 18:28:49,396] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2022-08-18 18:28:49,420] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2022-08-18 18:28:49,433] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Yey!! connected ^-^
ERROR [2022-08-18 18:43:49,639] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 18:43:49,639] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2022-08-18 18:44:09,506] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2022-08-18 18:44:09,517] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: connecting …
INFO [2022-08-18 18:44:09,517] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2022-08-18 18:44:09,525] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2022-08-18 18:44:09,537] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Yey!! connected ^-^
ERROR [2022-08-18 18:59:09,718] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 18:59:09,718] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2022-08-18 18:59:19,615] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2022-08-18 18:59:19,624] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: connecting …
INFO [2022-08-18 18:59:19,624] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2022-08-18 18:59:19,634] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2022-08-18 18:59:19,647] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Yey!! connected ^-^
ERROR [2022-08-18 19:14:19,862] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 19:14:19,862] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2022-08-18 19:14:39,731] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2022-08-18 19:14:39,752] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: connecting …
INFO [2022-08-18 19:14:39,753] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2022-08-18 19:14:39,762] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2022-08-18 19:14:39,774] se.kth.karamel.backend.machines.SshMachine: 10.208.57.174: Yey!! connected ^-^
ERROR [2022-08-18 19:29:40,012] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2022-08-18 19:29:40,012] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN

AbhiD · August 23, 2022, 4:24am

Hello Hopsworks team,

I could proceed with installation on EC2 after fixing sudoers file issue. But now facing the issue from hops__ndb install component and unable to proceed. I have run the hopsworks purge and cleaned the services and folders. After multiple tries, still getting same.

Running handlers:

[2022-08-23T09:14:03+05:30] ERROR: Running exception handlers

Running handlers complete

[2022-08-23T09:14:03+05:30] ERROR: Exception handlers complete

Chef Client failed. 10 resources updated in 08 seconds

[2022-08-23T09:14:03+05:30] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out

[2022-08-23T09:14:03+05:30] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report

[2022-08-23T09:14:03+05:30] FATAL: Mixlib::ShellOut::ShellCommandFailed: hops_ndb[install] (hops::ndb line 134) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[flyway_migrate] (/tmp/chef-solo/cookbooks/hops/providers/ndb.rb line 44) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’

---- Begin output of “bash” “/tmp/chef-script20220823-7920-gvd19r” ----

STDOUT: Flyway Community Edition 6.5.1 by Redgate

STDERR: ERROR:

Unable to obtain connection from database (jdbc:mysql://127.0.0.1:3306/hops) for user ‘kthfs’: Access denied for user ‘kthfs’@‘localhost’ (using password: YES)

SQL State : 28000

Error Code : 1045

Message : Access denied for user ‘kthfs’@‘localhost’ (using password: YES)

Any pointers would help.

Thanks