Hopsworks installation error on GCP

I tried to install hopsworks single node installation on personal GCP account.

Here is machine (VM) / os details on GCP

[rhel-7-v20200403]
DISK 500 GB

Machine type
e2-standard-16 (16 vCPUs, 64 GB memory)

I followed following URL to install on GCP

https://hopsworks.readthedocs.io/en/latest/getting_started/installation_guide/platforms/cloud.html

wget https://raw.githubusercontent.com/logicalclocks/karamel-chef/master/hopsworks-installer.sh chmod +x hopsworks-installer.sh ./hopsworks-installer.sh

I choose single host installation and GCP. After an hour I get following error. I refreshed and tried couple of more time with new VMs but was not successful.

Error message:

INFO [2020-04-18 16:33:38,498] se.kth.karamel.backend.machines.SshMachine: 10.142.0.3: Running task: hops_airflow::default

ERROR [2020-04-18 17:38:29,969] se.kth.karamel.backend.dag.DagNode: Failed ‘hops_airflow::default on 10.142.0.3’ be

cause '10.142.0.3: Command did not complete: mkdir -p /home/vksingh/.karamel/install ; cd /home/vksingh/.karamel/install; echo $$ > pid; echo '#!/bin/bash

set -eo pipefail

echo $(date ‘+%H:%M:%S’): ‘hops_airflow__default’ >> order

cat > hops_airflow__default.json <<-‘END_OF_FILE’

{

json file output

}

sudo chef-solo -c /home/vksingh/.karamel/install/solo.rb -j /home/vksingh/.karamel/install/hops_airflow__default.json 2>&1 | tee hops_airflow__default.log

echo ‘https://github.com/logicalclocks/airflow-chef/tree/master/hops_airflow::default’ >> succeed_list

’ > hops_airflow__default.sh ; chmod +x hops_airflow__default.sh ; ./hops_airflow__default.sh

', DAG is stuck here :frowning:

INFO [2020-04-18 17:38:32,720] se.kth.karamel.backend.machines.MachinesMonitor: Sending pause signal to all machines

When I look into hop_airflow log I see following error

  • bash[init_airflow_db] action run

================================================================================

Error executing action run on resource ‘bash[init_airflow_db]’

================================================================================

Hi. The hopsworks-installer.sh is installing our nightly build on master. We had an issue with SELinux on the latest centos image, so if you download it again and retry, it should be fixed. hopsworks-installer.sh will be officially supported from version 1.3.

This isn’t in readthedocs yet, but there is a new more convenient way to install hopsworks on GCP. If you have google-tools installed, you can do the following:

git clone https://github.com/logicalclocks/karamel-chef.git
cd karamel-chef/cloud/gcp
. setup.sh
./install.sh cpu community

This will spin up a 32GB/8-core VM with hopsworks. You can edit the config.sh file in the same folder to change the image/region/zone.
To destroy your VM, run:
./delete.sh cpu

You can install a version with a GPU using
./install.sh cpu community
and
./delete.sh gpu
to delete it.

thanks. It worked on single host GCP

1 Like