Hopsworks installation issues

Hopsworks
#1

Hi, I am facing a problem in installing the Hopworks following this guideline Hopsworks Installer — Documentation 2.4 documentation

I am using the following command to start the installation process:
./hopsworks-installer.sh -i localhost -c on-premises -ni -pwd

In the first attempt trying to install, I had an error in the kagent::default task. After reading about in this topic, "kagent::default" is not able to be installed in a cluster environment .

I purge the installation using the command ./hopsworks-installer.sh -i purge -cl -ni try to reinstall it, by rerun the previous command. Then, I had another error, this time in the hops_airflow::default task, where the log error said that could not initialize the database because the user airflow_db has access denied.

Lately, I try to remove everything related with mysql from the machine runing sudo apt purge -y mysql*, manullay remove all folders related to things related to mysql, and unfortanelly remove all folder in /home directory that was not the user I used folder. Then trying to reinstalling it again I am getting an error in the conda::install task because the home directory for anaconda does not exist.

I am using a Ubuntu 18.04 VM, with the recommend configuration.

Any help in installing Hopsworks, or at least help to purge everything and start the process again (without creating a new VM), will be very much appreciate.

#2

Hej @joaoferreira,

Can you try and manually remove:
/srv/hops
and
/home/<user>/.karamel
Best,
Alex

#3

Hi @Alex ,

Thank you for the quick reply. I follow your instructions, and besides that, remove /srv/hopsworks-data as well.

Trying to install again, I get the same error in conda::install , the log error is the following.
FATAL: Chef::Exceptions::EnclosingDirectoryDoesNotExist: template[/home/anaconda/.condarc] (conda::install line 74) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /home/anaconda does not exist.
Should I manually create a directory for anaconda in home? Is necessary a user to be owner of this folder? Or the installation process should take care of it?

Again, thank you for your reply, and appreciate your help!

#4

I was able to fix the previous error, and all related errors to home directory/user by deleting the user created with the command sudo deluser --remove-home <user>. However, now I am facing the error below in the hopsworks::default recipe.

FATAL: RuntimeError: ruby_block[check_db_empty] (hopsworks::default line 196) had an error: RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute

There is a way to delete the database created in previous installation attempts?
Thank you, in advance, for your help.

#5

Here is the steps I used to fully purge installation on Centos

Hops

./hopsworks-installer.sh -i purge -ni

Chef

rpm -qa chef
sudo yum remove [result of prev command]

MySQL if needed

rpm -qa mysql
sudo yum remove [result of prev command]

Docker

sudo yum remove docker-ce docker-ce-cli containerd.io -y
sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd

Old data and folder

sudo rm -rf /srv/hopsworks-data
sudo rm -rf /opt/chefdk/*
sudo rm -rf /opt/hops/*
sudo rm -rf /tmp/chef-solo/*

Some stuff

sudo rm -rf /home/[your user]/.berkshelf/
sudo rm -rf /home/[your user]/.pki/
sudo rm -rf /home/[your user]/.gitconfig

Users

sudo userdel -r anaconda
sudo userdel -r airflow
sudo userdel -r elastic
sudo userdel -r yarn
sudo userdel -r mysql
sudo userdel -r hdfs
sudo userdel -r mapred
sudo userdel -r yarnapp
sudo userdel -r rmyarn
sudo userdel -r elkadmin
sudo userdel -r hive
sudo userdel -r kagent
sudo userdel -r flink
sudo userdel -r kafka
sudo userdel -r livy
sudo userdel -r glassfish
sudo userdel -r tez
sudo userdel -r onlinefs

#6

Hi @FeSS thank you for your answer!

I try to follow the steps you suggest, however since I am using Ubuntu, I could not follow strictly your commands.

The error I am getting now is on hopsworks::default task. Here is the output of the task

RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute

It seems that the database created by other installations attempts was not deleted from the system. Could you help me with this on Ubuntu?

Thank you for your time.