Hopsworks-installer.sh Install Error

[jcolvin]

Hello,

I tried installing Hopsworks on an on-premise RHEL7 server today using the instructions from the readthedocs site. Everything seemed to be going smoothly, and the Karamel UI started up. However nothing really appeared on the web page. When I tailed the installation log, I could see that after a minute or so this error popped up:

ERROR [2020-04-29 15:15:56,424] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, elastic/opendistro_security/epipe/username, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, elastic/opendistro_security/admin/username, hopsworks/kagent_liveness/threshold, elastic/opendistro_security/audit/enable_transport, hopsworks/requests_verify, elastic/opendistro_security/elastic_exporter/username, kagent/python_conda_versions, elastic/opendistro_security/audit/enable_rest, mysql/password, hopsworks/featurestore_online, hopsworks/admin/password, hops/cgroups, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, elastic/opendistro_security/logstash/password, hopsworks/encryption_password, elastic/opendistro_security/epipe/password, alertmanager/email/to, elastic/opendistro_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, elastic/opendistro_security/kibana/password, hops/yarn/memory_mbs, alertmanager/email/from, hopsworks/master/password, hops/yarn/vcores, elastic/opendistro_security/kibana/username, hive2/mysql_password, install/kubernetes, hopsworks/kagent_liveness/enabled, hopsworks/https/port, elastic/opendistro_security/admin/password, install/cloud, elastic/opendistro_security/logstash/username, elastic/opendistro_security/elastic_exporter/password]

Seems like this means there’s a config error in the karamel-chef repo?

[Fabio]

Hi James,

Here the deployment tool (Karamel) is trying to reach Github to make sure that all the attributes are valid. The error you see usually happens if Karamel can’t reach Github.

Do you have to go through a proxy to reach Github? If so, you can export the http_proxy and https_proxy environment variables with credentials/location of the proxy and it should work.

–
Fabio

[Jim_Dowling]

Hi James
I just added support now for installing from behind a http_proxy - you can be our beta tester :slight_smile

If you download the latest version of the hopsworks-installer.sh file from master:
wget https://raw.githubusercontent.com/logicalclocks/karamel-chef/master/hopsworks-installer.sh

and run
./hopsworks-installer.sh

There is a switch for specifying the http-proxy as well:
./hopsworks-installer.sh --http-proxy http://your.proxy.server

Also, if you set the http_proxy or https_proxy environment variables before running hopsworks-installer.sh, that will also work (but karamel should have picked up one of those two environment variables if they had have been set).

[jcolvin]

Hi Jim,

Thanks for the quick response. the “http_proxy” envvar had already been set when I tried to install initially.

Now when trying to install with the latest hopsworks-installer.sh, I get a single line in the log file:

./hopsworks-installer.sh: line 1230: https_proxy=http://webproxy.bcbst.com: No such file or directory

[jcolvin]

Updating the following lines seem to have fixed that issue, but then I still get the “invalid YAML file” error.

line 946 change to
KARAMEL_HTTP_PROXY=“http_proxy=$PROXY”

line 1230 change to
export $KARAMEL_HTTP_PROXY && setsid ./bin/karamel -headless -launch …/$YML_FILE $SUDO_PWD > …/installation.log 2>&1 &

[Fabio]

Hi James,

Sorry for the late reply. There was an issue with Karamel not picking up the correct proxy configuration. I patched it and uploaded it a new version here: http://www.karamel.io/sites/default/files/downloads/karamel-0.6.tgz

The installer should automatically download the new version. Make sure you run the purge first ./hopswork-installer.sh -i purge -ni - otherwise you can always download and extract the package manually.

These are the correct environment variables to export

http_proxy                             
http_proxy_port
https_proxy
https_proxy_port

I didn’t manage to fix the hopsworks-installer.sh script yet. So please try to export the env variables and it should work. Let me know if it doesn’t.

–
Fabio

[Chen_Kepeng]

Hello,
karamel-0.6.tgz cannot be download because http://www.karamel.io/ is not accessible…looks like a problem with website itself.
any ideas to bypass karamel? It is really unstable.
thanks.