Hopswork 3.1 On-premises installation error ppcle64

Tomas_Kovacik · January 25, 2023, 8:36am

I’m installing Hopswork 3.1 on ppcle64le platform we wanted to test the functionality but we always end up with error

ERROR [2023-01-25 08:30:27,403] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, hopsworks/debug, e lastic/opensearch_security/audit/enable_rest, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, ndb/NumCPUs, ndb/TotalMemoryConfig, hopsworks/kagent_liveness/threshold, elastic/opensear ch_security/epipe/username, hopsworks/requests_verify, elastic/opensearch_security/logstash/username, elastic/opensearch_security/kibana/username, elastic/opensearch_security/admin/username, hops/yarn/pcores-vcores-mul tiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, airflow/mysql_password, hops/yarn /system-reserved-memory-mb, hopsworks/encryption_password, elastic/opensearch_security/elastic_exporter/username, alertmanager/email/to, elastic/opensearch_security/jwt/exp_ms, install/dir, prometheus/retention_time, a lertmanager/email/smtp_host, ndb/LockPagesInMainMemory, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opensearch_security/admin/password, elastic/opensearch_security/kibana/password, hiv e2/mysql_password, elastic/opensearch_security/logstash/password, hopsworks/hopsworks_public_host, elastic/opensearch_security/epipe/password, install/kubernetes, hopsworks/kagent_liveness/enabled, elastic/opensearch_s ecurity/elastic_exporter/password, hopsworks/https/port, kkafka/broker/host/public_ip, install/cloud, elastic/opensearch_security/audit/enable_transport]
INFO [2023-01-25 08:30:27,408] org.eclipse.jetty.server.ServerConnector: Stopped karamel-core@22b53226{HTTP/1.1}{0.0.0.0:9090}
INFO [2023-01-25 08:30:27,409] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@1c05a54d{/admin,null,UNAVAILABLE}
INFO [2023-01-25 08:30:27,410] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@6bd51ed8{/,null,UNAVAILABLE}

finally the 3.1 installer was used
wget https://raw.githubusercontent.com/logicalclocks/karamel-chef/3.1/hopsworks-installer.sh
but we got the some result with 3.0 , 2.0

LINUX VERSION
Red Hat Enterprise Linux Server release 7.9 (Maipo)

JAVA VERSION
openjdk version “1.8.0_262”
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

thanks

Tomas

antonios · January 30, 2023, 8:40am

Hi @Tomas_Kovacik

We have not tested any architecture other than amd64 - we have been using arm only for some submodules experimentally, so I don’t know how this will go.

The error you post here usually means your machine cannot does not have access to internet. Is this true?

Tomas_Kovacik · January 30, 2023, 4:38pm

HI Antonios,

we have access to the internet through a proxy, environment variables https_proxy , http_proxy
were exported before installation

later I tried to export http_proxy, http_proxy_port, https_proxy and https_port but the result was the same

thanks

Tomas

antonios · January 31, 2023, 8:14am

Can you export PROXY=http://myproxy:port before running the installer script?

Tomas_Kovacik · January 31, 2023, 3:20pm

i did

export PROXY=http://${myproxy}:8080

./hopsworks-installer.sh -i purge -ni
wget https://raw.githubusercontent.com/logicalclocks/karamel-chef/3.1/hopsworks-installer.sh
chmod u+x hopsworks-installer.sh
./hopsworks-installer.sh -hip $hip -kip $kip

tail -f installation.log

ERROR [2023-01-31 15:14:49,055] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, hopsworks/debug, elastic/opensearch_security/audit/enable_rest, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, ndb/NumCPUs, ndb/TotalMemoryConfig, hopsworks/kagent_liveness/threshold, elastic/opensearch_security/epipe/username, hopsworks/requests_verify, elastic/opensearch_security/logstash/username, elastic/opensearch_security/kibana/username, elastic/opensearch_security/admin/username, hops/yarn/pcores-vcores-multiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, airflow/mysql_password, hops/yarn/system-reserved-memory-mb, hopsworks/encryption_password, elastic/opensearch_security/elastic_exporter/username, alertmanager/email/to, elastic/opensearch_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, ndb/LockPagesInMainMemory, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opensearch_security/admin/password, elastic/opensearch_security/kibana/password, hive2/mysql_password, elastic/opensearch_security/logstash/password, hopsworks/hopsworks_public_host, elastic/opensearch_security/epipe/password, install/kubernetes, hopsworks/kagent_liveness/enabled, elastic/opensearch_security/elastic_exporter/password, hopsworks/https/port, kkafka/broker/host/public_ip, install/cloud, elastic/opensearch_security/audit/enable_transport]
INFO [2023-01-31 15:14:49,062] org.eclipse.jetty.server.ServerConnector: Stopped karamel-core@22b53226{HTTP/1.1}{0.0.0.0:9090}
INFO [2023-01-31 15:14:49,063] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@1c05a54d{/admin,null,UNAVAILABLE}
INFO [2023-01-31 15:14:49,063] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@6bd51ed8{/,null,UNAVAILABLE}

antonios · February 1, 2023, 8:35am

If you export the http_proxy environment variable pointing to your proxy, can you git clone https://github.com/logicalclocks/hopsworks-chef ?

Tomas_Kovacik · February 1, 2023, 5:52pm

yes

[hop@localhost ~]$ git clone GitHub - logicalclocks/hopsworks-chef: Chef Cookbook for Hopsworks
Cloning into ‘hopsworks-chef’…
remote: Enumerating objects: 14144, done.
remote: Counting objects: 100% (107/107), done.
remote: Compressing objects: 100% (64/64), done.
remote: Total 14144 (delta 38), reused 90 (delta 31), pack-reused 14037
Receiving objects: 100% (14144/14144), 10.46 MiB | 0 bytes/s, done.
Resolving deltas: 100% (8943/8943), done.

[hop@localhost ~]$ ls -al hopsworks-chef
total 116
drwxrwxr-x. 9 hop hop 4096 Feb 1 18:51 .
drwx------. 10 hop hop 4096 Feb 1 18:51 …
drwxrwxr-x. 2 hop hop 24 Feb 1 18:51 attributes
-rw-rw-r–. 1 hop hop 1801 Feb 1 18:51 Berksfile
-rwxrwxr-x. 1 hop hop 960 Feb 1 18:51 chefignore
-rwxrwxr-x. 1 hop hop 852 Feb 1 18:51 CONTRIBUTING.md
drwxrwxr-x. 3 hop hop 21 Feb 1 18:51 files
drwxrwxr-x. 8 hop hop 163 Feb 1 18:51 .git
-rwxrwxr-x. 1 hop hop 203 Feb 1 18:51 .gitignore
-rwxrwxr-x. 1 hop hop 486 Feb 1 18:51 Karamelfile
-rwxrwxr-x. 1 hop hop 5650 Feb 1 18:51 .kitchen.yml
-rwxrwxr-x. 1 hop hop 34522 Feb 1 18:51 LICENSE
-rw-rw-r–. 1 hop hop 37372 Feb 1 18:51 metadata.rb
drwxrwxr-x. 2 hop hop 54 Feb 1 18:51 providers
-rwxrwxr-x. 1 hop hop 754 Feb 1 18:51 README.md
drwxrwxr-x. 2 hop hop 199 Feb 1 18:51 recipes
drwxrwxr-x. 2 hop hop 54 Feb 1 18:51 resources
drwxrwxr-x. 3 hop hop 21 Feb 1 18:51 templates

antonios · February 6, 2023, 4:18pm

That’s a weird situation. Can you maybe check your exported environment variables and post them here (remove any sensitive information) with env?

unset all proxy related variables and the installer will ask you later. Maybe this way it will work.

Tomas_Kovacik · February 13, 2023, 9:40am

Antonios,

we get the same errors regardless of whether the http_proxy and https_proxy variable is exportted or unset
when proxy is exported we can see that karamel started with

Press ENTER to continue
Found karamel
Running command from /home/hop/karamel-0.7:
export http_proxy=http://xx.xx.xx.xx:pppp
export http_proxy_host=xx.xx.xx.xx
export http_proxy_port=pppp
export https_proxy=http://xx.xx.xx.xx:pppp
export https_proxy_host=xx.xx.xx.xx
export https_proxy_port=pppp
setsid ./bin/karamel -headless -launch …/cluster-defns/hopsworks-installation.yml > …/installation.log 2>&1 &

withoult proxy (unset http_proxy https_proxy)

karamel started (empty space where the proxy was)

Found karamel
Running command from /home/hop/karamel-0.7:

setsid ./bin/karamel -headless -launch …/cluster-defns/hopsworks-installation.yml > …/installation.log 2>&1 &

but in both cases the result is the same

ERROR [2023-02-13 09:32:35,343] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, hopsworks/debug, elastic/opensearch_security/audit/enable_rest, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, ndb/NumCPUs, ndb/TotalMemoryConfig, hopsworks/kagent_liveness/threshold, elastic/opensearch_security/epipe/username, hopsworks/requests_verify, elastic/opensearch_security/logstash/username, elastic/opensearch_security/kibana/username, elastic/opensearch_security/admin/username, hops/yarn/pcores-vcores-multiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, airflow/mysql_password, hops/yarn/system-reserved-memory-mb, hopsworks/encryption_password, elastic/opensearch_security/elastic_exporter/username, alertmanager/email/to, elastic/opensearch_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, ndb/LockPagesInMainMemory, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opensearch_security/admin/password, elastic/opensearch_security/kibana/password, hive2/mysql_password, elastic/opensearch_security/logstash/password, hopsworks/hopsworks_public_host, elastic/opensearch_security/epipe/password, install/kubernetes, hopsworks/kagent_liveness/enabled, elastic/opensearch_security/elastic_exporter/password, hopsworks/https/port, kkafka/broker/host/public_ip, install/cloud, elastic/opensearch_security/audit/enable_transport]
INFO [2023-02-13 09:32:35,348] org.eclipse.jetty.server.ServerConnector: Stopped karamel-core@22b53226{HTTP/1.1}{0.0.0.0:9090}

I’s possible to enable more verbose log output or can we debug somehow this state ?

thanks

Tomas

Tomas_Kovacik · February 13, 2023, 10:10am

one more thing during the installation we received warnings

WARNING: We recommend at least 30GB of disk space on the root partition as well as at least 50GB on the /mnt partition.
You have 41G space on ‘/’, and 0G on ‘/mnt’.

WARNING: Hopsworks installer only supports ‘ext4’ as a file system.
If you want to install Hopsworks on ‘zfs’ or ‘xfs’, contact www.logicalclocks.com for Enterprise support.

can these warnings cause a problem in the installation which we have ?

thanks

Tomas

antonios · February 13, 2023, 2:32pm

Hi guys. Summarizing our call earlier.

On-prem installation

To begin with, I would strongly suggest you provision an x86_64 machine to install Hopsworks with at least the following specs:

8 vcores
32GB RAM
300GB disk

The machine should allow egress to github.com, repo.hops.work among others. Also, the unix user who’s running the install script should be able to sudo su

Then I suggest you download again the download script for our 3.1 release FROM HERE

If you want to enable debugging logging please export the following environment variable before executing the install script

export JAVA_OPTS="$JAVA_OPTS -Ddw.logging.level=DEBUG"

Pre PoC on app.hopsworks.ai
As we discussed the easiest approach to begin with Hopsworks evaluation is to use our serverless platform at no cost.

You can’t use real data due to regulations but certainly you can use synthetic data to evaluate the Feature Store.

Hopsworks Feature Store on you Cloud
Another suggestion is to use our managed Enterprise solution on the Cloud. You will have to connect your account on any of AWS, GCP or Azure The control plane runs on our cloud but the cluster will be created on your account, so no data will leave your cloud. Registering will give you some free credits to evaluate.

If that is something interesting, let us know and we can have a short call for more details.

Tomas_Kovacik · February 21, 2023, 7:10am

Hi,

we migrated to x86 but without success the installation ends just like on ppcle64, extra debug does not provide any additional information why the installation does not continue

installer version we tried
wget https://raw.githubusercontent.com/logicalclocks/karamel-chef/3.1/hopsworks-installer.sh

ERROR
ERROR [2023-02-14 11:43:13,120] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, hopsworks/debug, elastic/opensearch_security/audit/enable_rest, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, ndb/NumCPUs, ndb/TotalMemoryConfig, hopsworks/kagent_liveness/threshold, elastic/opensearch_security/epipe/username, hopsworks/requests_verify, elastic/opensearch_security/logstash/username, elastic/opensearch_security/kibana/username, elastic/opensearch_security/admin/username, hops/yarn/pcores-vcores-multiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, airflow/mysql_password, hops/yarn/system-reserved-memory-mb, hopsworks/encryption_password, elastic/opensearch_security/elastic_exporter/username, alertmanager/email/to, elastic/opensearch_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, ndb/LockPagesInMainMemory, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opensearch_security/admin/password, elastic/opensearch_security/kibana/password, hive2/mysql_password, elastic/opensearch_security/logstash/password, hopsworks/hopsworks_public_host, elastic/opensearch_security/epipe/password, install/kubernetes, hopsworks/kagent_liveness/enabled, elastic/opensearch_security/elastic_exporter/password, hopsworks/https/port, kkafka/broker/host/public_ip, install/cloud, elastic/opensearch_security/audit/enable_transport]

we have access to github and the git clone command (git clone GitHub - logicalclocks/hopsworks-chef: Chef Cookbook for Hopsworks) works fine

do you have any other suggestions ?

thanks

Tomas

antonios · February 23, 2023, 11:18am

Hi @Tomas_Kovacik

Could you please try and download the installer script again? There was an error in our community version installer. I tried it on a VM and it worked this time.