ERROR : se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb

Freeman · November 23, 2020, 3:15am

OS Version:

CentOS Linux release 7.6.1810 (Core)

What would you like to do?

(1) Install a single-host Hopsworks cluster.

Where are you installing Hopsworks?

(1) On-premises or private cloud.

I encountered an error when I tried to install the Hopsworks, the error message is as follows:

ERROR [2020-11-23 02:40:39,766] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, elastic/opendistro_security/epipe/username, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, elastic/opendistro_security/admin/username, hopsworks/kagent_liveness/threshold, elastic/opendistro_security/audit/enable_transport, hopsworks/requests_verify, elastic/opendistro_security/elastic_exporter/username, elastic/opendistro_security/audit/enable_rest, hops/yarn/pcores-vcores-multiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, elastic/opendistro_security/logstash/password, hops/yarn/system-reserved-memory-mb, hopsworks/encryption_password, elastic/opendistro_security/epipe/password, alertmanager/email/to, elastic/opendistro_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, elastic/opendistro_security/kibana/password, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opendistro_security/kibana/username, hive2/mysql_password, install/kubernetes, hopsworks/kagent_liveness/enabled, hopsworks/https/port, ndb/DataMemory, elastic/opendistro_security/admin/password, install/cloud, elastic/opendistro_security/logstash/username, elastic/opendistro_security/elastic_exporter/password]

I’ve searched for the similar questions in Google, the links are:

Hopsworks-installer.sh Install Error
Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb

According to these topics, I used the Privoxy as the proxy software, and the following commands can be executed correctly,

$ nc -zv github.com 80
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.30.255.113:80.
Ncat: 0 bytes sent, 0 bytes received in 0.26 seconds.

$ nc -zv github.com 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.30.255.113:443.
Ncat: 0 bytes sent, 0 bytes received in 0.24 seconds.

The configration of Privoxy in /etc/profile

export http_proxy=http://127.0.0.1:8118
export https_proxy=https://127.0.0.1:8118

I’ve also tried to execute

"./hopsworks-installer.sh -p https://127.0.0.1:8118

", but the problem still exists.

It seems to be a common problem, has anyone solved this problem?

Freeman · November 23, 2020, 6:25am

Additional content:

It seems that the R&D team have noticed this problem.
Just now I downloaded the latest version of “hopsworks-installer.sh” and tried to execute it again, I found a new option:

Is the host running this installer behind a http proxy (y/n)? y
Enter the URL of the HTTP(S) PROXY: http://127.0.0.1:8118

Finally, I found that the original problem has disappeared, but there is still a new ERROR:

se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Cookbook-dependency ‘livy’ doesn’t refer to a valid url in Berksfile

I hope the R&D team will pay attention to this question.

BTW, I strongly suggest that the R&D team will be able to provide the installation method of docker, because Chinese users have to access GitHub through a proxy software, sometimes it’s a big problem.

moritzmeister · November 24, 2020, 8:59am

Hi,

what version of Hopsworks are you trying to install?

I tried to reproduce your issue, but Single Node Hopsworks installation using hopsworks-installer.sh is successful.

Please make sure to install from the latest version (2.0) with

wget https://raw.githubusercontent.com/logicalclocks/karamel-chef/2.0/hopsworks-installer.sh
chmod +x hopsworks-installer.sh
./hopsworks-installer.sh

Freeman · November 26, 2020, 8:35am

Hi Moritzmeister,
Thank you for your reply. Yes, the version is 2.0.
I did a test in the same way as you said. I found that original error message have disappeared, but there is a new problem:

INFO [2020-11-26 08:27:55,296] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2020-11-26 08:27:55,306] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2020-11-26 08:27:55,316] se.kth.karamel.backend.machines.SshMachine: 10.12.9.220: Yey!! connected ^-^
ERROR [2020-11-26 08:27:55,363] se.kth.karamel.backend.machines.MachinesMonitor:
! net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
! at net.schmizz.sshj.SSHClient.auth(SSHClient.java:225) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:340) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:359) ~[sshj-0.20.0.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:491) ~[karamel-core-0.6.jar:na]
! … 7 common frames omitted
! Causing: se.kth.karamel.common.exception.KaramelException: 10.12.9.220: Authentication problem using ssh keys.
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:523) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.ping(SshMachine.java:549) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.MachinesMonitor.run(MachinesMonitor.java:101) ~[karamel-core-0.6.jar:na]
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
! at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

The installation script continues to report this error, do you know how to fix it?
I’m looking forward to your reply, thanks a lot.

moritzmeister · November 26, 2020, 9:26am

Hi,

Does the user you use for installation have sudo rights? Also can you check if it is able to ssh into localhost?

The error is indicating that karamel is failing to SSH into localhost. It does that for running the setup. Can you double-check that ssh for that user is set up passwordless and localhost was added to trusted host by sshing into localhost with that user?

Alternatively, if your sudo account needs a sudo password, there should be an argument for that in hopsworks-installer.sh:
-pwd [PASSWORD] that you can pass to set the sudo password.

Let me know if that works.

Freeman · November 30, 2020, 1:00pm

Hi Moritzmeister,
Sorry for my late reply.

I have checked the permissions of the user(appadm), which is used to install Hopsworks, I confirm that the user has sudo rights.

And I executed the following commands:

 $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 
 $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Now this user is able to ssh into localhost without the password.

 [appadm@bogon ~]$ ssh localhost
 Last login: Mon Nov 30 20:45:26 2020 from 10.254.26.61

Then I tried to install Hopsworks:

./hopsworks-installer.sh

The installation script reported the following exception:

ERROR [2020-11-30 09:27:39,709] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, elastic/opendistro_security/epipe/username, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, elastic/opendistro_security/admin/username, hopsworks/kagent_liveness/threshold, elastic/opendistro_security/audit/enable_transport, hopsworks/requests_verify, elastic/opendistro_security/elastic_exporter/username, elastic/opendistro_security/audit/enable_rest, hops/yarn/pcores-vcores-multiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, elastic/opendistro_security/logstash/password, hops/yarn/system-reserved-memory-mb, hopsworks/encryption_password, elastic/opendistro_security/epipe/password, alertmanager/email/to, elastic/opendistro_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, elastic/opendistro_security/kibana/password, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opendistro_security/kibana/username, hive2/mysql_password, install/kubernetes, hopsworks/kagent_liveness/enabled, hopsworks/https/port, ndb/DataMemory, elastic/opendistro_security/admin/password, install/cloud, elastic/opendistro_security/logstash/username, elastic/opendistro_security/elastic_exporter/password]
INFO [2020-11-30 09:27:39,713] org.eclipse.jetty.server.ServerConnector: Stopped karamel-core@3918c187{HTTP/1.1}{0.0.0.0:9090}
INFO [2020-11-30 09:27:39,714] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@632aa1a3{/admin,null,UNAVAILABLE}
INFO [2020-11-30 09:27:39,714] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@3d9fc57a{/,null,UNAVAILABLE}

I also tried the second method. Because there are some special characters in the sudo password, I used double quotes :

 ./hopsworks-installer.sh -i purge -ni	 
 ./hopsworks-installer.sh -pwd "M7a194n6)"

The log information is as follows:

INFO [2020-11-30 08:52:59,644] se.kth.karamel.backend.machines.SshMachine: 10.12.9.220: connecting …
INFO [2020-11-30 08:52:59,645] net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0.20.0
INFO [2020-11-30 08:52:59,660] net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
INFO [2020-11-30 08:52:59,788] se.kth.karamel.backend.machines.SshMachine: 10.12.9.220: Yey!! connected ^-^
ERROR [2020-11-30 08:52:59,841] se.kth.karamel.backend.machines.MachinesMonitor:
! net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
! at net.schmizz.sshj.SSHClient.auth(SSHClient.java:225) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:340) ~[sshj-0.20.0.jar:na]
! at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:359) ~[sshj-0.20.0.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:491) ~[karamel-core-0.6.jar:na]
! … 7 common frames omitted
! Causing: se.kth.karamel.common.exception.KaramelException: 10.12.9.220: Authentication problem using ssh keys.
! at se.kth.karamel.backend.machines.SshMachine.connect(SshMachine.java:523) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.SshMachine.ping(SshMachine.java:549) ~[karamel-core-0.6.jar:na]
! at se.kth.karamel.backend.machines.MachinesMonitor.run(MachinesMonitor.java:101) ~[karamel-core-0.6.jar:na]
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
! at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
INFO [2020-11-30 08:53:04,625] se.kth.karamel.backend.machines.MachinesMonitor: Sending pause signal to all machines
ERROR [2020-11-30 08:54:59,660] net.schmizz.sshj.transport.TransportImpl: Dying because - {}
! net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
! at net.schmizz.sshj.transport.Reader.run(Reader.java:57) ~[sshj-0.20.0.jar:na]
INFO [2020-11-30 08:54:59,660] net.schmizz.sshj.transport.TransportImpl: Disconnected - UNKNOWN
INFO [2020-11-30 08:55:09,843] net.schmizz.sshj.transport.random.BouncyCastleRandom: Generating random seed from SecureRandom.
INFO [2020-11-30 08:55:09,848] se.kth.karamel.backend.machines.SshMachine: 10.12.9.220: connecting …

Did I do anything wrong? I am looking forward to your reply.

moritzmeister · December 1, 2020, 9:50am

Hey!

I can’t see you doing anything wrong now. Which Java version is installed on the machine?

Freeman · December 3, 2020, 5:19am

Hi Moritzmeister,
Java version is 1.8.0.

[appadm@bogon software]$ java -version
java version “1.8.0_231”
Java™ SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot™ 64-Bit Server VM (build 25.231-b11, mixed mode)

It seems that the installation script has been executed, but there are still some mistakes.
Because the log is long, I only extracted some ERROR messages:

if [ $RES -eq 0 ] ; then
echo ‘install chefdk’ >> succeed_list
fi
exit $RES
’ > install-chefdk.sh ; chmod +x install-chefdk.sh ; ./install-chefdk.sh
', DAG is stuck here
ERROR [2020-12-03 02:53:03,769] se.kth.karamel.backend.machines.SshMachine: Start log for Failed: ‘install chefdk’ ‘10.12.9.220’
ERROR [2020-12-03 02:53:03,772] se.kth.karamel.backend.machines.SshMachine: Loaded plugins: fastestmirror, langpacks
@ @ @ @

–2020-12-03 10:52:02-- https://hopsworks-distribution.s3-eu-west-1.amazonaws.com/chefdk-3.7.23-1.el7.x86_64.rpm
Connecting to 127.0.0.1:8118… connected.
Proxy request sent, awaiting response… 200 OK
Length: 89857896 (86M) [binary/octet-stream]
Saving to: ‘chefdk-3.7.23-1.el7.x86_64.rpm’
 0K .......... .......... .......... .......... ..........  0%  104K 14m1s
50K .......... .......... .......... .......... ..........  0%  668K 8m6s

87700K … … … … … 99% 12.1M 0s
87750K . 100% 3532G=40s

2020-12-03 10:52:44 (2.14 MB/s) - ‘chefdk-3.7.23-1.el7.x86_64.rpm’ saved [89857896/89857896]

sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
… … … 51% 4.89M 15s
45150K … … … … … 51% 4.60M 15s

85050K … … … …
ERROR [2020-12-03 02:53:03,772] se.kth.karamel.backend.machines.SshMachine: -------------------------------------------------------------------------------

ERROR [2020-12-03 02:53:03,772] se.kth.karamel.backend.machines.SshMachine: End Log for Failed: ‘install chefdk’ ‘10.12.9.220’
ERROR [2020-12-03 02:53:03,772] se.kth.karamel.backend.machines.SshMachine: -------------------------------------------------------------------------------

INFO [2020-12-03 02:53:07,720] se.kth.karamel.backend.machines.MachinesMonitor: Sending pause signal to all machines

I still used the latest version(2.0) to test. BTW, I want to know if 2.0 version is stable?
In comparison, we prefer to 1.4 version, because it’s an official release version. In other words, we hope that R&D team can analyze or fix this problem based on a stable version.

Thank you for your reply, thank you very much.

Theo · December 3, 2020, 10:17am

Hi,

2.0 version is stable, this issue does not seem related to a particular Hopsworks version.

Can you try to install chefdk-3.7.23 by yourself using the hopsworks installation user?

Freeman · December 10, 2020, 5:33am

If I have to install the Hopsworks manually, I am wondering if the R&D team could provide an installation document ?
You know the Hopsworks is complicated, it used a lot of open source frameworks. I probably need a manual to know how to install it step by step.

Freeman · January 2, 2021, 12:33pm

We have known the cause of this issue, since the domain name (raw.githubusercontent.com) has been contaminated by the ISP.
Thank you for your help.

111 · July 7, 2021, 11:59am

I met the same problem, have you fixed that issue?

Freeman · August 25, 2021, 5:19am

Hi @111 ,
Sorry for the late reply.
The domain name “raw.githubusercontent.com” has been contaminated by Chinese ISPs, you have to configure its real IP address in the host file.

hochan19960320 · May 8, 2023, 1:39am

终于找到了一个中国同胞，能留下个联系方式？有些hopswork的问题询问下你