The installation script reported an error : Can't open cluster-defns/hopsworks-installation.yml

Hopsworks
#1

OS : CentOS 7
Installation Options : Install a single-host Hopsworks cluster.
Where are you installing Hopsworks? : On-premises or private cloud

Details:

-------------------- Where are you installing Hopsworks? --------------------

On what platform are you installing Hopsworks?

(1) On-premises or private cloud.

(2) AWS.

(3) GCP.

(4) Azure.

Please enter your choice 1, 2, 3, 4 : 1

Press ENTER to continue
cp: cannot stat ‘cluster-defns/hopsworks-head.yml’: No such file or directory
Installing Karamel…
–2020-11-10 13:36:09-- http://www.karamel.io/sites/default/files/downloads/karamel-0.6.tgz
Resolving www.karamel.io (www.karamel.io)… 193.10.67.171
Connecting to www.karamel.io (www.karamel.io)|193.10.67.171|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 36943369 (35M) [application/x-gzip]
Saving to: ‘karamel-0.6.tgz’

100%[================================================================================================================================================================================>] 36,943,369 1.48MB/s in 21s

2020-11-10 13:36:30 (1.66 MB/s) - ‘karamel-0.6.tgz’ saved [36943369/36943369]

Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Can’t open cluster-defns/hopsworks-installation.yml: No such file or directory.
Running command from /home/appadm/hopsworks/karamel-0.6:

setsid ./bin/karamel -headless -launch …/cluster-defns/hopsworks-installation.yml > …/installation.log 2>&1 &

Installation has started, but may take 1 hour or more…

I live in China, I wonder if it’s because the download requests have been blocked by China’s firewall.
It’s my first time to install the Hopsworks, please help me.

#2

Hi Freeman,
The installation takes around 1 hours 50 minutes on a vm. Can you look in the log file if you have any errors.

/Ermias

#3

Hi Ermias,
The exception information is as follows:

INFO [2020-11-10 05:36:34,764] org.eclipse.jetty.server.ServerConnector: Started karamel-core@3918c187{HTTP/1.1}{0.0.0.0:9090}
Exception in thread “main” java.nio.file.NoSuchFileException: …/cluster-defns/hopsworks-installation.yml
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at se.kth.karamel.common.CookbookScaffolder.readFile(CookbookScaffolder.java:88)
at se.kth.karamel.webservice.KaramelServiceApplication.main(KaramelServiceApplication.java:254)

In the cluster-defns folder, there is no hopsworks-installation.yml. I don’t know what causes this file has not been downloaded.

#4

I’ve found the cause of the problem，the reason is that the download requests are blocked by China’s firewall.
Now I’ve solved the firewall problem, but there’s another issue, the log information is as follows :

INFO [2020-11-19 09:24:40,446] io.dropwizard.server.ServerFactory: Starting karamel-core
INFO [2020-11-19 09:24:40,494] org.eclipse.jetty.setuid.SetUIDListener: Opened karamel-core@3918c187{HTTP/1.1}{0.0.0.0:9090}
INFO [2020-11-19 09:24:40,496] org.eclipse.jetty.server.Server: jetty-9.0.7.v20131107
INFO [2020-11-19 09:24:40,606] com.sun.jersey.server.impl.application.WebApplicationImpl: Initiating Jersey application, version ‘Jersey: 1.18.1 02/19/2014 03:28 AM’
INFO [2020-11-19 09:24:40,677] io.dropwizard.jersey.DropwizardResourceConfig: The following paths were found for the configured resources:

PUT     /api/definition/yaml2json (se.kth.karamel.webservice.calls.definition.YamlToJson)
PUT     /api/definition/json2yaml (se.kth.karamel.webservice.calls.definition.JsonToYaml)
PUT     /api/definition/fetchCookbook (se.kth.karamel.webservice.calls.definition.FetchCookbook)
PUT     /api/ssh/loadKey (se.kth.karamel.webservice.calls.sshkeys.LoadSshKeys)
PUT     /api/ssh/registerKey (se.kth.karamel.webservice.calls.sshkeys.RegisterSshKeys)
PUT     /api/ssh/generateKey (se.kth.karamel.webservice.calls.sshkeys.GenerateSshKeys)
PUT     /api/ssh/setSudoPassword (se.kth.karamel.webservice.calls.sshkeys.SetSudoPassword)
PUT     /api/ec2/loadCredentials (se.kth.karamel.webservice.calls.ec2.LoadEc2Credentials)
PUT     /api/ec2/validateCredentials (se.kth.karamel.webservice.calls.ec2.ValidateEc2Credentials)
PUT     /api/gce/loadCredentials (se.kth.karamel.webservice.calls.gce.LoadGceCredentials)
PUT     /api/gce/validateCredentials (se.kth.karamel.webservice.calls.gce.ValidateGceCredentials)
PUT     /api/cluster/start (se.kth.karamel.webservice.calls.cluster.StartCluster)
PUT     /api/cluster/processCommand (se.kth.karamel.webservice.calls.cluster.ProcessCommand)
GET     /api/system/exit (se.kth.karamel.webservice.calls.system.ExitKaramel)
GET     /api/system/ping (se.kth.karamel.webservice.calls.system.PingServer)
GET     /api/github/getCredentials (se.kth.karamel.webservice.calls.github.GetGithubCredentials)
POST    /api/github/setCredentials (se.kth.karamel.webservice.calls.github.SetGithubCredentials)
POST    /api/github/getOrgs (se.kth.karamel.webservice.calls.github.GetGithubOrgs)
POST    /api/github/getRepos (se.kth.karamel.webservice.calls.github.GetGithubRepos)
POST    /api/github/removeRepository (se.kth.karamel.webservice.calls.github.RemoveRepository)
POST    /api/experiment/load (se.kth.karamel.webservice.calls.experiment.LoadExperiment)
PUT     /api/experiment/push (se.kth.karamel.webservice.calls.experiment.PushExperiment)
POST    /api/experiment/removeFile (se.kth.karamel.webservice.calls.experiment.RemoveFileFromExperiment)
PUT     /api/nova/loadCredentials (se.kth.karamel.webservice.calls.nova.LoadNovaCredentials)
PUT     /api/nova/validateCredentials (se.kth.karamel.webservice.calls.nova.ValidateNovaCredentials)
PUT     /api/occi/loadCredentials (se.kth.karamel.webservice.calls.occi.LoadOcciCredentials)
PUT     /api/occi/validateCredentials (se.kth.karamel.webservice.calls.occi.ValidateOcciCredentials)

INFO [2020-11-19 09:24:40,830] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@3d9fc57a{/,null,AVAILABLE}
INFO [2020-11-19 09:24:40,831] io.dropwizard.setup.AdminEnvironment: tasks =

POST    /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)

INFO [2020-11-19 09:24:40,835] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@632aa1a3{/admin,null,AVAILABLE}
INFO [2020-11-19 09:24:40,840] org.eclipse.jetty.server.ServerConnector: Started karamel-core@3918c187{HTTP/1.1}{0.0.0.0:9090}
INFO [2020-11-19 09:24:42,938] se.kth.karamel.client.api.CookbookCacheIml: 0-level cookbooks for Hops is 1
ERROR [2020-11-19 09:24:43,351] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, elastic/opendistro_security/epipe/username, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, elastic/opendistro_security/admin/username, hopsworks/kagent_liveness/threshold, elastic/opendistro_security/audit/enable_transport, hopsworks/requests_verify, elastic/opendistro_security/elastic_exporter/username, elastic/opendistro_security/audit/enable_rest, hops/yarn/pcores-vcores-multiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, elastic/opendistro_security/logstash/password, hops/yarn/system-reserved-memory-mb, hopsworks/encryption_password, elastic/opendistro_security/epipe/password, alertmanager/email/to, elastic/opendistro_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, elastic/opendistro_security/kibana/password, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opendistro_security/kibana/username, hive2/mysql_password, install/kubernetes, hopsworks/kagent_liveness/enabled, hopsworks/https/port, ndb/DataMemory, elastic/opendistro_security/admin/password, install/cloud, elastic/opendistro_security/logstash/username, elastic/opendistro_security/elastic_exporter/password]
INFO [2020-11-19 09:24:43,355] org.eclipse.jetty.server.ServerConnector: Stopped karamel-core@3918c187{HTTP/1.1}{0.0.0.0:9090}
INFO [2020-11-19 09:24:43,355] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@632aa1a3{/admin,null,UNAVAILABLE}
INFO [2020-11-19 09:24:43,356] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@3d9fc57a{/,null,UNAVAILABLE}

The key content is :

ERROR [2020-11-19 09:24:43,351] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [hops/tls/enabled, elastic/opendistro_security/epipe/username, hops/rmappsecurity/actor_class, hopsworks/application_certificate_validity_period, elastic/opendistro_security/admin/username, hopsworks/kagent_liveness/threshold, elastic/opendistro_security/audit/enable_transport, hopsworks/requests_verify, elastic/opendistro_security/elastic_exporter/username, elastic/opendistro_security/audit/enable_rest, hops/yarn/pcores-vcores-multiplier, mysql/password, hops/yarn/cgroups_strict_resource_usage, hopsworks/featurestore_online, hopsworks/admin/password, hops/yarn/detect-hardware-capabilities, hopsworks/admin/user, elastic/opendistro_security/logstash/password, hops/yarn/system-reserved-memory-mb, hopsworks/encryption_password, elastic/opendistro_security/epipe/password, alertmanager/email/to, elastic/opendistro_security/jwt/exp_ms, install/dir, prometheus/retention_time, alertmanager/email/smtp_host, elastic/opendistro_security/kibana/password, alertmanager/email/from, hopsworks/master/password, ndb/NoOfReplicas, elastic/opendistro_security/kibana/username, hive2/mysql_password, install/kubernetes, hopsworks/kagent_liveness/enabled, hopsworks/https/port, ndb/DataMemory, elastic/opendistro_security/admin/password, install/cloud, elastic/opendistro_security/logstash/username, elastic/opendistro_security/elastic_exporter/password]

What causes this?

#5

On this issue, I created a new topic to discuss: