Upgrading a Hopsworks 2.5 feature store to Hopsworks 3.0 with
./bin/karamel -launch hopsworks-upgrade-3.yml
I got an error
se.kth.karamel.webservice.KaramelServiceApplication:
Inalid yaml file; Invalid attributes,
all used attributes must be defined in metadata.rb files
Any hints are really appreciated.
What I did
I copied the generated cluster-defns/hopsworks-installation.yml file to the haramel-0.6/hopsworks-upgrade-3.yml,
changed the attribute
cookbooks/hopsworks/branch to 3.0
and added the attribute attrs/install/current_version: 2.5
The Karamel console Output is:
INFO [2022-09-15 14:12:13,067] io.dropwizard.server.ServerFactory: Starting karamel-core
INFO [2022-09-15 14:12:13,207] org.eclipse.jetty.setuid.SetUIDListener: Opened karamel-core@150ab4ed{HTTP/1.1}{0.0.0.0:9090}
INFO [2022-09-15 14:12:13,208] org.eclipse.jetty.server.Server: jetty-9.0.7.v20131107
INFO [2022-09-15 14:12:13,277] com.sun.jersey.server.impl.application.WebApplicationImpl: Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM'
INFO [2022-09-15 14:12:13,337] io.dropwizard.jersey.DropwizardResourceConfig: The following paths were found for the configured resources:
PUT /api/definition/yaml2json (se.kth.karamel.webservice.calls.definition.YamlToJson)
PUT /api/definition/json2yaml (se.kth.karamel.webservice.calls.definition.JsonToYaml)
PUT /api/definition/fetchCookbook (se.kth.karamel.webservice.calls.definition.FetchCookbook)
PUT /api/ssh/loadKey (se.kth.karamel.webservice.calls.sshkeys.LoadSshKeys)
PUT /api/ssh/registerKey (se.kth.karamel.webservice.calls.sshkeys.RegisterSshKeys)
PUT /api/ssh/generateKey (se.kth.karamel.webservice.calls.sshkeys.GenerateSshKeys)
PUT /api/ssh/setSudoPassword (se.kth.karamel.webservice.calls.sshkeys.SetSudoPassword)
PUT /api/ec2/loadCredentials (se.kth.karamel.webservice.calls.ec2.LoadEc2Credentials)
PUT /api/ec2/validateCredentials (se.kth.karamel.webservice.calls.ec2.ValidateEc2Credentials)
PUT /api/gce/loadCredentials (se.kth.karamel.webservice.calls.gce.LoadGceCredentials)
PUT /api/gce/validateCredentials (se.kth.karamel.webservice.calls.gce.ValidateGceCredentials)
PUT /api/cluster/start (se.kth.karamel.webservice.calls.cluster.StartCluster)
PUT /api/cluster/processCommand (se.kth.karamel.webservice.calls.cluster.ProcessCommand)
GET /api/system/exit (se.kth.karamel.webservice.calls.system.ExitKaramel)
GET /api/system/ping (se.kth.karamel.webservice.calls.system.PingServer)
GET /api/github/getCredentials (se.kth.karamel.webservice.calls.github.GetGithubCredentials)
POST /api/github/setCredentials (se.kth.karamel.webservice.calls.github.SetGithubCredentials)
POST /api/github/getOrgs (se.kth.karamel.webservice.calls.github.GetGithubOrgs)
POST /api/github/getRepos (se.kth.karamel.webservice.calls.github.GetGithubRepos)
POST /api/github/removeRepository (se.kth.karamel.webservice.calls.github.RemoveRepository)
POST /api/experiment/load (se.kth.karamel.webservice.calls.experiment.LoadExperiment)
PUT /api/experiment/push (se.kth.karamel.webservice.calls.experiment.PushExperiment)
POST /api/experiment/removeFile (se.kth.karamel.webservice.calls.experiment.RemoveFileFromExperiment)
PUT /api/nova/loadCredentials (se.kth.karamel.webservice.calls.nova.LoadNovaCredentials)
PUT /api/nova/validateCredentials (se.kth.karamel.webservice.calls.nova.ValidateNovaCredentials)
PUT /api/occi/loadCredentials (se.kth.karamel.webservice.calls.occi.LoadOcciCredentials)
PUT /api/occi/validateCredentials (se.kth.karamel.webservice.calls.occi.ValidateOcciCredentials)
INFO [2022-09-15 14:12:13,478] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@2c1dc8e{/,null,AVAILABLE}
INFO [2022-09-15 14:12:13,479] io.dropwizard.setup.AdminEnvironment: tasks =
POST /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)
INFO [2022-09-15 14:12:13,484] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@2472c7d8{/admin,null,AVAILABLE}
INFO [2022-09-15 14:12:13,490] org.eclipse.jetty.server.ServerConnector: Started karamel-core@150ab4ed{HTTP/1.1}{0.0.0.0:9090}
INFO [2022-09-15 14:12:15,625] se.kth.karamel.client.api.CookbookCacheIml: 0-level cookbooks for Hops is 1
INFO [2022-09-15 14:12:29,989] se.kth.karamel.client.api.CookbookCacheIml: 1-level cookbooks for Hops is 24
INFO [2022-09-15 14:12:30,851] se.kth.karamel.client.api.CookbookCacheIml: 2-level cookbooks for Hops is 14
ERROR [2022-09-15 14:12:30,857] se.kth.karamel.webservice.KaramelServiceApplication: Inalid yaml file; Invalid attributes, all used attributes must be defined in metadata.rb files: [elastic/opendistro_security/epipe/username, elastic/opendistro_security/admin/username, elastic/opendistro_security/audit/enable_transport, elastic/opendistro_security/elastic_exporter/username, elastic/opendistro_security/audit/enable_rest, elastic/opendistro_security/logstash/password, elastic/opendistro_security/epipe/password, elastic/opendistro_security/jwt/exp_ms, elastic/opendistro_security/kibana/password, elastic/opendistro_security/kibana/username, elastic/opendistro_security/admin/password, elastic/opendistro_security/logstash/username, elastic/opendistro_security/elastic_exporter/password]
INFO [2022-09-15 14:12:30,866] org.eclipse.jetty.server.ServerConnector: Stopped karamel-core@150ab4ed{HTTP/1.1}{0.0.0.0:9090}
INFO [2022-09-15 14:12:30,869] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@2472c7d8{/admin,null,UNAVAILABLE}
INFO [2022-09-15 14:12:30,869] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@2c1dc8e{/,null,UNAVAILABLE}