Hello,
I keep getting the following error:
[2021-01-12T18:29:08+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-01-12T18:29:08+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-01-12T18:29:08+00:00] FATAL: RuntimeError: ruby_block[check_db_empty] (hopsworks::default line 201) had an error: RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute
I have tried purging and re-installing and even starting with a fresh VM. I am using this installer:
https://raw.githubusercontent.com/logicalclocks/karamel-chef/2.0/hopsworks-installer.sh
NOTE: I have tried using https://raw.githubusercontent.com/logicalclocks/karamel-chef/master/hopsworks-installer.sh as well with he same result.
Any ideas ?
UPDATE: I ran the installtiona again after purging and now I cant get this error:
Chef Client failed. 7 resources updated in 02 minutes 11 secondse[0m
[2021-01-12T20:05:22+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-01-12T20:05:22+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-01-12T20:05:22+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[unpack_flyway] (hopsworks::install line 807) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20210112-32117-1navgip" ----
STDOUT:
STDERR: mv: inter-device move failed: 'flyway-6.5.1' to '/srv/hops/domains/domain1/flyway-6.5.1'; unable to remove target: Directory not empty
---- End output of "bash" "/tmp/chef-script20210112-32117-1navgip" ----
Ran "bash" "/tmp/chef-script20210112-32117-1navgip" returned
Hi.
The second message shows that you didn’t clean the installation directory, if you didn’t change it it’s /srv/hops Delete it and also delete .karamel directory in the home of the user performing the installation.
If the error appears again look at $HOME/.karamel/hopsworksvagrantsinglenode/logs/YOUR_IP for the failed recipe. Our orchestration framework will retry a failed recipe and the aggregated logs are there.
Also, please use the release version of the installer and not master as the later can be quite unstable.
Kind regards,
Hello,
I am now receiving the first error constantly:
RuntimeError: ruby_block[check_db_empty] (hopsworks::default line 201) had an error: RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute
/tmp/chef-solo/cookbooks/hopsworks/recipes/default.rb:203:in `block (2 levels) in from_file'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/ruby_block.rb:31:in `block in action_run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:227:in `converge_by'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/ruby_block.rb:30:in `action_run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:182:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource.rb:578:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:70:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `each'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:97:in `converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:720:in `block in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `catch'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:754:in `converge_and_save'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:286:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:303:in `run_with_graceful_exit_option'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:279:in `block in run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:261:in `run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:444:in `run_application'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:66:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/solo.rb:224:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/bin/chef-solo:24:in `<top (required)>'
/usr/bin/chef-solo:306:in `load'
/usr/bin/chef-solo:306:in `<main>'
This part of the code initializes the database. We have several guards protecting from data loss. Since it’s the first time you install Hopsworks - it’s not an upgrade - it’s safe to delete the database.
Execute /srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh -e "DELETE schema hopsworks" and retry again.
I still receive the same error after deleting the DB
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/hopsworks/recipes/default.rb:201:in `from_file'
e[0m
e[0m ruby_block("check_db_empty") do
e[0m action [:run]
e[0m default_guard_interpreter :default
e[0m declared_type :ruby_block
e[0m cookbook_name "hopsworks"
e[0m recipe_name "default"
e[0m block #<Proc:0x0000000006497d38@/tmp/chef-solo/cookbooks/hopsworks/recipes/default.rb:202>
e[0m block_name "check_db_empty"
e[0m only_if "/srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh hopsworks -e "SHOW TABLES" | grep project"
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=ubuntu
e[0m platform_version=18.04
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/usr/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m
e[0me[0m
Running handlers:e[0m
[2021-01-13T16:01:51+00:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2021-01-13T16:01:51+00:00] ERROR: Exception handlers complete
Chef Client failed. 6 resources updated in 16 secondse[0m
[2021-01-13T16:01:51+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-01-13T16:01:51+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-01-13T16:01:51+00:00] FATAL: RuntimeError: ruby_block[check_db_empty] (hopsworks::default line 201) had an error: RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute
I can’t be sure what’s the state of your VM. Something has failed after initializing the database and then it retries the recipe, that’s why you get the error above. It’s an affect, not the actual error.
Is it possible that you start from a fresh new VM? If you get the same error again, can you post the content of $HOME/.karamel/hopsworksvagrantsinglenode/logs/ ?
Hi, I tried with a fresh VM and received the same error. I do not see the log path you mentioned. I do have this path $HOME/.karamel/hops/logs/IP/. This directory has all the log files for the different recipes. Attached is the log file for the failed recipe.
hopsworks_default.log.zip (94.0 KB)
Hi.
I’ve looked at the logs you sent. The root problem is that the application server can’t be reached. Could you please do the following after the recipe has failed to identify the problem?
- Verify the application server is running.
sudo netstat -lpn | grep 443 you should see a service listening on that port. Also sudo jps should list a process ASMain with the same process ID.
-
sudo hostname -A this will give you back a list of hostnames your instance is associated with. For example “my_instance.internal.cloudapp.net my_instance”
- For each one of the hostnames above do
curl --insecure https://HOSTNAME:8182/hopsworks-api/api/variables/versions Note, you should replace the HOSTNAME. If the application server is running and the network setup is correct you should get a list of versions as a response.
Hi again.
Have you followed the instructions here to create a private DNS zone?
Hello, yes the private dns instructions were followed. When running the above commands i get this error:
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to HOSTNAME:8182
A nslookup on the hostname works fine
Hello,
Is there anymore suggestions I can try ? I’m stuck at this issue.
Hi.
The last point of my instructions was to execute curl but replace HOSTNAME with the actual hostname. Also, please give me the output of the second point - hostname -A
I did that. I just didn’t put my actual host name in the response. Still that is the error I got.
Output from sudo hostname -A
hostname.internal.cloudapp.net hostname
NOTE the hostname is the machine name which is 20 characters long
Also, is there an support ticket system or email that we could move too ?
If you are on azure, and you have permissions to create a VM using azure tools, you can use the
hopsworks-cloud-installer.sh . It will make sure you have setup a subscription, created a resource group if it doesn’t exist, the virtual network, the private dns zone, and validated the hostname before launching hopsworks-installer.sh
Otherwise, you can also use hopsworks.ai to launch a managed cluster in Azure.