Error in hopsworks-installer on Azure VM

Hello,

I keep getting the following error:

[2021-01-12T18:29:08+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-01-12T18:29:08+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-01-12T18:29:08+00:00] FATAL: RuntimeError: ruby_block[check_db_empty] (hopsworks::default line 201) had an error: RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute

I have tried purging and re-installing and even starting with a fresh VM. I am using this installer:
https://raw.githubusercontent.com/logicalclocks/karamel-chef/2.0/hopsworks-installer.sh

NOTE: I have tried using https://raw.githubusercontent.com/logicalclocks/karamel-chef/master/hopsworks-installer.sh as well with he same result.

Any ideas ?

UPDATE: I ran the installtiona again after purging and now I cant get this error:

Chef Client failed. 7 resources updated in 02 minutes 11 secondse[0m
[2021-01-12T20:05:22+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-01-12T20:05:22+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-01-12T20:05:22+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[unpack_flyway] (hopsworks::install line 807) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210112-32117-1navgip" ----
STDOUT: 
STDERR: mv: inter-device move failed: 'flyway-6.5.1' to '/srv/hops/domains/domain1/flyway-6.5.1'; unable to remove target: Directory not empty
---- End output of "bash"  "/tmp/chef-script20210112-32117-1navgip" ----
Ran "bash"  "/tmp/chef-script20210112-32117-1navgip" returned

Hi.

The second message shows that you didn’t clean the installation directory, if you didn’t change it it’s /srv/hops Delete it and also delete .karamel directory in the home of the user performing the installation.

If the error appears again look at $HOME/.karamel/hopsworksvagrantsinglenode/logs/YOUR_IP for the failed recipe. Our orchestration framework will retry a failed recipe and the aggregated logs are there.

Also, please use the release version of the installer and not master as the later can be quite unstable.

Kind regards,

Hello,

I am now receiving the first error constantly:

RuntimeError: ruby_block[check_db_empty] (hopsworks::default line 201) had an error: RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute
/tmp/chef-solo/cookbooks/hopsworks/recipes/default.rb:203:in `block (2 levels) in from_file'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/ruby_block.rb:31:in `block in action_run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:227:in `converge_by'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/ruby_block.rb:30:in `action_run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:182:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource.rb:578:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:70:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `each'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:97:in `converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:720:in `block in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `catch'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `converge'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:754:in `converge_and_save'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:286:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:303:in `run_with_graceful_exit_option'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:279:in `block in run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:261:in `run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:444:in `run_application'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:66:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/solo.rb:224:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/bin/chef-solo:24:in `<top (required)>'
/usr/bin/chef-solo:306:in `load'
/usr/bin/chef-solo:306:in `<main>'

This part of the code initializes the database. We have several guards protecting from data loss. Since it’s the first time you install Hopsworks - it’s not an upgrade - it’s safe to delete the database.

Execute /srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh -e "DELETE schema hopsworks" and retry again.

I still receive the same error after deleting the DB

e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /tmp/chef-solo/cookbooks/hopsworks/recipes/default.rb:201:in `from_file'
e[0m    
e[0m    ruby_block("check_db_empty") do
e[0m      action [:run]
e[0m      default_guard_interpreter :default
e[0m      declared_type :ruby_block
e[0m      cookbook_name "hopsworks"
e[0m      recipe_name "default"
e[0m      block #<Proc:0x0000000006497d38@/tmp/chef-solo/cookbooks/hopsworks/recipes/default.rb:202>
e[0m      block_name "check_db_empty"
e[0m      only_if "/srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh hopsworks -e "SHOW TABLES" | grep project"
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=14.10.9
e[0m    platform=ubuntu
e[0m    platform_version=18.04
e[0m    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m    program_name=/usr/bin/chef-solo
e[0m    executable=/opt/chefdk/bin/chef-soloe[0m
    
e[0me[0m
Running handlers:e[0m
[2021-01-13T16:01:51+00:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2021-01-13T16:01:51+00:00] ERROR: Exception handlers complete
Chef Client failed. 6 resources updated in 16 secondse[0m
[2021-01-13T16:01:51+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-01-13T16:01:51+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-01-13T16:01:51+00:00] FATAL: RuntimeError: ruby_block[check_db_empty] (hopsworks::default line 201) had an error: RuntimeError: You are trying to initialize the database, but the database is not empty. Either there is a failed migration, or you forgot to set the current_version attribute

I can’t be sure what’s the state of your VM. Something has failed after initializing the database and then it retries the recipe, that’s why you get the error above. It’s an affect, not the actual error.

Is it possible that you start from a fresh new VM? If you get the same error again, can you post the content of $HOME/.karamel/hopsworksvagrantsinglenode/logs/ ?

Hi, I tried with a fresh VM and received the same error. I do not see the log path you mentioned. I do have this path $HOME/.karamel/hops/logs/IP/. This directory has all the log files for the different recipes. Attached is the log file for the failed recipe.

hopsworks_default.log.zip (94.0 KB)