Hops:ndb Error while installing Azure VM Single host Installation: Access Denied Error

Hello,
I am trying to install hopsworks on an Azure VM (Single node installation). I believe the forward and reverse DNS is working well with instance (by checking dig commands).
I got stuck here.

INFO [2020-06-30 09:15:09,614] se.kth.karamel.backend.machines.SshMachine: x.x.x.x: Running task: hops::ndb
INFO [2020-06-30 09:15:29,738] se.kth.karamel.backend.machines.SshMachine: x.x.x.x: Running task: hops::ndb
ERROR [2020-06-30 09:15:50,933] se.kth.karamel.backend.dag.DagNode: Failed ‘hops::ndb on x.x.x.x’ because 'x.x.x.x: Command did not complete: mkdir -p /home/fmlstream/.karamel/install ; cd /home/fmlstream/.karamel/install; echo $$ > pid; echo '#!/bin/bash

When I opened and checked the .karamel/install/hops__ndb.log, below is the stacktrace

* bash[mysql-install-hops] action run
  - execute "bash"  "/tmp/chef-script20200630-125548-n5ma90"
* template[/srv/hops/ndb-hops/flyway.sql] action create (up to date)
* bash[flyway_baseline] action run (skipped due to not_if)
* bash[flyway_migrate] action run

  ================================================================================
  Error executing action `run` on resource 'bash[flyway_migrate]'
  ================================================================================

  Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Expected process to exit with [0], but received '1'
  ---- Begin output of "bash"  "/tmp/chef-script20200630-125548-kxp6x1" ----
  STDOUT: Flyway Community Edition 5.0.3 by Boxfuse
  STDERR: ERROR:
  Unable to obtain connection from database (jdbc:mysql://x.x.x.x:3306/hops) for user 'kthfs': Access denied for user 'kthfs'@'h1.internal.cloudapp.net' (using password: YES)
  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  SQL State  : 28000
  Error Code : 1045
  Message    : Access denied for user 'kthfs'@'h1.internal.cloudapp.net' (using password: YES)
  ---- End output of "bash"  "/tmp/chef-script20200630-125548-kxp6x1" ----
  Ran "bash"  "/tmp/chef-script20200630-125548-kxp6x1" returned 1

  Resource Declaration:
  ---------------------
  # In /tmp/chef-solo/cookbooks/hops/providers/ndb.rb

   44:   bash "flyway_migrate" do
   45:     user "root"
   46:     timeout 36000
   47:     code <<-EOF
   48:     set -e
   49:     cd #{flyway_dir}
   50:     #{flyway_dir}/flyway migrate
   51:   EOF
   52:   end
   53:

  Compiled Resource:
  ------------------
  # Declared in /tmp/chef-solo/cookbooks/hops/providers/ndb.rb:44:in `block in class_from_file'

  bash("flyway_migrate") do
    action [:run]
    default_guard_interpreter :default
    command nil
    backup 5
    interpreter "bash"
    declared_type :bash
    cookbook_name "hops"
    user "root"
    code "    set -e\n    cd /srv/hops/ndb-hops/flyway\n    /srv/hops/ndb-hops/flyway/flyway migrate\n"
    domain nil
    timeout 36000
  end

  System Info:
  ------------
  chef_version=14.10.9
  platform=redhat
  platform_version=7.8
  ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
  program_name=/bin/chef-solo
  executable=/opt/chefdk/bin/chef-solo


================================================================================
Error executing action `install_hops` on resource 'hops_ndb[install]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash[flyway_migrate] (/tmp/chef-solo/cookbooks/hops/providers/ndb.rb line 44) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200630-125548-kxp6x1" ----
STDOUT: Flyway Community Edition 5.0.3 by Boxfuse
STDERR: ERROR:
Unable to obtain connection from database (jdbc:mysql://x.x.x.x:3306/hops) for user 'kthfs': Access denied for user 'kthfs'@'h1.internal.cloudapp.net' (using password: YES)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State  : 28000
Error Code : 1045
Message    : Access denied for user 'kthfs'@'h1.internal.cloudapp.net' (using password: YES)
---- End output of "bash"  "/tmp/chef-script20200630-125548-kxp6x1" ----
Ran "bash"  "/tmp/chef-script20200630-125548-kxp6x1" returned 1

Resource Declaration:
---------------------
# In /tmp/chef-solo/cookbooks/hops/recipes/ndb.rb

144:   hops_ndb "install" do
145:     action :install_hops
146:   end
147: end

Compiled Resource:
------------------
# Declared in /tmp/chef-solo/cookbooks/hops/recipes/ndb.rb:144:in `from_file'

hops_ndb("install") do
  action [:install_hops]
  updated true
  updated_by_last_action true
  default_guard_interpreter :default
  declared_type :hops_ndb
  cookbook_name "hops"
  recipe_name "ndb"
end

System Info:
------------
chef_version=14.10.9
platform=redhat
platform_version=7.8
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/bin/chef-solo
executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2020-06-30T14:27:11+00:00] ERROR: Running exception handlers
Running handlers complete
[2020-06-30T14:27:11+00:00] ERROR: Exception handlers complete
Chef Client failed. 14 resources updated in 07 seconds
[2020-06-30T14:27:11+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-06-30T14:27:11+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-06-30T14:27:11+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: hops_ndb[install] (hops::ndb line 144) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[flyway_migrate] (/tmp/chef-solo/cookbooks/hops/providers/ndb.rb line 44) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20200630-125548-kxp6x1” ----
STDOUT: Flyway Community Edition 5.0.3 by Boxfuse
STDERR: ERROR:
Unable to obtain connection from database (jdbc:mysql://x.x.x.x:3306/hops) for user ‘kthfs’: Access denied for user ‘kthfs’@‘h1.internal.cloudapp.net’ (using password: YES)

SQL State : 28000
Error Code : 1045
Message : Access denied for user ‘kthfs’@‘h1.internal.cloudapp.net’ (using password: YES)
---- End output of “bash” “/tmp/chef-script20200630-125548-kxp6x1” ----
Ran “bash” “/tmp/chef-script20200630-125548-kxp6x1” returned 1

What might be the error while connecting to mysql?
Is this “Access Denied” error related to DNS issues in Azure or anything else?

Hi,

To be honest I’m not entirely sure what’s going on here. In Hopsworks 1.3 we still have some wide grants for the kthfs user which should grant privileges on % - which in MySQL world means “from everywhere”. So it should accept the location.

Did you try to run the hopsworks-installer.sh script twice? If so, did you run the purge command in between?

./hopswork-installer.sh -i purge-all -ni


Fabio