I tried to install with the following approch for an on-prem, single server on an Ubuntu 20.04.3 LTS VM:
wget https://raw.githubusercontent.com/logicalclocks/karamel-chef/master/hopsworks-installer.sh
chmod +x hopsworks-installer.sh
./hopsworks-installer.sh
I got an error:
ERROR [2021-11-11 18:23:20,744] se.kth.karamel.backend.dag.DagNode: Failed 'ndb::mysqld on 192.168.129.145' because '192.168.129.145: Command
did not complete: mkdir -p /home/ubuntu/.karamel/install ; cd /home/ubuntu/.karamel/install; echo $$ > pid; echo '#!/bin/bash
set -eo pipefail
echo $(date '+%H:%M:%S'): 'ndb__mysqld' >> order
cat > ndb__mysqld.json <<-'END_OF_FILE'
{
"hopsmonitor": {
"default": {
"private_ips": [
"192.168.129.145"
],
"public_ips": [
"192.168.129.145"
],
"private_ips_domainIds": {
"192.168.129.145": "0"
},
"hosts": {
"192.168.129.145": "192.168.129.145"
}
},
...
sudo chef-solo -c /home/ubuntu/.karamel/install/solo.rb -j /home/ubuntu/.karamel/install/ndb__mysqld.json 2>&1 | tee ndb__mysqld.log
echo 'https://github.com/logicalclocks/ndb-chef/tree/master/ndb::mysqld' >> succeed_list
' > ndb__mysqld.sh ; chmod +x ndb__mysqld.sh ; ./ndb__mysqld.sh
', DAG is stuck here :(
What does this mean? Karamel have already run more than 6 hours now. Does anyone has the same error? How can I stop the installation or better how can I fix this error?
Really appreciate your advice.
Here is some more detailed error.
e[0m * ndb_mysql_basic[create_users_grants] action install_grants
* ndb_waiter[wait_ndb_started] action wait_until_cluster_ready
* bash[wait_ndb_started] action run
e[0m
================================================================================e[0m
e[31mError executing action `run` on resource 'bash[wait_ndb_started]'e[0m
================================================================================e[0m
e[0m Mixlib::ShellOut::ShellCommandFailede[0m
------------------------------------e[0m
Expected process to exit with [0], but received '1'
e[0m ---- Begin output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
e[0m STDOUT: Connection to 192.168.129.198:1186 failed
e[0m STDERR:
e[0m ---- End output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
e[0m Ran "bash" "/tmp/chef-script20211112-38555-noekpf" returned 1e[0m
e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/ndb/providers/waiter.rb
e[0m
e[0m 3: bash new_resource.name do
e[0m 4: user node['ndb']['user']
e[0m 5: code <<-EOF
e[0m 6: #{node['mysql']['base_dir']}/bin/ndb_waiter -c #{node['ndb']['connectstring']} --timeout=#{node['ndb']['wait_startup']} 2>&1 > /dev/null
e[0m 7: EOF
e[0m 8: end
e[0m 9: new_resource.updated_by_last_action(true)
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/ndb/providers/waiter.rb:3:in `block in class_from_file'
e[0m
e[0m bash("wait_ndb_started") do
e[0m action [:run]
e[0m default_guard_interpreter :default
e[0m command nil
e[0m backup 5
e[0m interpreter "bash"
e[0m declared_type :bash
e[0m cookbook_name "ndb"
e[0m user "mysql"
e[0m code " /srv/hops/mysql/bin/ndb_waiter -c 192.168.129.198:1186 --timeout=10800 2>&1 > /dev/null\n"
e[0m domain nil
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=ubuntu
e[0m platform_version=20.04
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/usr/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m
e[0m e[0m
================================================================================e[0m
e[31mError executing action `wait_until_cluster_ready` on resource 'ndb_waiter[wait_ndb_started]'e[0m
================================================================================e[0m
e[0m Mixlib::ShellOut::ShellCommandFailede[0m
------------------------------------e[0m
bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
e[0m ---- Begin output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
e[0m STDOUT: Connection to 192.168.129.198:1186 failed
e[0m STDERR:
e[0m ---- End output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
e[0m Ran "bash" "/tmp/chef-script20211112-38555-noekpf" returned 1e[0m
e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb
e[0m
e[0m 3: ndb_waiter "wait_ndb_started" do
e[0m 4: action :wait_until_cluster_ready
e[0m 5: only_if { node['ndb']['enabled'] == "true" }
e[0m 6: end
e[0m 7:
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb:3:in `block in class_from_file'
e[0m
e[0m ndb_waiter("wait_ndb_started") do
e[0m action [:wait_until_cluster_ready]
e[0m updated true
e[0m updated_by_last_action true
e[0m default_guard_interpreter :default
e[0m declared_type :ndb_waiter
e[0m cookbook_name "ndb"
e[0m only_if { #code block }
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=ubuntu
e[0m platform_version=20.04
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/usr/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m
e[0m e[0m
================================================================================e[0m
e[31mError executing action `install_grants` on resource 'ndb_mysql_basic[create_users_grants]'e[0m
================================================================================e[0m
e[0m Mixlib::ShellOut::ShellCommandFailede[0m
------------------------------------e[0m
ndb_waiter[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
e[0m ---- Begin output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
e[0m STDOUT: Connection to 192.168.129.198:1186 failed
e[0m STDERR:
e[0m ---- End output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
e[0m Ran "bash" "/tmp/chef-script20211112-38555-noekpf" returned 1e[0m
e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb
e[0m
e[0m 138: ndb_mysql_basic "create_users_grants" do
e[0m 139: action :install_grants
e[0m 140: end
e[0m 141:
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb:138:in `from_file'
e[0m
e[0m ndb_mysql_basic("create_users_grants") do
e[0m action [:install_grants]
e[0m updated true
e[0m updated_by_last_action true
e[0m default_guard_interpreter :default
e[0m declared_type :ndb_mysql_basic
e[0m cookbook_name "ndb"
e[0m recipe_name "mysqld"
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=ubuntu
e[0m platform_version=20.04
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/usr/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m
e[0me[0m
Running handlers:e[0m
[2021-11-12T16:43:53+00:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2021-11-12T16:43:53+00:00] ERROR: Exception handlers complete
Chef Client failed. 14 resources updated in 04 minutes 22 secondse[0m
[2021-11-12T16:43:53+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-12T16:43:53+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-12T16:43:53+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: ndb_mysql_basic[create_users_grants] (ndb::mysqld line 138) had an error: Mixlib::ShellOut::ShellCommandFailed: ndb_waiter[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
STDOUT: Connection to 192.168.129.198:1186 failed
STDERR:
---- End output of "bash" "/tmp/chef-script20211112-38555-noekpf" ----
Ran "bash" "/tmp/chef-script20211112-38555-noekpf" returned 1
Starting Chef Client, version 14.10.9e[0m
resolving cookbooks for run list: ["ndb::mysqld"]e[0m
Synchronizing Cookbooks:e[0m
- ndb (2.4.0)e[0m
- kagent (2.4.0)e[0m
- consul (2.4.0)e[0m
- ulimit (2.4.0)e[0m
- hostsfile (2.4.6)e[0m
- openssl (4.4.0)e[0m
- ntp (2.0.3)e[0m
- sudo (4.0.1)e[0m
- magic_shell (1.0.0)e[0m
- conda (2.4.0)e[0m
- chef-sugar (5.1.12)e[0m
- windows (7.0.2)e[0m
- java (7.0.0)e[0m
- homebrew (5.2.2)e[0m
Installing Cookbook Gems:e[0m
Compiling Cookbooks...e[0m
Converging 33 resourcese[0m
Recipe: ndb::mysqlde[0m
* apt_package[libaio1] action install (up to date)
* directory[/srv/hopsworks-data/rondb/mysql] action create (skipped due to not_if)
* bash[Move MySQL data directory to data volume] action run (skipped due to only_if)
* bash[Delete MySQL data directory] action run (skipped due to only_if)
* link[/srv/hops/mysql-cluster/mysql] action create[2021-11-12T16:44:07+00:00] WARN: /srv/hops/mysql-cluster/mysql mode not changed: File.lchmod is unimplemented on this OS and Ruby version
e[32m- change mode from '0777' to '0700'e[0m
e[0m * template[/srv/hops/mysql-cluster/ndb/scripts/get-mysql-socket.sh] action create (up to date)
* template[/srv/hops/mysql-cluster/ndb/scripts/get-mysql-port.sh] action create (up to date)
* template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-start.sh] action create (up to date)
* template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-stop.sh] action create (up to date)
* template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-restart.sh] action create (up to date)
* template[/srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh] action create (up to date)
* template[/lib/systemd/system/mysqld.service] action create (up to date)
* service[mysqld] action nothing (skipped due to action :nothing)
* template[/srv/hops/mysql-cluster/my.cnf] action create (up to date)
* bash[mysql_install_db] action run
e[32m- execute "bash" "/tmp/chef-script20211112-39147-10yd2kt"e[0m
e[0m * kagent_config[mysqld] action systemd_reload
* bash[start-if-not-running-mysqld] action run
e[32m- execute "bash" "/tmp/chef-script20211112-39147-1h2xiua"e[0m
e[0m
e[0m * ndb_mysql_basic[create_users_grants] action install_grants
* ndb_waiter[wait_ndb_started] action wait_until_cluster_ready
* bash[wait_ndb_started] action run
e[0m
================================================================================e[0m
e[31mError executing action `run` on resource 'bash[wait_ndb_started]'e[0m
================================================================================e[0m
e[0m Mixlib::ShellOut::ShellCommandFailede[0m
------------------------------------e[0m
Expected process to exit with [0], but received '1'
e[0m ---- Begin output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
e[0m STDOUT: Connection to 192.168.129.198:1186 failed
e[0m STDERR:
e[0m ---- End output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
e[0m Ran "bash" "/tmp/chef-script20211112-39147-ta3y1t" returned 1e[0m
e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/ndb/providers/waiter.rb
e[0m
e[0m 3: bash new_resource.name do
e[0m 4: user node['ndb']['user']
e[0m 5: code <<-EOF
e[0m 6: #{node['mysql']['base_dir']}/bin/ndb_waiter -c #{node['ndb']['connectstring']} --timeout=#{node['ndb']['wait_startup']} 2>&1 > /dev/null
e[0m 7: EOF
e[0m 8: end
e[0m 9: new_resource.updated_by_last_action(true)
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/ndb/providers/waiter.rb:3:in `block in class_from_file'
e[0m
e[0m bash("wait_ndb_started") do
e[0m action [:run]
e[0m default_guard_interpreter :default
e[0m command nil
e[0m backup 5
e[0m interpreter "bash"
e[0m declared_type :bash
e[0m cookbook_name "ndb"
e[0m user "mysql"
e[0m code " /srv/hops/mysql/bin/ndb_waiter -c 192.168.129.198:1186 --timeout=10800 2>&1 > /dev/null\n"
e[0m domain nil
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=ubuntu
e[0m platform_version=20.04
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/usr/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m
e[0m e[0m
================================================================================e[0m
e[31mError executing action `wait_until_cluster_ready` on resource 'ndb_waiter[wait_ndb_started]'e[0m
================================================================================e[0m
e[0m Mixlib::ShellOut::ShellCommandFailede[0m
------------------------------------e[0m
bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
e[0m ---- Begin output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
e[0m STDOUT: Connection to 192.168.129.198:1186 failed
e[0m STDERR:
e[0m ---- End output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
e[0m Ran "bash" "/tmp/chef-script20211112-39147-ta3y1t" returned 1e[0m
e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb
e[0m
e[0m 3: ndb_waiter "wait_ndb_started" do
e[0m 4: action :wait_until_cluster_ready
e[0m 5: only_if { node['ndb']['enabled'] == "true" }
e[0m 6: end
e[0m 7:
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb:3:in `block in class_from_file'
e[0m
e[0m ndb_waiter("wait_ndb_started") do
e[0m action [:wait_until_cluster_ready]
e[0m updated true
e[0m updated_by_last_action true
e[0m default_guard_interpreter :default
e[0m declared_type :ndb_waiter
e[0m cookbook_name "ndb"
e[0m only_if { #code block }
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=ubuntu
e[0m platform_version=20.04
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/usr/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m
e[0m e[0m
================================================================================e[0m
e[31mError executing action `install_grants` on resource 'ndb_mysql_basic[create_users_grants]'e[0m
================================================================================e[0m
e[0m Mixlib::ShellOut::ShellCommandFailede[0m
------------------------------------e[0m
ndb_waiter[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
e[0m ---- Begin output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
e[0m STDOUT: Connection to 192.168.129.198:1186 failed
e[0m STDERR:
e[0m ---- End output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
e[0m Ran "bash" "/tmp/chef-script20211112-39147-ta3y1t" returned 1e[0m
e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb
e[0m
e[0m 138: ndb_mysql_basic "create_users_grants" do
e[0m 139: action :install_grants
e[0m 140: end
e[0m 141:
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb:138:in `from_file'
e[0m
e[0m ndb_mysql_basic("create_users_grants") do
e[0m action [:install_grants]
e[0m updated true
e[0m updated_by_last_action true
e[0m default_guard_interpreter :default
e[0m declared_type :ndb_mysql_basic
e[0m cookbook_name "ndb"
e[0m recipe_name "mysqld"
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=ubuntu
e[0m platform_version=20.04
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/usr/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m
e[0me[0m
Running handlers:e[0m
[2021-11-12T16:50:07+00:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2021-11-12T16:50:07+00:00] ERROR: Exception handlers complete
Chef Client failed. 4 resources updated in 06 minutes 07 secondse[0m
[2021-11-12T16:50:07+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-12T16:50:07+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-12T16:50:07+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: ndb_mysql_basic[create_users_grants] (ndb::mysqld line 138) had an error: Mixlib::ShellOut::ShellCommandFailed: ndb_waiter[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
STDOUT: Connection to 192.168.129.198:1186 failed
STDERR:
---- End output of "bash" "/tmp/chef-script20211112-39147-ta3y1t" ----
Ran "bash" "/tmp/chef-script20211112-39147-ta3y1t" returned 1
it seems to me that karamel is try to reach the port 1186
but on my vm there is no LISTEN port 1186
ss -l | grep 1186
and also no LISTEN port 3306 but a port 33060
ss -l | grep 3306
>>>
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 0 70 *:33060 *:*
In my htop the mysqld is launched with
srv/hops/mysql/bin/mysqld --defaults-file=/srv/hops/mysql-cluster/my.cnf
and the /srv/hops/mysql-cluster/my.cnf
#################################################
# MySQL Cluster client, Jim Dowling.
#################################################
[mysql]
default-character-set = utf8
[mysqld]
user = mysql
basedir = /srv/hops/mysql
datadir = /srv/hops/mysql-cluster/mysql
port = 3306
socket = /srv/hops/mysql-cluster/mysql.sock
bind-address = 0.0.0.0
pid-file = /srv/hops/mysql-cluster/log/mysql_52.pid
tmpdir = /tmp
#################################################
# mysql_cluster connection params
#################################################
[mysql_cluster]
# set connectstring to ndb management server (used by all executables)
ndb-connectstring=192.168.129.198:1186
# ndb-wait-setup=10800
...
[client]
default-character-set = utf8
socket = /srv/hops/mysql-cluster/mysql.sock
user = kthfs
password = 50b9e6e2_202
All the files are coming from the installer. I haven’t changed anything.
Hi.
Port 1186 is used by RonDB Management server. Most probably it is not running systemctl status ndb_mgmd You can check the logs under /srv/hops/mysql-cluster/log You can also check the systemd unit logs with journalctl -u ndb_mgmd
Kind regards,
Hi Yingding. We don’t officially support Ubuntu 20.04 yet. We’re still on 18.04.
@antonios Thanks and sorry for this later reply, there is actually no /srv/hops/mysql-cluster/log and also no entry in journalctl -u ndb_mgmd
@Jim_Dowling Thanks Jim for this heads up, I have successfully installed Hopswork on 18.04. Thanks again for the great work and I am really looking forward to the official support for Ubuntu 20.04.