ERROR: Validate failed: Detected failed migration to version 2.8.2.8 (hops)

Hopsworks
#1

HI, I am on ubuntu18.04 and I am trying to install Hopswork.
At the step of installing “hops::ndb” , the installation FAILED.
In the log I found this:

`[sudo] password for *****: Starting Chef Infra Client, version 15.7.32e[0m
resolving cookbooks for run list: ["hops::ndb"]e[0m
Synchronizing Cookbooks:e[0m
  - magic_shell (1.0.0)e[0m
  - java (7.0.0)e[0m
  - hops (1.3.0)e[0m
  - sysctl (1.0.5)e[0m
  - cmake (0.3.0)e[0m
  - kagent (1.3.0)e[0m
  - conda (1.3.0)e[0m
  - ndb (1.3.0)e[0m
  - consul (1.3.0)e[0m
  - elastic (1.3.0)e[0m
  - homebrew (5.0.8)e[0m
  - kzookeeper (1.3.0)e[0m
  - openssl (4.4.0)e[0m
  - windows (7.0.0)e[0m
  - hostsfile (2.4.6)e[0m
  - ulimit2 (0.2.0)e[0m
  - ntp (2.0.3)e[0m
  - ohai (5.3.0)e[0m
  - sudo (4.0.1)e[0m
  - ulimit (1.3.0)e[0m
  - chef-sugar (5.1.8)e[0m
  - elasticsearch (4.0.6)e[0m
  - apt (7.2.0)e[0m
  - yum (5.1.0)e[0m
  - ark (5.0.0)e[0m
  - seven_zip (3.1.2)e[0m
Installing Cookbook Gems:e[0m
Compiling Cookbooks...e[0m
[2020-04-19T22:41:01+02:00] WARN: Resource openssl_dhparam from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-19T22:41:01+02:00] WARN: Resource openssl_rsa_key from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-19T22:41:01+02:00] WARN: Resource openssl_x509 from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-19T22:41:01+02:00] WARN: Resource sudo from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-19T22:41:02+02:00] WARN: Resource sysctl_param from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
Converging 52 resourcese[0m
Recipe: hops::ndbe[0m
  * directory[/srv/hops/ndb-hops-2.8.2.10-SNAPSHOT-7.6.12] action create (up to date)
  * link[/srv/hops/ndb-hops] action create (up to date)
  * remote_file[/tmp/chef-solo/flyway-commandline-5.0.3-linux-x64.tar.gz] action create_if_missing (up to date)
  * bash[unpack_flyway] action run (skipped due to not_if)
  * template[/srv/hops/ndb-hops/flyway/conf/flyway.conf] action create (up to date)
  * directory[/srv/hops/ndb-hops/flyway/undo] action create (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V0.0.2__initial_tables.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.2__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.3__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.4__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.5__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.6__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.7__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.8__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.9__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.10__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/ndb-dal-2.8.2.10-SNAPSHOT-7.6.12.jar] action create_if_missing (up to date)
  * hops_ndb[extract_ndb_hops] action install_ndb_hops
    * remote_file[/srv/hops/ndb-hops-2.8.2.10-SNAPSHOT-7.6.12/libhopsyarn-2.8.2.10-SNAPSHOT-7.6.12.so] action create_if_missing (up to date)
    * link[/srv/hops/ndb-hops/libhopsyarn.so] action create (up to date)
    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jar] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jare[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jar] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jar to /srv/hops/ndb-hops/ndb-dal-2.8.2.10-SNAPSHOT-7.6.12.jare[0m
      e[32m- change owner from 'root' to 'hdfs'e[0m
      e[32m- change group from 'root' to 'hadoop'e[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jar] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jare[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jar] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jar to /srv/hops/ndb-hops/nvidia-management.jare[0m
      e[32m- change owner from '' to 'hdfs'e[0m
      e[32m- change group from '' to 'hadoop'e[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.so] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.soe[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.so] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.so to /srv/hops/mysql/lib/libndbclient.soe[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.so] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.soe[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.so] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.so to /srv/hops/ndb-hops-2.8.2.10-SNAPSHOT-7.6.12/libhopsyarn-2.8.2.10-SNAPSHOT-7.6.12.soe[0m
      e[32m- change owner from 'root' to 'hdfs'e[0m
      e[32m- change group from 'root' to 'hadoop'e[0m
e[0m  
e[0m  * link[/srv/hops/ndb-hops/ndb-dal.jar] action create (up to date)
  * template[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/etc/hadoop/ndb.props] action create (up to date)
  * hops_ndb[install] action install_hops
    * ndb_waiter[wait_mysql_started] action wait_until_cluster_ready
      * bash[wait_mysql_started] action run
        e[32m- execute "bash"  "/tmp/chef-script20200419-19255-orzw8j"e[0m
e[0m    
e[0m    * ndb_mysql_basic[mysqld_start_hop_install] action wait_until_started
      * bash[remove_mycnf_mysqld_start_hop_install] action run (skipped due to only_if)
      * bash[wait_mysqld_started] action run
        e[32m- execute "bash"  "/tmp/chef-script20200419-19255-onuzlu"e[0m
e[0m    
e[0m    * bash[mysql-install-hops] action run
      e[32m- execute "bash"  "/tmp/chef-script20200419-19255-cg5dga"e[0m
e[0m    * template[/srv/hops/ndb-hops/flyway.sql] action create (up to date)
    * bash[flyway_baseline] action run (skipped due to not_if)
    * bash[flyway_migrate] action run
      e[0m
      ================================================================================e[0m
      e[31mError executing action `run` on resource 'bash[flyway_migrate]'e[0m
      ================================================================================e[0m
      
e[0m      Mixlib::ShellOut::ShellCommandFailede[0m
      ------------------------------------e[0m
      Expected process to exit with [0], but received '1'
e[0m      ---- Begin output of "bash"  "/tmp/chef-script20200419-19255-bkletd" ----
e[0m      STDOUT: Flyway Community Edition 5.0.3 by Boxfuse
e[0m      
e[0m      Database: jdbc:mysql://10.0.2.15:3306/hops (MySQL 5.7)
e[0m      STDERR: ERROR: Validate failed: Detected failed migration to version 2.8.2.8 (hops)
e[0m      ---- End output of "bash"  "/tmp/chef-script20200419-19255-bkletd" ----
e[0m      Ran "bash"  "/tmp/chef-script20200419-19255-bkletd" returned 1e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /tmp/chef-solo/cookbooks/hops/providers/ndb.rb
e[0m      
e[0m       44:   bash "flyway_migrate" do
e[0m       45:     user "root"
e[0m       46:     timeout 36000
e[0m       47:     code <<-EOF
e[0m       48:     set -e
e[0m       49:     cd #{flyway_dir}
e[0m       50:     #{flyway_dir}/flyway migrate
e[0m       51:   EOF
e[0m       52:   end
e[0m       53:   
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /tmp/chef-solo/cookbooks/hops/providers/ndb.rb:44:in `block in class_from_file'
e[0m      
e[0m      bash("flyway_migrate") do
e[0m        action [:run]
e[0m        default_guard_interpreter :default
e[0m        command nil
e[0m        backup 5
e[0m        interpreter "bash"
e[0m        declared_type :bash
e[0m        cookbook_name "hops"
e[0m        user "root"
e[0m        code "    set -e\n    cd /srv/hops/ndb-hops/flyway\n    /srv/hops/ndb-hops/flyway/flyway migrate\n"
e[0m        domain nil
e[0m        timeout 36000
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.7.32
e[0m      platform=ubuntu
e[0m      platform_version=18.04
e[0m      ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m      program_name=/usr/bin/chef-solo
e[0m      executable=/opt/chefdk/bin/chef-soloe[0m
      
e[0m    e[0m
    ================================================================================e[0m
    e[31mError executing action `install_hops` on resource 'hops_ndb[install]'e[0m
    ================================================================================[0m
    
[0m    Mixlib::ShellOut::ShellCommandFailede[0m
    ------------------------------------e[0m
    bash[flyway_migrate] (/tmp/chef-solo/cookbooks/hops/providers/ndb.rb line 44) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
e[0m    ---- Begin output of "bash"  "/tmp/chef-script20200419-19255-bkletd" ----
e[0m    STDOUT: Flyway Community Edition 5.0.3 by Boxfuse
e[0m    
e[0m    Database: jdbc:mysql://10.0.2.15:3306/hops (MySQL 5.7)
e[0m    STDERR: ERROR: Validate failed: Detected failed migration to version 2.8.2.8 (hops)
e[0m    ---- End output of "bash"  "/tmp/chef-script20200419-19255-bkletd" ----
e[0m    Ran "bash"  "/tmp/chef-script20200419-19255-bkletd" returned 1e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /tmp/chef-solo/cookbooks/hops/recipes/ndb.rb
e[0m    
e[0m    142:   hops_ndb "install" do
e[0m    143:     action :install_hops
e[0m    144:   end
e[0m    145: end
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /tmp/chef-solo/cookbooks/hops/recipes/ndb.rb:142:in `from_file'
e[0m    
e[0m    hops_ndb("install") do
e[0m      action [:install_hops]
e[0m      updated true
e[0m      updated_by_last_action true
e[0m      default_guard_interpreter :default
e[0m      declared_type :hops_ndb
e[0m      cookbook_name "hops"
e[0m      recipe_name "ndb"
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=15.7.32
e[0m    platform=ubuntu
e[0m    platform_version=18.04
e[0m    ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m    program_name=/usr/bin/chef-solo
e[0m    executable=/opt/chefdk/bin/chef-soloe[0m
    
e[0me[0m
Running handlers:e[0m
[2020-04-19T22:41:03+02:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2020-04-19T22:41:03+02:00] ERROR: Exception handlers complete
Chef Infra Client failed. 14 resources updated in 06 secondse[0m
[2020-04-19T22:41:03+02:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-04-19T22:41:03+02:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-04-19T22:41:03+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: hops_ndb[install] (hops::ndb line 142) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[flyway_migrate] (/tmp/chef-solo/cookbooks/hops/providers/ndb.rb line 44) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200419-19255-bkletd" ----
STDOUT: Flyway Community Edition 5.0.3 by Boxfuse

Database: jdbc:mysql://10.0.2.15:3306/hops (MySQL 5.7)
STDERR: ERROR: Validate failed: Detected failed migration to version 2.8.2.8 (hops)
---- End output of "bash"  "/tmp/chef-script20200419-19255-bkletd" ----
Ran "bash"  "/tmp/chef-script20200419-19255-bkletd"

How can I throw it out ?

Best regards

#2

Hi!

Are you doing a fresh installation or an upgrade? If it’s a fresh installation and you don’t have any data stored the easiest solution is to drop hops database and retry the recipe.

To drop the database use /srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh as root and execute drop schema hops. Then you can re-run the recipe from Karamel UI. You can access it from your browser at http://YOUR_IP_ADDRESS:9090/index.html#/terminal Click on status and retry the failed recipe.

Kind regards,

#3

Also, how are you installing hopsworks (with hopsworks-installer.sh ?) and or what platform (GCP, AWS, Azure, On-premises)?

#4

Hi,
thanks to antonios, I solved the problem for hops::ndb .
I am doing a fresh installation.
I am installing hopswork by launching on my own ./bin/karamel for on-premises platform.

Now I have a problem on hops_airflow::default . The log gives me that:

   [sudo] password for nikokks: Starting Chef Infra Client, version 15.7.32e[0m
resolving cookbooks for run list: ["hops_airflow::default"]e[0m
Synchronizing Cookbooks:e[0m
  -

…

e[0m    MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'nikokks' (111)")
e[0m    
e[0m    The above exception was the direct cause of the following exception:
e[0m    
e[0m    Traceback (most recent call last):
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/bin/airflow", line 32, in <module>
e[0m        args.func(args)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
e[0m        return f(*args, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/bin/cli.py", line 1091, in upgradedb
e[0m        db.upgradedb()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py", line 346, in upgradedb
e[0m        command.upgrade(config, 'heads')
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/command.py", line 174, in upgrade
e[0m        script.run_env()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
e[0m        util.load_python_file(self.dir, 'env.py')
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
e[0m        module = load_module_py(module_id, path)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/util/compat.py", line 68, in load_module_py
e[0m        module_id, path).load_module(module_id)
e[0m      File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
e[0m      File "<frozen importlib._bootstrap_external>", line 823, in load_module
e[0m      File "<frozen importlib._bootstrap_external>", line 682, in load_module
e[0m      File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
e[0m      File "<frozen importlib._bootstrap>", line 684, in _load
e[0m      File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
e[0m      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
e[0m      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py", line 92, in <module>
e[0m        run_migrations_online()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py", line 78, in run_migrations_online
e[0m        with connectable.connect() as connection:
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2171, in connect
e[0m        return self._connection_cls(self, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
e[0m        else engine.raw_connection()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2259, in raw_connection
e[0m        self.pool.unique_connection, _connection
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect
e[0m        e, dialect, self
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1528, in _handle_dbapi_exception_noconnection
e[0m        util.raise_from_cause(sqlalchemy_exception, exc_info)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
e[0m        reraise(type(exception), exception, tb=exc_tb, cause=cause)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
e[0m        raise value.with_traceback(tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
e[0m        return fn()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 374, in unique_connection
e[0m        return _ConnectionFairy._checkout(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
e[0m        fairy = _ConnectionRecord.checkout(pool)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
e[0m        rec = pool._do_get()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
e[0m        self._dec_overflow()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
e[0m        compat.reraise(exc_type, exc_value, exc_tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
e[0m        raise value
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
e[0m        return self._create_connection()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
e[0m        return _ConnectionRecord(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
e[0m        self.__connect(first_connect_check=True)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
e[0m        connection = pool._invoke_creator(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
e[0m        return dialect.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
e[0m        return self.dbapi.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
e[0m        return Connection(*args, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
e[0m        super(Connection, self).__init__(*args, **kwargs2)
e[0m    sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'nikokks' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
e[0m    ---- End output of "bash"  "/tmp/chef-script20200420-13403-37y520" ----
e[0m    Ran "bash"  "/tmp/chef-script20200420-13403-37y520" returned 1e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /tmp/chef-solo/cookbooks/hops_airflow/recipes/default.rb
e[0m    
e[0m     65: bash 'init_airflow_db' do
e[0m     66:   user node['airflow']['user']
e[0m     67:   code <<-EOF
e[0m     68:       set -e
e[0m     69:       export AIRFLOW_HOME=#{node['airflow']['base_dir']}
e[0m     70:       #{node['airflow']['bin_path']}/airflow upgradedb
e[0m     71:     EOF
e[0m     72: end
e[0m     73: 
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /tmp/chef-solo/cookbooks/hops_airflow/recipes/default.rb:65:in `from_file'
e[0m    
e[0m    bash("init_airflow_db") do
e[0m      action [:run]
e[0m      default_guard_interpreter :default
e[0m      command nil
e[0m      backup 5
e[0m      interpreter "bash"
e[0m      declared_type :bash
e[0m      cookbook_name "hops_airflow"
e[0m      recipe_name "default"
e[0m      user "airflow"
e[0m      code "      set -e\n      export AIRFLOW_HOME=/srv/hops/airflow\n      /srv/hops/anaconda/anaconda/envs/airflow/bin/airflow upgradedb\n"
e[0m      domain nil
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=15.7.32
e[0m    platform=ubuntu
e[0m    platform_version=18.04
e[0m    ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m    program_name=/usr/bin/chef-solo
e[0m    executable=/opt/chefdk/bin/chef-soloe[0m
    
e[0me[0m
Running handlers:e[0m
[2020-04-20T12:42:49+02:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2020-04-20T12:42:49+02:00] ERROR: Exception handlers complete
Chef Infra Client failed. 54 resources updated in 05 minutes 30 secondse[0m
[2020-04-20T12:42:49+02:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-04-20T12:42:49+02:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-04-20T12:42:49+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[init_airflow_db] (hops_airflow::default line 65) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200420-13403-37y520" ----
STDOUT: [2020-04-20 12:42:47,726] {settings.py:174} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=3600, pid=14766
[2020-04-20 12:42:48,343] {__init__.py:51} INFO - Using executor LocalExecutor
[2020-04-20 12:42:49,012] {cli_action_loggers.py:69} ERROR - Failed on pre-execution callback using <function default_action_log at 0x7fccac3849d8>
Traceback (most recent call last):
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
    return fn()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 434, in connect
    return _ConnectionFairy._checkout(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
    rec = pool._do_get()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
    self._dec_overflow()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
    raise value
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
    return self._create_connection()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
    return _ConnectionRecord(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
    self.__connect(first_connect_check=True)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
    connection = pool._invoke_creator(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'nikokks' (111)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 67, in on_pre_execution
    cb(**kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 98, in default_action_log
    session.add(log)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py", line 44, in create_session
    session.commit()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1023, in commit
    self.transaction.commit()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 487, in commit
    self._prepare_impl()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 466, in _prepare_impl
    self.session.flush()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2446, in flush
    self._flush(objects)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2584, in _flush
    transaction.rollback(_capture_exception=True)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
    raise value
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2544, in _flush
    flush_context.execute()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute
    rec.execute(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute
    uow,
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 213, in save_obj
    ) in _organize_states_for_save(base_mapper, states, uowtransaction):
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 374, in _organize_states_for_save
    base_mapper, uowtransaction, states
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1558, in _connections_for_states
    connection = uowtransaction.transaction.connection(base_mapper)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 306, in connection
    return self._connection_for_bind(bind, execution_options)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 412, in _connection_for_bind
    conn = self._parent._connection_for_bind(bind, execution_options)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 424, in _connection_for_bind
    conn = bind.contextual_connect()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2192, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect
    e, dialect, self
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1528, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
    raise value.with_traceback(tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
    return fn()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 434, in connect
    return _ConnectionFairy._checkout(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
    rec = pool._do_get()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
    self._dec_overflow()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
    raise value
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
    return self._create_connection()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
    return _ConnectionRecord(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
    self.__connect(first_connect_check=True)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
    connection = pool._invoke_creator(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'nikokks' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
DB: mysql://airflow_db:***@nikokks:3306/airflow
[2020-04-20 12:42:49,265] {db.py:338} INFO - Creating tables
STDERR: Traceback (most recent call last):
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
    return fn()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 374, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
    rec = pool._do_get()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
    self._dec_overflow()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
    raise value
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
    return self._create_connection()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
    return _ConnectionRecord(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
    self.__connect(first_connect_check=True)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
    connection = pool._invoke_creator(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'nikokks' (111)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/hops/anaconda/anaconda/envs/airflow/bin/airflow", line 32, in <module>
    args.func(args)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/bin/cli.py", line 1091, in upgradedb
    db.upgradedb()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py", line 346, in upgradedb
    command.upgrade(config, 'heads')
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/util/compat.py", line 68, in load_module_py
    module_id, path).load_module(module_id)
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py", line 92, in <module>
    run_migrations_online()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py", line 78, in run_migrations_online
    with connectable.connect() as connection:
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2171, in connect
    return self._connection_cls(self, **kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
    else engine.raw_connection()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2259, in raw_connection
    self.pool.unique_connection, _connection
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect
    e, dialect, self
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1528, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
    raise value.with_traceback(tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
    return fn()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 374, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
    rec = pool._do_get()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
    self._dec_overflow()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
    raise value
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
    return self._create_connection()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
    return _ConnectionRecord(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
    self.__connect(first_connect_check=True)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
    connection = pool._invoke_creator(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'nikokks' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
---- End output of "bash"  "/tmp/chef-script20200420-13403-37y520" ----
Ran "bash"  "/tmp/chef-script20200420-13403-37y520" returned 1
#5

Is your MySQL daemon running (systemctl status mysqld)? It can’t connect.
You can try on your own doing mysql -h nikokks -u kthfs -p The password for the mysql database is https://github.com/logicalclocks/ndb-chef/blob/master/attributes/default.rb#L187 unless if you changed it.

Authentication problem
#6

I’ve just reboot the entire installation of karamel and found an other problem on hops::ndb .

[sudo] password for nikokks: Starting Chef Infra Client, version 15.7.32e[0m
resolving cookbooks for run list: ["hops::ndb"]e[0m
Synchronizing Cookbooks:e[0m
  - hops (1.3.0)e[0m
  - magic_shell (1.0.0)e[0m
  - cmake (0.3.0)e[0m
  - sysctl (1.0.5)e[0m
  - java (7.0.0)e[0m
  - kagent (1.3.0)e[0m
  - conda (1.3.0)e[0m
  - ndb (1.3.0)e[0m
  - consul (1.3.0)e[0m
  - homebrew (5.0.8)e[0m
  - windows (7.0.0)e[0m
  - ohai (5.3.0)e[0m
  - openssl (4.4.0)e[0m
  - hostsfile (2.4.6)e[0m
  - ntp (2.0.3)e[0m
  - sudo (4.0.1)e[0m
  - ulimit (1.3.0)e[0m
  - ulimit2 (0.2.0)e[0m
  - elasticsearch (4.0.6)e[0m
  - chef-sugar (5.1.8)e[0m
  - apt (7.2.0)e[0m
  - yum (5.1.0)e[0m
  - ark (5.0.0)e[0m
  - seven_zip (3.1.2)e[0m
  - kzookeeper (1.3.0)e[0m
  - elastic (1.3.0)e[0m
Installing Cookbook Gems:e[0m
Compiling Cookbooks...e[0m
[2020-04-20T18:10:50+02:00] WARN: Resource openssl_dhparam from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-20T18:10:50+02:00] WARN: Resource openssl_rsa_key from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-20T18:10:50+02:00] WARN: Resource openssl_x509 from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-20T18:10:50+02:00] WARN: Resource sudo from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-20T18:10:50+02:00] WARN: Resource sysctl_param from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
Converging 52 resourcese[0m
Recipe: hops::ndbe[0m
  * directory[/srv/hops/ndb-hops-2.8.2.10-SNAPSHOT-7.6.12] action create (up to date)
  * link[/srv/hops/ndb-hops] action create (up to date)
  * remote_file[/tmp/chef-solo/flyway-commandline-5.0.3-linux-x64.tar.gz] action create_if_missing (up to date)
  * bash[unpack_flyway] action run (skipped due to not_if)
  * template[/srv/hops/ndb-hops/flyway/conf/flyway.conf] action create (up to date)
  * directory[/srv/hops/ndb-hops/flyway/undo] action create (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V0.0.2__initial_tables.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.2__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.3__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.4__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.5__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.6__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.7__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.8__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.9__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/flyway/sql/V2.8.2.10__hops.sql] action create_if_missing (up to date)
  * remote_file[/srv/hops/ndb-hops/ndb-dal-2.8.2.10-SNAPSHOT-7.6.12.jar] action create_if_missing (up to date)
  * hops_ndb[extract_ndb_hops] action install_ndb_hops
    * remote_file[/srv/hops/ndb-hops-2.8.2.10-SNAPSHOT-7.6.12/libhopsyarn-2.8.2.10-SNAPSHOT-7.6.12.so] action create_if_missing (up to date)
    * link[/srv/hops/ndb-hops/libhopsyarn.so] action create (up to date)
    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jar] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jare[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jar] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/ndb-dal.jar to /srv/hops/ndb-hops/ndb-dal-2.8.2.10-SNAPSHOT-7.6.12.jare[0m
      e[32m- change owner from 'root' to 'hdfs'e[0m
      e[32m- change group from 'root' to 'hadoop'e[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jar] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jare[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jar] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/share/hadoop/common/lib/nvidia-management.jar to /srv/hops/ndb-hops/nvidia-management.jare[0m
      e[32m- change owner from '' to 'hdfs'e[0m
      e[32m- change group from '' to 'hadoop'e[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.so] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.soe[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.so] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libndbclient.so to /srv/hops/mysql/lib/libndbclient.soe[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.so] action delete
      e[32m- delete link to file at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.soe[0m
e[0m    * link[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.so] action create
      e[32m- create symlink at /srv/hops/hadoop-2.8.2.10-SNAPSHOT/lib/native/libhopsyarn.so to /srv/hops/ndb-hops-2.8.2.10-SNAPSHOT-7.6.12/libhopsyarn-2.8.2.10-SNAPSHOT-7.6.12.soe[0m
      e[32m- change owner from 'root' to 'hdfs'e[0m
      e[32m- change group from 'root' to 'hadoop'e[0m
e[0m  
e[0m  * link[/srv/hops/ndb-hops/ndb-dal.jar] action create (up to date)
  * template[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/etc/hadoop/ndb.props] action create (up to date)
  * hops_ndb[install] action install_hops
    * ndb_waiter[wait_mysql_started] action wait_until_cluster_ready
      * bash[wait_mysql_started] action run
        e[32m- execute "bash"  "/tmp/chef-script20200420-20768-8v1247"e[0m
e[0m    
e[0m    * ndb_mysql_basic[mysqld_start_hop_install] action wait_until_started
      * bash[remove_mycnf_mysqld_start_hop_install] action run (skipped due to only_if)
      * bash[wait_mysqld_started] action run
        e[32m- execute "bash"  "/tmp/chef-script20200420-20768-1dgdv17"e[0m
e[0m    
e[0m    * bash[mysql-install-hops] action run
      e[32m- execute "bash"  "/tmp/chef-script20200420-20768-womyzx"e[0m
e[0m    * template[/srv/hops/ndb-hops/flyway.sql] action create (up to date)
    * bash[flyway_baseline] action run (skipped due to not_if)
    * bash[flyway_migrate] action run
      e[32m- execute "bash"  "/tmp/chef-script20200420-20768-1rg4rxs"e[0m
e[0m  
e[0m  * template[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/sbin/start-nn.sh] action create (up to date)
  * template[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/sbin/stop-nn.sh] action create (up to date)
  * template[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/sbin/restart-nn.sh] action create (up to date)
  * template[/srv/hops/hadoop-2.8.2.10-SNAPSHOT/sbin/format-nn.sh] action create (up to date)
Recipe: java::notifye[0m
  * log[jdk-version-changed] action nothing (skipped due to action :nothing)
Recipe: java::openjdke[0m
  * apt_package[openjdk-8-jdk, openjdk-8-jre-headless] action install (up to date)
  * java_alternatives[set-java-alternatives] action set (up to date)
Recipe: java::default_java_symlinke[0m
  * link[/usr/lib/jvm/default-java] action create (up to date)
Recipe: java::set_java_homee[0m
  * directory[/etc/profile.d] action create (up to date)
  * template[/etc/profile.d/jdk.sh] action create (up to date)
  * ruby_block[Set JAVA_HOME in /etc/environment] action run
    e[32m- execute the ruby block Set JAVA_HOME in /etc/environmente[0m
e[0mRecipe: hops::defaulte[0m
  * ruby_block[discover_gpus] action run
    e[32m- execute the ruby block discover_gpuse[0m
e[0m  * template[/srv/hops/hadoop/etc/hadoop/log4j.properties] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/core-site.xml] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/hadoop-env.sh] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/jmxremote.access] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/jmxremote.password] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/yarn-jmxremote.password] action create (up to date)
  * template[/srv/hops/hadoop/sbin/kill-process.sh] action create (up to date)
  * template[/srv/hops/hadoop/sbin/set-env.sh] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/hdfs-site.xml] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/erasure-coding-site.xml] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/yarn-site.xml] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/container-executor.cfg] action create (up to date)
  * template[/srv/hops/hadoop/etc/hadoop/yarn-env.sh] action create (up to date)
  * bash[remove-hadoop-log-copy-cron] action run
    e[32m- execute "bash"  "/tmp/chef-script20200420-20768-16furw8"e[0m
e[0m  * cron[copy_hadoop_logs] action create
    e[32m- add crontab entry for cron[copy_hadoop_logs]e[0m
e[0m  * cron[delete_hadoop_logs] action create
    e[32m- add crontab entry for cron[delete_hadoop_logs]e[0m
e[0m  * cookbook_file[/srv/hops/hadoop/etc/hadoop/namenode.yaml] action create (up to date)
Recipe: hops::formate[0m
  * hops_ndb[format-nn] action format_nn
    * bash[format-nn] action run (skipped due to not_if)
     (up to date)
  * bash[validate_formatting] action run
    e[0m
    ================================================================================e[0m
    e[31mError executing action `run` on resource 'bash[validate_formatting]'e[0m
    ================================================================================e[0m
    
e[0m    Mixlib::ShellOut::ShellCommandFailede[0m
    ------------------------------------e[0m
    Expected process to exit with [0], but received '1'
e[0m    ---- Begin output of "bash"  "/tmp/chef-script20200420-20768-ch9sfe" ----
e[0m    STDOUT: 
e[0m    STDERR: mysql: [Warning] Using a password on the command line interface can be insecure.
e[0m    ---- End output of "bash"  "/tmp/chef-script20200420-20768-ch9sfe" ----
e[0m    Ran "bash"  "/tmp/chef-script20200420-20768-ch9sfe" returned 1e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /tmp/chef-solo/cookbooks/hops/recipes/format.rb
e[0m    
e[0m     22:     bash "validate_formatting" do
e[0m     23:       user "root"
e[0m     24:       code <<-EOF
e[0m     25:        #{exec} hops -e 'select count(*) from hdfs_variables' | tail -n 1 | egrep -v "^0$"
e[0m     26:       EOF
e[0m     27:     end
e[0m     28:   rescue
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /tmp/chef-solo/cookbooks/hops/recipes/format.rb:22:in `from_file'
e[0m    
e[0m    bash("validate_formatting") do
e[0m      action [:run]
e[0m      default_guard_interpreter :default
e[0m      command nil
e[0m      backup 5
e[0m      interpreter "bash"
e[0m      declared_type :bash
e[0m      cookbook_name "hops"
e[0m      recipe_name "format"
e[0m      user "root"
e[0m      code "       /srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh hops -e 'select count(*) from hdfs_variables' | tail -n 1 | egrep -v \"^0$\"\n"
e[0m      domain nil
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=15.7.32
e[0m    platform=ubuntu
e[0m    platform_version=18.04
e[0m    ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m    program_name=/usr/bin/chef-solo
e[0m    executable=/opt/chefdk/bin/chef-soloe[0m
    
e[0me[0m
Running handlers:e[0m
[2020-04-20T18:10:55+02:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2020-04-20T18:10:55+02:00] ERROR: Exception handlers complete
Chef Infra Client failed. 21 resources updated in 10 secondse[0m
[2020-04-20T18:10:55+02:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-04-20T18:10:55+02:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-04-20T18:10:55+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[validate_formatting] (hops::format line 22) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200420-20768-ch9sfe" ----
STDOUT: 
STDERR: mysql: [Warning] Using a password on the command line interface can be insecure.
---- End output of "bash"  "/tmp/chef-script20200420-20768-ch9sfe" ----
Ran "bash"  "/tmp/chef-script20200420-20768-ch9sfe" returned 1

Do you have a solution ?

#7

Can you run this command /srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh hops -e 'select count(*) from hdfs_variables' | tail -n 1 | egrep -v "^0$" ? I suspect something is off with the initialization of the Namenode.

If you restart the installation with Karamel it’s advised to (1) shutdown all services using this script /srv/hops/kagent/kagent/bin/shutdown-all-local-services.sh (2) rm -rf /srv/hops or whatever is the root of your installation (3) rm -rf $HOME/.karamel $HOME is the home of the user who’s running Karamel.

Could you send me the cluster definition you’re using to try it out myself?

#8

Hi antonios,
I have tried what you’ve said
this are the following results

root@nikokks:/usr/bin# mysql -h nikokks -u kthfs -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on 'nikokks' (111)
root@nikokks:/usr/bin# mysql  -u kthfs -p
Enter password: 
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

The password I used for these commands are :
the password mentioned in your link
the password for super user

#9

I suspect your hostname resolves to 127.0.0.1 instead of your private IP and mysql tries to use the unix socket instead of a network socket. Check your /etc/hosts. Also check on what IP your mysql server is listening to netstat -lpn | grep 3306

#10

Hi antonios,

these are the following results of the commands

nikokks@nikokks:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 nikokks

nikokks@nikokks:~$ sudo netstat -lpn | grep 3306
tcp 0 0 10.0.2.15:3306 0.0.0.0:* LISTEN 8368/mysqld

nikokks@nikokks:~$ sudo mysql
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)

#11

That’s what I suspected. Substitute the IP on the second line to look like 10.0.2.15 nikokks

#12

Thank you Antonios.
I have the following error now on ndb::mysqld .

[sudo] password for nikokks: Starting Chef Infra Client, version 15.7.32e[0m
resolving cookbooks for run list: ["ndb::mysqld"]e[0m
Synchronizing Cookbooks:e[0m
  - ndb (1.3.0)e[0m
  - kagent (1.3.0)e[0m
  - ulimit (1.3.0)e[0m
  - openssl (4.4.0)e[0m
  - hostsfile (2.4.6)e[0m
  - conda (1.3.0)e[0m
  - chef-sugar (5.1.8)e[0m
  - magic_shell (1.0.0)e[0m
  - ntp (2.0.3)e[0m
  - sudo (4.0.1)e[0m
  - windows (7.0.0)e[0m
  - java (7.0.0)e[0m
  - homebrew (5.0.8)e[0m
Installing Cookbook Gems:e[0m
Compiling Cookbooks...e[0m
[2020-04-23T10:09:10+02:00] WARN: Resource openssl_dhparam from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-23T10:09:10+02:00] WARN: Resource openssl_rsa_key from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-23T10:09:10+02:00] WARN: Resource openssl_x509 from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-23T10:09:10+02:00] WARN: Resource sudo from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
Converging 29 resourcese[0m
Recipe: ndb::mysqlde[0m
  * apt_package[libaio1] action install (up to date)
  * directory[/srv/hops/mysql-cluster/mysql] action create
    e[32m- change mode from '0750' to '0700'e[0m
e[0m  * template[/srv/hops/mysql-cluster/ndb/scripts/get-mysql-socket.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/get-mysql-port.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-start.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-stop.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-restart.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh] action create (up to date)
  * template[/lib/systemd/system/mysqld.service] action create (up to date)
  * service[mysqld] action nothing (skipped due to action :nothing)
  * template[/srv/hops/mysql-cluster/my.cnf] action create (up to date)
  * bash[mysql_install_db] action run
    e[32m- execute "bash"  "/tmp/chef-script20200423-20704-147i7xo"e[0m
e[0m  * kagent_config[mysqld] action systemd_reload
    * bash[start-if-not-running-mysqld] action run[2020-04-23T11:22:05+02:00] ERROR: bash[start-if-not-running-mysqld] (/tmp/chef-solo/cookbooks/kagent/providers/config.rb line 49) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command exceeded allowed execution time, process terminated
---- Begin output of "bash"  "/tmp/chef-script20200423-20704-yeco7u" ----
STDOUT: 
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200423-20704-yeco7u" ----
Ran "bash"  "/tmp/chef-script20200423-20704-yeco7u" returned ; ignore_failure is set, continuing

      e[0m
      ================================================================================e[0m
      e[31mError executing action `run` on resource 'bash[start-if-not-running-mysqld]'e[0m
      ================================================================================e[0m
      
e[0m      Mixlib::ShellOut::CommandTimeoute[0m
      --------------------------------e[0m
      Command timed out after 3600s:
e[0m      Command exceeded allowed execution time, process terminated
e[0m      ---- Begin output of "bash"  "/tmp/chef-script20200423-20704-yeco7u" ----
e[0m      STDOUT: 
e[0m      STDERR: 
e[0m      ---- End output of "bash"  "/tmp/chef-script20200423-20704-yeco7u" ----
e[0m      Ran "bash"  "/tmp/chef-script20200423-20704-yeco7u" returned e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /tmp/chef-solo/cookbooks/kagent/providers/config.rb
e[0m      
e[0m       49:   bash "start-if-not-running-#{new_resource.name}" do
e[0m       50:     user "root"
e[0m       51:     ignore_failure true
e[0m       52:     code <<-EOH
e[0m       53:      systemctl stop #{new_resource.name}
e[0m       54:      systemctl daemon-reload
e[0m       55:      systemctl reset-failed
e[0m       56:      systemctl start #{new_resource.name}
e[0m       57:     EOH
e[0m       58:   end
e[0m       59: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /tmp/chef-solo/cookbooks/kagent/providers/config.rb:49:in `block in class_from_file'
e[0m      
e[0m      bash("start-if-not-running-mysqld") do
e[0m        action [:run]
e[0m        default_guard_interpreter :default
e[0m        command nil
e[0m        backup 5
e[0m        interpreter "bash"
e[0m        declared_type :bash
e[0m        cookbook_name "ndb"
e[0m        user "root"
e[0m        code "     systemctl stop mysqld\n     systemctl daemon-reload\n     systemctl reset-failed\n     systemctl start mysqld\n"
e[0m        domain nil
e[0m        ignore_failure true
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.7.32
e[0m      platform=ubuntu
e[0m      platform_version=18.04
e[0m      ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m      program_name=/usr/bin/chef-solo
e[0m      executable=/opt/chefdk/bin/chef-soloe[0m
      
e[0m     (up to date)
  * ndb_mysql_basic[create_users_grants] action install_grants
    * ndb_waiter[wait_ndb_started] action wait_until_cluster_ready
      * bash[wait_ndb_started] action run
        e[32m- execute "bash"  "/tmp/chef-script20200423-20704-yw7fij"e[0m
e[0m    
e[0m    * ndb_mysql_basic[mysqld_start_grants] action wait_until_started
      * bash[remove_mycnf_mysqld_start_grants] action run
        e[32m- execute "bash"  "/tmp/chef-script20200423-20704-1p0xqci"e[0m
e[0m      * bash[wait_mysqld_started] action run
        e[0m
        ================================================================================e[0m
        e[31mError executing action `run` on resource 'bash[wait_mysqld_started]'e[0m
        ================================================================================e[0m
        
e[0m        Mixlib::ShellOut::ShellCommandFailede[0m
        ------------------------------------e[0m
        Expected process to exit with [0], but received '1'
e[0m        ---- Begin output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
e[0m        STDOUT: 
e[0m        STDERR: amysqladmin: connect to server at 'localhost' failed
e[0m        error: 'Can't connect to local MySQL server through socket '/srv/hops/mysql-cluster/mysql.sock' (2)'
e[0m        Check that mysqld is running and that the socket: '/srv/hops/mysql-cluster/mysql.sock' exists!
e[0m        ---- End output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
e[0m        Ran "bash"  "/tmp/chef-script20200423-20704-1p9595h" returned 1e[0m
        
e[0m        Resource Declaration:e[0m
        ---------------------e[0m
        # In /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb
e[0m        
e[0m         54:   bash 'wait_mysqld_started' do
e[0m         55:     user "root"
e[0m         56:     retries num_retry
e[0m         57:     retry_delay ret_delay 
e[0m         58:     code <<-EOF
e[0m         59:       #{node['mysql']['base_dir']}/bin/mysqladmin -u root -S #{node['ndb']['mysql_socket']} status
e[0m         60:     EOF
e[0m         61:   end
e[0m         62: 
e[0m        
e[0m        Compiled Resource:e[0m
        ------------------e[0m
        # Declared in /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb:54:in `block in class_from_file'
e[0m        
e[0m        bash("wait_mysqld_started") do
e[0m          action [:run]
e[0m          default_guard_interpreter :default
e[0m          command nil
e[0m          backup 5
e[0m          interpreter "bash"
e[0m          declared_type :bash
e[0m          cookbook_name "ndb"
e[0m          user "root"
e[0m          code "      /srv/hops/mysql/bin/mysqladmin -u root -S /srv/hops/mysql-cluster/mysql.sock status\n"
e[0m          domain nil
e[0m          retries 4
e[0m          retry_delay 5
e[0m        end
e[0m        
e[0m        System Info:e[0m
        ------------e[0m
        chef_version=15.7.32
e[0m        platform=ubuntu
e[0m        platform_version=18.04
e[0m        ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m        program_name=/usr/bin/chef-solo
e[0m        executable=/opt/chefdk/bin/chef-soloe[0m
        
e[0m      e[0m
      ================================================================================e[0m
      e[31mError executing action `wait_until_started` on resource 'ndb_mysql_basic[mysqld_start_grants]'e[0m
      ================================================================================e[0m
      
e[0m      Mixlib::ShellOut::ShellCommandFailede[0m
      ------------------------------------e[0m
      bash[wait_mysqld_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
e[0m      ---- Begin output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
e[0m      STDOUT: 
e[0m      STDERR: amysqladmin: connect to server at 'localhost' failed
e[0m      error: 'Can't connect to local MySQL server through socket '/srv/hops/mysql-cluster/mysql.sock' (2)'
e[0m      Check that mysqld is running and that the socket: '/srv/hops/mysql-cluster/mysql.sock' exists!
e[0m      ---- End output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
e[0m      Ran "bash"  "/tmp/chef-script20200423-20704-1p9595h" returned 1e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb
e[0m      
e[0m        8:   ndb_mysql_basic "mysqld_start_grants" do
e[0m        9:     wait_time 20
e[0m       10:     remove_mycnf 1
e[0m       11:     action :wait_until_started
e[0m       12:   end
e[0m       13: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb:8:in `block in class_from_file'
e[0m      
e[0m      ndb_mysql_basic("mysqld_start_grants") do
e[0m        action [:wait_until_started]
e[0m        updated true
e[0m        updated_by_last_action true
e[0m        default_guard_interpreter :default
e[0m        declared_type :ndb_mysql_basic
e[0m        cookbook_name "ndb"
e[0m        wait_time 20
e[0m        remove_mycnf 1
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.7.32
e[0m      platform=ubuntu
e[0m      platform_version=18.04
e[0m      ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m      program_name=/usr/bin/chef-solo
e[0m      executable=/opt/chefdk/bin/chef-soloe[0m
      
e[0m    e[0m
    ================================================================================e[0m
    e[31mError executing action `install_grants` on resource 'ndb_mysql_basic[create_users_grants]'e[0m
    ================================================================================e[0m
    
e[0m    Mixlib::ShellOut::ShellCommandFailede[0m
    ------------------------------------e[0m
    ndb_mysql_basic[mysqld_start_grants] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 8) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[wait_mysqld_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
e[0m    ---- Begin output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
e[0m    STDOUT: 
e[0m    STDERR: amysqladmin: connect to server at 'localhost' failed
e[0m    error: 'Can't connect to local MySQL server through socket '/srv/hops/mysql-cluster/mysql.sock' (2)'
e[0m    Check that mysqld is running and that the socket: '/srv/hops/mysql-cluster/mysql.sock' exists!
e[0m    ---- End output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
e[0m    Ran "bash"  "/tmp/chef-script20200423-20704-1p9595h" returned 1e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb
e[0m    
e[0m    107: ndb_mysql_basic "create_users_grants" do
e[0m    108:   action :install_grants
e[0m    109:   my_ip my_ip
e[0m    110: end
e[0m    111: 
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb:107:in `from_file'
e[0m    
e[0m    ndb_mysql_basic("create_users_grants") do
e[0m      action [:install_grants]
e[0m      updated true
e[0m      updated_by_last_action true
e[0m      default_guard_interpreter :default
e[0m      declared_type :ndb_mysql_basic
e[0m      cookbook_name "ndb"
e[0m      recipe_name "mysqld"
e[0m      my_ip "10.0.2.15"
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=15.7.32
e[0m    platform=ubuntu
e[0m    platform_version=18.04
e[0m    ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m    program_name=/usr/bin/chef-solo
e[0m    executable=/opt/chefdk/bin/chef-soloe[0m
    
e[0me[0m
Running handlers:e[0m
[2020-04-23T11:22:26+02:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2020-04-23T11:22:26+02:00] ERROR: Exception handlers complete
Chef Infra Client failed. 5 resources updated in 01 hours 13 minutes 20 secondse[0m
[2020-04-23T11:22:26+02:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-04-23T11:22:26+02:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-04-23T11:22:26+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: ndb_mysql_basic[create_users_grants] (ndb::mysqld line 107) had an error: Mixlib::ShellOut::ShellCommandFailed: ndb_mysql_basic[mysqld_start_grants] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 8) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[wait_mysqld_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
STDOUT: 
STDERR: amysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/srv/hops/mysql-cluster/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/srv/hops/mysql-cluster/mysql.sock' exists!
---- End output of "bash"  "/tmp/chef-script20200423-20704-1p9595h" ----
Ran "bash"  "/tmp/chef-script20200423-20704-1p9595h" returned 1
#13

It looks like your MySQL daemon is not running. systemctl status mysqld will print the status of the service. You can check the logs at /srv/hops/mysql-cluster/log

Also, make sure NDB nodes are up and running with /srv/hops/mysql-cluster/ndb/scripts/mgm-client.sh -e show There should not be any NDB nodes in state connecting or not connected

#14

I executed the following commands

nikokks@nikokks:~$ systemctl status mysqld
● mysqld.service - Mysql Server
   Loaded: loaded (/lib/systemd/system/mysqld.service; enabled; vendor preset: enabled)
   Active: activating (start) since Thu 2020-04-23 10:11:04 CEST; 1h 17min ago
  Process: 20898 ExecStart=/srv/hops/mysql-cluster/ndb/scripts/mysql-server-start.sh (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/mysqld.service

avril 23 10:11:04 nikokks systemd[1]: Starting Mysql Server...
avril 23 10:11:04 nikokks mysql-server-start.sh[20898]: Testing if a mysql server is already running on this host...
avril 23 10:11:04 nikokks systemd[1]: mysqld.service: Can't open PID file /srv/hops/mysql-cluster/log/mysql_52.pid (yet?) after start: No such file or directory

and this

root@nikokks:/srv/hops/mysql-cluster/log# /srv/hops/mysql-cluster/ndb/scripts/mgm-client.sh -e show 
Connected to Management Server at: 10.0.2.15:1186
Cluster Configuration
---------------------
[ndbd(NDB)]	1 node(s)
id=1	@10.0.2.15  (mysql-5.7.25 ndb-7.6.9, Nodegroup: 0, *)

[ndb_mgmd(MGM)]	1 node(s)
id=49	@10.0.2.15  (mysql-5.7.25 ndb-7.6.9)

[mysqld(API)]	13 node(s)
id=52 (not connected, accepting connect from 10.0.2.15)
id=190 (not connected, accepting connect from 10.0.2.15)
id=191 (not connected, accepting connect from 10.0.2.15)
id=192 (not connected, accepting connect from any host)
id=193 (not connected, accepting connect from any host)
id=194 (not connected, accepting connect from any host)
id=195 (not connected, accepting connect from any host)
id=196 (not connected, accepting connect from any host)
id=197 (not connected, accepting connect from any host)
id=198 (not connected, accepting connect from any host)
id=199 (not connected, accepting connect from any host)
id=200 (not connected, accepting connect from any host)
id=201 (not connected, accepting connect from any host)

in /srv/hops/mysql-cluster/log/ i found various logs:

root@nikokks:/srv/hops/mysql-cluster/log# ls
cluster.log  mysql_52_out.log  ndb_1_out.log  ndb_1.pid  ndb_49_out.log  ndb_49.pid
#15

Actually, I am trying to relaunch the installation stopped at ndb::mysqld after running these commands.

cd /srv/hops/mysql-cluster
touch mysqld.sock
chmod +x mysqld.sock
chown mysql:mysql -R *
cd /tmp
ln -s /var/run/mysqld/mysqld.sock mysqld.sock
/etc/init.d/mysql start

I found a symbolic link of a socket in /var/run/mysqld/mysqld.socket.
Moreover, the duration of the step ndb::mysqld is abnormally long (it is taking actually more than 2 hours).
Do you have an idea ?

#16

It ended with these errors after having create a socket.

[sudo] password for nikokks: Starting Chef Infra Client, version 15.7.32e[0m
resolving cookbooks for run list: ["ndb::mysqld"]e[0m
Synchronizing Cookbooks:e[0m
  - ndb (1.3.0)e[0m
  - kagent (1.3.0)e[0m
  - ntp (2.0.3)e[0m
  - ulimit (1.3.0)e[0m
  - magic_shell (1.0.0)e[0m
  - sudo (4.0.1)e[0m
  - conda (1.3.0)e[0m
  - openssl (4.4.0)e[0m
  - hostsfile (2.4.6)e[0m
  - chef-sugar (5.1.8)e[0m
  - windows (7.0.0)e[0m
  - homebrew (5.0.8)e[0m
  - java (7.0.0)e[0m
Installing Cookbook Gems:e[0m
Compiling Cookbooks...e[0m
[2020-04-23T14:15:13+02:00] WARN: Resource openssl_dhparam from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-23T14:15:13+02:00] WARN: Resource openssl_rsa_key from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-23T14:15:13+02:00] WARN: Resource openssl_x509 from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
[2020-04-23T14:15:13+02:00] WARN: Resource sudo from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
Converging 29 resourcese[0m
Recipe: ndb::mysqlde[0m
  * apt_package[libaio1] action install (up to date)
  * directory[/srv/hops/mysql-cluster/mysql] action create
    e[32m- change mode from '0750' to '0700'e[0m
e[0m  * template[/srv/hops/mysql-cluster/ndb/scripts/get-mysql-socket.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/get-mysql-port.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-start.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-stop.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-server-restart.sh] action create (up to date)
  * template[/srv/hops/mysql-cluster/ndb/scripts/mysql-client.sh] action create (up to date)
  * template[/lib/systemd/system/mysqld.service] action create (up to date)
  * service[mysqld] action nothing (skipped due to action :nothing)
  * template[/srv/hops/mysql-cluster/my.cnf] action create (up to date)
  * bash[mysql_install_db] action run
    e[32m- execute "bash"  "/tmp/chef-script20200423-25602-92e2tc"e[0m
e[0m  * kagent_config[mysqld] action systemd_reload
    * bash[start-if-not-running-mysqld] action run[2020-04-23T15:43:56+02:00] ERROR: bash[start-if-not-running-mysqld] (/tmp/chef-solo/cookbooks/kagent/providers/config.rb line 49) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command exceeded allowed execution time, process terminated
---- Begin output of "bash"  "/tmp/chef-script20200423-25602-ityeys" ----
STDOUT: 
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200423-25602-ityeys" ----
Ran "bash"  "/tmp/chef-script20200423-25602-ityeys" returned ; ignore_failure is set, continuing

      e[0m
      ================================================================================e[0m
      e[31mError executing action `run` on resource 'bash[start-if-not-running-mysqld]'e[0m
      ================================================================================e[0m
      
e[0m      Mixlib::ShellOut::CommandTimeoute[0m
      --------------------------------e[0m
      Command timed out after 3600s:
e[0m      Command exceeded allowed execution time, process terminated
e[0m      ---- Begin output of "bash"  "/tmp/chef-script20200423-25602-ityeys" ----
e[0m      STDOUT: 
e[0m      STDERR: 
e[0m      ---- End output of "bash"  "/tmp/chef-script20200423-25602-ityeys" ----
e[0m      Ran "bash"  "/tmp/chef-script20200423-25602-ityeys" returned e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /tmp/chef-solo/cookbooks/kagent/providers/config.rb
e[0m      
e[0m       49:   bash "start-if-not-running-#{new_resource.name}" do
e[0m       50:     user "root"
e[0m       51:     ignore_failure true
e[0m       52:     code <<-EOH
e[0m       53:      systemctl stop #{new_resource.name}
e[0m       54:      systemctl daemon-reload
e[0m       55:      systemctl reset-failed
e[0m       56:      systemctl start #{new_resource.name}
e[0m       57:     EOH
e[0m       58:   end
e[0m       59: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /tmp/chef-solo/cookbooks/kagent/providers/config.rb:49:in `block in class_from_file'
e[0m      
e[0m      bash("start-if-not-running-mysqld") do
e[0m        action [:run]
e[0m        default_guard_interpreter :default
e[0m        command nil
e[0m        backup 5
e[0m        interpreter "bash"
e[0m        declared_type :bash
e[0m        cookbook_name "ndb"
e[0m        user "root"
e[0m        code "     systemctl stop mysqld\n     systemctl daemon-reload\n     systemctl reset-failed\n     systemctl start mysqld\n"
e[0m        domain nil
e[0m        ignore_failure true
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.7.32
e[0m      platform=ubuntu
e[0m      platform_version=18.04
e[0m      ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m      program_name=/usr/bin/chef-solo
e[0m      executable=/opt/chefdk/bin/chef-soloe[0m
      
e[0m     (up to date)
  * ndb_mysql_basic[create_users_grants] action install_grants
    * ndb_waiter[wait_ndb_started] action wait_until_cluster_ready
      * bash[wait_ndb_started] action run
        e[0m
        ================================================================================e[0m
        e[31mError executing action `run` on resource 'bash[wait_ndb_started]'e[0m
        ================================================================================e[0m
        
e[0m        Mixlib::ShellOut::CommandTimeoute[0m
        --------------------------------e[0m
        Command timed out after 3600s:
e[0m        Command exceeded allowed execution time, process terminated
e[0m        ---- Begin output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
e[0m        STDOUT: 
e[0m        STDERR: 
e[0m        ---- End output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
e[0m        Ran "bash"  "/tmp/chef-script20200423-25602-jud2xt" returned e[0m
        
e[0m        Resource Declaration:e[0m
        ---------------------e[0m
        # In /tmp/chef-solo/cookbooks/ndb/providers/waiter.rb
e[0m        
e[0m          3:   bash new_resource.name do
e[0m          4:     user node['ndb']['user']
e[0m          5:     code <<-EOF
e[0m          6:       #{node['mysql']['base_dir']}/bin/ndb_waiter -c #{node['ndb']['connectstring']} --timeout=#{node['ndb']['wait_startup']}  2>&1 > /dev/null
e[0m          7:     EOF
e[0m          8:   end
e[0m          9:   new_resource.updated_by_last_action(true)
e[0m        
e[0m        Compiled Resource:e[0m
        ------------------e[0m
        # Declared in /tmp/chef-solo/cookbooks/ndb/providers/waiter.rb:3:in `block in class_from_file'
e[0m        
e[0m        bash("wait_ndb_started") do
e[0m          action [:run]
e[0m          default_guard_interpreter :default
e[0m          command nil
e[0m          backup 5
e[0m          interpreter "bash"
e[0m          declared_type :bash
e[0m          cookbook_name "ndb"
e[0m          user "mysql"
e[0m          code "      /srv/hops/mysql/bin/ndb_waiter -c 10.0.2.15:1186 --timeout=10800  2>&1 > /dev/null\n"
e[0m          domain nil
e[0m        end
e[0m        
e[0m        System Info:e[0m
        ------------e[0m
        chef_version=15.7.32
e[0m        platform=ubuntu
e[0m        platform_version=18.04
e[0m        ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m        program_name=/usr/bin/chef-solo
e[0m        executable=/opt/chefdk/bin/chef-soloe[0m
        
e[0m      e[0m
      ================================================================================e[0m
      e[31mError executing action `wait_until_cluster_ready` on resource 'ndb_waiter[wait_ndb_started]'e[0m
      ================================================================================e[0m
      
e[0m      Mixlib::ShellOut::CommandTimeoute[0m
      --------------------------------e[0m
      bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
e[0m      Command exceeded allowed execution time, process terminated
e[0m      ---- Begin output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
e[0m      STDOUT: 
e[0m      STDERR: 
e[0m      ---- End output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
e[0m      Ran "bash"  "/tmp/chef-script20200423-25602-jud2xt" returned e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb
e[0m      
e[0m        3:   ndb_waiter "wait_ndb_started" do
e[0m        4:     action :wait_until_cluster_ready
e[0m        5:     only_if { node['ndb']['enabled'] == "true" }
e[0m        6:   end
e[0m        7: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb:3:in `block in class_from_file'
e[0m      
e[0m      ndb_waiter("wait_ndb_started") do
e[0m        action [:wait_until_cluster_ready]
e[0m        updated true
e[0m        updated_by_last_action true
e[0m        default_guard_interpreter :default
e[0m        declared_type :ndb_waiter
e[0m        cookbook_name "ndb"
e[0m        only_if { #code block }
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.7.32
e[0m      platform=ubuntu
e[0m      platform_version=18.04
e[0m      ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m      program_name=/usr/bin/chef-solo
e[0m      executable=/opt/chefdk/bin/chef-soloe[0m
      
e[0m    e[0m
    ================================================================================e[0m
    e[31mError executing action `install_grants` on resource 'ndb_mysql_basic[create_users_grants]'e[0m
    ================================================================================e[0m
    
e[0m    Mixlib::ShellOut::CommandTimeoute[0m
    --------------------------------e[0m
    ndb_waiter[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 3) had an error: Mixlib::ShellOut::CommandTimeout: bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
e[0m    Command exceeded allowed execution time, process terminated
e[0m    ---- Begin output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
e[0m    STDOUT: 
e[0m    STDERR: 
e[0m    ---- End output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
e[0m    Ran "bash"  "/tmp/chef-script20200423-25602-jud2xt" returned e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb
e[0m    
e[0m    107: ndb_mysql_basic "create_users_grants" do
e[0m    108:   action :install_grants
e[0m    109:   my_ip my_ip
e[0m    110: end
e[0m    111: 
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /tmp/chef-solo/cookbooks/ndb/recipes/mysqld.rb:107:in `from_file'
e[0m    
e[0m    ndb_mysql_basic("create_users_grants") do
e[0m      action [:install_grants]
e[0m      updated true
e[0m      updated_by_last_action true
e[0m      default_guard_interpreter :default
e[0m      declared_type :ndb_mysql_basic
e[0m      cookbook_name "ndb"
e[0m      recipe_name "mysqld"
e[0m      my_ip "10.0.2.15"
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=15.7.32
e[0m    platform=ubuntu
e[0m    platform_version=18.04
e[0m    ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
e[0m    program_name=/usr/bin/chef-solo
e[0m    executable=/opt/chefdk/bin/chef-soloe[0m
    
e[0me[0m
Running handlers:e[0m
[2020-04-23T16:54:58+02:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2020-04-23T16:54:58+02:00] ERROR: Exception handlers complete
Chef Infra Client failed. 2 resources updated in 02 hours 44 minutes 16 secondse[0m
[2020-04-23T16:54:58+02:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-04-23T16:54:58+02:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-04-23T16:54:58+02:00] FATAL: Mixlib::ShellOut::CommandTimeout: ndb_mysql_basic[create_users_grants] (ndb::mysqld line 107) had an error: Mixlib::ShellOut::CommandTimeout: ndb_waiter[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/mysql_basic.rb line 3) had an error: Mixlib::ShellOut::CommandTimeout: bash[wait_ndb_started] (/tmp/chef-solo/cookbooks/ndb/providers/waiter.rb line 3) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command exceeded allowed execution time, process terminated
---- Begin output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
STDOUT: 
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200423-25602-jud2xt" ----
Ran "bash"  "/tmp/chef-script20200423-25602-jud2xt" returned
1 Like
#17

I found that this command

systemctl start mysqld

doesn’t run right.

#18

From the output of the NDB management server I can see that mysql is not running. Try systemctl restart mysqld The logs for mysql server is mysql_52_out.log

#19

the same for
systemctl restart mysqld

#20

Yes but is there anything meaningful in the logs?