Can't connect to MySQL server on machine

Hi,

I’m in the installation steps On-promise single installation. I’m using Karamel for that.
When it arrive to install "hops_airflow::default" it failed with the following error message:

Can’t connect to MySQL server on machine:

MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'My_machine' (111)")

Knowing that I have mysql is installed in my machine, but I can access on it only using super user privilege: sudo mysql when I try using only mysql I got this error message:

ERROR 1045 (28000): Access denied for user my_machine' (using password: NO)

Is the root cause because I’m using the sudo to access on mysql ? Should I disable it ?

Thanks to @Fabio and @antonios
Best regards,
Houda

Hi @smach_h,

Do you mean there was a MySQL server installed and running prior to starting the installation with Karamel? How many machines are used for the Hopsworks installation?

More log output will be available under /home/<install_user>/.karamel/install.hops_airflow__default.log

What the address listed if you do sudo netstat -ltpnu | grep 3306 ?

A simpler installation process compared to using Karamel directly would be to use hopsworks_installer https://hopsworks.readthedocs.io/en/latest/getting_started/installation_guide/platforms/baremetal.html#quickstart-single-host-installation

Hi @Theo,

Thanks for your answer.

Do you mean there was a MySQL server installed and running prior to starting the installation with Karamel? ==> I think yess, because I have Mysql server is installed.

The result of the command `dpkg --get-selections | grep mysql`:

libmysqlclient-dev				    install
libmysqlclient20:amd64				install
mysql-client					    install
mysql-client-5.7				    install
mysql-client-core-5.7			 	install
mysql-common					    install
mysql-server					    install
mysql-server-5.7				    install
mysql-server-core-5.7				install

The result of the command : sudo netstat -ltpnu | grep 3306

tcp        0      0 192.168.4.39:3306       0.0.0.0:*               LISTEN      13272/mysqld        
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      7404/mysqld    

The result of the command sudo service mysql status

● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-07-07 10:23:34 CEST; 1 day 3h ago
 Main PID: 7404 (mysqld)
    Tasks: 29 (limit: 4915)
   CGroup: /system.slice/mysql.service
           └─7404 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid

juil. 07 10:23:33 brooks-MS-7B98 systemd[1]: Starting MySQL Community Server...
juil. 07 10:23:34 brooks-MS-7B98 systemd[1]: Started MySQL Community Server.

I’m using only one machine for the Hopsworks installation.
I’m using already the link that you sent for the installation.
If I have a Mysql server installed, is impacted the installation with Karamel ?

Thanks,
Houda

Thank you, can you also post the output of /home/<install_user>/.karamel/install.hops_airflow__default.log to see to which MySQL server Airflow is trying to connect to?

I found 2 files, the first one is: hops_airflow__install.log:

Starting Chef Client, version 14.10.9e[0m
resolving cookbooks for run list: ["hops_airflow::install"]e[0m
Synchronizing Cookbooks:e[0m
  - hops_airflow (1.3.0)e[0m
  - apt (7.2.0)e[0m
  - poise-python (1.7.0)e[0m
  - java (7.0.0)e[0m
  - kagent (1.3.0)e[0m
  - hops (1.3.0)e[0m
  - ndb (1.3.0)e[0m
  - conda (1.3.0)e[0m
  - hive2 (1.3.0)e[0m
  - poise (2.8.2)e[0m
  - homebrew (5.0.8)e[0m
  - poise-languages (2.1.2)e[0m
  - windows (7.0.0)e[0m
  - openssl (4.4.0)e[0m
  - hostsfile (2.4.6)e[0m
  - ntp (2.0.3)e[0m
  - sudo (4.0.1)e[0m
  - magic_shell (1.0.0)e[0m
  - sysctl (1.0.5)e[0m
  - cmake (0.3.0)e[0m
  - kzookeeper (1.3.0)e[0m
  - ulimit (1.3.0)e[0m
  - elastic (1.3.0)e[0m
  - consul (1.3.0)e[0m
  - compat_resource (12.19.1)e[0m
  - authbind (0.1.10)e[0m
  - chef-sugar (5.1.8)e[0m
  - poise-archive (1.5.0)e[0m
  - ohai (5.3.0)e[0m
  - ulimit2 (0.2.0)e[0m
  - elasticsearch (4.0.6)e[0m
  - yum (5.1.0)e[0m
  - ark (5.0.0)e[0m
  - seven_zip (3.1.2)e[0m
Installing Cookbook Gems:e[0m
Compiling Cookbooks...e[0m
Converging 20 resourcese[0m
Recipe: apt::defaulte[0m
  * file[/var/lib/apt/periodic/update-success-stamp] action nothing (skipped due to action :nothing)
  * apt_update[periodic] action periodic (up to date)
  * execute[apt-get update] action nothing (skipped due to action :nothing)
  * execute[apt-get autoremove] action nothing (skipped due to action :nothing)
  * execute[apt-get autoclean] action nothing (skipped due to action :nothing)
  * directory[/var/cache/local] action create (up to date)
  * directory[/var/cache/local/preseeding] action create (up to date)
  * template[/etc/apt/apt.conf.d/10dpkg-options] action create (up to date)
  * template[/etc/apt/apt.conf.d/10recommends] action create (up to date)
  * apt_package[apt-transport-https, gnupg, dirmngr] action install (up to date)
Recipe: hops_airflow::usere[0m
  * group[airflow] action create (skipped due to not_if)
  * linux_user[airflow] action create (skipped due to not_if)
  * group[hadoop] action create (skipped due to not_if)
  * group[hadoop] action modify (up to date)
Recipe: hops_airflow::directoriese[0m
  * directory[/srv/hops] action create (skipped due to not_if)
  * directory[/srv/hops/airflow] action create (up to date)
  * directory[/srv/hops/airflow/dags] action create
    e[32m- change mode from '0770' to '0710'e[0m
e[0m  * directory[/srv/hops/airflow/plugins] action create (up to date)
  * directory[/srv/hops/airflow/run] action create (up to date)
  * directory[/srv/hops/airflow/secrets] action create (up to date)
e[0m
Running handlers:e[0m
Running handlers complete
e[0me[0m
Deprecated features used!e[0m
  Resource openssl_dhparam from a cookbook is overriding the resource from the client. Please upgrade your cookbook or remove the cookbook from your run_list before the next major release of Chef. at 1 location:e[0m
    - /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/log.rb:51:in `caller_location'e[0m
   See https://docs.chef.io/deprecations_map_collision.html for further details.e[0m
  Resource openssl_rsa_key from a cookbook is overriding the resource from the client. Please upgrade your cookbook or remove the cookbook from your run_list before the next major release of Chef. at 1 location:e[0m
    - /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/log.rb:51:in `caller_location'e[0m
   See https://docs.chef.io/deprecations_map_collision.html for further details.e[0m
  Resource sudo from a cookbook is overriding the resource from the client. Please upgrade your cookbook or remove the cookbook from your run_list before the next major release of Chef. at 1 location:e[0m
    - /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/log.rb:51:in `caller_location'e[0m
   See https://docs.chef.io/deprecations_map_collision.html for further details.e[0m
  Resource sysctl_param from a cookbook is overriding the resource from the client. Please upgrade your cookbook or remove the cookbook from your run_list before the next major release of Chef. at 1 location:e[0m
    - /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/log.rb:51:in `caller_location'e[0m
   See https://docs.chef.io/deprecations_map_collision.html for further details.e[0m
e[0m
Chef Client finished, 1/20 resources updated in 06 secondse[0m

The second file is: hops_airflow_default.log, is long a little:

Starting Chef Client, version 14.10.9e[0m
resolving cookbooks for run list: ["hops_airflow::default"]e[0m
Installing Cookbook Gems:e[0m
Compiling Cookbooks...e[0m
Converging 89 resourcese[0m
Recipe: hops_airflow::dbe[0m
  * bash[create_airflow_db] action run (skipped due to not_if)
  * cookbook_file[/home/airflow/create_db_idx_proc.sql] action create (up to date)
  * bash[import_create_idx_proc] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-14t4szf"e[0m
e[0mRecipe: hops_airflow::packagese[0m
  * apt_package[python-dev] action install (up to date)
  * apt_package[build-essential] action install (up to date)
  * apt_package[libssl-dev] action install (up to date)
  * apt_package[libffi-dev] action install (up to date)
  * apt_package[libffi-dev] action install (up to date)
  * apt_package[libkrb5-dev] action install (up to date)
  * apt_package[libsasl2-dev] action install (up to date)
  * apt_package[mysql-client] action install (up to date)
  * apt_package[libmysqlclient-dev] action install (up to date)
  * apt_package[libldap2-dev] action install (up to date)
  * apt_package[libffi-dev] action install (up to date)
  * bash[remove_airflow_env] action run
  
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-11e2p52"e[0m
e[0m  * bash[install_python__hive-thrift-py] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-1xtig1k"e[0m
e[0m  * bash[install_python__pyhive] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-eebxm8"e[0m
e[0m  * bash[install_python__markdown] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-fygkhy"e[0m
e[0m  * bash[install_python__futures] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-8kf8r1"e[0m
e[0m  * bash[install_python__werkzeug] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-18sbw37"e[0m
e[0m  * bash[install_python__impyla] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-besuoq"e[0m
e[0m  * bash[install_python__unicodecsv] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-q3qmvj"e[0m
e[0m  * bash[install_python__jaydebeapi] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-1jzlcke"e[0m
e[0m  * bash[install_python__mysqlclient] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-11u7lkk"e[0m
e[0m  * bash[install_python__librabbitmq] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-11n7kjh"e[0m
e[0m  * bash[install_python__boto] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-c0nghj"e[0m
e[0m  * bash[install_python__filechunkio] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-5zali"e[0m
e[0m  * bash[install_python__pysmbclient] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-1a2m07q"e[0m
e[0m  * bash[install_python__slackclient] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-85v0eu"e[0m
e[0m  * bash[install_python__statsd] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-7npp8v"e[0m
e[0m  * bash[install_python__ldap3] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-6glhds"e[0m
e[0m  * bash[install_python__bcrypt] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-1yzu27n"e[0m
e[0m  * bash[install_python__flask-bcrypt] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-bv7ue2"e[0m
e[0m  * bash[install_python__Flask-OAuthlib] action run
    e[32m- execute "bash"  "/tmp/chef-script20200708-27417-jmfa43"e[0m
e[0mRecipe: hops_airflow::defaulte[0m
  * group[airflow] action modify (up to date)
Recipe: hops_airflow::confige[0m
  * template[/srv/hops/airflow/airflow.cfg] action create (up to date)
Recipe: hops_airflow::servicese[0m
  * template[airflow_services_env] action create
    e[32m- change owner from 'airflow' to 'root'e[0m
e[0m  * template[/srv/hops/airflow/airflow_runner.sh] action create (up to date)
Recipe: hops_airflow::defaulte[0m
  * directory[/srv/hops/airflow/plugins] action create (up to date)
  * directory[/srv/hops/airflow/dags] action create (up to date)
  * template[airflow_services_env] action create
    e[32m- change owner from 'root' to 'airflow'e[0m
e[0m  * bash[init_airflow_db] action run
    e[0m
    ================================================================================e[0m
    e[31mError executing action `run` on resource 'bash[init_airflow_db]'e[0m
    ================================================================================e[0m
    
e[0m    Mixlib::ShellOut::ShellCommandFailede[0m
    ------------------------------------e[0m
    Expected process to exit with [0], but received '1'
e[0m    ---- Begin output of "bash"  "/tmp/chef-script20200708-27417-i8kw0a" ----
e[0m    STDOUT: [2020-07-08 14:52:24,673] {settings.py:174} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=3600, pid=32151
e[0m    [2020-07-08 14:52:24,842] {__init__.py:51} INFO - Using executor LocalExecutor
e[0m    [2020-07-08 14:52:24,971] {cli_action_loggers.py:69} ERROR - Failed on pre-execution callback using <function default_action_log at 0x7fbb5664dae8>
e[0m    Traceback (most recent call last):
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
e[0m        return fn()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 434, in connect
e[0m        return _ConnectionFairy._checkout(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
e[0m        fairy = _ConnectionRecord.checkout(pool)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
e[0m        rec = pool._do_get()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
e[0m        self._dec_overflow()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
e[0m        compat.reraise(exc_type, exc_value, exc_tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
e[0m        raise value
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
e[0m        return self._create_connection()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
e[0m        return _ConnectionRecord(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
e[0m        self.__connect(first_connect_check=True)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
e[0m        connection = pool._invoke_creator(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
e[0m        return dialect.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
e[0m        return self.dbapi.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 130, in Connect
e[0m        return Connection(*args, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 185, in __init__
e[0m        super().__init__(*args, **kwargs2)
e[0m    MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'brooks-MS-7B98' (111)")
e[0m    
e[0m    The above exception was the direct cause of the following exception:
e[0m    
e[0m    Traceback (most recent call last):
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 67, in on_pre_execution
e[0m        cb(**kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 98, in default_action_log
e[0m        session.add(log)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/contextlib.py", line 88, in __exit__
e[0m        next(self.gen)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py", line 44, in create_session
e[0m        session.commit()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1023, in commit
e[0m        self.transaction.commit()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 487, in commit
e[0m        self._prepare_impl()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 466, in _prepare_impl
e[0m        self.session.flush()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2446, in flush
e[0m        self._flush(objects)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2584, in _flush
e[0m        transaction.rollback(_capture_exception=True)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
e[0m        compat.reraise(exc_type, exc_value, exc_tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
e[0m        raise value
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2544, in _flush
e[0m        flush_context.execute()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute
e[0m        rec.execute(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute
e[0m        uow,
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 213, in save_obj
e[0m        ) in _organize_states_for_save(base_mapper, states, uowtransaction):
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 374, in _organize_states_for_save
e[0m        base_mapper, uowtransaction, states
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1558, in _connections_for_states
e[0m        connection = uowtransaction.transaction.connection(base_mapper)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 306, in connection
e[0m        return self._connection_for_bind(bind, execution_options)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 412, in _connection_for_bind
e[0m        conn = self._parent._connection_for_bind(bind, execution_options)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 424, in _connection_for_bind
e[0m        conn = bind.contextual_connect()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2192, in contextual_connect
e[0m        self._wrap_pool_connect(self.pool.connect, None),
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect
e[0m        e, dialect, self
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1528, in _handle_dbapi_exception_noconnection
e[0m        util.raise_from_cause(sqlalchemy_exception, exc_info)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
e[0m        reraise(type(exception), exception, tb=exc_tb, cause=cause)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
e[0m        raise value.with_traceback(tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
e[0m        return fn()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 434, in connect
e[0m        return _ConnectionFairy._checkout(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
e[0m        fairy = _ConnectionRecord.checkout(pool)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
e[0m        rec = pool._do_get()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
e[0m        self._dec_overflow()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
e[0m        compat.reraise(exc_type, exc_value, exc_tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
e[0m        raise value
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
e[0m        return self._create_connection()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
e[0m        return _ConnectionRecord(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
e[0m        self.__connect(first_connect_check=True)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
e[0m        connection = pool._invoke_creator(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
e[0m        return dialect.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
e[0m        return self.dbapi.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 130, in Connect
e[0m        return Connection(*args, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 185, in __init__
e[0m        super().__init__(*args, **kwargs2)
e[0m    sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'brooks-MS-7B98' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
e[0m    DB: mysql://airflow_db:***@brooks-MS-7B98:3306/airflow
e[0m    [2020-07-08 14:52:25,018] {db.py:338} INFO - Creating tables
e[0m    STDERR: Traceback (most recent call last):
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
e[0m        return fn()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 374, in unique_connection
e[0m        return _ConnectionFairy._checkout(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
e[0m        fairy = _ConnectionRecord.checkout(pool)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
e[0m        rec = pool._do_get()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
e[0m        self._dec_overflow()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
e[0m        compat.reraise(exc_type, exc_value, exc_tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
e[0m        raise value
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
e[0m        return self._create_connection()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
e[0m        return _ConnectionRecord(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
e[0m        self.__connect(first_connect_check=True)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
e[0m        connection = pool._invoke_creator(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
e[0m        return dialect.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
e[0m        return self.dbapi.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 130, in Connect
e[0m        return Connection(*args, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 185, in __init__
e[0m        super().__init__(*args, **kwargs2)
e[0m    MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'brooks-MS-7B98' (111)")
e[0m    
e[0m    The above exception was the direct cause of the following exception:
e[0m    
e[0m    Traceback (most recent call last):
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/bin/airflow", line 32, in <module>
e[0m        args.func(args)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
e[0m        return f(*args, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/bin/cli.py", line 1091, in upgradedb
e[0m        db.upgradedb()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py", line 346, in upgradedb
e[0m        command.upgrade(config, 'heads')
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/command.py", line 174, in upgrade
e[0m        script.run_env()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
e[0m        util.load_python_file(self.dir, 'env.py')
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
e[0m        module = load_module_py(module_id, path)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/alembic/util/compat.py", line 68, in load_module_py
e[0m        module_id, path).load_module(module_id)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py", line 92, in <module>
e[0m        run_migrations_online()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py", line 78, in run_migrations_online
e[0m        with connectable.connect() as connection:
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2171, in connect
e[0m        return self._connection_cls(self, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
e[0m        else engine.raw_connection()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2259, in raw_connection
e[0m        self.pool.unique_connection, _connection
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect
e[0m        e, dialect, self
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1528, in _handle_dbapi_exception_noconnection
e[0m        util.raise_from_cause(sqlalchemy_exception, exc_info)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
e[0m        reraise(type(exception), exception, tb=exc_tb, cause=cause)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
e[0m        raise value.with_traceback(tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
e[0m        return fn()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 374, in unique_connection
e[0m        return _ConnectionFairy._checkout(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
e[0m        fairy = _ConnectionRecord.checkout(pool)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
e[0m        rec = pool._do_get()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
e[0m        self._dec_overflow()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
e[0m        compat.reraise(exc_type, exc_value, exc_tb)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
e[0m        raise value
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
e[0m        return self._create_connection()
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
e[0m        return _ConnectionRecord(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
e[0m        self.__connect(first_connect_check=True)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
e[0m        connection = pool._invoke_creator(self)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
e[0m        return dialect.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
e[0m        return self.dbapi.connect(*cargs, **cparams)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 130, in Connect
e[0m        return Connection(*args, **kwargs)
e[0m      File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 185, in __init__
e[0m        super().__init__(*args, **kwargs2)
e[0m    sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'brooks-MS-7B98' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
e[0m    ---- End output of "bash"  "/tmp/chef-script20200708-27417-i8kw0a" ----
e[0m    Ran "bash"  "/tmp/chef-script20200708-27417-i8kw0a" returned 1e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /tmp/chef-solo/cookbooks/hops_airflow/recipes/default.rb
e[0m    
e[0m     65: bash 'init_airflow_db' do
e[0m     66:   user node['airflow']['user']
e[0m     67:   code <<-EOF
e[0m     68:       set -e
e[0m     69:       export AIRFLOW_HOME=#{node['airflow']['base_dir']}
e[0m     70:       #{node['airflow']['bin_path']}/airflow upgradedb
e[0m     71:     EOF
e[0m     72: end
e[0m     73: 
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /tmp/chef-solo/cookbooks/hops_airflow/recipes/default.rb:65:in `from_file'
e[0m    
e[0m    bash("init_airflow_db") do
e[0m      action [:run]
e[0m      default_guard_interpreter :default
e[0m      command nil
e[0m      backup 5
e[0m      interpreter "bash"
e[0m      declared_type :bash
e[0m      cookbook_name "hops_airflow"
e[0m      recipe_name "default"
e[0m      user "airflow"
e[0m      code "      set -e\n      export AIRFLOW_HOME=/srv/hops/airflow\n      /srv/hops/anaconda/anaconda/envs/airflow/bin/airflow upgradedb\n"
e[0m      domain nil
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=14.10.9
e[0m    platform=ubuntu
e[0m    platform_version=18.04
e[0m    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m    program_name=/usr/bin/chef-solo
e[0m    executable=/opt/chefdk/bin/chef-soloe[0m
    
e[0me[0m
Running handlers:e[0m
[2020-07-08T14:52:25+02:00] ERROR: Running exception handlers
Running handlers complete
e[0m[2020-07-08T14:52:25+02:00] ERROR: Exception handlers complete
Chef Client failed. 54 resources updated in 03 minutes 56 secondse[0m
[2020-07-08T14:52:25+02:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-07-08T14:52:25+02:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-07-08T14:52:25+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[init_airflow_db] (hops_airflow::default line 65) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200708-27417-i8kw0a" ----
STDOUT: [2020-07-08 14:52:24,673] {settings.py:174} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=3600, pid=32151
[2020-07-08 14:52:24,842] {__init__.py:51} INFO - Using executor LocalExecutor
[2020-07-08 14:52:24,971] {cli_action_loggers.py:69} ERROR - Failed on pre-execution callback using <function default_action_log at 0x7fbb5664dae8>
Traceback (most recent call last):
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
    return fn()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 434, in connect
    return _ConnectionFairy._checkout(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
    rec = pool._do_get()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
    self._dec_overflow()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
    raise value
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
    return self._create_connection()
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
    return _ConnectionRecord(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
    self.__connect(first_connect_check=True)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
    connection = pool._invoke_creator(self)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/__init__.py", line 130, in Connect
    return Connection(*args, **kwargs)
  File "/srv/hops/anaconda/anaconda/envs/airflow/lib/python3.6/site-packages/MySQLdb/connections.py", line 185, in __init__
    super().__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'brooks-MS-7B98' (111)")

The above exception was the direct cause of the following exception:

You could change the grants to the MySQL server to allow connections from all hosts and users but in general it is not recommended to have an existing MySQL server where Hopsworks is installed as Hopsworks will create MySQL users, databases, grants etc. At this stage it might be easier to purge the MySQL installation and start a fresh installation.