Error installing in Azure VM Single Host

Tangeda_Sai_Sharan · June 7, 2020, 7:09am

I’m trying to install Hopsworks in Azure VM and Installation always stops at hops_airflow_default.sh
I have tried removing password for sudo and using --password argument with password on
Also chef version is latest, and VM satisfies all the ram, name and space requirements
Below is the error log, please help
“run_list”: [

"hops_airflow::default"
  ]
}
END_OF_FILE
sudo chef-solo -c /home/sai/.karamel/install/solo.rb -j /home/sai/.karamel/install/hops_airflow__default.json 2>&1 | tee hops_airflow__default.log
echo 'https://github.com/logicalclocks/airflow-chef/tree/1.3/hops_airflow::default' >> succeed_list
' > hops_airflow__default.sh ; chmod +x hops_airflow__default.sh ; ./hops_airflow__default.sh
', DAG is stuck here :(
INFO  [2020-06-06 17:34:59,775] se.kth.karamel.backend.machines.MachinesMonitor: Sending pause signal to all machines

Thanks

antonios · June 8, 2020, 8:39am

Hi.

Can you paste the content of /home/sai/.karamel/install/hops_airflow__default.log ? It contains more detailed information for the failed Chef recipe.

Kind regards,

Tangeda_Sai_Sharan · June 8, 2020, 9:01am

Starting Chef Client, version 14.10.9
resolving cookbooks for run list: ["hops_airflow::default"]
Synchronizing Cookbooks:
  - hops_airflow (1.3.0)
  - apt (7.2.0)
  - poise-python (1.7.0)
  - java (7.0.0)
  - kagent (1.3.0)
  - hops (1.3.0)
  - ndb (1.3.0)
  - conda (1.3.0)
  - hive2 (1.3.0)
  - poise (2.8.2)
  - poise-languages (2.1.2)
  - homebrew (5.0.8)
  - windows (7.0.0)
  - hostsfile (2.4.6)
  - openssl (4.4.0)
  - ntp (2.0.3)
  - sudo (4.0.1)
  - sysctl (1.0.5)
  - magic_shell (1.0.0)
  - cmake (0.3.0)
  - elastic (1.3.0)
  - kzookeeper (1.3.0)
  - ulimit (1.3.0)
  - consul (1.3.0)
  - authbind (0.1.10)
  - compat_resource (12.19.1)
  - poise-archive (1.5.0)
  - ohai (5.3.0)
  - chef-sugar (5.1.8)
  - ulimit2 (0.2.0)
  - elasticsearch (4.0.6)
  - yum (5.1.0)
  - ark (5.0.0)
  - seven_zip (3.1.2)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 89 resources
Recipe: hops_airflow::db
  * bash[create_airflow_db] action run (skipped due to not_if)
  * cookbook_file[/home/airflow/create_db_idx_proc.sql] action create (up to date)
  * bash[import_create_idx_proc] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1xgor4v"
Recipe: hops_airflow::packages
  * apt_package[python-dev] action install (up to date)
  * apt_package[build-essential] action install (up to date)
  * apt_package[libssl-dev] action install (up to date)
  * apt_package[libffi-dev] action install (up to date)
  * apt_package[libffi-dev] action install (up to date)
  * apt_package[libkrb5-dev] action install (up to date)
  * apt_package[libsasl2-dev] action install (up to date)
  * apt_package[mysql-client] action install (up to date)
  * apt_package[libmysqlclient-dev] action install (up to date)
  * apt_package[libldap2-dev] action install (up to date)
  * apt_package[libffi-dev] action install (up to date)
  * bash[remove_airflow_env] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-noz6v5"
  * bash[create_airflow_env] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1yygxwn"
  * bash[install_airflow] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1j2fz1j"
  * bash[install_airflow_hive] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-an2z6n"
  * bash[install_airflow_mysql] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-acueqi"
  * bash[install_airflow_kubernetes] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-gydu1s"
  * bash[install_airflow_password] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-kcj9my"
  * bash[install_airflow_hdfs] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1uytce7"
  * bash[install_airflow_slack] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-zbnhuj"
  * bash[install_airflow_ssh] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1dpjp9p"
  * bash[install_airflow_jdbc] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-15hkvr7"
  * bash[install_airflow_mysql] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1xl2bi9"
  * bash[install_airflow_crypto] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-11lhv9s"
  * bash[install_python__prometheus_client] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-139typ4"
  * bash[install_python__greenlet] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1brqm7f"
  * bash[install_python__eventlet] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1ffmkwr"
  * bash[install_python__celery] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-frbbs2"
  * bash[install_python__cryptography] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1sddg4r"
  * bash[install_python__pyjwt] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-x1f1c4"
  * bash[install_python__sphinx] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-vek3i3"
  * bash[install_python__sphinx-argparse] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-7gxrsy"
  * bash[install_python__sphinx-rtd-theme] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-siqoiz"
  * bash[install_python__Sphinx-PyPI-upload] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1pj2kcc"
  * bash[install_python__docker-py] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-7iagok"
  * bash[install_python__druid] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-3us4ab"
  * bash[install_python__boto3] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1qoxq0b"
  * bash[install_python__httplib2] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1o4f8jr"
  * bash[install_python__google-api-python-client] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-19f7f6l"
  * bash[install_python__oauth2client] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1bb9gaf"
  * bash[install_python__PyOpenSSL] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-qgwq57"
  * bash[install_python__hdfs[dataframe,avro,kerberos]] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1um6jnx"
  * bash[install_python__cython] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-2qkd29"
  * bash[install_python__hive-thrift-py] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1u81hos"
  * bash[install_python__pyhive] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-5efvr0"
  * bash[install_python__markdown] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-15h7alm"
  * bash[install_python__futures] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1syb4sg"
  * bash[install_python__werkzeug] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-a9vkhe"
  * bash[install_python__impyla] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-iv94dj"
  * bash[install_python__unicodecsv] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1rwngta"
  * bash[install_python__jaydebeapi] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-108wcm5"
  * bash[install_python__mysqlclient] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1ch6uee"
  * bash[install_python__librabbitmq] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1xqzq38"
  * bash[install_python__boto] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-183km54"
  * bash[install_python__filechunkio] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1wtp24j"
  * bash[install_python__pysmbclient] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1bdpssm"
  * bash[install_python__slackclient] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-gzv3er"
  * bash[install_python__statsd] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1ibep78"
  * bash[install_python__ldap3] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-4c2pz0"
  * bash[install_python__bcrypt] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-7hds0s"
  * bash[install_python__flask-bcrypt] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-1easmlh"
  * bash[install_python__Flask-OAuthlib] action run
    - execute "bash"  "/tmp/chef-script20200606-30275-92cq9u"
Recipe: hops_airflow::default
  * group[airflow] action modify (up to date)
Recipe: hops_airflow::config
  * template[/srv/hops/airflow/airflow.cfg] action create (up to date)
Recipe: hops_airflow::services
  * template[airflow_services_env] action create
    - change owner from 'airflow' to 'root'
  * template[/srv/hops/airflow/airflow_runner.sh] action create (up to date)
Recipe: hops_airflow::default
  * directory[/srv/hops/airflow/plugins] action create (up to date)
  * directory[/srv/hops/airflow/dags] action create (up to date)
  * template[airflow_services_env] action create
    - change owner from 'root' to 'airflow'
  * bash[init_airflow_db] action run
    
    ================================================================================
    Error executing action `run` on resource 'bash[init_airflow_db]'
    ================================================================================
    
    Mixlib::ShellOut::CommandTimeout
    --------------------------------
    Command timed out after 3600s:
    Command exceeded allowed execution time, process terminated
    ---- Begin output of "bash"  "/tmp/chef-script20200606-30275-eis9jv" ----
    STDOUT: [2020-06-06 16:34:03,069] {settings.py:174} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=3600, pid=32157
    [2020-06-06 16:34:03,436] {__init__.py:51} INFO - Using executor LocalExecutor
    DB: mysql://airflow_db:***@h3.5kdebvbjgtde3hg0nvvkxz3hma.rx.internal.cloudapp.net:3306/airflow
    [2020-06-06 16:34:03,869] {db.py:338} INFO - Creating tables
    STDERR: INFO  [alembic.runtime.migration] Context impl MySQLImpl.
    INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
    INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
    ---- End output of "bash"  "/tmp/chef-script20200606-30275-eis9jv" ----
    Ran "bash"  "/tmp/chef-script20200606-30275-eis9jv" returned 
    
    Resource Declaration:
    ---------------------
    # In /tmp/chef-solo/cookbooks/hops_airflow/recipes/default.rb
    
     65: bash 'init_airflow_db' do
     66:   user node['airflow']['user']
     67:   code <<-EOF
     68:       set -e
     69:       export AIRFLOW_HOME=#{node['airflow']['base_dir']}
     70:       #{node['airflow']['bin_path']}/airflow upgradedb
     71:     EOF
     72: end
     73: 
    
    Compiled Resource:
    ------------------
    # Declared in /tmp/chef-solo/cookbooks/hops_airflow/recipes/default.rb:65:in `from_file'
    
    bash("init_airflow_db") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      interpreter "bash"
      declared_type :bash
      cookbook_name "hops_airflow"
      recipe_name "default"
      user "airflow"
      code "      set -e\n      export AIRFLOW_HOME=/srv/hops/airflow\n      /srv/hops/anaconda/anaconda/envs/airflow/bin/airflow upgradedb\n"
      domain nil
    end
    
    System Info:
    ------------
    chef_version=14.10.9
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/usr/bin/chef-solo
    executable=/opt/chefdk/bin/chef-solo
    

Running handlers:
[2020-06-06T17:34:51+00:00] ERROR: Running exception handlers
Running handlers complete
[2020-06-06T17:34:51+00:00] ERROR: Exception handlers complete
Chef Client failed. 54 resources updated in 01 hours 05 minutes 43 seconds
[2020-06-06T17:34:51+00:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-06-06T17:34:51+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-06-06T17:34:51+00:00] FATAL: Mixlib::ShellOut::CommandTimeout: bash[init_airflow_db] (hops_airflow::default line 65) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command exceeded allowed execution time, process terminated
---- Begin output of "bash"  "/tmp/chef-script20200606-30275-eis9jv" ----
STDOUT: [2020-06-06 16:34:03,069] {settings.py:174} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=3600, pid=32157
[2020-06-06 16:34:03,436] {__init__.py:51} INFO - Using executor LocalExecutor
DB: mysql://airflow_db:***@h3.5kdebvbjgtde3hg0nvvkxz3hma.rx.internal.cloudapp.net:3306/airflow
[2020-06-06 16:34:03,869] {db.py:338} INFO - Creating tables
STDERR: INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
---- End output of "bash"  "/tmp/chef-script20200606-30275-eis9jv" ----
Ran "bash"  "/tmp/chef-script20200606-30275-eis9jv" returned

This is log of hops_airflow__default

antonios · June 9, 2020, 10:03am

Can you also paste the content of /etc/hosts ?

Tangeda_Sai_Sharan · June 10, 2020, 8:36am

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Contents of hosts file

antonios · June 10, 2020, 1:33pm

Azure is a bit tricky when it comes to DNS. Please follow the instructions here

You should be able to perform a forward and reverse DNS lookup for your fully qualified domain name. So if your FQDN is my.awesome.fqdn.azure then by doing dig my.awesome.fqdn.azure you should get an answer with your private IP.

Similarly if you do dig -x PRIVATE_IP you should get your FQDN for an answer. If you get that, try running the installer again.

Tangeda_Sai_Sharan · June 12, 2020, 8:05am

Azure is a bit tricky when it comes to DNS. Please follow the instructions here

You should be able to perform a forward and reverse DNS lookup for your fully qualified domain name. So if your FQDN is my.awesome.fqdn.azure then by doing dig my.awesome.fqdn.azure you should get an answer with your private IP.

Similarly if you do dig -x PRIVATE_IP you should get your FQDN for an answer. If you get that, try running the installer again.

Hey, so I got my FQDN, also I have my private IP, but I could not figure out where do u want me to use it before running the installer again
PS - If u were referring to length restrictions I have made sure that there are small before creating the VM
From logs I saw that code was getting timed out, I could not figure out why?
Also tried it by removing password for sudo, incase that would help but it did not
VM Config - 32gb ram, 8CPU. Ubuntu 18.04 OS machine

km805u · July 2, 2020, 7:06am

Hello Sai_Sharan,
Did you able to resolve this error?

Thank you.