How to connect to hopsworks instance created on single machine by an image

Hopsworks
#21

here-
�[0m�[0m
Running handlers:�[0m
[2020-09-11T14:56:39+05:30] ERROR: Running exception handlers
Running handlers complete
�[0m[2020-09-11T14:56:39+05:30] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 07 seconds�[0m
[2020-09-11T14:56:39+05:30] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-09-11T14:56:39+05:30] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-09-11T14:56:39+05:30] FATAL: Net::HTTPServerException: remote_file[/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh] (conda::install line 47) had an error: Net::HTTPServerException: 404 “Not Found”
Starting Chef Client, version 14.10.9�[0m
resolving cookbooks for run list: [“conda::install”]�[0m
Synchronizing Cookbooks:�[0m

  • magic_shell (1.0.0)�[0m
  • conda (1.3.0)�[0m
  • java (7.0.0)�[0m
  • ulimit (1.3.0)�[0m
  • homebrew (5.0.8)�[0m
  • windows (7.0.2)�[0m
    Installing Cookbook Gems:�[0m
    Compiling Cookbooks…�[0m
    Converging 26 resources�[0m
    Recipe: conda::install�[0m

  • yum_package[epel-release] action install (up to date)

  • yum_package[bind-utils] action install (up to date)

  • yum_package[bzip2, vim, iftop, htop, iotop, rsync] action install (up to date)

  • group[anaconda] action create (up to date)

  • linux_user[anaconda] action create (skipped due to not_if)

  • directory[/srv/hops] action create (skipped due to not_if)

  • directory[/srv/hops/anaconda] action create (skipped due to not_if)

  • remote_file[/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh] action create_if_missing[2020-09-11T14:56:54+05:30] WARN: remote_file[/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh] cannot be downloaded from http://snurran.sics.se/hops/Anaconda3-2020.07-Linux-x86_64.sh: 404 “Not Found”

    �[0m
    ================================================================================�[0m
    �[31mError executing action create_if_missing on resource 'remote_file[/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh]'�[0m
    ================================================================================�[0m

�[0m Net::HTTPServerException�[0m
------------------------�[0m
404 "Not Found"�[0m

�[0m Resource Declaration:�[0m
---------------------�[0m
# In /tmp/chef-solo/cookbooks/conda/recipes/install.rb
�[0m
�[0m 47: remote_file installer_path do
�[0m 48: source node[‘conda’][‘url’]
�[0m 49: # checksum installer_checksum
�[0m 50: user node[‘conda’][‘user’]
�[0m 51: group node[‘conda’][‘group’]
�[0m 52: mode 0755
�[0m 53: action :create_if_missing
�[0m 54: end
�[0m 55:
�[0m
�[0m Compiled Resource:�[0m
------------------�[0m
# Declared in /tmp/chef-solo/cookbooks/conda/recipes/install.rb:47:in `from_file’
�[0m
�[0m remote_file("/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh") do
�[0m action [:create_if_missing]
�[0m default_guard_interpreter :default
�[0m source [“http://snurran.sics.se/hops/Anaconda3-2020.07-Linux-x86_64.sh”]
�[0m declared_type :remote_file
�[0m cookbook_name “conda”
�[0m recipe_name “install”
�[0m owner “anaconda”
�[0m group “anaconda”
�[0m mode 493
�[0m remote_domain nil
�[0m remote_user nil
�[0m path “/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh”
�[0m headers {}
�[0m verifications []
�[0m end
�[0m
�[0m System Info:�[0m
------------�[0m
chef_version=14.10.9
�[0m platform=centos
�[0m platform_version=7.4.1708
�[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
�[0m program_name=/bin/chef-solo
�[0m executable=/opt/chefdk/bin/chef-solo�[0m

�[0m�[0m
Running handlers:�[0m
[2020-09-11T14:56:54+05:30] ERROR: Running exception handlers
Running handlers complete
�[0m[2020-09-11T14:56:54+05:30] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 07 seconds�[0m
[2020-09-11T14:56:54+05:30] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-09-11T14:56:54+05:30] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-09-11T14:56:54+05:30] FATAL: Net::HTTPServerException: remote_file[/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh] (conda::install line 47) had an error: Net::HTTPServerException: 404 “Not Found”

#22

Thanks. This looks like there is something wrong with how we serve installation files. I will investigate and get back to you.

#23

Can you please retry the step. The download should be fixed.

#24

The conda install failed again

#25

logs-
Recipe: conda::install�[0m

  • yum_package[epel-release] action install (up to date)
  • yum_package[bind-utils] action install (up to date)
  • yum_package[bzip2, vim, iftop, htop, iotop, rsync] action install (up to date)
  • group[anaconda] action create (up to date)
  • linux_user[anaconda] action create (skipped due to not_if)
  • directory[/srv/hops] action create (skipped due to not_if)
  • directory[/srv/hops/anaconda] action create (skipped due to not_if)
  • remote_file[/tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh] action create_if_missing (up to date)
  • template[/home/anaconda/.condarc] action create (up to date)
  • directory[/home/anaconda/.pip] action create (up to date)
  • template[/home/anaconda/.pip/pip.conf] action create (up to date)
  • directory[/root/.pip] action create (up to date)
  • template[/root/.pip/pip.conf] action create (up to date)
  • bash[run_conda_installer] action run
    �[0m
    ================================================================================�[0m
    �[31mError executing action run on resource 'bash[run_conda_installer]'�[0m
    ================================================================================�[0m

�[0m Mixlib::ShellOut::ShellCommandFailed�[0m
------------------------------------�[0m
Expected process to exit with [0], but received ‘126’
�[0m ---- Begin output of “bash” “/tmp/chef-script20200911-12708-1oyyzab” ----
�[0m STDOUT:
�[0m STDERR: /tmp/chef-script20200911-12708-1oyyzab: line 1: /tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh: Permission denied
�[0m ---- End output of “bash” “/tmp/chef-script20200911-12708-1oyyzab” ----
�[0m Ran “bash” “/tmp/chef-script20200911-12708-1oyyzab” returned 126�[0m

�[0m Resource Declaration:�[0m
---------------------�[0m
# In /tmp/chef-solo/cookbooks/conda/recipes/install.rb
�[0m
�[0m 95: bash ‘run_conda_installer’ do
�[0m 96: user node[‘conda’][‘user’]
�[0m 97: group node[‘conda’][‘group’]
�[0m 98: umask “022”
�[0m 99: code <<-EOF
�[0m 100: #{installer_path} -b -p #{node[‘conda’][‘home’]}
�[0m 101: EOF
�[0m 102: not_if { File.directory?(node[‘conda’][‘home’]) }
�[0m 103: end
�[0m 104:
�[0m
�[0m Compiled Resource:�[0m
------------------�[0m
# Declared in /tmp/chef-solo/cookbooks/conda/recipes/install.rb:95:in `from_file’
�[0m
�[0m bash(“run_conda_installer”) do
�[0m action [:run]
�[0m default_guard_interpreter :default
�[0m command nil
�[0m backup 5
�[0m interpreter “bash”
�[0m declared_type :bash
�[0m cookbook_name “conda”
�[0m recipe_name “install”
�[0m user “anaconda”
�[0m group “anaconda”
�[0m umask “022”
�[0m code " /tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh -b -p /srv/hops/anaconda/anaconda-3-2020.07\n"
�[0m domain nil
�[0m not_if { #code block }
�[0m end
�[0m
�[0m System Info:�[0m
------------�[0m
chef_version=14.10.9
�[0m platform=centos
�[0m platform_version=7.4.1708
�[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
�[0m program_name=/bin/chef-solo
�[0m executable=/opt/chefdk/bin/chef-solo�[0m

�[0m�[0m
Running handlers:�[0m
[2020-09-11T16:20:40+05:30] ERROR: Running exception handlers
Running handlers complete
�[0m[2020-09-11T16:20:40+05:30] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 06 seconds�[0m

#26

Can you check the file permissions of /tmp/chef-solo/Anaconda3-2020.07-Linux-x86_64.sh. It should be 755.

#27

the tensorflow installation also failed -

Installing Cookbook Gems:
Compiling Cookbooks…
Converging 24 resources
Recipe: tensorflow::install

  • yum_package[epel-release] action install (up to date)

  • yum_package[kernel-devel] action install

    • No version specified, and no candidate version available for kernel-devel[ 2020-09-11T18:03:06+05:30] ERROR: yum_package[kernel-devel] (tensorflow::install line 185) had an error: Chef::Exceptions::package: No version specified, and no candidate version available for kernel-devel; ignore_failure is set, continuing

    ============================================================================ ====
    Error executing action install on resource ‘yum_package[kernel-devel]’
    ============================================================================ ====

    Chef::Exceptions::Package

    No version specified, and no candidate version available for kernel-devel

    Resource Declaration:

    In /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb

    185: package ‘kernel-devel’ do
    186: version node[‘kernel’][‘release’].sub(/.#{node[‘kernel’][‘machine’ ]}/, “”)
    187: arch node[‘kernel’][‘machine’]
    188: action :install
    189: ignore_failure true
    190: end
    191:

    Compiled Resource:

    Declared in /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb:185:in `from_file’

    yum_package(“kernel-devel”) do
    package_name “kernel-devel”
    action [:install]
    default_guard_interpreter :default
    declared_type :package
    cookbook_name “tensorflow”
    recipe_name “install”
    version “3.10.0-862.6.3.el7”
    arch “x86_64”
    ignore_failure true
    end

    System Info:

    chef_version=14.10.9
    platform=centos
    platform_version=7.4.1708
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/bin/chef-solo
    executable=/opt/chefdk/bin/chef-solo

  • yum_package[kernel-devel] action install (skipped due to not_if)

  • yum_package[pciutils, python-pip, mlocate, gcc, gcc-c++, openssl, openssl-de vel, python, python-devel, python-lxml, python-pillow, libcurl-devel, python-whe el, python-six] action install

    • No candidate version available for python-wheel
      ============================================================================ ====
      Error executing action install on resource ‘yum_package[pciutils, python-p ip, mlocate, gcc, gcc-c++, openssl, openssl-devel, python, python-devel, python- lxml, python-pillow, libcurl-devel, python-wheel, python-six]’
      ============================================================================ ====

    Chef::Exceptions::Package

    No candidate version available for python-wheel

    Resource Declaration:

    In /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb

    197: package [‘pciutils’, ‘python-pip’, ‘mlocate’, ‘gcc’, ‘gcc-c++’, ‘open ssl’, ‘openssl-devel’, ‘python’, ‘python-devel’, ‘python-lxml’, ‘python-pillow’, ‘libcurl-devel’, ‘python-wheel’, ‘python-six’]
    198: end

    Compiled Resource:

    Declared in /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb:197:in `from_file’

    yum_package(“pciutils, python-pip, mlocate, gcc, gcc-c++, openssl, openssl-d evel, python, python-devel, python-lxml, python-pillow, libcurl-devel, python-wh eel, python-six”) do
    package_name [“pciutils”, “python-pip”, “mlocate”, “gcc”, “gcc-c++”, “open ssl”, “openssl-devel”, “python”, “python-devel”, “python-lxml”, “python-pillow”, “libcurl-devel”, “python-wheel”, “python-six”]
    action [:install]
    default_guard_interpreter :default
    declared_type :package
    cookbook_name “tensorflow”
    recipe_name “install”
    end

    System Info:

    chef_version=14.10.9
    platform=centos
    platform_version=7.4.1708
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/bin/chef-solo
    executable=/opt/chefdk/bin/chef-solo

#28

It looks like your operating system is outdated and the packages are not available anymore. Can you run yum update.

#29

More installs like kagent, tensorflow, hopslogs, and hopsworks utils are gettting stuck

#30

logs -

e[31mError executing action install on resource 'yum_package[kernel-devel]'e[0m
================================================================================e[0m

e[0m Chef::Exceptions::Packagee[0m
-------------------------e[0m
No version specified, and no candidate version available for kernel-devele[0m

e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb
e[0m
e[0m 185: package ‘kernel-devel’ do
e[0m 186: version node[‘kernel’][‘release’].sub(/.#{node[‘kernel’][‘machine’]}/, “”)
e[0m 187: arch node[‘kernel’][‘machine’]
e[0m 188: action :install
e[0m 189: ignore_failure true
e[0m 190: end
e[0m 191:
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb:185:in `from_file’
e[0m
e[0m yum_package(“kernel-devel”) do
e[0m package_name “kernel-devel”
e[0m action [:install]
e[0m default_guard_interpreter :default
e[0m declared_type :package
e[0m cookbook_name “tensorflow”
e[0m recipe_name “install”
e[0m version “3.10.0-862.6.3.el7”
e[0m arch “x86_64”
e[0m ignore_failure true
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=centos
e[0m platform_version=7.4.1708
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m

e[0m * yum_package[kernel-devel] action install (skipped due to not_if)

  • yum_package[pciutils, python-pip, mlocate, gcc, gcc-c++, openssl, openssl-devel, python, python-devel, python-lxml, python-pillow, libcurl-devel, python-wheel, python-six] action install
    e[31m* No candidate version available for python-wheele[0me[0m
    ================================================================================e[0m
    e[31mError executing action install on resource 'yum_package[pciutils, python-pip, mlocate, gcc, gcc-c++, openssl, openssl-devel, python, python-devel, python-lxml, python-pillow, libcurl-devel, python-wheel, python-six]'e[0m
    ================================================================================e[0m

e[0m Chef::Exceptions::Packagee[0m
-------------------------e[0m
No candidate version available for python-wheele[0m

e[0m Resource Declaration:e[0m
---------------------e[0m
# In /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb
e[0m
e[0m 197: package [‘pciutils’, ‘python-pip’, ‘mlocate’, ‘gcc’, ‘gcc-c++’, ‘openssl’, ‘openssl-devel’, ‘python’, ‘python-devel’, ‘python-lxml’, ‘python-pillow’, ‘libcurl-devel’, ‘python-wheel’, ‘python-six’]
e[0m 198: end
e[0m
e[0m Compiled Resource:e[0m
------------------e[0m
# Declared in /tmp/chef-solo/cookbooks/tensorflow/recipes/install.rb:197:in `from_file’
e[0m
e[0m yum_package(“pciutils, python-pip, mlocate, gcc, gcc-c++, openssl, openssl-devel, python, python-devel, python-lxml, python-pillow, libcurl-devel, python-wheel, python-six”) do
e[0m package_name [“pciutils”, “python-pip”, “mlocate”, “gcc”, “gcc-c++”, “openssl”, “openssl-devel”, “python”, “python-devel”, “python-lxml”, “python-pillow”, “libcurl-devel”, “python-wheel”, “python-six”]
e[0m action [:install]
e[0m default_guard_interpreter :default
e[0m declared_type :package
e[0m cookbook_name “tensorflow”
e[0m recipe_name “install”
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=14.10.9
e[0m platform=centos
e[0m platform_version=7.4.1708
e[0m ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
e[0m program_name=/bin/chef-solo
e[0m executable=/opt/chefdk/bin/chef-soloe[0m

e[0me[0m
Running handlers:e[0m
[2020-09-12T12:55:36+05:30] ERROR: Running exception handlers
Running handlers complete
e[0m[2020-09-12T12:55:36+05:30] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 10 secondse[0m
[2020-09-12T12:55:36+05:30] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2020-09-12T12:55:36+05:30] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-09-12T12:55:36+05:30] FATAL: Chef::Exceptions::package: yum_package[pciutils, python-pip, mlocate, gcc, gcc-c++, openssl, openssl-devel, python, python-devel, python-lxml, python-pillow, libcurl-devel, python-wheel, python-six] (tensorflow::install line 197) had an error: Chef::Exceptions::package: No candidate version available for python-wheel

#31

That looks like more issues with your version of Centos. Would you be able to try it on an up to date installation of Centos or Ubuntu?

#32

I did an yum update, still tensorflow install failed

#33

Could you try to install python-wheel manually and see what you get?

#34

I got-
Requirement already satisfied: wheel in /home/subexgpu5/anaconda3/lib/python3.7/site-packages (0.34.2)

#35

Can you do it while not having conda activated?

#36

how to install with conda deactivated?

#37

conda deactivate before running the installation should do the trick. See: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

#38

still getting the same response with conda deactivated-
Requirement already satisfied: wheel in /home/subexgpu5/anaconda3/lib/python3.7/site-packages (0.34.2)

#39

Looking at the path, it is still finding the package in anaconda. I don’t know how your environment is set up but it looks like you are still using conda.

#40

HI! I did-
conda uninstall wheel, and conda deactivate, still getting the same response to pip install wheel