Recipe "docker_registry" failed

Hi, I am trying to install hopswork in one VM, and the installation failed while running docker_registry, and below are the error info, could u please help me out?

starting Chef Client, version 14.10.9
resolving cookbooks for run list: [“hops::docker_registry”]
Synchronizing Cookbooks:

  • hops (2.4.0)
  • magic_shell (1.0.0)
  • sysctl (1.0.5)
  • cmake (0.3.0)
  • kagent (2.4.0)
  • ndb (2.4.0)
  • conda (2.4.0)
  • kzookeeper (2.4.0)
  • elastic (2.4.0)
  • consul (2.4.0)
  • java (7.0.0)
  • ohai (5.3.0)
  • openssl (4.4.0)
  • ntp (2.0.3)
  • hostsfile (2.4.6)
  • ulimit (2.4.0)
  • ulimit2 (0.2.0)
  • sudo (4.0.1)
  • elasticsearch (4.0.6)
  • ark (5.1.1)
  • yum (6.1.1)
  • homebrew (5.2.2)
  • windows (7.0.2)
  • chef-sugar (5.1.12)
  • seven_zip (3.2.0)
  • apt (7.2.0)
    Installing Cookbook Gems:
    Compiling Cookbooks…
    Converging 12 resources
    Recipe: hops::docker_registry
  • remote_file[/tmp/chef-solo/registry_image.tar] action create_if_missing (skipped due to not_if)

  • bash[import_image] action run (skipped due to not_if)

  • file[/tmp/chef-solo/registry_image.tar] action delete

    • delete file /tmp/chef-solo/registry_image.tar
  • bash[start_docker_registry] action run (skipped due to not_if)

  • template[/srv/hops/hadoop/bin/consul/registry-health.sh] action create (up to date)

  • consul_service[Registering Registry with Consul] action register

    • template[/srv/hops/consul/consul.d/registry-consul.hcl] action create (up to date)
    • systemd_unit[consul.service] action reload
      • reloading unit: consul.service
    • systemd_unit[consul.service] action restart (skipped due to only_if)
  • bash[download_images] action run (skipped due to not_if)

  • bash[import_images] action run

    ================================================================================
    Error executing action run on resource ‘bash[import_images]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
    STDOUT:
    STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-081669584/repositories: no such file or directory
    ---- End output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
    Ran “bash” “/tmp/chef-script20211117-61035-l34p6s” returned 1

    Resource Declaration:

    In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

    126: bash “import_images” do
    127: user “root”
    128: code <<-EOF
    129: docker load -i #{Chef::Config[‘file_cache_path’]}/#{base_filename}
    130: EOF
    131: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
    132: end
    133:
    134: bash “tag_images” do
    135: user “root”
    136: code <<-EOF
    137: docker tag #{base_image} #{registry_address}/#{base_image}
    138: docker rmi #{base_image}
    139: docker tag #{base_image_python} #{registry_address}/#{base_image_python}
    140: docker rmi #{base_image_python}
    141: EOF
    142: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
    143: end
    144:
    145: bash “push_images” do
    146: user “root”
    147: code <<-EOF
    148: docker push #{registry_address}/#{base_image}
    149: docker push #{registry_address}/#{base_image_python}
    150: EOF
    151: end
    152:
    153: #delete tar
    154: file “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” do
    155: action :delete
    156: only_if { File.exist? “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” }
    157: end
    158:
    159: # We add docker in kagent in this recipe as the hops::docker recipe runs during the install phase and it might run
    160: # before kagent::install
    161: service_name=‘docker’
    162: if node[‘kagent’][‘enabled’] == “true”
    163: kagent_config service_name do
    164: service “docker”
    165: end
    166: end
    167:
    168: if conda_helpers.is_upgrade
    169: kagent_config service_name do
    170: action :systemd_reload
    171: end
    172: end

    Compiled Resource:

    Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

    bash(“import_images”) do
    action [:run]
    default_guard_interpreter :default
    command nil
    backup 5
    interpreter “bash”
    declared_type :bash
    cookbook_name “hops”
    recipe_name “docker_registry”
    user “root”
    code " docker load -i /tmp/chef-solo/base.tar\n"
    domain nil
    not_if “docker image inspect registry.service.consul:4443/base:2.4.0 registry.service.consul:4443/python37:2.4.0”
    end

    System Info:

    chef_version=14.10.9
    platform=centos
    platform_version=7.9.2009
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/bin/chef-solo
    executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-17T13:58:29-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-17T13:58:29-05:00] ERROR: Exception handlers complete
Chef Client failed. 3 resources updated in 08 seconds
[2021-11-17T13:58:29-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-17T13:58:29-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-17T13:58:29-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[import_images] (hops::docker_registry line 126) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0]
, but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
STDOUT:
STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-081669584/repositories: no such file or directory
---- End output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
Ran “bash” “/tmp/chef-script20211117-61035-l34p6s” returned 1

Hi @david

Could you check if docker is actually running? systemctl status docker ? Also, what filesystem are you using?

Hi @antonios,

yes, the docker is actually running. and I am using ext4 filesystem in /srv.

Thanks,
-David

Hi @antonios,

in fact, some additional error before above error I posted, which is about download_images:

Error executing action run on resource ‘bash[download_images]’
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Resource Declaration:
---------------------
suppressed sensitive resource output

Compiled Resource:
------------------
suppressed sensitive resource output

System Info:
------------
chef_version=14.10.9
platform=centos
platform_version=7.9.2009
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/bin/chef-solo
executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-18T11:23:07-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-18T11:23:07-05:00] ERROR: Exception handlers complete
Chef Client failed. 9 resources updated in 17 seconds
[2021-11-18T11:23:07-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-18T11:23:07-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-18T11:23:07-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[download_images] (hops::docker_registry line 115) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOU
T/STDERR suppressed for sensitive resource

I’m wondering whether you have enough available disk space. Can you go do /srv/hops and do df -h . ?

Hi @antonios,

Here is the output per your requirement:

[root@poc hops]# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 92G 27G 61G 31% /
[root@poc hops]# pwd
/srv/hops

Can you try removing /tmp/chef-solo/base.tar and re-try the docker_registry recipe?

Hi @antonios,

same error as below, both download image and import image encounter the error.

Starting Chef Client, version 14.10.9
resolving cookbooks for run list: [“hops::docker_registry”]
Synchronizing Cookbooks:

  • magic_shell (1.0.0)
  • sysctl (1.0.5)
  • cmake (0.3.0)
  • hops (2.4.0)
  • kagent (2.4.0)
  • conda (2.4.0)
  • ndb (2.4.0)
  • kzookeeper (2.4.0)
  • elastic (2.4.0)
  • consul (2.4.0)
  • java (7.0.0)
  • openssl (4.4.0)
  • ohai (5.3.0)
  • hostsfile (2.4.6)
  • ntp (2.0.3)
  • sudo (4.0.1)
  • ulimit (2.4.0)
  • ulimit2 (0.2.0)
  • elasticsearch (4.0.6)
  • ark (5.1.1)
  • yum (6.1.1)
  • homebrew (5.2.2)
  • windows (7.0.2)
  • chef-sugar (5.1.12)
  • apt (7.2.0)
  • seven_zip (3.2.0)
    Installing Cookbook Gems:
    Compiling Cookbooks…
    Converging 12 resources
    Recipe: hops::docker_registry
  • remote_file[/tmp/chef-solo/registry_image.tar] action create_if_missing (skipped due to not_if)

  • bash[import_image] action run (skipped due to not_if)

  • file[/tmp/chef-solo/registry_image.tar] action delete (skipped due to only_if)

  • bash[start_docker_registry] action run (skipped due to not_if)

  • template[/srv/hops/hadoop/bin/consul/registry-health.sh] action create (up to date)

  • consul_service[Registering Registry with Consul] action register

    • template[/srv/hops/consul/consul.d/registry-consul.hcl] action create (up to date)
    • systemd_unit[consul.service] action reload
      • reloading unit: consul.service
    • systemd_unit[consul.service] action restart (skipped due to only_if)
  • bash[download_images] action run

    ================================================================================
    Error executing action run on resource ‘bash[download_images]’

    Mixlib::ShellOut::ShellCommandFailed

    Command execution failed. STDOUT/STDERR suppressed for sensitive resource

    Resource Declaration:

    suppressed sensitive resource output

    Compiled Resource:

    suppressed sensitive resource output

    System Info:

    chef_version=14.10.9
    platform=centos
    platform_version=7.9.2009
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/bin/chef-solo
    executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-19T09:10:21-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-19T09:10:21-05:00] ERROR: Exception handlers complete
Chef Client failed. 2 resources updated in 10 seconds
[2021-11-19T09:10:21-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-19T09:10:21-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-19T09:10:21-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[download_images] (hops::docker_registry line 115) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed.
STDOUT/STDERR suppressed for sensitive resource

resolving cookbooks for run list: [“hops::docker_registry”]
Synchronizing Cookbooks:

  • hops (2.4.0)
  • magic_shell (1.0.0)
  • sysctl (1.0.5)
  • cmake (0.3.0)
  • kagent (2.4.0)
  • ndb (2.4.0)
  • conda (2.4.0)
  • consul (2.4.0)
  • kzookeeper (2.4.0)
  • elastic (2.4.0)
  • java (7.0.0)
  • ohai (5.3.0)
  • openssl (4.4.0)
  • hostsfile (2.4.6)
  • ntp (2.0.3)
  • sudo (4.0.1)
  • ulimit (2.4.0)
  • elasticsearch (4.0.6)
  • ulimit2 (0.2.0)
  • ark (5.1.1)
  • yum (6.1.1)
  • homebrew (5.2.2)
  • windows (7.0.2)
  • seven_zip (3.2.0)
  • chef-sugar (5.1.12)
  • apt (7.2.0)
    Installing Cookbook Gems:
    Compiling Cookbooks…
    Converging 12 resources
    Recipe: hops::docker_registry
  • remote_file[/tmp/chef-solo/registry_image.tar] action create_if_missing (skipped due to not_if)

  • bash[import_image] action run (skipped due to not_if)

  • file[/tmp/chef-solo/registry_image.tar] action delete (up to date)

  • bash[start_docker_registry] action run (skipped due to not_if)

  • template[/srv/hops/hadoop/bin/consul/registry-health.sh] action create (up to date)

  • consul_service[Registering Registry with Consul] action register

    • template[/srv/hops/consul/consul.d/registry-consul.hcl] action create (up to date)
    • systemd_unit[consul.service] action reload
      • reloading unit: consul.service
    • systemd_unit[consul.service] action restart (skipped due to only_if)
  • bash[download_images] action run (skipped due to not_if)

  • bash[import_images] action run

    ================================================================================
    Error executing action run on resource ‘bash[import_images]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
    STDOUT:
    STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-627887742/repositories: no such file or directory
    ---- End output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
    Ran “bash” “/tmp/chef-script20211119-69761-1wbkts2” returned 1

    Resource Declaration:

    In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

    126: bash “import_images” do
    127: user “root”
    128: code <<-EOF
    129: docker load -i #{Chef::Config[‘file_cache_path’]}/#{base_filename}
    130: EOF
    131: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
    132: end
    133:
    134: bash “tag_images” do
    135: user “root”
    136: code <<-EOF
    137: docker tag #{base_image} #{registry_address}/#{base_image}
    138: docker rmi #{base_image}
    139: docker tag #{base_image_python} #{registry_address}/#{base_image_python}
    140: docker rmi #{base_image_python}
    141: EOF
    142: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
    143: end
    144:
    145: bash “push_images” do
    146: user “root”
    147: code <<-EOF
    148: docker push #{registry_address}/#{base_image}
    149: docker push #{registry_address}/#{base_image_python}
    150: EOF
    151: end
    152:
    153: #delete tar
    154: file “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” do
    155: action :delete
    156: only_if { File.exist? “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” }
    157: end
    158:
    159: # We add docker in kagent in this recipe as the hops::docker recipe runs during the install phase and it might run
    160: # before kagent::install
    161: service_name=‘docker’
    162: if node[‘kagent’][‘enabled’] == “true”
    163: kagent_config service_name do
    164: service “docker”
    165: end
    166: end)
    167:
    168: if conda_helpers.is_upgrade
    169: kagent_config service_name do
    170: action :systemd_reload
    171: end
    172: end

    Compiled Resource:

    Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

    bash(“import_images”) do
    action [:run]
    default_guard_interpreter :default
    command nil
    backup 5
    interpreter “bash”
    declared_type :bash
    cookbook_name “hops”
    recipe_name “docker_registry”
    user “root”
    code " docker load -i /tmp/chef-solo/base.tar\n"
    domain nil
    not_if “docker image inspect registry.service.consul:4443/base:2.4.0 registry.service.consul:4443/python37:2.4.0”
    end

    System Info:

    chef_version=14.10.9
    platform=centos
    platform_version=7.9.2009
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/bin/chef-solo
    executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-19T09:10:36-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-19T09:10:36-05:00] ERROR: Exception handlers complete
Chef Client failed. 2 resources updated in 08 seconds
[2021-11-19T09:10:36-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-19T09:10:36-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-19T09:10:36-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[import_images] (hops::docker_registry line 126) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit wi
th [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
STDOUT:
STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-627887742/repositories: no such file or directory
---- End output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
Ran “bash” “/tmp/chef-script20211119-69761-1wbkts2” returned 1

Can you download https://repo.hops.works/master/kube/docker-images/2.4.0/base.tar ?

Thanks @antonios very much.

after downloading the base.tar manually, the recipe ran successfully. Really appreciate.

That’s a bit strange though. I don’t understand why it would fail to download the images. In any case, it’s positive if you made progress.

yeah, before downloading the file manually, the original size is zero. :grinning:

1 Like