Recipe "docker_registry" failed

david · November 17, 2021, 7:06pm

Hi, I am trying to install hopswork in one VM, and the installation failed while running docker_registry, and below are the error info, could u please help me out?

starting Chef Client, version 14.10.9
resolving cookbooks for run list: [“hops::docker_registry”]
Synchronizing Cookbooks:

hops (2.4.0)
magic_shell (1.0.0)
sysctl (1.0.5)
cmake (0.3.0)
kagent (2.4.0)
ndb (2.4.0)
conda (2.4.0)
kzookeeper (2.4.0)
elastic (2.4.0)
consul (2.4.0)
java (7.0.0)
ohai (5.3.0)
openssl (4.4.0)
ntp (2.0.3)
hostsfile (2.4.6)
ulimit (2.4.0)
ulimit2 (0.2.0)
sudo (4.0.1)
elasticsearch (4.0.6)
ark (5.1.1)
yum (6.1.1)
homebrew (5.2.2)
windows (7.0.2)
chef-sugar (5.1.12)
seven_zip (3.2.0)
apt (7.2.0)
Installing Cookbook Gems:
Compiling Cookbooks…
Converging 12 resources
Recipe: hops::docker_registry

remote_file[/tmp/chef-solo/registry_image.tar] action create_if_missing (skipped due to not_if)
bash[import_image] action run (skipped due to not_if)
file[/tmp/chef-solo/registry_image.tar] action delete
- delete file /tmp/chef-solo/registry_image.tar
bash[start_docker_registry] action run (skipped due to not_if)
template[/srv/hops/hadoop/bin/consul/registry-health.sh] action create (up to date)
consul_service[Registering Registry with Consul] action register
- template[/srv/hops/consul/consul.d/registry-consul.hcl] action create (up to date)
- systemd_unit[consul.service] action reload
  - reloading unit: consul.service
- systemd_unit[consul.service] action restart (skipped due to only_if)
bash[download_images] action run (skipped due to not_if)
bash[import_images] action run

================================================================================
Error executing action run on resource ‘bash[import_images]’

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
STDOUT:
STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-081669584/repositories: no such file or directory
---- End output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
Ran “bash” “/tmp/chef-script20211117-61035-l34p6s” returned 1

Resource Declaration:

In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

126: bash “import_images” do
127: user “root”
128: code <<-EOF
129: docker load -i #{Chef::Config[‘file_cache_path’]}/#{base_filename}
130: EOF
131: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
132: end
133:
134: bash “tag_images” do
135: user “root”
136: code <<-EOF
137: docker tag #{base_image} #{registry_address}/#{base_image}
138: docker rmi #{base_image}
139: docker tag #{base_image_python} #{registry_address}/#{base_image_python}
140: docker rmi #{base_image_python}
141: EOF
142: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
143: end
144:
145: bash “push_images” do
146: user “root”
147: code <<-EOF
148: docker push #{registry_address}/#{base_image}
149: docker push #{registry_address}/#{base_image_python}
150: EOF
151: end
152:
153: #delete tar
154: file “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” do
155: action :delete
156: only_if { File.exist? “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” }
157: end
158:
159: # We add docker in kagent in this recipe as the hops::docker recipe runs during the install phase and it might run
160: # before kagent::install
161: service_name=‘docker’
162: if node[‘kagent’][‘enabled’] == “true”
163: kagent_config service_name do
164: service “docker”
165: end
166: end
167:
168: if conda_helpers.is_upgrade
169: kagent_config service_name do
170: action :systemd_reload
171: end
172: end

Compiled Resource:

Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

bash(“import_images”) do
action [:run]
default_guard_interpreter :default
command nil
backup 5
interpreter “bash”
declared_type :bash
cookbook_name “hops”
recipe_name “docker_registry”
user “root”
code " docker load -i /tmp/chef-solo/base.tar\n"
domain nil
not_if “docker image inspect registry.service.consul:4443/base:2.4.0 registry.service.consul:4443/python37:2.4.0”
end

System Info:

chef_version=14.10.9
platform=centos
platform_version=7.9.2009
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/bin/chef-solo
executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-17T13:58:29-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-17T13:58:29-05:00] ERROR: Exception handlers complete
Chef Client failed. 3 resources updated in 08 seconds
[2021-11-17T13:58:29-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-17T13:58:29-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-17T13:58:29-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[import_images] (hops::docker_registry line 126) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0]
, but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
STDOUT:
STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-081669584/repositories: no such file or directory
---- End output of “bash” “/tmp/chef-script20211117-61035-l34p6s” ----
Ran “bash” “/tmp/chef-script20211117-61035-l34p6s” returned 1

antonios · November 18, 2021, 12:38pm

Hi @david

Could you check if docker is actually running? systemctl status docker ? Also, what filesystem are you using?

david · November 18, 2021, 1:34pm

Hi @antonios,

yes, the docker is actually running. and I am using ext4 filesystem in /srv.

Thanks,
-David

david · November 18, 2021, 5:13pm

Hi @antonios,

in fact, some additional error before above error I posted, which is about download_images:

Error executing action run on resource ‘bash[download_images]’
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Resource Declaration:
---------------------
suppressed sensitive resource output

Compiled Resource:
------------------
suppressed sensitive resource output

System Info:
------------
chef_version=14.10.9
platform=centos
platform_version=7.9.2009
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/bin/chef-solo
executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-18T11:23:07-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-18T11:23:07-05:00] ERROR: Exception handlers complete
Chef Client failed. 9 resources updated in 17 seconds
[2021-11-18T11:23:07-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-18T11:23:07-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-18T11:23:07-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[download_images] (hops::docker_registry line 115) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOU
T/STDERR suppressed for sensitive resource

antonios · November 19, 2021, 1:23pm

I’m wondering whether you have enough available disk space. Can you go do /srv/hops and do df -h . ?

david · November 19, 2021, 1:45pm

Hi @antonios,

Here is the output per your requirement:

[root@poc hops]# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 92G 27G 61G 31% /
[root@poc hops]# pwd
/srv/hops

antonios · November 19, 2021, 2:06pm

Can you try removing /tmp/chef-solo/base.tar and re-try the docker_registry recipe?

david · November 19, 2021, 2:14pm

Hi @antonios,

same error as below, both download image and import image encounter the error.

Starting Chef Client, version 14.10.9
resolving cookbooks for run list: [“hops::docker_registry”]
Synchronizing Cookbooks:

magic_shell (1.0.0)
sysctl (1.0.5)
cmake (0.3.0)
hops (2.4.0)
kagent (2.4.0)
conda (2.4.0)
ndb (2.4.0)
kzookeeper (2.4.0)
elastic (2.4.0)
consul (2.4.0)
java (7.0.0)
openssl (4.4.0)
ohai (5.3.0)
hostsfile (2.4.6)
ntp (2.0.3)
sudo (4.0.1)
ulimit (2.4.0)
ulimit2 (0.2.0)
elasticsearch (4.0.6)
ark (5.1.1)
yum (6.1.1)
homebrew (5.2.2)
windows (7.0.2)
chef-sugar (5.1.12)
apt (7.2.0)
seven_zip (3.2.0)
Installing Cookbook Gems:
Compiling Cookbooks…
Converging 12 resources
Recipe: hops::docker_registry

remote_file[/tmp/chef-solo/registry_image.tar] action create_if_missing (skipped due to not_if)
bash[import_image] action run (skipped due to not_if)
file[/tmp/chef-solo/registry_image.tar] action delete (skipped due to only_if)
bash[start_docker_registry] action run (skipped due to not_if)
template[/srv/hops/hadoop/bin/consul/registry-health.sh] action create (up to date)
consul_service[Registering Registry with Consul] action register
- template[/srv/hops/consul/consul.d/registry-consul.hcl] action create (up to date)
- systemd_unit[consul.service] action reload
  - reloading unit: consul.service
- systemd_unit[consul.service] action restart (skipped due to only_if)
bash[download_images] action run

================================================================================
Error executing action run on resource ‘bash[download_images]’

Mixlib::ShellOut::ShellCommandFailed

Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Resource Declaration:

suppressed sensitive resource output

Compiled Resource:

suppressed sensitive resource output

System Info:

chef_version=14.10.9
platform=centos
platform_version=7.9.2009
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/bin/chef-solo
executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-19T09:10:21-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-19T09:10:21-05:00] ERROR: Exception handlers complete
Chef Client failed. 2 resources updated in 10 seconds
[2021-11-19T09:10:21-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-19T09:10:21-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-19T09:10:21-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[download_images] (hops::docker_registry line 115) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed.
STDOUT/STDERR suppressed for sensitive resource

resolving cookbooks for run list: [“hops::docker_registry”]
Synchronizing Cookbooks:

hops (2.4.0)
magic_shell (1.0.0)
sysctl (1.0.5)
cmake (0.3.0)
kagent (2.4.0)
ndb (2.4.0)
conda (2.4.0)
consul (2.4.0)
kzookeeper (2.4.0)
elastic (2.4.0)
java (7.0.0)
ohai (5.3.0)
openssl (4.4.0)
hostsfile (2.4.6)
ntp (2.0.3)
sudo (4.0.1)
ulimit (2.4.0)
elasticsearch (4.0.6)
ulimit2 (0.2.0)
ark (5.1.1)
yum (6.1.1)
homebrew (5.2.2)
windows (7.0.2)
seven_zip (3.2.0)
chef-sugar (5.1.12)
apt (7.2.0)
Installing Cookbook Gems:
Compiling Cookbooks…
Converging 12 resources
Recipe: hops::docker_registry

remote_file[/tmp/chef-solo/registry_image.tar] action create_if_missing (skipped due to not_if)
bash[import_image] action run (skipped due to not_if)
file[/tmp/chef-solo/registry_image.tar] action delete (up to date)
bash[start_docker_registry] action run (skipped due to not_if)
template[/srv/hops/hadoop/bin/consul/registry-health.sh] action create (up to date)
consul_service[Registering Registry with Consul] action register
- template[/srv/hops/consul/consul.d/registry-consul.hcl] action create (up to date)
- systemd_unit[consul.service] action reload
  - reloading unit: consul.service
- systemd_unit[consul.service] action restart (skipped due to only_if)
bash[download_images] action run (skipped due to not_if)
bash[import_images] action run

================================================================================
Error executing action run on resource ‘bash[import_images]’

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
STDOUT:
STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-627887742/repositories: no such file or directory
---- End output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
Ran “bash” “/tmp/chef-script20211119-69761-1wbkts2” returned 1

Resource Declaration:

In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

126: bash “import_images” do
127: user “root”
128: code <<-EOF
129: docker load -i #{Chef::Config[‘file_cache_path’]}/#{base_filename}
130: EOF
131: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
132: end
133:
134: bash “tag_images” do
135: user “root”
136: code <<-EOF
137: docker tag #{base_image} #{registry_address}/#{base_image}
138: docker rmi #{base_image}
139: docker tag #{base_image_python} #{registry_address}/#{base_image_python}
140: docker rmi #{base_image_python}
141: EOF
142: not_if “docker image inspect #{registry_address}/#{base_image} #{registry_address}/#{base_image_python}”
143: end
144:
145: bash “push_images” do
146: user “root”
147: code <<-EOF
148: docker push #{registry_address}/#{base_image}
149: docker push #{registry_address}/#{base_image_python}
150: EOF
151: end
152:
153: #delete tar
154: file “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” do
155: action :delete
156: only_if { File.exist? “#{Chef::Config[‘file_cache_path’]}/#{base_filename}” }
157: end
158:
159: # We add docker in kagent in this recipe as the hops::docker recipe runs during the install phase and it might run
160: # before kagent::install
161: service_name=‘docker’
162: if node[‘kagent’][‘enabled’] == “true”
163: kagent_config service_name do
164: service “docker”
165: end
166: end)
167:
168: if conda_helpers.is_upgrade
169: kagent_config service_name do
170: action :systemd_reload
171: end
172: end

Compiled Resource:

Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

bash(“import_images”) do
action [:run]
default_guard_interpreter :default
command nil
backup 5
interpreter “bash”
declared_type :bash
cookbook_name “hops”
recipe_name “docker_registry”
user “root”
code " docker load -i /tmp/chef-solo/base.tar\n"
domain nil
not_if “docker image inspect registry.service.consul:4443/base:2.4.0 registry.service.consul:4443/python37:2.4.0”
end

System Info:

chef_version=14.10.9
platform=centos
platform_version=7.9.2009
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/bin/chef-solo
executable=/opt/chefdk/bin/chef-solo

Running handlers:
[2021-11-19T09:10:36-05:00] ERROR: Running exception handlers
Running handlers complete
[2021-11-19T09:10:36-05:00] ERROR: Exception handlers complete
Chef Client failed. 2 resources updated in 08 seconds
[2021-11-19T09:10:36-05:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[2021-11-19T09:10:36-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-11-19T09:10:36-05:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[import_images] (hops::docker_registry line 126) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit wi
th [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
STDOUT:
STDERR: open /srv/hopsworks-data/docker/tmp/docker-import-627887742/repositories: no such file or directory
---- End output of “bash” “/tmp/chef-script20211119-69761-1wbkts2” ----
Ran “bash” “/tmp/chef-script20211119-69761-1wbkts2” returned 1

antonios · November 19, 2021, 2:28pm

Can you download https://repo.hops.works/master/kube/docker-images/2.4.0/base.tar ?

david · November 19, 2021, 3:00pm

Thanks @antonios very much.

after downloading the base.tar manually, the recipe ran successfully. Really appreciate.

antonios · November 19, 2021, 3:09pm

That’s a bit strange though. I don’t understand why it would fail to download the images. In any case, it’s positive if you made progress.

david · November 19, 2021, 3:15pm

yeah, before downloading the file manually, the original size is zero.

Recipe "docker_registry" failed

================================================================================
Error executing action `run` on resource ‘bash[import_images]’

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

Compiled Resource:

Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

System Info:

================================================================================
Error executing action `run` on resource ‘bash[download_images]’

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

Compiled Resource:

System Info:

================================================================================
Error executing action `run` on resource ‘bash[import_images]’

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

Compiled Resource:

Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

System Info:

Recipe "docker_registry" failed

================================================================================ Error executing action run on resource ‘bash[import_images]’

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

Compiled Resource:

Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

System Info:

================================================================================ Error executing action run on resource ‘bash[download_images]’

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

Compiled Resource:

System Info:

================================================================================ Error executing action run on resource ‘bash[import_images]’

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

In /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb

Compiled Resource:

Declared in /tmp/chef-solo/cookbooks/hops/recipes/docker_registry.rb:126:in `from_file’

System Info:

================================================================================
Error executing action `run` on resource ‘bash[import_images]’

================================================================================
Error executing action `run` on resource ‘bash[download_images]’

================================================================================
Error executing action `run` on resource ‘bash[import_images]’