Can I add a worker node separately?

Hopsworks
#1

I’ve already deployed a cluster environment (1 Master Node and 2 Worker Nodes) in our private cloud (VMs). If I want to add 2 worker nodes into this cluster, do I have to re-install the entire cluster environment? If not, is there any document to tell me how to do it?

Any suggestions will be much appreciated.

#2

Hi,

You do not need to re-install the entire cluster, you would need to prepare a karamel cluster definition with the new worker nodes and run the installation with Karamel and this new cluster definition. Did you use karamel or the installer script to for the initial installation?

#3

Hi Theo,

Thank you for your reply.
Yes, I used hopsworks-installer.sh and karamel to install the Hopsworks 2.0, next I’m going to take a look at how to write a karamel cluster definition.
BTW, I’d like to confirm if each worker node runs independently? In other words, if I install a new worker node, will it synchronize data with other nodes automatically?

#4

Hello,

Karamel coordinates installation of the different services and installation on the nodes is done in parallel. You would have to have a karamel cluster definition with all the correct chef attributes set from your first installation, then add the new nodes you want to add, the run chef. It would be much easier to re-install with all the nodes or use hopsworks.ai which has support for adding/removing nodes.

#5

Hi Theo,

Thank you very much for your reply and I apologize for my late reply.

So the better choice is to re-install with all the nodes, right?

BTW，is there any way to upgrade the version of the Jypyter? In other words, if I want to install a specified version of Jypyter, do I have to re-install the entire cluster environment? I’ve looked at the chef file (hopsworks-chef/templates/default/jupyter-launch.sh.erb), but I haven’t found where to specify the version.

#6

Hi,

Yes it would be better to re-install in this case.

It is not possible to install another Jupyter version. Jupyter is pre-installed and configured within a Hopsworks project and that configuration is done in the Python env docker image which is loaded when Hopsworks is installed.

#7

Hi Theo,

Okay. I think I understand.
Thanks for your help. :slight_smile:

#8

@Theo @Freeman How to add nodes using Karamel installation. Assume I have 3 nodes in the cluster. I want to add one more node. For adding the node should I create Karamel cluster definition for all the 4 nodes or one node I want to add? How it will add the new data node to the master node? Can I add a node manually?

1 Like
#9

I’d like to know the answer on this question too. I’ve tried to add new worker node to existing cluster definition and then run the installation with Karamel - it started to re-install entire cluster. I’ve tried to create new cluster definition with only worker node, but it fails with

ERROR: Invalid cookbook/recipe name: hopsworks/default

================================================================================
Recipe Compile Error in /tmp/chef-solo/cookbooks/consul/recipes/slave.rb
================================================================================

ArgumentError
-------------
Invalid Recipe fo cookbook hopsworks

System Info:
------------
chef_version=14.10.9
platform=centos
platform_version=7.9.2009
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/bin/chef-solo
executable=/opt/chefdk/bin/chef-solo