Hi,
I am trying to set up a single node instance using Docker. May I know if anyone has tried before? If you have, could you share the Dockerfile? Thanks.
Hi,
Running Hopsworks itself in Docker is currently not supported. Doing so will require quite some effort but let us now if you get something working.
Hello, Steffen!
I was also trying to setup a single node instance using Docker. Could you please give a simple explanation why it isn’t possible to setup Hopsworks this way?
Hi!
We simply do not officially test and support it. It’s surely doable but not straightforward. If you try and get stuck, let us know where and we’ll see if we can help.
One big technical issue is that many services in Hopsworks are systemd services. We manage their lifecycle with systemd and it is not supported in Docker.
Hi,
I am interested in this too and in theory it is possible to start a privileged container to support systemd services as stated in this post. But there are really lots of services coming with a single node of hopsworks. I am not sure if a container is supposed to run all of the services.
A better way would be to separate the Karamel and DAG controller into a container to orchestrate further services in containers. This can be done on a Kubernetes cluster. I can imagine that would be really a lot of work for the Hopsworks to do. I would be grateful to know whether this refactoring is on the road map.
@martaq My org switched to Jetbrain Datalore, Airflow, Feast on Kubernetes cluster to achieve the same MLOps task as using Hopsworks a long time ago, since we were not able to upgrade the hopsworks feature store to newer version on-prem. So i gave up on containerizing hopsworks.
I see, thanks for the info!