PySpark jobs and hsfs problem


using hsfs on my local machine I created a feature group and save a dataframe with the following code:

fg_target = fs.create_feature_group(
    description="customerID, Churn",

The PySpark Job is successfully created, however it fails after ca. 30 seconds.

Similarly, PySpark Jupyter notebook crashes just after executing “import hsfs” command:

What could be the source of this problem? The Hopsworks version we use is 2.2, hsfs library 2.2.21.


Hi @Krzysztof

The import failure logs at the second screenshot do not look good. Could you please make sure every service is up and running?

To do so, assuming you are an Administrator, go to the traditional Hopsworks UI at https://IP_ADDRESS/hopsworks, click on your email at the top right corner and then Services. All services should be green.

Kind regards,

Thank you @antonios for your reply.
In the meantime we have fully reinstalled Hopsworks (it is single node on-premises installation on Ubuntu 18.04 LTS). The errors however are still valid. I have checked the services, as you suggested, everything seems correct:

Is there any procedure to verify that the services are up and running? Is there any other way to check logs for services?


Hi @Krzysztof!

When you installed hopsworks did you use the installer script? ( or Also are you still on hopsworks 2.2 after the reinstall?

Best regards, Robin

Hi @Robin_Andersson

After the reinstall have upgraded Hopsworks to 2.4 with use of the following script:

Kind regards

Hi @Krzysztof, we don’t recommend installing hopsworks using the master branch as it may be unstable in periods. Can you instead use

The above will install hopsworks 2.3. But also the 2.4 installation script should be ready now, which you can find at

Best regards,

Hi @Robin_Andersson,
OK, we will reinstall Hopsworks from one of the links you provided, hopefully it will work.

Thank you,

Great @Krzysztof! Can you also let me know which one you choose? And I’ll deploy it to and make sure everything is working correctly.

Sure, we will install 2.4 version:

Kind regards,

Hi @Robin_Andersson, we have reinstalled Hopsworks to version 2.4 from the link I sent previously - unfortunately I have the same error:

During installation all packages have been successfully installed… Do you have any idea on what can be the reason for such behaviour?

Kind regards

Thanks @Krzysztof I’m looking into it now. To make sure it’s not a hardware problem, can you let me know how much total and free RAM you have on the machine? Also if you switch to the Python kernel in your notebook I guess it works to import hsfs?

We have 33 GB RAM memory, 150 GB disk size and 8 CPUs on our machine.
In fact if I switch to “pure” Python kernel the cell “import hsfs” does not start at all - it attempts to start but stops without any warning or error being shown below the cell. This is how it looks after execution of whole notebook:
It seems to me that the issue is just the HSFS library as any other command is working both on Python and PySpark kernel.

Kind regards,

Thanks @Krzysztof!

That’s really helpful. Can you show me the configuration you used to start Jupyter with? You will find it if you go back to Hopsworks. I would like to try and see what happens if you increase the Spark driver memory and try to import hsfs in the PySpark kernel again.

I have used the default configuration - please find it attached below:


Kind regards,

Hi @Robin_Andersson,
I attach the stdout.log file for the PySpark job, which fails. Maybe that can help a little.
Processing: stdout.log… (1.2 KB)

Kind regards,

Thanks @Krzysztof! Can you also send the stderr?

Also does it work to run Spark applications that don’t utilize hsfs? Like SparkPI for example: spark/SparkPi.scala at master · apache/spark · GitHub

Sure, stderr is attached. However, I don’t think it will help… (288 Bytes)
The applications that don’t utilize hsfs (2.4.5 in my case) do work - both Python and Scala (PI value computation) didn’t fail.