Dependency Issues with Hopsworks 3.2.0 and fastavro

#1

Issue

I have mighty issues installing hopsworks on my local machine with poetry or pip. One package seem to be required with specific version numbers that cannot be installed at the moment:

fastavro

Error Logs

The following error code is returned.

1 error generated. error: command '/usr/bin/clang' failed with exit code 1

Note: This error originates from the build backend, and is likely not a problem with poetry but with fastavro (1.7.3) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "fastavro (==1.7.3)"'.

Attempted Remedy

I have so far tried to install the packages separately but this runs into version issues with hopsworks 3.2.0.

Machine:
Hardware: MacBook M2 (2023), Apple M2 Max, 64 GB Memory, Ventura 13.4

If more logs are required I will added them as soon as possible.

#2

Hi @llctrautmann,

Thanks for flagging this. What Python version are you using?

–
Fabio

#3

Hi Fabio,

I am using python version 3.10.1 within a poetry virtual environment. Fwiw, I had the same error with confluent-kafka, and there is a solution in your documentation. So maybe that is related but I couldn’t get it to work with fastarvo.

This is the error code I get when installing the packages seperately and the most recent version. This was for confulent-kafka, but I got the same error for fastavro.

Because hsfs (3.2.0) depends on confluent-kafka (<=1.9.0) and no versions of hsfs match >3.2.0,<3.3.0, hsfs (>=3.2.0,<3.3.0) requires confluent-kafka (<=1.9.0). And because hopsworks (3.2.0) depends on hsfs (>=3.2.0,<3.3.0), hopsworks (3.2.0) requires confluent-kafka (<=1.9.0). So, because <PROJECT> depends on both confluent-kafka (2.1.0) and hopsworks (3.2.0), version solving failed.

i was also able to install hopsworks 3.3.0rc (not 100% sure on the letters in the version number) but that gave me HttpsConnectionError on port 433 issues and I cannot seem to replicate the installation again at the moment for some reason.

Hope this helps.

#4

Hi @llctrautmann ,

The issue is related to fastavro doesn't compile with Cython 3 · Issue #701 · fastavro/fastavro · GitHub

The Hopsworks library has a dependency on: "fastavro>=1.4.11,<=1.7.3". The fix for the issue above is available in fastavro 1.8.1. We are making a new release of the Hopsworks library to relax the fastavro version constraints.

I’ll let you know when the release is going to be available on PyPi.

In the meantime you can try this workaround to get fastavro installed: fastavro doesn't compile with Cython 3 · Issue #701 · fastavro/fastavro · GitHub

–
Fabio

#5

Hi @llctrautmann

This should be solved in the latest release of hsfs - https://pypi.org/project/hsfs/3.2.1/
Let me know if that works for you

–
Fabio

#6

Hi Fabio,

I can confirm that the install now works. Thank you for the great support and quick fix.

Luca

EDIT: The package install now works but I cannot import the package. I get the following error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 7
      5 import pandas as pd
      6 from functions import *
----> 7 import hsfs

File ~/Desktop/ML/TaxiDemandML/.venv/lib/python3.10/site-packages/hsfs/__init__.py:20
     17 import warnings
     19 from hsfs import util, version
---> 20 from hsfs.connection import Connection
     22 __version__ = version.__version__
     24 connection = Connection.connection

File ~/Desktop/ML/TaxiDemandML/.venv/lib/python3.10/site-packages/hsfs/connection.py:23
     20 from requests.exceptions import ConnectionError
     22 from hsfs.decorators import connected, not_connected
---> 23 from hsfs import engine, client, util
     24 from hsfs.core import feature_store_api, project_api, hosts_api, services_api
     26 AWS_DEFAULT_REGION = "default"

File ~/Desktop/ML/TaxiDemandML/.venv/lib/python3.10/site-packages/hsfs/engine/__init__.py:17
      1 #
      2 #   Copyright 2020 Logical Clocks AB
      3 #
...
    110     )
    112 @property
    113 def id(self) -> Optional[int]:

AttributeError: module 'great_expectations' has no attribute 'core'
#7

Restarting the kernel solved the issue. So I think this works now.

1 Like