I have tried to connect to feature store in a python3.8.10 env according to this doc: Python - Hopsworks Documentation
While installing
!{sys.executable} -m pip install --no-cache-dir --user --upgrade hsfs[hive]~=2.4.0
I got an error with twofish:
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas->hsfs[hive]~=2.4.0) (2021.1)
Collecting future
Downloading future-0.18.2.tar.gz (829 kB)
|████████████████████████████████| 829 kB 68.2 MB/s eta 0:00:01
Collecting thrift>=0.10.0
Downloading thrift-0.15.0.tar.gz (59 kB)
|████████████████████████████████| 59 kB 78.4 MB/s eta 0:00:01
Requirement already satisfied: pyasn1>=0.3.5 in /opt/conda/lib/python3.8/site-packages (from pyjks->hsfs[hive]~=2.4.0) (0.4.8)
Collecting javaobj-py3
Downloading javaobj_py3-0.4.3-py2.py3-none-any.whl (57 kB)
|████████████████████████████████| 57 kB 58.5 MB/s eta 0:00:01
Collecting twofish
Downloading twofish-0.3.0.tar.gz (26 kB)
Collecting pycryptodomex
Downloading pycryptodomex-3.12.0-cp35-abi3-manylinux2010_x86_64.whl (2.0 MB)
|████████████████████████████████| 2.0 MB 56.3 MB/s eta 0:00:01
Requirement already satisfied: pyasn1-modules in /opt/conda/lib/python3.8/site-packages (from pyjks->hsfs[hive]~=2.4.0) (0.2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests->hsfs[hive]~=2.4.0) (2021.10.8)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->hsfs[hive]~=2.4.0) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests->hsfs[hive]~=2.4.0) (4.0.0)
Collecting greenlet!=0.4.17
Downloading greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (156 kB)
|████████████████████████████████| 156 kB 74.5 MB/s eta 0:00:01
Building wheels for collected packages: hsfs, pyhopshive, thrift, future, twofish
Building wheel for hsfs (setup.py) ... done
Created wheel for hsfs: filename=hsfs-2.4.7-py3-none-any.whl size=146223 sha256=6f5df988fef81ca0cd6f0e85f4bcc7c133a17865ead4a3d6dc9c58089751fd45
Stored in directory: /tmp/pip-ephem-wheel-cache-a7517smc/wheels/28/f8/42/a4f98a10597f1862b5f25b0108826f2cb965adbaf340651fb4
Building wheel for pyhopshive (setup.py) ... done
Created wheel for pyhopshive: filename=PyHopsHive-0.6.4.dev0-py3-none-any.whl size=48438 sha256=ad1da2497a90e80f5d101f893c20f39aa7ccc8b606f54d9838da834b68156ae2
Stored in directory: /tmp/pip-ephem-wheel-cache-a7517smc/wheels/df/bd/31/fe991868ddc1a45261f5b8e0c41a9e0dcc688106ce50ecfecc
Building wheel for thrift (setup.py) ... done
Created wheel for thrift: filename=thrift-0.15.0-py3-none-any.whl size=155878 sha256=da1617d50c9f77fb22ddb42779aeab95dfe57b57fa0666cd6216a8a02ba640c6
Stored in directory: /tmp/pip-ephem-wheel-cache-a7517smc/wheels/4c/b5/5b/10af165d7e0895afdfe25ad487422ae8ada6ea422b0dc444ab
Building wheel for future (setup.py) ... done
Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=a76e31114d24948b52a8be7624621466349f983eeee3ac937e6ef2dd15d7ee40
Stored in directory: /tmp/pip-ephem-wheel-cache-a7517smc/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4
Building wheel for twofish (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ss34cwr8
cwd: /tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/
Complete output (60 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
copying twofish.py -> build/lib.linux-x86_64-3.8
running build_ext
building '_twofish' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/twofish-0.3
gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Itwofish-0.3 -I/opt/conda/include/python3.8 -c twofish-0.3/twofish.c -o build/temp.linux-x86_64-3.8/twofish-0.3/twofish.o
In file included from twofish-0.3/twofish.c:174:
twofish-0.3/twofish.h:86:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
extern void Twofish_initialise();
^~~~~~
twofish-0.3/twofish.c:479:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_platform()
^~~~~~~~~~~~~
twofish-0.3/twofish.c:658:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_vectors()
^~~~~~~~~~~~
twofish-0.3/twofish.c:794:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_sequences()
^~~~~~~~~~~~~~
twofish-0.3/twofish.c:827:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_odd_sized_keys()
^~~~~~~~~~~~~~~~~~~
twofish-0.3/twofish.c:885:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void self_test()
^~~~~~~~~
twofish-0.3/twofish.c:999:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void initialise_q_boxes() {
^~~~~~~~~~~~~~~~~~
twofish-0.3/twofish.c:1043:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void initialise_mds_tables()
^~~~~~~~~~~~~~~~~~~~~
twofish-0.3/twofish.c:1231:6: warning: function declaration isn't a prototype [-Wstrict-prototypes]
void Twofish_initialise()
^~~~~~~~~~~~~~~~~~
gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Itwofish-0.3 -I/opt/conda/include/python3.8 -c twofish.c -o build/temp.linux-x86_64-3.8/twofish.o
In file included from twofish.c:12:
twofish-0.3/twofish.h:86:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
extern void Twofish_initialise();
^~~~~~
twofish.c:25:17: warning: function declaration isn't a prototype [-Wstrict-prototypes]
DL_EXPORT(void) exp_Twofish_initialise() {
^~~~~~~~~~~~~~~~~~~~~~
twofish.c: In function 'init_twofish':
twofish.c:45:1: warning: control reaches end of non-void function [-Wreturn-type]
PyMODINIT_FUNC init_twofish(void) { }
^~~~~~~~~~~~~~
twofish.c: In function 'PyInit__twofish':
twofish.c:46:1: warning: control reaches end of non-void function [-Wreturn-type]
PyMODINIT_FUNC PyInit__twofish(void) { }
^~~~~~~~~~~~~~
gcc -pthread -shared -B /opt/conda/compiler_compat -L/opt/conda/lib -Wl,-rpath=/opt/conda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/twofish-0.3/twofish.o build/temp.linux-x86_64-3.8/twofish.o -o build/lib.linux-x86_64-3.8/_twofish.cpython-38-x86_64-linux-gnu.so
/opt/conda/compiler_compat/ld: cannot find /lib64/libpthread.so.0
/opt/conda/compiler_compat/ld: cannot find /usr/lib64/libpthread_nonshared.a
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for twofish
Running setup.py clean for twofish
Successfully built hsfs pyhopshive thrift future
Failed to build twofish
Installing collected packages: twofish, pycryptodomex, javaobj-py3, s3transfer, pyjks, orderedmultidict, greenlet, future, thrift, sqlalchemy, PyMySQL, pyhumps, pyhopshive, mock, furl, boto3, hsfs
Running setup.py install for twofish ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-03d5x7so/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/jovyan/.local/include/python3.8/twofish
cwd: /tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/
Complete output (60 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
copying twofish.py -> build/lib.linux-x86_64-3.8
running build_ext
building '_twofish' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/twofish-0.3
gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Itwofish-0.3 -I/opt/conda/include/python3.8 -c twofish-0.3/twofish.c -o build/temp.linux-x86_64-3.8/twofish-0.3/twofish.o
In file included from twofish-0.3/twofish.c:174:
twofish-0.3/twofish.h:86:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
extern void Twofish_initialise();
^~~~~~
twofish-0.3/twofish.c:479:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_platform()
^~~~~~~~~~~~~
twofish-0.3/twofish.c:658:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_vectors()
^~~~~~~~~~~~
twofish-0.3/twofish.c:794:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_sequences()
^~~~~~~~~~~~~~
twofish-0.3/twofish.c:827:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void test_odd_sized_keys()
^~~~~~~~~~~~~~~~~~~
twofish-0.3/twofish.c:885:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void self_test()
^~~~~~~~~
twofish-0.3/twofish.c:999:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void initialise_q_boxes() {
^~~~~~~~~~~~~~~~~~
twofish-0.3/twofish.c:1043:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
static void initialise_mds_tables()
^~~~~~~~~~~~~~~~~~~~~
twofish-0.3/twofish.c:1231:6: warning: function declaration isn't a prototype [-Wstrict-prototypes]
void Twofish_initialise()
^~~~~~~~~~~~~~~~~~
gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Itwofish-0.3 -I/opt/conda/include/python3.8 -c twofish.c -o build/temp.linux-x86_64-3.8/twofish.o
In file included from twofish.c:12:
twofish-0.3/twofish.h:86:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
extern void Twofish_initialise();
^~~~~~
twofish.c:25:17: warning: function declaration isn't a prototype [-Wstrict-prototypes]
DL_EXPORT(void) exp_Twofish_initialise() {
^~~~~~~~~~~~~~~~~~~~~~
twofish.c: In function 'init_twofish':
twofish.c:45:1: warning: control reaches end of non-void function [-Wreturn-type]
PyMODINIT_FUNC init_twofish(void) { }
^~~~~~~~~~~~~~
twofish.c: In function 'PyInit__twofish':
twofish.c:46:1: warning: control reaches end of non-void function [-Wreturn-type]
PyMODINIT_FUNC PyInit__twofish(void) { }
^~~~~~~~~~~~~~
gcc -pthread -shared -B /opt/conda/compiler_compat -L/opt/conda/lib -Wl,-rpath=/opt/conda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/twofish-0.3/twofish.o build/temp.linux-x86_64-3.8/twofish.o -o build/lib.linux-x86_64-3.8/_twofish.cpython-38-x86_64-linux-gnu.so
/opt/conda/compiler_compat/ld: cannot find /lib64/libpthread.so.0
/opt/conda/compiler_compat/ld: cannot find /usr/lib64/libpthread_nonshared.a
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /opt/conda/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mjpfyvnb/twofish_391887dcf3854563b11eddc863288449/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-03d5x7so/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/jovyan/.local/include/python3.8/twofish Check the logs for full command output.
Is there a particular version of gcc I should use?