Connecting to Hopsworks(feature store) on EC2 instance from local IDE (Python)

Feature Store
#1

I have pip installed hopsworks-cloud-sdk on my machine and was able to successfully connect to Hopsworks feature store running on an AWS EC2 instance. I verified the connection using

import hops.featurestore as fs
'connection code to feature store  is omitted'
fs.project_featurestore()

output: ‘demo_featurestore_admin000_featurestore’

But I get the below error when I try to use other methods like get_feature_list. Please help.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/anaconda/lib/python3.6/site-packages/hops/featurestore.py in get_features_list(featurestore, online)
    375         return fs_utils._do_get_features_list(core._get_featurestore_metadata(featurestore,
--> 376                                                                               update_cache=update_cache_default,),
    377                                               online=online)

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/core.py in _get_featurestore_metadata(featurestore, update_cache)
     74         response_object = rest_rpc._get_featurestore_metadata(featurestore)
---> 75         metadata_cache = FeaturestoreMetadata(response_object)
     76     return metadata_cache

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/dao/common/featurestore_metadata.py in __init__(self, metadata_json)
     24         featuregroups, training_datasets, features_to_featuregroups, featurestore, settings, storage_connectors, \
---> 25             online_featurestore_connector = self._parse_featurestore_metadata(metadata_json)
     26         self.featuregroups = featuregroups

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/dao/common/featurestore_metadata.py in _parse_featurestore_metadata(self, metadata_json)
     65 
---> 66         settings = FeaturestoreSettings(metadata_json[constants.REST_CONFIG.JSON_FEATURESTORE_SETTINGS])
     67         for sc in metadata_json[constants.REST_CONFIG.JSON_FEATURESTORE_STORAGE_CONNECTORS]:

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/dao/settings/featurestore_settings.py in __init__(self, settings_json)
     17         self.entity_name_max_len = \
---> 18             settings_json[constants.REST_CONFIG.JSON_FEATURESTORE_SETTINGS_ENTITY_NAME_MAX_LENGTH]
     19         self.entity_description_max_len = \

KeyError: 'featurestoreEntityNameMaxLength'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-18-a2904ede16de> in <module>()
----> 1 fs.get_features_list()

/anaconda/lib/python3.6/site-packages/hops/featurestore.py in get_features_list(featurestore, online)
    377                                               online=online)
    378     except:
--> 379         return fs_utils._do_get_features_list(core._get_featurestore_metadata(featurestore, update_cache=True,),
    380                                               online=online)
    381 

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/core.py in _get_featurestore_metadata(featurestore, update_cache)
     73     if metadata_cache is None or update_cache:
     74         response_object = rest_rpc._get_featurestore_metadata(featurestore)
---> 75         metadata_cache = FeaturestoreMetadata(response_object)
     76     return metadata_cache
     77 

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/dao/common/featurestore_metadata.py in __init__(self, metadata_json)
     23         """
     24         featuregroups, training_datasets, features_to_featuregroups, featurestore, settings, storage_connectors, \
---> 25             online_featurestore_connector = self._parse_featurestore_metadata(metadata_json)
     26         self.featuregroups = featuregroups
     27         self.training_datasets = training_datasets

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/dao/common/featurestore_metadata.py in _parse_featurestore_metadata(self, metadata_json)
     64                 TrainingDataset(td)
     65 
---> 66         settings = FeaturestoreSettings(metadata_json[constants.REST_CONFIG.JSON_FEATURESTORE_SETTINGS])
     67         for sc in metadata_json[constants.REST_CONFIG.JSON_FEATURESTORE_STORAGE_CONNECTORS]:
     68             if sc[constants.REST_CONFIG.JSON_FEATURESTORE_CONNECTOR_TYPE] == \

/anaconda/lib/python3.6/site-packages/hops/featurestore_impl/dao/settings/featurestore_settings.py in __init__(self, settings_json)
     16         """
     17         self.entity_name_max_len = \
---> 18             settings_json[constants.REST_CONFIG.JSON_FEATURESTORE_SETTINGS_ENTITY_NAME_MAX_LENGTH]
     19         self.entity_description_max_len = \
     20             settings_json[constants.REST_CONFIG.JSON_FEATURESTORE_SETTINGS_ENTITY_DESCRIPTION_MAX_LENGTH]

KeyError: 'featurestoreEntityNameMaxLength'
#2

Hi,
Which version of Hopsworks are you using? From the error message it seems that you are using a newer version of hopsworks-cloud-sdk with an older version of Hopsworks

You can see the Hopsworks version in your Project > Settings > Versions

#3

Here are the versions:

hopsworks-cloud-sdk-1.1.0.0.tar.gz

hopsworks 1.0.0

#4

The version of hopsworks-cloud-sdk you are using is for the Hopsworks 1.1 release can you try with 1.0: https://pypi.org/project/hopsworks-cloud-sdk/1.0.0.4/.

#5

It looks like the connect method for the older version doesn’t have an argument to specify the api-key. Here is how the connect piece looks like and I don’t know how to provide api key from my local workstation using the older SDK version.’

DEFAULT_REGION='us-east-1'
fs.connect('instance address', 'demo_featurestore_admin000',
    port = 443, secrets_store='local',
    region_name=DEFAULT_REGION,
    api_key_file=os.path.abspath('api-key.txt'),
    hostname_verification=False)
#6

Yes that’s correct we introduced that feature later. I see two options for you going forward. You can use hopsworks.ai to spin up a new instance as it will help you to set up the latest release. If you are unable to do so then you can place the secret in the secrets manager as described here: https://hopsworks.readthedocs.io/en/latest/featurestore/featurestore.html#connecting-from-amazon-sagemaker. Look especially for the part Storing the API Key in the AWS Secrets Manager.

#7

I missed the point that you are trying to access from you local device. In that case you’ll have to deploy the newer version to utilize this feature. Hopsworks.ai will ease that a lot if you are able to use it.

#8

Yea I was able to connect to SageMaker with no issues. I will check out the Hopsworks.ai option. Thanks.