I’ve been trying to create an online feature group and have been running into a lot of problems. I couldn’t find a thread on this specific error so I decided to start one.
I’m using hopsworks 3.0.5 on a Python 3.9.10 virtual environment on my macbook. The offending code:
aqi_fg = fs.get_or_create_feature_group(
name=fg_name,
version=1,
description=f’historical air quality index with predictors for {fg_name}',
primary_key=[‘id’],
event_time=‘datetime’,
partition_key=[‘date’],
online_enabled=True
)
aqi_fg.insert(data)
And the exception:
Exception has occurred: RestAPIError
Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/13468/featurestores/13388/featuregroups). Server response:
HTTP code: 500, HTTP reason: Internal Server Error, error code: 120005, error msg: Persistence error occurred, user msg: Duplicate entry ‘chicago_60603_aqi_1_offline_fg_backfill-13468’ for key ‘jobs.name_project_idx’
File “/Users/giorgio/CodingProjects/AQIPrediction/1_feature_gen.py”, line 54, in
aqi_fg.insert(data)
I can see in the website UI that there is this lingering job that I have no option to delete:
On https://app.hopsworks.ai the Job (Ingestion) functionality is read only.
You can bump the version number in the get_or_create_feature_group() and it will generate a new job for the new feature group version.
So I don’t have any feature groups in my feature store right now. I was using an offline feature group, but I needed to query the feature group and the query was taking a long time to run (about two minutes to return a single value). I figured I needed to switch to online storage. I deleted the old feature group to switch to an online one. Then I started to run into the error I already mentioned.
Since I don’t have any feature groups I decided to change the group name in get_or_create_feature_group() rather than bump up the version like you said. That seems to have moved things forward, but I’m now getting an exception related to secrets:
Exception has occurred: RestAPIError
Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/13468/featurestores/13388/featuregroups). Server response:
HTTP code: 500, HTTP reason: Internal Server Error, error code: 270063, error msg: Could not get JDBC connection for the online featurestore, user msg: Problem getting secrets for the JDBC connection to the online FS
File “/Users/giorgio/CodingProjects/AQIPrediction/1_feature_gen.py”, line 54, in
aqi_fg.insert(data)
I saw another thread on this issue but I couldn’t figure things out. I did not have any secrets in my profile, so I tried creating one using the default password for the JDBC online storage connector as a value. Don’t think I did it correctly.
I also can see that there is now another unfinished job in the ingestions list. Any tips? I’m fairly new to all of this so I’m learning as I go.
I think something went wrong during the project setup and the credentials for the online feature store were not generated correctly. I can remove your project and you can try to create a new one if you don’t have anything in it.
Hi,
I have modified you account so that if you log out and log back in app.hopsworks.ai you will be asked to provide a project name again.
Enter a new project name. You will then be redirected to the project page where you will be able to switch between your existing (broken) project and the new one (at the top left).
