No valid role found for this invocation - add_training_dataset_tag

jbdddsai · October 26, 2023, 7:13am

Hello,

when addings tags for the training dataset in the feature store, I am getting an error since the last update to hopsworks to version 3.4.1 with this error: “No valid role found for this invocation”

This is the full stacktrace:

[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Logged in to project, explore it here https://c.app.hopsworks.ai:443/p/112798
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Connected. Call `.close()` to terminate connection gracefully.
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Traceback (most recent call last):
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 50, in <module>
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     res = train_from_best_config(*arg_dict["args"], **arg_dict["kwargs"])
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 44, in train_from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return train.from_best_config(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 165, in from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     model_version = attach_best_model_to_feature_store(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 326, in attach_best_model_to_feature_store
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     feature_view.add_training_dataset_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/feature_view.py", line 2205, in add_training_dataset_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return self._feature_view_engine.add_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/feature_view_engine.py", line 546, in add_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     self._tags_api.add(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/tags_api.py", line 54, in add
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     _client._send_request("PUT", path_params, headers=headers, data=json_value)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/decorators.py", line 35, in if_connected
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return fn(inst, *args, **kwargs)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/client/base.py", line 179, in _send_request
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     raise exceptions.RestAPIError(url, response)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - hsfs.client.exceptions.RestAPIError: Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response:
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg:
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Traceback (most recent call last):
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 50, in <module>
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     res = train_from_best_config(*arg_dict["args"], **arg_dict["kwargs"])
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 44, in train_from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return train.from_best_config(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 165, in from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     model_version = attach_best_model_to_feature_store(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 326, in attach_best_model_to_feature_store
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     feature_view.add_training_dataset_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/feature_view.py", line 2205, in add_training_dataset_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return self._feature_view_engine.add_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/feature_view_engine.py", line 546, in add_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     self._tags_api.add(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/tags_api.py", line 54, in add
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     _client._send_request("PUT", path_params, headers=headers, data=json_value)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/decorators.py", line 35, in if_connected
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return fn(inst, *args, **kwargs)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/client/base.py", line 179, in _send_request
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     raise exceptions.RestAPIError(url, response)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - hsfs.client.exceptions.RestAPIError: Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response:
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg:
[2023-10-25, 15:35:23 UTC] {taskinstance.py:1943} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/decorators/base.py", line 221, in execute
    return_value = super().execute(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 395, in execute
    return super().execute(context=serializable_context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 192, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 602, in execute_callable
    result = self._execute_python_callable_in_subprocess(python_path, tmp_path)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 463, in _execute_python_callable_in_subprocess
    raise AirflowException(error_msg) from None
airflow.exceptions.AirflowException: Process returned non-zero exit status 1.
Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response: 
HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg: 
[2023-10-25, 15:35:23 UTC] {taskinstance.py:1400} INFO - Marking task as FAILED. dag_id=ml_pipeline, task_id=train_from_best_config, execution_date=20231025T121644, start_date=20231025T152642, end_date=20231025T153523
[2023-10-25, 15:35:23 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 35 for task train_from_best_config (Process returned non-zero exit status 1.
Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response: 
HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg: ; 6699)
[2023-10-25, 15:35:23 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2023-10-25, 15:35:23 UTC] {taskinstance.py:2784} INFO - 0 downstream tasks scheduled from follow-on schedule check

Any help is highly appreciated.

mahmoud · October 27, 2023, 7:24am

Hello @jbdddsai ,

Let me take a look. Was it working normally before the update?

Regards,
Mahmoud

jbdddsai · October 27, 2023, 7:41am

Hi @mahmoud,

yeah it was working before when i run the code in airflow before the update.

mahmoud · October 30, 2023, 3:46pm

Hello @jbdddsai ,

We have found an issue with the tag endpoint after update and we have fixed it now, so you should be able to rerun your code

Regards,
Mahmoud

Pavan_Kulkarni · December 30, 2023, 5:53am

Hello,

Getting this error while creating new project in same user, if anyone can help would be appreciated.

Connected. Call .close() to terminate connection gracefully.
Traceback (most recent call last):
File “/home/ubuntu/PycharmProjects/Hopsworks/create_project_hw.py”, line 13, in
connection.create_project(“my_hopsworks_project”, description=“An example Hopsworks project”)
File “/home/ubuntu/PycharmProjects/Hopsworks/lib/python3.8/site-packages/hopsworks/decorators.py”, line 35, in if_connected
return fn(inst, *args, **kwargs)
File “/home/ubuntu/PycharmProjects/Hopsworks/lib/python3.8/site-packages/hopsworks/connection.py”, line 151, in create_project
return self._project_api._create_project(name, description, feature_store_topic)
File “/home/ubuntu/PycharmProjects/Hopsworks/lib/python3.8/site-packages/hopsworks/core/project_api.py”, line 102, in _create_project
_client._send_request(
File “/home/ubuntu/PycharmProjects/Hopsworks/lib/python3.8/site-packages/hopsworks/decorators.py”, line 35, in if_connected
return fn(inst, *args, **kwargs)
File “/home/ubuntu/PycharmProjects/Hopsworks/lib/python3.8/site-packages/hopsworks/client/base.py”, line 174, in _send_request
raise exceptions.RestAPIError(url, response)
hopsworks.client.exceptions.RestAPIError: Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project). Server response:
HTTP code: 403, HTTP reason: Forbidden, body: b’{“errorCode”:320003,“errorMsg”:“No valid role found for this invocation”}', error code: 320003, error msg: No valid role found for this invocation, user msg:

mahmoud · January 4, 2024, 1:57pm

Hello @Pavan_Kulkarni ,

I see that you already have a project under your user, the error above is intentional since you cannot create a new project on the same account programmatically. Can you just use the existing project or do you need to create a new project?

Regards,
Mahmoud