No valid role found for this invocation - add_training_dataset_tag

Hello,

when addings tags for the training dataset in the feature store, I am getting an error since the last update to hopsworks to version 3.4.1 with this error: “No valid role found for this invocation”

This is the full stacktrace:

[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Logged in to project, explore it here https://c.app.hopsworks.ai:443/p/112798
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Connected. Call `.close()` to terminate connection gracefully.
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Traceback (most recent call last):
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 50, in <module>
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     res = train_from_best_config(*arg_dict["args"], **arg_dict["kwargs"])
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 44, in train_from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return train.from_best_config(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 165, in from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     model_version = attach_best_model_to_feature_store(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 326, in attach_best_model_to_feature_store
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     feature_view.add_training_dataset_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/feature_view.py", line 2205, in add_training_dataset_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return self._feature_view_engine.add_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/feature_view_engine.py", line 546, in add_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     self._tags_api.add(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/tags_api.py", line 54, in add
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     _client._send_request("PUT", path_params, headers=headers, data=json_value)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/decorators.py", line 35, in if_connected
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return fn(inst, *args, **kwargs)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/client/base.py", line 179, in _send_request
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     raise exceptions.RestAPIError(url, response)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - hsfs.client.exceptions.RestAPIError: Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response:
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg:
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - Traceback (most recent call last):
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 50, in <module>
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     res = train_from_best_config(*arg_dict["args"], **arg_dict["kwargs"])
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/script.py", line 44, in train_from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return train.from_best_config(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 165, in from_best_config
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     model_version = attach_best_model_to_feature_store(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/training_pipeline/train.py", line 326, in attach_best_model_to_feature_store
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     feature_view.add_training_dataset_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/feature_view.py", line 2205, in add_training_dataset_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return self._feature_view_engine.add_tag(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/feature_view_engine.py", line 546, in add_tag
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     self._tags_api.add(
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/core/tags_api.py", line 54, in add
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     _client._send_request("PUT", path_params, headers=headers, data=json_value)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/decorators.py", line 35, in if_connected
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     return fn(inst, *args, **kwargs)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -   File "/tmp/venv8fwwme8m/lib/python3.9/site-packages/hsfs/client/base.py", line 179, in _send_request
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO -     raise exceptions.RestAPIError(url, response)
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - hsfs.client.exceptions.RestAPIError: Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response:
[2023-10-25, 15:35:20 UTC] {process_utils.py:190} INFO - HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg:
[2023-10-25, 15:35:23 UTC] {taskinstance.py:1943} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/decorators/base.py", line 221, in execute
    return_value = super().execute(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 395, in execute
    return super().execute(context=serializable_context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 192, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 602, in execute_callable
    result = self._execute_python_callable_in_subprocess(python_path, tmp_path)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 463, in _execute_python_callable_in_subprocess
    raise AirflowException(error_msg) from None
airflow.exceptions.AirflowException: Process returned non-zero exit status 1.
Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response: 
HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg: 
[2023-10-25, 15:35:23 UTC] {taskinstance.py:1400} INFO - Marking task as FAILED. dag_id=ml_pipeline, task_id=train_from_best_config, execution_date=20231025T121644, start_date=20231025T152642, end_date=20231025T153523
[2023-10-25, 15:35:23 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 35 for task train_from_best_config (Process returned non-zero exit status 1.
Metadata operation error: (url: https://c.app.hopsworks.ai/hopsworks-api/api/project/112798/featurestores/112717/featureview/energy_consumption_denmark_view/version/1/trainingdatasets/version/1/tags/wandb). Server response: 
HTTP code: 403, HTTP reason: Forbidden, body: b'{"errorCode":320003,"errorMsg":"No valid role found for this invocation"}', error code: 320003, error msg: No valid role found for this invocation, user msg: ; 6699)
[2023-10-25, 15:35:23 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2023-10-25, 15:35:23 UTC] {taskinstance.py:2784} INFO - 0 downstream tasks scheduled from follow-on schedule check

Any help is highly appreciated.

Hello @jbdddsai ,

Let me take a look. Was it working normally before the update?

Regards,
Mahmoud

Hi @mahmoud,

yeah it was working before when i run the code in airflow before the update.

Hello @jbdddsai ,

We have found an issue with the tag endpoint after update and we have fixed it now, so you should be able to rerun your code

Regards,
Mahmoud