Stuck on starting when deploy

Hopsworks
#1

It’s been a long time starting on a very simple model, how can I stop the deployment?

Yes, I already tried to stop with the UI

image
image961×654 35.7 KB

#2

Hi @datorresb ,

Thanks for flagging this - this is a known issue which we need to fix. The deployment didn’t start properly I suspect that’s related to Deployment Error - Scikit-Learn Version Mismatch .

If you give me the project name and the deployment name and version I can clean it up for you.

Regards,
Fabio

#3

https://c.app.hopsworks.ai/p/79010/deployments/52232#models
irisdeployed.dev-mlops.hopsworks.ai

I hope this answer the question :slight_smile:

#4

Hi sir i have the issue too, can you clean it for me, its project name is Recomendation_assets,
Deployment: rankingdeployment version :1 . Tks

#5

Hi,
Same issue here; deployment stuck on starting. Project name: ‘recommender’ , deployment : rankingdeployment version :1
Thanks in advance

#6

Hello @Khanh_Vo_Van,

Sorry for the late response. I checked both deployments you have and both of them have failed to start with some error on the transformer script and that somehow made the deployments stuck into a pending state with no error being reported back. I have stopped both deployments and attached the errors in this message, so you can fix them accordingly and rerun your deployment(s)

 File "../kserve-component-server.py", line 118, in __init__
    super().__init__(name, component)
  File "../kserve-component-server.py", line 90, in __init__
    self._load_component(component)
  File "../kserve-component-server.py", line 127, in _load_component
    self._component = component.Transformer()
  File "/mnt/transformers/transformer-ranking_transformer.py", line 17, in __init__
    self.customer_fv = self.fs.get_feature_view("customers", 1)
  File "/srv/hops/anaconda/envs/theenv/lib/python3.8/site-packages/hsfs/feature_store.py", line 1382, in get_feature_view
    return self._feature_view_engine.get(name, version)
  File "/srv/hops/anaconda/envs/theenv/lib/python3.8/site-packages/hsfs/core/feature_view_engine.py", line 141, in get
    fv = self._feature_view_api.get_by_name_version(name, version)
  File "/srv/hops/anaconda/envs/theenv/lib/python3.8/site-packages/hsfs/core/feature_view_api.py", line 113, in get_by_name_version
    raise e
  File "/srv/hops/anaconda/envs/theenv/lib/python3.8/site-packages/hsfs/core/feature_view_api.py", line 101, in get_by_name_version
    self._client._send_request(
  File "/srv/hops/anaconda/envs/theenv/lib/python3.8/site-packages/hsfs/decorators.py", line 35, in if_connected
    return fn(inst, *args, **kwargs)
  File "/srv/hops/anaconda/envs/theenv/lib/python3.8/site-packages/hsfs/client/base.py", line 171, in _send_request
    raise exceptions.RestAPIError(url, response)
hsfs.client.exceptions.RestAPIError: Metadata operation error: (url: https://hopsworks.glassfish.service.consul:8182/hopsworks-api/api/project/98457/featurestores/98376/featureview/customers/version/1). Server response:
HTTP code: 404, HTTP reason: Not Found, body: b'{"errorCode":270181,"usrMsg":"There exists no feature view with the name customers and version 1.","errorMsg":"Feature view wasn\'t found."}', error code: 270181, error msg: Feature view wasn't found., user msg: There exists no feature view with the name customers and version 1.

and here is the other error:

root INFO [<module>():180] Loading serving script
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Traceback (most recent call last):
  File "../kserve-component-server.py", line 184, in <module>
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/mnt/transformers/transformer-ranking_transformer.py", line 8, in <module>
    import hopsworks
ModuleNotFoundError: No module named 'hopsworks'
#7

Hello @Metin_Turgal

I have checked your deployment and it has failed on the transformer script with the following error:

  File "../kserve-component-server.py", line 191, in <module>
    c_wrapper = TransformerWrapper(args.model_name, args.predictor_host, mod)
  File "../kserve-component-server.py", line 118, in __init__
    super().__init__(name, component)
  File "../kserve-component-server.py", line 90, in __init__
    self._load_component(component)
  File "../kserve-component-server.py", line 127, in _load_component
    self._component = component.Transformer()
  File "/mnt/transformers/transformer-ranking_transformer.py", line 50, in __init__
    self.candidate_index = opensearch_api.get_project_index("candidate_index")
NameError: name 'opensearch_api' is not defined

Are you trying to run the recommender tutorial against app? If so, then that is the issue then- this advanced tutorial is not supported on app at the moment, and you need to run it on your own Hopsworks cluster instance

#8

Hello, I have same situation. It is stuck in starting the deployment. I am not able to proceed. Could you please do necessary cleanup ? Thanks.
https://c.app.hopsworks.ai/p/147605/deployments