I would like to understand the integration of airflow within Hopsworks (specifically hopsworks enterprise). Can hopsworks jobs in the UI point to an already existing airflow server on k8s?
Do you maybe mean the other way? Can airflow pipelines point and start jobs in the Hopsworks UI? Jobs are not aware of pipelines and airflow.
Running Hopsworks jobs from a airflow pipeline:
and example of such a operator usage:
If this is not what you meant, could you please re-phrase your question.
Thanks for reply Alex. I mean are we limited to use only airflow server which is setup on the hopsworks cluster or can we point hopsworks to another airflow server which we have?
Unfortunately we currently do not support connecting to an existing airflow server.