Is airflow configurable in hopsworks enterprise?

dsiegel · February 15, 2022, 6:10pm

Hello,
I would like to understand the integration of airflow within Hopsworks (specifically hopsworks enterprise). Can hopsworks jobs in the UI point to an already existing airflow server on k8s?
Thank you.

Alex · February 21, 2022, 1:47pm

Hi @dsiegel,

Do you maybe mean the other way? Can airflow pipelines point and start jobs in the Hopsworks UI? Jobs are not aware of pipelines and airflow.

Running Hopsworks jobs from a airflow pipeline:
https://hopsworks.readthedocs.io/en/stable/user_guide/hopsworks/airflow.html?highlight=airflow%20operator#hopsworkslaunchoperator
and example of such a operator usage:

github.com

logicalclocks/hops-examples/blob/release-1.1/airflow/launch_jobs.py

## NOTES:
##
## Example Airflow DAG for launching jobs on Hopsworks
## This example assumes that you have already defined
## four jobs from the Jobs UI.
##
## You can create a Spark tour project and copy the
## demo job.
##
## In this example we will launch one job and wait for
## it to finish successfully. Then launch two more jobs
## in parallel. We will wait only for one of them and
## only on successful completion we will launch the last
## job.

import airflow

from datetime import datetime, timedelta
from airflow import DAG

This file has been truncated. show original

If this is not what you meant, could you please re-phrase your question.

Regards,
Alex

dsiegel · March 2, 2022, 12:32am

Thanks for reply Alex. I mean are we limited to use only airflow server which is setup on the hopsworks cluster or can we point hopsworks to another airflow server which we have?

Alex · March 7, 2022, 1:49pm

Hi @dsiegel,

Unfortunately we currently do not support connecting to an existing airflow server.

Alex