Deploying Airflow 2 on EKS using Terraform, Helm and ArgoCD — Part 2/2

Learn how to deploy Apache Airflow 2.x on Kubernets using ArgoCD, git-sync, Terraform and helm charts.

airflow 2 deployment

Project Structure

kubernetes folder
  • terraform: it’s the terraform configuration project we have talked about on part 1 of this artigle
  • kubenetes: here we have the helm projects of the applications we want to deploy. In this case, apps/airflow

Airflow Deployment

As mentioned before, we are going to use Airflow Helm chart. We can use the project directly from artifacthub.io or download the source and customize as we want. We going to deploy using the later option.

  1. Go to kubertes/apps/airflow folder
  2. We will use this Helm: https://artifacthub.io/packages/helm/airflow-helm/airflow
  3. In order to download, please follow the commands below:
helm repo add airflow-helm https://airflow-helm.github.io/charts
helm pull airflow-helm/airflow --version 8.5.2
airflow helm folder

Helm Chart Configuration

There are several customization we can do since we have the source helm chart project into our repository.

  • executor: we will use KubernetesExecutor;
  • fernetKey: it’s a critical security value. In this example, the raw value is exposed. Don’t do that in production environment. Please consider using AWS Secret Manager, Kubernetes secrets or any other way you prefer to hide this value;
  • postgresql: enabled: false ( we are setting false because we want to use the RDS we have created before)
  • redis: enable: false
  • externalDatabase: we need to setup the RDS we have created before. Remember we have set up a kubernetes secret for this

ArgoCD Configuration

It’s finally time to configure the ArgoCD and set up the Airflow Helm chart to be deployed on Kubernetes Cluster.

  1. Login into ArgoCD Web UI (http://localhost:8001/api/v1/namespaces/argocd/services/https:argocd-server:443/proxy/)
  2. On the left bar, click on “Manage your repositories, projects, settings”
  3. You will see a page like below. Click on “Repositories”
Argo CD Configuration
Generating ssh-keygen
cat /home/your-user/.ssh/id_rsa_argo.pub
adding new key
cat /home/your-user/.ssh/id_rsa_argo
adding new git
Repo already configured
  • Application Name: airflow2
  • Project: default
  • Sync Policy: Automatic
  • Check Prune Resources
  • Prune Propagation Policy: foreground
  • Source: select the repository we have configured before
  • Revision: HEAD
  • Path: infrastructure/kubernetes/apps/airflow/
  • Destination: select the kubernetes cluster (you will see only one)
  • Namespace: airflow
adding new app
Applications home screen
airflow 2 deployment in progress

Access Airflow Web UI

Now we have Airflow up and running in our AWS EKS cluster. It’s time to see Airflow web UI.

  1. Open a terminal and run below command. It will map the Airflow running on Kubernetes to your localhost:8080
kubectl port-forward svc/airflow2-web 8080:8080 -n airflow
http://localhost:8080
airflow login page
airflow home screen

Summary

At the end of this article you have learned how to deploy a Kubernetes Cluster on AWS EKS using Terraform, how to deploy ArgoCD on EKS using Helm and Terraform and how to deploy Apache Airflow 2.x on EKS using ArgoCD.

Attention

If you let this infrastructure up and running you will be billed by the time it was kept up. So please make sure you destroy everything after your study.

terraform destroy -var-file terraform.tfvars

--

--

I’m a Data Engineer and guitar player.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store