diff --git a/bitnami/airflow/README.md b/bitnami/airflow/README.md index 3dacc4545..eeac45eb1 100644 --- a/bitnami/airflow/README.md +++ b/bitnami/airflow/README.md @@ -74,7 +74,7 @@ The following tables lists the configurable parameters of the Kafka chart and th | `gitImage.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | | `updateStrategy` | Update strategy for the stateful set | `RollingUpdate` | | `rollingUpdatePartition` | Partition update strategy | `nil` | -| `airflow.airflowConfigurationConfigMap` | Name of an existing config map containing the Airflow config file | `nil` | +| `airflow.configurationConfigMap` | Name of an existing config map containing the Airflow config file | `nil` | | `airflow.dagsConfigMap` | Name of an existing config map containing all the DAGs files you want to load in Airflow. | `nil` | | `airflow.loadExamples` | Switch to load some Airflow examples | `true` | | `airflow.cloneDagFilesFromGit.enabled` | Enable in order to download DAG files from git repository. | `false` | @@ -84,8 +84,8 @@ The following tables lists the configurable parameters of the Kafka chart and th | `airflow.worker.port` | Airflow Worker port | `8793` | | `airflow.worker.replicas` | Number of Airflow Worker replicas | `2` | | `airflow.auth.forcePassword` | Force users to specify a password | `false` | -| `airflow.auth.airflowUsername` | Username to access web UI | `user` | -| `airflow.auth.airflowPassword` | Password to access web UI | `nil` | +| `airflow.auth.username` | Username to access web UI | `user` | +| `airflow.auth.password` | Password to access web UI | `nil` | | `airflow.auth.fernetKey` | Fernet key to secure connections | `nil` | | `airflow.auth.existingSecret` | Name of an existing secret containing airflow password and fernet key | `nil` | | `airflow.extraEnvVars` | Extra environment variables to add to airflow web, worker and scheduler pods | `nil` | @@ -148,8 +148,10 @@ Specify each parameter using the `--set key=value[,key=value]` argument to `helm ```console $ helm install --name my-release \ - --set airflow.auth.airflowUsername=my-user,airflow.auth.airflowPassword=my-passsword \ - bitnami/airflow + --set airflow.auth.username=my-user \ + --set airflow.auth.password=my-passsword \ + --set airflow.auth.fernetKey=my-fernet-key \ + bitnami/airflow ``` The above command sets the credentials to access the Airflow web UI. @@ -166,9 +168,15 @@ $ helm install --name my-release -f values.yaml bitnami/airflow The Bitnami Airflow chart relies on the PostgreSQL chart persistence. This means that Airflow does not persist anything. +## Generate a Fernet key + +A Fernet key is required in order to encrypt password within connections. The Fernet key must be a base64-encoded 32-byte key. + +Learn how to generate one [here](https://bcb.github.io/airflow/fernet-key) + ## Load DAG files -There are three different ways to load your custom DAG files into the Airflow chart: +There are three different ways to load your custom DAG files into the Airflow chart. All of them are compatible so you can use more than one at the same time. ### Option 1: Load locally from the `files` folder diff --git a/bitnami/airflow/templates/NOTES.txt b/bitnami/airflow/templates/NOTES.txt index ee4d54101..67023e306 100644 --- a/bitnami/airflow/templates/NOTES.txt +++ b/bitnami/airflow/templates/NOTES.txt @@ -44,7 +44,7 @@ host. To configure Airflow with the URL of your service: 2. Complete your Airflow deployment by running: helm upgrade {{ .Release.Name }} bitnami/{{ .Chart.Name }} \ - --set service.type={{ .Values.service.type }},airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.airflowPassword=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD{{- if .Values.global }}{{- if .Values.global.imagePullSecrets }},global.imagePullSecrets={{ .Values.global.imagePullSecrets }}{{- end }}{{- end }} + --set service.type={{ .Values.service.type }},airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.password=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD{{- if .Values.global }}{{- if .Values.global.imagePullSecrets }},global.imagePullSecrets={{ .Values.global.imagePullSecrets }}{{- end }}{{- end }} {{- else -}} @@ -72,7 +72,7 @@ host. To configure Airflow with the URL of your service: 2. Get your Airflow login credentials by running: - echo User: {{ .Values.airflow.auth.airflowUsername }} + echo User: {{ .Values.airflow.auth.username }} echo Password: $(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} -o jsonpath="{.data.airflow-password}" | base64 --decode) {{- end }} \ No newline at end of file diff --git a/bitnami/airflow/templates/_helpers.tpl b/bitnami/airflow/templates/_helpers.tpl index 80c5c29d6..095610006 100644 --- a/bitnami/airflow/templates/_helpers.tpl +++ b/bitnami/airflow/templates/_helpers.tpl @@ -224,14 +224,14 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} {{- end -}} -{{ template "airflow.dagFilesConfigMap" . }} + {{/* -Get the DAG files ConfigMap name. +Get the secret name */}} -{{- define "airflow.dagFilesConfigMap" -}} -{{- if .Values.airflow.dagsConfigMap -}} -{{- printf "%s" .Values.airflow.dagsConfigMap -}} +{{- define "airflow.secretName" -}} +{{- if .Values.airflow.auth.existingSecret -}} +{{- printf "%s" .Values.airflow.auth.existingSecret -}} {{- else -}} -{{- printf "%s-dag-files" (include "airflow.fullname" .) -}} +{{- printf "%s" (include "airflow.fullname" .) -}} {{- end -}} {{- end -}} \ No newline at end of file diff --git a/bitnami/airflow/templates/configmap-dags.yaml b/bitnami/airflow/templates/configmap-dags.yaml index 3dab10817..569377c32 100644 --- a/bitnami/airflow/templates/configmap-dags.yaml +++ b/bitnami/airflow/templates/configmap-dags.yaml @@ -1,4 +1,4 @@ -{{- if and (.Files.Glob "files/dags/*.py") (not .Values.airflow.dagsConfigMap) }} +{{- if .Files.Glob "files/dags/*.py" }} apiVersion: v1 kind: ConfigMap metadata: @@ -9,5 +9,5 @@ metadata: app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/managed-by: {{ .Release.Service }} data: -{{ (.Files.Glob "files/dags/*.{py}").AsConfig | indent 2 }} +{{ (.Files.Glob "files/dags/*.py").AsConfig | indent 2 }} {{ end }} \ No newline at end of file diff --git a/bitnami/airflow/templates/deployment-scheduler.yaml b/bitnami/airflow/templates/deployment-scheduler.yaml index f659b9edd..67a95fafc 100644 --- a/bitnami/airflow/templates/deployment-scheduler.yaml +++ b/bitnami/airflow/templates/deployment-scheduler.yaml @@ -45,14 +45,14 @@ spec: fsGroup: {{ .Values.securityContext.fsGroup }} runAsUser: {{ .Values.securityContext.runAsUser }} {{- end }} - {{- if and .Values.airflow.cloneDagFilesFromGit.enabled (not .Values.airflow.dagsConfigMap) }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} initContainers: - name: git-clone-repository image: "{{ template "git.image" . }}" imagePullPolicy: {{ .Values.git.pullPolicy | quote }} command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}'] volumeMounts: - - name: custom-dag-files + - name: github-cloned-dag-files mountPath: /dags {{- end }} containers: @@ -149,11 +149,19 @@ spec: readinessProbe: {{- end }} volumeMounts: - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }} - - name: custom-dag-files - mountPath: /opt/bitnami/airflow/dags + {{- if .Files.Glob "files/dags/*.py" }} + - name: local-dag-files + mountPath: /opt/bitnami/airflow/dags/local {{- end }} - {{- if .Values.airflow.airflowConfigurationConfigMap }} + {{- if .Values.airflow.dagsConfigMap }} + - name: external-dag-files + mountPath: /opt/bitnami/airflow/dags/external + {{- end }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} + - name: github-cloned-dag-files + mountPath: /opt/bitnami/airflow/dags/github + {{- end }} + {{- if .Values.airflow.configurationConfigMap }} - name: custom-configuration-file mountPath: /opt/bitnami/airflow/airflow.cfg subPath: airflow.cfg @@ -161,17 +169,22 @@ spec: resources: {{ toYaml .Values.resources | indent 10 }} volumes: - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }} - - name: custom-dag-files - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap }} + {{- if .Files.Glob "files/dags/*.py" }} + - name: local-dag-files configMap: - name: {{ template "airflow.dagFilesConfigMap" . }} - {{- else if .Values.airflow.cloneDagFilesFromGit.enabled }} - emptyDir: {} - {{- end }} + name: {{ template "airflow.fullname" . }}-dag-files {{- end }} - {{- if .Values.airflow.airflowConfigurationConfigMap }} + {{- if .Values.airflow.dagsConfigMap }} + - name: external-dag-files + configMap: + name: {{ .Values.airflow.dagsConfigMap }} + {{- end }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} + - name: github-cloned-dag-files + emptyDir: {} + {{- end }} + {{- if .Values.airflow.configurationConfigMap }} - name: custom-configuration-file configMap: - name: {{ .Values.airflow.airflowConfigurationConfigMap }} + name: {{ .Values.airflow.configurationConfigMap }} {{- end }} \ No newline at end of file diff --git a/bitnami/airflow/templates/deployment-web.yaml b/bitnami/airflow/templates/deployment-web.yaml index be25e27cc..7a917f57b 100644 --- a/bitnami/airflow/templates/deployment-web.yaml +++ b/bitnami/airflow/templates/deployment-web.yaml @@ -45,14 +45,14 @@ spec: fsGroup: {{ .Values.securityContext.fsGroup }} runAsUser: {{ .Values.securityContext.runAsUser }} {{- end }} - {{- if and .Values.airflow.cloneDagFilesFromGit.enabled (not .Values.airflow.dagsConfigMap) }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} initContainers: - name: git-clone-repository image: "{{ template "git.image" . }}" imagePullPolicy: {{ .Values.git.pullPolicy | quote }} command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}'] volumeMounts: - - name: custom-dag-files + - name: github-cloned-dag-files mountPath: /dags {{- end }} containers: @@ -127,7 +127,7 @@ spec: - name: AIRFLOW_EXECUTOR value: "CeleryExecutor" - name: AIRFLOW_USERNAME - value: {{ .Values.airflow.auth.airflowUsername }} + value: {{ .Values.airflow.auth.username }} - name: AIRFLOW_PASSWORD valueFrom: secretKeyRef: @@ -179,11 +179,19 @@ spec: failureThreshold: {{ .Values.readinessProbe.failureThreshold }} {{- end }} volumeMounts: - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }} - - name: custom-dag-files - mountPath: /opt/bitnami/airflow/dags + {{- if .Files.Glob "files/dags/*.py" }} + - name: local-dag-files + mountPath: /opt/bitnami/airflow/dags/local {{- end }} - {{- if .Values.airflow.airflowConfigurationConfigMap }} + {{- if .Values.airflow.dagsConfigMap }} + - name: external-dag-files + mountPath: /opt/bitnami/airflow/dags/external + {{- end }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} + - name: github-cloned-dag-files + mountPath: /opt/bitnami/airflow/dags/github + {{- end }} + {{- if .Values.airflow.configurationConfigMap }} - name: custom-configuration-file mountPath: /opt/bitnami/airflow/airflow.cfg subPath: airflow.cfg @@ -191,17 +199,22 @@ spec: resources: {{ toYaml .Values.resources | indent 10 }} volumes: - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }} - - name: custom-dag-files - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap }} + {{- if .Files.Glob "files/dags/*.py" }} + - name: local-dag-files configMap: - name: {{ template "airflow.dagFilesConfigMap" . }} - {{- else if .Values.airflow.cloneDagFilesFromGit.enabled }} - emptyDir: {} - {{- end }} + name: {{ template "airflow.fullname" . }}-dag-files {{- end }} - {{- if .Values.airflow.airflowConfigurationConfigMap }} + {{- if .Values.airflow.dagsConfigMap }} + - name: external-dag-files + configMap: + name: {{ .Values.airflow.dagsConfigMap }} + {{- end }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} + - name: github-cloned-dag-files + emptyDir: {} + {{- end }} + {{- if .Values.airflow.configurationConfigMap }} - name: custom-configuration-file configMap: - name: {{ .Values.airflow.airflowConfigurationConfigMap }} + name: {{ .Values.airflow.configurationConfigMap }} {{- end }} \ No newline at end of file diff --git a/bitnami/airflow/templates/secrets.yaml b/bitnami/airflow/templates/secrets.yaml index c7336f38d..d3678825a 100644 --- a/bitnami/airflow/templates/secrets.yaml +++ b/bitnami/airflow/templates/secrets.yaml @@ -10,12 +10,12 @@ metadata: app.kubernetes.io/managed-by: {{ .Release.Service }} type: Opaque data: - {{- if .Values.airflow.auth.airflowPassword }} - airflow-password: {{ .Values.airflow.auth.airflowPassword | b64enc | quote }} + {{- if .Values.airflow.auth.password }} + airflow-password: {{ .Values.airflow.auth.password | b64enc | quote }} {{ else if (not .Values.airflow.auth.forcePassword) }} airflow-password: {{ randAlphaNum 10 | b64enc | quote }} {{ else }} - airflow-password: {{ required "An Airflow Password is required!" .Values.airflow.auth.airflowPassword }} + airflow-password: {{ required "An Airflow Password is required!" .Values.airflow.auth.password }} {{- end }} {{- if .Values.airflow.auth.fernetKey }} airflow-fernetKey: {{ .Values.airflow.auth.fernetKey | b64enc | quote }} diff --git a/bitnami/airflow/templates/statefulset-worker.yaml b/bitnami/airflow/templates/statefulset-worker.yaml index c00f4c3f0..7caed12c0 100644 --- a/bitnami/airflow/templates/statefulset-worker.yaml +++ b/bitnami/airflow/templates/statefulset-worker.yaml @@ -49,14 +49,14 @@ spec: affinity: {{ toYaml .Values.affinity | indent 8 }} {{- end }} - {{- if and .Values.airflow.cloneDagFilesFromGit.enabled (not .Values.airflow.dagsConfigMap) }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} initContainers: - name: git-clone-repository image: "{{ template "git.image" . }}" imagePullPolicy: {{ .Values.git.pullPolicy | quote }} command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}'] volumeMounts: - - name: custom-dag-files + - name: github-cloned-dag-files mountPath: /dags {{- end }} containers: @@ -166,27 +166,40 @@ spec: failureThreshold: {{ .Values.readinessProbe.failureThreshold }} {{- end }} volumeMounts: - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }} - - name: custom-dag-files - mountPath: /opt/bitnami/airflow/dags + {{- if .Files.Glob "files/dags/*.py" }} + - name: local-dag-files + mountPath: /opt/bitnami/airflow/dags/local {{- end }} - {{- if .Values.airflow.airflowConfigurationConfigMap }} + {{- if .Values.airflow.dagsConfigMap }} + - name: external-dag-files + mountPath: /opt/bitnami/airflow/dags/external + {{- end }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} + - name: github-cloned-dag-files + mountPath: /opt/bitnami/airflow/dags/github + {{- end }} + {{- if .Values.airflow.configurationConfigMap }} - name: custom-configuration-file mountPath: /opt/bitnami/airflow/airflow.cfg subPath: airflow.cfg {{- end }} volumes: - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }} - - name: custom-dag-files - {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap }} + {{- if .Files.Glob "files/dags/*.py" }} + - name: local-dag-files configMap: - name: {{ template "airflow.dagFilesConfigMap" . }} - {{- else if .Values.airflow.cloneDagFilesFromGit.enabled }} - emptyDir: {} - {{- end }} + name: {{ template "airflow.fullname" . }}-dag-files {{- end }} - {{- if .Values.airflow.airflowConfigurationConfigMap }} + {{- if .Values.airflow.dagsConfigMap }} + - name: external-dag-files + configMap: + name: {{ .Values.airflow.dagsConfigMap }} + {{- end }} + {{- if .Values.airflow.cloneDagFilesFromGit.enabled }} + - name: github-cloned-dag-files + emptyDir: {} + {{- end }} + {{- if .Values.airflow.configurationConfigMap }} - name: custom-configuration-file configMap: - name: {{ .Values.airflow.airflowConfigurationConfigMap }} + name: {{ .Values.airflow.configurationConfigMap }} {{- end }} \ No newline at end of file diff --git a/bitnami/airflow/values-production.yaml b/bitnami/airflow/values-production.yaml index 163c505f4..d3ca6d1b0 100644 --- a/bitnami/airflow/values-production.yaml +++ b/bitnami/airflow/values-production.yaml @@ -109,7 +109,7 @@ updateStrategy: RollingUpdate airflow: ## Name of an existing config map containing the Airflow config file ## - # airflowConfigurationConfigMap: + # configurationConfigMap: ## Name of an existing config map containing all the DAGs files you want to load in Airflow. ## # dagsConfigMap: @@ -137,10 +137,10 @@ airflow: forcePassword: true ## Specify username to acccess web UI ## - airflowUsername: user + username: user ## Specify password to acccess web UI ## - # airflowPassword: + # password: ## Specify a fernet key to secure airflow connections ## Fernet key must be 32 url-safe base64-encoded bytes. ## More info at: @@ -148,7 +148,7 @@ airflow: ## - https://bcb.github.io/airflow/fernet-key ## # fernetKey: - ## Use existing secret (ignores previous airflowPassword and fernetKey) + ## Use existing secret (ignores previous password and fernetKey) ## # existingSecret: diff --git a/bitnami/airflow/values.yaml b/bitnami/airflow/values.yaml index 46d98a97e..fbee75473 100644 --- a/bitnami/airflow/values.yaml +++ b/bitnami/airflow/values.yaml @@ -109,19 +109,19 @@ updateStrategy: RollingUpdate airflow: ## Name of an existing config map containing the Airflow config file ## - # airflowConfigurationConfigMap: + # configurationConfigMap: ## Name of an existing config map containing all the DAGs files you want to load in Airflow. ## - # dagsConfigMap: + dagsConfigMap: patata ## Airflow generic configuration ## - loadExamples: true + loadExamples: false ## Enable in order to download DAG files from git repository. ## cloneDagFilesFromGit: - enabled: false - repository: - revision: + enabled: true + repository: https://github.com/tompizmor/sample-airflow-dag + revision: master ## URL used to access to airflow web ui ## # baseUrl: @@ -137,10 +137,10 @@ airflow: forcePassword: false ## Specify username to acccess web UI ## - airflowUsername: user + username: user ## Specify password to acccess web UI ## - # airflowPassword: + # password: ## Specify a fernet key to secure airflow connections ## Fernet key must be 32 url-safe base64-encoded bytes. ## More info at: @@ -148,7 +148,7 @@ airflow: ## - https://bcb.github.io/airflow/fernet-key ## # fernetKey: - ## Use existing secret (ignores previous airflowPassword and fernetKey) + ## Use existing secret (ignores previous password and fernetKey) ## # existingSecret: