home/charts
2
0
Fork 0
mirror of https://github.com/wahyd4/charts.git synced 2026-08-22 11:35:58 +10:00
Code Issues Packages Projects Releases Wiki Activity

[bitnami/airflow] MAJOR: refactor, standards, apiVersion: v2, and adding functionalities (#4069)

Browse Source 
* [bitnami/airflow] MAJOR: refactor, standards and adding functionalities

Signed-off-by: darteaga <darteaga@vmware.com>

* fix: NOTES.txt wrong indentation and missing exports

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add redis and postgresql new major versions

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: restructure folders

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add network policies

Signed-off-by: darteaga <darteaga@vmware.com>

* fix: add executor to values

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: refactor git

Signed-off-by: darteaga <darteaga@vmware.com>

* fix: wrong image template

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add different executors

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add pod autoscaling for workers

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add pod disruption budget to the components

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: refactor values and add standardization

Signed-off-by: darteaga <darteaga@vmware.com>

* fix: redefine extradeploy

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add priorityclassname

Signed-off-by: darteaga <darteaga@vmware.com>

* fix: error with pod template

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add notable changes and upgrade instructions

Signed-off-by: darteaga <darteaga@vmware.com>

* Update bitnami/airflow/README.md

Co-authored-by: Marcos Bjoerkelund <marcosbc@users.noreply.github.com>

* Include changes Helm v2 -> v3

Signed-off-by: juan131 <juanariza@vmware.com>

* Rename git_helpers

* Improve docs about 'auth.existingSecret'

* Format/Fix linter issues in yaml

* feat(airflow): removing ./files/dags/*.py support

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: using git syncer notice

Signed-off-by: darteaga <darteaga@vmware.com>

* feat: add container port

Signed-off-by: darteaga <darteaga@vmware.com>

Co-authored-by: Juan Ariza Toledano <juanariza@vmware.com>
Co-authored-by: Marcos Bjoerkelund <marcosbc@users.noreply.github.com>
Co-authored-by: Carlos Rodriguez Hernandez <crhernandez@bitnami.com>
This commit is contained in:
Daniel Arteaga
2020-11-23 16:25:20 +01:00
committed by GitHub
co-authored by Marcos Bjoerkelund Juan Ariza Toledano Carlos Rodriguez Hernandez
parent ab46e49574
commit 166ac3fa7c
47 changed files with 3327 additions and 2651 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bitnami/airflow/requirements.yaml → bitnami/airflow/Chart.lock
+5 -7
View File
@@ -1,14 +1,12 @@
dependencies:
- name: common
version: 0.x.x
repository: https://charts.bitnami.com/bitnami
tags:
- bitnami-common
version: 0.10.0
- name: postgresql
version: 8.x.x
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
version: 10.0.2
- name: redis
version: 10.x.x
repository: https://charts.bitnami.com/bitnami
condition: redis.enabled
version: 12.1.1
digest: sha256:b4d71b4de0c1c179a7aa8f32f901f2765b0733a4a8a601f3b813d97be5be64b8
generated: '2020-11-19T15:46:30.410005455Z'
bitnami/airflow/Chart.yaml
+25 -11
View File
@@ -1,21 +1,35 @@
apiVersion: v1
name: airflow
version: 6.7.2
annotations:
category: WorkFlow
apiVersion: v2
appVersion: 1.10.12
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
tags:
- bitnami-common
version: 0.x.x
- condition: postgresql.enabled
name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 10.x.x
- condition: redis.enabled
name: redis
repository: https://charts.bitnami.com/bitnami
version: 12.x.x
description: Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
engine: gotpl
home: https://github.com/bitnami/charts/tree/master/bitnami/airflow
icon: https://bitnami.com/assets/stacks/airflow/img/airflow-stack-110x117.png
keywords:
- apache
- airflow
- workflow
- dag
home: https://github.com/bitnami/charts/tree/master/bitnami/airflow
maintainers:
- email: containers@bitnami.com
name: Bitnami
name: airflow
sources:
- https://github.com/bitnami/bitnami-docker-airflow
- https://airflow.apache.org/
maintainers:
- name: Bitnami
email: containers@bitnami.com
engine: gotpl
icon: https://bitnami.com/assets/stacks/airflow/img/airflow-stack-110x117.png
annotations:
category: WorkFlow
version: 7.0.0
bitnami/airflow/README.md
+462 -203
View File
@@ -17,7 +17,7 @@ Bitnami charts can be used with [Kubeapps](https://kubeapps.com/) for deployment
## Prerequisites
- Kubernetes 1.12+
- Helm 2.12+ or Helm 3.0-beta3+
- Helm 3.0-beta3+
## Installing the Chart
@@ -46,193 +46,299 @@ The command removes all the Kubernetes components associated with the chart and
The following tables lists the configurable parameters of the Airflow chart and their default values.
| Parameter | Description | Default |
|--------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|
| `global.imageRegistry` | Global Docker image registry | `nil` |
| `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `global.storageClass` | Global storage class for dynamic provisioning | `nil` |
| `image.registry` | Airflow image registry | `docker.io` |
| `image.repository` | Airflow image name | `bitnami/airflow` |
| `image.tag` | Airflow image tag | `{TAG_NAME}` |
| `image.pullPolicy` | Airflow image pull policy | `IfNotPresent` |
| `image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `image.debug` | Specify if debug values should be set | `false` |
| `schedulerImage.registry` | Airflow Scheduler image registry | `docker.io` |
| `schedulerImage.repository` | Airflow Scheduler image name | `bitnami/airflow-scheduler` |
| `schedulerImage.tag` | Airflow Scheduler image tag | `{TAG_NAME}` |
| `schedulerImage.pullPolicy` | Airflow Scheduler image pull policy | `IfNotPresent` |
| `schedulerImage.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `schedulerImage.debug` | Specify if debug values should be set | `false` |
| `workerImage.registry` | Airflow Worker image registry | `docker.io` |
| `workerImage.repository` | Airflow Worker image name | `bitnami/airflow-worker` |
| `workerImage.tag` | Airflow Worker image tag | `{TAG_NAME}` |
| `workerImage.pullPolicy` | Airflow Worker image pull policy | `IfNotPresent` |
| `workerImage.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `workerImage.debug` | Specify if debug values should be set | `false` |
| `git.registry` | Git image registry | `docker.io` |
| `git.repository` | Git image name | `bitnami/git` |
| `git.tag` | Git image tag | `{TAG_NAME}` |
| `git.pullPolicy` | Git image pull policy | `IfNotPresent` |
| `git.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `nameOverride` | String to partially override airflow.fullname template with a string (will prepend the release name) | `nil` |
| `fullnameOverride` | String to fully override ariflow.fullname template with a string | `nil` |
| `commonLabels` | Labels to add to all deployed objects | `{}` |
| `commonAnnotations` | Annotations to add to all deployed objects | `{}` |
| `updateStrategy` | Update strategy for the stateful set | `RollingUpdate` |
| `rollingUpdatePartition` | Partition update strategy | `nil` |
| `rbac.create` | If true, create & use RBAC resources | `false` |
| `serviceAccount.create` | If true, create a service account | `false` |
| `serviceAccount.name` | The name of the service account to use. If not set and create is true, a name is generated using the fullname template. | `` |
| `serviceAccount.annotations` | Annotations for service account | `{}` |
| `airflow.configurationConfigMap` | Name of an existing config map containing the Airflow config file | `nil` |
| `airflow.dagsConfigMap` | Name of an existing config map containing all the DAGs files you want to load in Airflow. | `nil` |
| `airflow.loadExamples` | Switch to load some Airflow examples | `true` |
| `airflow.gitSyncInterval` | Interval (in seconds) to pull the git repository containing the plugins and/or DAG files | `60` |
| `airflow.gitCloneContainer.extraEnvVars` | Array containing extra env vars | `nil` |
| `airflow.gitCloneContainer.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `airflow.gitCloneContainer.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `airflow.gitCloneContainer.command` | Override default container command (useful when using custom images) | `nil` |
| `airflow.gitCloneContainer.args` | Override default container args (useful when using custom images) | `nil` |
| `airflow.gitCloneContainer.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `airflow.gitSyncContainer.extraEnvVars` | Array containing extra env vars | `nil` |
| `airflow.gitSyncContainer.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `airflow.gitSyncContainer.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `airflow.gitSyncContainer.command` | Override default container command (useful when using custom images) | `nil` |
| `airflow.gitSyncContainer.args` | Override default container args (useful when using custom images) | `nil` |
| `airflow.gitSyncContainer.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `airflow.cloneDagFilesFromGit.enabled` | Enable in order to download DAG files from git repository. | `false` |
| `airflow.cloneDagFilesFromGit.repository` | Repository where download DAG files from | `nil` |
| `airflow.cloneDagFilesFromGit.branch` | Branch from repository to checkout | `nil` |
| `airflow.cloneDagFilesFromGit.path` | Path to a folder in the repository containing DAGs. If not set, all DAGS from the repo are loaded. | `nil` |
| `airflow.cloneDagFilesFromGit.repositories[0].repository` | Repository where download DAG files from | `nil` |
| `airflow.cloneDagFilesFromGit.repositories[0].name` | An unique identifier for repository, must be unique for each repository | `nil` |
| `airflow.cloneDagFilesFromGit.repositories[0].branch` | Branch from repository to checkout | `nil` |
| `airflow.cloneDagFilesFromGit.repositories[0].path` | Path to a folder in the repository containing DAGs. If not set, all DAGS from the repo are loaded. | `nil` |
| `airflow.clonePluginsFromGit.enabled` | Enable in order to download plugins from git repository. | `false` |
| `airflow.clonePluginsFromGit.repository` | Repository where download plugins from | `nil` |
| `airflow.clonePluginsFromGit.branch` | Branch from repository to checkout | `nil` |
| `airflow.clonePluginsFromGit.path` | Path to a folder in the repository containing the plugins. | `nil` |
| `airflow.baseUrl` | URL used to access to airflow web ui | `nil` |
| `airflow.worker.port` | Airflow Worker port | `8793` |
| `airflow.worker.replicas` | Number of Airflow Worker replicas | `2` |
| `airflow.worker.podManagementPolicy` | podManagementPolicy for Worker replicas | `OrderedReady` |
| `airflow.worker.resources.limits` | The resources limits for the Worker containers | `{}` |
| `airflow.worker.resources.requests` | The requested resources for the Worker containers | `{}` |
| `airflow.worker.extraEnvVars` | Array containing extra env vars | `nil` |
| `airflow.worker.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `airflow.worker.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `airflow.worker.command` | Override default container command (useful when using custom images) | `nil` |
| `airflow.worker.args` | Override default container args (useful when using custom images) | `nil` |
| `airflow.worker.podAnnotations` | Additional annotations for Airflow worker pods | [] |
| `airflow.worker.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `airflow.worker.extraVolumes` | Array of extra volumes to be added deployment (evaluated as template). Requires setting `extraVolumeMounts` | `nil` |
| `airflow.auth.forcePassword` | Force users to specify a password | `false` |
| `airflow.auth.username` | Username to access web UI | `user` |
| `airflow.auth.password` | Password to access web UI | `nil` |
| `airflow.auth.fernetKey` | Fernet key to secure connections | `nil` |
| `airflow.auth.existingSecret` | Name of an existing secret containing airflow password and fernet key | `nil` |
| `airflow.extraEnvVars` | Extra environment variables to add to airflow web, worker and scheduler pods | `nil` |
| `airflow.web.resources.limits` | The resources limits for the web containers | `{}` |
| `airflow.web.resources.requests` | The requested resources for the web containers | `{}` |
| `airflow.web.extraEnvVars` | Array containing extra env vars | `nil` |
| `airflow.web.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `airflow.web.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `airflow.web.command` | Override default container command (useful when using custom images) | `nil` |
| `airflow.web.args` | Override default container args (useful when using custom images) | `nil` |
| `airflow.web.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `airflow.web.extraVolumes` | Array of extra volumes to be added deployment (evaluated as template). Requires setting `extraVolumeMounts` | `nil` |
| `airflow.scheduler.resources.limits` | The resources limits for the scheduler containers | `{}` |
| `airflow.scheduler.resources.requests` | The requested resources for the scheduler containers | `{}` |
| `airflow.scheduler.extraEnvVars` | Array containing extra env vars | `nil` |
| `airflow.scheduler.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `airflow.scheduler.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `airflow.scheduler.command` | Override default container command (useful when using custom images) | `nil` |
| `airflow.scheduler.args` | Override default container args (useful when using custom images) | `nil` |
| `airflow.scheduler.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `airflow.scheduler.extraVolumes` | Array of extra volumes to be added deployment (evaluated as template). Requires setting `extraVolumeMounts` | `nil` |
| `airflow.webserverConfigMap` | Config map name for ~/airflow/webserver_config.py | `nil` |
| `ldap.enabled` | Enable LDAP support | `false` |
| `ldap.uri` | LDAP URL beginning in the form `ldap[s]://<hostname>:<port>` | `nil` |
| `ldap.base` | LDAP search base DN | `nil` |
| `ldap.binddn` | LDAP bind DN | `nil` |
| `ldap.bindpw` | LDAP bind password | `nil` |
| `ldap.uidField` | LDAP field used for uid | `uid` |
| `ldap.tls.enabled` | Enable LDAP over TLS (LDAPS) | `False` |
| `ldap.tls.allowSelfSigned` | Allow self signed certicates for LDAPS | `True` |
| `ldap.tls.CAcertificateSecret` | Name of the secret that contains the LDAPS CA cert file | `uid` |
| `ldap.tls.CAcertificateFilename` | LDAPS CA cert filename | `uid` |
| `securityContext.enabled` | Enable security context | `true` |
| `securityContext.fsGroup` | Group ID for the container | `1001` |
| `securityContext.runAsUser` | User ID for the container | `1001` |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Airflow Web port | `8080` |
| `service.nodePort` | Kubernetes Service nodePort | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Airflow Service | `nil` |
| `service.annotations` | Service annotations | `` |
| `ingress.enabled` | Enable ingress controller resource | `false` |
| `ingress.certManager` | Add annotations for cert-manager | `false` |
| `ingress.annotations` | Ingress annotations | `[]` |
| `ingress.hosts[0].name` | Hostname to your Airflow installation | `airflow.local` |
| `ingress.hosts[0].path` | Path within the url structure | `/` |
| `ingress.hosts[0].tls` | Utilize TLS backend in ingress | `false` |
| `ingress.hosts[0].tlsHosts` | Array of TLS hosts for ingress record (defaults to `ingress.hosts[0].name` if `nil`) | `nil` |
| `ingress.hosts[0].tlsSecret` | TLS Secret (certificates) | `airflow.local-tls` |
| `ingress.secrets[0].name` | TLS Secret Name | `nil` |
| `ingress.secrets[0].certificate` | TLS Secret Certificate | `nil` |
| `ingress.secrets[0].key` | TLS Secret Key | `nil` |
| `podAffinityPreset` | Pod affinity preset. Ignored if `affinity` is set. Allowed values: `soft` or `hard` | `""` |
| `podAntiAffinityPreset` | Pod anti-affinity preset. Ignored if `affinity` is set. Allowed values: `soft` or `hard` | `soft` |
| `nodeAffinityPreset.type` | Node affinity preset type. Ignored if `affinity` is set. Allowed values: `soft` or `hard` | `""` |
| `nodeAffinityPreset.key` | Node label key to match Ignored if `affinity` is set. | `""` |
| `nodeAffinityPreset.values` | Node label values to match. Ignored if `affinity` is set. | `[]` |
| `affinity` | Affinity for pod assignment | `{}` (evaluated as a template) |
| `nodeSelector` | Node labels for pod assignment | `{}` (evaluated as a template) |
| `tolerations` | Tolerations for pod assignment | `[]` (evaluated as a template) |
| `livenessProbe.enabled` | would you like a livessProbed to be enabled | `true` |
| `livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 180 |
| `livenessProbe.periodSeconds` | How often to perform the probe | 20 |
| `livenessProbe.timeoutSeconds` | When the probe times out | 5 |
| `livenessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `livenessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` |
| `readinessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 30 |
| `readinessProbe.periodSeconds` | How often to perform the probe | 10 |
| `readinessProbe.timeoutSeconds` | When the probe times out | 5 |
| `readinessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `readinessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `postgresql.enabled` | Switch to enable or disable the PostgreSQL helm chart | `true` |
| `postgresql.postgresqlUsername` | Airflow Postgresql username | `bn_airflow` |
| `postgresql.postgresqlPassword` | Airflow Postgresql password | `nil` |
| `postgresql.postgresqlDatabase` | Airflow Postgresql database | `bitnami_airflow` |
| `postgresql.existingSecret` | Name of an existing secret containing the PostgreSQL password ('postgresql-password' key) . This secret is used in case of postgresql.enabled=true and we would like to specify password for newly created postgresql instance | `nil` |
| `externalDatabase.host` | External PostgreSQL host | `nil` |
| `externalDatabase.user` | External PostgreSQL user | `nil` |
| `externalDatabase.password` | External PostgreSQL password | `nil` |
| `externalDatabase.database` | External PostgreSQL database name | `nil` |
| `externalDatabase.port` | External PostgreSQL port | `nil` |
| `externalDatabase.existingSecret` | Name of an existing secret containing the PostgreSQL password ('postgresql-password' key) | `nil` |
| `redis.enabled` | Switch to enable or disable the Redis helm chart | `true` |
| `redis.existingSecret` | Name of an existing secret containing the Redis password ('redis-password' key) . This secret is used in case of redis.enabled=true and we would like to specify password for newly created redis instance | `nil` |
| `externalRedis.host` | External Redis host | `nil` |
| `externalRedis.port` | External Redis port | `nil` |
| `externalRedis.password` | External Redis password | `nil` |
| `externalRedis.username` | External Redis username (not required on most Redis implementations) | `nil` |
| `externalRedis.existingSecret` | Name of an existing secret containing the Redis password ('redis-password' key) | `nil` |
| `metrics.enabled` | Start a side-car prometheus exporter | `false` |
| `metrics.image.registry` | Airflow exporter image registry | `docker.io` |
| `metrics.image.repository` | Airflow exporter image name | `bitnami/airflow-exporter` |
| `metrics.image.tag` | Airflow exporter image tag | `{TAG_NAME}` |
| `metrics.image.pullPolicy` | Image pull policy | `IfNotPresent` |
| `metrics.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `metrics.podAnnotations` | Additional annotations for Metrics exporter | `{prometheus.io/scrape: "true", prometheus.io/port: "9112"}` |
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `global.imageRegistry` | Global Docker image registry | `nil` |
| `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `global.storageClass` | Global storage class for dynamic provisioning | `nil` |
### Common parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `affinity` | Affinity for pod assignment (evaluated as a template) | `{}` |
| `commonAnnotations` | Annotations to add to all deployed objects | `{}` |
| `commonLabels` | Labels to add to all deployed objects | `{}` |
| `containerSecurityContext.enabled` | Enable container security context | `true` |
| `containerSecurityContext.runAsUser` | User ID for the container | `1001` |
| `extraDeploy` | A list of extra kubernetes resources to be deployed | `[]` |
| `extraEnvVars` | Extra environment variables to add to web, worker and scheduler pods | `nil` |
| `extraEnvVarsCM` | ConfigMap containing extra env vars to add to web, worker and scheduler pods | `nil` |
| `extraEnvVarsSecret` | Secret containing extra env vars to add to web, worker and scheduler pods | `nil` |
| `fullnameOverride` | String to fully override ariflow.fullname template with a string | `nil` |
| `initContainers` | List of init containers to be added to the web, worker and scheduler pods | `nil` |
| `nameOverride` | String to partially override airflow.fullname template with a string (will prepend the release name) | `nil` |
| `networkPolicies.enabled` | Switch to enable network policies | `false` |
| `nodeAffinityPreset.key` | Node label key to match Ignored if `affinity` is set. | `""` |
| `nodeAffinityPreset.type` | Node affinity preset type. Ignored if `affinity` is set. Allowed values: `soft` or `hard` | `""` |
| `nodeAffinityPreset.values` | Node label values to match. Ignored if `affinity` is set. | `[]` |
| `nodeSelector` | Node labels for pod assignment | `{}` (evaluated as a template) |
| `podAffinityPreset` | Pod affinity preset. Ignored if `affinity` is set. Allowed values: `soft` or `hard` | `""` |
| `podAntiAffinityPreset` | Pod anti-affinity preset. Ignored if `affinity` is set. Allowed values: `soft` or `hard` | `soft` |
| `podSecurityContext.enabled` | Enable pod security context | `true` |
| `podSecurityContext.fsGroup` | fsGroup ID for the pod | `1001` |
| `rbac.create` | If true, create & use RBAC resources | `false` |
| `serviceAccount.annotations` | Annotations for service account | `{}` |
| `serviceAccount.create` | If true, create a service account | `false` |
| `serviceAccount.name` | The name of the service account to use. If not set and create is true, a name is generated using the fullname template. | `` |
| `sidecars` | List of sidecar containers to be adde to web, worker and scheduler pods | `nil` |
| `tolerations` | Tolerations for pod assignment | `[]` (evaluated as a template) |
### Airflow common parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `auth.existingSecret` | Name of an existing secret containing password and fernet key ('airflow-password and 'airflow-fernetKey' keys) | `nil` |
| `auth.fernetKey` | Fernet key to secure connections | `nil` |
| `auth.forcePassword` | Force users to specify a password | `false` |
| `auth.password` | Password to access web UI | `nil` |
| `auth.username` | Username to access web UI | `user` |
| `configurationConfigMap` | Name of an existing config map containing the Airflow config file | `nil` |
| `dagsConfigMap` | Name of an existing config map containing all the DAGs files you want to load in Airflow. | `nil` |
| `executor` | Airflow executor, it should be one of `SequentialExecutor`, `Local Executor`, `CeleryExecutor`, `KubernetesExecutor` | `"CeleryExecutor"` |
| `loadExamples` | Switch to load some Airflow examples | `false` |
## Airflow web parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `web.args` | Override default container args (useful when using custom images) | `nil` |
| `web.baseUrl` | URL used to access to airflow web ui | `nil` |
| `web.command` | Override default container command (useful when using custom images) | `nil` |
| `web.configMap` | Config map name for ~/airflow/webserver_config.py | `nil` |
| `web.containerPort` | Container port to be used for exposing http server. | `8080` |
| `web.customLivenessProbe` | Custom liveness probe for the web component | `{}` |
| `web.customReadinessProbe` | Custom rediness probe for the web component | `{}` |
| `web.extraEnvVars` | Array containing extra env vars | `nil` |
| `web.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `web.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `web.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `web.extraVolumes` | Array of extra volumes to be added (evaluated as template). | `nil` |
| `web.image.debug` | Specify if debug values should be set | `false` |
| `web.image.pullPolicy` | Airflow web image pull policy | `IfNotPresent` |
| `web.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `web.image.registry` | Airflow web image registry | `docker.io` |
| `web.image.repository` | Airflow web image name | `bitnami/airflow` |
| `web.image.tag` | Airflow web image tag | `{TAG_NAME}` |
| `web.initContainers` | List of init containers to be added to the web's pods | `nil` |
| `web.livenessProbe.enabled` | Switch to enable livess probe | `true` |
| `web.livenessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `web.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 180 |
| `web.livenessProbe.periodSeconds` | How often to perform the probe | 20 |
| `web.livenessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `web.livenessProbe.timeoutSeconds` | When the probe times out | 5 |
| `web.podAnnotations` | Annotations to add to the web's pods | `nil` |
| `web.podDisruptionBudget.enabled` | Switch to enable Pod Disruption Budget for Airflow web component | `false` |
| `web.podDisruptionBudget.minAvailable` | Set the minimum amount of pods available | `1` |
| `web.podLabels` | Labels to add to the web's pods | `nil` |
| `web.priorityClassName` | Priority class name for the web's pods | `""` |
| `web.readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` |
| `web.readinessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `web.readinessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 30 |
| `web.readinessProbe.periodSeconds` | How often to perform the probe | 10 |
| `web.readinessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `web.readinessProbe.timeoutSeconds` | When the probe times out | 5 |
| `web.replicaCount` | Number of Airflow web replicas | `2` |
| `web.resources.limits` | The resources limits for the web containers | `{}` |
| `web.resources.requests` | The requested resources for the web containers | `{}` |
| `web.sidecars` | List of sidecar containers to be added to the web's pods | `nil` |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Airflow Web port | `8080` |
| `service.nodePort` | Kubernetes Service nodePort | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Airflow Service | `nil` |
| `service.annotations` | Service annotations | `` |
### Airflow scheduler parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `scheduler.args` | Override default container args (useful when using custom images) | `nil` |
| `scheduler.command` | Override default container command (useful when using custom images) | `nil` |
| `scheduler.customLivenessProbe` | Custom liveness probe for the Airflow scheduler component | `{}` |
| `scheduler.customReadinessProbe` | Custom rediness probe for the Airflow scheduler component | `{}` |
| `scheduler.extraEnvVars` | Array containing extra env vars | `nil` |
| `scheduler.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `scheduler.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `scheduler.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `scheduler.extraVolumes` | Array of extra volumes to be added (evaluated as template). | `nil` |
| `scheduler.image.debug` | Specify if debug values should be set | `false` |
| `scheduler.image.pullPolicy` | Airflow Scheduler image pull policy | `IfNotPresent` |
| `scheduler.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `scheduler.image.registry` | Airflow Scheduler image registry | `docker.io` |
| `scheduler.image.repository` | Airflow Scheduler image name | `bitnami/airflow-scheduler` |
| `scheduler.image.tag` | Airflow Scheduler image tag | `{TAG_NAME}` |
| `scheduler.initContainers` | List of init containers to be added to the scheduler's pods | `nil` |
| `scheduler.podAnnotations` | Annotations to add to the scheduler's pods | `nil` |
| `scheduler.podDisruptionBudget.enabled` | Switch to enable Pod Disruption Budget for Airflow scheduler component | `false` |
| `scheduler.podDisruptionBudget.minAvailable` | Set the minimum amount of pods available | `1` |
| `scheduler.podLabels` | Labels to add to the scheduler's pods | `nil` |
| `scheduler.priorityClassName` | Priority class name for the scheduler's pods | `""` |
| `scheduler.replicaCount` | Number of Airflow scheduler replicas | `2` |
| `scheduler.resources.limits` | The resources limits for the scheduler containers | `{}` |
| `scheduler.resources.requests` | The requested resources for the scheduler containers | `{}` |
| `scheduler.sidecars` | List of sidecar containers to be added to the scheduler's pods | `nil` |
### Airflow worker parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `worker.args` | Override default container args (useful when using custom images) | `nil` |
| `worker.autoscaling.enabled` | Switch to enable Horizontal Pod Autoscaler for Airflow worker component (only when executor is `CeleryExecutor`). When enable you should also set `resources.requests` | `false` |
| `worker.autoscaling.replicas.max` | Maximun amount of replicas | `3` |
| `worker.autoscaling.replicas.min` | Minimum amount of replicas | `1` |
| `worker.autoscaling.targets.cpu` | Target cpu that will trigger an scaling action (unit: %) | `80` |
| `worker.autoscaling.targets.memory` | Target memory that will trigger an scaling action (unit: %) | `80` |
| `worker.command` | Override default container command (useful when using custom images) | `nil` |
| `worker.customLivenessProbe` | Custom liveness probe for the Airflow worker component | `{}` |
| `worker.customReadinessProbe` | Custom rediness probe for the Airflow worker component | `{}` |
| `worker.extraEnvVars` | Array containing extra env vars | `nil` |
| `worker.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `worker.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `worker.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `worker.extraVolumes` | Array of extra volumes to be added (evaluated as template). | `nil` |
| `worker.image.debug` | Specify if debug values should be set | `false` |
| `worker.image.pullPolicy` | Airflow worker image pull policy | `IfNotPresent` |
| `worker.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `worker.image.registry` | Airflow worker image registry | `docker.io` |
| `worker.image.repository` | Airflow worker image name | `bitnami/airflow-worker` |
| `worker.image.tag` | Airflow worker image tag | `{TAG_NAME}` |
| `worker.initContainers` | List of init containers to be added to the worker's pods | `nil` |
| `worker.livenessProbe.enabled` | Switch to enable livess probe | `true` |
| `worker.livenessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `worker.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 180 |
| `worker.livenessProbe.periodSeconds` | How often to perform the probe | 20 |
| `worker.livenessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `worker.livenessProbe.timeoutSeconds` | When the probe times out | 5 |
| `worker.podAnnotations` | Annotations to add to the worker's pods | `nil` |
| `worker.podDisruptionBudget.enabled` | Switch to enable Pod Disruption Budget for Airflow worker component | `false` |
| `worker.podDisruptionBudget.minAvailable` | Set the minimum amount of pods available | `1` |
| `worker.podLabels` | Labels to add to the worker's pods | `nil` |
| `worker.podManagementPolicy` | Pod management policy to manage scaling operation of worker pods | `nil` |
| `worker.port` | Port to bind Arflow workers | `8793` |
| `worker.priorityClassName` | Priority class name for the worker's pods | `""` |
| `worker.readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` |
| `worker.readinessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `worker.readinessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 30 |
| `worker.readinessProbe.periodSeconds` | How often to perform the probe | 10 |
| `worker.readinessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `worker.readinessProbe.timeoutSeconds` | When the probe times out | 5 |
| `worker.replicaCount` | Number of Airflow worker replicas | `2` |
| `worker.resources.limits` | The resources limits for the worker containers | `{}` |
| `worker.resources.requests` | The requested resources for the worker containers | `{}` |
| `worker.rollingUpdatePartition` | Partition update strategy | `nil` |
| `worker.sidecars` | List of sidecar containers to be added to the worker's pods | `nil` |
| `worker.updateStrategy` | pdate strategy for the statefulset | `"RollingUpdate"` |
### Airflow database parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `externalDatabase.database` | External PostgreSQL database name | `nil` |
| `externalDatabase.existingSecret` | Name of an existing secret containing the PostgreSQL password ('postgresql-password' key) | `nil` |
| `externalDatabase.host` | External PostgreSQL host | `nil` |
| `externalDatabase.password` | External PostgreSQL password | `nil` |
| `externalDatabase.port` | External PostgreSQL port | `nil` |
| `externalDatabase.user` | External PostgreSQL user | `nil` |
| `externalRedis.existingSecret` | Name of an existing secret containing the Redis password ('redis-password' key) | `nil` |
| `externalRedis.host` | External Redis host | `nil` |
| `externalRedis.password` | External Redis password | `nil` |
| `externalRedis.port` | External Redis port | `nil` |
| `externalRedis.username` | External Redis username (not required on most Redis implementations) | `nil` |
| `postgresql.enabled` | Switch to enable or disable the PostgreSQL helm chart | `true` |
| `postgresql.existingSecret` | Name of an existing secret containing the PostgreSQL password ('postgresql-password' key) . This secret is used in case of postgresql.enabled=true and we would like to specify password for newly created postgresql instance | `nil` |
| `postgresql.postgresqlDatabase` | Airflow Postgresql database | `bitnami_airflow` |
| `postgresql.postgresqlPassword` | Airflow Postgresql password | `nil` |
| `postgresql.postgresqlUsername` | Airflow Postgresql username | `bn_airflow` |
| `redis.cluster.enabled` | Switch to enable a clustered redis | `false` |
| `redis.enabled` | Switch to enable or disable the Redis helm chart | `true` |
| `redis.existingSecret` | Name of an existing secret containing the Redis password ('redis-password' key) . This secret is used in case of redis.enabled=true and we would like to specify password for newly created redis instance | `nil` |
### Airflow exposing parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `ingress.annotations` | Ingress annotations | `[]` |
| `ingress.certManager` | Add annotations for cert-manager | `false` |
| `ingress.enabled` | Enable ingress controller resource | `false` |
| `ingress.hosts[0].name` | Hostname to your Airflow installation | `airflow.local` |
| `ingress.hosts[0].path` | Path within the url structure | `/` |
| `ingress.hosts[0].tls` | Utilize TLS backend in ingress | `false` |
| `ingress.hosts[0].tlsHosts` | Array of TLS hosts for ingress record (defaults to `ingress.hosts[0].name` if `nil`) | `nil` |
| `ingress.hosts[0].tlsSecret` | TLS Secret (certificates) | `airflow.local-tls` |
| `ingress.secrets[0].name` | TLS Secret Name | `nil` |
| `ingress.secrets[0].certificate` | TLS Secret Certificate | `nil` |
| `ingress.secrets[0].key` | TLS Secret Key | `nil` |
### Airflow metrics parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `metrics.enabled` | Start a side-car prometheus exporter | `false` |
| `metrics.image.pullPolicy` | Image pull policy | `IfNotPresent` |
| `metrics.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `metrics.image.registry` | Airflow exporter image registry | `docker.io` |
| `metrics.image.repository` | Airflow exporter image name | `bitnami/airflow-exporter` |
| `metrics.image.tag` | Airflow exporter image tag | `{TAG_NAME}` |
| `metrics.podAnnotations` | Annotations to add to the metrics's pods | `nil` |
| `metrics.podLabels` | Labels to add to the worker's pods | `{}` |
| `metrics.resources` | The resources for the metrics containers | `{}` |
| `metrics.tolerations` | The tolerations for the metrics pod | `[]` |
### Airflow ldap parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `ldap.base` | LDAP search base DN | `nil` |
| `ldap.binddn` | LDAP bind DN | `nil` |
| `ldap.bindpw` | LDAP bind password | `nil` |
| `ldap.enabled` | Enable LDAP support | `false` |
| `ldap.tls.enabled` | Enable LDAP over TLS (LDAPS) | `False` |
| `ldap.tls.allowSelfSigned` | Allow self signed certicates for LDAPS | `True` |
| `ldap.tls.CAcertificateSecret` | Name of the secret that contains the LDAPS CA cert file | `uid` |
| `ldap.tls.CAcertificateFilename` | LDAPS CA cert filename | `uid` |
| `ldap.uidField` | LDAP field used for uid | `uid` |
| `ldap.uri` | LDAP URL beginning in the form `ldap[s]://<hostname>:<port>` | `nil` |
### Airflow git sync parameters
| Parameter | Description | Default |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| `git.clone.args` | Override default container args (useful when using custom images) | `nil` |
| `git.clone.command` | Override default container command (useful when using custom images) | `nil` |
| `git.clone.extraEnvVars` | Array containing extra env vars | `nil` |
| `git.clone.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `git.clone.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `git.clone.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `git.dags.enabled` | Enable in order to download DAG files from git repository. | `false` |
| `git.dags.repositories[0].branch` | Branch from repository to checkout | `nil` |
| `git.dags.repositories[0].name` | An unique identifier for repository, must be unique for each repository, by default: `[0].repository` in kebacase | `nil` |
| `git.dags.repositories[0].path` | Path to a folder in the repository containing the dags. | `nil` |
| `git.dags.repositories[0].repository` | Repository where download plugins from | `nil` |
| `git.image.pullPolicy` | Git image pull policy | `"IfNotPresent"` |
| `git.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` |
| `git.image.registry` | Git image registry | `"docker.io"` |
| `git.image.repository` | Git image name | `"bitnami/git"` |
| `git.image.tag` | Git image tag | `"2.29.0-debian-10-r0"` |
| `git.plugins.enabled` | Enable in order to download plugins from git repository. | `false` |
| `git.plugins.repositories[0].branch` | Branch from repository to checkout | `nil` |
| `git.plugins.repositories[0].name` | An unique identifier for repository, must be unique for each repository, by default: `[0].repository` in kebacase | `nil` |
| `git.plugins.repositories[0].path` | Path to a folder in the repository containing the plugins. | `nil` |
| `git.plugins.repositories[0].repository` | Repository where download plugins from | `nil` |
| `git.sync.args` | Override default container args (useful when using custom images) | `nil` |
| `git.sync.command` | Override default container command (useful when using custom images) | `nil` |
| `git.sync.extraEnvVars` | Array containing extra env vars | `nil` |
| `git.sync.extraEnvVarsCM` | ConfigMap containing extra env vars | `nil` |
| `git.sync.extraEnvVarsSecret` | Secret containing extra env vars (in case of sensitive data) | `nil` |
| `git.sync.extraVolumeMounts` | Array of extra volume mounts to be added (evaluated as template). Normally used with `extraVolumes`. | `nil` |
| `git.sync.interval` | Interval (in seconds) to pull the git repository containing the plugins and/or DAG files | `60` |
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
```console
$ helm install my-release \
--set airflow.auth.username=my-user \
--set airflow.auth.password=my-passsword \
--set airflow.auth.fernetKey=my-fernet-key \
--set auth.username=my-user \
--set auth.password=my-passsword \
--set auth.fernetKey=my-fernet-key \
bitnami/airflow
```
@@ -265,11 +371,17 @@ This chart includes a `values-production.yaml` file where you can find some para
+ airflow.baseUrl: http://airflow.local
```
- Number of Airflow Worker replicas:
- Number of Airflow Worker replicas and enable autoscaling:
```diff
- airflow.worker.replicas: 1
+ airflow.worker.replicas: 3
- worker.replicaCount: 1
+ worker.replicaCount: 3
- worker.autoscaling.enabled=false
+ worker.autoscaling.enabled=true
- worker.autoscaling.replicas.max=3
+ worker.autoscaling.replicas.max=6
- worker.autoscaling.replicas.min=1
+ worker.autoscaling.replicas.min=3
```
- Force users to specify a password:
@@ -294,25 +406,23 @@ Learn how to generate one [here](https://bcb.github.io/airflow/fernet-key)
### Load DAG files
There are three different ways to load your custom DAG files into the Airflow chart. All of them are compatible so you can use more than one at the same time.
There are two different ways to load your custom DAG files into the Airflow chart. All of them are compatible so you can use more than one at the same time.
#### Option 1: Load locally from the `files` folder
#### Option 1: Specify an existing config map
If you plan to deploy the chart from your filesystem, you can copy your DAG files inside the `files/dags` directory. A config map will be created with those files and it will be mounted in all airflow nodes.
You can manually create a config map containing all your DAG files and then pass the name when deploying Airflow chart. For that, you can pass the option `dagsConfigMap`.
#### Option 2: Specify an existing config map
You can manually create a config map containing all your DAG files and then pass the name when deploying Airflow chart. For that, you can pass the option `airflow.dagsConfigMap`.
#### Option 3: Get your DAG files from a git repository
#### Option 2: Get your DAG files from a git repository
You can store all your DAG files on GitHub repositories and then clone to the Airflow pods with an initContainer. The repositories will be periodically updated using a sidecar container. In order to do that, you can deploy airflow with the following options:
> NOTE: When enabling git synchronization, an init container and sidecar container will be added for all the pods running airflow, this will allow scheduler, worker and web component to reach dags if it was needed.
```console
airflow.cloneDagFilesFromGit.enabled=true
airflow.cloneDagFilesFromGit.repositories[0].repository=https://github.com/USERNAME/REPOSITORY
airflow.cloneDagFilesFromGit.repositories[0].name=REPO-IDENTIFIER
airflow.cloneDagFilesFromGit.repositories[0].branch=master
git.dags.enabled=true
git.dags.repositories[0].repository=https://github.com/USERNAME/REPOSITORY
git.dags.repositories[0].name=REPO-IDENTIFIER
git.dags.repositories[0].branch=master
```
If you use a private repository from GitHub, a possible option to clone the files is using a [Personal Access Token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and using it as part of the URL: https://USERNAME:PERSONAL_ACCESS_TOKEN@github.com/USERNAME/REPOSITORY
@@ -321,11 +431,13 @@ If you use a private repository from GitHub, a possible option to clone the file
You can load plugins into the chart by specifying a git repository containing the plugin files. The repository will be periodically updated using a sidecar container. In order to do that, you can deploy airflow with the following options:
> NOTE: When enabling git synchronization, an init container and sidecar container will be added for all the pods running airflow, this will allow scheduler, worker and web component to reach plugins if it was needed.
```console
airflow.clonePluginsFromGit.enabled=true
airflow.clonePluginsFromGit.repository=https://github.com/teamclairvoyant/airflow-rest-api-plugin.git
airflow.clonePluginsFromGit.branch=v1.0.9-branch
airflow.clonePluginsFromGit.path=plugins
git.plugins.enabled=true
git.plugins.repositories[0].repository=https://github.com/teamclairvoyant/airflow-rest-api-plugin.git
git.plugins.repositories[0].branch=v1.0.9-branch
git.plugins.repositories[0].path=plugins
```
### Existing Secrets
@@ -343,7 +455,7 @@ redis.enabled=false
externalRedis.host=my.external.redis.host
externalRedis.existingSecret=all-my-secrets
airflow.auth.existingSecret=all-my-secrets
auth.existingSecret=all-my-secrets
```
The expected secret resource looks as follows:
@@ -373,6 +485,60 @@ As an alternative, you can use of the preset configurations for pod affinity, po
This chart allows you to mount volumes using `extraVolumes` and `extraVolumeMounts` in all 3 airflow components (web, scheduler, worker). Mounting a requirements.txt using these options to `/bitnami/python/requirements.txt` will execute `pip install -r /bitnami/python/requirements.txt` on container start. [Reference](https://github.com/bitnami/bitnami-docker-airflow/blob/cafc8eab1efddb5efda5a00cc861ef10f35f1d49/1/debian-10/rootfs/run.sh#L14)
### Enabling network policies
This chart allows you to set network policies that will rectrict the access to the deployed pods in the cluster. Basically, no other pods apart from Scheduler's pods may access Worker's pods and no other pods apart from Web's pods may access Worker's ones. To do so, set `networkPolicies.enabled=true`.
### Executors
Airflow supports different executors runtimes and this chart provides support for the following ones.
#### CeleryExecutor
Celery executor is the default value for this chart with it you can scale out the number of workers. To point the `executor` parameter to `CeleryExecutor` you need to do something, you just install the chart with default parameters.
#### KubernetesExecutor
The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance using the `pod_template.yaml` that you can find [templates/config/configmap.yaml](), otherwise you can override this template using `worker.podTemplate`. To enable `KubernetesExecutor` set the following parameters.
```console
executor=KubernetesExecutor
redis.enabled=false
rbac.create=true
serviceaccount.create=true
```
#### LocalExecutor
Local executor runs tasks by spawning processes in the Scheduler pods. To enable `LocalExecutor` set the following parameters.
```console
executor=LocalExecutor
redis.enabled=false
```
#### SequentialExecutor
This executor will only run one task instance at a time in the Scheduler pods. For production use case, please use other executors. To enable `SequentialExecutor` set the following parameters.
```console
executor=SequentialExecutor
redis.enabled=false
```
### Scaling worker pods
Sometime when using large workloads a fixed number of worker pods may make task to take a long time to be executed. This chart provide two ways for scaling worker pods.
- If you are using `KubernetesExecutor` auto scaling pods would be done by the Scheduler without adding anything more.
- If you are using `SequentialExecutor` you would have to enable `worker.autoscaling` to do so, please, set the following parameters. It will use autoscaling by defualt configuration that you can change using `worker.autoscaling.replicas.*` and `worker.autoscaling.targets.*`.
```console
worker.autoscaling.enabled=true
worker.resources.requests.cpu=200m
worker.resources.requests.memory=250Mi
```
## Persistence
The Bitnami Airflow chart relies on the PostgreSQL chart persistence. This means that Airflow does not persist anything.
@@ -383,6 +549,99 @@ Find more information about how to deal with common errors related to Bitnami’
## Notable changes
### 7.0.0
[On November 13, 2020, Helm v2 support was formally finished](https://github.com/helm/charts#status-of-the-project), this major version is the result of the required changes applied to the Helm Chart to be able to incorporate the different features added in Helm v3 and to be consistent with the Helm project itself regarding the Helm v2 EOL.
#### What changes were introduced in this major version?
- Previous versions of this Helm Chart use `apiVersion: v1` (installable by both Helm 2 and 3), this Helm Chart was updated to `apiVersion: v2` (installable by Helm 3 only). [Here](https://helm.sh/docs/topics/charts/#the-apiversion-field) you can find more information about the `apiVersion` field.
- Move dependency information from the *requirements.yaml* to the *Chart.yaml*.
- After running `helm dependency update`, a *Chart.lock* file is generated containing the same structure used in the previous *requirements.lock*.
- The different fields present in the *Chart.yaml* file has been ordered alphabetically in a homogeneous way for all the Bitnami Helm Charts.
- Several parameters were renamed or dissapeared in favor of new ones on this major version:
- The image objects have been moved to its corresponding component object, e.g: `workerImage.*` now is located at `worker.image.*`.
- The prefix *airflow* has been removed. Therefore, parameters prefixed with `airflow` are now at root level, e.g. `airflow.loadExamples` now is `loadExamples` or `airflow.worker.resources` now is `worker.resources`.
- Parameters related to the *git* features has completely been refactored, please see how to configure git for [dags](#loaddagfiles) and [plugins](#loadingplugins) sections for more details.
- They have been moved to `git.*` prefix.
- `airflow.cloneDagsFromGit.*` no longer exists, instead you must use `git.dags.*` and `git.dags.repositories[*]` has been introduced that will add support for multiple repositories.
- `airflow.clonePluginsFromGit.*` no longer exists, instead you must use `git.plugins.*`. `airflow.clonePluginsFromGit.repository`, `airflow.clonePluginsFromGit.branch` and `airflow.clonePluginsFromGit.path` have been removed in favour of `git.dags.repositories[*].*`.
- Liveness and rediness probe have been separated by components `airflow.livenessProbe.*` and `airflow.redinessProbe` have been removed in favour of `web.livenessProbe`, `worker.livenessProbe`, `web.redinessProbe` and `worker.redinessProbe`.
- `airflow.baseUrl` has been moved to `web.baseUrl`.
- Security context has been migrated to the bitnami standard way so that `securityContext.*` has been divided into `podSecurityContext.*` that will define the `fsGroup` for all the containers in the pod and `containerSecurityContext.*` that will define the user id that will run the main containers.
- Both `bitnami/postgresql` and `bitnami/redis` have been upgraded to their latest major versions, `9.x.x` and `11.x.x` respectively, find more info in their READMEs [`bitnami/postgresql`](https://github.com/bitnami/charts/tree/master/bitnami/postgresql#900) and [`bitnami/redis`](https://github.com/bitnami/charts/tree/master/bitnami/redis#to-1100)
- `./files/dags/*.py` will not be include in the deployment any more.
- Some new features:
- The folder structure has changed, we separated yaml manifest into folders by components.
- Network policies has been added see more in the [Enabling network policies](#enablingnetworkpolicies) section.
- Executors, this new version add support for the different executors that airflow implements, see more in the [Executors](#executors) section.
- Worker scaling functionality has been added, see more in the [Scaling worker pods](#scalingworkerpods).
- Pod disruption budget has been added.
#### Considerations when upgrading to this version
- If you want to upgrade to this version using Helm v2, this scenario is not supported as this version doesn't support Helm v2 anymore
- If you installed the previous version with Helm v2 and wants to upgrade to this version with Helm v3, please refer to the [official Helm documentation](https://helm.sh/docs/topics/v2_v3_migration/#migration-use-cases) about migrating from Helm v2 to v3
- If you want to upgrade to this version from a previous one installed with Helm v3, you can try to follow the following steps:
> NOTE: Please, create a backup of your database before running any of those actions.
Having an already existing chart release called `airflow` and deployed like
```console
$ helm install airflow bitnami/airflow \
--set airflow.loadExamples=true \
--set airflow.baseUrl=http://127.0.0.1:8080
```
##### Export secrets and required values to update
```console
$ export AIRFLOW_PASSWORD=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-password}" | base64 --decode)
$ export AIRFLOW_FERNETKEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-fernetKey}" | base64 --decode)
$ export POSTGRESQL_PASSWORD=$(kubectl get secret --namespace default airflow-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$ export REDIS_PASSWORD=$(kubectl get secret --namespace default airflow-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
```
##### Delete statefulsets
Once the new version change fields in the statefulset that will make the upgrade action to fail you will need to remove them
> NOTE: Don't forget to set `--cascade=false`
```
$ kubectl delete statefulsets.apps --cascade=false airflow-postgresql
$ kubectl delete statefulsets.apps --cascade=false airflow-worker
```
##### Upgrade the chart release
> NOTE: Please remember to migrate all the values to its new path following the above notes, e.g: `airflow.loadExamples` -> `loadExamples` or `airflow.baseUrl=http://127.0.0.1:8080` -> `web.baseUrl=http://127.0.0.1:8080`.
```console
$ helm upgrade airflow bitnami/airflow \
--set loadExamples=true \
--set web.baseUrl=http://127.0.0.1:8080 \
--set auth.password=$AIRFLOW_PASSWORD \
--set auth.fernetKey=$AIRFLOW_FERNETKEY \
--set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD \
--set redis.password=$REDIS_PASSWORD \
--set redis.cluster.enabled=true
```
##### Force new statefulset to create a new pod for postgresql
```console
$ kubectl delete pod airflow-postgresql-0
```
#### Useful links
- https://docs.bitnami.com/tutorials/resolve-helm2-helm3-post-migration-issues/
- https://helm.sh/docs/topics/v2_v3_migration/
- https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/
### 6.5.0
This version also introduces `bitnami/common`, a [library chart](https://helm.sh/docs/topics/library_charts/#helm) as a dependency. More documentation about this new utility could be found [here](https://github.com/bitnami/charts/tree/master/bitnami/common#bitnami-common-library-chart). Please, make sure that you have updated the chart dependencies before executing any upgrade.
bitnami/airflow/requirements.lock
-12
View File
@@ -1,12 +0,0 @@
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
version: 0.10.0
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 8.10.14
- name: redis
repository: https://charts.bitnami.com/bitnami
version: 10.9.0
digest: sha256:34d539b4a96a21c9045a5167cf308b02356113fa5088754de74270e2c1b019ff
generated: "2020-11-19T23:19:38.772630879Z"
bitnami/airflow/templates/NOTES.txt
+57 -29
View File
@@ -1,4 +1,7 @@
{{- if not .Values.airflow.baseUrl -}}
{{- $airflowSecretName := include "airflow.secretName" . -}}
{{- $postgresqlSecretName := include "airflow.postgresql.secretName" . -}}
{{- $redisSecretName := include "airflow.redis.secretName" . -}}
{{- if not .Values.web.baseUrl -}}
###############################################################################
### ERROR: You did not provide an external URL in your 'helm install' call ###
###############################################################################
@@ -10,41 +13,47 @@ host. To configure Airflow with the URL of your service:
1. Get the Airflow URL indicated on the Ingress Rule and associate it to your cluster external IP:
export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
export HOSTNAME=$(kubectl get ingress --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} -o jsonpath='{.spec.rules[0].host}')
echo "Airflow URL: http://$HOSTNAME/"
echo "$CLUSTER_IP $HOSTNAME" | sudo tee -a /etc/hosts
export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
export AIRFLOW_HOST=$(kubectl get ingress --namespace {{ .Release.Namespace }} {{ include "common.names.fullname" . }} -o jsonpath='{.spec.rules[0].host}')
echo "Airflow URL: http://$AIRFLOW_HOST/"
echo "$CLUSTER_IP $AIRFLOW_HOST" | sudo tee -a /etc/hosts
{{- else }}
1. Get the Airflow URL by running:
{{- if contains "NodePort" .Values.service.type }}
{{- if eq "NodePort" .Values.service.type }}
export APP_HOST=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
export APP_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "airflow.fullname" . }})
export AIRFLOW_HOST=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
export AIRFLOW_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "common.names.fullname" . }})
{{- else if contains "ClusterIP" .Values.service.type }}
{{- else if eq "ClusterIP" .Values.service.type }}
export APP_HOST=127.0.0.1
export APP_PORT=8080
export AIRFLOW_HOST=127.0.0.1
export AIRFLOW_PORT=8080
{{- else if contains "LoadBalancer" .Values.service.type }}
{{- else if eq "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
Watch the status with: 'kubectl get svc --namespace {{ .Release.Namespace }} -w {{ template "airflow.fullname" . }}'
Watch the status with: 'kubectl get svc --namespace {{ .Release.Namespace }} -w {{ include "common.names.fullname" . }}'
export APP_HOST=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} --template "{{ "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}" }}")
export APP_PORT=80
export AIRFLOW_HOST=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "common.names.fullname" . }} --template "{{ "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}" }}")
export AIRFLOW_PORT=80
{{- end }}
export APP_PASSWORD=$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.secretName" . }} -o jsonpath="{.data.airflow-password}" | base64 --decode)
export APP_DATABASE_PASSWORD=$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.postgresql.secretName" . }} -o jsonpath="{.data.postgresql-password}" | base64 --decode)
export APP_REDIS_PASSWORD=$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.redis.secretName" . }} -o jsonpath="{.data.redis-password}" | base64 --decode)
{{- end }}
{{ include "common.utils.secret.getvalue" (dict "secret" $airflowSecretName "field" "airflow-password" "context" $) }}
{{ include "common.utils.secret.getvalue" (dict "secret" $airflowSecretName "field" "airflow-fernetKey" "context" $) }}
{{ include "common.utils.secret.getvalue" (dict "secret" $postgresqlSecretName "field" "postgresql-password" "context" $) }}
{{ include "common.utils.secret.getvalue" (dict "secret" $redisSecretName "field" "redis-password" "context" $) }}
2. Complete your Airflow deployment by running:
helm upgrade {{ .Release.Name }} bitnami/{{ .Chart.Name }} \
--set service.type={{ .Values.service.type }},airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.password=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD{{- if .Values.global }}{{- if .Values.global.imagePullSecrets }},global.imagePullSecrets={{ .Values.global.imagePullSecrets }}{{- end }}{{- end }}
--set service.type={{ .Values.service.type }} \
--set web.baseUrl=http://$AIRFLOW_HOST:$AIRFLOW_PORT \
--set auth.password=$AIRFLOW_PASSWORD \
--set auth.fernetKey=$AIRFLOW_FERNETKEY \
--set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD \
--set redis.password=$REDIS_PASSWORD
{{- else -}}
@@ -52,30 +61,49 @@ host. To configure Airflow with the URL of your service:
{{- if .Values.ingress.enabled }}
echo URL : {{ .Values.airflow.baseUrl }}
echo URL : {{ .Values.web.baseUrl }}
{{- else if eq .Values.service.type "ClusterIP" }}
echo URL : http://127.0.0.1:{{ default "8080" .Values.service.port }}
kubectl port-forward --namespace {{ .Release.Namespace }} svc/{{ template "airflow.fullname" . }} {{ default "8080" .Values.service.port }}:8080
kubectl port-forward --namespace {{ .Release.Namespace }} svc/{{ include "common.names.fullname" . }} {{ default "8080" .Values.service.port }}:8080
{{- else if eq .Values.service.type "NodePort" }}
export APP_HOST=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
export APP_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "airflow.fullname" . }})
export AIRFLOW_HOST=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
export AIRFLOW_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "common.names.fullname" . }})
echo URL : http://$APP_HOST:$APP_PORT
echo URL : http://$AIRFLOW_HOST:$AIRFLOW_PORT
{{- else }}
echo URL : {{ .Values.airflow.baseUrl }}
echo URL : {{ .Values.web.baseUrl }}
{{- end }}
2. Get your Airflow login credentials by running:
echo User: {{ .Values.airflow.auth.username }}
echo Password: $(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.secretName" . }} -o jsonpath="{.data.airflow-password}" | base64 --decode)
{{ include "common.utils.secret.getvalue" (dict "secret" $airflowSecretName "field" "airflow-password" "context" $) }}
echo User: {{ .Values.auth.username }}
echo Password: $AIRFLOW_PASSWORD
{{- end }}
{{ include "airflow.validateValues" . }}
{{ include "airflow.checkRollingTags" . }}
{{- $passwordValidationErrors := list -}}
{{- if not .Values.auth.existingSecret -}}
{{- $requiredAirflowPassword := dict "valueKey" "auth.password" "secret" $airflowSecretName "field" "airflow-password" "context" $ -}}
{{- $requiredAirflowPasswordError := include "common.validations.values.single.empty" $requiredAirflowPassword -}}
{{- $passwordValidationErrors = append $passwordValidationErrors $requiredAirflowPasswordError -}}
{{- $requiredFernetKeyPassword := dict "valueKey" "auth.fernetKey" "secret" $airflowSecretName "field" "airflow-fernetKey" "context" $ -}}
{{- $requiredFernetKeyPasswordError := include "common.validations.values.single.empty" $requiredFernetKeyPassword -}}
{{- $passwordValidationErrors = append $passwordValidationErrors $requiredFernetKeyPasswordError -}}
{{- end -}}
{{- $postgresqlPasswordValidationErrors := include "common.validations.values.postgresql.passwords" (dict "secret" $postgresqlSecretName "subchart" true "context" $) -}}
{{- $passwordValidationErrors = append $passwordValidationErrors $postgresqlPasswordValidationErrors -}}
{{- $redisPasswordValidationErrors := include "common.validations.values.redis.passwords" (dict "secret" $redisSecretName "subchart" true "context" $) -}}
{{- $passwordValidationErrors = append $passwordValidationErrors $redisPasswordValidationErrors -}}
{{- include "common.errors.upgrade.passwords.empty" (dict "validationErrors" $passwordValidationErrors "context" $) -}}
bitnami/airflow/templates/_git_helpers.tpl
+174
View File
@@ -0,0 +1,174 @@
{{/* vim: set filetype=mustache: */}}
{{/*
Returns the name that will identify the repository internaly and it will be used to create folders or
volume names
*/}}
{{- define "airflow.git.repository.name" -}}
{{- $defaultName := regexFind "/.*$" .repository | replace "//" "" | replace "/" "-" | replace "." "-" -}}
{{- .name | default $defaultName | kebabcase -}}
{{- end -}}
{{/*
Returns the volume mounts that will be used by git containers (clone and sync)
*/}}
{{- define "airflow.git.volumeMounts" -}}
{{- if .Values.git.dags.enabled }}
{{- range .Values.git.dags.repositories }}
- name: git-cloned-dag-files-{{ include "airflow.git.repository.name" . }}
mountPath: /dags-{{ include "airflow.git.repository.name" . }}
{{- end }}
{{- end }}
{{- if .Values.git.plugins.enabled }}
{{- range .Values.git.plugins.repositories }}
- name: git-cloned-plugins-files-{{ include "airflow.git.repository.name" . }}
mountPath: /plugins-{{ include "airflow.git.repository.name" . }}
{{- end }}
{{- end }}
{{- end -}}
{{/*
Returns the volume mounts that will be used by the main container
*/}}
{{- define "airflow.git.maincontainer.volumeMounts" -}}
{{- if .Values.git.dags.enabled }}
{{- range .Values.git.dags.repositories }}
- name: git-cloned-dag-files-{{ include "airflow.git.repository.name" . }}
mountPath: /opt/bitnami/airflow/dags/git-{{ include "airflow.git.repository.name" . }}
{{- if .path }}
subPath: {{ .path }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.git.plugins.enabled }}
{{- range .Values.git.plugins.repositories }}
- name: git-cloned-plugins-files-{{ include "airflow.git.repository.name" . }}
mountPath: /opt/bitnami/airflow/git-{{ include "airflow.git.repository.name" . }}
{{- if .path }}
subPath: {{ .path }}
{{- end }}
{{- end }}
{{- end }}
{{- end -}}
{{/*
Returns the volumes that will be attached to the workload resources (deployment, statefulset, etc)
*/}}
{{- define "airflow.git.volumes" -}}
{{- if .Values.git.dags.enabled }}
{{- range .Values.git.dags.repositories }}
- name: git-cloned-dag-files-{{ include "airflow.git.repository.name" . }}
emptyDir: {}
{{- end }}
{{- end }}
{{- if .Values.git.plugins.enabled }}
{{- range .Values.git.plugins.repositories }}
- name: git-cloned-plugins-files-{{ include "airflow.git.repository.name" . }}
emptyDir: {}
{{- end }}
{{- end }}
{{- end }}
{{/*
Returns the init container that will clone repositories files from a given list of git repositories
*/}}
{{- define "airflow.git.containers.clone" -}}
{{- if or .Values.git.dags.enabled .Values.git.plugins.enabled }}
- name: clone-repositories
image: {{ include "git.image" . | quote }}
imagePullPolicy: {{ .Values.git.image.pullPolicy | quote }}
{{- if .Values.git.clone.command }}
command: {{- include "common.tplvalues.render" (dict "value" .Values.git.clone.command "context" $) | nindent 4 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
{{- if .Values.git.dags.enabled }}
{{- range .Values.git.dags.repositories }}
git clone {{ .repository }} --branch {{ .branch }} /dags-{{ include "airflow.git.repository.name" . }}
{{- end }}
{{- end }}
{{- if .Values.git.plugins.enabled }}
{{- range .Values.git.plugins.repositories }}
git clone {{ .repository }} --branch {{ .branch }} /plugins-{{ include "airflow.git.repository.name" . }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.git.clone.args }}
args: {{- include "common.tplvalues.render" (dict "value" .Values.git.clone.args "context" $) | nindent 4 }}
{{- end }}
volumeMounts:
{{- include "airflow.git.volumeMounts" . | trim | nindent 4 }}
{{- if .Values.git.clone.extraVolumeMounts }}
{{- include "common.tplvalues.render" (dict "value" .Values.git.clone.extraVolumeMounts "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.git.clone.extraEnvVars }}
env: {{- include "common.tplvalues.render" (dict "value" .Values.git.clone.extraEnvVars "context" $) | nindent 4 }}
{{- end }}
{{- if or .Values.git.clone.extraEnvVarsCM .Values.git.clone.extraEnvVarsSecret }}
envFrom:
{{- if .Values.git.clone.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.git.clone.extraEnvVarsCM }}
{{- end }}
{{- if .Values.git.clone.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.git.clone.extraEnvVarsSecret }}
{{- end }}
{{- end }}
{{- end }}
{{- end -}}
{{/*
Returns the a container that will pull and sync repositories files from a given list of git repositories
*/}}
{{- define "airflow.git.containers.sync" -}}
{{- if or .Values.git.dags.enabled .Values.git.plugins.enabled }}
- name: sync-repositories
image: {{ include "git.image" . | quote }}
imagePullPolicy: {{ .Values.git.image.pullPolicy | quote }}
{{- if .Values.git.sync.command }}
command: {{- include "common.tplvalues.render" (dict "value" .Values.git.sync.command "context" $) | nindent 4 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
while true; do
{{- if .Values.git.dags.enabled }}
{{- range .Values.git.dags.repositories }}
cd /dags-{{ include "airflow.git.repository.name" . }} && git pull origin {{ .branch }}
{{- end }}
{{- end }}
{{- if .Values.git.plugins.enabled }}
{{- range .Values.git.plugins.repositories }}
cd /plugins-{{ include "airflow.git.repository.name" . }} && git pull origin {{ .branch }}
{{- end }}
{{- end }}
sleep {{ default "60" .Values.git.sync.interval }}
done
{{- end }}
{{- if .Values.git.sync.args }}
args: {{- include "common.tplvalues.render" (dict "value" .Values.git.sync.args "context" $) | nindent 4 }}
{{- end }}
volumeMounts:
{{- include "airflow.git.volumeMounts" . | trim | nindent 4 }}
{{- if .Values.git.sync.extraVolumeMounts }}
{{- include "common.tplvalues.render" (dict "value" .Values.git.sync.extraVolumeMounts "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.git.sync.extraEnvVars }}
env: {{- include "common.tplvalues.render" (dict "value" .Values.git.sync.extraEnvVars "context" $) | nindent 4 }}
{{- end }}
{{- if or .Values.git.sync.extraEnvVarsCM .Values.git.sync.extraEnvVarsSecret }}
envFrom:
{{- if .Values.git.sync.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.git.sync.extraEnvVarsCM }}
{{- end }}
{{- if .Values.git.sync.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.git.sync.extraEnvVarsSecret }}
{{- end }}
{{- end }}
{{- end }}
{{- end -}}
bitnami/airflow/templates/_helpers.tpl
+203 -232
View File
@@ -6,24 +6,6 @@ Expand the name of the chart.
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "airflow.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
@@ -31,19 +13,6 @@ Create chart name and version as used by the chart label.
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Renders a value that contains template.
Usage:
{{ include "airflow.tplValue" ( dict "value" .Values.path.to.the.Value "context" $) }}
*/}}
{{- define "airflow.tplValue" -}}
{{- if typeIs "string" .value }}
{{- tpl .value .context }}
{{- else }}
{{- tpl (.value | toYaml) .context }}
{{- end }}
{{- end -}}
{{/*
Full path to CA Cert file
*/}}
@@ -55,7 +24,7 @@ Full path to CA Cert file
Fully qualified app name for LDAP
*/}}
{{- define "airflow.ldap" -}}
{{- printf "%s-ldap" (include "airflow.fullname" .) -}}
{{- printf "%s-ldap" (include "common.names.fullname" .) -}}
{{- end -}}
{{/*
@@ -90,184 +59,49 @@ Also, we can't use a single if because lazy evaluation is not an option
{{- end -}}
{{- end -}}
{{/*
Common labels
*/}}
{{- define "airflow.labels" -}}
app.kubernetes.io/name: {{ template "airflow.name" . }}
helm.sh/chart: {{ template "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
{{/*
Return the proper Airflow image name
*/}}
{{- define "airflow.image" -}}
{{- $registryName := .Values.image.registry -}}
{{- $repositoryName := .Values.image.repository -}}
{{- $tag := .Values.image.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- include "common.images.image" (dict "imageRoot" .Values.web.image "global" .Values.global) -}}
{{- end -}}
{{/*
Return the proper Airflow Scheduler image name
*/}}
{{- define "airflow.schedulerImage" -}}
{{- $registryName := .Values.schedulerImage.registry -}}
{{- $repositoryName := .Values.schedulerImage.repository -}}
{{- $tag := .Values.schedulerImage.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- include "common.images.image" (dict "imageRoot" .Values.scheduler.image "global" .Values.global) -}}
{{- end -}}
{{/*
Return the proper Airflow Worker image name
*/}}
{{- define "airflow.workerImage" -}}
{{- $registryName := .Values.workerImage.registry -}}
{{- $repositoryName := .Values.workerImage.repository -}}
{{- $tag := .Values.workerImage.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- include "common.images.image" (dict "imageRoot" .Values.worker.image "global" .Values.global) -}}
{{- end -}}
{{/*
Return the proper git image name
*/}}
{{- define "git.image" -}}
{{- $registryName := .Values.git.registry -}}
{{- $repositoryName := .Values.git.repository -}}
{{- $tag := .Values.git.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- include "common.images.image" (dict "imageRoot" .Values.git.image "global" .Values.global) -}}
{{- end -}}
{{/*
Return the proper Airflow Metrics image name
*/}}
{{- define "airflow.metrics.image" -}}
{{- $registryName := .Values.metrics.image.registry -}}
{{- $repositoryName := .Values.metrics.image.repository -}}
{{- $tag := .Values.metrics.image.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- include "common.images.image" (dict "imageRoot" .Values.metrics.image "global" .Values.global) -}}
{{- end -}}
{{/*
Return the proper Docker Image Registry Secret Names
*/}}
{{- define "airflow.imagePullSecrets" -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 does not support it, so we need to implement this if-else logic.
Also, we can not use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imagePullSecrets }}
imagePullSecrets:
{{- range .Values.global.imagePullSecrets }}
- name: {{ . }}
{{- end }}
{{- else if or .Values.image.pullSecrets .Values.schedulerImage.pullSecrets .Values.workerImage.pullSecrets .Values.git.pullSecrets .Values.metrics.image.pullSecrets }}
imagePullSecrets:
{{- range .Values.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.schedulerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.workerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.git.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.metrics.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- end -}}
{{- else if or .Values.image.pullSecrets .Values.schedulerImage.pullSecrets .Values.workerImage.pullSecrets .Values.git.pullSecrets .Values.metrics.image.pullSecrets }}
imagePullSecrets:
{{- range .Values.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.schedulerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.workerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.git.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.metrics.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- end -}}
{{- include "common.images.pullSecrets" (dict "images" (list .Values.web.image .Values.scheduler.image .Values.worker.image .Values.git .Values.metrics.image) "global" .Values.global) -}}
{{- end -}}
{{/*
Create a default fully qualified postgresql name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
@@ -276,6 +110,7 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified redis name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
@@ -289,7 +124,7 @@ Get the Redis credentials secret.
*/}}
{{- define "airflow.redis.secretName" -}}
{{- if and (.Values.redis.enabled) (not .Values.redis.existingSecret) -}}
{{/* Create a template for the redis secret
{{/* Create a include for the redis secret
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- $name := default "redis" .Values.redis.nameOverride -}}
@@ -326,31 +161,172 @@ Get the Postgresql credentials secret.
Get the secret name
*/}}
{{- define "airflow.secretName" -}}
{{- if .Values.airflow.auth.existingSecret -}}
{{- printf "%s" .Values.airflow.auth.existingSecret -}}
{{- if .Values.auth.existingSecret -}}
{{- printf "%s" .Values.auth.existingSecret -}}
{{- else -}}
{{- printf "%s" (include "airflow.fullname" .) -}}
{{- printf "%s" (include "common.names.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Get the configmap name
*/}}
{{- define "airflow.configMapName" -}}
{{- if .Values.configurationConfigMap -}}
{{- printf "%s" .Values.configurationConfigMap -}}
{{- else -}}
{{- printf "%s-configuration" (include "common.names.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Should use config from the configmap
*/}}
{{- define "airflow.shouldUseConfigFromConfigMap" -}}
{{- if or .Values.config .Values.configurationConfigMap -}}
true
{{- else -}}{{- end -}}
{{- end -}}
{{/*
Create the name of the service account to use
*/}}
{{- define "airflow.serviceAccountName" -}}
{{- if .Values.serviceAccount.create -}}
{{ default (include "airflow.fullname" .) .Values.serviceAccount.name }}
{{ default (include "common.names.fullname" .) .Values.serviceAccount.name }}
{{- else -}}
{{ default "default" .Values.serviceAccount.name }}
{{- end -}}
{{- end -}}
{{/*
Add environmnet variables to configure database values
*/}}
{{- define "airflow.database.host" -}}
{{- ternary (include "airflow.postgresql.fullname" .) .Values.externalDatabase.host .Values.postgresql.enabled | quote -}}
{{- end -}}
{{/*
Add environmnet variables to configure database values
*/}}
{{- define "airflow.database.user" -}}
{{- ternary .Values.postgresql.postgresqlUsername .Values.externalDatabase.user .Values.postgresql.enabled | quote -}}
{{- end -}}
{{/*
Add environmnet variables to configure database values
*/}}
{{- define "airflow.database.name" -}}
{{- ternary .Values.postgresql.postgresqlDatabase .Values.externalDatabase.database .Values.postgresql.enabled | quote -}}
{{- end -}}
{{/*
Add environmnet variables to configure database values
*/}}
{{- define "airflow.database.port" -}}
{{- ternary "5432" .Values.externalDatabase.port .Values.postgresql.enabled | quote -}}
{{- end -}}
{{/*
Add environmnet variables to configure database values
*/}}
{{- define "airflow.configure.database" -}}
- name: AIRFLOW_DATABASE_NAME
value: {{ include "airflow.database.name" . }}
- name: AIRFLOW_DATABASE_USERNAME
value: {{ include "airflow.database.user" . }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "airflow.postgresql.secretName" . }}
key: postgresql-password
- name: AIRFLOW_DATABASE_HOST
value: {{ include "airflow.database.host" . }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
value: {{ include "airflow.database.port" . }}
{{- end -}}
{{/*
Add environmnet variables to configure redis values
*/}}
{{- define "airflow.configure.redis" -}}
- name: REDIS_HOST
value: {{ ternary (include "airflow.redis.fullname" .) .Values.externalRedis.host .Values.redis.enabled | quote }}
- name: REDIS_PORT_NUMBER
value: {{ ternary "6379" .Values.externalRedis.port .Values.redis.enabled | quote }}
{{- if and (not .Values.redis.enabled) .Values.externalRedis.username }}
- name: REDIS_USER
value: {{ .Values.externalRedis.username | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "airflow.redis.secretName" . }}
key: redis-password
{{- end -}}
{{/*
Add environmnet variables to configure airflow common values
*/}}
{{- define "airflow.configure.airflow.common" -}}
- name: AIRFLOW_EXECUTOR
value: {{ .Values.executor }}
- name: AIRFLOW_FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ include "airflow.secretName" . }}
key: airflow-fernetKey
- name: AIRFLOW_WEBSERVER_HOST
value: {{ include "common.names.fullname" . }}
- name: AIRFLOW_WEBSERVER_PORT_NUMBER
value: {{ .Values.service.port | quote }}
- name: AIRFLOW_LOAD_EXAMPLES
value: {{ ternary "yes" "no" .Values.loadExamples | quote }}
{{- if .Values.web.image.debug }}
- name: BASH_DEBUG
value: "1"
- name: NAMI_DEBUG
value: "1"
- name: NAMI_LOG_LEVEL
value: "trace8"
{{- end }}
{{- end -}}
{{/*
Add environmnet variables to configure airflow kubernetes executor
*/}}
{{- define "airflow.configure.airflow.kubernetesExecutor" -}}
{{- if eq .Values.executor "KubernetesExecutor" }}
- name: AIRFLOW__KUBERNETES__NAMESPACE
value: {{ .Release.Namespace }}
- name: AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY
value: {{ printf "%s/%s" .Values.worker.image.registry .Values.worker.image.repository }}
- name: AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG
value: {{ .Values.worker.image.tag }}
- name: AIRFLOW__KUBERNETES__IMAGE_PULL_POLICY
value: {{ .Values.worker.image.pullPolicy }}
- name: AIRFLOW__KUBERNETES__DAGS_IN_IMAGE
value: "True"
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS
value: "True"
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE
value: "False"
- name: AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME
value: {{ include "airflow.serviceAccountName" . }}
- name: AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE
value: "/opt/bitnami/airflow/pod_template.yaml"
{{- end }}
{{- end -}}
{{/*
Compile all warnings into a single message, and call fail.
*/}}
{{- define "airflow.validateValues" -}}
{{- $messages := list -}}
{{- $messages := append $messages (include "airflow.validateValues.cloneDagFilesFromGit.repositories" .) -}}
{{- $messages := append $messages (include "airflow.validateValues.cloneDagFilesFromGit.repository_details" .) -}}
{{- $messages := append $messages (include "airflow.validateValues.dags.repositories" .) -}}
{{- $messages := append $messages (include "airflow.validateValues.dags.repository_details" .) -}}
{{- $messages := append $messages (include "airflow.validateValues.plugins.repositories" .) -}}
{{- $messages := append $messages (include "airflow.validateValues.plugins.repository_details" .) -}}
{{- $messages := without $messages "" -}}
{{- $message := join "\n" $messages -}}
@@ -359,74 +335,69 @@ Compile all warnings into a single message, and call fail.
{{- end -}}
{{- end -}}
{{/* Validate values of Airflow - Atleast one repository details must be provided when "airflow.cloneDagFilesFromGit.enabled" is "true" */}}
{{- define "airflow.validateValues.cloneDagFilesFromGit.repositories" -}}
{{- if and .Values.airflow.cloneDagFilesFromGit.enabled (empty .Values.airflow.cloneDagFilesFromGit.repositories) -}}
{{- if or (empty .Values.airflow.cloneDagFilesFromGit.repository) (empty .Values.airflow.cloneDagFilesFromGit.branch) -}}
airflow: airflow.cloneDagFilesFromGit.repositories
{{/* Validate values of Airflow - Atleast one repository details must be provided when "git.dags.enabled" is "true" */}}
{{- define "airflow.validateValues.dags.repositories" -}}
{{- if and .Values.git.dags.enabled (empty .Values.git.dags.repositories) -}}
airflow: git.dags.repositories
At least one repository must be provided when enabling downloading DAG files
from git repository (--set airflow.cloneDagFilesFromGit.repositories[0].repository="xxx"
--set airflow.cloneDagFilesFromGit.repositories[0].name="xxx"
--set airflow.cloneDagFilesFromGit.repositories[0].branch="name")
from git repository (--set git.dags.repositories[0].repository="xxx"
--set git.dags.repositories[0].name="xxx"
--set git.dags.repositories[0].branch="name")
{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/* Validate values of Airflow - "airflow.cloneDagFilesFromGit.repositories.repository", "airflow.cloneDagFilesFromGit.repositories.name", "airflow.cloneDagFilesFromGit.repositories.branch" must be provided when "airflow.cloneDagFilesFromGit.enabled" is "true" */}}
{{- define "airflow.validateValues.cloneDagFilesFromGit.repository_details" -}}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled -}}
{{- range $index, $repository_detail := .Values.airflow.cloneDagFilesFromGit.repositories }}
{{/* Validate values of Airflow - "git.dags.repositories.repository", "git.dags.repositories.name", "git.dags.repositories.branch" must be provided when "git.dags.enabled" is "true" */}}
{{- define "airflow.validateValues.dags.repository_details" -}}
{{- if .Values.git.dags.enabled -}}
{{- range $index, $repository_detail := .Values.git.dags.repositories }}
{{- if empty $repository_detail.repository -}}
airflow: airflow.cloneDagFilesFromGit.repositories[$index].repository
airflow: git.dags.repositories[$index].repository
The repository must be provided when enabling downloading DAG files
from git repository (--set airflow.cloneDagFilesFromGit.repositories[$index].repository="xxx")
{{- end -}}
{{- if empty $repository_detail.name -}}
airflow: airflow.cloneDagFilesFromGit.repositories[$index].name
The name must be provided when enabling downloading DAG files
from git repository (--set airflow.cloneDagFilesFromGit.repositories[$index].name="xxx")
from git repository (--set git.dags.repositories[$index].repository="xxx")
{{- end -}}
{{- if empty $repository_detail.branch -}}
airflow: airflow.cloneDagFilesFromGit.repositories[$index].branch
airflow: git.dags.repositories[$index].branch
The branch must be provided when enabling downloading DAG files
from git repository (--set airflow.cloneDagFilesFromGit.repositories[$index].branch="xxx")
from git repository (--set git.dags.repositories[$index].branch="xxx")
{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/* Validate values of Airflow - "airflow.clonePluginsFromGit.repository" must be provided when "airflow.clonePluginsFromGit.enabled" is "true" */}}
{{- define "airflow.validateValues.clonePluginsFromGit.repository" -}}
{{- if and .Values.airflow.clonePluginsFromGit.enabled (empty .Values.airflow.clonePluginsFromGit.repository) -}}
airflow: airflow.clonePluginsFromGit.repository
The repository must be provided when enabling downloading plugins
from git repository (--set airflow.clonePluginsFromGit.repository="xxx")
{{/* Validate values of Airflow - "git.plugins.repositories" must be provided when "git.plugins.enabled" is "true" */}}
{{- define "airflow.validateValues.plugins.repositories" -}}
{{- if and .Values.git.plugins.enabled (empty .Values.git.plugins.repositories) -}}
airflow: git.plugins.repositories
At least one repository must be provided when enabling downloading DAG files
from git repository (--set git.plugins.repositories[0].repository="xxx"
--set git.plugins.repositories[0].name="xxx"
--set git.plugins.repositories[0].branch="name")
{{- end -}}
{{- end -}}
{{/* Validate values of Airflow - "git.plugins.repositories.repository", "git.plugins.repositories.name", "git.plugins.repositories.branch" must be provided when "git.plugins.enabled" is "true" */}}
{{- define "airflow.validateValues.plugins.repository_details" -}}
{{- if .Values.git.plugins.enabled -}}
{{- range $index, $repository_detail := .Values.git.plugins.repositories }}
{{- if empty $repository_detail.repository -}}
airflow: git.plugins.repositories[$index].repository
The repository must be provided when enabling downloading DAG files
from git repository (--set git.plugins.repositories[$index].repository="xxx")
{{- end -}}
{{- if empty $repository_detail.branch -}}
airflow: git.plugins.repositories[$index].branch
The branch must be provided when enabling downloading DAG files
from git repository (--set git.plugins.repositories[$index].branch="xxx")
{{- end -}}
{{- end -}}
{{/* Validate values of Airflow - "airflow.clonePluginsFromGit.branch" must be provided when "airflow.clonePluginsFromGit.enabled" is "true" */}}
{{- define "airflow.validateValues.clonePluginsFromGit.branch" -}}
{{- if and .Values.airflow.clonePluginsFromGit.enabled (empty .Values.airflow.clonePluginsFromGit.branch) -}}
airflow: airflow.clonePluginsFromGit.branch
The branch must be provided when enabling downloading plugins
from git repository (--set airflow.clonePluginsFromGit.branch="xxx")
{{- end -}}
{{- end -}}
{{/* Check if there are rolling tags in the images */}}
{{- define "airflow.checkRollingTags" -}}
{{- if and (contains "bitnami/" .Values.image.repository) (not (.Values.image.tag | toString | regexFind "-r\\d+$|sha256:")) }}
WARNING: Rolling tag detected ({{ .Values.image.repository }}:{{ .Values.image.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment.
+info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/
{{- end }}
{{- if and (contains "bitnami/" .Values.schedulerImage.repository) (not (.Values.schedulerImage.tag | toString | regexFind "-r\\d+$|sha256:")) }}
WARNING: Rolling tag detected ({{ .Values.schedulerImage.repository }}:{{ .Values.schedulerImage.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment.
+info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/
{{- end }}
{{- if and (contains "bitnami/" .Values.workerImage.repository) (not (.Values.workerImage.tag | toString | regexFind "-r\\d+$|sha256:")) }}
WARNING: Rolling tag detected ({{ .Values.workerImage.repository }}:{{ .Values.workerImage.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment.
+info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/
{{- end }}
{{- if and (contains "bitnami/" .Values.git.repository) (not (.Values.git.tag | toString | regexFind "-r\\d+$|sha256:")) }}
WARNING: Rolling tag detected ({{ .Values.git.repository }}:{{ .Values.git.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment.
+info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/
{{- end }}
{{- include "common.warnings.rollingTag" .Values.web.image }}
{{- include "common.warnings.rollingTag" .Values.scheduler.image }}
{{- include "common.warnings.rollingTag" .Values.worker.image }}
{{- include "common.warnings.rollingTag" .Values.git.image }}
{{- include "common.warnings.rollingTag" .Values.metrics.image }}
{{- end -}}
bitnami/airflow/templates/config/configmap.yaml
+187
View File
@@ -0,0 +1,187 @@
{{- $kube := eq .Values.executor "KubernetesExecutor" -}}
{{- if or .Values.config $kube }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "common.names.fullname" . }}-configuration
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
data:
{{- if .Values.config }}
airflow.cfg: |-
{{ .Values.config | indent 4 }}
{{- end }}
{{- if $kube }}
pod_template.yaml: |-
{{- if .Values.worker.podTemplate }}
{{ .Values.worker.podTemplate | nindent 4 }}
{{- else }}
apiVersion: v1
kind: Pod
metadata:
name: k8s-executor-pod
labels: {{- include "common.labels.standard" . | nindent 8 }}
app.kubernetes.io/component: worker
{{- if .Values.worker.podLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.podLabels "context" $) | nindent 8 }}
{{- end }}
{{- if .Values.worker.podAnnotations }}
annotations:
{{- include "common.tplvalues.render" (dict "value" .Values.worker.podAnnotations "context" $) | nindent 8 }}
{{- end }}
spec:
priorityClassName: {{ .Values.priorityClassName | quote }}
initContainers: {{- include "airflow.git.containers.clone" . | trim | nindent 8 }}
- name: k8s-executor-init-config
image: {{ include "airflow.workerImage" . }}
imagePullPolicy: {{ .Values.worker.image.pullPolicy }}
{{- if .Values.containerSecurityContext.enabled }}
securityContext: {{- omit .Values.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
command:
- "/bin/bash"
- "-ce"
- |
. /opt/bitnami/base/helpers
nami_initialize airflow-worker
cp /opt/bitnami/airflow/airflow.cfg /opt/bitnami/airflow/k8s-executor-config/airflow.cfg
env:
{{- include "airflow.configure.airflow.common" . | nindent 12 }}
{{- include "airflow.configure.database" . | nindent 12 }}
{{- include "airflow.configure.redis" . | nindent 12 }}
{{- include "airflow.configure.airflow.kubernetesExecutor" . | nindent 12 }}
{{- if .Values.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.worker.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.extraEnvVarsCM }}
{{- end }}
{{- if .Values.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.worker.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.worker.extraEnvVarsCM }}
{{- end }}
{{- if .Values.worker.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.worker.extraEnvVarsSecret }}
{{- end }}
volumeMounts:
- name: k8s-executor-config
mountPath: /opt/bitnami/airflow/k8s-executor-config
{{- if .Values.worker.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
containers:
- name: airflow-worker
image: {{ include "airflow.workerImage" . }}
imagePullPolicy: {{ .Values.worker.image.pullPolicy }}
{{- if .Values.containerSecurityContext.enabled }}
securityContext: {{- omit .Values.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
env:
- name: AIRFLOW__CORE__EXECUTOR
value: LocalExecutor
{{- include "airflow.configure.airflow.common" . | nindent 12 }}
{{- include "airflow.configure.database" . | nindent 12 }}
{{- include "airflow.configure.redis" . | nindent 12 }}
{{- if .Values.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.worker.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.extraEnvVarsCM }}
{{- end }}
{{- if .Values.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.worker.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.worker.extraEnvVarsCM }}
{{- end }}
{{- if .Values.worker.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.worker.extraEnvVarsSecret }}
{{- end }}
ports:
- name: worker
containerPort: {{ .Values.worker.port }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if include "airflow.shouldUseConfigFromConfigMap" . }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- else }}
- name: k8s-executor-config
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
{{- include "airflow.git.maincontainer.volumeMounts" . | trim | nindent 12 }}
{{- if .Values.worker.extraVolumeMounts }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
{{- if .Values.worker.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
restartPolicy: Never
{{- if .Values.podSecurityContext.enabled }}
securityContext: {{- omit .Values.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
{{- if .Values.affinity }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "airflow.serviceAccountName" . }}
volumes:
- name: k8s-executor-config
emptyDir: {}
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.dagsConfigMap }}
{{- end }}
{{- if include "airflow.shouldUseConfigFromConfigMap" . }}
- name: custom-configuration-file
configMap:
name: {{ include "airflow.configMapName" . }}
{{- end }}
{{- include "airflow.git.volumes" . | trim | nindent 8 }}
{{- if .Values.worker.extraVolumes }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.extraVolumes "context" $) | nindent 8 }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
bitnami/airflow/templates/config/secret-external-db.yaml
+16
View File
@@ -0,0 +1,16 @@
{{- if and (not .Values.postgresql.enabled) (not .Values.externalDatabase.existingSecret) (not .Values.postgresql.existingSecret)}}
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
type: Opaque
data:
postgresql-password: {{ .Values.externalDatabase.password | b64enc | quote }}
{{- end }}
bitnami/airflow/templates/config/secret-external-redis.yaml
+16
View File
@@ -0,0 +1,16 @@
{{- if and (not .Values.redis.enabled) (not .Values.externalRedis.existingSecret) (not .Values.redis.existingSecret)}}
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
type: Opaque
data:
redis-password: {{ .Values.externalRedis.password | b64enc | quote }}
{{- end }}
bitnami/airflow/templates/ldap-secrets.yaml → bitnami/airflow/templates/config/secret-ldap.yaml
+3 -4
View File
@@ -3,13 +3,12 @@ apiVersion: v1
kind: Secret
metadata:
name: {{ include "airflow.ldap" . }}
labels:
{{- include "airflow.labels" . | nindent 4 }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
type: Opaque
data:
bitnami/airflow/templates/config/secret.yaml
+29
View File
@@ -0,0 +1,29 @@
{{- if (not .Values.auth.existingSecret) -}}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "common.names.fullname" . }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
type: Opaque
data:
{{- if .Values.auth.password }}
airflow-password: {{ .Values.auth.password | b64enc | quote }}
{{ else if (not .Values.auth.forcePassword) }}
airflow-password: {{ randAlphaNum 10 | b64enc | quote }}
{{ else }}
airflow-password: {{ required "An Airflow Password is required!" .Values.auth.password }}
{{- end }}
{{- if .Values.auth.fernetKey }}
airflow-fernetKey: {{ .Values.auth.fernetKey | b64enc | quote }}
{{ else if (not .Values.auth.forcePassword) }}
airflow-fernetKey: {{ randAlphaNum 32 | b64enc | b64enc | quote }}
{{ else }}
airflow-fernetKey: {{ required "An Airflow Fernet Key is required!" .Values.auth.fernetKey }}
{{- end }}
{{- end }}
bitnami/airflow/templates/configmap-dags.yaml
-19
View File
@@ -1,19 +0,0 @@
{{- if .Files.Glob "files/dags/*.py" }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "airflow.fullname" . }}-dag-files
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
data:
{{ (.Files.Glob "files/dags/*.py").AsConfig | indent 2 }}
{{ end }}
bitnami/airflow/templates/configmap.yaml
-20
View File
@@ -1,20 +0,0 @@
{{- if .Values.config }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "airflow.fullname" . }}-configuration
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
data:
airflow.cfg: |-
{{ .Values.config | indent 4 }}
{{- end -}}
bitnami/airflow/templates/deployment-scheduler.yaml
-366
View File
@@ -1,366 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "airflow.fullname" . }}-scheduler
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: scheduler
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: scheduler
replicas: {{ .Values.replicaCount }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: scheduler
{{- if .Values.podAnnotations }}
annotations:
{{ toYaml .Values.podAnnotations | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
{{- if .Values.securityContext.enabled }}
securityContext:
fsGroup: {{ .Values.securityContext.fsGroup }}
runAsUser: {{ .Values.securityContext.runAsUser }}
{{- end }}
serviceAccountName: {{ template "airflow.serviceAccountName" . }}
{{- if .Values.affinity }}
affinity: {{- include "airflow.tplValue" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "scheduler" "context" $) | nindent 10 }}
podAntiAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAntiAffinityPreset "component" "scheduler" "context" $) | nindent 10 }}
nodeAffinity: {{- include "common.affinities.nodes" (dict "type" .Values.nodeAffinityPreset.type "key" .Values.nodeAffinityPreset.key "values" .Values.nodeAffinityPreset.values) | nindent 10 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector: {{- include "airflow.tplValue" (dict "value" .Values.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations: {{- include "airflow.tplValue" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
{{- if or .Values.airflow.cloneDagFilesFromGit.enabled .Values.airflow.clonePluginsFromGit.enabled }}
initContainers:
- name: git-clone-repository
image: {{ include "git.image" . }}
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
{{- if .Values.airflow.gitCloneContainer.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.command "context" $) | nindent 12 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} --branch {{ .Values.airflow.cloneDagFilesFromGit.branch }} /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
git clone {{ .repository }} --branch {{ .branch }} /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
git clone {{ .Values.airflow.clonePluginsFromGit.repository }} --branch {{ .Values.airflow.clonePluginsFromGit.branch }} /plugins
{{- end }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.args }}
args: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.args "context" $) | nindent 12 }}
{{- end }}
volumeMounts:
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /plugins
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraEnvVars }}
env: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if or .Values.airflow.gitCloneContainer.extraEnvVarsCM .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
envFrom:
{{- if .Values.airflow.gitCloneContainer.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.gitCloneContainer.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
{{- end }}
{{- end }}
containers:
- name: git-repo-syncer
image: "{{ template "git.image" . }}"
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
{{- if .Values.airflow.gitSyncContainer.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.command "context" $) | nindent 12 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
while true; do
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
cd /dags && git pull origin {{ .Values.airflow.cloneDagFilesFromGit.branch }}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
cd /dags-{{ .name | kebabcase }} && git pull origin {{ .branch }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
cd /plugins && git pull origin {{ .Values.airflow.clonePluginsFromGit.branch }}
{{- end }}
sleep {{ default "60" .Values.airflow.gitSyncInterval }}
done
{{- end }}
volumeMounts:
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /plugins
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraEnvVars }}
env: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if or .Values.airflow.gitSyncContainer.extraEnvVarsCM .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
envFrom:
{{- if .Values.airflow.gitSyncContainer.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.gitSyncContainer.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
{{- end }}
{{- end }}
{{- else }}
containers:
{{- end }}
- name: airflow-scheduler
image: {{ include "airflow.schedulerImage" . }}
imagePullPolicy: {{ .Values.schedulerImage.pullPolicy | quote }}
{{- if .Values.airflow.scheduler.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.scheduler.command "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.scheduler.args }}
args: {{- include "airflow.tplValue" (dict "value" .Values.airflow.scheduler.args "context" $) | nindent 12 }}
{{- end }}
env:
{{- if .Values.schedulerImage.debug }}
- name: BASH_DEBUG
value: "1"
- name: NAMI_DEBUG
value: "1"
- name: NAMI_LOG_LEVEL
value: "trace8"
{{- end }}
- name: AIRFLOW_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "airflow.postgresql.secretName" . }}
key: postgresql-password
- name: AIRFLOW_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: REDIS_HOST
{{- if .Values.redis.enabled }}
value: {{ template "airflow.redis.fullname" . }}
{{- else }}
value: {{ .Values.externalRedis.host | quote }}
{{- end }}
- name: REDIS_PORT_NUMBER
{{- if .Values.redis.enabled }}
value: "6379"
{{- else }}
value: {{ .Values.externalRedis.port | quote }}
{{- end }}
{{- if and (not .Values.redis.enabled) .Values.externalRedis.username }}
- name: REDIS_USER
value: {{ .Values.externalRedis.username | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
- name: AIRFLOW_EXECUTOR
value: "CeleryExecutor"
- name: AIRFLOW_FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-fernetKey
- name: AIRFLOW_WEBSERVER_HOST
value: {{ template "airflow.fullname" . }}
- name: AIRFLOW_WEBSERVER_PORT_NUMBER
value: {{ .Values.service.port | quote }}
- name: AIRFLOW_LOAD_EXAMPLES
{{- if .Values.airflow.loadExamples }}
value: "yes"
{{- else }}
value: "no"
{{- end }}
{{- if .Values.airflow.extraEnvVars }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.scheduler.extraEnvVars }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.scheduler.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.airflow.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.airflow.scheduler.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.scheduler.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.scheduler.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.scheduler.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.livenessProbe.enabled }}
livenessProbe:
{{- end }}
{{- if .Values.readinessProbe.enabled }}
readinessProbe:
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /opt/bitnami/airflow/dags/git
{{- if .Values.airflow.cloneDagFilesFromGit.path }}
subPath: {{ .Values.airflow.cloneDagFilesFromGit.path }}
{{- end }}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /opt/bitnami/airflow/dags/git-{{ .name | kebabcase }}
{{- if .path }}
subPath: {{ .path }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /opt/bitnami/airflow/plugins
{{- if .Values.airflow.clonePluginsFromGit.path }}
subPath: {{ .Values.airflow.clonePluginsFromGit.path }}
{{- end }}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
{{- if .Values.airflow.scheduler.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.scheduler.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.scheduler.resources }}
resources: {{- toYaml .Values.airflow.scheduler.resources | nindent 12 }}
{{- end }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
configMap:
name: {{ template "airflow.fullname" . }}-dag-files
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.airflow.dagsConfigMap }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
emptyDir: {}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
emptyDir: {}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
emptyDir: {}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
configMap:
name: {{ .Values.airflow.configurationConfigMap }}
{{- end }}
{{- if .Values.airflow.scheduler.extraVolumes }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.scheduler.extraVolumes "context" $) | nindent 8 }}
{{- end }}
bitnami/airflow/templates/deployment-web.yaml
-451
View File
@@ -1,451 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "airflow.fullname" . }}-web
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: web
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: web
replicas: {{ .Values.replicaCount }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: web
{{- if .Values.podAnnotations }}
annotations:
{{ toYaml .Values.podAnnotations | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
{{- if .Values.securityContext.enabled }}
securityContext:
fsGroup: {{ .Values.securityContext.fsGroup }}
runAsUser: {{ .Values.securityContext.runAsUser }}
{{- end }}
serviceAccountName: {{ template "airflow.serviceAccountName" . }}
{{- if .Values.affinity }}
affinity: {{- include "airflow.tplValue" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "web" "context" $) | nindent 10 }}
podAntiAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAntiAffinityPreset "component" "web" "context" $) | nindent 10 }}
nodeAffinity: {{- include "common.affinities.nodes" (dict "type" .Values.nodeAffinityPreset.type "key" .Values.nodeAffinityPreset.key "values" .Values.nodeAffinityPreset.values) | nindent 10 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector: {{- include "airflow.tplValue" (dict "value" .Values.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations: {{- include "airflow.tplValue" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
{{- if or .Values.airflow.cloneDagFilesFromGit.enabled .Values.airflow.clonePluginsFromGit.enabled }}
initContainers:
- name: git-clone-repository
image: {{ include "git.image" . }}
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
{{- if .Values.airflow.gitCloneContainer.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.command "context" $) | nindent 12 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} --branch {{ .Values.airflow.cloneDagFilesFromGit.branch }} /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
git clone {{ .repository }} --branch {{ .branch }} /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
git clone {{ .Values.airflow.clonePluginsFromGit.repository }} --branch {{ .Values.airflow.clonePluginsFromGit.branch }} /plugins
{{- end }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.args }}
args: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.args "context" $) | nindent 12 }}
{{- end }}
volumeMounts:
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /plugins
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraEnvVars }}
env: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if or .Values.airflow.gitCloneContainer.extraEnvVarsCM .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
envFrom:
{{- if .Values.airflow.gitCloneContainer.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.gitCloneContainer.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
{{- end }}
{{- end }}
containers:
- name: git-repo-syncer
image: "{{ template "git.image" . }}"
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
{{- if .Values.airflow.gitSyncContainer.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.command "context" $) | nindent 12 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
while true; do
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
cd /dags && git pull origin {{ .Values.airflow.cloneDagFilesFromGit.branch }}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
cd /dags-{{ .name | kebabcase }} && git pull origin {{ .branch }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
cd /plugins && git pull origin {{ .Values.airflow.clonePluginsFromGit.branch }}
{{- end }}
sleep {{ default "60" .Values.airflow.gitSyncInterval }}
done
{{- end }}
volumeMounts:
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /plugins
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraEnvVars }}
env: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if or .Values.airflow.gitSyncContainer.extraEnvVarsCM .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
envFrom:
{{- if .Values.airflow.gitSyncContainer.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.gitSyncContainer.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
{{- end }}
{{- end }}
{{- else }}
containers:
{{- end }}
- name: airflow-web
image: {{ template "airflow.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
{{- if .Values.airflow.web.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.web.command "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.web.args }}
args: {{- include "airflow.tplValue" (dict "value" .Values.airflow.web.args "context" $) | nindent 12 }}
{{- end }}
env:
{{- if .Values.image.debug }}
- name: BASH_DEBUG
value: "1"
- name: NAMI_DEBUG
value: "1"
- name: NAMI_LOG_LEVEL
value: "trace8"
{{- end }}
- name: AIRFLOW_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "airflow.postgresql.secretName" . }}
key: postgresql-password
- name: AIRFLOW_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: REDIS_HOST
{{- if .Values.redis.enabled }}
value: {{ template "airflow.redis.fullname" . }}
{{- else }}
value: {{ .Values.externalRedis.host | quote }}
{{- end }}
- name: REDIS_PORT_NUMBER
{{- if .Values.redis.enabled }}
value: "6379"
{{- else }}
value: {{ .Values.externalRedis.port | quote }}
{{- end }}
{{- if and (not .Values.redis.enabled) .Values.externalRedis.username }}
- name: REDIS_USER
value: {{ .Values.externalRedis.username | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
- name: AIRFLOW_EXECUTOR
value: "CeleryExecutor"
- name: AIRFLOW_USERNAME
value: {{ .Values.airflow.auth.username }}
- name: AIRFLOW_PASSWORD
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-password
- name: AIRFLOW_FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-fernetKey
- name: AIRFLOW_WEBSERVER_HOST
value: {{ template "airflow.fullname" . }}
{{- if .Values.airflow.baseUrl }}
- name: AIRFLOW_BASE_URL
value: {{ .Values.airflow.baseUrl }}
{{- end }}
- name: AIRFLOW_LOAD_EXAMPLES
{{- if .Values.airflow.loadExamples }}
value: "yes"
{{- else }}
value: "no"
{{- end }}
- name: AIRFLOW_LDAP_ENABLE
value: {{ ternary "yes" "no" .Values.ldap.enabled | quote }}
{{- if .Values.ldap.enabled }}
- name: AIRFLOW_LDAP_URI
value: {{ .Values.ldap.uri }}
- name: AIRFLOW_LDAP_SEARCH
value: {{ .Values.ldap.base }}
- name: AIRFLOW_LDAP_BIND_USER
value: {{ .Values.ldap.binddn }}
- name: AIRFLOW_LDAP_BIND_PASSWORD
value: {{ .Values.ldap.bindpw }}
valueFrom:
secretKeyRef:
name: {{ include "airflow.ldapSecretName" . }}
key: bind-password
- name: AIRFLOW_LDAP_UID_FIELD
value: {{ .Values.ldap.uidField }}
- name: AIRFLOW_LDAP_USE_TLS
value: {{ ternary "yes" "no" .Values.ldap.tls.enabled | quote }}
{{- if .Values.ldap.tls.enabled }}
- name: AIRFLOW_LDAP_ALLOW_SELF_SIGNED
value: {{ .Values.ldap.tls.allowSelfSigned }}
- name: AIRFLOW_LDAP_TLS_CA_CERTIFICATE
value: {{ include "airflow.ldapCAFilename" . | quote }}
{{- end }}
{{- end }}
{{- if .Values.airflow.extraEnvVars }}
{{- toYaml .Values.airflow.extraEnvVars | nindent 12 }}
{{- end }}
{{- if .Values.airflow.web.extraEnvVars }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.web.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.airflow.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.airflow.web.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.web.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.web.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.web.extraEnvVarsSecret }}
{{- end }}
ports:
- name: http
containerPort: 8080
{{- if .Values.livenessProbe.enabled }}
livenessProbe:
{{- if .Values.airflow.baseUrl }}
tcpSocket:
port: http
{{- else }}
httpGet:
path: /health
port: http
{{- end }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
{{- end }}
{{- if .Values.readinessProbe.enabled }}
readinessProbe:
{{- if .Values.airflow.baseUrl }}
tcpSocket:
port: http
{{- else }}
httpGet:
path: /health
port: http
{{- end }}
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /opt/bitnami/airflow/dags/git
{{- if .Values.airflow.cloneDagFilesFromGit.path }}
subPath: {{ .Values.airflow.cloneDagFilesFromGit.path }}
{{- end }}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /opt/bitnami/airflow/dags/git-{{ .name | kebabcase }}
{{- if .path }}
subPath: {{ .path }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /opt/bitnami/airflow/plugins
{{- if .Values.airflow.clonePluginsFromGit.path }}
subPath: {{ .Values.airflow.clonePluginsFromGit.path }}
{{- end }}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
{{- if .Values.airflow.webserverConfigMap }}
- name: custom-webserver-configuration-file
mountPath: /opt/bitnami/airflow/webserver_config.py
subPath: webserver_config.py
{{- end }}
{{- if .Values.ldap.tls.enabled }}
- name: airflow-ldap-ca-certificate
mountPath: /opt/bitnami/airflow/conf/certs/
readOnly: true
{{- end }}
{{- if .Values.airflow.web.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.web.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.web.resources }}
resources: {{- toYaml .Values.airflow.web.resources | nindent 12 }}
{{- end }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
configMap:
name: {{ template "airflow.fullname" . }}-dag-files
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.airflow.dagsConfigMap }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
emptyDir: {}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
emptyDir: {}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
emptyDir: {}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
configMap:
name: {{ .Values.airflow.configurationConfigMap }}
{{- end }}
{{- if .Values.airflow.webserverConfigMap }}
- name: custom-webserver-configuration-file
configMap:
name: {{ .Values.airflow.webserverConfigMap }}
{{- end }}
{{- if .Values.ldap.tls.enabled }}
- name: airflow-ldap-ca-certificate
secret:
secretName: {{ required "A secret containing the LDAP CA certificate. It is required when SSL in enabled" .Values.ldap.tls.CAcertificateSecret }}
defaultMode: 256
{{- end }}
{{- if .Values.airflow.web.extraVolumes }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.web.extraVolumes "context" $) | nindent 8 }}
{{- end }}
bitnami/airflow/templates/extradeploy.yaml
+4
View File
@@ -0,0 +1,4 @@
{{- range .Values.extraDeploy }}
---
{{ include "common.tplvalues.render" (dict "value" . "context" $) }}
{{- end }}
bitnami/airflow/templates/metrics-deployment.yaml
-74
View File
@@ -1,74 +0,0 @@
{{- if .Values.metrics.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "airflow.fullname" . }}-exporter
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: metrics
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
annotations: {{- toYaml .Values.metrics.podAnnotations | nindent 8 }}
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
containers:
- image: {{ template "airflow.metrics.image" . }}
name: airflow-exporter
env:
- name: AIRFLOW_PROMETHEUS_DATABASE_BACKEND
value: "postgres"
- name: AIRFLOW_PROMETHEUS_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_PROMETHEUS_DATABASE_PORT
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: AIRFLOW_PROMETHEUS_DATABASE_USER
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_PROMETHEUS_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "airflow.postgresql.secretName" . }}
key: postgresql-password
- name: AIRFLOW_PROMETHEUS_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
ports:
- name: metrics
containerPort: 9112
{{- if .Values.metrics.resources}}
resources: {{- toYaml .Values.metrics.resources | nindent 12 }}
{{- end }}
{{- end }}
bitnami/airflow/templates/metrics-svc.yaml
-28
View File
@@ -1,28 +0,0 @@
{{- if .Values.metrics.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "airflow.fullname" . }}-exporter
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: metrics
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
type: ClusterIP
ports:
- name: metrics
port: 9112
targetPort: metrics
selector:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: metrics
{{- end }}
bitnami/airflow/templates/metrics/deployment.yaml
+60
View File
@@ -0,0 +1,60 @@
{{- if .Values.metrics.enabled }}
apiVersion: {{ include "common.capabilities.deployment.apiVersion" . }}
kind: Deployment
metadata:
name: {{ include "common.names.fullname" . }}-exporter
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/component: metrics
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
replicas: 1
selector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: metrics
template:
metadata:
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.metrics.podAnnotations "context" $) | nindent 8 }}
labels: {{- include "common.labels.standard" . | nindent 8 }}
app.kubernetes.io/component: metrics
{{- if .Values.metrics.podLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.metrics.podLabels "context" $) | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
{{- if .Values.podSecurityContext.enabled }}
securityContext: {{- omit .Values.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
containers:
- image: {{ include "airflow.metrics.image" . | quote }}
name: airflow-exporter
{{- if .Values.containerSecurityContext.enabled }}
securityContext: {{- omit .Values.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
env:
- name: AIRFLOW_PROMETHEUS_DATABASE_BACKEND
value: "postgres"
- name: AIRFLOW_PROMETHEUS_DATABASE_HOST
value: {{ include "airflow.database.host" . }}
- name: AIRFLOW_PROMETHEUS_DATABASE_PORT
value: {{ include "airflow.database.port" . }}
- name: AIRFLOW_PROMETHEUS_DATABASE_USER
value: {{ include "airflow.database.user" . }}
- name: AIRFLOW_PROMETHEUS_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "airflow.postgresql.secretName" . }}
key: postgresql-password
- name: AIRFLOW_PROMETHEUS_DATABASE_NAME
value: {{ include "airflow.database.name" . }}
ports:
- name: metrics
containerPort: 9112
{{- if .Values.metrics.resources}}
resources: {{- toYaml .Values.metrics.resources | nindent 12 }}
{{- end }}
{{- end }}
bitnami/airflow/templates/metrics/service.yaml
+22
View File
@@ -0,0 +1,22 @@
{{- if .Values.metrics.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "common.names.fullname" . }}-exporter
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/component: metrics
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
type: ClusterIP
ports:
- name: metrics
port: 9112
targetPort: metrics
selector: {{- include "common.labels.matchLabels" . | nindent 4 }}
app.kubernetes.io/component: metrics
{{- end }}
bitnami/airflow/templates/rbac/role.yaml
+39
View File
@@ -0,0 +1,39 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: {{ include "common.names.fullname" . }}
namespace: {{ .Release.Namespace }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
rules:
- apiGroups:
- ""
resources:
- "pods"
verbs:
- "create"
- "list"
- "get"
- "watch"
- "delete"
- apiGroups:
- ""
resources:
- "pods/log"
verbs:
- "get"
- apiGroups:
- ""
resources:
- "pods/exec"
verbs:
- "create"
- "get"
{{- end -}}
bitnami/airflow/templates/rbac/rolebinding.yaml
+22
View File
@@ -0,0 +1,22 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: {{ include "common.names.fullname" . }}
namespace: {{ .Release.Namespace }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ include "common.names.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ include "airflow.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- end -}}
bitnami/airflow/templates/rbac/serviceaccount.yaml
+20
View File
@@ -0,0 +1,20 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "airflow.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if or .Values.serviceAccount.annotations .Values.commonAnnotations }}
annotations:
{{- if .Values.commonAnnotations }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.serviceAccount.annotations }}
{{- include "common.tplvalues.render" (dict "value" .Values.serviceAccount.annotations "context" $) | nindent 4 }}
{{- end }}
{{- end }}
{{- end -}}
bitnami/airflow/templates/role.yaml
-30
View File
@@ -1,30 +0,0 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: {{ template "airflow.fullname" . }}
namespace: {{ .Release.Namespace }}
labels: {{- include "airflow.labels" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
rules:
- apiGroups:
- ""
resources:
- pods
- pods/log
- pods/exec
- configmaps
- secrets
verbs:
- get
- list
- watch
- create
- delete
{{- end -}}
bitnami/airflow/templates/rolebinding.yaml
-22
View File
@@ -1,22 +0,0 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: {{ template "airflow.fullname" . }}
namespace: {{ .Release.Namespace }}
labels: {{- include "airflow.labels" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ template "airflow.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ template "airflow.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- end -}}
bitnami/airflow/templates/scheduler/deployment.yaml
+157
View File
@@ -0,0 +1,157 @@
{{- $kube := eq .Values.executor "KubernetesExecutor" -}}
apiVersion: {{ include "common.capabilities.deployment.apiVersion" . }}
kind: Deployment
metadata:
name: {{ include "common.names.fullname" . }}-scheduler
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: scheduler
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | trim | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: scheduler
replicas: {{ default 1 .Values.scheduler.replicaCount }}
template:
metadata:
labels: {{- include "common.labels.standard" . | nindent 8 }}
app.kubernetes.io/component: scheduler
{{- if .Values.scheduler.podLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.podLabels "context" $) | nindent 8 }}
{{- end }}
annotations:
checksum/configmap: {{ include (print $.Template.BasePath "/config/configmap.yaml") . | sha256sum }}
{{- if .Values.scheduler.podAnnotations }}
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.podAnnotations "context" $) | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
priorityClassName: {{ .Values.priorityClassName | quote }}
{{- if .Values.podSecurityContext.enabled }}
securityContext: {{- omit .Values.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "airflow.serviceAccountName" . }}
{{- if .Values.affinity }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "scheduler" "context" $) | nindent 10 }}
podAntiAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAntiAffinityPreset "component" "scheduler" "context" $) | nindent 10 }}
nodeAffinity: {{- include "common.affinities.nodes" (dict "type" .Values.nodeAffinityPreset.type "key" .Values.nodeAffinityPreset.key "values" .Values.nodeAffinityPreset.values) | nindent 10 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector: {{- include "common.tplvalues.render" (dict "value" .Values.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
initContainers: {{- include "airflow.git.containers.clone" . | trim | nindent 8 }}
{{- if .Values.scheduler.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
containers: {{- include "airflow.git.containers.sync" . | trim | nindent 8 }}
- name: airflow-scheduler
image: {{ include "airflow.schedulerImage" . | quote }}
imagePullPolicy: {{ .Values.scheduler.image.pullPolicy | quote }}
{{- if .Values.containerSecurityContext.enabled }}
securityContext: {{- omit .Values.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
{{- if .Values.scheduler.command }}
command: {{- include "common.tplvalues.render" (dict "value" .Values.scheduler.command "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.scheduler.args }}
args: {{- include "common.tplvalues.render" (dict "value" .Values.scheduler.args "context" $) | nindent 12 }}
{{- end }}
env:
{{- include "airflow.configure.airflow.common" . | nindent 12 }}
{{- include "airflow.configure.database" . | nindent 12 }}
{{- include "airflow.configure.redis" . | nindent 12 }}
{{- include "airflow.configure.airflow.kubernetesExecutor" . | nindent 12 }}
{{- if .Values.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.scheduler.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.extraEnvVarsCM }}
{{- end }}
{{- if .Values.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.scheduler.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.scheduler.extraEnvVarsCM }}
{{- end }}
{{- if .Values.scheduler.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.scheduler.extraEnvVarsSecret }}
{{- end }}
ports:
- name: task-logs
containerPort: {{ .Values.worker.port }}
{{- if .Values.scheduler.customLivenessProbe }}
livenessProbe:
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.customLivenessProbe "context" $) | trim | nindent 12 }}
{{- end }}
{{- if .Values.scheduler.customRedinessProbe }}
readinessProbe:
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.customRedinessProbe "context" $) | trim | nindent 12 }}
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if include "airflow.shouldUseConfigFromConfigMap" . }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
{{- if $kube }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/pod_template.yaml
subPath: pod_template.yaml
{{- end }}
{{- include "airflow.git.maincontainer.volumeMounts" . | trim | nindent 12 }}
{{- if .Values.scheduler.extraVolumeMounts }}
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.scheduler.resources }}
resources: {{- toYaml .Values.scheduler.resources | nindent 12 }}
{{- end }}
{{- if .Values.scheduler.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
volumes:
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.dagsConfigMap }}
{{- end }}
{{- if or (include "airflow.shouldUseConfigFromConfigMap" .) $kube }}
- name: custom-configuration-file
configMap:
name: {{ include "airflow.configMapName" . }}
{{- end }}
{{- include "airflow.git.volumes" . | trim | nindent 8 }}
{{- if .Values.scheduler.extraVolumes }}
{{- include "common.tplvalues.render" (dict "value" .Values.scheduler.extraVolumes "context" $) | nindent 8 }}
{{- end }}
bitnami/airflow/templates/scheduler/networkpolicy.yaml
+27
View File
@@ -0,0 +1,27 @@
{{- if .Values.networkPolicies.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "common.names.fullname" . }}-scheduler
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: scheduler
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
spec:
podSelector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: scheduler
policyTypes:
- Ingress
{{- if (or (eq .Values.executor "LocalExecutor") (eq .Values.executor "SequentialExecutor")) }}
ingress:
- from:
- podSelector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 14 }}
app.kubernetes.io/component: web
ports:
- protocol: TCP
port: task-logs
{{- end }}
{{- end }}
bitnami/airflow/templates/scheduler/poddisruptionbudget.yaml
+16
View File
@@ -0,0 +1,16 @@
{{- if and .Values.scheduler.podDisruptionBudget.enabled }}
kind: PodDisruptionBudget
apiVersion: policy/v1beta1
metadata:
name: {{ include "common.names.fullname" . }}-scheduler
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: scheduler
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: scheduler
{{- include "common.tplvalues.render" (dict "value" (omit .Values.scheduler.podDisruptionBudget "enabled") "context" $) | nindent 2 }}
{{- end }}
bitnami/airflow/templates/scheduler/service-headless.yaml
+21
View File
@@ -0,0 +1,21 @@
{{- if (or (eq .Values.executor "LocalExecutor") (eq .Values.executor "SequentialExecutor")) }}
kind: Service
apiVersion: v1
metadata:
name: {{ include "common.names.fullname" . }}-scheduler-headless
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
clusterIP: None
ports:
- name: task-logs
port: {{ .Values.worker.port }}
targetPort: task-logs
selector: {{- include "common.labels.matchLabels" . | nindent 4 }}
app.kubernetes.io/component: scheduler
{{- end }}
bitnami/airflow/templates/secret-externaldb.yaml
-20
View File
@@ -1,20 +0,0 @@
{{- if and (not .Values.postgresql.enabled) (not .Values.externalDatabase.existingSecret) (not .Values.postgresql.existingSecret)}}
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
type: Opaque
data:
postgresql-password: {{ .Values.externalDatabase.password | b64enc | quote }}
{{- end }}
bitnami/airflow/templates/secret-externalredis.yaml
-20
View File
@@ -1,20 +0,0 @@
{{- if and (not .Values.redis.enabled) (not .Values.externalRedis.existingSecret) (not .Values.redis.existingSecret)}}
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
type: Opaque
data:
redis-password: {{ .Values.externalRedis.password | b64enc | quote }}
{{- end }}
bitnami/airflow/templates/secrets.yaml
-33
View File
@@ -1,33 +0,0 @@
{{- if (not .Values.airflow.auth.existingSecret) -}}
apiVersion: v1
kind: Secret
metadata:
name: {{ template "airflow.fullname" . }}
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
type: Opaque
data:
{{- if .Values.airflow.auth.password }}
airflow-password: {{ .Values.airflow.auth.password | b64enc | quote }}
{{ else if (not .Values.airflow.auth.forcePassword) }}
airflow-password: {{ randAlphaNum 10 | b64enc | quote }}
{{ else }}
airflow-password: {{ required "An Airflow Password is required!" .Values.airflow.auth.password }}
{{- end }}
{{- if .Values.airflow.auth.fernetKey }}
airflow-fernetKey: {{ .Values.airflow.auth.fernetKey | b64enc | quote }}
{{ else if (not .Values.airflow.auth.forcePassword) }}
airflow-fernetKey: {{ randAlphaNum 32 | b64enc | b64enc | quote }}
{{ else }}
airflow-fernetKey: {{ required "An Airflow Fernet Key is required!" .Values.airflow.auth.fernetKey }}
{{- end }}
{{- end }}
bitnami/airflow/templates/serviceaccount.yaml
-21
View File
@@ -1,21 +0,0 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ template "airflow.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "airflow.labels" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if or .Values.serviceAccount.annotations .Values.commonAnnotations }}
annotations:
{{- if .Values.commonAnnotations }}
{{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.serviceAccount.annotations }}
{{- include "airflow.tplValue" (dict "value" .Values.serviceAccount.annotations "context" $) | nindent 4 }}
{{- end }}
{{- end }}
{{- end -}}
bitnami/airflow/templates/statefulset-worker.yaml
-388
View File
@@ -1,388 +0,0 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ template "airflow.fullname" . }}-worker
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: worker
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
{{- if .Values.airflow.worker.podManagementPolicy }}
podManagementPolicy: {{ .Values.airflow.worker.podManagementPolicy | quote }}
{{- end }}
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: worker
serviceName: {{ template "airflow.fullname" . }}-headless
replicas: {{ .Values.airflow.worker.replicas }}
updateStrategy:
type: {{ .Values.updateStrategy | quote }}
{{- if (eq "Recreate" .Values.updateStrategy) }}
rollingUpdate: null
{{- else if .Values.rollingUpdatePartition }}
rollingUpdate:
partition: {{ .Values.rollingUpdatePartition }}
{{- end }}
template:
metadata:
annotations: {{- include "airflow.tplValue" (dict "value" .Values.airflow.worker.podAnnotations "context" $) | nindent 8 }}
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: worker
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
{{- if .Values.securityContext.enabled }}
securityContext:
fsGroup: {{ .Values.securityContext.fsGroup }}
runAsUser: {{ .Values.securityContext.runAsUser }}
{{- end }}
serviceAccountName: {{ template "airflow.serviceAccountName" . }}
{{- if .Values.affinity }}
affinity: {{- include "airflow.tplValue" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "worker" "context" $) | nindent 10 }}
podAntiAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAntiAffinityPreset "component" "worker" "context" $) | nindent 10 }}
nodeAffinity: {{- include "common.affinities.nodes" (dict "type" .Values.nodeAffinityPreset.type "key" .Values.nodeAffinityPreset.key "values" .Values.nodeAffinityPreset.values) | nindent 10 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector: {{- include "airflow.tplValue" (dict "value" .Values.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations: {{- include "airflow.tplValue" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
{{- if or .Values.airflow.cloneDagFilesFromGit.enabled .Values.airflow.clonePluginsFromGit.enabled }}
initContainers:
- name: git-clone-repository
image: {{ include "git.image" . }}
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
{{- if .Values.airflow.gitCloneContainer.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.command "context" $) | nindent 12 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} --branch {{ .Values.airflow.cloneDagFilesFromGit.branch }} /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
git clone {{ .repository }} --branch {{ .branch }} /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
git clone {{ .Values.airflow.clonePluginsFromGit.repository }} --branch {{ .Values.airflow.clonePluginsFromGit.branch }} /plugins
{{- end }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.args }}
args: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.args "context" $) | nindent 12 }}
{{- end }}
volumeMounts:
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /plugins
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraEnvVars }}
env: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitCloneContainer.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if or .Values.airflow.gitCloneContainer.extraEnvVarsCM .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
envFrom:
{{- if .Values.airflow.gitCloneContainer.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.gitCloneContainer.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.gitCloneContainer.extraEnvVarsSecret }}
{{- end }}
{{- end }}
containers:
- name: git-repo-syncer
image: "{{ template "git.image" . }}"
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
{{- if .Values.airflow.gitSyncContainer.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.command "context" $) | nindent 12 }}
{{- else }}
command:
- /bin/bash
- -ec
- |
while true; do
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
cd /dags && git pull origin {{ .Values.airflow.cloneDagFilesFromGit.branch }}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
cd /dags-{{ .name | kebabcase }} && git pull origin {{ .branch }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
cd /plugins && git pull origin {{ .Values.airflow.clonePluginsFromGit.branch }}
{{- end }}
sleep {{ default "60" .Values.airflow.gitSyncInterval }}
done
{{- end }}
{{- if .Values.airflow.gitSyncContainer.args }}
args: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.args "context" $) | nindent 12 }}
{{- end }}
volumeMounts:
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /dags-{{ .name | kebabcase }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /plugins
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraEnvVars }}
env: {{- include "airflow.tplValue" (dict "value" .Values.airflow.gitSyncContainer.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
{{- if or .Values.airflow.gitSyncContainer.extraEnvVarsCM .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
envFrom:
{{- if .Values.airflow.gitSyncContainer.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.gitSyncContainer.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.gitSyncContainer.extraEnvVarsSecret }}
{{- end }}
{{- end }}
{{- else }}
containers:
{{- end }}
- name: airflow-worker
image: {{ include "airflow.workerImage" . }}
imagePullPolicy: {{ .Values.workerImage.pullPolicy | quote }}
{{- if .Values.airflow.worker.command }}
command: {{- include "airflow.tplValue" (dict "value" .Values.airflow.worker.command "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.worker.args }}
args: {{- include "airflow.tplValue" (dict "value" .Values.airflow.worker.args "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.airflow.worker.resources }}
resources: {{- toYaml .Values.airflow.worker.resources | nindent 12 }}
{{- end }}
env:
{{- if .Values.workerImage.debug }}
- name: BASH_DEBUG
value: "1"
- name: NAMI_DEBUG
value: "1"
- name: NAMI_LOG_LEVEL
value: "trace8"
{{- end }}
- name: AIRFLOW_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "airflow.postgresql.secretName" . }}
key: postgresql-password
- name: AIRFLOW_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: REDIS_HOST
{{- if .Values.redis.enabled }}
value: {{ template "airflow.redis.fullname" . }}
{{- else }}
value: {{ .Values.externalRedis.host | quote }}
{{- end }}
- name: REDIS_PORT_NUMBER
{{- if .Values.redis.enabled }}
value: "6379"
{{- else }}
value: {{ .Values.externalRedis.port | quote }}
{{- end }}
{{- if and (not .Values.redis.enabled) .Values.externalRedis.username }}
- name: REDIS_USER
value: {{ .Values.externalRedis.username | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
- name: AIRFLOW_EXECUTOR
value: "CeleryExecutor"
- name: AIRFLOW_FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-fernetKey
- name: AIRFLOW_WEBSERVER_HOST
value: {{ template "airflow.fullname" . }}
- name: AIRFLOW_WEBSERVER_PORT_NUMBER
value: {{ .Values.service.port | quote }}
{{- if .Values.airflow.extraEnvVars }}
{{ toYaml .Values.airflow.extraEnvVars | nindent 12 }}
{{- end }}
{{- if .Values.airflow.worker.extraEnvVars }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.worker.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.airflow.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.airflow.worker.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.airflow.worker.extraEnvVarsCM }}
{{- end }}
{{- if .Values.airflow.worker.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.airflow.worker.extraEnvVarsSecret }}
{{- end }}
ports:
- name: worker
containerPort: {{ .Values.airflow.worker.port }}
{{- if .Values.livenessProbe.enabled }}
livenessProbe:
tcpSocket:
port: worker
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
{{- end }}
{{- if .Values.readinessProbe.enabled }}
readinessProbe:
tcpSocket:
port: worker
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
mountPath: /opt/bitnami/airflow/dags/git
{{- if .Values.airflow.cloneDagFilesFromGit.path }}
subPath: {{ .Values.airflow.cloneDagFilesFromGit.path }}
{{- end }}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
mountPath: /opt/bitnami/airflow/dags/git-{{ .name | kebabcase }}
{{- if .path }}
subPath: {{ .path }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
mountPath: /opt/bitnami/airflow/plugins
{{- if .Values.airflow.clonePluginsFromGit.path }}
subPath: {{ .Values.airflow.clonePluginsFromGit.path }}
{{- end }}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
{{- if .Values.airflow.worker.extraVolumeMounts }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.worker.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
configMap:
name: {{ template "airflow.fullname" . }}-dag-files
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.airflow.dagsConfigMap }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
{{- if and .Values.airflow.cloneDagFilesFromGit.repository .Values.airflow.cloneDagFilesFromGit.branch }}
- name: git-cloned-dag-files
emptyDir: {}
{{- end }}
{{- range .Values.airflow.cloneDagFilesFromGit.repositories }}
- name: git-cloned-dag-files-{{ .name | kebabcase }}
emptyDir: {}
{{- end }}
{{- end }}
{{- if .Values.airflow.clonePluginsFromGit.enabled }}
- name: git-cloned-plugins
emptyDir: {}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
configMap:
name: {{ .Values.airflow.configurationConfigMap }}
{{- end }}
{{- if .Values.airflow.worker.extraVolumes }}
{{- include "airflow.tplValue" (dict "value" .Values.airflow.worker.extraVolumes "context" $) | nindent 6 }}
{{- end }}
bitnami/airflow/templates/svc-headless-worker.yaml
-26
View File
@@ -1,26 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "airflow.fullname" . }}-headless
labels:
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
type: ClusterIP
clusterIP: None
ports:
- name: worker
port: 8793
targetPort: worker
selector:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: worker
bitnami/airflow/templates/web/deployment.yaml
+231
View File
@@ -0,0 +1,231 @@
apiVersion: {{ include "common.capabilities.deployment.apiVersion" . }}
kind: Deployment
metadata:
name: {{ include "common.names.fullname" . }}-web
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: web
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: web
replicas: {{ default 1 .Values.web.replicaCount }}
template:
metadata:
labels: {{- include "common.labels.standard" . | nindent 8 }}
app.kubernetes.io/component: web
{{- if .Values.web.podLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.web.podLabels "context" $) | nindent 8 }}
{{- end }}
annotations:
checksum/configmap: {{ include (print $.Template.BasePath "/config/configmap.yaml") . | sha256sum }}
{{- if .Values.web.podAnnotations }}
{{- include "common.tplvalues.render" (dict "value" .Values.web.podAnnotations "context" $) | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
priorityClassName: {{ .Values.priorityClassName | quote }}
{{- if .Values.podSecurityContext.enabled }}
securityContext: {{- omit .Values.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "airflow.serviceAccountName" . }}
{{- if .Values.affinity }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "web" "context" $) | nindent 10 }}
podAntiAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAntiAffinityPreset "component" "web" "context" $) | nindent 10 }}
nodeAffinity: {{- include "common.affinities.nodes" (dict "type" .Values.nodeAffinityPreset.type "key" .Values.nodeAffinityPreset.key "values" .Values.nodeAffinityPreset.values) | nindent 10 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector: {{- include "common.tplvalues.render" (dict "value" .Values.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
initContainers: {{- include "airflow.git.containers.clone" . | trim | nindent 8 }}
{{- if .Values.web.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.web.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
containers: {{- include "airflow.git.containers.sync" . | trim | nindent 8 }}
- name: airflow-web
image: {{ include "airflow.image" . }}
imagePullPolicy: {{ .Values.web.image.pullPolicy | quote }}
{{- if .Values.containerSecurityContext.enabled }}
securityContext: {{- omit .Values.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
{{- if .Values.web.command }}
command: {{- include "common.tplvalues.render" (dict "value" .Values.web.command "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.web.args }}
args: {{- include "common.tplvalues.render" (dict "value" .Values.web.args "context" $) | nindent 12 }}
{{- end }}
env:
{{- include "airflow.configure.airflow.common" . | nindent 12 }}
{{- include "airflow.configure.database" . | nindent 12 }}
{{- include "airflow.configure.redis" . | nindent 12 }}
{{- include "airflow.configure.airflow.kubernetesExecutor" . | nindent 12 }}
- name: AIRFLOW_USERNAME
value: {{ .Values.auth.username }}
- name: AIRFLOW_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "airflow.secretName" . }}
key: airflow-password
{{- if .Values.web.baseUrl }}
- name: AIRFLOW_BASE_URL
value: {{ .Values.web.baseUrl }}
{{- end }}
- name: AIRFLOW_LDAP_ENABLE
value: {{ ternary "yes" "no" .Values.ldap.enabled | quote }}
{{- if .Values.ldap.enabled }}
- name: AIRFLOW_LDAP_URI
value: {{ .Values.ldap.uri }}
- name: AIRFLOW_LDAP_SEARCH
value: {{ .Values.ldap.base }}
- name: AIRFLOW_LDAP_BIND_USER
value: {{ .Values.ldap.binddn }}
- name: AIRFLOW_LDAP_BIND_PASSWORD
value: {{ .Values.ldap.bindpw }}
valueFrom:
secretKeyRef:
name: {{ include "airflow.ldapSecretName" . }}
key: bind-password
- name: AIRFLOW_LDAP_UID_FIELD
value: {{ .Values.ldap.uidField }}
- name: AIRFLOW_LDAP_USE_TLS
value: {{ ternary "yes" "no" .Values.ldap.tls.enabled | quote }}
{{- if .Values.ldap.tls.enabled }}
- name: AIRFLOW_LDAP_ALLOW_SELF_SIGNED
value: {{ .Values.ldap.tls.allowSelfSigned }}
- name: AIRFLOW_LDAP_TLS_CA_CERTIFICATE
value: {{ include "airflow.ldapCAFilename" . | quote }}
{{- end }}
{{- end }}
{{- if .Values.extraEnvVars }}
{{- toYaml .Values.extraEnvVars | nindent 12 }}
{{- end }}
{{- if .Values.web.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.web.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.extraEnvVarsCM }}
{{- end }}
{{- if .Values.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.web.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.web.extraEnvVarsCM }}
{{- end }}
{{- if .Values.web.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.web.extraEnvVarsSecret }}
{{- end }}
ports:
- name: http
containerPort: {{ .Values.web.containerPort }}
{{- if .Values.web.livenessProbe.enabled }}
livenessProbe:
{{- if .Values.web.baseUrl }}
tcpSocket:
port: http
{{- else }}
httpGet:
path: /health
port: http
{{- end }}
{{ omit .Values.web.livenessProbe "enabled" | toYaml | nindent 12 }}
{{- else if .Values.web.customLivenessProbe }}
livenessProbe:
{{- include "common.tplvalues.render" (dict "value" .Values.web.customLivenessProbe "context" $) | trim | nindent 12 }}
{{- end }}
{{- if .Values.web.readinessProbe.enabled }}
readinessProbe:
{{- if .Values.web.baseUrl }}
tcpSocket:
port: http
{{- else }}
httpGet:
path: /health
port: http
{{- end }}
{{ omit .Values.web.readinessProbe "enabled" | toYaml | nindent 12 }}
{{- else if .Values.web.customRedinessProbe }}
readinessProbe:
{{- include "common.tplvalues.render" (dict "value" .Values.web.customRedinessProbe "context" $) | trim | nindent 12 }}
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if include "airflow.shouldUseConfigFromConfigMap" . }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
{{- if .Values.web.configMap }}
- name: custom-webserver-configuration-file
mountPath: /opt/bitnami/airflow/webserver_config.py
subPath: webserver_config.py
{{- end }}
{{- if .Values.ldap.tls.enabled }}
- name: airflow-ldap-ca-certificate
mountPath: /opt/bitnami/airflow/conf/certs/
readOnly: true
{{- end }}
{{- include "airflow.git.maincontainer.volumeMounts" . | trim | nindent 12 }}
{{- if .Values.web.extraVolumeMounts }}
{{- include "common.tplvalues.render" (dict "value" .Values.web.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.web.resources }}
resources: {{- toYaml .Values.web.resources | nindent 12 }}
{{- end }}
{{- if .Values.web.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.web.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
volumes:
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.dagsConfigMap }}
{{- end }}
{{- if include "airflow.shouldUseConfigFromConfigMap" . }}
- name: custom-configuration-file
configMap:
name: {{ include "airflow.configMapName" . }}
{{- end }}
{{- if .Values.web.configMap }}
- name: custom-webserver-configuration-file
configMap:
name: {{ .Values.web.configMap }}
{{- end }}
{{- if .Values.ldap.tls.enabled }}
- name: airflow-ldap-ca-certificate
secret:
secretName: {{ required "A secret containing the LDAP CA certificate. It is required when SSL in enabled" .Values.ldap.tls.CAcertificateSecret }}
defaultMode: 256
{{- end }}
{{- include "airflow.git.volumes" . | trim | nindent 8 }}
{{- if .Values.web.extraVolumes }}
{{- include "common.tplvalues.render" (dict "value" .Values.web.extraVolumes "context" $) | nindent 8 }}
{{- end }}
bitnami/airflow/templates/ingress.yaml → bitnami/airflow/templates/web/ingress.yaml
+7 -11
View File
@@ -1,25 +1,21 @@
{{- if .Values.ingress.enabled }}
apiVersion: extensions/v1beta1
apiVersion: {{ include "common.capabilities.ingress.apiVersion" . }}
kind: Ingress
metadata:
name: {{ template "airflow.fullname" . }}
labels:
name: {{ include "common.names.fullname" . }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
annotations:
{{- if .Values.commonAnnotations }}
{{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.ingress.certManager }}
kubernetes.io/tls-acme: "true"
{{- end }}
{{- range $key, $value := .Values.ingress.annotations }}
{{ $key }}: {{ $value | quote }}
{{ $key }}: {{ include "common.tplvalues.render" (dict "value" $value "context" $) | quote }}
{{- end }}
spec:
rules:
@@ -29,7 +25,7 @@ spec:
paths:
- path: {{ default "/" .path }}
backend:
serviceName: {{ template "airflow.fullname" $ }}
serviceName: {{ include "common.names.fullname" $ }}
servicePort: http
{{- end }}
tls:
bitnami/airflow/templates/web/poddisruptionbudget.yaml
+16
View File
@@ -0,0 +1,16 @@
{{- if and .Values.web.podDisruptionBudget.enabled (eq .Values.executor "CeleryExecutor") }}
kind: PodDisruptionBudget
apiVersion: policy/v1beta1
metadata:
name: {{ include "common.names.fullname" . }}-web
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: web
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: web
{{- include "common.tplvalues.render" (dict "value" (omit .Values.web.podDisruptionBudget "enabled") "context" $) | nindent 2 }}
{{- end }}
bitnami/airflow/templates/svc.yaml → bitnami/airflow/templates/web/service.yaml
+8 -12
View File
@@ -1,21 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "airflow.fullname" . }}
labels:
name: {{ include "common.names.fullname" . }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "airflow.tplValue" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
annotations:
{{- if .Values.commonAnnotations }}
{{- include "airflow.tplValue" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
{{- range $key, $value := .Values.service.annotations }}
{{ $key }}: {{ $value | quote }}
{{ $key }}: {{ include "common.tplvalues.render" (dict "value" $value "context" $) | quote }}
{{- end }}
spec:
type: {{ .Values.service.type }}
@@ -29,9 +25,9 @@ spec:
port: {{ .Values.service.port }}
{{- if and .Values.service.nodePort (or (eq .Values.service.type "NodePort") (eq .Values.service.type "LoadBalancer")) }}
nodePort: {{ .Values.service.nodePort }}
{{- else }}
nodePort: null
{{- end }}
targetPort: http
selector:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
selector: {{- include "common.labels.matchLabels" . | nindent 4 }}
app.kubernetes.io/component: web
bitnami/airflow/templates/worker/horizontalpodautoscaler.yaml
+27
View File
@@ -0,0 +1,27 @@
{{- if (and .Values.worker.autoscaling.enabled (eq .Values.executor "CeleryExecutor")) }}
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "common.names.fullname" . }}-worker
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: worker
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: {{ include "common.names.fullname" . }}-worker
minReplicas: {{ .Values.worker.autoscaling.replicas.min }}
maxReplicas: {{ .Values.worker.autoscaling.replicas.max }}
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: {{ .Values.worker.autoscaling.targets.cpu }}
- type: Resource
resource:
name: memory
targetAverageUtilization: {{ .Values.worker.autoscaling.targets.memory }}
{{- end }}
bitnami/airflow/templates/worker/networkpolicy.yaml
+25
View File
@@ -0,0 +1,25 @@
{{- if (and .Values.networkPolicies.enabled (eq .Values.executor "CeleryExecutor")) }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "common.names.fullname" . }}-worker
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: worker
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
spec:
podSelector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: worker
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 14 }}
app.kubernetes.io/component: web
ports:
- protocol: TCP
port: worker
{{- end }}
bitnami/airflow/templates/worker/poddisruptionbudget.yaml
+16
View File
@@ -0,0 +1,16 @@
{{- if and .Values.worker.podDisruptionBudget.enabled (eq .Values.executor "CeleryExecutor") }}
kind: PodDisruptionBudget
apiVersion: policy/v1beta1
metadata:
name: {{ include "common.names.fullname" . }}-worker
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: worker
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: worker
{{- include "common.tplvalues.render" (dict "value" (omit .Values.worker.podDisruptionBudget "enabled") "context" $) | nindent 2 }}
{{- end }}
bitnami/airflow/templates/worker/service-headless.yaml
+20
View File
@@ -0,0 +1,20 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "common.names.fullname" . }}-worker-headless
labels: {{- include "common.labels.standard" . | nindent 4 }}
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
type: ClusterIP
clusterIP: None
ports:
- name: worker
port: {{ .Values.worker.port }}
targetPort: worker
selector: {{- include "common.labels.matchLabels" . | nindent 4 }}
app.kubernetes.io/component: worker
bitnami/airflow/templates/worker/statefulset.yaml
+174
View File
@@ -0,0 +1,174 @@
{{- if eq .Values.executor "CeleryExecutor" }}
apiVersion: {{ include "common.capabilities.statefulset.apiVersion" . }}
kind: StatefulSet
metadata:
name: {{ include "common.names.fullname" . }}-worker
labels: {{- include "common.labels.standard" . | nindent 4 }}
app.kubernetes.io/component: worker
{{- if .Values.commonLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.commonLabels "context" $) | nindent 4 }}
{{- end }}
{{- if .Values.commonAnnotations }}
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
{{- end }}
spec:
{{- if .Values.worker.podManagementPolicy }}
podManagementPolicy: {{ .Values.worker.podManagementPolicy | quote }}
{{- end }}
selector:
matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
app.kubernetes.io/component: worker
serviceName: {{ include "common.names.fullname" . }}-worker-headless
replicas: {{ default 1 .Values.worker.replicaCount }}
updateStrategy:
type: {{ .Values.worker.updateStrategy | quote }}
{{- if (eq "Recreate" .Values.worker.updateStrategy) }}
rollingUpdate: null
{{- else if .Values.worker.rollingUpdatePartition }}
rollingUpdate:
partition: {{ .Values.worker.rollingUpdatePartition }}
{{- end }}
template:
metadata:
annotations:
checksum/configmap: {{ include (print $.Template.BasePath "/config/configmap.yaml") . | sha256sum }}
{{- if .Values.worker.podAnnotations }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.podAnnotations "context" $) | nindent 8 }}
{{- end }}
labels: {{- include "common.labels.standard" . | nindent 8 }}
app.kubernetes.io/component: worker
{{- if .Values.worker.podLabels }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.podLabels "context" $) | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
priorityClassName: {{ .Values.priorityClassName | quote }}
{{- if .Values.podSecurityContext.enabled }}
securityContext: {{- omit .Values.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "airflow.serviceAccountName" . }}
{{- if .Values.affinity }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "worker" "context" $) | nindent 10 }}
podAntiAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAntiAffinityPreset "component" "worker" "context" $) | nindent 10 }}
nodeAffinity: {{- include "common.affinities.nodes" (dict "type" .Values.nodeAffinityPreset.type "key" .Values.nodeAffinityPreset.key "values" .Values.nodeAffinityPreset.values) | nindent 10 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector: {{- include "common.tplvalues.render" (dict "value" .Values.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
initContainers: {{- include "airflow.git.containers.clone" . | trim | nindent 8 }}
{{- if .Values.worker.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.initContainer }}
{{- include "common.tplvalues.render" (dict "value" .Values.initContainer "context" $) | trim | nindent 8 }}
{{- end }}
containers: {{- include "airflow.git.containers.sync" . | trim | nindent 8 }}
- name: airflow-worker
image: {{ include "airflow.workerImage" . }}
imagePullPolicy: {{ .Values.worker.image.pullPolicy | quote }}
{{- if .Values.containerSecurityContext.enabled }}
securityContext: {{- omit .Values.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
{{- if .Values.worker.command }}
command: {{- include "common.tplvalues.render" (dict "value" .Values.worker.command "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.worker.args }}
args: {{- include "common.tplvalues.render" (dict "value" .Values.worker.args "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.worker.resources }}
resources: {{- toYaml .Values.worker.resources | nindent 12 }}
{{- end }}
env:
{{- include "airflow.configure.airflow.common" . | nindent 12 }}
{{- include "airflow.configure.database" . | nindent 12 }}
{{- include "airflow.configure.redis" . | nindent 12 }}
{{- if .Values.extraEnvVars }}
{{ toYaml .Values.extraEnvVars | nindent 12 }}
{{- end }}
{{- if .Values.worker.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.extraEnvVars "context" $) | nindent 12 }}
{{- end }}
envFrom:
{{- if .Values.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.extraEnvVarsCM }}
{{- end }}
{{- if .Values.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.extraEnvVarsSecret }}
{{- end }}
{{- if .Values.worker.extraEnvVarsCM }}
- configMapRef:
name: {{ .Values.worker.extraEnvVarsCM }}
{{- end }}
{{- if .Values.worker.extraEnvVarsSecret }}
- secretRef:
name: {{ .Values.worker.extraEnvVarsSecret }}
{{- end }}
ports:
- name: worker
containerPort: {{ .Values.worker.port }}
{{- if .Values.worker.livenessProbe.enabled }}
livenessProbe:
tcpSocket:
port: worker
{{ omit .Values.worker.livenessProbe "enabled" | toYaml | nindent 12 }}
{{- else if .Values.worker.customLivenessProbe }}
livenessProbe:
{{- include "common.tplvalues.render" (dict "value" .Values.worker.customLivenessProbe "context" $) | trim | nindent 12 }}
{{- end }}
{{- if .Values.worker.readinessProbe.enabled }}
readinessProbe:
tcpSocket:
port: worker
{{ omit .Values.worker.readinessProbe "enabled" | toYaml | nindent 12 }}
{{- else if .Values.worker.customRedinessProbe }}
readinessProbe:
{{- include "common.tplvalues.render" (dict "value" .Values.worker.customRedinessProbe "context" $) | trim | nindent 12 }}
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if include "airflow.shouldUseConfigFromConfigMap" . }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
{{- include "airflow.git.maincontainer.volumeMounts" . | trim | nindent 12 }}
{{- if .Values.worker.extraVolumeMounts }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.extraVolumeMounts "context" $) | nindent 12 }}
{{- end }}
{{- if .Values.worker.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
{{- if .Values.sidecars }}
{{- include "common.tplvalues.render" (dict "value" .Values.sidecars "context" $) | trim | nindent 8 }}
{{- end }}
volumes:
{{- if .Values.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.dagsConfigMap }}
{{- end }}
{{- if include "airflow.shouldUseConfigFromConfigMap" . }}
- name: custom-configuration-file
configMap:
name: {{ include "airflow.configMapName" . }}
{{- end }}
{{- include "airflow.git.volumes" . | trim | nindent 8 }}
{{- if .Values.worker.extraVolumes }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.extraVolumes "context" $) | nindent 8 }}
{{- end }}
{{- end }}
bitnami/airflow/values-production.yaml
+651 -272
View File
File diff suppressed because it is too large Load Diff
bitnami/airflow/values.yaml
+587 -340
View File
File diff suppressed because it is too large Load Diff