diff --git a/bitnami/kafka/Chart.yaml b/bitnami/kafka/Chart.yaml index b11463e9f..f250379f9 100644 --- a/bitnami/kafka/Chart.yaml +++ b/bitnami/kafka/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v1 name: kafka -version: 8.0.1 +version: 9.0.0 appVersion: 2.4.1 description: Apache Kafka is a distributed streaming platform. keywords: diff --git a/bitnami/kafka/README.md b/bitnami/kafka/README.md index ec605aa50..db951a63e 100644 --- a/bitnami/kafka/README.md +++ b/bitnami/kafka/README.md @@ -5,8 +5,8 @@ ## TL;DR; ```console -$ helm repo add bitnami https://charts.bitnami.com/bitnami -$ helm install my-release bitnami/kafka +helm repo add bitnami https://charts.bitnami.com/bitnami +helm install my-release bitnami/kafka ``` ## Introduction @@ -26,8 +26,8 @@ Bitnami charts can be used with [Kubeapps](https://kubeapps.com/) for deployment To install the chart with the release name `my-release`: ```console -$ helm repo add bitnami https://charts.bitnami.com/bitnami -$ helm install my-release bitnami/kafka +helm repo add bitnami https://charts.bitnami.com/bitnami +helm install my-release bitnami/kafka ``` These commands deploy Kafka on the Kubernetes cluster in the default configuration. The [Parameters](#parameters) section lists the parameters that can be configured during installation. @@ -39,179 +39,229 @@ These commands deploy Kafka on the Kubernetes cluster in the default configurati To uninstall/delete the `my-release` deployment: ```console -$ helm delete my-release +helm delete my-release ``` The command removes all the Kubernetes components associated with the chart and deletes the release. ## Parameters -The following tables lists the configurable parameters of the Kafka chart and their default values. +The following tables lists the configurable parameters of the Kafka chart and their default values per section/component: -| Parameter | Description | Default | -|-----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------| -| `global.imageRegistry` | Global Docker image registry | `nil` | -| `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | -| `global.storageClass` | Global storage class for dynamic provisioning | `nil` | -| `image.registry` | Kafka image registry | `docker.io` | -| `image.repository` | Kafka Image name | `bitnami/kafka` | -| `image.tag` | Kafka Image tag | `{VERSION}` | -| `image.pullPolicy` | Kafka image pull policy | `IfNotPresent` | -| `image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | -| `image.debug` | Specify if debug values should be set | `false` | -| `nameOverride` | String to partially override kafka.fullname template with a string (will append the release name) | `nil` | -| `fullnameOverride` | String to fully override kafka.fullname template with a string | `nil` | -| `volumePermissions.enabled` | Enable init container that changes volume permissions in the data directory (for cases where the default k8s `runAsUser` and `fsUser` values do not work) | `false` | -| `volumePermissions.image.registry` | Init container volume-permissions image registry | `docker.io` | -| `volumePermissions.image.repository` | Init container volume-permissions image name | `bitnami/minideb` | -| `volumePermissions.image.tag` | Init container volume-permissions image tag | `buster` | -| `volumePermissions.image.pullPolicy` | Init container volume-permissions image pull policy | `Always` | -| `volumePermissions.resources` | Init container resource requests/limit | `nil` | -| `updateStrategy` | Update strategy for the stateful set | `RollingUpdate` | -| `rollingUpdatePartition` | Partition update strategy | `nil` | -| `podDisruptionBudget.maxUnavailable` | Max number of pods down simultaneously | `1` | -| `replicaCount` | Number of Kafka nodes | `1` | -| `config` | Configuration file for Kafka | `nil` | -| `allowPlaintextListener` | Allow to use the PLAINTEXT listener | `true` | -| `listeners` | The address the socket server listens on. | `nil` | -| `advertisedListeners` | Hostname and port the broker will advertise to producers and consumers. | `nil` | -| `listenerSecurityProtocolMap` | The protocol->listener mapping. | `nil` | -| `interBrokerListenerName` | The listener that the brokers should communicate on. | `nil` | -| `brokerId` | ID of the Kafka node | `-1` | -| `deleteTopicEnable` | Switch to enable topic deletion or not. | `false` | -| `heapOpts` | Kafka's Java Heap size. | `-Xmx1024m -Xms1024m` | -| `logFlushIntervalMessages` | The number of messages to accept before forcing a flush of data to disk. | `10000` | -| `logFlushIntervalMs` | The maximum amount of time a message can sit in a log before we force a flush. | `1000` | -| `logRetentionBytes` | A size-based retention policy for logs. | `_1073741824` | -| `logRetentionCheckIntervalMs` | The interval at which log segments are checked to see if they can be deleted. | `300000` | -| `logRetentionHours` | The minimum age of a log file to be eligible for deletion due to age. | `168` | -| `logSegmentBytes` | The maximum size of a log segment file. When this size is reached a new log segment will be created. | `_1073741824` | -| `logMessageFormatVersion` | Logging message format version. | `` | -| `logsDirs` | A comma separated list of directories under which to store log files. | `/bitnami/kafka/data` | -| `maxMessageBytes` | The largest record batch size allowed by Kafka. | `1000012` | -| `defaultReplicationFactor` | Default replication factors for automatically created topics | `1` | -| `offsetsTopicReplicationFactor` | The replication factor for the offsets topic | `1` | -| `transactionStateLogReplicationFactor ` | The replication factor for the transaction topic | `1` | -| `transactionStateLogMinIsr ` | Overridden min.insync.replicas config for the transaction topic | `1` | -| `numIoThreads` | The number of threads doing disk I/O. | `8` | -| `numNetworkThreads` | The number of threads handling network requests. | `3` | -| `numPartitions` | The default number of log partitions per topic. | `1` | -| `numRecoveryThreadsPerDataDir` | The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. | `1` | -| `socketReceiveBufferBytes` | The receive buffer (SO_RCVBUF) used by the socket server. | `102400` | -| `socketRequestMaxBytes` | The maximum size of a request that the socket server will accept (protection against OOM). | `_104857600` | -| `socketSendBufferBytes` | The send buffer (SO_SNDBUF) used by the socket server. | `102400` | -| `zookeeperConnectionTimeoutMs` | Timeout in ms for connecting to Zookeeper. | `6000` | -| `extraEnvVars` | Extra environment variables to add to kafka pods | `nil` | -| `sslEndpointIdentificationAlgorithm` | The endpoint identification algorithm to validate server hostname using server certificate. | `https` | -| `auth.enabled` | Switch to enable the kafka authentication. | `false` | -| `auth.existingSecret` | Name of the existing secret containing credentials for brokerUser, interBrokerUser and zookeeperUser. | `nil` | -| `auth.certificatesSecret` | Name of the existing secret containing the certificate files that will be used by Kafka. | `nil` | -| `auth.certificatesPassword` | Password for the above certificates if they are password protected. | `nil` | -| `auth.brokerUser` | Kafka client user. | `user` | -| `auth.brokerPassword` | Kafka client password. | `nil` | -| `auth.interBrokerUser` | Kafka inter broker communication user | `admin` | -| `auth.interBrokerPassword` | Kafka inter broker communication password. | `nil` | -| `auth.zookeeperUser` | Kafka Zookeeper user. | `nil` | -| `auth.zookeeperPassword` | Kafka Zookeeper password. | `nil` | -| `securityContext.enabled` | Enable security context | `true` | -| `securityContext.fsGroup` | Group ID for the container | `1001` | -| `securityContext.runAsUser` | User ID for the container | `1001` | -| `clusterDomain` | Kubernetes cluster domain | `cluster.local` | -| `service.type` | Kubernetes Service type | `ClusterIP` | -| `service.port` | Kafka port | `9092` | -| `service.nodePort` | Kubernetes Service nodePort | `nil` | -| `service.loadBalancerIP` | loadBalancerIP for Kafka Service | `nil` | -| `service.annotations` | Service annotations | `` | -| `externalAccess.enabled` | Enable Kubernetes external cluster access to Kafka brokers | `false` | -| `externalAccess.service.type` | Kubernetes Servive type for external access. It can be NodePort or LoadBalancer | `LoadBalancer` | -| `externalAccess.service.port` | Kafka port used for external access when service type is LoadBalancer | `19092` | -| `externalAccess.service.loadBalancerIP` | Array of load balancer IPs for Kafka brokers. | `[]` | -| `externalAccess.service.domain` | Domain or external ip used to configure Kafka external listener when service type is NodePort | `nil` | -| `externalAccess.service.nodePort` | Array of node ports used to configure Kafka external listener when service type is NodePort | `[]` | -| `externalAccess.service.annotations` | Service annotations for external access | `` | -| `serviceAccount.create` | Enable creation of ServiceAccount for kafka pod | `false` | -| `serviceAccount.name` | Name of the created serviceAccount | Generated using the `kafka.fullname` template | -| `persistence.enabled` | Enable Kafka persistence using PVC, note that Zookeeper perisstency is unaffected | `true` | -| `persistence.existingClaim` | Provide an existing `PersistentVolumeClaim`, the value is evaluated as a template. | `nil` | -| `persistence.storageClass` | PVC Storage Class for Kafka volume | `nil` | -| `persistence.accessMode` | PVC Access Mode for Kafka volume | `ReadWriteOnce` | -| `persistence.size` | PVC Storage Request for Kafka volume | `8Gi` | -| `persistence.annotations` | Annotations for the PVC | `{}` | -| `nodeSelector` | Node labels for pod assignment | `{}` | -| `tolerations` | Toleration labels for pod assignment | `[]` | -| `affinity` | Map of node/pod affinities | `{}` | -| `resources` | CPU/Memory resource requests/limits | Memory: `256Mi`, CPU: `250m` | -| `livenessProbe.enabled` | would you like a livessProbed to be enabled | `true` | -| `livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 30 | -| `livenessProbe.periodSeconds` | How often to perform the probe | 10 | -| `livenessProbe.timeoutSeconds` | When the probe times out | 5 | -| `livenessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 | -| `livenessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 | -| `readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` | -| `readinessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 5 | -| `readinessProbe.periodSeconds` | How often to perform the probe | 10 | -| `readinessProbe.timeoutSeconds` | When the probe times out | 5 | -| `readinessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 | -| `readinessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 | -| `metrics.kafka.enabled` | Whether or not to create a standalone Kafka exporter to expose Kafka metrics | `false` | -| `metrics.kafka.image.registry` | Kafka exporter image registry | `docker.io` | -| `metrics.kafka.image.repository` | Kafka exporter image name | `bitnami/kafka-exporter` | -| `metrics.kafka.image.tag` | Kafka exporter image tag | `{TAG_NAME}` | -| `metrics.kafka.image.pullPolicy` | Kafka exporter image pull policy | `IfNotPresent` | -| `metrics.kafka.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | -| `metrics.kafka.interval` | Interval that Prometheus scrapes Kafka metrics when using Prometheus Operator | `10s` | -| `metrics.kafka.port` | Kafka Exporter Port which exposes metrics in Prometheus format for scraping | `9308` | -| `metrics.kafka.resources` | Allows setting resource limits for kafka-exporter pod | `{}` | -| `metrics.kafka.annotations` | Annotations for Prometheus metrics deployment | `{}` | -| `metrics.kafka.podAnnotations` | Annotations for Prometheus metrics pods | `{}` | -| `metrics.kafka.service.type` | Kubernetes service type (`ClusterIP`, `NodePort` or `LoadBalancer`) for Kafka Exporter | `ClusterIP` | -| `metrics.kafka.service.port` | Kafka Exporter Prometheus port | `9308` | -| `metrics.kafka.service.nodePort` | Kubernetes HTTP node port | `""` | -| `metrics.kafka.service.annotations` | Annotations for Prometheus metrics service | `Check values.yaml file` | -| `metrics.kafka.service.loadBalancerIP` | loadBalancerIP if service type is `LoadBalancer` | `nil` | -| `metrics.kafka.service.clusterIP` | Static clusterIP or None for headless services | `nil` | -| `metrics.jmx.enabled` | Whether or not to expose JMX metrics to Prometheus | `false` | -| `metrics.jmx.image.registry` | JMX exporter image registry | `docker.io` | -| `metrics.jmx.image.repository` | JMX exporter image name | `bitnami/jmx-exporter` | -| `metrics.jmx.image.tag` | JMX exporter image tag | `{TAG_NAME}` | -| `metrics.jmx.image.pullPolicy` | JMX exporter image pull policy | `IfNotPresent` | -| `metrics.jmx.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | -| `metrics.jmx.interval` | Interval that Prometheus scrapes JMX metrics when using Prometheus Operator | `10s` | -| `metrics.jmx.exporterPort` | JMX Exporter Port which exposes metrics in Prometheus format for scraping | `5556` | -| `metrics.jmx.resources` | Allows setting resource limits for jmx sidecar container | `{}` | -| `metrics.jmx.service.type` | Kubernetes service type (`ClusterIP`, `NodePort` or `LoadBalancer`) for JMX Exporter | `ClusterIP` | -| `metrics.jmx.service.port` | JMX Exporter Prometheus port | `5556` | -| `metrics.jmx.service.nodePort` | Kubernetes HTTP node port | `""` | -| `metrics.jmx.service.annotations` | Annotations for Prometheus metrics service | `Check values.yaml file` | -| `metrics.jmx.service.loadBalancerIP` | loadBalancerIP if service type is `LoadBalancer` | `nil` | -| `metrics.jmx.service.clusterIP` | Static clusterIP or None for headless services | `nil` | -| `metrics.jmx.configMap.enabled` | Enable the default ConfigMap for JMX | `true` | -| `metrics.jmx.configMap.overrideConfig` | Allows config file to be generated by passing values to ConfigMap | `{}` | -| `metrics.jmx.configMap.overrideName` | Allows setting the name of the ConfigMap to be used | `""` | -| `metrics.jmx.jmxPort` | The jmx port which JMX style metrics are exposed (note: these are not scrapeable by Prometheus) | `5555` | -| `metrics.jmx.whitelistObjectNames` | Allows setting which JMX objects you want to expose to via JMX stats to JMX Exporter | (see `values.yaml`) | -| `metrics.serviceMonitor.enabled` | if `true`, creates a Prometheus Operator ServiceMonitor (also requires `metrics.kafka.enabled` or `metrics.jmx.enabled` to be `true`) | `false` | -| `metrics.serviceMonitor.namespace` | Namespace which Prometheus is running in | `monitoring` | -| `metrics.serviceMonitor.interval` | How frequently to scrape metrics (use by default, falling back to Prometheus' default) | `nil` | -| `metrics.serviceMonitor.selector` | Default to kube-prometheus install (CoreOS recommended), but should be set according to Prometheus install | `{ prometheus: kube-prometheus }` | -| `zookeeper.enabled` | Switch to enable or disable the Zookeeper helm chart | `true` | -| `zookeeper.persistence.enabled` | Enable Zookeeper persistence using PVC | `true` | -| `externalZookeeper.servers` | Server or list of external Zookeeper servers to use. | `nil` | +### Global parameters + +| Parameter | Description | Default | +|---------------------------|-------------------------------------------------|---------------------------------------------------------| +| `global.imageRegistry` | Global Docker image registry | `nil` | +| `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | +| `global.storageClass` | Global storage class for dynamic provisioning | `nil` | + +### Common parameters + +| Parameter | Description | Default | +|---------------------|----------------------------------------------|-----------------| +| `nameOverride` | String to partially override kafka.fullname | `nil` | +| `fullnameOverride` | String to fully override kafka.fullname | `nil` | +| `clusterDomain` | Default Kubernetes cluster domain | `cluster.local` | + +### Kafka parameters + +| Parameter | Description | Default | +|----------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------------------------------------| +| `image.registry` | Kafka image registry | `docker.io` | +| `image.repository` | Kafka image name | `bitnami/kafka` | +| `image.tag` | Kafka image tag | `{TAG_NAME}` | +| `image.pullPolicy` | Kafka image pull policy | `IfNotPresent` | +| `image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | +| `image.debug` | Set to true if you would like to see extra information on logs | `false` | +| `config` | Configuration file for Kafka. Auto-generated based on other parameters when not specified | `nil` | +| `existingConfigmap` | Name of existing ConfigMap with Kafka configuration | `nil` | +| `allowPlaintextListener` | Allow to use the PLAINTEXT listener | `true` | +| `listeners` | The address(es) the socket server listens on | `[]` | +| `advertisedListeners` | The address(es) (hostname:port) the broker will advertise to producers and consumers | `[]` | +| `listenerSecurityProtocolMap` | The protocol->listener mapping | `nil` | +| `interBrokerListenerName` | The listener that the brokers should communicate on | `nil` | +| `brokerId` | ID of the Kafka node | `nil` | +| `heapOpts` | Kafka's Java Heap size | `-Xmx1024m -Xms1024m` | +| `deleteTopicEnable` | Switch to enable topic deletion or not | `false` | +| `logFlushIntervalMessages` | The number of messages to accept before forcing a flush of data to disk | `10000` | +| `logFlushIntervalMs` | The maximum amount of time a message can sit in a log before we force a flush | `1000` | +| `logRetentionBytes` | A size-based retention policy for logs | `_1073741824` | +| `logRetentionCheckIntervalMs` | The interval at which log segments are checked to see if they can be deleted | `300000` | +| `logRetentionHours` | The minimum age of a log file to be eligible for deletion due to age | `168` | +| `logSegmentBytes` | The maximum size of a log segment file. When this size is reached a new log segment will be created | `_1073741824` | +| `logsDirs` | A comma separated list of directories under which to store log files | `/bitnami/kafka/data` | +| `maxMessageBytes` | The largest record batch size allowed by Kafka | `1000012` | +| `defaultReplicationFactor` | Default replication factors for automatically created topics | `1` | +| `offsetsTopicReplicationFactor` | The replication factor for the offsets topic | `1` | +| `transactionStateLogReplicationFactor` | The replication factor for the transaction topic | `1` | +| `transactionStateLogMinIsr` | Overridden min.insync.replicas config for the transaction topic | `1` | +| `numIoThreads` | The number of threads doing disk I/O | `8` | +| `numNetworkThreads` | The number of threads handling network requests | `3` | +| `numPartitions` | The default number of log partitions per topic | `1` | +| `numRecoveryThreadsPerDataDir` | The number of threads per data directory to be used for log recovery at startup and flushing at shutdown | `1` | +| `socketReceiveBufferBytes` | The receive buffer (SO_RCVBUF) used by the socket server | `102400` | +| `socketRequestMaxBytes` | The maximum size of a request that the socket server will accept (protection against OOM) | `_104857600` | +| `socketSendBufferBytes` | The send buffer (SO_SNDBUF) used by the socket server | `102400` | +| `zookeeperConnectionTimeoutMs` | Timeout in ms for connecting to Zookeeper | `6000` | +| `sslEndpointIdentificationAlgorithm` | The endpoint identification algorithm to validate server hostname using server certificate | `https` | +| `extraEnvVars` | Extra environment variables to add to kafka pods | `[]` | +| `auth.enabled` | Switch to enable the kafka authentication | `false` | +| `auth.certificatesSecret` | Name of the existing secret containing the certificate files that will be used by Kafka | `nil` | +| `auth.certificatesPassword` | Password for the above certificates if they are password protected | `nil` | +| `auth.brokerUser` | Kafka client user | `user` | +| `auth.brokerPassword` | Kafka client password | `nil` | +| `auth.interBrokerUser` | Kafka inter broker communication user | `admin` | +| `auth.interBrokerPassword` | Kafka inter broker communication password | `nil` | +| `auth.zookeeperUser` | Kafka Zookeeper user | `nil` | +| `auth.zookeeperPassword` | Kafka Zookeeper password | `nil` | +| `auth.existingSecret` | Name of the existing secret containing credentials for brokerUser, interBrokerUser and zookeeperUser | `nil` | + +### Statefulset parameters + +| Parameter | Description | Default | +|-----------------------------|--------------------------------------------------------------------|--------------------------------| +| `replicaCount` | Number of Kafka nodes | `1` | +| `updateStrategy` | Update strategy for the stateful set | `RollingUpdate` | +| `rollingUpdatePartition` | Partition update strategy | `nil` | +| `podAnnotations` | Kafka Pod annotations | `{}` (evaluated as a template) | +| `affinity` | Affinity for pod assignment | `{}` (evaluated as a template) | +| `nodeSelector` | Node labels for pod assignment | `{}` (evaluated as a template) | +| `tolerations` | Tolerations for pod assignment | `[]` (evaluated as a template) | +| `podSecurityContext` | Kafka pods' Security Context | `{}` | +| `containerSecurityContext` | Kafka containers' Security Context | `{}` | +| `resources.limits` | The resources limits for Kafka containers | `{}` | +| `resources.requests` | The requested resources for Kafka containers | `{}` | +| `livenessProbe` | Liveness probe configuration for Kafka | `Check values.yaml file` | +| `readinessProbe` | Readiness probe configuration for Kafka | `Check values.yaml file` | +| `pdb.create` | Enable/disable a Pod Disruption Budget creation | `false` | +| `pdb.minAvailable` | Minimum number/percentage of pods that should remain scheduled | `nil` | +| `pdb.maxUnavailable` | Maximum number/percentage of pods that may be made unavailable | `1` | +| `sidecars` | Attach additional sidecar containers to the Kafka pod | `{}` | + +### Exposure parameters + +| Parameter | Description | Default | +|---------------------------------------------------|----------------------------------------------------------------------------------------------------------|-------------------------------| +| `service.type` | Kubernetes Service type | `ClusterIP` | +| `service.port` | Kafka port | `9092` | +| `service.sslPort` | Kafka SSL port | `9093` | +| `service.nodePorts.kafka` | Kubernetes Kafka node port | `""` | +| `service.nodePorts.ssl` | Kubernetes Kafka SSL node port | `""` | +| `service.loadBalancerIP` | loadBalancerIP for Kafka Service | `nil` | +| `service.loadBalancerSourceRanges` | Address(es) that are allowed when service is LoadBalancer | `[]` | +| `service.annotations` | Service annotations | `{}`(evaluated as a template) | +| `externalAccess.enabled` | Enable Kubernetes external cluster access to Kafka brokers | `false` | +| `externalAccess.autoDiscovery.enabled` | Enable using an init container to auto-detect external IPs/ports by querying the K8s API | `false` | +| `externalAccess.autoDiscovery.image.registry` | Init container auto-discovery image registry (kubectl) | `docker.io` | +| `externalAccess.autoDiscovery.image.repository` | Init container auto-discovery image name (kubectl) | `bitnami/kubectl` | +| `externalAccess.autoDiscovery.image.tag` | Init container auto-discovery image tag (kubectl) | `{TAG_NAME}` | +| `externalAccess.autoDiscovery.image.pullPolicy` | Init container auto-discovery image pull policy (kubectl) | `Always` | +| `externalAccess.autoDiscovery.resources.limits` | Init container auto-discovery resource limits | `{}` | +| `externalAccess.autoDiscovery.resources.requests` | Init container auto-discovery resource requests | `{}` | +| `externalAccess.service.type` | Kubernetes Servive type for external access. It can be NodePort or LoadBalancer | `LoadBalancer` | +| `externalAccess.service.port` | Kafka port used for external access when service type is LoadBalancer | `19092` | +| `externalAccess.service.loadBalancerIPs` | Array of load balancer IPs for Kafka brokers | `[]` | +| `externalAccess.service.loadBalancerSourceRanges` | Address(es) that are allowed when service is LoadBalancer | `[]` | +| `externalAccess.service.domain` | Domain or external ip used to configure Kafka external listener when service type is NodePort | `nil` | +| `externalAccess.service.nodePorts` | Array of node ports used to configure Kafka external listener when service type is NodePort | `[]` | +| `externalAccess.service.annotations` | Service annotations for external access | `{}`(evaluated as a template) | + +### Persistence parameters + +| Parameter | Description | Default | +|-----------------------------|----------------------------------------------------------------------------------------|-------------------------------| +| `persistence.enabled` | Enable Kafka data persistence using PVC, note that Zookeeper persistence is unaffected | `true` | +| `persistence.existingClaim` | Provide an existing `PersistentVolumeClaim`, the value is evaluated as a template | `nil` | +| `persistence.storageClass` | PVC Storage Class for Kafka data volume | `nil` | +| `persistence.accessMode` | PVC Access Mode for Kafka data volume | `ReadWriteOnce` | +| `persistence.size` | PVC Storage Request for Kafka data volume | `8Gi` | +| `persistence.annotations` | Annotations for the PVC | `{}`(evaluated as a template) | + +### RBAC parameters + +| Parameter | Description | Default | +|-------------------------|--------------------------------------------------|-----------------------------------------------| +| `serviceAccount.create` | Enable creation of ServiceAccount for Kafka pods | `true` | +| `serviceAccount.name` | Name of the created serviceAccount | Generated using the `kafka.fullname` template | +| `rbac.create` | Weather to create & use RBAC resources or not | `false` | + +### Volume Permissions parameters + +| Parameter | Description | Default | +|----------------------------------------|----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------| +| `volumePermissions.enabled` | Enable init container that changes the owner and group of the persistent volume(s) mountpoint to `runAsUser:fsGroup` | `false` | +| `volumePermissions.image.registry` | Init container volume-permissions image registry | `docker.io` | +| `volumePermissions.image.repository` | Init container volume-permissions image name | `bitnami/minideb` | +| `volumePermissions.image.tag` | Init container volume-permissions image tag | `buster` | +| `volumePermissions.image.pullPolicy` | Init container volume-permissions image pull policy | `Always` | +| `volumePermissions.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | +| `volumePermissions.resources.limits` | Init container volume-permissions resource limits | `{}` | +| `volumePermissions.resources.requests` | Init container volume-permissions resource requests | `{}` | + +### Metrics parameters + +| Parameter | Description | Default | +|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------| +| `metrics.kafka.enabled` | Whether or not to create a standalone Kafka exporter to expose Kafka metrics | `false` | +| `metrics.kafka.image.registry` | Kafka exporter image registry | `docker.io` | +| `metrics.kafka.image.repository` | Kafka exporter image name | `bitnami/kafka-exporter` | +| `metrics.kafka.image.tag` | Kafka exporter image tag | `{TAG_NAME}` | +| `metrics.kafka.image.pullPolicy` | Kafka exporter image pull policy | `IfNotPresent` | +| `metrics.kafka.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | +| `metrics.kafka.resources.limits` | Kafka Exporter container resource limits | `{}` | +| `metrics.kafka.resources.requests` | Kafka Exporter container resource requests | `{}` | +| `metrics.kafka.service.type` | Kubernetes service type (`ClusterIP`, `NodePort` or `LoadBalancer`) for Kafka Exporter | `ClusterIP` | +| `metrics.kafka.service.port` | Kafka Exporter Prometheus port | `9308` | +| `metrics.kafka.service.nodePort` | Kubernetes HTTP node port | `""` | +| `metrics.kafka.service.annotations` | Annotations for Prometheus metrics service | `Check values.yaml file` | +| `metrics.kafka.service.loadBalancerIP` | loadBalancerIP if service type is `LoadBalancer` | `nil` | +| `metrics.kafka.service.clusterIP` | Static clusterIP or None for headless services | `nil` | +| `metrics.jmx.enabled` | Whether or not to expose JMX metrics to Prometheus | `false` | +| `metrics.jmx.image.registry` | JMX exporter image registry | `docker.io` | +| `metrics.jmx.image.repository` | JMX exporter image name | `bitnami/jmx-exporter` | +| `metrics.jmx.image.tag` | JMX exporter image tag | `{TAG_NAME}` | +| `metrics.jmx.image.pullPolicy` | JMX exporter image pull policy | `IfNotPresent` | +| `metrics.jmx.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) | +| `metrics.jmx.resources.limits` | JMX Exporter container resource limits | `{}` | +| `metrics.jmx.resources.requests` | JMX Exporter container resource requests | `{}` | +| `metrics.jmx.service.type` | Kubernetes service type (`ClusterIP`, `NodePort` or `LoadBalancer`) for JMX Exporter | `ClusterIP` | +| `metrics.jmx.service.port` | JMX Exporter Prometheus port | `5556` | +| `metrics.jmx.service.nodePort` | Kubernetes HTTP node port | `""` | +| `metrics.jmx.service.annotations` | Annotations for Prometheus metrics service | `Check values.yaml file` | +| `metrics.jmx.service.loadBalancerIP` | loadBalancerIP if service type is `LoadBalancer` | `nil` | +| `metrics.jmx.service.clusterIP` | Static clusterIP or None for headless services | `nil` | +| `metrics.jmx.whitelistObjectNames` | Allows setting which JMX objects you want to expose to via JMX stats to JMX Exporter | (see `values.yaml`) | +| `metrics.jmx.config` | Configuration file for JMX exporter | (see `values.yaml`) | +| `metrics.jmx.existingConfigmap` | Name of existing ConfigMap with JMX exporter configuration | `nil` | +| `metrics.serviceMonitor.enabled` | if `true`, creates a Prometheus Operator ServiceMonitor (requires `metrics.kafka.enabled` or `metrics.jmx.enabled` to be `true`) | `false` | +| `metrics.serviceMonitor.namespace` | Namespace which Prometheus is running in | `monitoring` | +| `metrics.serviceMonitor.interval` | Interval at which metrics should be scraped | `nil` | +| `metrics.serviceMonitor.scrapeTimeout` | Timeout after which the scrape is ended | `nil` (Prometheus Operator default value) | +| `metrics.serviceMonitor.selector` | ServiceMonitor selector labels | `nil` (Prometheus Operator default value) | + +### Zookeeper chart parameters + +| Parameter | Description | Default | +|----------------------------------|------------------------------------------------------|-----------| +| `zookeeper.enabled` | Switch to enable or disable the Zookeeper helm chart | `true` | +| `zookeeper.persistence.enabled` | Enable Zookeeper persistence using PVC | `true` | +| `externalZookeeper.servers` | Server or list of external Zookeeper servers to use | `[]` | Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, ```console -$ helm install my-release \ - --set kafkaPassword=secretpassword,kafkaDatabase=my-database \ - bitnami/kafka +helm install my-release \ + --set replicaCount=3 \ + bitnami/kafka ``` -The above command sets the Kafka `kafka` account password to `secretpassword`. Additionally it creates a database named `my-database`. +The above command deploys Kafka with 3 brokers (replicas). Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example, ```console -$ helm install my-release -f values.yaml bitnami/kafka +helm install my-release -f values.yaml bitnami/kafka ``` > **Tip**: You can use the default [values.yaml](values.yaml) @@ -229,60 +279,70 @@ Bitnami will release a new chart updating its containers if a new version of the This chart includes a `values-production.yaml` file where you can find some parameters oriented to production configuration in comparison to the regular `values.yaml`. You can use this file instead of the default one. - Number of Kafka nodes: + ```diff - replicaCount: 1 + replicaCount: 3 ``` - Allow to use the PLAINTEXT listener: + ```diff - allowPlaintextListener: true + allowPlaintextListener: false ``` - Default replication factors for automatically created topics: + ```diff - defaultReplicationFactor: 1 + defaultReplicationFactor: 3 ``` - The replication factor for the offsets topic: + ```diff - offsetsTopicReplicationFactor: 1 + offsetsTopicReplicationFactor: 3 ``` - The replication factor for the transaction topic: + ```diff - transactionStateLogReplicationFactor: 1 + transactionStateLogReplicationFactor: 3 ``` - Overridden min.insync.replicas config for the transaction topic: + ```diff - transactionStateLogMinIsr: 1 + transactionStateLogMinIsr: 3 ``` - Switch to enable the kafka authentication: + ```diff - auth.enabled: false + auth.enabled: true ``` - Whether or not to create a separate Kafka exporter: + ```diff - metrics.kafka.enabled: false + metrics.kafka.enabled: true ``` - Whether or not to expose JMX metrics to Prometheus: + ```diff - metrics.jmx.enabled: false + metrics.jmx.enabled: true ``` - Zookeeper chart metrics configuration: + ```diff + zookeeper.metrics.enabled: true ``` @@ -298,11 +358,11 @@ In order to pass custom environment variables use the `extraEnvVars` property. If you enabled the authentication for Kafka, the SASL_SSL listener will be configured with your provided inputs. In particular you can set the following pair of credentials: - * brokerUser/brokerPassword: To authenticate kafka clients against kafka brokers - * interBrokerUser/interBrokerPassword: To authenticate kafka brokers between them. - * zookeeperUser/zookeeperPassword: In the case that the Zookeeper chart is deployed with SASL authentication enabled. +- brokerUser/brokerPassword: To authenticate kafka clients against kafka brokers +- interBrokerUser/interBrokerPassword: To authenticate kafka brokers between them. +- zookeeperUser/zookeeperPassword: In the case that the Zookeeper chart is deployed with SASL authentication enabled. -In order to configure the authentication, you **must** create a secret containing the *kafka.keystore.jks* and *kafka.truststore.jks* certificates and pass the secret name with the `--auth.certificatesSecret` option when deploying the chart. +In order to configure the authentication, you **must** create a secret containing the *kafka.keystore.jks* and *kafka.trustore.jks* certificates and pass the secret name with the `--auth.certificatesSecret` option when deploying the chart. You can create the secret and deploy the chart with authentication using the following parameters: @@ -332,27 +392,76 @@ There are two ways of configuring external access. Using LoadBalancer services o #### Using LoadBalancer services +You have two alternatives to use LoadBalancer services: + +- Option A) Use random load balancer IPs using an **initContainer** that waits for the IPs to be ready and discover them automatically. + ```console externalAccess.enabled=true externalAccess.service.type=LoadBalancer externalAccess.service.port=19092 -externalAccess.service.loadBalancerIP={'external-ip-1', 'external-ip-2'} +externalAccess.autoDiscovery.enabled=true +serviceAccount.create=true +rbac.create=true ``` -You need to know in advance the load balancer IPs so each Kafka broker advertised listener is configured with it. +Note: This option requires creating RBAC rules on clusters where RBAC policies are enabled. + +- Option B) Manually specify the load balancer IPs: + +```console +externalAccess.enabled=true +externalAccess.service.type=LoadBalancer +externalAccess.service.port=19092 +externalAccess.service.loadBalancerIPs[0]='external-ip-1' +externalAccess.service.loadBalancerIPs[1]='external-ip-2'} +``` + +Note: You need to know in advance the load balancer IPs so each Kafka broker advertised listener is configured with it. #### Using NodePort services +You have two alternatives to use NodePort services: + +- Option A) Use random node ports using an **initContainer** that discover them automatically. + ```console externalAccess.enabled=true externalAccess.service.type=NodePort -externalAccess.service.nodePort={'node-port-1', 'node-port-2'} +externalAccess.autoDiscovery.enabled=true +serviceAccount.create=true +rbac.create=true ``` -You need to know in advance the NodePort that will be exposed for each Kafka broker. It will be used to configure the advertised listener of each broker. +Note: This option requires creating RBAC rules on clusters where RBAC policies are enabled. + +- Option B) Manually specify the node ports: + +```console +externalAccess.enabled=true +externalAccess.service.type=NodePort +externalAccess.serivce.nodePorts[0]='node-port-1' +externalAccess.serivce.nodePorts[1]='node-port-2' +``` + +Note: You need to know in advance the node ports that will be exposed so each Kafka broker advertised listener is configured with it. The pod will try to get the external ip of the node using `curl -s https://ipinfo.io/ip` unless `externalAccess.service.domain` is provided. +### Sidecars + +If you have a need for additional containers to run within the same pod as Kafka (e.g. an additional metrics or logging exporter), you can do so via the `sidecars` config parameter. Simply define your container according to the Kubernetes container spec. + +```yaml +sidecars: + - name: your-image-name + image: your-image + imagePullPolicy: Always + ports: + - name: portname + containerPort: 1234 +``` + ## Persistence The [Bitnami Kafka](https://github.com/bitnami/bitnami-docker-kafka) image stores the Kafka data at the `/bitnami/kafka` path of the container. @@ -371,6 +480,28 @@ You can enable this initContainer by setting `volumePermissions.enabled` to `tru ## Upgrading +### To 9.0.0 + +Backwards compatibility is not guaranteed you adapt your values.yaml to the new format. Here you can find some parameters that were renamed on this major version: + +```diff +- securityContext.enabled +- securityContext.fsGroup +- securityContext.fsGroup ++ podSecurityContext +- externalAccess.service.loadBalancerIP ++ externalAccess.service.loadBalancerIPs +- externalAccess.service.nodePort ++ externalAccess.service.nodePorts +- metrics.jmx.configMap.enabled +- metrics.jmx.configMap.overrideConfig ++ metrics.jmx.config +- metrics.jmx.configMap.overrideName ++ metrics.jmx.existingConfigmap +``` + +Ports names were prefixed with the protocol to comply with Istio (see https://istio.io/docs/ops/deployment/requirements/). + ### To 8.0.0 There is not backwards compatibility since the brokerID changes to the POD_NAME. For more information see [this PR](https://github.com/bitnami/charts/pull/2028). @@ -381,8 +512,8 @@ Backwards compatibility is not guaranteed when Kafka metrics are enabled, unless Use the workaround below to upgrade from versions previous to 7.0.0. The following example assumes that the release name is kafka: ```console -$ helm upgrade kafka bitnami/kafka --version 6.1.8 --set metrics.kafka.enabled=false -$ helm upgrade kafka bitnami/kafka --version 7.0.0 --set metrics.kafka.enabled=true +helm upgrade kafka bitnami/kafka --version 6.1.8 --set metrics.kafka.enabled=false +helm upgrade kafka bitnami/kafka --version 7.0.0 --set metrics.kafka.enabled=true ``` ### To 2.0.0 @@ -391,8 +522,8 @@ Backwards compatibility is not guaranteed unless you modify the labels used on t Use the workaround below to upgrade from versions previous to 2.0.0. The following example assumes that the release name is kafka: ```console -$ kubectl delete statefulset kafka-kafka --cascade=false -$ kubectl delete statefulset kafka-zookeeper --cascade=false +kubectl delete statefulset kafka-kafka --cascade=false +kubectl delete statefulset kafka-zookeeper --cascade=false ``` ### To 1.0.0 @@ -401,6 +532,6 @@ Backwards compatibility is not guaranteed unless you modify the labels used on t Use the workaround below to upgrade from versions previous to 1.0.0. The following example assumes that the release name is kafka: ```console -$ kubectl delete statefulset kafka-kafka --cascade=false -$ kubectl delete statefulset kafka-zookeeper --cascade=false +kubectl delete statefulset kafka-kafka --cascade=false +kubectl delete statefulset kafka-zookeeper --cascade=false ``` diff --git a/bitnami/kafka/requirements.lock b/bitnami/kafka/requirements.lock index c8a11664c..92822b732 100644 --- a/bitnami/kafka/requirements.lock +++ b/bitnami/kafka/requirements.lock @@ -1,6 +1,6 @@ dependencies: - name: zookeeper repository: https://charts.bitnami.com/bitnami - version: 5.7.0 -digest: sha256:a17751aa80bc652c15978e24209b4d5dca3cb478fefd581affddca1ac8eabdbc -generated: "2020-03-20T10:19:20.537345164Z" + version: 5.7.1 +digest: sha256:3bfa6357c14e0e9605defd4325f5721b94bd2730edab1a3730baa740e44889a8 +generated: "2020-03-23T17:16:30.676937467Z" diff --git a/bitnami/kafka/templates/NOTES.txt b/bitnami/kafka/templates/NOTES.txt index 2ecbaeb79..82e912d3d 100644 --- a/bitnami/kafka/templates/NOTES.txt +++ b/bitnami/kafka/templates/NOTES.txt @@ -1,6 +1,41 @@ -{{- if contains .Values.service.type "LoadBalancer" }} -{{- if not .Values.kafkaPassword }} -------------------------------------------------------------------------------- +{{- $replicaCount := int .Values.replicaCount }} +{{- $releaseNamespace := .Release.Namespace }} +{{- $fullName := include "kafka.fullname" . }} +{{- $loadBalancerIPListLength := len .Values.externalAccess.service.loadBalancerIPs }} +{{- if and .Values.externalAccess.enabled (not .Values.externalAccess.autoDiscovery.enabled) (not (eq $replicaCount $loadBalancerIPListLength )) (eq .Values.externalAccess.service.type "LoadBalancer") }} + +############################################################################### +### ERROR: You enabled external access to Kafka brokers without specifying ### +### the array of load balancer IPs for Kafka brokers. ### +############################################################################### + +This deployment will be incomplete until you configure the array of load balancer +IPs for Kafka brokers. To complete your deployment follow the steps below: + +1. Wait for the load balancer IPs (it may take a few minutes for them to be available): +. + kubectl get svc --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka,pod" -w +. +2. Obtain the load balancer IPs and upgrade your chart: +. + {{- range $i, $e := until $replicaCount }} + LOAD_BALANCER_IP_{{ add $i 1 }}="$(kubectl get svc --namespace {{ $releaseNamespace }} {{ $fullName }}-{{ $i }}-external -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" + {{- end }} +. +3. Upgrade you chart: +. + helm upgrade {{ .Release.Name }} bitnami/{{ .Chart.Name }} \ + --set replicaCount={{ $replicaCount }} \ + --set externalAccess.enabled=true \ + {{- range $i, $e := until $replicaCount }} + --set externalAccess.service.loadBalancerIPs[{{ $i }}]=$LOAD_BALANCER_IP_{{ add $i 1 }} \ + {{- end }} + --set externalAccess.service.type=LoadBalancer +. +{{- else }} + +{{- if and (or (eq .Values.service.type "LoadBalancer") .Values.externalAccess.enabled) (not .Values.auth.enabled) }} +--------------------------------------------------------------------------------------------- WARNING By specifying "serviceType=LoadBalancer" and not configuring the authentication @@ -10,50 +45,24 @@ For security reasons, we strongly suggest that you switch to "ClusterIP" or "NodePort". As alternative, you can also configure the Kafka authentication. -------------------------------------------------------------------------------- -{{- end }} +--------------------------------------------------------------------------------------------- {{- end }} ** Please be patient while the chart is being deployed ** -Kafka can be accessed via port 9092 on the following DNS name from within your cluster: +Kafka can be accessed via port {{ .Values.service.port }} on the following DNS name from within your cluster: {{ template "kafka.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }} -To create a topic run the following command: +To create a a pod that you can use as a Kafka client run the following command: - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka" -o jsonpath="{.items[0].metadata.name}") - kubectl --namespace {{ .Release.Namespace }} exec -it $POD_NAME -- kafka-topics.sh --create --zookeeper {{ template "kafka.zookeeper.fullname" . }}:2181 --replication-factor 1 --partitions 1 --topic test - -To list all the topics run the following command: - - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka" -o jsonpath="{.items[0].metadata.name}") - kubectl --namespace {{ .Release.Namespace }} exec -it $POD_NAME -- kafka-topics.sh --list --zookeeper {{ template "kafka.zookeeper.fullname" . }}:2181 - -To start a kafka producer run the following command: - - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka" -o jsonpath="{.items[0].metadata.name}") - {{- if .Values.auth.enabled }} - kubectl --namespace {{ .Release.Namespace }} exec -it $POD_NAME -- kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test --producer.config /opt/bitnami/kafka/conf/producer.properties - {{- else }} - kubectl --namespace {{ .Release.Namespace }} exec -it $POD_NAME -- kafka-console-producer.sh --broker-list localhost:9092 --topic test - {{- end }} - -To start a kafka consumer run the following command: - - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka" -o jsonpath="{.items[0].metadata.name}") - {{- if .Values.auth.enabled }} - kubectl --namespace {{ .Release.Namespace }} exec -it $POD_NAME -- kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --consumer.config /opt/bitnami/kafka/conf/consumer.properties - {{- else }} - kubectl --namespace {{ .Release.Namespace }} exec -it $POD_NAME -- kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning - {{- end }} + kubectl run {{ template "kafka.fullname" . }}-client --rm --tty -i --restart='Never' --image {{ template "kafka.image" . }} --namespace {{ .Release.Namespace }} --command -- bash {{- if .Values.externalAccess.enabled }} -To connect to your Kafka server from outside the cluster check the following information: - -{{- if contains "NodePort" .Values.externalAccess.service.type }} +To connect to your Kafka server from outside the cluster, follow the instructions below: +{{- if eq "NodePort" .Values.externalAccess.service.type }} {{- if .Values.externalAccess.service.domain }} Kafka brokers domain: Use your provided hostname to reach Kafka brokers, {{ .Values.externalAccess.service.domain }} @@ -63,27 +72,34 @@ To connect to your Kafka server from outside the cluster check the following inf Kafka brokers domain: You can get the external node IP from the Kafka configuration file with the following commands (Check the EXTERNAL listener) 1. Obtain the pod name: + kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka" 2. Obtain pod configuration: + kubectl exec -it KAFKA_POD -- cat /opt/bitnami/kafka/conf/server.properties | grep advertised.listeners {{- end }} - Kafka brokers port: You should have a different node port for each Kafka broker. The list of configured node ports is {{ .Values.externalAccess.service.nodePort }} + Kafka brokers port: You will have a different node port for each Kafka broker. You can get the list of configured node ports using the command below: + + echo "$(kubectl get svc --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka,pod" -o jsonpath='{.items[*].spec.ports[0].nodePort}' | tr ' ' '\n')" {{- else if contains "LoadBalancer" .Values.externalAccess.service.type }} - NOTE: It may take a few minutes for the LoadBalancer IP to be available. - Watch the status with: 'kubectl get svc --namespace {{ .Release.Namespace }} -w {{ template "kafka.fullname" . }}' + NOTE: It may take a few minutes for the LoadBalancer IPs to be available. + Watch the status with: 'kubectl get svc --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka,pod" -w' + + Kafka Brokers domain: You will have a different external IP for each Kafka broker. You can get the list of external IPs using the command below: + + echo "$(kubectl get svc --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka,pod" -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}' | tr ' ' '\n')" - Kafka Brokers domain: You will have a different external IP for each Kafka broker. Get the external ip from `-external` suffixed services: `kubectl get svc`. Kafka Brokers port: {{ .Values.externalAccess.service.port }} {{- end }} {{- end }} -{{ if .Values.auth.enabled }} +{{- if .Values.auth.enabled }} PRODUCER: kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test --producer.config /opt/bitnami/kafka/conf/producer.properties CONSUMER: @@ -98,18 +114,13 @@ To connect to your Kafka server from outside the cluster check the following inf Finally, before using your client you need to export the following env var: export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/kafka_jaas.conf" -{{ else }} +{{- else }} PRODUCER: kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test CONSUMER: kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --from-beginning -{{ end }} - -{{- if and (contains "bitnami/" .Values.image.repository) (not (.Values.image.tag | toString | regexFind "-r\\d+$|sha256:")) }} - -WARNING: Rolling tag detected ({{ .Values.image.repository }}:{{ .Values.image.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment. -+info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/ - +{{- end }} {{- end }} -{{ include "kafka.validateValues" . }} \ No newline at end of file +{{- include "kafka.checkRollingTags" . }} +{{- include "kafka.validateValues" . }} diff --git a/bitnami/kafka/templates/_helpers.tpl b/bitnami/kafka/templates/_helpers.tpl index b777857c3..9ae2973d9 100644 --- a/bitnami/kafka/templates/_helpers.tpl +++ b/bitnami/kafka/templates/_helpers.tpl @@ -31,17 +31,6 @@ Create chart name and version as used by the chart label. {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} {{- end -}} -{{/* - Create the name of the service account to use - */}} -{{- define "kafka.serviceAccountName" -}} -{{- if .Values.serviceAccount.create -}} - {{ default (include "kafka.fullname" .) .Values.serviceAccount.name }} -{{- else -}} - {{ default "default" .Values.serviceAccount.name }} -{{- end -}} -{{- end -}} - {{/* Common labels */}} @@ -60,6 +49,30 @@ app.kubernetes.io/name: {{ include "kafka.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end -}} +{{/* +Create a default fully qualified zookeeper name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +*/}} +{{- define "kafka.zookeeper.fullname" -}} +{{- if .Values.zookeeper.fullnameOverride -}} +{{- .Values.zookeeper.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default "zookeeper" .Values.zookeeper.nameOverride -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} + +{{/* + Create the name of the service account to use + */}} +{{- define "kafka.serviceAccountName" -}} +{{- if .Values.serviceAccount.create -}} + {{ default (include "kafka.fullname" .) .Values.serviceAccount.name }} +{{- else -}} + {{ default "default" .Values.serviceAccount.name }} +{{- end -}} +{{- end -}} + {{/* Return the proper Kafka image name */}} @@ -83,6 +96,52 @@ Also, we can't use a single if because lazy evaluation is not an option {{- end -}} {{- end -}} +{{/* +Return the proper image name (for the init container auto-discovery image) +*/}} +{{- define "kafka.externalAccess.autoDiscovery.image" -}} +{{- $registryName := .Values.externalAccess.autoDiscovery.image.registry -}} +{{- $repositoryName := .Values.externalAccess.autoDiscovery.image.repository -}} +{{- $tag := .Values.externalAccess.autoDiscovery.image.tag | toString -}} +{{/* +Helm 2.11 supports the assignment of a value to a variable defined in a different scope, +but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic. +Also, we can't use a single if because lazy evaluation is not an option +*/}} +{{- if .Values.global }} + {{- if .Values.global.imageRegistry }} + {{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}} + {{- else -}} + {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}} + {{- end -}} +{{- else -}} + {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}} +{{- end -}} +{{- end -}} + +{{/* +Return the proper image name (for the init container volume-permissions image) +*/}} +{{- define "kafka.volumePermissions.image" -}} +{{- $registryName := .Values.volumePermissions.image.registry -}} +{{- $repositoryName := .Values.volumePermissions.image.repository -}} +{{- $tag := .Values.volumePermissions.image.tag | toString -}} +{{/* +Helm 2.11 supports the assignment of a value to a variable defined in a different scope, +but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic. +Also, we can't use a single if because lazy evaluation is not an option +*/}} +{{- if .Values.global }} + {{- if .Values.global.imageRegistry }} + {{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}} + {{- else -}} + {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}} + {{- end -}} +{{- else -}} + {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}} +{{- end -}} +{{- end -}} + {{/* Return the proper Kafka exporter image name */}} @@ -129,19 +188,6 @@ Also, we can't use a single if because lazy evaluation is not an option {{- end -}} {{- end -}} -{{/* -Create a default fully qualified zookeeper name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -*/}} -{{- define "kafka.zookeeper.fullname" -}} -{{- if .Values.zookeeper.fullnameOverride -}} -{{- .Values.zookeeper.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default "zookeeper" .Values.zookeeper.nameOverride -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} - {{/* Return the proper Docker Image Registry Secret Names */}} @@ -157,58 +203,41 @@ imagePullSecrets: {{- range .Values.global.imagePullSecrets }} - name: {{ . }} {{- end }} -{{- else if or .Values.image.pullSecrets .Values.metrics.kafka.image.pullSecrets .Values.metrics.jmx.image.pullSecrets .Values.volumePermissions.image.pullSecrets }} +{{- else if or .Values.image.pullSecrets .Values.externalAccess.autoDiscovery.image.pullSecrets .Values.volumePermissions.image.pullSecrets .Values.metrics.kafka.image.pullSecrets .Values.metrics.jmx.image.pullSecrets }} imagePullSecrets: {{- range .Values.image.pullSecrets }} - name: {{ . }} {{- end }} +{{- range .Values.externalAccess.autoDiscovery.image.pullSecrets }} + - name: {{ . }} +{{- end }} +{{- range .Values.volumePermissions.image.pullSecrets }} + - name: {{ . }} +{{- end }} {{- range .Values.metrics.kafka.image.pullSecrets }} - name: {{ . }} {{- end }} {{- range .Values.metrics.jmx.image.pullSecrets }} - name: {{ . }} {{- end }} -{{- range .Values.volumePermissions.image.pullSecrets }} - - name: {{ . }} -{{- end }} {{- end -}} -{{- else if or .Values.image.pullSecrets .Values.metrics.kafka.image.pullSecrets .Values.metrics.jmx.image.pullSecrets .Values.volumePermissions.image.pullSecrets }} +{{- else if or .Values.image.pullSecrets .Values.externalAccess.autoDiscovery.image.pullSecrets .Values.volumePermissions.image.pullSecrets .Values.metrics.kafka.image.pullSecrets .Values.metrics.jmx.image.pullSecrets }} imagePullSecrets: {{- range .Values.image.pullSecrets }} - name: {{ . }} {{- end }} +{{- range .Values.externalAccess.autoDiscovery.image.pullSecrets }} + - name: {{ . }} +{{- end }} +{{- range .Values.volumePermissions.image.pullSecrets }} + - name: {{ . }} +{{- end }} {{- range .Values.metrics.kafka.image.pullSecrets }} - name: {{ . }} {{- end }} {{- range .Values.metrics.jmx.image.pullSecrets }} - name: {{ . }} {{- end }} -{{- range .Values.volumePermissions.image.pullSecrets }} - - name: {{ . }} -{{- end }} -{{- end -}} -{{- end -}} - -{{/* -Return the proper image name (for the init container volume-permissions image) -*/}} -{{- define "kafka.volumePermissions.image" -}} -{{- $registryName := .Values.volumePermissions.image.registry -}} -{{- $repositoryName := .Values.volumePermissions.image.repository -}} -{{- $tag := .Values.volumePermissions.image.tag | toString -}} -{{/* -Helm 2.11 supports the assignment of a value to a variable defined in a different scope, -but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic. -Also, we can't use a single if because lazy evaluation is not an option -*/}} -{{- if .Values.global }} - {{- if .Values.global.imageRegistry }} - {{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}} - {{- else -}} - {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}} - {{- end -}} -{{- else -}} - {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}} {{- end -}} {{- end -}} @@ -247,6 +276,66 @@ but Helm 2.9 and 2.10 does not support it, so we need to implement this if-else {{- end -}} {{- end -}} +{{/* +Return the Kafka auth credentials secret +*/}} +{{- define "kafka.secretName" -}} +{{- if .Values.auth.existingSecret -}} + {{- printf "%s" (tpl .Values.auth.existingSecret $) -}} +{{- else -}} + {{- printf "%s" (include "kafka.fullname" .) -}} +{{- end -}} +{{- end -}} + +{{/* +Return true if a secret object should be created +*/}} +{{- define "kafka.createSecret" -}} +{{- if and .Values.auth.enabled (not .Values.auth.existingSecret) }} + {{- true -}} +{{- end -}} +{{- end -}} + +{{/* +Return the Kafka configuration configmap +*/}} +{{- define "kafka.configmapName" -}} +{{- if .Values.existingConfigmap -}} + {{- printf "%s" (tpl .Values.existingConfigmap $) -}} +{{- else -}} + {{- printf "%s-configuration" (include "kafka.fullname" .) -}} +{{- end -}} +{{- end -}} + +{{/* +Return true if a configmap object should be created +*/}} +{{- define "kafka.createConfigmap" -}} +{{- if and .Values.config (not .Values.existingConfigmap) }} + {{- true -}} +{{- end -}} +{{- end -}} + +{{/* +Return the Kafka configuration configmap +*/}} +{{- define "kafka.metrics.jmx.configmapName" -}} +{{- if .Values.metrics.jmx.existingConfigmap -}} + {{- printf "%s" (tpl .Values.metrics.jmx.existingConfigmap $) -}} +{{- else -}} + {{- printf "%s-jmx-configuration" (include "kafka.fullname" .) -}} +{{- end -}} +{{- end -}} + +{{/* +Return true if a configmap object should be created +*/}} +{{- define "kafka.metrics.jmx.createConfigmap" -}} +{{- if and .Values.metrics.jmx.enabled .Values.metrics.jmx.config (not .Values.metrics.jmx.existingConfigmap) }} + {{- true -}} +{{- end -}} +{{- end -}} + {{/* Renders a value that contains template. Usage: @@ -260,14 +349,36 @@ Usage: {{- end }} {{- end -}} +{{/* +Check if there are rolling tags in the images +*/}} +{{- define "kafka.checkRollingTags" -}} +{{- if and (contains "bitnami/" .Values.image.repository) (not (.Values.image.tag | toString | regexFind "-r\\d+$|sha256:")) }} +WARNING: Rolling tag detected ({{ .Values.image.repository }}:{{ .Values.image.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment. ++info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/ +{{- end }} +{{- if and (contains "bitnami/" .Values.externalAccess.autoDiscovery.image.repository) (not (.Values.externalAccess.autoDiscovery.image.tag | toString | regexFind "-r\\d+$|sha256:")) }} +WARNING: Rolling tag detected ({{ .Values.externalAccess.autoDiscovery.image.repository }}:{{ .Values.externalAccess.autoDiscovery.image.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment. ++info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/ +{{- end }} +{{- if and (contains "bitnami/" .Values.metrics.kafka.image.repository) (not (.Values.metrics.kafka.image.tag | toString | regexFind "-r\\d+$|sha256:")) }} +WARNING: Rolling tag detected ({{ .Values.metrics.kafka.image.repository }}:{{ .Values.metrics.kafka.image.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment. ++info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/ +{{- end }} +{{- if and (contains "bitnami/" .Values.metrics.jmx.image.repository) (not (.Values.metrics.jmx.image.tag | toString | regexFind "-r\\d+$|sha256:")) }} +WARNING: Rolling tag detected ({{ .Values.metrics.jmx.image.repository }}:{{ .Values.metrics.jmx.image.tag }}), please note that it is strongly recommended to avoid using rolling tags in a production environment. ++info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/ +{{- end }} +{{- end -}} + {{/* Compile all warnings into a single message, and call fail. */}} {{- define "kafka.validateValues" -}} {{- $messages := list -}} {{- $messages := append $messages (include "kafka.validateValues.nodePortListLength" .) -}} -{{- $messages := append $messages (include "kafka.validateValues.loadBalancerIPListLength" .) -}} {{- $messages := append $messages (include "kafka.validateValues.externalAccessServiceType" .) -}} +{{- $messages := append $messages (include "kafka.validateValues.externalAccessAutoDiscoveryRBAC" .) -}} {{- $messages := without $messages "" -}} {{- $message := join "\n" $messages -}} @@ -276,21 +387,11 @@ Compile all warnings into a single message, and call fail. {{- end -}} {{- end -}} -{{/* Validate values of Kafka - number of replicas must be the same than loadBalancerIP list */}} -{{- define "kafka.validateValues.loadBalancerIPListLength" -}} -{{- $replicaCount := int .Values.replicaCount }} -{{- $loadBalancerIPListLength := len .Values.externalAccess.service.loadBalancerIP }} -{{- if and ( .Values.externalAccess.enabled ) ( not (eq $replicaCount $loadBalancerIPListLength )) (eq .Values.externalAccess.service.type "LoadBalancer") -}} -kafka: externalAccess.service.loadBalancerIP - Number of replicas and loadBalancerIP array length must be the same. -{{- end -}} -{{- end -}} - {{/* Validate values of Kafka - number of replicas must be the same than NodePort list */}} {{- define "kafka.validateValues.nodePortListLength" -}} {{- $replicaCount := int .Values.replicaCount }} -{{- $nodePortListLength := len .Values.externalAccess.service.nodePort }} -{{- if and ( .Values.externalAccess.enabled ) ( not (eq $replicaCount $nodePortListLength )) (eq .Values.externalAccess.service.type "NodePort") -}} +{{- $nodePortListLength := len .Values.externalAccess.service.nodePorts }} +{{- if and .Values.externalAccess.enabled (not .Values.externalAccess.autoDiscovery.enabled) (not (eq $replicaCount $nodePortListLength )) (eq .Values.externalAccess.service.type "NodePort") -}} kafka: .Values.externalAccess.service.nodePort Number of replicas and nodePort array length must be the same. {{- end -}} @@ -302,4 +403,15 @@ kafka: .Values.externalAccess.service.nodePort kafka: externalAccess.service.type Available servive type for external access are NodePort or LoadBalancer. {{- end -}} -{{- end -}} \ No newline at end of file +{{- end -}} + +{{/* Validate values of Kafka - RBAC should be enabled when autoDiscovery is enabled */}} +{{- define "kafka.validateValues.externalAccessAutoDiscoveryRBAC" -}} +{{- if and .Values.externalAccess.enabled .Values.externalAccess.autoDiscovery.enabled (not .Values.rbac.create )}} +kafka: rbac.create + By specifying "externalAccess.enabled=true" and "externalAccess.autoDiscovery.enabled=true" + an initContainer will be used to autodetect the external IPs/ports by querying the + K8s API. Please note this initContainer requires specific RBAC resources. You can create them + by specifying "--set rbac.create=true". +{{- end -}} +{{- end -}} diff --git a/bitnami/kafka/templates/configmap.yaml b/bitnami/kafka/templates/configmap.yaml index 812c0d9ee..ff9b32162 100644 --- a/bitnami/kafka/templates/configmap.yaml +++ b/bitnami/kafka/templates/configmap.yaml @@ -1,4 +1,4 @@ -{{- if .Values.config }} +{{- if (include "kafka.createConfigmap" .) }} apiVersion: v1 kind: ConfigMap metadata: @@ -6,5 +6,5 @@ metadata: labels: {{- include "kafka.labels" . | nindent 4 }} data: server.properties: |- -{{ .Values.config | indent 4 }} + {{ .Values.config | nindent 4 }} {{- end -}} diff --git a/bitnami/kafka/templates/jmx-configmap.yaml b/bitnami/kafka/templates/jmx-configmap.yaml index ba0caf94e..1924ef1ae 100644 --- a/bitnami/kafka/templates/jmx-configmap.yaml +++ b/bitnami/kafka/templates/jmx-configmap.yaml @@ -1,25 +1,12 @@ -{{- if and .Values.metrics.jmx.enabled .Values.metrics.jmx.configMap.enabled }} +{{- if (include "kafka.metrics.jmx.createConfigmap" .) }} apiVersion: v1 kind: ConfigMap metadata: name: {{ template "kafka.fullname" . }}-jmx-configuration - labels: - app.kubernetes.io/name: {{ template "kafka.name" . }} - helm.sh/chart: {{ template "kafka.chart" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/managed-by: {{ .Release.Service }} + labels: {{- include "kafka.labels" . | nindent 4 }} data: - jmx-kafka-prometheus.yml: |+ -{{- if .Values.metrics.jmx.configMap.overrideConfig }} -{{ toYaml .Values.metrics.jmx.configMap.overrideConfig | indent 4 }} -{{- else }} - jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:{{ .Values.metrics.jmx.jmxPort }}/jmxrmi - lowercaseOutputName: true - lowercaseOutputLabelNames: true - ssl: false - {{ if .Values.metrics.jmx.whitelistObjectNames }} - whitelistObjectNames: ["{{ join "\",\"" .Values.metrics.jmx.whitelistObjectNames }}"] - {{ end }} + jmx-kafka-prometheus.yml: |- + {{- include "kafka.tplValue" ( dict "value" .Values.metrics.jmx.config "context" $ ) | nindent 4 }} rules: - pattern: kafka.controller<>(Value) name: kafka_controller_$1_$2_$4 @@ -63,5 +50,4 @@ data: labels: topic: $3 partition: $4 -{{- end }} {{- end -}} diff --git a/bitnami/kafka/templates/kafka-jmx-metrics-svc.yaml b/bitnami/kafka/templates/jmx-metrics-svc.yaml similarity index 62% rename from bitnami/kafka/templates/kafka-jmx-metrics-svc.yaml rename to bitnami/kafka/templates/jmx-metrics-svc.yaml index 9445c5126..11dadf441 100644 --- a/bitnami/kafka/templates/kafka-jmx-metrics-svc.yaml +++ b/bitnami/kafka/templates/jmx-metrics-svc.yaml @@ -5,20 +5,28 @@ metadata: name: {{ template "kafka.fullname" . }}-jmx-metrics labels: {{- include "kafka.labels" . | nindent 4 }} app.kubernetes.io/component: kafka + {{- if .Values.metrics.jmx.service.annotations }} annotations: {{ include "kafka.tplValue" ( dict "value" .Values.metrics.jmx.service.annotations "context" $) | nindent 4 }} + {{- end }} spec: type: {{ .Values.metrics.jmx.service.type }} - {{- if and .Values.metrics.jmx.service.loadBalancerIP (eq .Values.metrics.jmx.service.type "LoadBalancer") }} + {{- if eq .Values.metrics.jmx.service.type "LoadBalancer" }} + {{- if .Values.metrics.jmx.service.loadBalancerIP }} loadBalancerIP: {{ .Values.metrics.jmx.service.loadBalancerIP }} {{- end }} + {{- if .Values.metrics.jmx.service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: {{- toYaml .Values.metrics.jmx.service.loadBalancerSourceRanges | nindent 4 }} + {{- end }} + {{- end }} {{- if and (eq .Values.metrics.jmx.service.type "ClusterIP") .Values.metrics.jmx.service.clusterIP }} clusterIP: {{ .Values.metrics.jmx.service.clusterIP }} {{- end }} ports: - - name: metrics + - name: http-metrics port: {{ .Values.metrics.jmx.service.port }} + protocol: TCP targetPort: metrics - {{- if (and (or (eq .Values.metrics.jmx.service.type "NodePort") (eq .Values.metrics.jmx.service.type "LoadBalancer")) (not (empty .Values.metrics.jmx.service.nodePort)))}} + {{- if and (or (eq .Values.metrics.jmx.service.type "NodePort") (eq .Values.metrics.jmx.service.type "LoadBalancer")) (not (empty .Values.metrics.jmx.service.nodePort)) }} nodePort: {{ .Values.metrics.jmx.service.nodePort }} {{- else if eq .Values.metrics.jmx.service.type "ClusterIP" }} nodePort: null diff --git a/bitnami/kafka/templates/kafka-exporter.yaml b/bitnami/kafka/templates/kafka-metrics-deployment.yaml similarity index 65% rename from bitnami/kafka/templates/kafka-exporter.yaml rename to bitnami/kafka/templates/kafka-metrics-deployment.yaml index 91141dd48..dcd6e0fdc 100644 --- a/bitnami/kafka/templates/kafka-exporter.yaml +++ b/bitnami/kafka/templates/kafka-metrics-deployment.yaml @@ -5,7 +5,6 @@ metadata: name: {{ template "kafka.fullname" . }}-exporter labels: {{- include "kafka.labels" . | nindent 4 }} app.kubernetes.io/component: metrics - annotations: {{ include "kafka.tplValue" ( dict "value" .Values.metrics.kafka.annotations "context" $) | nindent 4 }} spec: replicas: 1 selector: @@ -15,18 +14,18 @@ spec: metadata: labels: {{- include "kafka.labels" . | nindent 8 }} app.kubernetes.io/component: metrics - annotations: {{ include "kafka.tplValue" ( dict "value" .Values.metrics.kafka.podAnnotations "context" $) | nindent 8 }} spec: {{- include "kafka.imagePullSecrets" . | indent 6 }} containers: - - image: {{ template "kafka.metrics.kafka.image" . }} - name: kafka-exporter + - name: kafka-exporter + image: {{ include "kafka.metrics.kafka.image" . }} + imagePullPolicy: {{ .Values.metrics.kafka.image.pullPolicy | quote }} args: - --kafka.server={{ template "kafka.fullname" . }}:{{ .Values.service.port }} - - --web.listen-address=:{{ .Values.metrics.kafka.port }} + - --web.listen-address=:9308 ports: - name: metrics - containerPort: {{ .Values.metrics.kafka.port }} + containerPort: 9308 {{- if .Values.metrics.kafka.resources }} resources: {{ toYaml .Values.metrics.kafka.resources | nindent 12 }} {{- end }} diff --git a/bitnami/kafka/templates/kafka-metrics-svc.yaml b/bitnami/kafka/templates/kafka-metrics-svc.yaml index 24e866e64..44e7dabad 100644 --- a/bitnami/kafka/templates/kafka-metrics-svc.yaml +++ b/bitnami/kafka/templates/kafka-metrics-svc.yaml @@ -5,20 +5,28 @@ metadata: name: {{ template "kafka.fullname" . }}-metrics labels: {{- include "kafka.labels" . | nindent 4 }} app.kubernetes.io/component: metrics + {{- if .Values.metrics.kafka.service.annotations }} annotations: {{ include "kafka.tplValue" ( dict "value" .Values.metrics.kafka.service.annotations "context" $) | nindent 4 }} + {{- end }} spec: type: {{ .Values.metrics.kafka.service.type }} - {{- if and .Values.metrics.kafka.service.loadBalancerIP (eq .Values.metrics.kafka.service.type "LoadBalancer") }} + {{- if eq .Values.metrics.kafka.service.type "LoadBalancer" }} + {{- if .Values.metrics.kafka.service.loadBalancerIP }} loadBalancerIP: {{ .Values.metrics.kafka.service.loadBalancerIP }} {{- end }} + {{- if .Values.metrics.kafka.service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: {{- toYaml .Values.metrics.kafka.service.loadBalancerSourceRanges | nindent 4 }} + {{- end }} + {{- end }} {{- if and (eq .Values.metrics.kafka.service.type "ClusterIP") .Values.metrics.kafka.service.clusterIP }} clusterIP: {{ .Values.metrics.kafka.service.clusterIP }} {{- end }} ports: - - name: metrics + - name: http-metrics port: {{ .Values.metrics.kafka.service.port }} + protocol: TCP targetPort: metrics - {{- if (and (or (eq .Values.metrics.kafka.service.type "NodePort") (eq .Values.metrics.kafka.service.type "LoadBalancer")) (not (empty .Values.metrics.kafka.service.nodePort)))}} + {{- if and (or (eq .Values.metrics.kafka.service.type "NodePort") (eq .Values.metrics.kafka.service.type "LoadBalancer")) (not (empty .Values.metrics.kafka.service.nodePort)) }} nodePort: {{ .Values.metrics.kafka.service.nodePort }} {{- else if eq .Values.metrics.kafka.service.type "ClusterIP" }} nodePort: null diff --git a/bitnami/kafka/templates/poddisruptionbudget.yaml b/bitnami/kafka/templates/poddisruptionbudget.yaml index ccaeb7005..6ba639bf4 100644 --- a/bitnami/kafka/templates/poddisruptionbudget.yaml +++ b/bitnami/kafka/templates/poddisruptionbudget.yaml @@ -1,5 +1,5 @@ {{- $replicaCount := int .Values.replicaCount }} -{{- if gt $replicaCount 1 }} +{{- if and .Values.pdb.create (gt $replicaCount 1) }} apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: @@ -7,8 +7,13 @@ metadata: labels: {{- include "kafka.labels" . | nindent 4 }} app.kubernetes.io/component: kafka spec: + {{- if .Values.pdb.minAvailable }} + minAvailable: {{ .Values.pdb.minAvailable }} + {{- end }} + {{- if .Values.pdb.maxUnavailable }} + maxUnavailable: {{ .Values.pdb.maxUnavailable }} + {{- end }} selector: matchLabels: {{- include "kafka.matchLabels" . | nindent 6 }} app.kubernetes.io/component: kafka -{{ toYaml .Values.podDisruptionBudget | indent 2 }} {{- end }} diff --git a/bitnami/kafka/templates/role.yaml b/bitnami/kafka/templates/role.yaml new file mode 100644 index 000000000..ac9ab1d52 --- /dev/null +++ b/bitnami/kafka/templates/role.yaml @@ -0,0 +1,18 @@ +{{- if .Values.rbac.create -}} +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ template "kafka.fullname" . }} + labels: {{- include "kafka.labels" . | nindent 4 }} + app.kubernetes.io/component: kafka +rules: + - apiGroups: + - "" + resources: + - services + verbs: + - get + - list + - watch + - patch +{{- end -}} diff --git a/bitnami/kafka/templates/rolebinding.yaml b/bitnami/kafka/templates/rolebinding.yaml new file mode 100644 index 000000000..705a17ace --- /dev/null +++ b/bitnami/kafka/templates/rolebinding.yaml @@ -0,0 +1,16 @@ +{{- if and .Values.serviceAccount.create .Values.rbac.create }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ template "kafka.fullname" . }} + labels: {{- include "kafka.labels" . | nindent 4 }} + app.kubernetes.io/component: kafka +roleRef: + kind: Role + name: {{ template "kafka.fullname" . }} + apiGroup: rbac.authorization.k8s.io +subjects: + - kind: ServiceAccount + name: {{ template "kafka.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/bitnami/kafka/templates/scripts-configmap.yaml b/bitnami/kafka/templates/scripts-configmap.yaml index ca7a4b584..0496d7c67 100644 --- a/bitnami/kafka/templates/scripts-configmap.yaml +++ b/bitnami/kafka/templates/scripts-configmap.yaml @@ -1,19 +1,72 @@ -{{- if .Values.externalAccess.enabled }} apiVersion: v1 kind: ConfigMap metadata: name: {{ template "kafka.fullname" . }}-scripts labels: {{- include "kafka.labels" . | nindent 4 }} data: + {{- $fullname := include "kafka.fullname" . }} + {{- $releaseNamespace := .Release.Namespace }} + {{- if .Values.externalAccess.autoDiscovery.enabled }} + auto-discovery.sh: |- + #!/bin/bash + + SVC_NAME="${MY_POD_NAME}-external" + + {{- if eq .Values.externalAccess.service.type "LoadBalancer" }} + # Auxiliar functions + retry_while() { + local -r cmd="${1:?cmd is missing}" + local -r retries="${2:-12}" + local -r sleep_time="${3:-5}" + local return_value=1 + + read -r -a command <<< "$cmd" + for ((i = 1 ; i <= retries ; i+=1 )); do + "${command[@]}" && return_value=0 && break + sleep "$sleep_time" + done + return $return_value + } + k8s_svc_lb_ip() { + local namespace=${1:?namespace is missing} + local service=${2:?service is missing} + echo "$(kubectl get svc "$service" -n "$namespace" -o jsonpath="{.status.loadBalancer.ingress[0].ip}")" + } + k8s_svc_lb_ip_ready() { + local namespace=${1:?namespace is missing} + local service=${2:?service is missing} + [[ -n "$(k8s_svc_lb_ip "$namespace" "$service")" ]] + } + # Wait until LoadBalancer IP is ready + retry_while "k8s_svc_lb_ip_ready {{ $releaseNamespace }} $SVC_NAME" || exit 1 + # Obtain LoadBalancer external IP + k8s_svc_lb_ip "{{ $releaseNamespace }}" "$SVC_NAME" | tee "$SHARED_FILE" + {{- else if eq .Values.externalAccess.service.type "NodePort" }} + k8s_svc_node_port() { + local namespace=${1:?namespace is missing} + local service=${2:?service is missing} + local index=${3:-0} + local node_port="$(kubectl get svc "$service" -n "$namespace" -o jsonpath="{.spec.ports[$index].nodePort}")" + kubectl patch svc "$service" -n "$namespace" --type='json' -p="[{\"op\": \"replace\", \"path\": \"/spec/ports/0/targetPort\", \"value\": $node_port}]" + echo "$node_port" + } + k8s_svc_node_port "{{ $releaseNamespace }}" "$SVC_NAME" | tee "$SHARED_FILE" + {{- end }} + {{- end }} setup.sh: |- #!/bin/bash - HOSTNAME=$(hostname -s) - ID=${HOSTNAME:(-1)} + ID="${MY_POD_NAME#"{{ $fullname }}-"}" + export KAFKA_CFG_BROKER_ID="$ID" + {{- if .Values.externalAccess.enabled }} # Configure external ip and port {{- if eq .Values.externalAccess.service.type "LoadBalancer" }} - export EXTERNAL_ACCESS_IP=$(echo '{{ .Values.externalAccess.service.loadBalancerIP }}' | tr -d '[]' | cut -d ' ' -f "$(($ID + 1))") + {{- if .Values.externalAccess.autoDiscovery.enabled }} + export EXTERNAL_ACCESS_IP="$(<${SHARED_FILE})" + {{- else }} + export EXTERNAL_ACCESS_IP=$(echo '{{ .Values.externalAccess.service.loadBalancerIPs }}' | tr -d '[]' | cut -d ' ' -f "$(($ID + 1))") + {{- end }} export EXTERNAL_ACCESS_PORT={{ .Values.externalAccess.service.port }} {{- else if eq .Values.externalAccess.service.type "NodePort" }} {{- if .Values.externalAccess.service.domain }} @@ -21,19 +74,20 @@ data: {{- else }} export EXTERNAL_ACCESS_IP=$(curl -s https://ipinfo.io/ip) {{- end }} - export EXTERNAL_ACCESS_PORT=$(echo '{{ .Values.externalAccess.service.nodePort }}' | tr -d '[]' | cut -d ' ' -f "$(($ID + 1))") + {{- if .Values.externalAccess.autoDiscovery.enabled }} + export EXTERNAL_ACCESS_PORT="$(<${SHARED_FILE})" + {{- else }} + export EXTERNAL_ACCESS_PORT=$(echo '{{ .Values.externalAccess.service.nodePorts }}' | tr -d '[]' | cut -d ' ' -f "$(($ID + 1))") + {{- end }} {{- end }} # Configure Kafka internal and external listeners - export KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT export KAFKA_CFG_LISTENERS=INTERNAL://:{{ .Values.service.port }},EXTERNAL://:${EXTERNAL_ACCESS_PORT} - export KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL {{- if .Values.advertisedListeners }} export KAFKA_CFG_ADVERTISED_LISTENERS={{ .Values.advertisedListeners }} {{- else }} export KAFKA_CFG_ADVERTISED_LISTENERS="INTERNAL://${MY_POD_NAME}.{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:{{ .Values.service.port }},EXTERNAL://${EXTERNAL_ACCESS_IP}:${EXTERNAL_ACCESS_PORT}" {{- end }} + {{- end }} exec /entrypoint.sh /run.sh - -{{- end }} \ No newline at end of file diff --git a/bitnami/kafka/templates/secrets.yaml b/bitnami/kafka/templates/secrets.yaml index 4b0d26f6d..1b5fdd5de 100644 --- a/bitnami/kafka/templates/secrets.yaml +++ b/bitnami/kafka/templates/secrets.yaml @@ -1,4 +1,4 @@ -{{- if and .Values.auth.enabled (not .Values.auth.existingSecret) -}} +{{- if (include "kafka.createSecret" .) }} apiVersion: v1 kind: Secret metadata: @@ -6,21 +6,9 @@ metadata: labels: {{- include "kafka.labels" . | nindent 4 }} type: Opaque data: - {{- if .Values.auth.brokerPassword }} - kafka-broker-password: {{ .Values.auth.brokerPassword | b64enc | quote }} - {{ else }} - kafka-broker-password: {{ randAlphaNum 10 | b64enc | quote }} - {{- end }} - {{- if .Values.auth.interBrokerPassword }} - kafka-inter-broker-password: {{ .Values.auth.interBrokerPassword | b64enc | quote }} - {{ else }} - kafka-inter-broker-password: {{ randAlphaNum 10 | b64enc | quote }} - {{- end }} + kafka-broker-password: {{ ternary (randAlphaNum 10) .Values.auth.brokerPassword (empty .Values.auth.brokerPassword) | b64enc | quote }} + kafka-inter-broker-password: {{ ternary (randAlphaNum 10) .Values.auth.interBrokerPassword (empty .Values.auth.interBrokerPassword) | b64enc | quote }} {{- if .Values.auth.zookeeperUser }} - {{- if .Values.auth.zookeeperPassword }} - kafka-zookeeper-password: {{ .Values.auth.zookeeperPassword | b64enc | quote }} - {{ else }} - kafka-zookeeper-password: {{ randAlphaNum 10 | b64enc | quote }} - {{- end }} + kafka-zookeeper-password: {{ ternary (randAlphaNum 10) .Values.auth.zookeeperPassword (empty .Values.auth.zookeeperPassword) | b64enc | quote }} {{- end }} {{- end }} diff --git a/bitnami/kafka/templates/serviceaccount.yaml b/bitnami/kafka/templates/serviceaccount.yaml index 9a6a3979a..5d1cfb0cd 100644 --- a/bitnami/kafka/templates/serviceaccount.yaml +++ b/bitnami/kafka/templates/serviceaccount.yaml @@ -4,5 +4,5 @@ kind: ServiceAccount metadata: name: {{ template "kafka.serviceAccountName" . }} labels: {{- include "kafka.labels" . | nindent 4 }} - role: kafka + app.kubernetes.io/component: kafka {{- end }} diff --git a/bitnami/kafka/templates/statefulset.yaml b/bitnami/kafka/templates/statefulset.yaml index 624f2e5b6..4f182fb23 100644 --- a/bitnami/kafka/templates/statefulset.yaml +++ b/bitnami/kafka/templates/statefulset.yaml @@ -1,20 +1,22 @@ +{{- $replicaCount := int .Values.replicaCount }} +{{- $loadBalancerIPListLength := len .Values.externalAccess.service.loadBalancerIPs }} +{{- if not (and .Values.externalAccess.enabled (not .Values.externalAccess.autoDiscovery.enabled) (not (eq $replicaCount $loadBalancerIPListLength )) (eq .Values.externalAccess.service.type "LoadBalancer")) }} apiVersion: apps/v1 kind: StatefulSet metadata: name: {{ include "kafka.fullname" . }} labels: {{- include "kafka.labels" . | nindent 4 }} app.kubernetes.io/component: kafka - role: kafka spec: + podManagementPolicy: Parallel + replicas: {{ .Values.replicaCount }} selector: matchLabels: {{- include "kafka.matchLabels" . | nindent 6 }} app.kubernetes.io/component: kafka serviceName: {{ template "kafka.fullname" . }}-headless - podManagementPolicy: "Parallel" - replicas: {{ .Values.replicaCount }} updateStrategy: type: {{ .Values.updateStrategy | quote }} - {{- if (eq "Recreate" .Values.updateStrategy) }} + {{- if (eq "OnDelete" .Values.updateStrategy) }} rollingUpdate: null {{- else if .Values.rollingUpdatePartition }} rollingUpdate: @@ -24,45 +26,93 @@ spec: metadata: labels: {{- include "kafka.labels" . | nindent 8 }} app.kubernetes.io/component: kafka + {{- if or (include "kafka.createConfigmap" .) (include "kafka.createSecret" .) .Values.externalAccess.enabled (include "kafka.metrics.jmx.createConfigmap" .) .Values.podAnnotations }} + annotations: + {{- if (include "kafka.createConfigmap" .) }} + checksum/configuration: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + {{- end }} + {{- if (include "kafka.createSecret" .) }} + checksum/secret: {{ include (print $.Template.BasePath "/secrets.yaml") . | sha256sum }} + {{- end }} + {{- if .Values.externalAccess.enabled }} + checksum/scripts: {{ include (print $.Template.BasePath "/scripts-configmap.yaml") . | sha256sum }} + {{- end }} + {{- if (include "kafka.metrics.jmx.createConfigmap" .) }} + checksum/jmx-configuration: {{ include (print $.Template.BasePath "/jmx-configmap.yaml") . | sha256sum }} + {{- end }} + {{- if .Values.podAnnotations }} + {{- include "kafka.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + {{- end }} + {{- end }} spec: {{- include "kafka.imagePullSecrets" . | indent 6 }} + {{- if .Values.affinity }} + affinity: {{- include "kafka.tplValue" ( dict "value" .Values.affinity "context" $ ) | nindent 8 }} + {{- end }} + {{- if .Values.nodeSelector }} + nodeSelector: {{- include "kafka.tplValue" ( dict "value" .Values.nodeSelector "context" $ ) | nindent 8 }} + {{- end }} + {{- if .Values.tolerations }} + tolerations: {{- include "kafka.tplValue" ( dict "value" .Values.tolerations "context" $ ) | nindent 8 }} + {{- end }} + {{- if .Values.podSecurityContext }} + securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }} + {{- end }} {{- if .Values.serviceAccount.create }} serviceAccountName: {{ template "kafka.serviceAccountName" . }} {{- end }} - {{- if .Values.securityContext.enabled }} - securityContext: - fsGroup: {{ .Values.securityContext.fsGroup }} - runAsUser: {{ .Values.securityContext.runAsUser }} - {{- end }} - nodeSelector: {{- include "kafka.tplValue" ( dict "value" .Values.nodeSelector "context" $ ) | nindent 8 }} - tolerations: {{- include "kafka.tplValue" ( dict "value" .Values.tolerations "context" $ ) | nindent 8 }} - affinity: {{- include "kafka.tplValue" ( dict "value" .Values.affinity "context" $ ) | nindent 8 }} - {{- if and .Values.volumePermissions.enabled .Values.persistence.enabled }} + {{- if or (and .Values.volumePermissions.enabled .Values.persistence.enabled) (and .Values.externalAccess.enabled .Values.externalAccess.autoDiscovery.enabled) }} initContainers: + {{- if and .Values.volumePermissions.enabled .Values.persistence.enabled }} - name: volume-permissions image: {{ include "kafka.volumePermissions.image" . }} - imagePullPolicy: {{ default "" .Values.volumePermissions.image.pullPolicy | quote }} - command: ["chown", "-R", "{{ .Values.securityContext.runAsUser }}:{{ .Values.securityContext.fsGroup }}", "/bitnami/kafka"] + imagePullPolicy: {{ .Values.volumePermissions.image.pullPolicy | quote }} + command: + - /bin/bash + args: + - -ec + - | + mkdir -p /bitnami/kafka + chown -R "{{ .Values.podSecurityContext.runAsUser }}:{{ .Values.podSecurityContext.fsGroup }}" "/bitnami/kafka" securityContext: runAsUser: 0 {{- if .Values.volumePermissions.resources }} - resources: {{ toYaml .Values.volumePermissions.resources | nindent 12 }} + resources: {{- toYaml .Values.volumePermissions.resources | nindent 12 }} {{- end }} volumeMounts: - name: data - mountPath: "/bitnami/kafka" + mountPath: /bitnami/kafka + {{- end }} + {{- if and .Values.externalAccess.enabled .Values.externalAccess.autoDiscovery.enabled }} + - name: auto-discovery + image: {{ include "kafka.externalAccess.autoDiscovery.image" . }} + imagePullPolicy: {{ .Values.externalAccess.autoDiscovery.image.pullPolicy | quote }} + command: + - /scripts/auto-discovery.sh + env: + - name: MY_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: SHARED_FILE + value: "/shared/info.txt" + {{- if .Values.externalAccess.autoDiscovery.resources }} + resources: {{- toYaml .Values.externalAccess.autoDiscovery.resources | nindent 12 }} + {{- end }} + volumeMounts: + - name: shared + mountPath: /shared + - name: scripts + mountPath: /scripts/auto-discovery.sh + subPath: auto-discovery.sh + {{- end }} {{- end }} containers: - name: kafka image: {{ include "kafka.image" . }} imagePullPolicy: {{ .Values.image.pullPolicy | quote }} - {{- if .Values.resources }} - resources: {{ toYaml .Values.resources | nindent 12 }} - {{- end }} - {{- if .Values.externalAccess.enabled }} command: - /scripts/setup.sh - {{- end }} env: - name: BITNAMI_DEBUG value: {{ ternary "true" "false" .Values.image.debug | quote }} @@ -76,20 +126,39 @@ spec: fieldPath: metadata.name - name: KAFKA_CFG_ZOOKEEPER_CONNECT {{- if .Values.zookeeper.enabled }} - value: {{ template "kafka.zookeeper.fullname" . }} + value: {{ include "kafka.zookeeper.fullname" . | quote }} {{- else }} value: {{ .Values.externalZookeeper.servers | quote }} {{- end }} - name: KAFKA_PORT_NUMBER value: {{ .Values.service.port | quote }} - {{- if not .Values.externalAccess.enabled }} + {{- if and .Values.externalAccess.enabled .Values.externalAccess.autoDiscovery.enabled }} + - name: SHARED_FILE + value: "/shared/info.txt" + {{- end }} + {{- if .Values.externalAccess.enabled }} + - name: KAFKA_INTER_BROKER_LISTENER_NAME + value: "INTERNAL" + - name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP + {{- if and .Values.auth.enabled .Values.auth.ssl }} + value: "INTERNAL:SASL_SSL,EXTERNAL:SASL_SSL" + {{- else if and .Values.auth.ssl (not .Values.auth.enabled) }} + value: "INTERNAL:SSL,EXTERNAL:SSL" + {{- else if and .Values.auth.enabled (not .Values.auth.ssl) }} + value: "INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT" + {{- else }} + value: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT" + {{- end }} + {{- else }} - name: KAFKA_CFG_LISTENERS {{- if .Values.listeners }} value: {{ .Values.listeners }} {{- else if and .Values.auth.ssl .Values.auth.enabled }} value: "SASL_SSL://:$(KAFKA_PORT_NUMBER),SSL://:9093" - {{- else if .Values.auth.enabled }} - value: "SASL_SSL://:$(KAFKA_PORT_NUMBER)" + {{- else if and .Values.auth.ssl (not .Values.auth.enabled) }} + value: "SSL://:9093" + {{- else if and .Values.auth.enabled (not .Values.auth.ssl) }} + value: "SASL_PLAINTEXT://:$(KAFKA_PORT_NUMBER)" {{- else }} value: "PLAINTEXT://:$(KAFKA_PORT_NUMBER)" {{- end }} @@ -97,11 +166,13 @@ spec: {{- if .Values.advertisedListeners }} value: {{ .Values.advertisedListeners }} {{- else if and .Values.auth.ssl .Values.auth.enabled }} - value: 'SASL_SSL://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:$(KAFKA_PORT_NUMBER),SSL://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:9093' - {{- else if .Values.auth.enabled }} - value: 'SASL_SSL://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:$(KAFKA_PORT_NUMBER)' + value: "SASL_SSL://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:$(KAFKA_PORT_NUMBER),SSL://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:9093" + {{- else if and .Values.auth.ssl (not .Values.auth.enabled) }} + value: "SSL://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:9093" + {{- else if and .Values.auth.enabled (not .Values.auth.ssl) }} + value: "SASL_PLAINTEXT://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:$(KAFKA_PORT_NUMBER)" {{- else }} - value: 'PLAINTEXT://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:$(KAFKA_PORT_NUMBER)' + value: "PLAINTEXT://$(MY_POD_NAME).{{ template "kafka.fullname" . }}-headless.{{.Release.Namespace}}.svc.{{ .Values.clusterDomain }}:$(KAFKA_PORT_NUMBER)" {{- end }} {{- end }} {{- if .Values.listenerSecurityProtocolMap }} @@ -114,7 +185,7 @@ spec: {{- end }} {{- if .Values.metrics.jmx.enabled }} - name: JMX_PORT - value: {{ .Values.metrics.jmx.jmxPort | quote }} + value: "5555" {{- end }} {{- if .Values.auth.enabled }} - name: KAFKA_OPTS @@ -124,14 +195,14 @@ spec: - name: KAFKA_BROKER_PASSWORD valueFrom: secretKeyRef: - name: {{ if .Values.auth.existingSecret }}{{ .Values.auth.existingSecret }}{{ else }}{{ template "kafka.fullname" . }}{{ end }} + name: {{ include "kafka.secretName" . }} key: kafka-broker-password - name: KAFKA_INTER_BROKER_USER value: {{ .Values.auth.interBrokerUser | quote }} - name: KAFKA_INTER_BROKER_PASSWORD valueFrom: secretKeyRef: - name: {{ if .Values.auth.existingSecret }}{{ .Values.auth.existingSecret }}{{ else }}{{ template "kafka.fullname" . }}{{ end }} + name: {{ include "kafka.secretName" . }} key: kafka-inter-broker-password {{- if .Values.auth.zookeeperUser }} - name: KAFKA_ZOOKEEPER_USER @@ -141,7 +212,7 @@ spec: - name: KAFKA_ZOOKEEPER_PASSWORD valueFrom: secretKeyRef: - name: {{ if .Values.auth.existingSecret }}{{ .Values.auth.existingSecret }}{{ else }}{{ template "kafka.fullname" . }}{{ end }} + name: {{ include "kafka.secretName" . }} key: kafka-zookeeper-password {{- end }} {{- end }} @@ -151,6 +222,8 @@ spec: {{- end }} - name: ALLOW_PLAINTEXT_LISTENER value: {{ ternary "yes" "no" (or .Values.auth.enabled .Values.allowPlaintextListener) | quote }} + - name: KAFKA_CFG_BROKER_ID + value: {{ .Values.brokerId | quote }} - name: KAFKA_CFG_DELETE_TOPIC_ENABLE value: {{ .Values.deleteTopicEnable | quote }} - name: KAFKA_HEAP_OPTS @@ -165,16 +238,12 @@ spec: value: {{ .Values.logRetentionCheckIntervalMs | quote }} - name: KAFKA_CFG_LOG_RETENTION_HOURS value: {{ .Values.logRetentionHours | quote }} - {{- if .Values.logMessageFormatVersion }} - - name: KAFKA_CFG_LOG_MESSAGE_FORMAT_VERSION - value: {{ .Values.logMessageFormatVersion | quote }} - {{- end }} - name: KAFKA_CFG_MESSAGE_MAX_BYTES value: {{ .Values.maxMessageBytes | replace "_" "" | quote }} - name: KAFKA_CFG_LOG_SEGMENT_BYTES value: {{ .Values.logSegmentBytes | replace "_" "" | quote }} - name: KAFKA_CFG_LOG_DIRS - value: {{ .Values.logsDirs }} + value: {{ .Values.logsDirs | quote }} - name: KAFKA_CFG_DEFAULT_REPLICATION_FACTOR value: {{ .Values.defaultReplicationFactor | quote }} - name: KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR @@ -204,13 +273,9 @@ spec: {{- if .Values.extraEnvVars }} {{ include "kafka.tplValue" ( dict "value" .Values.extraEnvVars "context" $) | nindent 12 }} {{- end }} - command: ["/bin/bash", "-c"] - args: - - export KAFKA_CFG_BROKER_ID=${MY_POD_NAME##*-}; - exec /entrypoint.sh /run.sh; ports: - name: kafka - containerPort: {{ .Values.service.port }} + containerPort: 9092 {{- if .Values.auth.ssl }} - name: kafka-ssl containerPort: 9093 @@ -219,50 +284,41 @@ spec: - name: kafka-external containerPort: 19092 {{- end }} - {{- if .Values.livenessProbe.enabled }} - livenessProbe: - tcpSocket: - port: kafka - initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.livenessProbe.periodSeconds }} - timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }} - successThreshold: {{ .Values.livenessProbe.successThreshold }} - failureThreshold: {{ .Values.livenessProbe.failureThreshold }} + {{- if .Values.livenessProbe }} + livenessProbe: {{- include "kafka.tplValue" (dict "value" .Values.livenessProbe "context" $) | nindent 12 }} {{- end }} - {{- if .Values.readinessProbe.enabled }} - readinessProbe: - tcpSocket: - port: kafka - initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} - periodSeconds: {{ .Values.readinessProbe.periodSeconds }} - timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }} - successThreshold: {{ .Values.readinessProbe.successThreshold }} - failureThreshold: {{ .Values.readinessProbe.failureThreshold }} + {{- if .Values.readinessProbe }} + readinessProbe: {{- include "kafka.tplValue" (dict "value" .Values.readinessProbe "context" $) | nindent 12 }} + {{- end }} + {{- if .Values.resources }} + resources: {{- toYaml .Values.resources | nindent 12 }} {{- end }} volumeMounts: - {{- if .Values.externalAccess.enabled }} + {{- if or .Values.config .Values.existingConfigmap }} + - name: kafka-config + mountPath: /opt/bitnami/kafka/conf/server.properties + subPath: server.properties + {{- end }} - name: scripts mountPath: /scripts/setup.sh subPath: setup.sh + {{- if and .Values.externalAccess.enabled .Values.externalAccess.autoDiscovery.enabled }} + - name: shared + mountPath: /shared {{- end }} {{- if .Values.persistence.enabled }} - name: data mountPath: /bitnami/kafka {{- end }} - {{- if .Values.config }} - - name: kafka-config - mountPath: /opt/bitnami/kafka/conf/server.properties - subPath: server.properties - {{- end }} - {{- if .Values.auth.enabled }} + {{- if .Values.auth.ssl }} - name: kafka-certificates mountPath: /opt/bitnami/kafka/conf/certs/ readOnly: true {{- end }} - {{ if .Values.metrics.jmx.enabled }} + {{- if .Values.metrics.jmx.enabled }} - name: jmx-exporter - image: "{{ template "kafka.metrics.jmx.image" . }}" - imagePullPolicy: "{{ .Values.metrics.jmx.image.pullPolicy }}" + image: {{ template "kafka.metrics.jmx.image" . }} + imagePullPolicy: {{ .Values.metrics.jmx.image.pullPolicy | quote }} command: - java - -XX:+UnlockExperimentalVMOptions @@ -271,64 +327,60 @@ spec: - -XshowSettings:vm - -jar - jmx_prometheus_httpserver.jar - - {{ .Values.metrics.jmx.exporterPort | quote }} + - "5556" - /etc/jmx-kafka/jmx-kafka-prometheus.yml ports: - name: metrics - containerPort: {{ .Values.metrics.jmx.exporterPort }} + containerPort: 5556 {{- if .Values.metrics.jmx.resources }} - resources: {{ toYaml .Values.metrics.jmx.resources | nindent 12 }} + resources: {{- toYaml .Values.metrics.jmx.resources | nindent 12 }} {{- end }} volumeMounts: - name: jmx-config mountPath: /etc/jmx-kafka - {{ end }} + {{- end }} + {{- if .Values.sidecars }} + {{- include "kafka.tplValue" (dict "value" .Values.sidecars "context" $) | nindent 8 }} + {{- end }} volumes: - {{- if .Values.externalAccess.enabled }} + {{- if or .Values.config .Values.existingConfigmap }} + - name: kafka-config + configMap: + name: {{ include "kafka.configmapName" . }} + {{- end }} - name: scripts configMap: name: {{ include "kafka.fullname" . }}-scripts defaultMode: 0755 + {{- if and .Values.externalAccess.enabled .Values.externalAccess.autoDiscovery.enabled }} + - name: shared + emptyDir: {} {{- end }} - {{ if .Values.metrics.jmx.enabled }} + {{- if .Values.metrics.jmx.enabled }} - name: jmx-config configMap: - {{- if .Values.metrics.jmx.configMap.overrideName }} - name: {{ .Values.metrics.jmx.configMap.overrideName }} - {{- else }} - name: {{ template "kafka.fullname" . }}-jmx-configuration - {{- end }} - {{ end }} - {{ if .Values.config }} - - name: kafka-config - configMap: - name: {{ template "kafka.fullname" . }}-configuration - {{ end }} - {{ if .Values.auth.enabled }} + name: {{ include "kafka.metrics.jmx.configmapName" . }} + {{- end }} + {{- if .Values.auth.ssl }} - name: kafka-certificates secret: - secretName: {{ required "A secret containing the Kafka JKS certificates is required when authentication in enabled" .Values.auth.certificatesSecret }} + secretName: {{ required "A secret containing the Kafka JKS certificates is required when SSL in enabled" .Values.auth.certificatesSecret }} defaultMode: 256 - {{ end }} + {{- end }} {{- if not .Values.persistence.enabled }} - name: data emptyDir: {} {{- else if .Values.persistence.existingClaim }} - name: data persistentVolumeClaim: -{{- with .Values.persistence.existingClaim }} - claimName: {{ tpl . $ }} -{{- end }} + claimName: {{ printf "%s" (tpl .Values.persistence.existingClaim .) }} {{- else }} volumeClaimTemplates: - metadata: name: data - {{- with .Values.persistence.annotations }} - annotations: - {{- range $key, $value := . }} - {{ $key }}: {{ $value }} + {{- if .Values.persistence.annotations }} + annotations: {{- include "kafka.tplValue" (dict "value" .Values.persistence.annotations "context" $) | nindent 10 }} {{- end }} - {{- end }} spec: accessModes: {{- range .Values.persistence.accessModes }} @@ -339,3 +391,4 @@ spec: storage: {{ .Values.persistence.size | quote }} {{ include "kafka.storageClass" . | nindent 8 }} {{- end }} +{{- end }} diff --git a/bitnami/kafka/templates/svc-external-access.yaml b/bitnami/kafka/templates/svc-external-access.yaml index 8a602b43f..f022f10a8 100644 --- a/bitnami/kafka/templates/svc-external-access.yaml +++ b/bitnami/kafka/templates/svc-external-access.yaml @@ -5,7 +5,6 @@ {{- range $i, $e := until $replicaCount }} {{- $targetPod := printf "%s-%d" (printf "%s" $fullName) $i }} - --- apiVersion: v1 kind: Service @@ -14,26 +13,31 @@ metadata: labels: {{- include "kafka.labels" $ | nindent 4 }} app.kubernetes.io/component: kafka pod: {{ $targetPod }} - annotations: {{ include "kafka.tplValue" ( dict "value" $root.Values.externalAccess.service.annotations "context" $) | nindent 4 }} + {{- if $root.Values.externalAccess.service.annotations }} + annotations: {{- include "kafka.tplValue" ( dict "value" $root.Values.externalAccess.service.annotations "context" $) | nindent 4 }} + {{- end }} spec: type: {{ $root.Values.externalAccess.service.type }} {{- if eq $root.Values.externalAccess.service.type "LoadBalancer" }} - {{- if $root.Values.externalAccess.service.loadBalancerIP }} - loadBalancerIP: {{ index $root.Values.externalAccess.service.loadBalancerIP $i }} + {{- if not (empty $root.Values.externalAccess.service.loadBalancerIPs) }} + loadBalancerIP: {{ index $root.Values.externalAccess.service.loadBalancerIPs $i }} + {{- end }} + {{- if $root.Values.externalAccess.service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: {{- toYaml $root.Values.service.loadBalancerSourceRanges | nindent 4 }} {{- end }} {{- end }} ports: - - name: kafka - {{- if eq $root.Values.externalAccess.service.type "LoadBalancer" }} + - name: tcp-kafka port: {{ $root.Values.externalAccess.service.port }} + {{- if not (empty $root.Values.externalAccess.service.nodePorts) }} + nodePort: {{ index $root.Values.externalAccess.service.nodePorts $i }} {{- else }} - port: {{ index $root.Values.externalAccess.service.nodePort $i }} + nodePort: null {{- end }} - {{- if $root.Values.externalAccess.service.nodePort }} - nodePort: {{ index $root.Values.externalAccess.service.nodePort $i }} - {{- end }} - {{- if eq $root.Values.externalAccess.service.type "LoadBalancer" }} + {{- if or (eq $root.Values.externalAccess.service.type "LoadBalancer") }} targetPort: kafka-external + {{- else if $root.Values.externalAccess.autoDiscovery.enabled }} + targetPort: null {{- else }} targetPort: {{ index $root.Values.externalAccess.service.nodePort $i }} {{- end }} diff --git a/bitnami/kafka/templates/svc-headless.yaml b/bitnami/kafka/templates/svc-headless.yaml index df126c4db..de6524f64 100644 --- a/bitnami/kafka/templates/svc-headless.yaml +++ b/bitnami/kafka/templates/svc-headless.yaml @@ -8,12 +8,14 @@ spec: type: ClusterIP clusterIP: None ports: - - name: kafka + - name: tcp-kafka port: {{ .Values.service.port }} + protocol: TCP targetPort: kafka {{- if .Values.auth.ssl }} - - name: kafka-ssl - port: 9093 + - name: tls-kafka + port: {{ .Values.service.sslPort }} + protocol: TCP targetPort: kafka-ssl {{- end }} selector: {{- include "kafka.matchLabels" . | nindent 4 }} diff --git a/bitnami/kafka/templates/svc.yaml b/bitnami/kafka/templates/svc.yaml index e75590ad7..12a48c3fd 100644 --- a/bitnami/kafka/templates/svc.yaml +++ b/bitnami/kafka/templates/svc.yaml @@ -4,25 +4,39 @@ metadata: name: {{ template "kafka.fullname" . }} labels: {{- include "kafka.labels" . | nindent 4 }} app.kubernetes.io/component: kafka - annotations: {{ include "kafka.tplValue" ( dict "value" .Values.service.annotations "context" $) | nindent 4 }} + {{- if .Values.service.annotations }} + annotations: {{- include "kafka.tplValue" (dict "value" .Values.service.annotations "context" $) | nindent 4 }} + {{- end }} spec: type: {{ .Values.service.type }} {{- if eq .Values.service.type "LoadBalancer" }} {{- if .Values.service.loadBalancerIP }} loadBalancerIP: {{ .Values.service.loadBalancerIP }} {{- end }} + {{- if .Values.service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: {{- toYaml .Values.service.loadBalancerSourceRanges | nindent 4 }} + {{- end }} {{- end }} ports: - - name: kafka + - name: tcp-kafka port: {{ .Values.service.port }} - {{- if and .Values.service.nodePort (or (eq .Values.service.type "NodePort") (eq .Values.service.type "LoadBalancer")) }} - nodePort: {{ .Values.service.nodePort }} - {{- end }} + protocol: TCP targetPort: kafka + {{- if and (or (eq .Values.service.type "NodePort") (eq .Values.service.type "LoadBalancer")) (not (empty .Values.service.nodePorts.kafka)) }} + nodePort: {{ .Values.service.nodePorts.kafka }} + {{- else if eq .Values.service.type "ClusterIP" }} + nodePort: null + {{- end }} {{- if .Values.auth.ssl }} - - name: kafka-ssl - port: 9093 + - name: tls-kafka + port: {{ .Values.service.sslPort }} + protocol: TCP targetPort: kafka-ssl + {{- if and (or (eq .Values.service.type "NodePort") (eq .Values.service.type "LoadBalancer")) (not (empty .Values.service.nodePorts.ssl)) }} + nodePort: {{ .Values.service.nodePorts.ssl }} + {{- else if eq .Values.service.type "ClusterIP" }} + nodePort: null + {{- end }} {{- end }} selector: {{- include "kafka.matchLabels" . | nindent 4 }} app.kubernetes.io/component: kafka diff --git a/bitnami/kafka/values-production.yaml b/bitnami/kafka/values-production.yaml index e1a9c455f..048781026 100644 --- a/bitnami/kafka/values-production.yaml +++ b/bitnami/kafka/values-production.yaml @@ -14,190 +14,202 @@ image: registry: docker.io repository: bitnami/kafka - tag: 2.4.1-debian-10-r8 + tag: 2.4.1-debian-10-r12 ## Specify a imagePullPolicy ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName ## - # pullSecrets: - # - myRegistryKeySecretName + pullSecrets: [] ## Set to true if you would like to see extra information on logs - ## It turns BASH and NAMI debugging in minideb - ## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging + ## debug: false ## String to partially override kafka.fullname template (will maintain the release name) +## # nameOverride: ## String to fully override kafka.fullname template +## # fullnameOverride: -## Init containers parameters: -## volumePermissions: Change the owner and group of the persistent volume mountpoint to runAsUser:fsGroup values from the securityContext section. +## Kubernetes Cluster Domain ## -volumePermissions: - enabled: false - image: - registry: docker.io - repository: bitnami/minideb - tag: buster - pullPolicy: Always - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ - ## - # pullSecrets: - # - myRegistryKeySecretName - resources: {} +clusterDomain: cluster.local -## StatefulSet controller supports automated updates. There are two valid update strategies: RollingUpdate and OnDelete -## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets +## Kafka Configuration +## Specify content for server.properties +## The server.properties is auto-generated based on other parameters when this paremeter is not specified ## -updateStrategy: RollingUpdate - -## Partition update strategy -## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions +## Example: +## config: |- +## broker.id=-1 +## listeners=PLAINTEXT://:9092 +## advertised.listeners=PLAINTEXT://KAFKA_IP:9092 +## num.network.threads=3 +## num.io.threads=8 +## socket.send.buffer.bytes=102400 +## socket.receive.buffer.bytes=102400 +## socket.request.max.bytes=104857600 +## log.dirs=/bitnami/kafka/data +## num.partitions=1 +## num.recovery.threads.per.data.dir=1 +## offsets.topic.replication.factor=1 +## transaction.state.log.replication.factor=1 +## transaction.state.log.min.isr=1 +## log.flush.interval.messages=10000 +## log.flush.interval.ms=1000 +## log.retention.hours=168 +## log.retention.bytes=1073741824 +## log.segment.bytes=1073741824 +## log.retention.check.interval.ms=300000 +## zookeeper.connect=ZOOKEEPER_SERVICE_NAME +## zookeeper.connection.timeout.ms=6000 +## group.initial.rebalance.delay.ms=0 ## -# rollingUpdatePartition: +# config: -## Limits the number of pods of the replicated application that are down simultaneously from voluntary disruptions -## The PDB will only be created if replicaCount is greater than 1 -## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions +## ConfigMap with Kafka Configuration +## NOTE: This will override config ## -podDisruptionBudget: - maxUnavailable: 1 +# existingConfigmap: -replicaCount: 3 - -config: |- -# broker.id=-1 -# listeners=PLAINTEXT://:9092 -# advertised.listeners=PLAINTEXT://KAFKA_IP:9092 -# num.network.threads=3 -# num.io.threads=8 -# socket.send.buffer.bytes=102400 -# socket.receive.buffer.bytes=102400 -# socket.request.max.bytes=104857600 -# log.dirs=/bitnami/kafka/data -# num.partitions=1 -# num.recovery.threads.per.data.dir=1 -# offsets.topic.replication.factor=1 -# transaction.state.log.replication.factor=1 -# transaction.state.log.min.isr=1 -# log.flush.interval.messages=10000 -# log.flush.interval.ms=1000 -# log.retention.hours=168 -# log.retention.bytes=1073741824 -# log.segment.bytes=1073741824 -# log.retention.check.interval.ms=300000 -# zookeeper.connect=ZOOKEEPER_SERVICE_NAME -# zookeeper.connection.timeout.ms=6000 -# group.initial.rebalance.delay.ms=0 - -## Kafka docker image available customizations -## https://github.com/bitnami/bitnami-docker-kafka#configuration -## ## Allow to use the PLAINTEXT listener. +## allowPlaintextListener: false -## The address the socket server listens on. -# listeners: +## The address(es) the socket server listens on. +## +listeners: [] -## Hostname and port the broker will advertise to producers and consumers. -# advertisedListeners: +## The address(es) (hostname:port) the broker will advertise to producers and consumers. +## +advertisedListeners: [] ## The protocol->listener mapping +## Example: +## listenerSecurityProtocolMap: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT +## # listenerSecurityProtocolMap: -## The listener that the brokers should communicate on +## Name of listener used for communication between brokers. +## # interBrokerListenerName: -## ID of the Kafka node. -brokerId: -1 +## Kafka broker ID. +## If unset, a unique broker id will be generated. +## +# brokerId: -1 -## Switch to enable topic deletion or not. -deleteTopicEnable: false - -## Kafka's Java Heap size. +## Kafka's Java Heap size +## heapOpts: -Xmx1024m -Xms1024m +## Switch to enable topic deletion or not. +## +deleteTopicEnable: false + ## The number of messages to accept before forcing a flush of data to disk. +## logFlushIntervalMessages: 10000 ## The maximum amount of time a message can sit in a log before we force a flush. +## logFlushIntervalMs: 1000 ## A size-based retention policy for logs. +## logRetentionBytes: _1073741824 ## The interval at which log segments are checked to see if they can be deleted. +## logRetentionCheckIntervalMs: 300000 ## The minimum age of a log file to be eligible for deletion due to age. +## logRetentionHours: 168 ## The maximum size of a log segment file. When this size is reached a new log segment will be created. +## logSegmentBytes: _1073741824 -## Log message format version -logMessageFormatVersion: "" - ## A comma separated list of directories under which to store log files. +## logsDirs: /bitnami/kafka/data ## The largest record batch size allowed by Kafka +## maxMessageBytes: _1000012 ## Default replication factors for automatically created topics +## defaultReplicationFactor: 3 ## The replication factor for the offsets topic +## offsetsTopicReplicationFactor: 3 ## The replication factor for the transaction topic +## transactionStateLogReplicationFactor: 3 ## Overridden min.insync.replicas config for the transaction topic +## transactionStateLogMinIsr: 3 ## The number of threads doing disk I/O. +## numIoThreads: 8 ## The number of threads handling network requests. +## numNetworkThreads: 3 ## The default number of log partitions per topic. +## numPartitions: 1 ## The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. +## numRecoveryThreadsPerDataDir: 1 ## The receive buffer (SO_RCVBUF) used by the socket server. +## socketReceiveBufferBytes: 102400 ## The maximum size of a request that the socket server will accept (protection against OOM). +## socketRequestMaxBytes: _104857600 ## The send buffer (SO_SNDBUF) used by the socket server. +## socketSendBufferBytes: 102400 ## Timeout in ms for connecting to zookeeper. +## zookeeperConnectionTimeoutMs: 6000 ## The endpoint identification algorithm to validate server hostname using server certificate. +## Disable server host name verification by setting it to an empty string +## See: https://docs.confluent.io/current/kafka/authentication_ssl.html#optional-settings +## sslEndpointIdentificationAlgorithm: https ## All the parameters from the configuration file can be overwritten by using environment variables with this format: KAFKA_CFG_{KEY} +## ref: https://github.com/bitnami/bitnami-docker-kafka#configuration +## Example: +## extraEnvVars: +## - name: KAFKA_CFG_BACKGROUND_THREADS +## value: "10" ## -# extraEnvVars: -# - name: KAFKA_CFG_BACKGROUND_THREADS -# value: "10" +extraEnvVars: [] ## Authentication parameteres ## https://github.com/bitnami/bitnami-docker-kafka#security @@ -207,164 +219,369 @@ auth: enabled: true ## Enable SSL to be used with brokers and consumers + ## # ssl: false - ## Name of the existing secret containing credentials for brokerUser, interBrokerUser and zookeeperUser. - # existingSecret: - ## Name of the existing secret containing the certificate files that will be used by Kafka. + ## # certificatesSecret: ## Password for the above certificates if they are password protected. + ## # certificatesPassword: ## Kafka client user. + ## brokerUser: user - ## Kafka client password. + ## Kafka client password + ## # brokerPassword: - ## Kafka inter broker communication user. + ## Kafka inter broker communication user + ## interBrokerUser: admin - ## Kafka inter broker communication password. + ## Kafka inter broker communication password + ## # interBrokerPassword: ## Kafka Zookeeper user. + ## # zookeeperUser: ## Kafka Zookeeper password. + ## # zookeeperPassword: -## Kubernetes Security Context -## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ + ## Name of the existing secret containing credentials for brokerUser, interBrokerUser and zookeeperUser. + ## + # existingSecret: + +## Number of Kafka brokers to deploy ## -securityContext: - enabled: true +replicaCount: 3 + +## StrategyType, can be set to RollingUpdate or OnDelete by default. +## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets +## +updateStrategy: RollingUpdate + +## Partition update strategy +## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions +## +# rollingUpdatePartition: + +## Pod annotations. Evaluated as a template +## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/ +## +podAnnotations: {} + +## Affinity for pod assignment. Evaluated as a template +## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity +## +affinity: {} + +## Node labels for pod assignment. Evaluated as a template +## ref: https://kubernetes.io/docs/user-guide/node-selection/ +## +nodeSelector: {} + +## Tolerations for pod assignment. Evaluated as a template +## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ +## +tolerations: [] + +## Kafka pods' Security Context +## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod +## +podSecurityContext: fsGroup: 1001 runAsUser: 1001 -# Cluster domain -clusterDomain: cluster.local +## Kafka containers' Security Context +## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container +## Example: +## containerSecurityContext: +## capabilities: +## drop: ["NET_RAW"] +## readOnlyRootFilesystem: true +## +containerSecurityContext: {} -## Kubernetes configuration -## For minikube, set this to NodePort, elsewhere use LoadBalancer +## Kafka containers' resource requests and limits +## ref: http://kubernetes.io/docs/user-guide/compute-resources/ +## +resources: + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + limits: {} + # cpu: 250m + # memory: 1Gi + requests: {} + # cpu: 250m + # memory: 256Mi + +## Kafka containers' liveness and readiness probes. Evaluated as a template. +## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes +## +livenessProbe: + tcpSocket: + port: kafka + initialDelaySeconds: 10 + timeoutSeconds: 5 + # failureThreshold: 3 + # periodSeconds: 10 + # successThreshold: 1 +readinessProbe: + tcpSocket: + port: kafka + initialDelaySeconds: 5 + failureThreshold: 6 + timeoutSeconds: 5 + # periodSeconds: 10 + # successThreshold: 1 + +## Pod Disruption Budget configuration +## The PDB will only be created if replicaCount is greater than 1 +## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions +## +pdb: + create: true + ## Min number of pods that must still be available after the eviction + ## + # minAvailable: 1 + ## Max number of pods that can be unavailable after the eviction + ## + maxUnavailable: 1 + +## Add sidecars to the pod. +## Example: +## sidecars: +## - name: your-image-name +## image: your-image +## imagePullPolicy: Always +## ports: +## - name: portname +## containerPort: 1234 +## +sidecars: {} + +## Service parameters ## service: + ## Service type + ## type: ClusterIP + ## Kafka port + ## port: 9092 - - ## Specify the NodePort value for the LoadBalancer and NodePort service types. + ## Kafka SSL port + ## + sslPort: 9093 + ## Specify the nodePort(s) value for the LoadBalancer and NodePort service types. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport ## - # nodePort: - - ## Use loadBalancerIP to request a specific static IP, + nodePorts: + kafka: "" + ssl: "" + ## Set the LoadBalancer service type to internal only. + ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer + ## # loadBalancerIP: - - ## Service annotations done as key:value pairs + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] + ## Provide any additional annotations which may be required. Evaluated as a template + ## annotations: {} +## External Access to Kafka brokers configuration +## externalAccess: + ## Enable Kubernetes external cluster access to Kafka brokers + ## enabled: false + + ## External IPs auto-discovery configuration + ## An init container is used to auto-detect LB IPs or node ports by querying the K8s API + ## Note: RBAC might be required + ## + autoDiscovery: + ## Enable external IPs auto-discovery + ## + enabled: false + ## Bitnami Kubectl image + ## ref: https://hub.docker.com/r/bitnami/kubectl/tags/ + ## + image: + registry: docker.io + repository: bitnami/kubectl + tag: 1.17.3-debian-10-r20 + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images + ## + pullPolicy: IfNotPresent + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] + ## Init Container resource requests and limits + ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ + ## + resources: + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + limits: {} + # cpu: 100m + # memory: 128Mi + requests: {} + # cpu: 100m + # memory: 128Mi + + ## Parameters to configure K8s service(s) used to externally access Kafka brokers + ## A new service per broker will be created + ## service: - ## Type of service for external access. It can be LoadBalancer or NodePort. + ## Service type. Allowed values: LoadBalancer or NodePort ## type: LoadBalancer ## Port used when service type is LoadBalancer ## port: 19092 - ## Array of load balancer IPs for each Kafka broker. Length must be the same as replicas + ## Array of load balancer IPs for each Kafka broker. Length must be the same as replicaCount + ## Example: + ## loadBalancerIPs: + ## - X.X.X.X + ## - Y.Y.Y.Y ## - loadBalancerIP: [] + loadBalancerIPs: [] + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] + ## Array of node ports used for each Kafka broker. Length must be the same as replicaCount + ## Example: + ## nodePorts: + ## - 30001 + ## - 30002 + ## + nodePorts: [] ## When service type is NodePort, you can specify the domain used for Kafka advertised listeners. - ## If not specified, the container will try to get the kubernetes node external IP using: 'curl -s https://ipinfo.io/ip' + ## If not specified, the container will try to get the kubernetes node external IP + ## # domain: mydomain.com - ## Array of node ports used for each Kafka broker. Length must be the same as replicas + ## Provide any additional annotations which may be required. Evaluated as a template ## - nodePort: [] - - ## Service annotations done as key:value pairs annotations: {} -## Service account for Kafka to use. -## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ -## -serviceAccount: - ## Specifies whether a ServiceAccount should be created - ## - create: false - ## The name of the ServiceAccount to use. - ## If not set and create is true, a name is generated using the kafka.fullname template - # name: - -## Kafka data Persistent Volume Storage Class -## If defined, storageClassName: -## If set to "-", storageClassName: "", which disables dynamic provisioning -## If undefined (the default) or set to null, no storageClassName spec is -## set, choosing the default provisioner. (gp2 on AWS, standard on -## GKE, AWS & OpenStack) +## Persistence paramaters ## persistence: enabled: true ## A manually managed Persistent Volume and Claim ## If defined, PVC must be created manually before volume will be bound - ## The value is evaluated as a template, so, for example, the name can depend on .Release or .Chart + ## The value is evaluated as a template ## # existingClaim: - + ## PV Storage Class + ## If defined, storageClassName: + ## If set to "-", storageClassName: "", which disables dynamic provisioning + ## If undefined (the default) or set to null, no storageClassName spec is + ## set, choosing the default provisioner. + ## # storageClass: "-" + ## PV Access Mode + ## accessModes: - ReadWriteOnce + ## PVC size + ## size: 8Gi + ## PVC annotations + ## annotations: {} -## Node labels for pod assignment -## Ref: https://kubernetes.io/docs/user-guide/node-selection/ +## Init Container paramaters +## Change the owner and group of the persistent volume(s) mountpoint(s) to 'runAsUser:fsGroup' on each component +## values from the securityContext section of the component ## -nodeSelector: {} +volumePermissions: + enabled: false + ## Bitnami Minideb image + ## ref: https://hub.docker.com/r/bitnami/minideb/tags/ + ## + image: + registry: docker.io + repository: bitnami/minideb + tag: buster + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images + ## + pullPolicy: Always + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] + ## Init Container resource requests and limits + ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ + ## + resources: + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + limits: {} + # cpu: 100m + # memory: 128Mi + requests: {} + # cpu: 100m + # memory: 128Mi -## Tolerations for pod assignment -## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ +## Kafka pods ServiceAccount +## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ ## -tolerations: [] +serviceAccount: + ## Specifies whether a ServiceAccount should be created + ## + create: true + ## The name of the ServiceAccount to use. + ## If not set and create is true, a name is generated using the fluentd.fullname template + ## + # name: -## Affinity for pod assignment -## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity +## Role Based Access +## ref: https://kubernetes.io/docs/admin/authorization/rbac/ ## -affinity: {} - -## Configure resource requests and limits -## ref: http://kubernetes.io/docs/user-guide/compute-resources/ -## -resources: {} -# limits: -# cpu: 200m -# memory: 1Gi -# requests: -# memory: 256Mi -# cpu: 250m - -## Configure extra options for liveness and readiness probes -## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes) -livenessProbe: - enabled: true - initialDelaySeconds: 10 - periodSeconds: 10 - timeoutSeconds: 5 - failureThreshold: 2 - successThreshold: 1 - -readinessProbe: - enabled: true - initialDelaySeconds: 5 - periodSeconds: 10 - timeoutSeconds: 5 - failureThreshold: 6 - successThreshold: 1 +rbac: + ## Specifies whether RBAC rules should be created + ## binding Kafka ServiceAccount to a role + ## that allows Kafka pods querying the K8s API + ## + create: false ## Prometheus Exporters / Metrics ## metrics: ## Prometheus Kafka Exporter: exposes complimentary metrics to JMX Exporter + ## kafka: enabled: true @@ -374,20 +591,19 @@ metrics: image: registry: docker.io repository: bitnami/kafka-exporter - tag: 1.2.0-debian-10-r52 - pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + tag: 1.2.0-debian-10-r55 + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## - # pullSecrets: - # - myRegistryKeySecretName - - ## Interval at which Prometheus scrapes metrics, note: only used by Prometheus Operator - interval: 10s - - ## Port kafka-exporter exposes for Prometheus to scrape metrics - port: 9308 + pullPolicy: IfNotPresent + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] ## Prometheus Kafka Exporter' resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ @@ -404,14 +620,8 @@ metrics: # cpu: 100m # memory: 128Mi - ## Annotations for the Kafka Exporter Prometheus metrics deployment + ## Service configuration ## - annotations: {} - - ## Annotations for the Kafka Exporter pods - ## - podAnnotations: {} - service: ## Kafka Exporter Service type ## @@ -427,6 +637,13 @@ metrics: ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer ## # loadBalancerIP: + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] ## Set the Cluster IP to use ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address ## @@ -449,22 +666,19 @@ metrics: image: registry: docker.io repository: bitnami/jmx-exporter - tag: 0.12.0-debian-10-r51 - pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + tag: 0.12.0-debian-10-r54 + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## - # pullSecrets: - # - myRegistryKeySecretName - - ## Interval at which Prometheus scrapes metrics, note: only used by Prometheus Operator - ## - interval: 10s - - ## Port jmx-exporter exposes Prometheus format metrics to scrape - ## - exporterPort: 5556 + pullPolicy: IfNotPresent + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] ## Prometheus JMX Exporter' resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ @@ -481,6 +695,8 @@ metrics: # cpu: 100m # memory: 128Mi + ## Service configuration + ## service: ## JMX Exporter Service type ## @@ -496,6 +712,13 @@ metrics: ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer ## # loadBalancerIP: + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] ## Set the Cluster IP to use ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address ## @@ -507,38 +730,8 @@ metrics: prometheus.io/port: "{{ .Values.metrics.jmx.exporterPort }}" prometheus.io/path: "/" - ## Credits to the incubator/kafka chart for the JMX configuration. - ## https://github.com/helm/charts/tree/master/incubator/kafka - ## - ## Rules to apply to the Prometheus JMX Exporter. Note while lots of stats have been cleaned and exposed, - ## there are still more stats to clean up and expose, others will never get exposed. They keep lots of duplicates - ## that can be derived easily. The configMap in this chart cleans up the metrics it exposes to be in a Prometheus - ## format, eg topic, broker are labels and not part of metric name. Improvements are gladly accepted and encouraged. - ## - configMap: - ## Allows disabling the default configmap, note a configMap is needed - ## - enabled: true - ## Allows setting values to generate confimap - ## To allow all metrics through (warning its crazy excessive) comment out below `overrideConfig` and set - ## `whitelistObjectNames: []` - ## - overrideConfig: {} - # jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:5555/jmxrmi - # lowercaseOutputName: true - # lowercaseOutputLabelNames: true - # ssl: false - # rules: - # - pattern: ".*" - ## If you would like to supply your own ConfigMap for JMX metrics, supply the name of that - ## ConfigMap as an `overrideName` here. - ## - overrideName: "" - ## Port the jmx metrics are exposed in native jmx format, not in Prometheus format - ## - jmxPort: 5555 - ## JMX Whitelist Objects, can be set to control which JMX metrics are exposed. Only whitelisted - ## values will be exposed via JMX Exporter. They must also be exposed via Rules. To expose all metrics + ## JMX Whitelist Objects, can be set to control which JMX metrics are exposed. Only whitelisted + ## values will be exposed via JMX Exporter. They must also be exposed via Rules. To expose all metrics ## (warning its crazy excessive and they aren't formatted in a prometheus style) (1) `whitelistObjectNames: []` ## (2) commented out above `overrideConfig`. ## @@ -549,7 +742,28 @@ metrics: - kafka.network:* - kafka.log:* - # Enable this if you're using https://github.com/coreos/prometheus-operator + ## Prometheus JMX exporter configuration + ## Specify content for jmx-kafka-prometheus.yml. Evaluated as a template + ## + ## Credits to the incubator/kafka chart for the JMX configuration. + ## https://github.com/helm/charts/tree/master/incubator/kafka + ## + config: |- + jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:5555/jmxrmi + lowercaseOutputName: true + lowercaseOutputLabelNames: true + ssl: false + {{- if .Values.metrics.jmx.whitelistObjectNames }} + whitelistObjectNames: ["{{ join "\",\"" .Values.metrics.jmx.whitelistObjectNames }}"] + {{- end }} + + ## ConfigMap with Prometheus JMX exporter configuration + ## NOTE: This will override metrics.jmx.config + ## + # existingConfigmap: + + ## Prometheus Operator ServiceMonitor configuration + ## serviceMonitor: enabled: false ## Namespace in which Prometheus is running @@ -579,8 +793,9 @@ zookeeper: metrics: enabled: true +## This value is only used when zookeeper.enabled is set to false +## externalZookeeper: - ## This value is only used when zookeeper.enabled is set to false - ## Server or list of external zookeeper servers to use. - servers: + ## + servers: [] diff --git a/bitnami/kafka/values.yaml b/bitnami/kafka/values.yaml index 026f45d11..96ddf1ba8 100644 --- a/bitnami/kafka/values.yaml +++ b/bitnami/kafka/values.yaml @@ -14,190 +14,202 @@ image: registry: docker.io repository: bitnami/kafka - tag: 2.4.1-debian-10-r8 + tag: 2.4.1-debian-10-r12 ## Specify a imagePullPolicy ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName ## - # pullSecrets: - # - myRegistryKeySecretName + pullSecrets: [] ## Set to true if you would like to see extra information on logs - ## It turns BASH and NAMI debugging in minideb - ## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging + ## debug: false ## String to partially override kafka.fullname template (will maintain the release name) +## # nameOverride: ## String to fully override kafka.fullname template +## # fullnameOverride: -## Init containers parameters: -## volumePermissions: Change the owner and group of the persistent volume mountpoint to runAsUser:fsGroup values from the securityContext section. +## Kubernetes Cluster Domain ## -volumePermissions: - enabled: false - image: - registry: docker.io - repository: bitnami/minideb - tag: buster - pullPolicy: Always - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ - ## - # pullSecrets: - # - myRegistryKeySecretName - resources: {} +clusterDomain: cluster.local -## StatefulSet controller supports automated updates. There are two valid update strategies: RollingUpdate and OnDelete -## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets +## Kafka Configuration +## Specify content for server.properties +## The server.properties is auto-generated based on other parameters when this paremeter is not specified ## -updateStrategy: RollingUpdate - -## Partition update strategy -## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions +## Example: +## config: |- +## broker.id=-1 +## listeners=PLAINTEXT://:9092 +## advertised.listeners=PLAINTEXT://KAFKA_IP:9092 +## num.network.threads=3 +## num.io.threads=8 +## socket.send.buffer.bytes=102400 +## socket.receive.buffer.bytes=102400 +## socket.request.max.bytes=104857600 +## log.dirs=/bitnami/kafka/data +## num.partitions=1 +## num.recovery.threads.per.data.dir=1 +## offsets.topic.replication.factor=1 +## transaction.state.log.replication.factor=1 +## transaction.state.log.min.isr=1 +## log.flush.interval.messages=10000 +## log.flush.interval.ms=1000 +## log.retention.hours=168 +## log.retention.bytes=1073741824 +## log.segment.bytes=1073741824 +## log.retention.check.interval.ms=300000 +## zookeeper.connect=ZOOKEEPER_SERVICE_NAME +## zookeeper.connection.timeout.ms=6000 +## group.initial.rebalance.delay.ms=0 ## -# rollingUpdatePartition: +# config: -## Limits the number of pods of the replicated application that are down simultaneously from voluntary disruptions -## The PDB will only be created if replicaCount is greater than 1 -## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions +## ConfigMap with Kafka Configuration +## NOTE: This will override config ## -podDisruptionBudget: - maxUnavailable: 1 +# existingConfigmap: -replicaCount: 1 - -config: |- -# broker.id=-1 -# listeners=PLAINTEXT://:9092 -# advertised.listeners=PLAINTEXT://KAFKA_IP:9092 -# num.network.threads=3 -# num.io.threads=8 -# socket.send.buffer.bytes=102400 -# socket.receive.buffer.bytes=102400 -# socket.request.max.bytes=104857600 -# log.dirs=/bitnami/kafka/data -# num.partitions=1 -# num.recovery.threads.per.data.dir=1 -# offsets.topic.replication.factor=1 -# transaction.state.log.replication.factor=1 -# transaction.state.log.min.isr=1 -# log.flush.interval.messages=10000 -# log.flush.interval.ms=1000 -# log.retention.hours=168 -# log.retention.bytes=1073741824 -# log.segment.bytes=1073741824 -# log.retention.check.interval.ms=300000 -# zookeeper.connect=ZOOKEEPER_SERVICE_NAME -# zookeeper.connection.timeout.ms=6000 -# group.initial.rebalance.delay.ms=0 - -## Kafka docker image available customizations -## https://github.com/bitnami/bitnami-docker-kafka#configuration -## ## Allow to use the PLAINTEXT listener. +## allowPlaintextListener: true -## The address the socket server listens on. -# listeners: +## The address(es) the socket server listens on. +## +listeners: [] -## Hostname and port the broker will advertise to producers and consumers. -# advertisedListeners: +## The address(es) (hostname:port) the broker will advertise to producers and consumers. +## +advertisedListeners: [] ## The protocol->listener mapping +## Example: +## listenerSecurityProtocolMap: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT +## # listenerSecurityProtocolMap: -## The listener that the brokers should communicate on +## Name of listener used for communication between brokers. +## # interBrokerListenerName: -## ID of the Kafka node. -# brokerId: -1 +## Kafka broker ID. +## If unset, a unique broker id will be generated. +## +brokerId: -1 -## Switch to enable topic deletion or not. -deleteTopicEnable: false - -## Kafka's Java Heap size. +## Kafka's Java Heap size +## heapOpts: -Xmx1024m -Xms1024m +## Switch to enable topic deletion or not. +## +deleteTopicEnable: false + ## The number of messages to accept before forcing a flush of data to disk. +## logFlushIntervalMessages: 10000 ## The maximum amount of time a message can sit in a log before we force a flush. +## logFlushIntervalMs: 1000 ## A size-based retention policy for logs. +## logRetentionBytes: _1073741824 ## The interval at which log segments are checked to see if they can be deleted. +## logRetentionCheckIntervalMs: 300000 ## The minimum age of a log file to be eligible for deletion due to age. +## logRetentionHours: 168 ## The maximum size of a log segment file. When this size is reached a new log segment will be created. +## logSegmentBytes: _1073741824 -## Log message format version -logMessageFormatVersion: "" - ## A comma separated list of directories under which to store log files. +## logsDirs: /bitnami/kafka/data ## The largest record batch size allowed by Kafka +## maxMessageBytes: _1000012 ## Default replication factors for automatically created topics +## defaultReplicationFactor: 1 ## The replication factor for the offsets topic +## offsetsTopicReplicationFactor: 1 ## The replication factor for the transaction topic +## transactionStateLogReplicationFactor: 1 ## Overridden min.insync.replicas config for the transaction topic +## transactionStateLogMinIsr: 1 ## The number of threads doing disk I/O. +## numIoThreads: 8 ## The number of threads handling network requests. +## numNetworkThreads: 3 ## The default number of log partitions per topic. +## numPartitions: 1 ## The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. +## numRecoveryThreadsPerDataDir: 1 ## The receive buffer (SO_RCVBUF) used by the socket server. +## socketReceiveBufferBytes: 102400 ## The maximum size of a request that the socket server will accept (protection against OOM). +## socketRequestMaxBytes: _104857600 ## The send buffer (SO_SNDBUF) used by the socket server. +## socketSendBufferBytes: 102400 ## Timeout in ms for connecting to zookeeper. +## zookeeperConnectionTimeoutMs: 6000 ## The endpoint identification algorithm to validate server hostname using server certificate. +## Disable server host name verification by setting it to an empty string +## See: https://docs.confluent.io/current/kafka/authentication_ssl.html#optional-settings +## sslEndpointIdentificationAlgorithm: https ## All the parameters from the configuration file can be overwritten by using environment variables with this format: KAFKA_CFG_{KEY} +## ref: https://github.com/bitnami/bitnami-docker-kafka#configuration +## Example: +## extraEnvVars: +## - name: KAFKA_CFG_BACKGROUND_THREADS +## value: "10" ## -# extraEnvVars: -# - name: KAFKA_CFG_BACKGROUND_THREADS -# value: "10" +extraEnvVars: [] ## Authentication parameteres ## https://github.com/bitnami/bitnami-docker-kafka#security @@ -207,166 +219,371 @@ auth: enabled: false ## Enable SSL to be used with brokers and consumers + ## # ssl: false - ## Name of the existing secret containing credentials for brokerUser, interBrokerUser and zookeeperUser. - # existingSecret: - ## Name of the existing secret containing the certificate files that will be used by Kafka. + ## # certificatesSecret: ## Password for the above certificates if they are password protected. + ## # certificatesPassword: ## Kafka client user. + ## brokerUser: user - ## Kafka client password. + ## Kafka client password + ## # brokerPassword: - ## Kafka inter broker communication user. + ## Kafka inter broker communication user + ## interBrokerUser: admin - ## Kafka inter broker communication password. + ## Kafka inter broker communication password + ## # interBrokerPassword: ## Kafka Zookeeper user. + ## # zookeeperUser: ## Kafka Zookeeper password. + ## # zookeeperPassword: -## Kubernetes Security Context -## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ + ## Name of the existing secret containing credentials for brokerUser, interBrokerUser and zookeeperUser. + ## + # existingSecret: + +## Number of Kafka brokers to deploy ## -securityContext: - enabled: true +replicaCount: 1 + +## StrategyType, can be set to RollingUpdate or OnDelete by default. +## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets +## +updateStrategy: RollingUpdate + +## Partition update strategy +## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions +## +# rollingUpdatePartition: + +## Pod annotations. Evaluated as a template +## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/ +## +podAnnotations: {} + +## Affinity for pod assignment. Evaluated as a template +## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity +## +affinity: {} + +## Node labels for pod assignment. Evaluated as a template +## ref: https://kubernetes.io/docs/user-guide/node-selection/ +## +nodeSelector: {} + +## Tolerations for pod assignment. Evaluated as a template +## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ +## +tolerations: [] + +## Kafka pods' Security Context +## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod +## +podSecurityContext: fsGroup: 1001 runAsUser: 1001 -# Cluster domain -clusterDomain: cluster.local +## Kafka containers' Security Context +## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container +## Example: +## containerSecurityContext: +## capabilities: +## drop: ["NET_RAW"] +## readOnlyRootFilesystem: true +## +containerSecurityContext: {} -## Kubernetes configuration -## For minikube, set this to NodePort, elsewhere use LoadBalancer +## Kafka containers' resource requests and limits +## ref: http://kubernetes.io/docs/user-guide/compute-resources/ +## +resources: + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + limits: {} + # cpu: 250m + # memory: 1Gi + requests: {} + # cpu: 250m + # memory: 256Mi + +## Kafka containers' liveness and readiness probes. Evaluated as a template. +## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes +## +livenessProbe: + tcpSocket: + port: kafka + initialDelaySeconds: 10 + timeoutSeconds: 5 + # failureThreshold: 3 + # periodSeconds: 10 + # successThreshold: 1 +readinessProbe: + tcpSocket: + port: kafka + initialDelaySeconds: 5 + failureThreshold: 6 + timeoutSeconds: 5 + # periodSeconds: 10 + # successThreshold: 1 + +## Pod Disruption Budget configuration +## The PDB will only be created if replicaCount is greater than 1 +## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions +## +pdb: + create: true + ## Min number of pods that must still be available after the eviction + ## + # minAvailable: 1 + ## Max number of pods that can be unavailable after the eviction + ## + maxUnavailable: 1 + +## Add sidecars to the pod. +## Example: +## sidecars: +## - name: your-image-name +## image: your-image +## imagePullPolicy: Always +## ports: +## - name: portname +## containerPort: 1234 +## +sidecars: {} + +## Service parameters ## service: + ## Service type + ## type: ClusterIP + ## Kafka port + ## port: 9092 - - ## Specify the NodePort value for the LoadBalancer and NodePort service types. + ## Kafka SSL port + ## + sslPort: 9093 + ## Specify the nodePort(s) value for the LoadBalancer and NodePort service types. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport ## - # nodePort: - - ## Use loadBalancerIP to request a specific static IP, + nodePorts: + kafka: "" + ssl: "" + ## Set the LoadBalancer service type to internal only. + ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer + ## # loadBalancerIP: - - ## Service annotations done as key:value pairs + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] + ## Provide any additional annotations which may be required. Evaluated as a template + ## annotations: {} +## External Access to Kafka brokers configuration +## externalAccess: + ## Enable Kubernetes external cluster access to Kafka brokers + ## enabled: false + + ## External IPs auto-discovery configuration + ## An init container is used to auto-detect LB IPs or node ports by querying the K8s API + ## Note: RBAC might be required + ## + autoDiscovery: + ## Enable external IP/ports auto-discovery + ## + enabled: false + ## Bitnami Kubectl image + ## ref: https://hub.docker.com/r/bitnami/kubectl/tags/ + ## + image: + registry: docker.io + repository: bitnami/kubectl + tag: 1.17.3-debian-10-r20 + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images + ## + pullPolicy: IfNotPresent + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] + ## Init Container resource requests and limits + ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ + ## + resources: + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + limits: {} + # cpu: 100m + # memory: 128Mi + requests: {} + # cpu: 100m + # memory: 128Mi + + ## Parameters to configure K8s service(s) used to externally access Kafka brokers + ## A new service per broker will be created + ## service: - ## Type of service for external access. It can be LoadBalancer or NodePort. + ## Service type. Allowed values: LoadBalancer or NodePort ## type: LoadBalancer ## Port used when service type is LoadBalancer ## port: 19092 - ## Array of load balancer IPs for each Kafka broker. Length must be the same as replicas + ## Array of load balancer IPs for each Kafka broker. Length must be the same as replicaCount + ## Example: + ## loadBalancerIPs: + ## - X.X.X.X + ## - Y.Y.Y.Y ## - loadBalancerIP: [] + loadBalancerIPs: [] + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] + ## Array of node ports used for each Kafka broker. Length must be the same as replicaCount + ## Example: + ## nodePorts: + ## - 30001 + ## - 30002 + ## + nodePorts: [] ## When service type is NodePort, you can specify the domain used for Kafka advertised listeners. - ## If not specified, the container will try to get the kubernetes node external IP using: 'curl -s https://ipinfo.io/ip' + ## If not specified, the container will try to get the kubernetes node external IP + ## # domain: mydomain.com - ## Array of node ports used for each Kafka broker. Length must be the same as replicas + ## Provide any additional annotations which may be required. Evaluated as a template ## - nodePort: [] - - ## Service annotations done as key:value pairs annotations: {} -## Service account for Kafka to use. -## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ -## -serviceAccount: - ## Specifies whether a ServiceAccount should be created - ## - create: false - ## The name of the ServiceAccount to use. - ## If not set and create is true, a name is generated using the kafka.fullname template - # name: - -## Kafka data Persistent Volume Storage Class -## If defined, storageClassName: -## If set to "-", storageClassName: "", which disables dynamic provisioning -## If undefined (the default) or set to null, no storageClassName spec is -## set, choosing the default provisioner. (gp2 on AWS, standard on -## GKE, AWS & OpenStack) +## Persistence paramaters ## persistence: enabled: true ## A manually managed Persistent Volume and Claim ## If defined, PVC must be created manually before volume will be bound - ## The value is evaluated as a template, so, for example, the name can depend on .Release or .Chart + ## The value is evaluated as a template ## # existingClaim: - + ## PV Storage Class + ## If defined, storageClassName: + ## If set to "-", storageClassName: "", which disables dynamic provisioning + ## If undefined (the default) or set to null, no storageClassName spec is + ## set, choosing the default provisioner. + ## # storageClass: "-" + ## PV Access Mode + ## accessModes: - ReadWriteOnce + ## PVC size + ## size: 8Gi + ## PVC annotations + ## annotations: {} -## Node labels for pod assignment -## Ref: https://kubernetes.io/docs/user-guide/node-selection/ +## Init Container paramaters +## Change the owner and group of the persistent volume(s) mountpoint(s) to 'runAsUser:fsGroup' on each component +## values from the securityContext section of the component ## -nodeSelector: {} +volumePermissions: + enabled: false + ## Bitnami Minideb image + ## ref: https://hub.docker.com/r/bitnami/minideb/tags/ + ## + image: + registry: docker.io + repository: bitnami/minideb + tag: buster + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images + ## + pullPolicy: Always + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] + ## Init Container resource requests and limits + ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ + ## + resources: + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + limits: {} + # cpu: 100m + # memory: 128Mi + requests: {} + # cpu: 100m + # memory: 128Mi -## Tolerations for pod assignment -## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ +## Kafka pods ServiceAccount +## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ ## -tolerations: [] +serviceAccount: + ## Specifies whether a ServiceAccount should be created + ## + create: true + ## The name of the ServiceAccount to use. + ## If not set and create is true, a name is generated using the fluentd.fullname template + ## + # name: -## Affinity for pod assignment -## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity +## Role Based Access +## ref: https://kubernetes.io/docs/admin/authorization/rbac/ ## -affinity: {} - -## Configure resource requests and limits -## ref: http://kubernetes.io/docs/user-guide/compute-resources/ -## -resources: {} -# limits: -# cpu: 200m -# memory: 1Gi -# requests: -# memory: 256Mi -# cpu: 250m - -## Configure extra options for liveness and readiness probes -## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes) -livenessProbe: - enabled: true - initialDelaySeconds: 10 - periodSeconds: 10 - timeoutSeconds: 5 - failureThreshold: 2 - successThreshold: 1 - -readinessProbe: - enabled: true - initialDelaySeconds: 5 - periodSeconds: 10 - timeoutSeconds: 5 - failureThreshold: 6 - successThreshold: 1 +rbac: + ## Specifies whether RBAC rules should be created + ## binding Kafka ServiceAccount to a role + ## that allows Kafka pods querying the K8s API + ## + create: false ## Prometheus Exporters / Metrics ## metrics: ## Prometheus Kafka Exporter: exposes complimentary metrics to JMX Exporter + ## kafka: enabled: false @@ -376,20 +593,19 @@ metrics: image: registry: docker.io repository: bitnami/kafka-exporter - tag: 1.2.0-debian-10-r52 - pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + tag: 1.2.0-debian-10-r55 + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## - # pullSecrets: - # - myRegistryKeySecretName - - ## Interval at which Prometheus scrapes metrics, note: only used by Prometheus Operator - interval: 10s - - ## Port kafka-exporter exposes for Prometheus to scrape metrics - port: 9308 + pullPolicy: IfNotPresent + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] ## Prometheus Kafka Exporter' resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ @@ -406,14 +622,8 @@ metrics: # cpu: 100m # memory: 128Mi - ## Annotations for the Kafka Exporter Prometheus metrics deployment + ## Service configuration ## - annotations: {} - - ## Annotations for the Kafka Exporter pods - ## - podAnnotations: {} - service: ## Kafka Exporter Service type ## @@ -429,6 +639,13 @@ metrics: ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer ## # loadBalancerIP: + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] ## Set the Cluster IP to use ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address ## @@ -451,22 +668,19 @@ metrics: image: registry: docker.io repository: bitnami/jmx-exporter - tag: 0.12.0-debian-10-r51 - pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + tag: 0.12.0-debian-10-r54 + ## Specify a imagePullPolicy + ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' + ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## - # pullSecrets: - # - myRegistryKeySecretName - - ## Interval at which Prometheus scrapes metrics, note: only used by Prometheus Operator - ## - interval: 10s - - ## Port jmx-exporter exposes Prometheus format metrics to scrape - ## - exporterPort: 5556 + pullPolicy: IfNotPresent + ## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace) + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Example: + ## pullSecrets: + ## - myRegistryKeySecretName + ## + pullSecrets: [] ## Prometheus JMX Exporter' resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ @@ -483,6 +697,8 @@ metrics: # cpu: 100m # memory: 128Mi + ## Service configuration + ## service: ## JMX Exporter Service type ## @@ -498,6 +714,13 @@ metrics: ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer ## # loadBalancerIP: + ## Load Balancer sources + ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service + ## Example: + ## loadBalancerSourceRanges: + ## - 10.10.10.0/24 + ## + loadBalancerSourceRanges: [] ## Set the Cluster IP to use ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address ## @@ -509,38 +732,8 @@ metrics: prometheus.io/port: "{{ .Values.metrics.jmx.exporterPort }}" prometheus.io/path: "/" - ## Credits to the incubator/kafka chart for the JMX configuration. - ## https://github.com/helm/charts/tree/master/incubator/kafka - ## - ## Rules to apply to the Prometheus JMX Exporter. Note while lots of stats have been cleaned and exposed, - ## there are still more stats to clean up and expose, others will never get exposed. They keep lots of duplicates - ## that can be derived easily. The configMap in this chart cleans up the metrics it exposes to be in a Prometheus - ## format, eg topic, broker are labels and not part of metric name. Improvements are gladly accepted and encouraged. - ## - configMap: - ## Allows disabling the default configmap, note a configMap is needed - ## - enabled: true - ## Allows setting values to generate confimap - ## To allow all metrics through (warning its crazy excessive) comment out below `overrideConfig` and set - ## `whitelistObjectNames: []` - ## - overrideConfig: {} - # jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:5555/jmxrmi - # lowercaseOutputName: true - # lowercaseOutputLabelNames: true - # ssl: false - # rules: - # - pattern: ".*" - ## If you would like to supply your own ConfigMap for JMX metrics, supply the name of that - ## ConfigMap as an `overrideName` here. - ## - overrideName: "" - ## Port the jmx metrics are exposed in native jmx format, not in Prometheus format - ## - jmxPort: 5555 - ## JMX Whitelist Objects, can be set to control which JMX metrics are exposed. Only whitelisted - ## values will be exposed via JMX Exporter. They must also be exposed via Rules. To expose all metrics + ## JMX Whitelist Objects, can be set to control which JMX metrics are exposed. Only whitelisted + ## values will be exposed via JMX Exporter. They must also be exposed via Rules. To expose all metrics ## (warning its crazy excessive and they aren't formatted in a prometheus style) (1) `whitelistObjectNames: []` ## (2) commented out above `overrideConfig`. ## @@ -551,7 +744,28 @@ metrics: - kafka.network:* - kafka.log:* - # Enable this if you're using https://github.com/coreos/prometheus-operator + ## Prometheus JMX exporter configuration + ## Specify content for jmx-kafka-prometheus.yml. Evaluated as a template + ## + ## Credits to the incubator/kafka chart for the JMX configuration. + ## https://github.com/helm/charts/tree/master/incubator/kafka + ## + config: |- + jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:5555/jmxrmi + lowercaseOutputName: true + lowercaseOutputLabelNames: true + ssl: false + {{- if .Values.metrics.jmx.whitelistObjectNames }} + whitelistObjectNames: ["{{ join "\",\"" .Values.metrics.jmx.whitelistObjectNames }}"] + {{- end }} + + ## ConfigMap with Prometheus JMX exporter configuration + ## NOTE: This will override metrics.jmx.config + ## + # existingConfigmap: + + ## Prometheus Operator ServiceMonitor configuration + ## serviceMonitor: enabled: false ## Namespace in which Prometheus is running @@ -582,8 +796,9 @@ metrics: zookeeper: enabled: true +## This value is only used when zookeeper.enabled is set to false +## externalZookeeper: - ## This value is only used when zookeeper.enabled is set to false - ## Server or list of external zookeeper servers to use. - servers: + ## + servers: []