<#import "/templates/guide.adoc" as tmpl>
<#import "/templates/kc.adoc" as kc>
<#import "/templates/options.adoc" as opts>
<#import "/templates/links.adoc" as links>
<#import "/templates/profile.adoc" as profile>

<@tmpl.guide
title="Advanced configuration"
summary="Tune advanced aspects of the Keycloak CR.">

== Advanced configuration
This {section} describes how to use Custom Resources (CRs) for advanced configuration of your {project_name} deployment.

=== Server configuration details

Many server options are exposed as first-class citizen fields in the Keycloak CR. The structure of the CR is based on the configuration structure of {project_name}. For example, to configure the `https-port` of the server, follow a
similar pattern in the CR and use the `httpsPort` field. The following example is a complex server configuration; however, it illustrates the relationship between server options and the Keycloak CR:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  db:
    vendor: postgres
    usernameSecret:
      name: usernameSecret
      key: usernameSecretKey
    passwordSecret:
      name: passwordSecret
      key: passwordSecretKey
    host: host
    database: database
    port: 123
    schema: schema
    poolInitialSize: 1
    poolMinSize: 2
    poolMaxSize: 30
  http:
    httpEnabled: true
    httpPort: 8180
    httpsPort: 8543
    tlsSecret: my-tls-secret
  hostname:
    hostname: https://my-hostname.tld
    admin: https://my-hostname.tld/admin
    strict: false
    backchannelDynamic: true
  features:
    enabled:
      - docker
      - authorization
    disabled:
      - admin
      - step-up-authentication
  transaction:
    xaEnabled: false
----

WARNING: These values are illustrative; choose values appropriate for your environment and workload.

For a list of options, see the Keycloak CRD, and for guidance on each option and its acceptable values, see <@links.server id="all-config"/>.

==== Additional options

Some expert server options are unavailable as dedicated fields in the Keycloak CR. The following are examples of omitted fields:

* Fields that require deep understanding of the underlying {project_name} implementation
* Fields that are not relevant to
<@profile.ifCommunity>
a Kubernetes
</@profile.ifCommunity>
<@profile.ifProduct>
an OpenShift
</@profile.ifProduct>
environment
* Fields for provider configuration because they are dynamic based on the used provider implementation

The `additionalOptions` field of the Keycloak CR enables {project_name} to accept any available configuration in the form of key-value pairs.
You can use this field to include any option that is omitted in the Keycloak CR.
For details on configuring options, see <@links.server id="all-config"/>.

The values can be expressed as plain text strings or Secret object references as shown in this example:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  ...
  additionalOptions:
    - name: spi-connections-http-client--default--connection-pool-size
      secret: # Secret reference
        name: http-client-secret # Name of the Secret
        key: poolSize # Name of the Key in the Secret
    - name: spi-email-template--mycustomprovider--enabled
      value: 'true' # Plain text value. Use quotes for numbers and boolean values.
----
NOTE:  The name format of options defined in this way is identical to the key format of options specified in the configuration file.
       For details on various configuration formats, see <@links.server id="configuration"/>.

NOTE: If you configure mTLS with `https-client-auth` set to `required`, this configuration is inherited by the management interface.
If you have not otherwise configured the usage of HTTP for the health endpoints, you will likely want to also set the additionalOption `https-management-client-auth` to `request` or `none` so that a valid client certificate is not required for probe requests.

==== Custom environment variables

You may find a need to set custom environment variables - such as for theme properties or `kc.[sh|bat]` script variables.
The `spec.env` field of the Keycloak CR allows you to directly set any environment variable.
Logic in the operator based upon looking for value of a particular setting does not consult `spec.env`,
therefore do not use `spec.env` for anything that has a first-class configuration in the CR or may be specified as an `additionalOption`.

Here's an example setting JAVA_OPTS_APPEND:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  ...
  env:
    - name: JAVA_OPTS_APPEND
      value: -Djava.net.preferIPv6Addresses=true
----

Similar to `additionalOptions` you may specify either a value or reference a Secret.

=== Secret References

Secret References are used by some dedicated options in the Keycloak CR, such as a `tlsSecret`, or as a value in `additionalOptions`.

Similarly ConfigMap References are used by options such as the `configMapFile`.

When specifying a Secret or ConfigMap Reference, make sure that a Secret or ConfigMap containing the referenced keys is present in the same namespace as the CR referencing it.

The operator will poll approximately every minute for changes to referenced Secrets or ConfigMaps. When a meaningful change is detected, the Operator performs a rolling restart of the {project_name} Deployment to pick up the changes.

=== Unsupported features

The `unsupported` field of the CR contains highly experimental configuration options that are not completely tested and are Tech Preview.

==== Pod Template

The Pod Template is a raw API representation that is used for the Deployment Template.
This field is a temporary workaround in case no supported field exists at the top level of the CR for your use case.

The Operator merges the fields of the provided template with the values generated by the Operator for the specific Deployment.
With this feature, you have access to a high level of customizations. However, no guarantee exists that the Deployment will work as expected.

The following example illustrates injecting labels, annotations, volumes, and volume mounts:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  ...
  unsupported:
    podTemplate:
      metadata:
        labels:
          my-label: "keycloak"
      spec:
        containers:
          - volumeMounts:
              - name: test-volume
                mountPath: /mnt/test
        volumes:
          - name: test-volume
            secret:
              secretName: keycloak-additional-secret
----

===== Probe Configuration

The Keycloak CR exposes options to set periodSeconds and failureThreshold on each of the three probes (readiness, liveness and startup)

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  readinessProbe:
    periodSeconds: 20
    failureThreshold: 5
  livenessProbe:
    periodSeconds: 20
    failureThreshold: 5
  startupProbe:
    periodSeconds: 20
    failureThreshold: 5
----

=== Disabling required options

{project_name} and the {project_name} Operator provide the best production-ready experience with security in mind.
However, during the development phase, you can disable key security features.

Specifically, you can disable the hostname and TLS as shown in the following example:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  ...
  http:
    httpEnabled: true
  hostname:
    strict: false
----

=== Resource requirements

The Keycloak CR allows specifying the `resources` options for managing compute resources for the {project_name} container.
It provides the ability to request and limit resources independently for the main Keycloak deployment via the Keycloak CR, and for the realm import Job via the Realm Import CR.

When no values are specified, the default `requests` memory is set to `1700MiB`, and the `limits` memory is set to `2GiB`.
These values were chosen based on a deeper analysis of {project_name} memory management.

If no values are specified in the Realm Import CR, it falls back to the values specified in the Keycloak CR, or to the defaults as defined above.

You can specify your custom values based on your requirements as follows:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  ...
  resources:
    requests:
      cpu: 1200m
      memory: 896Mi
    limits:
      cpu: 6
      memory: 3Gi
----

Moreover, the {project_name} container manages the heap size more effectively by providing relative values for the heap size.
It is achieved by providing certain JVM options.

For more details, see <@links.server id="containers" />.

=== Scheduling

You may control several aspects of the server Pod scheduling via the Keycloak CR. The scheduling stanza exposes optional standard Kubernetes affinity, tolerations, topology spread constraints, and the priority class name to fine tune the scheduling and placement of your server Pods.

An example utilizing all scheduling fields:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  scheduling:
    priorityClassName: custom-high
    affinity:
      podAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app: keycloak
                app.kubernetes.io/managed-by: keycloak-operator
                app.kubernetes.io/component: server
                topologyKey: topology.kubernetes.io/zone
              weight: 10
    tolerations:
    - key: "some-taint"
      operator: "Exists"
      effect: "NoSchedule"
    topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
      ...
  ...
----

Please see https://kubernetes.io/docs/concepts/scheduling-eviction[the kubernetes docs] for more on scheduling concepts.

If you do not specify a custom topology spread constraint, your Pods will have a spread constraint across zones and nodes to improve availability. These defaults do not affect import and update Jobs, only server Pods.

==== Job Scheduling

Realm import and update Jobs inherit the scheduling of the server Pods if it is specified in the Keycloak CR `spec.scheduling` field. If your server Pod scheduling is highly restrictive this may result in the
Job Pods being unschedulable. In this situation you may override the Job scheduling using the Keycloak CR `spec.update.scheduling` and `spec.import.scheduling` fields.

For example the following will remove the use of any server Pod scheduling for the update Job:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  update:
    scheduling: {}
  ...
----

=== Management Interface

To change the port of the management interface, use the first-class citizen field `httpManagement.port` in the Keycloak CR.
To change the properties of the management interface, you can do it by providing `additionalOptions` field.

You can specify the `port` and the `additionalOptions` as follows:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  httpManagement:
    port: 9001
  additionalOptions:
    - name: http-management-relative-path
      value: /management
----

NOTE: If you are using a custom image, the Operator is *unaware* of any configuration options that might've been specified there.
For instance, the management interface may use `https`, but the Operator accesses it via `http` when the TLS settings are specified in the custom image.
To ensure proper TLS configuration, use the `tlsSecret` and `truststores` fields in the Keycloak CR so that the Operator can reflect that.

For more details, see <@links.server id="management-interface" />.

=== Truststores

If you need to provide trusted certificates, the Keycloak CR provides a top level feature for configuring the server's truststore as discussed in <@links.server id="keycloak-truststore"/>.

Use the truststores stanza of the Keycloak spec to specify Secrets or ConfigMaps containing PEM encoded files, or PKCS12 files with extension `.p12`, `.pfx`, or `.pkcs12`, for example:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  ...
  truststores:
    my-truststore:
      secret:
        name: my-secret
----

Where the contents of my-secret could be a PEM file, for example:

[source,yaml]
------
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
stringData:
  cert.pem: |
    -----BEGIN CERTIFICATE-----
    ...
------

When running on a Kubernetes or OpenShift environment, the server automatically includes well-known locations of trusted certificates if they are present:
`/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` and `/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt`.

This behavior is now implemented by the server and enabled by default. You can disable it using server options in the Keycloak CR `additionalOptions`, for example:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: example-kc
spec:
  additionalOptions:
    - name: truststore-kubernetes-enabled
      value: "false"
----

To prevent the service account token from being mounted in the pod, set `automountServiceAccountToken` to `false` in the Keycloak CR spec.
This also disables the Kubernetes CA truststore auto-discovery described above.
This option cannot be `false` if you plan to use an external Infinispan cluster, the Kubernetes service accounts identity provider, or any custom provider logic that expects to implicitly use the Kubernetes API.

=== Admin Bootstrapping

When you create a new instance the Keycloak CR spec.bootstrapAdmin stanza may be used to configure the bootstrap user and/or service account. If you do not specify anything for the spec.bootstrapAdmin, the operator will create a Secret named "metadata.name"-initial-admin with a username temp-admin and a generated password. If you specify a Secret name for the bootstrap admin user, then the Secret will need to contain `username` and `password` key value pairs. If you specify a Secret name for bootstrap admin service account, then the Secret will need to contain `client-id` and `client-secret` key value pairs.

If a master realm has already been created for your cluster, then the spec.bootstrapAdmin is effectively ignored. If you need to create a recovery admin account, then you'll need to run the CLI command against a Pod directly.

For more information on how to bootstrap a temporary admin user or service account and recover lost admin access, refer to the <@links.server id="bootstrap-admin-recovery"/> guide.

=== Tracing (OpenTelemetry)

Tracing allows for detailed monitoring of each request's lifecycle, which helps quickly identify and diagnose issues, leading to more efficient debugging and maintenance.

You can change tracing configuration via Keycloak CR fields as follows:

[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  tracing:
    enabled: true                             # default 'false'
    endpoint: http://my-tracing:4317          # default 'http://localhost:4317'
    samplerType: parentbased_traceidratio     # default 'traceidratio'
    samplerRatio: 0.01                        # default '1'
    resourceAttributes:
      some.attribute: something
  additionalOptions:
    - name: tracing-jdbc-enabled
      value: false                            # default 'true'
    - name: tracing-header-My-Custom-header
      value: keycloak-custom-header
    - name: tracing-header-Authorization
      secret:
        name: tracing-secret
        key: token

----

These fields should reflect 1:1 association with `tracing-*` options that contain more information.

NOTE: The `tracing-jdbc-enabled` is not promoted as a first-class citizen as it might not be well managed in the future, so it needs to be set via the `additionalOptions` field.

For more details about tracing, see <@links.observability id="tracing" />.

=== Network Policies

NetworkPolicies allow you to specify rules for traffic flow within your cluster, and also between Pods and the outside world.
Your cluster must use a network plugin that supports NetworkPolicy enforcement to restrict the network traffic.

The operator automatically creates a NetworkPolicy to deny access to the clustering port of your {project_name} Pods.
The HTTP(S) endpoint is open to traffic from any namespace and the outside world.

To disable the NetworkPolicy, set `spec.networkPolicy.enabled` in your Keycloak CR, as shown in the example below.

.Keycloak CR with Network Policies enabled
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  networkPolicy:
    enabled: false
----

By default, traffic to the HTTP endpoints and the management endpoint is allowed from all sources.
The Keycloak CR can be extended to include a list of rules for each of the endpoints exposed by {project_name}.
These rules specify from where (the source) the traffic is allowed, and it is possible to communicate with the {project_name} Pods.

.Extended Network Policy configuration
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  networkPolicy:
    enabled: true
    http: <list of rules> #<1>
    https: <list of rules> #<2>
    management: <list of rules> #<3>
----
<1> It defines the rules for HTTP endpoint (port 8080 by default).
Due to security reasons, the HTTP endpoint is disabled by default.
<2> It defines the access rules for HTTPS endpoint (port 8443 by default.
<3> It defines the access rules for management endpoint (port 9000 by default).
The management endpoint is used by the Kubernetes Probes and to expose the {project_name} metrics.

The rule syntax is the same as the one used by the Kubernetes Network Policy.
It makes it easy to migrate your existing rules into your Keycloak CP.
For more information, check the https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors[rule syntax].

==== Example with OpenShift

For a concrete example, let's imagine we have a {project_name} deployment running in a OpenShift cluster.
Users have to access {project_name} to login, so {project_name} must be accessible from the Internet.

To make this example more interesting, let's assume the {project_name} is monitored too.
The monitoring is enabled as described in this OpenShift documentation page:
https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/monitoring/configuring-user-workload-monitoring#enabling-monitoring-for-user-defined-projects_preparing-to-configure-the-monitoring-stack-uwm[enabling Monitoring for user defined projects].

Based on those requirements, the Keycloak CR would be like this (most parts are omitted, like DB connection and security):

.Keycloak CR
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  ingress:
    enabled: true #<1>
  networkPolicy:
    enabled: true
    https:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-ingress #<2>
    management:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-user-workload-monitoring #<3>
----
<1> Enables Ingress for outside access.
<2> The default OpenShift Ingress class pods are running in `openshift-ingress` namespace.
We allow traffic from these pods to access the {project_name} HTTPS endpoint.
The traffic from outside the OpenShift cluster goes through these pods.
<3> Prometheus pods are running in `openshift-user-workload-monitoring`.
They need to access {project_name} to scrape the available metrics.

Check the https://kubernetes.io/docs/concepts/services-networking/network-policies/[Kubernetes Network Policies documentation] for more information about NetworkPolicies.

=== Parameterizing service labels and annotations

If you need to set custom labels or annotations to keycloak service you can do that through `spec.http.labels` and `spec.http.annotations`

.Custom service labels and annotations
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  http:
    labels:
      label1: label-value1
      label2: label-value2
    annotations:
      annotation1: annotation-value1
      annotation2: annotation-value2
----

=== Configuring the Kubernetes Service name and port

By default, the operator creates a Kubernetes Service named `<cr-name>-service` (for example, `example-kc-service`) whose port numbers match the container ports.

You can override the Service name and expose different port numbers on the Service while the Pods continue to listen on their original ports.
This is useful when you want a cleaner URL (for example, `https://keycloak.mynamespace.svc` instead of `https://keycloak-service.mynamespace.svc:8443`), or when migrating from an existing deployment where the Service name must remain stable.

.Custom Service name and port
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: example-kc
spec:
  http:
    httpsPort: 8443          # port the pod listens on (default: 8443)
    serviceHttpsPort: 443    # port exposed on the Service (default: same as httpsPort)
    serviceName: keycloak    # overrides the default "example-kc-service"
----

The resulting Service will be:

[source,yaml]
----
metadata:
  name: keycloak
spec:
  ports:
  - name: https
    port: 443
    targetPort: 8443
----

All three fields are optional. Omitting them preserves the existing behavior exactly.
Similarly, `serviceHttpPort` can be used to override the port for the HTTP endpoint.

=== ServiceMonitor
A `ServiceMonitor` resource is used to define how a service's metrics are discovered and scraped by Prometheus.
The {project_name} Operator automatically generates a `ServiceMonitor` resource for your deployment. In order for the
`ServiceMonitor` resource to be created, at least version `v0.72.0` of the `monitoring.coreos.com/v1:ServiceMonitor`
Custom Resource Definition (CRD) must be installed on your {kubernetes} cluster.

A ServiceMonitor is only created if the `metrics-enabled` option is set.

.Enable Keycloak metrics
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  additionalOptions:
    - name: metrics-enabled
      value: "true"
----

The Operator generates a `ServiceMonitor` with the following spec:

.Default ServiceMonitor
[source,yaml]
----
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-kc # <1>
  namespace: example-namespace # <2>
spec:
  endpoints:
  - interval: 30s
    path: /metrics # <3>
    port: management
    scheme: https # <4>
    scrapeTimeout: 10s
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - example-namespace # <2>
  selector:
    matchLabels:
      app: keycloak
      app.kubernetes.io/instance: example-kc # <2>
      app.kubernetes.io/managed-by: keycloak-operator
  scrapeProtocols:
    - OpenMetricsText1.0.0
----
<1> The `ServiceMonitor` is created with the same name as the Keycloak CR.
<2> The `ServiceMonitor` is always deployed to the same namespace as the Keycloak CR and will only match a Service
in that namespace.
<3> The configured path defaults to `/metrics`, but respects the `http-management-relative-path` value if configured.
<4> The configured scheme is changed automatically depending on whether TLS is enabled or not.


==== Modify ServiceMonitor configuration
You can configure the `interval` and `scrapeTimeout` fields and also `annotations` and `labels` of the generated `ServiceMonitor` by modifying the Keycloak CR spec.

.Modified ServiceMonitor
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  serviceMonitor:
    interval: 10s
    scrapeTimeout: 5s
    labels:
      label1: label-value1
      label2: label-value2
    annotations:
      annotation1: annotation-value1
      annotation2: annotation-value2
----

NOTE: You can use service monitor labels to determine which Prometheus instance will scrape metrics from {project_name}.

==== Disable ServiceMonitor creation
You can prevent the Operator from creating a ServiceMonitor by configuring the Keycloak CR as follows:

.Disabled ServiceMonitor
[source,yaml]
----
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
  name: example-kc
spec:
  serviceMonitor:
    enabled: false
----

</@tmpl.guide>
