github.com/piraeusdatastore/piraeus-operator

Added

• pv-hostpath: automatically determine on which nodes PVs should be created if no override is given.
• Automatically add labels on Kubernetes Nodes to LINSTOR satellites as Auxiliary Properties. This enables using Kubernetes labels for volume scheduling, for example using replicasOnSame: topology.kubernetes.io/zone.
• Support LINSTORs k8s backend by adding the necessary RBAC resources and documentation.
• Automatically create a LINSTOR passphrase when none is configured.
• Automatic eviction and deletion of offline satellites if the Kubernetes node object was also deleted.

Changed

• Default images:
  • quay.io/piraeusdatastore/piraeus-server:v1.17.0
  • quay.io/piraeusdatastore/piraeus-csi:v0.17.0
  • quay.io/piraeusdatastore/drbd9-bionic:v9.1.4
  • quay.io/piraeusdatastore/drbd-reactor:v0.4.4
• Recreates or updates to the satellite pods are now applied at once, instead of waiting for a node to complete before moving to the next.
• Enable CSI topology by default, allowing better volume scheduling with volumeBindingMode: WaitForFirstConsumer.
• Disable STORK by default. Instead, we recommend using volumeBindingMode: WaitForFirstConsumer in storage classes.

Added

• Allow CSI to work with distributions that use a kubelet working directory other than /var/lib/kubelet. See the csi.kubeletPath option.
• Enable Storage Capacity Tacking. This enables Kubernetes to base Pod scheduling decisions on remaining storage capacity. The feature is in beta and enabled by default starting with Kubernetes 1.21.

Changed

• Disable Stork Health Monitoring by default. Stork cannot distinguish between control plane and data plane issues, which can lead to instances where Stork will migrate a volume that is still mounted on another node, making the volume effectively unusable.

• Updated operator to kubernetes v1.21 components.

• Default images:

  • quay.io/piraeusdatastore/piraeus-server:v1.14.0
  • quay.io/piraeusdatastore/drbd9-bionic:v9.0.30
  • quay.io/piraeusdatastore/drbd-reactor:v0.4.3
  • quay.io/piraeusdatastore/piraeus-ha-controller:v0.2.0
  • external CSI images

Removed

• The cluster-wide snapshot controller is no longer deployed as a dependency of the piraeus-operator chart. Instead, separate charts are available on artifacthub.io that deploy the snapshot controller and extra validation for snapshot resources.

  The subchart was removed, as it unnecessarily tied updates of the snapshot controller to piraeus and vice versa. With the tightened validation starting with snapshot CRDs v1, moving the snapshot controller to a proper chart seems like a good solution.

Changed

• Default images:
  • Piraeus Server v1.13.0
  • Piraeus CSI v0.13.1
  • CSI Provisioner v2.1.2

Added

• All operator-managed workloads apply recommended labels. This requires the recreation of Deployments and DaemonSets on upgrade. This is automatically handled by the operator, however any customizations applied to the deployments not managed by the operator will be reverted in the process.
• Use drbd-reactor to expose Prometheus endpoints on each satellite.
• Configure ServiceMonitor resources if they are supported by the cluster (i.e. prometheus operator is configured)

Changed

• CSI Nodes no longer use hostNetwork: true. The pods already got the correct hostname via the downwardAPI and do not talk to DRBD's netlink interface directly.
• External: CSI snapshotter subchart now packages v1 CRDs. Fixes deprecation warnings when installing the snapshot controller.
• Default images:
  • Piraeus Server v1.12.3
  • Piraeus CSI v0.13.0
  • DRBD v9.0.29

Added

• Additional environment variables and Linstor properties can now be set in the LinstorController CRD.
• Set node name variable for Controller Pods, enabling k8s-await-election to correctly set up the endpoint for hairpin mode.

Fixed

• Update the network address of controller pods if they diverged between Linstor and kubernetes. This can happen after a node restart, where a pod is recreated with the same name but different IP address.

Added

• New guide on host preparation here.

Changed

• Default image updated:
  • operator.satelliteSet.kernelModuleInjectionImage: quay.io/piraeusdatastore/drbd9-bionic:v9.0.27
  • operator.satelliteSet.satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.11.1
  • operator.controller.controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.11.1
  • haController.image: quay.io/piraeusdatastore/piraeus-ha-controller:v0.1.3
  • pv-hostpath: chownerImage: quay.io/centos/centos:8

Added

• New component: haController will deploy the Piraeus High Availability Controller. More information is available in the optional components page
• Enable strict checking of DRBD parameter to disable usermode helper in container environments.
• Override the image used in "chown" jobs in the pv-hostpath chart by using --set chownerImage=<my-image>.

Changed

• Updated operator-sdk to v0.19.4
• Set CSI component timeout to 1 minute to reduce the number of retries in the CSI driver
• Default images updated:
  • operator.controller.controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.11.0
  • operator.satelliteSet.satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.11.0
  • operator.satelliteSet.kernelModuleInjectionImage: quay.io/piraeusdatastore/drbd9-bionic:v9.0.26
  • csi.pluginImage: quay.io/piraeusdatastore/piraeus-csi:v0.11.0

Fixed

• Fixed Helm warnings when setting "csi.controllerAffinity", "operator.controller.affinity" and "operator.satelliteSet.storagePools".

Added

• storagePools can now also set up devices similar to automaticStorageType, but with more fine grained control. See the updated storage guide
• New Helm options to disable creation of LinstorController and LinstorSatelliteSet resource operator.controller.enabled and operator.satelliteSet.enabled.
• New Helm option to override the generated controller endpoint: controllerEndpoint
• Allow overriding the default securityContext on a component basis:
  • etcd.podsecuritycontext sets the securityContext of etcd pods
  • stork.podsecuritycontext sets the securityContext of stork plugin and scheduler pods
  • csi-snapshotter.podsecuritycontext sets the securityContext of the CSI-Snapshotter pods
  • operator.podsecuritycontext sets the securityContext of the operator pods
• Example settings for openshift
• LINSTOR controller runs with additional GID 1000, to ensure write access to log directory

Changed

• Fixed a bug in pv-hostpath where permissions on the created directory are not applied on all nodes.
• Volumes created by pv-hostpath are now group writable. This makes them easier to integrate with fsGroup settings.
• Default value for affinity on LINSTOR controller and CSI controller changed. The new default is to distribute the pods across all available nodes.
• Default value for tolerations for etcd pods changed. They are now able to run on master nodes.
• Updates to LinstorController, LinstorSatelliteSet and LinstorCSIDriver are now propagated across all created resources
• Updated default images:
  • csi sidecar containers updated (compatible with Kubernetes v1.17+)
  • LINSTOR 1.10.0
  • LINSTOR CSI 0.10.0

Deprecation

• Using automaticStorageType is deprecated. Use the storagePools values instead.

Breaking

• The LINSTOR controller image given in operator.controller.controllerImage has to have its entrypoint set to k8s-await-election v0.2.0 or newer. Learn more in the upgrade guide.

Added

• LINSTOR controller can be started with multiple replicas. See operator.controller.replicas. NOTE: This requires support from the container. You need piraeus-server:v1.8.0 or newer.
• The pv-hostpath helper chart automatically sets up permissions for non-root etcd containers.
• Disable securityContext enforcement by setting global.setSecurityContext=false.
• Add cluster roles to work with OpenShift's SCC system.
• Control volume placement and accessibility by using CSIs Topology feature. Controlled by setting csi.enableTopology.
• All pods use a dedicated service account to allow for fine-grained permission control.
• The new helm section psp.* can automatically configure the ServiceAccount of all components to use the appropriate PSP roles.

Changed

• Default values:
  • operator.controller.controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.9.0
  • operator.satelliteSet.satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.9.0
  • operator.satelliteSet.kernelModuleInjectionImage: quay.io/piraeusdatastore/drbd9-bionic:v9.0.25
  • stork.storkImage: docker.io/openstorage/stork:2.5.0
• linstor-controller no longer starts in a privileged container.

Removed

• legacy CRDs (LinstorControllerSet, LinstorNodeSet) have been removed.
• v1alpha CRD versions have been removed.
• default pull secret drbdiocred removed. To keep using it, use --set drbdRepoCred=drbdiocred.

Added

• v1 of all CRDs
• Central value for controller image pull policy of all pods. Use --set global.imagePullPolicy=<value> on helm deployment.
• charts/piraeus/values.cn.yaml a set of helm values for faster image download for CN users.
• Allow specifying resource requirements for all pods. In helm you can set:
  • etcd.resources for etcd containers
  • stork.storkResources for stork plugin resources
  • stork.schedulerResources for the kube-scheduler deployed for use with stork
  • csi-snapshotter.resources for the cluster snapshotter controller
  • csi.resources for all CSI related containers. for brevity, there is only one setting for ALL CSI containers. They are all stateless go process which use the same amount of resources.
  • operator.resources for operator containers
  • operator.controller.resources for LINSTOR controller containers
  • operator.satelliteSet.resources for LINSTOR satellite containers
  • operator.satelliteSet.kernelModuleInjectionResources for kernel module injector/builder containers
• Components deployed by the operator can now run with multiple replicas. Components elect a leader, that will take on the actual work as long as it is active. Should one pod go down, another replica will take over. Currently these components support multiple replicas:
  • etcd => set etcd.replicas to the desired count
  • stork => set stork.replicas to the desired count for stork scheduler and controller
  • snapshot-controller => set csi-snapshotter.replicas to the desired count for cluster-wide CSI snapshot controller
  • csi-controller => set csi.controllerReplicas to the desired count for the linstor CSI controller
  • operator => set operator.replicas to have multiple replicas of the operator running
• Reference docs for all helm settings. Link
• stork.schedulerTag can override the automatically chosen tag for the kube-scheduler image. Previously, the tag always matched the kubernetes release.

Changed

• Renamed LinstorNodeSet to LinstorSatelliteSet. This brings the operator in line with other LINSTOR resources. Existing LinstorNodeSet resources will automatically be migrated to LinstorSatelliteSet.
• Renamed LinstorControllerSet to LinstorController. The old name implied the existence of multiple (separate) controllers. Existing LinstorControllerSet resources will automatically be migrated to LinstorController.
• Helm values renamed to align with new CRD names:
  • operator.controllerSet to operator.controller
  • operator.nodeSet to operator.satelliteSet
• Node scheduling no longer relies on linstor.linbit.com/piraeus-node labels. Instead, all CRDs support setting pod affinity and tolerations. In detail:
  • linstorcsidrivers gained 4 new resource keys, with no change in default behaviour:
    • nodeAffinity affinity passed to the csi nodes
    • nodeTolerations tolerations passed to the csi nodes
    • controllerAffinity affinity passed to the csi controller
    • controllerTolerations tolerations passed to the csi controller
  • linstorcontrollerset gained 2 new resource keys, with no change in default behaviour:
    • affinity affinity passed to the linstor controller pod
    • tolerations tolerations passed to the linstor controller pod
  • linstornodeset gained 2 new resource keys, with change in default behaviour:
    • affinity affinity passed to the linstor controller pod
    • tolerations tolerations passed to the linstor controller pod

• Controller is now a Deployment instead of StatefulSet.
• Renamed kernelModImage to kernelModuleInjectionImage
• Renamed drbdKernelModuleInjectionMode to KernelModuleInjectionMode

Added

• Support volume resizing with newer CSI versions.

• A new Helm chart csi-snapshotter that deploys extra components needed for volume snapshots.

• Add new kmod injection mode DepsOnly. Will try load kmods for LINSTOR layers from the host. Deprecates None.

• Automatic deployment of Stork scheduler configured for LINSTOR.

Removed

Changed

• Replaced bitnami/etcd dependency with vendored custom version Some important keys for the etcd helm chart have changed:
  • statefulset.replicaCount -> replicas
  • persistence.enabled -> persistentVolume.enabled
  • persistence.size -> persistentVolume.storage
  • àuth.rbac was removed: use tls certificates
  • auth.peer.useAutoTLS was removed
  • envVarsConfigMap was removed
  • When using etcd with TLS enabled:
    • For peer communication, peers need valid certificates for *.<release-name>-etcd (was .<release-name>>-etcd-headless.<namespace>.svc.cluster.local)
    • For client communication, servers need valid certificates for *.<release-name>-etcd (was .<release-name>>-etcd.<namespace>.svc.cluster.local)

Added

• Automatic storage pool creation via automaticStorageType on LinstorNodeSet. If this option is set, LINSTOR will create a storage pool based on all available devices on a node.

Changed

• Moved storage documentation to the storage guide
• Helm: update default images

Added

• Secured database connection for Linstor: When using the etcd connector, you can specify a secret containing a CA certificate to switch from HTTP to HTTPS communication.
• Secured connection between Linstor components: You can specify TLS keys to secure the communication between controller and satellite
• Secure storage with LUKS: You can specify the master passphrase used by Linstor when creating encrypted volumes when installing via Helm.
• Authentication with etcd using TLS client certificates.
• Secured connection between linstor-client and controller (HTTPS). More in the security guide
• Linstor controller endpoint can now be customized for all resources. If not specified, the old default values will be filled in.

Removed

• NodeSet service (piraeus-op-ns) was replaced by the ControllerSet service (piraeus-op-cs) everywhere

Changed

• CSI storage driver setup: move setup from helm to go operator. This is mostly an internal change. These changes may be of note if you used a non-default CSI configuration:
  • helm value csi.image was renamed to csi.pluginImage
  • CSI deployment can be controlled by a new resource linstorcsidrivers.piraeus.linbit.com
• PriorityClasses are not automatically created. When not specified, the priority class is:
  • "system-node-critical", if deployed in "kube-system" namespace
  • default PriorityClass in other namespaces
• RBAC rules for CSI: creation moved to deployment step (Helm/OLM). ServiceAccounts should be specified in CSI resource. If no ServiceAccounts are named, the implicitly created accounts from previous deployments will be used.
• Helm: update default images

Changed

• Use single values for images in CRDs instead of specifying the version separately
• Helm: Use single values for images instead of specifying repo, name and version separately
• Helm: Replace fixed storage pool configuration with list
• Helm: Do not create any storage pools by default
• Helm: Replace operator.nodeSet.spec and operator.controllerSet.spec by just operator.nodeSet and operator.controllerSet.

Changed

• Fix reporting of errors in LinstorControllerSet status

Changed

• Helm: Update LINSTOR server dependencies to fix startup problems

Added

• Helm: Allow an existing database to be used instead of always setting up a dedicated etcd instance

Changed

• Rename etcdURL parameter of LinstorControllerSet to dbConnectionURL to reflect the fact that it can be used for any database type
• Upgrade to operator-sdk v0.16.0
• Helm: Create multiple volumes with a single pv-hostchart installation
• Helm: Update dependencies

Added

• Helm: Add support for hostPath PersistentVolume persistence of etcd

Removed

• Helm: Remove vendored etcd chart from repository

Changed

• Rename CRDs from Piraeus* to Linstor*
• Make priority classes configurable
• Fix LINSTOR Controller/Satellite arguments
• Helm: Make etcd persistent by default
• Helm: Fix deployment of permissions objects into a non-default namespace
• Helm: Set default etcd size to 1Gi
• Helm: Update dependent image versions
• Docker: Change base image to Debian Buster

Added

• Initial public version with docs