github.com/openshift/installer

Azure disk selection fix for hive

Pre-release tag for k8s pkg bump

Installer 5.0 Dev Branch Opens

4.22 Engineering Candidate 5

4.21 Release Candidate 2

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.4.21 Pre-release

Includes CRD definitions for hive usage.

Tag for konflux for acb3420

Added

• Documentation for user-provided infrastructure on bare metal.

Changed

• Authorized SSH keys are now supplied via installer-generated MachineConfig manifests instead of the stub Ignition configurations. Additional MachineConfig manifests may be provided during a staged install.
• [The certificate signer][kubecsr] used for etcd bootstrapping is now sourced from the release image instead of from quay.io/coreos/kube-etcd-signer-server.
• The pinned RHCOS bootimage has been bumped from 400.7.20190306.0 to 410.8.20190325.0 to transition from a RHEL 7 base to RHEL 8.
• The detailed networking configuration manifest has been moved from networkconfigs.networkoperator.openshift.io to networks.operator.openshift.io.
• When installation fails after bootstrap removal, the installer will now pass along the status reported by [the cluster-version operator][cluster-version-operator], to make it easier to identify underlying issues.
• On AWS, when specific availability zones are requested for all machine pools in install-config.yaml, the installer will now only create per-zone resources for those zones. This allows for clusters in high-zone regions like us-east-1 without requiring limit bumps.
• On OpenStack, we have restored the ability to create Machine(Set)s with trunk support enabled.
• On OpenStack, machines are now tagged with Name and openshiftClusterID.
• Several cleanups to docs, internal code, and user-provided infrastructure support.

Fixed

• On AWS, creating a new cluster using the same Kubernetes API URL as an existing cluster will now error out instead of clobbering public Route 53 records for the existing cluster. This adds an additional install-time permission requirement: s3:GetBucketObjectLockConfiguration.
• On AWS, install-config validation now requires any explicitly configured machine-pool zones to be in the configured platform region. Installs never worked for zones from other regions, but the improved validation gives a more obivous error message and avoids partially provisioning a cluster before hitting the error.

Added

• We now initialize TLS certificates for etcd metrics over TCP on port 9979. We also store etcd certificates in various kube-system Secrets and ConfigMaps.
• The Kubernetes client is now extracted from the release image's kube-client-agent reference, replacing quay.io/coreos/kube-client-agent.
• On the control-plane and compute nodes, CRI-O's pause image is now extracted from the release image's pod reference.
• Initial work for user-provided infrastructure on AWS and vSphere, including a new user-provided-infrastructure subcommand.

Changed

• The install-config version has been bumped to v1beta4 for changes to more closely align with Network.config.openshift.io:
  • serviceCIDR is now serviceNetwork.
  • clusterNetworks is now clusterNetwork.
  • type is now networkType.
  • hostSubnetLength is now hostPrefix. v1beta3 is deprecated and will be removed in a future release.
• On AWS and OpenStack, ports 9000 through 9999 are now open for UDP. They had been open for TCP since 0.4.0, with a bugfix for 9990 -> 9999 in 0.13.0.
• On AWS, we now create network interfaces for the control-plane nodes explicitly, which allows for faster resource-creation time by allowing greater parallelization.
• On AWS, we now ask the machine-API operator to use aws-cloud-credentials (created by [the credential operator][credential-operator]) to fulfill our Machine(Set)s.
• On OpenStack, resources are prefixed with the cluster ID to avoid conflicts when running multiple clusters under the same tenant.
• On OpenStack, machines are now configured with hostnames to allow inter-VM communication.
• On OpenStack, machines are now configured with default DNS nameservers (1.1.1.1 and 208.67.222.222).
• Several doc and internal cleanups.

Fixed

• On AWS, the credentials-checking logic now allows root credentials, although it logs a warning because this approach is not recommended.
• On AWS, we only consider available zones when calculating defaults. This reduces the chance of errors from attempting resource creation in impaired or unavailable zones, although there's still a possibility for a zone going unavailable after our check but before resource creation.
• On AWS, the bootstrap machine is now created in the first public subnet, restoring SSH and journald access, and fixing a bug from 0.14.0.
• On AWS, the Kubernetes API load balancers now use /readyz instead of /healthz for health checks, which allows for more graceful control-plane rotation.
• On AWS, destroy cluster has some fixes for:
  • Removing snapshots associated with copied AMIs, fixing a bug from 0.14.0.
  • Deleting network interfaces, where we now remove all network interfaces in an owned VPC regardless of whether those network interfaces were themselves tagged as owned.
  • Instance termination, where we now attempt to terminate instances which are stopped, stopping, or shutting down in addition to those which are pending or running.
  • Instance profiles (which cannot be tagged directly) are now removed by name in a final deletion step, covering cases where they slipped through tag-based deletion because some external actor removed both the referencing instances and roles but left the instance profiles.
  • InvalidGroup.NotFound is now caught and considered a succesful deletion in more situations than with previous releases.
  • Error handling where subsequent successes no longer mask earlier errors.
  • Rate-limiting delete cycles, to reduce excessive AWS API usage (and associated throttling) while waiting for removed dependencies to resolve.
• On OpenStack, Machine(Set)s now use the correct security group name.
• On OpenStack, we now set api and *.apps DNS entries for internal IPs when a floating IP is not configured.
• The none platform no longer creates Machine(Set)s, because there is, by definition, no machine-API support for that platform.

Removed

• The deprecated cluster-config-v1 ConfigMap no longer contains the pull secret, now that all pull-secret consumers have been migrated to the coreos-pull-secret Secret.
• On AWS, control-plane nodes no longer allow ingress on ports 12379 or 12380 (which had, in the distant past, been used for etcd bootstrapping).

Changed

• A new, long-lived, self-signed certificate authority has been added to sign kubelet certificate-signing requests. This works around the current lack of certificate rotation in the machine-config operator.
• Machine(Set) labels have been migrated from sigs.k8s.io/cluster-api-... to machine.openshift.io, continuing the transition begun in 0.13.0.
• On AWS, control-plane nodes are now based on encrypted AMIs. These AMIs are copied into the target account from unencrypted, public AMIs provided by Red Hat. To support the copy and post-cluster cleanup, the installer requires the following additional AWS credentials: ec2:CopyImage, ec2:DeregisterImage, and ec2:DeleteSnapshot. 0.14.0 doesn't actually clean up the snapshots associated with the copied AMIs yet, but we have a fix for that landed for the next release. In the meantime, you should manually prune your snapshots after destroying a cluster.
• On AWS, the security-group simplification from 0.13.1 accidentially removed global SSH access to the bootstrap machine. We've fixed that with this release. Unfortunately, this release also moves the bootstrap machine into the same subnet as the first control-plane node, and since 0.13.0, control-plane nodes are in private subnets. So SSH access to the bootstrap machine from outside the cluster is still broken, but we've landed a fix to get it working again in the next release. In the meantime, you can set up a SSH bastion or debug pod if you need SSH access to cluster machines.
• On OpenStack, the Machine(Set)s have been updated to track provider changes. For example, the SecurityGroups schema has changed, as has the schema for selecting subnets.
• Several doc and internal cleanups.

Fixed

• On AWS, we now respect the availability zones configured in the control-plane Machine manifests, which are in turn fed by the install-config (previously control-plane nodes were always striped over zones regardless of the configuration).
• On AWS, the credentials-checking logic now uses the standard logger instead of creating its own custom logger.

Changed

• The aggregator and etcd-client certificate authorities are now self-signed authorities decoupled from the root certificate authority, continuing the transition begun in 0.13.0.
• On AWS, Route 53 A records for the API load balancer no longer use health checks.
• On AWS, the security group configuration has been simplified, with several stale rules being removed.

Fixed

• When rendering manifests before pushing them to the cluster, the bootstrap machine now correctly cleans up broken renders before re-rendering.
• The bootstrap machine now uses an etcdctl referenced from the release image, instead of hard-coding its own version.

Removed

• The nominal install-config compatibility with v1beta1 and v1beta2 has been removed, so the installer will error out if provided with an older install-config.yaml. v1beta1 was deprecated in 0.12.0 and v1beta2 was deprecated in 0.13.0. In both cases, the installer would ignore removed properties but not error out.

Added

• When cluster-creation times out waiting for cluster-version completion, the installer now logs the last failing-operator message (if any).
• The installer now invokes the [cluster-config operator][cluster-config-operator] on the bootstrap machine to generate config.openshift.io custom resource definitions.

Changed

• The install-config version has been bumped from v1beta2 to v1beta3. All users will need to update any saved install-config.yaml to use the new schema.

  • machines has been split into controlPlane and compute. Multiple compute pools are now supported (previously, only a single worker pool was supported). Every compute pool will use the same Ignition configuration. The installer will warn about but allow configurations where there are zero compute replicas.
  • On libvirt, the masterIPs property has been removed, since you cannot configure master IPs via the libvirt machine API provider.
  • On OpenStack, there is also a new lbFloatingIP property, which allows you to provide an IP address to be used by the load balancer. This allows you to create local DNS entries ahead of time before calling create cluster.

• Cluster domain names have been adjusted so that the cluster lives entirely within a per-cluster subdomain. This keeps split-horizon DNS from masking other clusters with the same base domain.

• The cluster-version update URL has been changed from the dummy http://localhost:8080/graph to the functioning https://api.openshift.com/api/upgrades_info/v1/graph and the channel has been changed from fast to stable-4.0, to opt clusters in to 4.0 upgrades.

• Machine-API resources have been moved from cluster.k8s.io to machine.openshift.io to clarify our divergence from the upstream types while they are unstable. The openshift-cluster-api namespace has been replaced with openshift-machine-api as well.

• The installer now uses etcd and OS images referenced by the update payload when configuring the machine-config operator.

• The etcd, aggregator, and other certificate authorities are now self-signed, decoupling their chains of trust from the root certificate authority.

• The installer no longer creates a service-serving certificate authority. The certificate authority is now created by the [service-CA operator][service-ca-operator].

• On AWS, the worker IAM role permissions were reduced to a smaller set required for kubelet initialization.

• On AWS, the worker security group has been expanded to allow ports 9000-9999 for for host network services. This matches the approach we have been using for masters since 0.4.0. The master security group has also been adjusted to fix a 9990 -> 9999 typo from 0.4.0.

• On libvirt, the default compute nodes have been bumped from 2 to 4 GiB of memory and the control-plane nodes have been bumped from 4 to 6 GiB of memory and 2 to 4 vCPUs.

• Several doc and internal cleanups and minor fixes.

Fixed

• The router certificate authority is appended to the admin kubeconfig to fix the OAuth flow behind oc login.

• The install-config.yaml validation is now more robust, with the installer:

  • Validating cluster names (it previously only validated cluster names provided via the install-config wizard).
  • Validating networking.clusterNetworks[].cidr and explicitly checking for nil machineCIDR and serviceCIDR.

• Terraform variables are now generated from master machine configurations instead of from the install configuration. This allows them to reflect changes made by editing master machine configurations during staged installs.

• metadata.json is generated before the Terraform invocation, fixing a bug introduced in 0.12.0 which made it hard to clean up after failed Terraform creation.

• The machine-config server has moved its Ignition-config service from port 49500 to 22623 to avoid the dynamic-port range starting at [49152][rfc-6335-s6].

• When the installer prompts for AWS credentials, it now respects AWS_PROFILE and will update an existing credentials file instead of erroring out.

• On AWS, the default [instance types][aws-instance-types] now depend on the selected region, with regions that do not support m4 types falling back to m5.

• On AWS, the installer now verifies that the user-supplied credentials have sufficient permissions for creating a cluster. Previously, permissions issues would surface as Terraform errors or broken cluster functionality after a nominally successful install.

• On AWS, the destroy cluster implementation is now more robust, fixing several bugs from 0.10.1:

  • The destroy code now checks for nil before dereferencing, avoiding panics when removing internet gateways which had not yet been associated with a VPC, and in other similar cases.
  • The destoy code now treats already-deleted instances as successfully deleted, instead of looping forever while trying to delete them.
  • The destroy code now treats a non-existant public DNS zone as success, instead of looping forever while trying to delete records from it.

• On AWS and OpenStack, there is a new infra ID that is a uniqified, possibly-abbreviated form of the cluster name. The infra ID is used to name and tag cluster resources, allowing for multiple clusters that share the same cluster name in a single account without naming conflicts (beyond DNS conflicts if both clusters also share the same base domain).

• On OpenStack, the HAProxy configuration on the service VM now only balances ports 80 and 443 across compute nodes (it used to also balance them across control-plane nodes).

• On OpenStack, the service VM now uses CoreDNS instead of dnsmasq. And it now includes records for *.apps.{cluster-domain} and the Kubernetes API.

• On OpenStack, the service VM has been moved to its own subnet.

Removed

• On AWS, control-plane nodes have been moved to private subnets and no longer have public IPs. Use a VPN or bastion host if you need SSH access to them.

Changed

• We now wait for [ClusterVersion][ClusterVersion] to report all operators as available before returning from create cluster.
• We now configure the network operator via networks.config.openshift.io and reserve networkconfigs.networkoperator.openshift.io for lower-level configuration (although we still generate it as well).
• We now set apiServerURL and etcdDiscoveryDomain in infrastructures.config.openshift.io.
• Release binaries are now stripped, which dramatically reduces their size. Builds with MODE=dev remain unstripped if you want to attach a debugger.
• On AWS, destroy cluster no longer depends directly on the cluster name (although it still depends on the cluster name indirectly via the kubernetes.io/cluster/{name} tag). This makes it easier to reconstruct metadata.json for destroy cluster if you accidentally removed the file before destroying your cluster.
• On AWS, the default worker MachineSets have been bumped to 120 GiB volumes to increase our baseline performance from on [gp2's sliding IOPS scale][aws-ebs-gp2-iops]. The new default worker volumes match our master bump from 0.5.0.
• On OpenStack, the HAProxy configuration on the service VM is dynamically updated as masters and workers are added and removed. This supports console access, among other things.
• Several doc and internal cleanups.

Fixed

• We no longer write distracting ERROR: logging before flag.Parse... messages from our underlying Kubernetes libraries.
• On loading install-config.yaml, we now error on CIDRs whose IP is not at the beginning of the masked subnet. For example, we now error for 192.168.126.10/24, since the beginning of that subnet is 192.168.126.0.
• On loading install-config.yaml, we now fill in defaults for replicas when it is unset or explicitly null.
• We have fixed some issues with round-tripping assets between the installer and the asset directory which lead to the reloaded assets being falsely identified as dirty and rebuilt.
• On OpenStack, a new security rule exposes port 443 to allow OpenShift web-console access.
• On OpenStack, credentials secret generation now respects the install configuration's cloud value, and the secret name has been updated from openstack-creds to openstack-credentials.
• On OpenStack, the local-dns service will now restart on failure (e.g. when the initial image pull fails) and it no longer sets the name of the container (so we can always re-run it without running into duplicate name issues).

Removed

• On loading install-config.yaml, the installer no longer restricts networking.type to a known value. If the network operator sees an unrecognized type, it assumes the user is configurating networking and doesn't react.
• We no longer seed ~core/.bash_history on the bootstrap node, as part of becoming less opinionated about which users are present on the underlying operating system.
• On AWS, the iamRoleName machine-pool property is gone, and the podCIDR networking property (deprecated in 0.4.0) is gone. The install-config version has been bumped from v1beta1 to v1beta2. All users, regardless of platform, will need to update any saved install-config.yaml to use the new version. IAM roles are being replaced by [the credential operator][credential-operator], and while we still create IAM roles for our master, worker, and bootstrap machines, we're removing the user-facing property now to avoid making this breaking change later.
• On AWS, the bootstrap machine security group allowing kubelet access (added in 0.10.1) has been removed. Static pod logs should soon be available from journald (although they aren't yet).

Changed

• We now wait for [ClusterVersion][ClusterVersion] to report all operators as available before returning from create cluster.
• We now configure the network operator via networks.config.openshift.io and reserve networkconfigs.networkoperator.openshift.io for lower-level configuration (although we still generate it as well).
• We now set apiServerURL and etcdDiscoveryDomain in infrastructures.config.openshift.io.
• Release binaries are now stripped, which dramatically reduces their size. Builds with MODE=dev remain unstripped if you want to attach a debugger.
• On AWS, destroy cluster no longer depends directly on the cluster name (although it still depends on the cluster name indirectly via the kubernetes.io/cluster/{name} tag). This makes it easier to reconstruct metadata.json for destroy cluster if you accidentally removed the file before destroying your cluster.
• On AWS, the default worker MachineSets have been bumped to 120 GiB volumes to increase our baseline performance from on [gp2's sliding IOPS scale][aws-ebs-gp2-iops]. The new default worker volumes match our master bump from 0.5.0.
• On OpenStack, the HAProxy configuration on the service VM is dynamically updated as masters and workers are added and removed. This supports console access, among other things.
• Several doc and internal cleanups.

Fixed

• We no longer write distracting ERROR: logging before flag.Parse... messages from our underlying Kubernetes libraries.
• On loading install-config.yaml, we now error on CIDRs whose IP is not at the beginning of the masked subnet. For example, we now error for 192.168.126.10/24, since the beginning of that subnet is 192.168.126.0.
• On loading install-config.yaml, we now fill in defaults for replicas when it is unset or explicitly null.
• We have fixed some issues with round-tripping assets between the installer and the asset directory which lead to the reloaded assets being falsely identified as dirty and rebuilt.
• On OpenStack, a new security rule exposes port 443 to allow OpenShift web-console access.
• On OpenStack, credentials secret generation now respects the install configuration's cloud value, and the secret name has been updated from openstack-creds to openstack-credentials.
• On OpenStack, the local-dns service will now restart on failure (e.g. when the initial image pull fails) and it no longer sets the name of the container (so we can always re-run it without running into duplicate name issues).

Removed

• On loading install-config.yaml, the installer no longer restricts networking.type to a known value. If the network operator sees an unrecognized type, it assumes the user is configurating networking and doesn't react.
• We no longer seed ~core/.bash_history on the bootstrap node, as part of becoming less opinionated about which users are present on the underlying operating system.
• On AWS, the iamRoleName machine-pool property is gone, and the podCIDR networking property (deprecated in 0.4.0) is gone. The install-config version has been bumped from v1beta1 to v1beta2. All users, regardless of platform, will need to update any saved install-config.yaml to use the new version. IAM roles are being replaced by [the credentials operator][credential-operator], and while we still create IAM roles for our master, worker, and bootstrap machines, we're removing the user-facing property now to avoid making this breaking change later.
• On AWS, the bootstrap machine security group allowing kubelet access (added in 0.10.1) has been removed. Static pod logs should soon be available from journald (although they aren't yet).

Added

• On AWS, the installer creates [DHCP options][aws-dhcp-options] for the VPC to support internal unqualified-hostname resolution. This works around some limitations with oc rsh and Kubernetes node registration in the face of inappropriate default DHCP options. And because [the AWS domain-name logic is region-specific][aws-dhcp-options], there is no single DHCP options configuration that provides internal unqualified-hostname resolution for multiple regions.

Changed

• On AWS, the installer now prompts for missing credentials even if you supplied an install-config.yaml. Previously, only the install-config wizard would prompt.
• On OpenStack, the developer-only internal DNS server which was removed in 0.10.0 has been restored, because the approach taken in 0.10.0 broke etcd cluster formation for some users.
• Several doc and internal cleanups.

Fixed

• openshift-install has improved error handling for various invalid command lines. It now errors when additional positional arguments are passed to commands that do not take positional arguments (previously those commands silently ignored the presence of positional arguments). And it logs an error and exits 1 when an invalid value is provided to --log-level (previously it exited 1 but did not write to the standard error stream).

• The slow-input issues for the install-config wizard have been fixed.

• On AWS, destroy cluster fixed a bug in the 0.10.1 refactor which could lead to leaked resources and a claim of successful deletion if a call to get tagged resources failed (for example, because the caller lacked the tag:GetResources permission).

• On AWS, a new explicit dependency in the Terraform modules prevents errors like:

    * module.vpc.aws_lb.api_external: 1 error occurred:
    * aws_lb.api_external: Error creating Application Load Balancer: InvalidSubnet: VPC vpc-0765c67bbc82a1b7d has no internet gateway
    status code: 400, request id: 5a...d5
  

• On libvirt, the installer no longer holds the OS image in memory after it has been written to disk. Ideally it would stream the OS image to disk instead of ever holding it in memory, but this fix mitigates our current in-memory buffering.

Changed

• create ignition-configs now also writes metadata.json to the asset directory, which allows [Hive][] to more reliably destroy clusters.
• destroy cluster now removes .openshift_install_state.json on success, clearing the way for future create cluster runs in the same asset directory.
• On AWS, we now default to m4.xlarge masters. The increased CPU reduces etcd latencies, which in turn helps with cluster stability.
• On AWS, the bootstrap machine has a new security-group allowing journald-gateway and kubelet access, for easier debugging when bootstrapping fails.
• Several doc and internal cleanups.

Removed

• The SSH public key is no longer inserted in the pointer Ignition configurations, now that authorized public keys are [managed by the machine-config daemon][machine-config-daemon-ssh-keys].

Fixed

• On AWS, the cluster-API provider now supports configuring machine volumes, so rootVolume settings in install-config.yaml will be respected.

• On AWS, the generated Terraform variables no longer clobber master instance type and root volume configuration set via install-config.yaml. You can now use:

  machines:
  - name: master
    platform:
      aws:
        type: m5.large
        rootVolume:
          iops: 3000
          size: 220
          type: io1
    replicas: 3
  - name: worker
    ...
  

  and similar to successfully customize your master machines.

• On AWS, destroy cluster has been adjusted to use more efficient tag-based lookup and fix several bugs due to previously-missing pagination. This should address some issues we had been seeing with leaking AWS resources despite destroy cluster claiming success.

Added

• The installer pushes an Infrastructure object to infrastructures.config.openshift.io with platform information. Cluster components should use this instead of the deprecated cluster-config-v1 resource.
• openshift-install has a new completion subcommand, to generation shell-completion code (currently only for Bash).
• On AWS, destroy cluster now also removed IAM users with the usual tags. We don't create these users yet, but the removal sets the stage for the coming [credential operator][credential-operator].

Changed

• Install configuration now includes a new apiVersion property which must be set to v1beta1. Future changes to the install-config schema will result in new versions, allowing new installers to continue to support older install-config schema (and older installers to error out when presented with newer install-config schema). Changes to the schema since 0.9.0:

  • clusterID has been removed. This should be a new UUID for every cluster, so there is no longer an easy way for users to configure it.
  • Image configuration has been removed. Almost all users should be fine with the installer-chosen RHCOS. Users who need to override the RHCOS build (because they're testing new RHCOS releases) can set a new OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE environment variable.
  • Per-platform network properties have been consolidated into the platform-agnostic machineCIDR from which node IP addresses are assigned.
  • On libvirt, all machine-pool configuration has been removed, as no remaining properties were supported by the libvirt cluster-API provider.

• install-config.yaml read during staged installs will now have installer-defaults applied for missing properties. This allows you to set only the properties you are interested in overriding, and allow the installer to manage the remaining properties.

• create ignition-configs now also writes the admin kubeconfig to the asset directory, to support bring-your-own-infrastructure use cases.

• The bootstrap node now serves journals for easier troubleshooting.

• The validity for the initial kubelet TLS certificate has been increased from one hour to 24 hours, to give bring-your-own-infrastructure users longer to manually distribute the certificate before it expires.

• The key for the root certificate authority is no longer pushed into the cluster (not even to the bootstrap node).

• Machine(set)s generated by the installer now use providerSpec instead of the deprecated providerConfig.

• On AWS, the load balancers now use HTTPS health checks to reduce log noise like:

    http: TLS handshake error from 10.0.20.86:28372: EOF
  

• On AWS, IAM roles are now tagged with the usual resource tags (openshiftClusterID, etc.). Some other resources have had their tags updated to match those conventions (e.g. the internal Route 53 hosted zone was updated from KubernetesCluster to kubernetes.io/cluster/{name}: owned).

• The OpenStack platform has been removed from the install-config wizard while it remains experimental. It is still available for users who supply their own install-config.yaml.

• On OpenStack, the service VP now respects any SSH key specified in the install configuration.

• On OpenStack, a developer-only internal DNS server has been removed, so users need to configure additional records for the existing external DNS zone.

• On OpenStack, Neutron trunk ports are now used for VM network interfaces if Neutron supports them to support future Kuryr integration.

• On OpenStack, masters and workers have been consolidated in a single subnet to simplify the deployment.

• On OpenStack, the Ignition security group now only allows internal connections, and no longer allows connections from outside the cluster network.

• On OpenStack, the machine(set) templates have been updated to set cloudName and some other properties.

• On libvirt, destroy cluster is now more robust in the face of domains which were already shutdown.

• Lots of doc and internal cleanup and minor fixes.

Removed

• Support for install-config.yml (deprecated in 0.8.0) has been removed.

Fixed

• On AWS, domain pagination for the wizard's base-domain select widget has been fixed. Previously, it would continuously fetch the first page of hosted zones (for accounts with multiple pages of zones) until it hit an error like:

  ERROR list hosted zones: Throttling: Rate exceeded
          status code: 400, request id: ...
  

  before falling back to a free-form base-domain input.

Added

• The installer pushes an Infrastructure object to infrastructures.config.openshift.io with platform information. Cluster components should use this instead of the deprecated cluster-config-v1 resource.
• openshift-install has a new completion subcommand, to generation shell-completion code (currently only for Bash).
• On AWS, destroy cluster now also removed IAM users with the usual tags. We don't create these users yet, but the removal sets the stage for the coming credentials operator.

Changed

• Install configuration now includes a new apiVersion property which must be set to v1beta1. Future changes to the install-config schema will result in new versions, allowing new installers to continue to support older install-config schema (and older installers to error out when presented with newer install-config schema). Changes to the schema since 0.9.0:

  • clusterID has been removed. This should be a new UUID for every cluster, so there is no longer an easy way for users to configure it.
  • Image configuration has been removed. Almost all users should be fine with the installer-chosen RHCOS. Users who need to override the RHCOS build (because they're testing new RHCOS releases) can set a new OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE environment variable.
  • Per-platform network properties have been consolidated into the platform-agnostic machineCIDR from which node IP addresses are assigned.
  • On libvirt, all machine-pool configuration has been removed, as no remaining properties were supported by the libvirt cluster-API provider.

• install-config.yaml read during staged installs will now have installer-defaults applied for missing properties. This allows you to set only the properties you are interested in overriding, and allow the installer to manage the remaining properties.

• create ignition-configs now also writes the admin kubeconfig to the asset directory, to support bring-your-own-infrastructure use cases.

• The bootstrap node now serves journals for easier troubleshooting.

• The validity for the initial kubelet TLS certificate has been increased from one hour to 24 hours, to give bring-your-own-infrastructure users longer to manually distribute the certificate before it expires.

• The key for the root certificate authority is no longer pushed into the cluster (not even to the bootstrap node).

• Machine(set)s generated by the installer now use providerSpec instead of the deprecated providerConfig.

• On AWS, the load balancers now use HTTPS health checks to reduce log noise like:

    http: TLS handshake error from 10.0.20.86:28372: EOF
  

• On AWS, IAM roles are now tagged with the usual resource tags (openshiftClusterID, etc.). Some other resources have had their tags updated to match those conventions (e.g. the internal Route 53 hosted zone was updated from KubernetesCluster to kubernetes.io/cluster/{name}: owned).

• The OpenStack platform has been removed from the install-config wizard while it remains experimental. It is still available for users who supply their own install-config.yaml.

• On OpenStack, the service VP now respects any SSH key specified in the install configuration.

• On OpenStack, a developer-only internal DNS server has been removed, so users need to configure additional records for the existing external DNS zone.

• On OpenStack, Neutron trunk ports are now used for VM network interfaces if Neutron supports them to support future Kuryr integration.

• On OpenStack, masters and workers have been consolidated in a single subnet to simplify the deployment.

• On OpenStack, the Ignition security group now only allows internal connections, and no longer allows connections from outside the cluster network.

• On OpenStack, the machine(set) templates have been updated to set cloudName and some other properties.

• On libvirt, destroy cluster is now more robust in the face of domains which were already shutdown.

• Lots of doc and internal cleanup and minor fixes.

Removed

• Support for install-config.yml (deprecated in 0.8.0) has been removed.

Fixed

• On AWS, domain pagination for the wizard's base-domain select widget has been fixed. Previously, it would continuously fetch the first page of hosted zones (for accounts with multiple pages of zones) until it hit an error like:

  ERROR list hosted zones: Throttling: Rate exceeded
          status code: 400, request id: ...
  

  before falling back to a free-form base-domain input.

Added

• There is a new none platform for bring-your-own infrastructure users who want to generate Ignition configurations. The new platform is mostly undocumented; users will usually interact with it via [OpenShift Ansible][openshift-ansible].

Changed

• On OpenStack, there's no longer a default flavor, because flavor names are not standardized. Instead, there's a new prompt to choose among flavors supported by the target OpenStack implementation.
• On libvirt, we now use the host-passthrough CPU type, which should improve performance for some use-cases.
• Some doc and internal cleanup and minor fixes.