Action required: Users should migrate the configs in values.yaml of previous chart releases to the new values.yaml of the new chart. Otherwise, the monitor pods might fail when you upgrade the monitor with the new chart.

For example, configs in the old values.yaml file:

monitor:
  ...
  initializer:
    image: pingcap/tidb-monitor-initializer:v3.0.5
    imagePullPolicy: IfNotPresent
  ...

After migration, configs in the new values.yaml file should be as follows:

monitor:
  ...
  initializer:
    image: pingcap/tidb-monitor-initializer:v3.0.5
    imagePullPolicy: Always
    config:
      K8S_PROMETHEUS_URL: http://prometheus-k8s.monitoring.svc:9090
  ...

Monitor

• Enable alert rule persistence (#898)
• Add node & pod info in TiDB Grafana (#885)

TiDB Scheduler

• Refine scheduler error messages (#1373)

Compatibility

• Fix the compatibility issue in Kubernetes v1.17 (#1241)
• Bind the system:kube-scheduler ClusterRole to the tidb-scheduler service account (#1355)

TiKV Importer

• Fix the default tikv-importer configuration (#1415)

E2E

• Ensure pods unaffected when upgrading (#955)

CI

• Move the release CI script from Jenkins into the tidb-operator repository (#1237)
• Adjust the release CI script for the release-1.0 branch (#1320)

TiDB Operator v1.0.5 Release Notes

There is no action required if you are upgrading from v1.0.4.

Scheduled Backup

• Fix the issue that backup failed when clusterName is too long (#1229)

TiDB Binlog

• It is recommended that TiDB and Pump be deployed on the same node through the affinity feature and Pump be dispersed on different nodes through the anti-affinity feature. At most only one Pump instance is allowed on each node. We added a guide to the chart. (#1251)

Compatibility

• Fix tidb-scheduler RBAC permission in Kubernetes v1.16 (#1282)
• Do not set DNSPolicy if hostNetwork is disabled to keep backward compatibility (#1287)

E2E

• Fix e2e nil point dereference (#1221)

TiDB Operator v1.0.4 Release Notes

Action Required

There is no action required if you are upgrading from v1.0.3.

Highlights

#1202 introduced HostNetwork support, which offers better performance compared to the Pod network. Check out our benchmark report for details.

Note:

Due to this issue of Kubernetes, the Kubernetes cluster must be one of the following versions to enable HostNetwork of the TiDB cluster:

• v1.13.11 or later
• v1.14.7 or later
• v1.15.4 or later
• any version since v1.16.0

#1175 added the podSecurityContext support for TiDB cluster Pods. We recommend setting the namespaced kernel parameters for TiDB cluster Pods according to our Environment Recommendation.

New Helm chart tidb-lightning brings TiDB Lightning support for TiDB in Kubernetes. Check out the document for detailed user guide.

Another new Helm chart tidb-drainer brings multiple drainers support for TiDB Binlog in Kubernetes. Check out the document for detailed user guide.

Improvements

• Support HostNetwork (#1202)
• Support configuring sysctls for Pods and enable net.* (#1175)
• Add tidb-lightning support (#1161)
• Add new helm chart tidb-drainer to support multiple drainers (#1160)

Action Required

ACTION REQUIRED: This release upgrades default TiDB version to v3.0.5 which fixed a serious bug in TiDB. So if you are using TiDB v3.0.4 or prior versions, you must upgrade to v3.0.5.

ACTION REQUIRED: This release adds the timezone support for all charts.

For existing TiDB clusters. If the timezone in tidb-cluster/values.yaml has been customized to other timezones instead of the default UTC, then upgrading tidb-operator will trigger a rolling update for the related pods.

The related pods include pump, drainer, discovery, monitor, scheduled backup, tidb-initializer, and tikv-importer.

The time zone for all images maintained by tidb-operator should be UTC. If you use your own images, you need to make sure that the corresponding time zones are UTC.

Improvements

• Add the timezone support for all containers of the TiDB cluster
• Support configuring resource requests and limits for all containers of the TiDB cluster

Action Required

The AWS Terraform script uses auto-scaling-group for all components (PD/TiKV/TiDB/monitor). When an ec2 instance fails the health check, the instance will be replaced. This is helpful for those applications that are stateless or use EBS volumes to store data.

But a TiKV Pod uses instance store to store its data. When an instance is replaced, all the data on its store will be lost. TiKV has to resync all data to the newly added instance. Though TiDB is a distributed database and can work when a node fails, resyncing data can cost much if the dataset is large. Besides, the ec2 instance may be recovered to a healthy state by rebooting.

So we disabled the auto-scaling-group's replacing behavior in v1.0.2.

Auto-scaling-group scaling process can also be suspended according to its documentation if you are using v1.0.1 or prior versions.

Improvements

• Suspend ReplaceUnhealthy process for AWS TiKV auto-scaling-group
• Add a new VM manager qm in stability test
• Add tikv.maxFailoverCount limit to TiKV
• Set the default externalTrafficPolicy to be Local for TiDB service in AWS/GCP/Aliyun
• Add provider and module versions for AWS

Bug fixes

• Fix the issue that tkctl version does not work when the release name is un-wanted
• Migrate statefulsets apiVersion to app/v1 which fixes compatibility with Kubernetes 1.16 and above versions
• Fix the issue that the create_tidb_cluster_release variable in AWS Terraform script does not work
• Fix compatibility issues by adding v1beta1 to statefulset apiVersions
• Fix the issue that TiDB Loadbalancer is empty in Terraform output
• Fix a compatibility issue of TiKV maxFailoverCount
• Fix Terraform providers version constraint issues for GCP and Aliyun
• Fix values file customization for tidb-operator on Aliyun
• Fix tidb-operator crash when users modify statefulset upgrade strategy improperly
• Fix drainer misconfiguration

Action Required

• ACTION REQUIRED: We fixed a serious bug (#878) that could cause all PD and TiKV pods to be accidentally deleted when kube-apiserver fails. This would cause TiDB service outage. So if you are using v1.0.0 or prior versions, you must upgrade to v1.0.1.
• ACTION REQUIRED: The backup tool image pingcap/tidb-cloud-backup uses a forked version of Mydumper. The current version pingcap/tidb-cloud-backup:20190610 contains a serious bug that could result in a missing column in the exported data. This is fixed in #29. And the default image used now contains this fixed version. So if you are using the old version image for backup, you must upgrade to use pingcap/tidb-cloud-backup:201908028 and do a new full backup to avoid potential data inconsistency.

Improvements

• Modularize GCP Terraform
• Add a script to remove orphaned k8s disks
• Support binlog.pump.config, binlog.drainer.config configurations for Pump and Drainer
• Set the resource limit for the tidb-backup job
• Add affinity to Pump and Drainer configurations
• Upgrade local-volume-provisioner to v2.3.2
• Reduce e2e run time from 60m to 20m
• Prevent the Pump process from exiting with 0 if the Pump becomes offline
• Support expanding cloud storage PV dynamically by increasing PVC storage size
• Add the tikvGCLifeTime option to do backup
• Add important parameters to tikv.config and tidb.config in values.yaml
• Support restoring the TiDB cluster from a specified scheduled backup directory
• Enable cloud storage volume expansion & label local volume
• Document and improve HA algorithm
• Support specifying the permit host in the values.tidb.permitHost chart
• Add the zone label and reserved resources arguments to kubelet
• Update the default backup image to pingcap/tidb-cloud-backup:20190828

Bug fixes

• Fix the TiKV scale-in failure in some cases after the TiKV failover
• Fix error handling for UpdateService
• Fix some orphan pods cleaner bugs
• Fix the bug of setting the StatefulSet partition
• Fix ad-hoc full backup failure due to incorrect claimName
• Fix the offline Pump: the Pump process will exit with 0 if going offline
• Fix an incorrect condition judgment

Action Required

• ACTION REQUIRED: tikv.storeLabels was removed from values.yaml. You can directly set it with location-labels in pd.config.
• ACTION REQUIRED: the --features flag of tidb-scheduler has been updated to the key={true,false} format. You can enable the feature by appending =true.
• ACTION REQUIRED: you need to change the configurations in values.yaml of previous chart releases to the new values.yaml of the new chart. Otherwise, the configurations will be ignored when upgrading the TiDB cluster with the new chart.

The pd section in old values.yaml:

pd:
  logLevel: info
  maxStoreDownTime: 30m
  maxReplicas: 3

The pd section in new values.yaml:

pd:
  config: |
    [log]
    level = "info"
    [schedule]
    max-store-down-time = "30m"
    [replication]
    max-replicas = 3

The tikv section in old values.yaml:

tikv:
  logLevel: info
  syncLog: true
  readpoolStorageConcurrency: 4
  readpoolCoprocessorConcurrency: 8
  storageSchedulerWorkerPoolSize: 4

The tikv section in new values.yaml:

tikv:
  config: |
    log-level = "info"
    [server]
    status-addr = "0.0.0.0:20180"
    [raftstore]
    sync-log = true
    [readpool.storage]
    high-concurrency = 4
    normal-concurrency = 4
    low-concurrency = 4
    [readpool.coprocessor]
    high-concurrency = 8
    normal-concurrency = 8
    low-concurrency = 8
    [storage]
    scheduler-worker-pool-size = 4

The tidb section in old values.yaml:

tidb:
  logLevel: info
  preparedPlanCacheEnabled: false
  preparedPlanCacheCapacity: 100
  txnLocalLatchesEnabled: false
  txnLocalLatchesCapacity: "10240000"
  tokenLimit: "1000"
  memQuotaQuery: "34359738368"
  txnEntryCountLimit: "300000"
  txnTotalSizeLimit: "104857600"
  checkMb4ValueInUtf8: true
  treatOldVersionUtf8AsUtf8mb4: true
  lease: 45s
  maxProcs: 0

The tidb section in new values.yaml:

tidb:
  config: |
    token-limit = 1000
    mem-quota-query = 34359738368
    check-mb4-value-in-utf8 = true
    treat-old-version-utf8-as-utf8mb4 = true
    lease = "45s"
    [log]
    level = "info"
    [prepared-plan-cache]
    enabled = false
    capacity = 100
    [txn-local-latches]
    enabled = false
    capacity = 10240000
    [performance]
    txn-entry-count-limit = 300000
    txn-total-size-limit = 104857600
    max-procs = 0

The monitor section in old values.yaml:

monitor:
  create: true
  ...

The monitor section in new values.yaml:

monitor:
  create: true
   initializer:
     image: pingcap/tidb-monitor-initializer:v3.0.5
     imagePullPolicy: IfNotPresent
   reloader:
     create: true
     image: pingcap/tidb-monitor-reloader:v1.0.0
     imagePullPolicy: IfNotPresent
     service:
       type: NodePort
  ...

Please check cluster configuration for detailed configuration.

Stability test cases added

• Stop all etcds and kubelets

Improvements

• Simplify GKE SSD setup
• Modularization for AWS Terraform scripts
• Turn on the automatic failover feature by default
• Enable configmap rollout by default
• Enable stable scheduling by default
• Support multiple TiDB clusters management in Alibaba Cloud
• Enable AWS NLB cross zone load balancing by default

Bug fixes

• Fix sysbench installation on bastion machine of AWS deployment
• Fix TiKV metrics monitoring in default setup

Stability test cases added

• stop kube-proxy
• upgrade tidb-operator

Improvements

• get the TS first and increase the TiKV GC life time to 3 hours before the full backup
• Add endpoints list and watch permission for controller-manager
• Scheduler image is updated to use "registry.k8s.io/kube-scheduler" which is much smaller than "gcr.io/google-containers/hyperkube". You must pre-pull the new scheduler image into your airgap environment before upgrading.
• Full backup data can be uploaded to or downloaded from Amazon S3
• The terraform scripts support manage multiple TiDB clusters in one EKS cluster.
• Add tikv.storeLables setting
• on GKE one can use COS for TiKV nodes with small data for faster startup
• Support force upgrade when PD cluster is unavailable.

Bug fixes

• fix unbound variable in the backup script
• Give kube-scheduler permission to update/patch pod status
• fix tidb user of scheduled backup script
• fix scheduled backup to ceph object storage
• Fix several usability problems for AWS terraform deployment
• fix scheduled backup bug: segmentation fault when backup user's password is empty

Action Required

• ACTION REQUIRED: nodeSelectorRequired was removed from values.yaml.
• ACTION REQUIRED: Comma-separated values support in nodeSelector has been dropped, please use new-added affinity field which has a more expressive syntax.

A lot of stability cases added

• ConfigMap rollout
• One PD replicas
• Stop TiDB Operator itself
• TiDB stable scheduling
• Disaster tolerance and data regions disaster tolerance
• Fix many bugs of stability test

Features added

• Introduce ConfigMap rollout management. With the feature gate open, configuration file changes will be automatically applied to the cluster via a rolling update. Currently, the scheduler and replication configurations of PD can not be changed via ConfigMap rollout. You can use pd-ctl to change these values instead, see #487 for details.
• Support stable scheduling for pods of TiDB members in tidb-scheduler.
• Support adding additional pod annotations for PD/TiKV/TiDB, e.g. fluentbit.io/parser.
• Support the affinity feature of k8s which can define the rule of assigning pods to nodes
• Allow pausing during TiDB upgrade

Documentation improved

• GCP one-command deployment
• Refine user guides
• Improve GKE, AWS, Aliyun guide

Pass User Acceptance Tests

Other improvements

• Upgrade default TiDB version to v3.0.0-rc.1
• fix bug in reporting assigned nodes of tidb members
• tkctl get can show cpu usage correctly now
• Adhoc backup now appends the start time to the PVC name by default.
• add the privileged option for TiKV pod
• tkctl upinfo can show nodeIP podIP port now
• get TS and use it before full backup using mydumper
• Fix capabilities issue for tkctl debug command

Stability has been greatly enhanced

• Refactored e2e test
• Added stability test, 7x24 running

Greatly improved ease of use

• One-command deployment for AWS, Aliyun
• Minikube deployment for testing
• Tkctl cli tool
• Refactor backup chart for ease use
• Refine initializer job
• Grafana monitor dashboard improved, support multi-version
• Improved user guide
• Contributing documentation

Bug fixes

• Fix PD start script, add join file when startup
• Fix TiKV failover take too long
• Fix PD ha when replcias is less than 3
• Fix a tidb-scheduler acquireLock bug and emit event when scheduled failed
• Fix scheduler ha bug with defer deleting pods
• Fix bug when using shareinformer without deepcopy

Other improvements

• Remove pushgateway from TiKV pod
• Add GitHub templates for issue reporting and PR
• Automatically set the scheduler K8s version
• Swith to go module
• Support slow log of TiDB