Release notes

https://xgboost.readthedocs.io/en/latest/changes/v3.3.0.html

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

22d4fba822fba5cd02299bf0c63ec68ff72606bc1b1bd910423d4b83c2f108ff  xgboost-src-3.3.0.tar.gz
df276bf14ebda98319da70fa88874746d9275ffbcde77e18e171d809cdbda86a  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux.tar.gz: Download

Source tarball

• xgboost-src-3.3.0.tar.gz: Download

Release note

https://xgboost.readthedocs.io/en/latest/changes/v3.2.0.html

Additional artifacts

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

16a31dfbc0c54544c9c36ab5f696fa7b646c125f161c52c814d757a58241a404  xgboost-src-3.2.0.tar.gz
41ce6798ed032380d4efed08cb1e4fadb87a5eba401b530fefcb90f1deb367d0  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.2.0.tar.gz: Download

Source tarball

• xgboost-src-3.2.0.tar.gz: Download

What's Changed

• Scikit-learn 1.8 compatibility fix (#11858)
• Add ARM CUDA wheels for PyPI. (#11827) Add nccl as dep for aarch64. (#11753)
• [R] Fix off-by-one bug: nrounds=0 resulted in 2 iterations #11856
• [R] Fix mingw warnings, winbuilder check warnings, memory safety issues. (#11859, #11847, #11830, #11906)
• Avoid overflow in rounding estimation. (#11910)
• Workaround compiler issue on Windows, affects the use of max_delta_step with CUDA. (#11916)

Full Changelog: v3.1.2...v3.1.3

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

67800a7c1c0455c22c9be73dbf3c39bfd9ac9627b2cb617eb2795fd675a9d49e  xgboost-src-3.1.3.tar.gz
f3586dc2da415bba7c3a632b290d653b74eea0caf2ea9e8ffb488cacb57a1dcf  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.1.3.tar.gz: Download

Source tarball

• xgboost-src-3.1.3.tar.gz: Download

What's Changed

• Fix ordering of Python callbacks. (#11812)
• Fix loading nccl 2.28. (#11806)
• Infer the enable_categorical during model load. (#11816)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

12f2d6f735fa71e007c40171fd926c12306276dd299dc48f6c923e4f3891c33e  xgboost-src-3.1.2.tar.gz
2f83f1b24affb50bf65a8dd80d4ac9d19fe95cf181df35fa8a335a06d2eb9cfd  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.1.2.tar.gz: Download

Source tarball

• xgboost-src-3.1.2.tar.gz: Download

What's Changed

• Emit correct error when performing inplace-predict using a CPU-only version of XGBoost, but with a GPU input. (#11761)
• Enhance the error message for loading the removed binary model format. (#11760)
• Use the correct group ID for SHAP when the intercept is a vector. (#11764)

Full Changelog: v3.1.0...v3.1.1

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

b2bb9c93f28fe7e401dbe592eb7990f5382baa712b02301eb8fd4cdb6c676731  xgboost-src-3.1.1.tar.gz
ae6f2f2397aea02c77e77435cd9f617b5990756d5800218ff44f4ff5eba9104a  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.1.1.tar.gz: Download

Source tarball

• xgboost-src-3.1.1.tar.gz: Download

Release note

https://xgboost.readthedocs.io/en/latest/changes/v3.1.0.html

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

4c42d35976067270a9255bf9ee290a706917bb3929a60cdd74d4dd3f1a9c86cc  xgboost-src-3.1.0.tar.gz
79b3407f19ccfa7344ee1a7ae9afb845cff9472c5a736fbdbdf95d98950c8290  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.1.0.tar.gz: Download

Source tarball

• xgboost-src-3.1.0.tar.gz: Download

See https://github.com/dmlc/xgboost/issues/11704 for details.

What's Changed

• Remove restrict. #11669
• Compile with CTK13. #11670

Full Changelog: v3.0.4...v3.0.5

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

0776b59fad03548c447cb1e188469761241ffb3b36154dc8a59735f11d262dc2  xgboost-src-3.0.5.tar.gz
516759a0dd40da18d46fa84a945dce48a7612c9ddc4cfb3bc99df7575e889318  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.0.5.tar.gz: Download

Source tarball

• xgboost-src-3.0.5.tar.gz: Download

What's Changed

• Remove the use of all __restrict__. (#11616)
• Make CUDA lineinfo optional. (#11606)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

5388cc28f4f7725edc7d9eed4c4794a818df7c76c2d39652debe6fca7df770cf  xgboost-src-3.0.4.tar.gz
e43482127db15039f2ea2eb834adde885fa6a1d685a0526fed4293f863a793d5  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.0.4.tar.gz: Download

Source tarball

• xgboost-src-3.0.4.tar.gz: Download

• Fix NDCG metric with non-exp gain. (#11534)
• Avoid using mean intercept for rmsle. (#11588)
• [jvm-packages] add setNumEarlyStoppingRounds API (#11571)
• Avoid implicit synchronization in GPU evaluation. (#11542)
• Remove CUDA check in the array interface handler (#11386)
• Fix check in GPU histogram. (#11574)
• Support Rapids 25.06 (#11504)
• Adding enable_categorical to the sklearn .apply method (#11550)
• Make xgboost.testing compatible with scikit-learn 1.7 (#11502)
• Add support for building xgboost wheels on Win-ARM64 (#11572, #11597, #11559)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

6598adf6a073a55cc87a31e6712fc6dab938a5317aeae7134a07067d51acdf3a  xgboost-src-3.0.3.tar.gz
162eb7811313eac5c55f686920b32c5c29c929872bdbc65af147c6f4f19bc38d  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.0.3.tar.gz: Download

Source tarball

• xgboost-src-3.0.3.tar.gz: Download

What's Changed

• Dask 2025.4.0 scheduler info compatibility (#11462) by @jrbourbeau
• Fix VM fallback logic on WSL2 (#11471) by @hcho3

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

8f909899f5dc64d4173662a3efa307100713e3c2e2b831177c2e56af0e816caf  xgboost-src-3.0.2.tar.gz
c169cb92fe378d99f1938da5d2830da1cef731129701db480b03dbbd04333ae2  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.0.2.tar.gz: Download

Source tarball

• xgboost-src-3.0.2.tar.gz: Download

• Use nvidia-smi to detect the driver version and handle old drivers that don't support virtual memory. (#11391)
• Optimize deep trees for GPU external memory. (#11387)
• Small fix for page concatenation with external memory (#11338)
• Build xgboost-cpu for manylinux_2_28_x86_64 (#11406)
• Workaround for different Dask versions (#11436)
• Output models now use denormal floating-point instead of nan. (#11428)
• Fix aarch64 CI. (#11454)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

46e6815fd24dec7e17ed6e9327cc062da098387ee36358e3e0a43fc43939a8b1  xgboost-src-3.0.1.tar.gz
c00bc34a070d25557b06ccb684b4dff44a0f578cbe84442957742f3aba4f0c32  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.0.1.tar.gz: Download

Source tarball

• xgboost-src-3.0.1.tar.gz: Download

Release note

https://xgboost.readthedocs.io/en/latest/changes/v3.0.0.html

Additional artifacts:

You can verify the downloaded packages by running the following command with your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

431222b47085b9c3504d77ef59cfa23ae4fe9d701085313f47217e49e8823326  xgboost-src-3.0.0.tar.gz
a5dafa6ccc1a3df3d7e3c84d61dae4dcc6921b56e0b1932309ebd519253e11b1  xgboost_r_gpu_linux.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_3.0.0.tar.gz: Download

Source tarball

• xgboost-src-3.0.0.tar.gz: Download

See https://github.com/dmlc/xgboost/issues/11286 .

The 2.1.4 patch release incorporates the following fixes on top of the 2.1.3 release:

• XGBoost is now compatible with scikit-learn 1.6 (#11021, #11162)
• Build wheels with CUDA 12.8 and enable Blackwell support (#11187, #11202)
• Adapt to RMM 25.02 logger changes (#11153)

Full Changelog: https://github.com/dmlc/xgboost/compare/v2.1.3...v2.1.4

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

b6ce5870d03cc1233cad5ff8460f670a2aff78625adfb578c0b9eec3b8b88406  xgboost-2.1.4.tar.gz
9780ba8314824eac7b8565cc2af8ea692fd4898712052a49132ac3fdf7c0ab2b  xgboost_r_gpu_linux_2.1.4.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.1.4.tar.gz: Download

Source tarball

• xgboost.tar.gz: Download(base)

The 2.1.3 patch release makes the following bug fixes:

• [pyspark] Support large model size (#10984).
• Fix rng for the column sampler (#10998).
• Handle cudf.pandas proxy objects properly (#11014).

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

90b1b7b770803299b337dd9b9206760d9c16f418403c77acce74b350c6427667  xgboost-2.1.3.tar.gz
96b41da84769920408c5733d05fa2d56b53feeefd209e3d96842cf9c266e27ea  xgboost_r_gpu_linux_2.1.3.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.1.3.tar.gz: Download

Source tarball

• xgboost.tar.gz: Download

The 2.1.2 patch release makes the following bug fixes:

• Clean up and modernize release-artifacts.py (#10818)
• Fix ellpack categorical feature with missing values. (#10906)
• Fix unbiased ltr with training continuation. (#10908)
• Fix potential race in feature constraint. (#10719)
• Fix boolean array for arrow-backed DF. (#10527)
• Ensure that pip check does not fail due to a bad platform tag (#10755)
• Check cub errors (#10721)
• Limit the maximum number of threads. (#10872)
• Fixes for large size clusters. (#10880)
• POSIX compliant poll.h and mmap (#10767)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

a84fc7d9846c24659a2ad16788a7eefa9640b19eea9bbc65f30e0a9d53c52453  xgboost-2.1.2.tar.gz
999eff38533ea79ab3a1f0da524c54f6d0abd2ef220b6dbb9ba1331703e898bc  xgboost_r_gpu_linux_2.1.2.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.1.2.tar.gz: Download

Source tarball

• xgboost.tar.gz: Download

The 2.1.1 patch release make the following bug fixes:

• [Dask] Disable broadcast in the scatter call so that predict function won't hang (#10632) by @trivialfis
• [Dask] Handle empty partitions correctly (#10559) by @trivialfis
• Fix federated learning for the encrypted GRPC backend (#10503) by @trivialfis
• Fix a race condition in column splitter (#10572) by @trivialfis
• Gracefully handle cases where system files like /sys/fs/cgroup/cpu.max are not readable by the user (#10623) by @trivialfis
• Fix build and C++ tests for FreeBSD (#10480) by @hcho3, @trivialfis
• Clarify the requirement Pandas 1.2+ (#10476) by @hcho3
• More robust endianness detection in R package build (#10642) by @jakirkham, @hcho3

In addition, it contains several enhancements:

• Publish JVM packages targeting Linux ARM64 (#10487) by @hcho3
• Publish a CPU-only wheel under name xgboost-cpu (#10603) by @hcho3
• Support building with CUDA Toolkit 12.5 and latest CCCL (#10624, #10633, #10574) by @hcho3, @trivialfis, @jakirkham

Full Changelog: https://github.com/dmlc/xgboost/compare/v2.1.0...v2.1.1

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

eddbc5200b7c5210f2b8974b9d2a0328a30753416bfb81fdaf5040f4f7abb222  xgboost-2.1.1.tar.gz
3ba5a6e0c609bd5cc0a667d83c57457c06778bece50863e58c8bc1b4eb415fc6  xgboost_r_gpu_linux_2.1.1.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.1.1.tar.gz: Download

Source tarball

• xgboost.tar.gz: Download

2.1.0 (2024 Jun 20)

We are thrilled to announce the XGBoost 2.1 release. This note will start by summarizing some general changes and then highlighting specific package updates. As we are working on a new R interface, this release will not include the R package. We'll update the R package as soon as it's ready. Stay tuned!

Networking Improvements

An important ongoing work for XGBoost, which we've been collaborating on, is to support resilience for improved scaling and federated learning on various platforms. The existing networking library in XGBoost, adopted from the RABIT project, can no longer meet the feature demand. We've revamped the RABIT module in this release to pave the way for future development. The choice of using an in-house version instead of an existing library is due to the active development status with frequent new feature requests like loading extra plugins for federated learning. The new implementation features:

• Both CPU and GPU communication (based on NCCL).
• A reusable tracker for both the Python package and JVM packages. With the new release, the JVM packages no longer require Python as a runtime dependency.
• Supports federated communication patterns for both CPU and GPU.
• Supports timeout. The high-level interface parameter is currently hard-coded to 30 minutes, which we plan to improve.
• Supports significantly more data types.
• Supports thread-based workers.
• Improved handling for worker errors, including better error messages when one of the peers dies during training.
• Work with IPv6. Currently, this is only supported by the dask interface.
• Built-in support for various operations like broadcast, allgatherV, allreduce, etc.

Related PRs (#9597, #9576, #9523, #9524, #9593, #9596, #9661, #10319, #10152, #10125, #10332, #10306, #10208, #10203, #10199, #9784, #9777, #9773, #9772, #9759, #9745, #9695, #9738, #9732, #9726, #9688, #9681, #9679, #9659, #9650, #9644, #9649, #9917, #9990, #10313, #10315, #10112, #9531, #10075, #9805, #10198, #10414).

The existing option of using MPI in RABIT is removed in the release. (#9525)

NCCL is now fetched from PyPI.

In the previous version, XGBoost statically linked NCCL, which significantly increased the binary size and led to hitting the PyPI repository limit. With the new release, we have made a significant improvement. The new release can now dynamically load NCCL from an external source, reducing the binary size. For the PyPI package, the nvidia-nccl-cu12 package will be fetched during installation. With more downstream packages reusing NCCL, we expect the user environments to be slimmer in the future as well. (#9796, #9804, #10447)

Parts of the Python package now require glibc 2.28+

Starting from 2.1.0, XGBoost Python package will be distributed in two variants:

• manylinux_2_28: for recent Linux distros with glibc 2.28 or newer. This variant comes with all features enabled.
• manylinux2014: for old Linux distros with glibc older than 2.28. This variant does not support GPU algorithms or federated learning.

The pip package manager will automatically choose the correct variant depending on your system.

Starting from May 31, 2025, we will stop distributing the manylinux2014 variant and exclusively distribute the manylinux_2_28 variant. We made this decision so that our CI/CD pipeline won't have depend on software components that reached end-of-life (such as CentOS 7). We strongly encourage everyone to migrate to recent Linux distros in order to use future versions of XGBoost.

Note. If you want to use GPU algorithms or federated learning on an older Linux distro, you have two alternatives:

1. Upgrade to a recent Linux distro with glibc 2.28+. OR
2. Build XGBoost from the source.

Multi-output

We continue the work on multi-target and vector leaf in this release:

• Revise the support for custom objectives with a new API, XGBoosterTrainOneIter. This new function supports strided matrices and CUDA inputs. In addition, custom objectives now return the correct shape for prediction. (#9508)
• The hinge objective now supports multi-target regression (#9850)
• Fix the gain calculation with vector leaf (#9978)
• Support graphviz plot for multi-target tree. (#10093)
• Fix multi-output with alternating strategies. (#9933)

Please note that the feature is still in progress and not suitable for production use.

Federated Learning

Progress has been made on federated learning with improved support for column-split, including the following updates:

• Column split work for both CPU and GPU. In addition, categorical data is now compatible with column split. (#9562, #9609, #9611, #9628, #9539, #9578, #9685, #9623, #9613, #9511, #9384, #9595)
• The use of UBJson to serialize split entries for column split has been implemented, aiding vector-leaf with column-based data split. (#10059, #10055, #9702)
• Documentation and small fixes. (#9610, #9552, #9614, #9867)

Ongoing work for SYCL support.

XGBoost is developing a SYCL plugin for SYCL devices, starting with the hist tree method. (#10216, #9800, #10311, #9691, #10269, #10251, #10222, #10174, #10080, #10057, #10011, #10138, #10119, #10045, #9876, #9846, #9682) XGBoost now supports launchable inference on SYCL devices, and work on adding SYCL support for training is ongoing.

Looking ahead, we plan to complete the training in the coming releases and then focus on improving test coverage for SYCL, particularly for Python tests.

Optimizations

• Implement column sampler in CUDA for GPU-based tree methods. This helps us get faster training time when column sampling is employed (#9785)
• CMake LTO and CUDA arch (#9677)
• Small optimization to external memory with a thread pool. This reduces the number of threads launched during iteration. (#9605, #10288, #10374)

Deprecation and breaking changes

Package-specific breaking changes are outlined in respective sections. Here we list general breaking changes in this release:

• The command line interface is deprecated due to the increasing complexity of the machine learning ecosystem. Building a machine learning model using a command shell is no longer feasible and could mislead newcomers. (#9485)
• Universal binary JSON is now the default format for saving models (#9947, #9958, #9954, #9955). See https://github.com/dmlc/xgboost/issues/7547 for more info.
• The XGBoosterGetModelRaw is now removed after deprecation in 1.6. (#9617)
• Drop support for loading remote files. Users are encouraged to use dedicated libraries to fetch remote content. (#9504)
• Remove the dense libsvm parser plugin. This plugin is never tested or documented (#9799)
• XGDMatrixSetDenseInfo and XGDMatrixSetUIntInfo are now deprecated. Use the array interface based alternatives instead.

Features

This section lists some new features that are general to all language bindings. For package-specific changes, please visit respective sections.

• Adopt a new XGBoost logo (#10270)
• Now supports dataframe data format in native XGBoost. This improvement enhances performance and reduces memory usage when working with dataframe-based structures such as pandas, arrow, and R dataframe. (#9828, #9616, #9905)
• Change default metric for gamma regression to deviance. (#9757)
• Normalization for learning to rank is now optional with the introduction of the new lambdarank_normalization parameter. (#10094)
• Contribution prediction with QuantileDMatrix on CPU. (#10043)
• XGBoost on macos no longer bundles OpenMP runtime. Users can install the latest runtime from their dependency manager of choice. (https://github.com/dmlc/xgboost/pull/10440). Along with which, JVM packages on MacoOS are now built with OpenMP support (https://github.com/dmlc/xgboost/pull/10449).

Bug fixes

• Fix training with categorical data from external memory. (https://github.com/dmlc/xgboost/pull/10433)
• Fix compilation with CTK-12. (#10123)
• Fix inconsistent runtime library on Windows. (#10404)
• Fix default metric configuration. (#9575)
• Fix feature names with special characters. (#9923)
• Fix global configuration for external memory training. (#10173)
• Disable column sample by node for the exact tree method. (#10083)
• Fix the FieldEntry constructor specialization syntax error (#9980)
• Fix pairwise objective with NDCG metric along with custom gain. (#10100)
• Fix the default value for lambdarank_pair_method. (#10098)
• Fix UBJSON with boolean values. No existing code is affected by this fix. (#10054)
• Be more lenient on floating point errors for AUC. This prevents the AUC > 1.0 error. (#10264)
• Check support status for categorical features. This prevents gblinear from treating categorical features as numerical. (#9946)

Document

Here is a list of documentation changes not specific to any XGBoost package.

• A new coarse map for XGBoost features to assist development. (#10310)
• New language binding consistency guideline. (#9755, #9866)
• Fixes, cleanups, small updates (#9501, #9988, #10023, #10013, #10143, #9904, #10179, #9781, #10340, #9658, #10182, #9822)
• Update document for parameters (#9900)
• Brief introduction to base_score. (#9882)
• Mention data consistency for categorical features. (#9678)

Python package

• Dask Other than the changes in networking, we have some optimizations and document updates in dask:

• Filter models on workers instead of clients; this prevents an OOM error on the client machine. (#9518)
• Users are now encouraged to use from xgboost import dask instead of import xgboost.dask to avoid drawing in unnecessary dependencies for non-dask users. (#9742)
• Add seed to demos. (#10009)
• New document for using dask XGBoost with k8s. (#10271)
• Workaround potentially unaligned pointer from an empty partition. (#10418)
• Workaround a race condition in the latest dask. (#10419)
• Add typing to dask demos. (#10207)

• PySpark PySpark has several new features along with some small fixes:

• Support stage-level scheduling for training on various platforms, including yarn/k8s. (#9519, #10209, #9786, #9727)
• Support GPU-based transform methods (#9542)
• Avoid expensive repartition when appropriate. (#10408)
• Refactor the logging and the GPU code path (#10077, 9724)
• Sort workers by task ID. This helps the PySpark interface obtain deterministic results. (#10220)
• Fix PySpark with verbosity=3. (#10172)
• Fix spark estimator doc. (#10066)
• Rework transform for improved code reusing. (#9292)

• Breaking changes For the Python package, eval_metric, early_stopping_rounds, and callbacks from now removed from the fit method in the sklearn interface. They were deprecated in 1.6. Use the parameters with the same name in constructors instead. (#9986)

• Features Following is a list of new features in the Python package:

• Support sample weight in sklearn custom objective. (#10050)
• New supported data types, including cudf.pandas (#9602), torch.Tensor (#9971), and more scipy types (#9881).
• Support pandas 2.2 and numpy 2.0. (#10266, #9557, #10252, #10175)
• Support the latest rapids including rmm. (#10435)
• Improved data cache option in data iterator. (#10286)
• Accept numpy generators as random_state (#9743)
• Support returning base score as intercept in the sklearn interface. (#9486)
• Support arrow through pandas ext types. This is built on top of the new DataFrame API in XGBoost. See general features for more info. (#9612)
• Handle np integer in model slice and prediction. (#10007)
• Improved sklearn tags support. (#10230)
• The base image for building Linux binary wheels is updated to rockylinux8. (#10399)
• Improved handling for float128. (#10322)

• Fixes

• Fix DMatrix with None input. (#10052)
• Fix native library discovery logic. (#9712, #9860)
• Fix using categorical data with the score function for the ranker. (#9753)

• Document

• Clarify the effect of enable_categorical (#9877, #9884)
• Update the Python introduction. (#10033)
• Fixes. (#10058, #9991, #9573)

JVM package

Here is a list of JVM-specific changes. Like the PySpark package, the JVM package also gains stage-level scheduling.

• Features and related documents

• [breaking] Remove rabit checkpoint. (#9599)
• Support stage-level scheduling (#9775)
• Allow JVM-Package to access inplace predict method (#9167)
• Support JDK 17 for test (#9959)
• Various dependency updates.(#10211, #10210, #10217, #10156, #10070, #9809, #9517, #10235, #10276, #9331, #10335, #10309, #10240, #10244, #10260, #9489, #9326, #10294, #10197, #10196, #10193, #10202, #10191, #10188, #9328, #9311, #9951, #10151, #9827, #9820, #10253)
• Update and fixes for document. (#9752, #10385)

• Bug Fixes

• Fixes memory leak in error handling. (#10307)
• Fixes group col for GPU packages (#10254)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

28bec8e821b1fefcea722d96add66024adba399063f723bc5c815f7af4a5f5e4  xgboost-2.1.0.tar.gz
60c715d8c97ef710185469b27f30303b6efa655600d035963f96e6acf65f4dac  xgboost_r_gpu_linux_2.1.0.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.1.0.tar.gz: Download

Source tarball

• xgboost.tar.gz: Download

See https://github.com/dmlc/xgboost/issues/10356 for details.

The 2.0.3 patch release make the following bug fixes:

• [backport][sklearn] Fix loading model attributes. (#9808) by @trivialfis in https://github.com/dmlc/xgboost/pull/9880
• [backport][py] Use the first found native library. (#9860) by @trivialfis in https://github.com/dmlc/xgboost/pull/9879
• [backport] [CI] Upload libxgboost4j.dylib (M1) to S3 bucket by @hcho3 in https://github.com/dmlc/xgboost/pull/9887
• [jvm-packages] Fix POM for xgboost-jvm metapackage by @hcho3 in https://github.com/dmlc/xgboost/pull/9893 https://github.com/dmlc/xgboost/pull/9897

Full Changelog: https://github.com/dmlc/xgboost/compare/v2.0.2...v2.0.3

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

7c4bd1cf6162d335fd20a8168a54dd11508342f82fbf381a80c02ac57be0bce4  xgboost-2.0.3.tar.gz
d0c3499504133a8ea0043da2974c51cc71aae792f0719080bc227d7add8fb881  xgboost_r_gpu_win64_2.0.3.tar.gz
ee47da5b21231965b1f054d191a5418543377f4ba0d0615a593a6f99d1832ca1  xgboost_r_gpu_linux_2.0.3.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.0.3.tar.gz: Download
• xgboost_r_gpu_win64_2.0.3.tar.gz: Download

The 2.0.2 patch releases make the following bug fixes:

• [jvm-packages] Add Scala version suffix to xgboost-jvm package (#9776). The JVM packages had incorrect metadata, and the 2.0.2 patch version fixes the metadata.
• [backport] Fix using categorical data with the ranker. (#9753)

This is a patch release for bug fixes.

Bug fixes

• Support pandas 2.1.0. by @trivialfis in https://github.com/dmlc/xgboost/pull/9655
• Fix default metric configuration. by @trivialfis in https://github.com/dmlc/xgboost/pull/9590
• [R] Fix method name. by @trivialfis in https://github.com/dmlc/xgboost/pull/9592
• Use array interface for testing NumPy arrays. by @trivialfis in https://github.com/dmlc/xgboost/pull/9635
• Workaround Apple clang issue. by @trivialfis in https://github.com/dmlc/xgboost/pull/9636
• Add support for cgroupv2. by @trivialfis in https://github.com/dmlc/xgboost/pull/9656
• Fix build for GCC 8.x by @hcho3 in https://github.com/dmlc/xgboost/pull/9670
• [pyspark] Support stage-level scheduling by @wbo4958 in https://github.com/dmlc/xgboost/pull/9686
• Fix build for AppleClang 11 by @hcho3 in https://github.com/dmlc/xgboost/pull/9684
• Fix libpath logic for Windows by @hcho3 in https://github.com/dmlc/xgboost/pull/9687, https://github.com/dmlc/xgboost/pull/9711
• Remove hard dependency on libjvm by @hcho3 in https://github.com/dmlc/xgboost/pull/9705

In addition, this is the first release where the JVM package is distributed with native support for Apple Silicon.

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

529e9d0f88c2a7abae833f05b7d1e7e7ce01de20481ea60f6ebb6eb7fc96ba69  xgboost.tar.gz
25342c91e7cda98b1362b70282b286c2e4f3e996b518fb590c1303f53f39f188  xgboost_r_gpu_win64_2.0.1.tar.gz
3d8cde1160ab135c393b8092ce0475709dff318024022b735a253d968f9711b3  xgboost_r_gpu_linux_2.0.1.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.0.1.tar.gz: Download
• xgboost_r_gpu_win64_2.0.1.tar.gz: Download

Source tarball

• xgboost.tar.gz: Download

2.0.0 (2023 Sep 12)

We are excited to announce the release of XGBoost 2.0. This note will begin by covering some overall changes and then highlight specific updates to the package.

Initial work on multi-target trees with vector-leaf outputs

We have been working on vector-leaf tree models for multi-target regression, multi-label classification, and multi-class classification in version 2.0. Previously, XGBoost would build a separate model for each target. However, with this new feature that's still being developed, XGBoost can build one tree for all targets. The feature has multiple benefits and trade-offs compared to the existing approach. It can help prevent overfitting, produce smaller models, and build trees that consider the correlation between targets. In addition, users can combine vector leaf and scalar leaf trees during a training session using a callback. Please note that the feature is still a working in progress, and many parts are not yet available. See #9043 for the current status. Related PRs: (#8538, #8697, #8902, #8884, #8895, #8898, #8612, #8652, #8698, #8908, #8928, #8968, #8616, #8922, #8890, #8872, #8889, #9509) Please note that, only the hist (default) tree method on CPU can be used for building vector leaf trees at the moment.

New device parameter.

A new device parameter is set to replace the existing gpu_id, gpu_hist, gpu_predictor, cpu_predictor, gpu_coord_descent, and the PySpark specific parameter use_gpu. Onward, users need only the device parameter to select which device to run along with the ordinal of the device. For more information, please see our document page (https://xgboost.readthedocs.io/en/stable/parameter.html#general-parameters) . For example, with device="cuda", tree_method="hist", XGBoost will run the hist tree method on GPU. (#9363, #8528, #8604, #9354, #9274, #9243, #8896, #9129, #9362, #9402, #9385, #9398, #9390, #9386, #9412, #9507, #9536). The old behavior of gpu_hist is preserved but deprecated. In addition, the predictor parameter is removed.

hist is now the default tree method

Starting from 2.0, the hist tree method will be the default. In previous versions, XGBoost chooses approx or exact depending on the input data and training environment. The new default can help XGBoost train models more efficiently and consistently. (#9320, #9353)

GPU-based approx tree method

There's initial support for using the approx tree method on GPU. The performance of the approx is not yet well optimized but is feature complete except for the JVM packages. It can be accessed through the use of the parameter combination device="cuda", tree_method="approx". (#9414, #9399, #9478). Please note that the Scala-based Spark interface is not yet supported.

Optimize and bound the size of the histogram on CPU, to control memory footprint

XGBoost has a new parameter max_cached_hist_node for users to limit the CPU cache size for histograms. It can help prevent XGBoost from caching histograms too aggressively. Without the cache, performance is likely to decrease. However, the size of the cache grows exponentially with the depth of the tree. The limit can be crucial when growing deep trees. In most cases, users need not configure this parameter as it does not affect the model's accuracy. (#9455, #9441, #9440, #9427, #9400).

Along with the cache limit, XGBoost also reduces the memory usage of the hist and approx tree method on distributed systems by cutting the size of the cache by half. (#9433)

Improved external memory support

There is some exciting development around external memory support in XGBoost. It's still an experimental feature, but the performance has been significantly improved with the default hist tree method. We replaced the old file IO logic with memory map. In addition to performance, we have reduced CPU memory usage and added extensive documentation. Beginning from 2.0.0, we encourage users to try it with the hist tree method when the memory saving by QuantileDMatrix is not sufficient. (#9361, #9317, #9282, #9315, #8457)

Learning to rank

We created a brand-new implementation for the learning-to-rank task. With the latest version, XGBoost gained a set of new features for ranking task including:

• A new parameter lambdarank_pair_method for choosing the pair construction strategy.
• A new parameter lambdarank_num_pair_per_sample for controlling the number of samples for each group.
• An experimental implementation of unbiased learning-to-rank, which can be accessed using the lambdarank_unbiased parameter.
• Support for custom gain function with NDCG using the ndcg_exp_gain parameter.
• Deterministic GPU computation for all objectives and metrics.
• NDCG is now the default objective function.
• Improved performance of metrics using caches.
• Support scikit-learn utilities for XGBRanker.
• Extensive documentation on how learning-to-rank works with XGBoost.

For more information, please see the tutorial. Related PRs: (#8771, #8692, #8783, #8789, #8790, #8859, #8887, #8893, #8906, #8931, #9075, #9015, #9381, #9336, #8822, #9222, #8984, #8785, #8786, #8768)

Automatically estimated intercept

In the previous version, base_score was a constant that could be set as a training parameter. In the new version, XGBoost can automatically estimate this parameter based on input labels for optimal accuracy. (#8539, #8498, #8272, #8793, #8607)

Quantile regression

The XGBoost algorithm now supports quantile regression, which involves minimizing the quantile loss (also called "pinball loss"). Furthermore, XGBoost allows for training with multiple target quantiles simultaneously with one tree per quantile. (#8775, #8761, #8760, #8758, #8750)

L1 and Quantile regression now supports learning rate

Both objectives use adaptive trees due to the lack of proper Hessian values. In the new version, XGBoost can scale the leaf value with the learning rate accordingly. (#8866)

Export cut value

Using the Python or the C package, users can export the quantile values (not to be confused with quantile regression) used for the hist tree method. (#9356)

column-based split and federated learning

We made progress on column-based split for federated learning. In 2.0, both approx, hist, and hist with vector leaf can work with column-based data split, along with support for vertical federated learning. Work on GPU support is still on-going, stay tuned. (#8576, #8468, #8442, #8847, #8811, #8985, #8623, #8568, #8828, #8932, #9081, #9102, #9103, #9124, #9120, #9367, #9370, #9343, #9171, #9346, #9270, #9244, #8494, #8434, #8742, #8804, #8710, #8676, #9020, #9002, #9058, #9037, #9018, #9295, #9006, #9300, #8765, #9365, #9060)

PySpark

After the initial introduction of the PySpark interface, it has gained some new features and optimizations in 2.0.

• GPU-based prediction. (#9292, #9542)
• Optimization for data initialization by avoiding the stack operation. (#9088)
• Support predict feature contribution. (#8633)
• Python typing support. (#9156, #9172, #9079, #8375)
• use_gpu is deprecated. The device parameter is preferred.
• Update eval_metric validation to support list of strings (#8826)
• Improved logs for training (#9449)
• Maintenance, including refactoring and document updates (#8324, #8465, #8605, #9202, #9460, #9302, #8385, #8630, #8525, #8496)
• Fix for GPU setup. (#9495)

Other General New Features

Here's a list of new features that don't have their own section and yet are general to all language bindings.

• Use array interface for CSC matrix. This helps XGBoost to use a consistent number of threads and align the interface of the CSC matrix with other interfaces. In addition, memory usage is likely to decrease with CSC input thanks to on-the-fly type conversion. (#8672)
• CUDA compute 90 is now part of the default build.. (#9397)

Other General Optimization

These optimizations are general to all language bindings. For language-specific optimization, please visit the corresponding sections.

• Performance for input with array_interface on CPU (like numpy) is significantly improved. (#9090)
• Some optimization with CUDA for data initialization. (#9199, #9209, #9144)
• Use the latest thrust policy to prevent synchronizing GPU devices. (#9212)
• XGBoost now uses a per-thread CUDA stream, which prevents synchronization with other streams. (#9416, #9396, #9413)

Notable breaking change

Other than the aforementioned change with the device parameter, here's a list of breaking changes affecting all packages.

• Users must specify the format for text input (#9077). However, we suggest using third-party data structures such as numpy.ndarray instead of relying on text inputs. See https://github.com/dmlc/xgboost/issues/9472 for more info.

Notable bug fixes

Some noteworthy bug fixes that are not related to specific language bindings are listed in this section.

• Some language environments use a different thread to perform garbage collection, which breaks the thread-local cache used in XGBoost. XGBoost 2.0 implements a new thread-safe cache using a light weight lock to replace the thread-local cache. (#8851)
• Fix model IO by clearing the prediction cache. (#8904)
• inf is checked during data construction. (#8911)
• Preserve order of saved updaters configuration. Usually, this is not an issue unless the updater parameter is used instead of the tree_method parameter (#9355)
• Fix GPU memory allocation issue with categorical splits. (#9529)
• Handle escape sequence like \t\n in feature names for JSON model dump. (#9474)
• Normalize file path for model IO and text input. This handles short paths on Windows and paths that contain ~ on Unix (#9463). In addition, all path inputs are required to be encoded in UTF-8 (#9448, #9443)
• Fix integer overflow on H100. (#9380)
• Fix weighted sketching on GPU with categorical features. (#9341)
• Fix metric serialization. The bug might cause some of the metrics to be dropped during evaluation. (#9405)
• Fixes compilation errors on MSVC x86 targets (#8823)
• Pick up the dmlc-core fix for the CSV parser. (#8897)

Documentation

Aside from documents for new features, we have many smaller updates to improve user experience, from troubleshooting guides to typo fixes.

• Explain CPU/GPU interop. (#8450)
• Guide to troubleshoot NCCL errors. (#8943, #9206)
• Add a note for rabit port selection. (#8879)
• How to build the docs using conda (#9276)
• Explain how to obtain reproducible results on distributed systems. (#8903)

• Fixes and small updates to document and demonstration scripts. (#8626, #8436, #8995, #8907, #8923, #8926, #9358, #9232, #9201, #9469, #9462, #9458, #8543, #8597, #8401, #8784, #9213, #9098, #9008, #9223, #9333, #9434, #9435, #9415, #8773, #8752, #9291, #9549)

Python package

• New Features and Improvements

• Support primitive types of pyarrow-backed pandas dataframe. (#8653)
• Warning messages emitted by XGBoost are now emitted using Python warnings. (#9387)
• User can now format the value printed near the bars on the plot_importance plot (#8540)
• XGBoost has improved half-type support (float16) with pandas, cupy, and cuDF. With GPU input, the handling is through CUDA __half type, and no data copy is made. (#8487, #9207, #8481)
• Support Series and Python primitive types in inplace_predict and QuantileDMatrix (#8547, #8542)
• Support all pandas' nullable integer types. (#8480)
• Custom metric with the scikit-learn interface now supports sample_weight. (#8706)
• Enable Installation of Python Package with System lib in a Virtual Environment (#9349)
• Raise if expected workers are not alive in xgboost.dask.train (#9421)

• Optimization

• Cache transformed data in QuantileDMatrix for efficiency. (#8666, #9445)
• Take datatable as row-major input. (#8472)
• Remove unnecessary conversions between data structures (#8546)

• Adopt modern Python packaging conventions (PEP 517, PEP 518, PEP 621)

• XGBoost adopted the modern Python packaging conventions. The old setup script setup.py is now replaced with the new configuration file pyproject.toml. Along with this, XGBoost now supports Python 3.11. (#9021, #9112, #9114, #9115) Consult the latest documentation for the updated instructions to build and install XGBoost.

• Fixes

• DataIter now accepts only keyword arguments. (#9431)
• Fix empty DMatrix with categorical features. (#8739)
• Convert DaskXGBClassifier.classes_ to an array (#8452)
• Define best_iteration only if early stopping is used to be consistent with documented behavior. (#9403)
• Make feature validation immutable. (#9388)

• Breaking changes

• Discussed in the new device parameter section, the predictor parameter is now removed. (#9129)
• Remove support for single-string feature info. Feature type and names should be a sequence of strings (#9401)
• Remove parameters in the save_model call for the scikit-learn interface. (#8963)
• Remove the ntree_limit in the python package. This has been deprecated in previous versions. (#8345)

• Maintenance including formatting and refactoring along with type hints.

• More consistent use of black and isort for code formatting (#8420, #8748, #8867)
• Improved type support. Most of the type changes happen in the PySpark module; here, we list the remaining changes. (#8444, #8617, #9197, #9005)
• Set enable_categorical to True in predict. (#8592)
• Some refactoring and updates for tests (#8395, #8372, #8557, #8379, #8702, #9459, #9316, #8446, #8695, #8409, #8993, #9480)

• Documentation

• Add introduction and notes for the sklearn interface. (#8948)
• Demo for using dask for hyper-parameter optimization. (#8891)
• Document all supported Python input types. (#8643)
• Other documentation updates (#8944, #9304)

R package

• Use the new data consumption interface for CSR and CSC. This provides better control for the number of threads and improves performance. (#8455, #8673)
• Accept multiple evaluation metrics during training. (#8657)
• Fix integer inputs with NA. (#9522)
• Some refactoring for the R package (#8545, #8430, #8614, #8624, #8613, #9457, #8689, #8563, #9461, #8647, #8564, #8565, #8736, #8610, #8609, #8599, #8704, #9456, #9450, #9476, #9477, #9481). Special thanks to @jameslamb.
• Document updates (#8886, #9323, #9437, #8998)

JVM packages

Following are changes specific to various JVM-based packages.

• Stop using Rabit in prediction (#9054)
• Set feature_names and feature_types in jvm-packages. This is to prepare support for categorical features (#9364)
• Scala 2.13 support. (#9099)
• Change training stage from ResultStage to ShuffleMapStage (#9423)
• Automatically set the max/min direction for the best score during early stopping. (#9404)

• Revised support for flink (#9046)

• Breaking changes

• Scala-based tracker is removed. (#9078, #9045)
• Change DeviceQuantileDmatrix into QuantileDMatrix (#8461)

• Maintenance (#9253, #9166, #9395, #9389, #9224, #9233, #9351, #9479)

• CI bot PRs We employed GitHub dependent bot to help us keep the dependencies up-to-date for JVM packages. With the help from the bot, we have cleared up all the dependencies that are lagging behind (#8501, #8507).

Here's a list of dependency update PRs including those made by dependent bots (#8456, #8560, #8571, #8561, #8562, #8600, #8594, #8524, #8509, #8548, #8549, #8533, #8521, #8534, #8532, #8516, #8503, #8531, #8530, #8518, #8512, #8515, #8517, #8506, #8504, #8502, #8629, #8815, #8813, #8814, #8877, #8876, #8875, #8874, #8873, #9049, #9070, #9073, #9039, #9083, #8917, #8952, #8980, #8973, #8962, #9252, #9208, #9131, #9136, #9219, #9160, #9158, #9163, #9184, #9192, #9265, #9268, #8882, #8837, #8662, #8661, #8390, #9056, #8508, #8925, #8920, #9149, #9230, #9097, #8648, #9203, #8593).

Maintenance

Maintenance work includes refactoring, fixing small issues that don't affect end users. (#9256, #8627, #8756, #8735, #8966, #8864, #8747, #8892, #9057, #8921, #8949, #8941, #8942, #9108, #9125, #9155, #9153, #9176, #9447, #9444, #9436, #9438, #9430, #9200, #9210, #9055, #9014, #9004, #8999, #9154, #9148, #9283, #9246, #8888, #8900, #8871, #8861, #8858, #8791, #8807, #8751, #8703, #8696, #8693, #8677, #8686, #8665, #8660, #8386, #8371, #8410, #8578, #8574, #8483, #8443, #8454, #8733)

CI

• Build pip wheel with RMM support (#9383)
• Other CI updates including updating dependencies and work on the CI infrastructure. (#9464, #9428, #8767, #9394, #9278, #9214, #9234, #9205, #9034, #9104, #8878, #9294, #8625, #8806, #8741, #8707, #8381, #8382, #8388, #8402, #8397, #8445, #8602, #8628, #8583, #8460, #9544)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

de3a56c3d08a818bc1ea90c0476e28b937e10e0736b3ed4e27e22b43e8072ec1  xgboost-2.0.0.tar.gz
a23d965005e494ad9147cfaed1153e52ae238a8ad03ae9aa9aed83526ce7e150  xgboost_r_gpu_win64_2.0.0.tar.gz
c1a633a02cd7de14701b7814e9d81220716592d1891a33e265e76e54ce0e8e11  xgboost_r_gpu_linux_2.0.0.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_2.0.0.tar.gz: Download
• xgboost_r_gpu_win64_2.0.0.tar.gz: Download

Source tarball

• xgboost.tar.gz: Download

Roadmap: https://github.com/dmlc/xgboost/projects/2 Release note: https://github.com/dmlc/xgboost/pull/9484 Release status: https://github.com/dmlc/xgboost/issues/9497

This is a patch release for bug fixes. The CRAN package for the R binding is kept at 1.7.5.

Bug Fixes

• Fix distributed training with mixed dense and sparse partitions. (#9272)
• Fix monotone constraints on CPU with large trees. (#9122)
• [spark] Make the spark model have the same UID as its estimator (#9022)
• Optimize prediction with QuantileDMatrix. (#9096)

Document

• Improve doxygen (#8959)
• Update the cuDF pip index URL. (#9106)

Maintenance

• Fix tests with pandas 2.0. (#9014)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

0a54300dd274b98b7f039acffa006bec4875dace041fd9288422306fe7c379ca  xgboost.tar.gz
990fb3c54be7ce53365389f2eb82ce3c1f2e78735b4605ddd2ddb0d47a15d3c3  xgboost_r_gpu_linux_1.7.6.tar.gz
a48fc64bce774bb76eddade6dc6df1d4fc25199a0c17dc66cdfa50cedd3282ad  xgboost_r_gpu_win64_1.7.6.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_1.7.6.tar.gz: Download
• xgboost_r_gpu_win64_1.7.6.tar.gz: Download

Source tarball Link in GitHub release assets

1.7.5 (2023 Mar 30)

This is a patch release for bug fixes.

• C++ requirement is updated to C++-17, along with which, CUDA 11.8 is used as the default CTK. (#8860, #8855, #8853)
• Fix import for pyspark ranker. (#8692)
• Fix Windows binary wheel to be compatible with Poetry (#8991)
• Fix GPU hist with column sampling. (#8850)
• Make sure the iterative DMatrix is properly initialized. (#8997)
• [R] Update link in a document. (#8998)

Additional artifacts:

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

69a8cf4958e2cea5d492948968d765b856f60d336fbd4367d8176de95898ad7a  xgboost.tar.gz
0098f8d1cf5646d75c7d9dafa7e11b8d57441384f86a004b181cd679ef9677d1  xgboost_r_gpu_linux_1.7.5.tar.gz
a23b9744fcff8b53325604935b239c4cfef8a047ca5f4e57ea2b1011382314ee  xgboost_r_gpu_win64_1.7.5.tar.gz

Experimental binary packages for R with CUDA enabled

• xgboost_r_gpu_linux_1.7.5.tar.gz: Download
• xgboost_r_gpu_win64_1.7.5.tar.gz: Download

Source tarball Link in GitHub release assets

1.7.4 (2023 Feb 16)

This is a patch release for bug fixes.

• [R] Fix OpenMP detection on macOS. #8684
• [Python] Make sure input numpy array is aligned. #8690
• Fix feature interaction with column sampling in gpu_hist evaluator. #8754
• Fix GPU L1 error. #8749
• [PySpark] Fix feature types param #8772
• Fix ranking with quantile dmatrix and group weight. #8762
• Fix CPU bin compression with categorical data. #8809

Artifacts

xgboost_r_gpu_win64_1.7.4.tar.gz: Download

1.7.3 (2023 Jan 6)

This is a patch release for bug fixes.

• [Breaking] XGBoost Sklearn estimator method get_params no longer returns internally configured values. (#8634)
• Fix linalg iterator, which may crash the L1 error. (#8603)
• Fix loading pickled GPU sklearn estimator with a CPU-only XGBoost build. (#8632)
• Fix inference with unseen categories with categorical features. (#8591, #8602)
• CI fixes. (#8620, #8631, #8579)

Artifacts

You can verify the downloaded packages by running the following command on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

0b6aa86b93aec2b3e7ec6f53a696f8bbb23e21a03b369dc5a332c55ca57bc0c4  xgboost.tar.gz

v1.7.2 (2022 Dec 8)

This is a patch release for bug fixes.

• Work with newer thrust and libcudacxx (#8432)

• Support null value in CUDA array interface namespace. (#8486)

• Use getsockname instead of SO_DOMAIN on AIX. (#8437)

• [pyspark] Make QDM optional based on a cuDF check (#8471)

• [pyspark] sort qid for SparkRanker. (#8497)

• [dask] Properly await async method client.wait_for_workers. (#8558)

• [R] Fix CRAN test notes. (#8428)

• [doc] Fix outdated document [skip ci]. (#8527)

• [CI] Fix github action mismatched glibcxx. (#8551)

Artifacts

You can verify the downloaded packages by running this on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

15be5a96e86c3c539112a2052a5be585ab9831119cd6bc3db7048f7e3d356bac  xgboost_r_gpu_linux_1.7.2.tar.gz
0dd38b08f04ab15298ec21c4c43b17c667d313eada09b5a4ac0d35f8d9ba15d7  xgboost_r_gpu_win64_1.7.2.tar.gz

v1.7.1 (2022 November 3)

This is a patch release to incorporate the following hotfix:

• Add back xgboost.rabit for backwards compatibility (#8411)

Note. The source distribution of Python XGBoost 1.7.0 was defective (#8415). Since PyPI does not allow us to replace existing artifacts, we released 1.7.0.post0 version to upload the new source distribution. Everything in 1.7.0.post0 is identical to 1.7.0 otherwise.

v1.7.0 (2022 Oct 20)

We are excited to announce the feature packed XGBoost 1.7 release. The release note will walk through some of the major new features first, then make a summary for other improvements and language-binding-specific changes.

PySpark

XGBoost 1.7 features initial support for PySpark integration. The new interface is adapted from the existing PySpark XGBoost interface developed by databricks with additional features like QuantileDMatrix and the rapidsai plugin (GPU pipeline) support. The new Spark XGBoost Python estimators not only benefit from PySpark ml facilities for powerful distributed computing but also enjoy the rest of the Python ecosystem. Users can define a custom objective, callbacks, and metrics in Python and use them with this interface on distributed clusters. The support is labeled as experimental with more features to come in future releases. For a brief introduction please visit the tutorial on XGBoost's document page. (#8355, #8344, #8335, #8284, #8271, #8283, #8250, #8231, #8219, #8245, #8217, #8200, #8173, #8172, #8145, #8117, #8131, #8088, #8082, #8085, #8066, #8068, #8067, #8020, #8385)

Due to its initial support status, the new interface has some limitations; categorical features and multi-output models are not yet supported.

Development of categorical data support

More progress on the experimental support for categorical features. In 1.7, XGBoost can handle missing values in categorical features and features a new parameter max_cat_threshold, which limits the number of categories that can be used in the split evaluation. The parameter is enabled when the partitioning algorithm is used and helps prevent over-fitting. Also, the sklearn interface can now accept the feature_types parameter to use data types other than dataframe for categorical features. (#8280, #7821, #8285, #8080, #7948, #7858, #7853, #8212, #7957, #7937, #7934)

Experimental support for federated learning and new communication collective

An exciting addition to XGBoost is the experimental federated learning support. The federated learning is implemented with a gRPC federated server that aggregates allreduce calls, and federated clients that train on local data and use existing tree methods (approx, hist, gpu_hist). Currently, this only supports horizontal federated learning (samples are split across participants, and each participant has all the features and labels). Future plans include vertical federated learning (features split across participants), and stronger privacy guarantees with homomorphic encryption and differential privacy. See Demo with NVFlare integration for example usage with nvflare.

As part of the work, XGBoost 1.7 has replaced the old rabit module with the new collective module as the network communication interface with added support for runtime backend selection. In previous versions, the backend is defined at compile time and can not be changed once built. In this new release, users can choose between rabit and federated. (#8029, #8351, #8350, #8342, #8340, #8325, #8279, #8181, #8027, #7958, #7831, #7879, #8257, #8316, #8242, #8057, #8203, #8038, #7965, #7930, #7911)

The feature is available in the public PyPI binary package for testing.

Quantile DMatrix

Before 1.7, XGBoost has an internal data structure called DeviceQuantileDMatrix (and its distributed version). We now extend its support to CPU and renamed it to QuantileDMatrix. This data structure is used for optimizing memory usage for the hist and gpu_hist tree methods. The new feature helps reduce CPU memory usage significantly, especially for dense data. The new QuantileDMatrix can be initialized from both CPU and GPU data, and regardless of where the data comes from, the constructed instance can be used by both the CPU algorithm and GPU algorithm including training and prediction (with some overhead of conversion if the device of data and training algorithm doesn't match). Also, a new parameter ref is added to QuantileDMatrix, which can be used to construct validation/test datasets. Lastly, it's set as default in the scikit-learn interface when a supported tree method is specified by users. (#7889, #7923, #8136, #8215, #8284, #8268, #8220, #8346, #8327, #8130, #8116, #8103, #8094, #8086, #7898, #8060, #8019, #8045, #7901, #7912, #7922)

Mean absolute error

The mean absolute error is a new member of the collection of objectives in XGBoost. It's noteworthy since MAE has zero hessian value, which is unusual to XGBoost as XGBoost relies on Newton optimization. Without valid Hessian values, the convergence speed can be slow. As part of the support for MAE, we added line searches into the XGBoost training algorithm to overcome the difficulty of training without valid Hessian values. In the future, we will extend the line search to other objectives where it's appropriate for faster convergence speed. (#8343, #8107, #7812, #8380)

XGBoost on Browser

With the help of the pyodide project, you can now run XGBoost on browsers. (#7954, #8369)

Experimental IPv6 Support for Dask

With the growing adaption of the new internet protocol, XGBoost joined the club. In the latest release, the Dask interface can be used on IPv6 clusters, see XGBoost's Dask tutorial for details. (#8225, #8234)

Optimizations

We have new optimizations for both the hist and gpu_hist tree methods to make XGBoost's training even more efficient.

• Hist Hist now supports optional by-column histogram build, which is automatically configured based on various conditions of input data. This helps the XGBoost CPU hist algorithm to scale better with different shapes of training datasets. (#8233, #8259). Also, the build histogram kernel now can better utilize CPU registers (#8218)

• GPU Hist GPU hist performance is significantly improved for wide datasets. GPU hist now supports batched node build, which reduces kernel latency and increases throughput. The improvement is particularly significant when growing deep trees with the default depthwise policy. (#7919, #8073, #8051, #8118, #7867, #7964, #8026)

Breaking Changes

Breaking changes made in the 1.7 release are summarized below.

• The grow_local_histmaker updater is removed. This updater is rarely used in practice and has no test. We decided to remove it and focus have XGBoot focus on other more efficient algorithms. (#7992, #8091)
• Single precision histogram is removed due to its lack of accuracy caused by significant floating point error. In some cases the error can be difficult to detect due to log-scale operations, which makes the parameter dangerous to use. (#7892, #7828)
• Deprecated CUDA architectures are no longer supported in the release binaries. (#7774)
• As part of the federated learning development, the rabit module is replaced with the new collective module. It's a drop-in replacement with added runtime backend selection, see the federated learning section for more details (#8257)

General new features and improvements

Before diving into package-specific changes, some general new features other than those listed at the beginning are summarized here.

• Users of DMatrix and QuantileDMatrix can get the data from XGBoost. In previous versions, only getters for meta info like labels are available. The new method is available in Python (DMatrix::get_data) and C. (#8269, #8323)
• In previous versions, the GPU histogram tree method may generate phantom gradient for missing values due to floating point error. We fixed such an error in this release and XGBoost is much better equated to handle floating point errors when training on GPU. (#8274, #8246)
• Parameter validation is no longer experimental. (#8206)
• C pointer parameters and JSON parameters are vigorously checked. (#8254, #8254)
• Improved handling of JSON model input. (#7953, #7918)
• Support IBM i OS (#7920, #8178)

Fixes

Some noteworthy bug fixes that are not related to specific language binding are listed in this section.

• Rename misspelled config parameter for pseudo-Huber (#7904)
• Fix feature weights with nested column sampling. (#8100)
• Fix loading DMatrix binary in distributed env. (#8149)
• Force auc.cc to be statically linked for unusual compiler platforms. (#8039)
• New logic for detecting libomp on macos (#8384).

Python Package

• Python 3.8 is now the minimum required Python version. (#8071)

• More progress on type hint support. Except for the new PySpark interface, the XGBoost module is fully typed. (#7742, #7945, #8302, #7914, #8052)

• XGBoost now validates the feature names in inplace_predict, which also affects the predict function in scikit-learn estimators as it uses inplace_predict internally. (#8359)

• Users can now get the data from DMatrix using DMatrix::get_data or QuantileDMatrix::get_data.

• Show libxgboost.so path in build info. (#7893)

• Raise import error when using the sklearn module while scikit-learn is missing. (#8049)

• Use config_context in the sklearn interface. (#8141)

• Validate features for inplace prediction. (#8359)

• Pandas dataframe handling is refactored to reduce data fragmentation. (#7843)

• Support more pandas nullable types (#8262)

• Remove pyarrow workaround. (#7884)

• Binary wheel size We aim to enable as many features as possible in XGBoost's default binary distribution on PyPI (package installed with pip), but there's a upper limit on the size of the binary wheel. In 1.7, XGBoost reduces the size of the wheel by pruning unused CUDA architectures. (#8179, #8152, #8150)

• Fixes Some noteworthy fixes are listed here:

  • Fix the Dask interface with the latest cupy. (#8210)
  • Check cuDF lazily to avoid potential errors with cuda-python. (#8084)

• Fix potential error in DMatrix constructor on 32-bit platform. (#8369)

• Maintenance work

  • Linter script is moved from dmlc-core to XGBoost with added support for formatting, mypy, and parallel run, along with some fixes (#7967, #8101, #8216)
  • We now require the use of isort and black for selected files. (#8137, #8096)
  • Code cleanups. (#7827)
  • Deprecate use_label_encoder in XGBClassifier. The label encoder has already been deprecated and removed in the previous version. These changes only affect the indicator parameter (#7822)
  • Remove the use of distutils. (#7770)
  • Refactor and fixes for tests (#8077, #8064, #8078, #8076, #8013, #8010, #8244, #7833)

• Documents

  • [dask] Fix potential error in demo. (#8079)
  • Improved documentation for the ranker. (#8356, #8347)
  • Indicate lack of py-xgboost-gpu on Windows (#8127)
  • Clarification for feature importance. (#8151)
  • Simplify Python getting started example (#8153)

R Package

We summarize improvements for the R package briefly here:

• Feature info including names and types are now passed to DMatrix in preparation for categorical feature support. (#804)
• XGBoost 1.7 can now gracefully load old R models from RDS for better compatibility with 3-party tuning libraries (#7864)
• The R package now can be built with parallel compilation, along with fixes for warnings in CRAN tests. (#8330)
• Emit error early if DiagrammeR is missing (#8037)
• Fix R package Windows build. (#8065)

JVM Packages

The consistency between JVM packages and other language bindings is greatly improved in 1.7, improvements range from model serialization format to the default value of hyper-parameters.

• Java package now supports feature names and feature types for DMatrix in preparation for categorical feature support. (#7966)
• Models trained by the JVM packages can now be safely used with other language bindings. (#7896, #7907)
• Users can specify the model format when saving models with a stream. (#7940, #7955)
• The default value for training parameters is now sourced from XGBoost directly, which helps JVM packages be consistent with other packages. (#7938)
• Set the correct objective if the user doesn't explicitly set it (#7781)
• Auto-detection of MUSL is replaced by system properties (#7921)
• Improved error message for launching tracker. (#7952, #7968)
• Fix a race condition in parameter configuration. (#8025)
• [Breaking] timeoutRequestWorkers is now removed. With the support for barrier mode, this parameter is no longer needed. (#7839)
• Dependencies updates. (#7791, #8157, #7801, #8240)

Documents

• Document for the C interface is greatly improved and is now displayed at the sphinx document page. Thanks to the breathe project, you can view the C API just like the Python API. (#8300)
• We now avoid having XGBoost internal text parser in demos and recommend users use dedicated libraries for loading data whenever it's feasible. (#7753)
• Python survival training demos are now displayed at sphinx gallery. (#8328)
• Some typos, links, format, and grammar fixes. (#7800, #7832, #7861, #8099, #8163, #8166, #8229, #8028, #8214, #7777, #7905, #8270, #8309, d70e59fef, #7806)
• Updated winning solution under readme.md (#7862)
• New security policy. (#8360)
• GPU document is overhauled as we consider CUDA support to be feature-complete. (#8378)

Maintenance

• Code refactoring and cleanups. (#7850, #7826, #7910, #8332, #8204)
• Reduce compiler warnings. (#7768, #7916, #8046, #8059, #7974, #8031, #8022)
• Compiler workarounds. (#8211, #8314, #8226, #8093)
• Dependencies update. (#8001, #7876, #7973, #8298, #7816)
• Remove warnings emitted in previous versions. (#7815)
• Small fixes occurred during development. (#8008)

CI and Tests

• We overhauled the CI infrastructure to reduce the CI cost and lift the maintenance burdens. Jenkins is replaced with buildkite for better automation, with which, finer control of test runs is implemented to reduce overall cost. Also, we refactored some of the existing tests to reduce their runtime, drooped the size of docker images, and removed multi-GPU C++ tests. Lastly, pytest-timeout is added as an optional dependency for running Python tests to keep the test time in check. (#7772, #8291, #8286, #8276, #8306, #8287, #8243, #8313, #8235, #8288, #8303, #8142, #8092, #8333, #8312, #8348)
• New documents for how to reproduce the CI environment (#7971, #8297)
• Improved automation for JVM release. (#7882)
• GitHub Action security-related updates. (#8263, #8267, #8360)
• Other fixes and maintenance work. (#8154, #7848, #8069, #7943)
• Small updates and fixes to GitHub action pipelines. (#8364, #8321, #8241, #7950, #8011)

Roadmap: https://github.com/dmlc/xgboost/issues/8282 Release note: https://github.com/dmlc/xgboost/pull/8374

Release status: https://github.com/dmlc/xgboost/issues/8366

This is a patch release for bug fixes.

• Remove pyarrow workaround. (#7884)
• Fix monotone constraint with tuple input. (#7891)
• Verify shared object version at load. (#7928)
• Fix LTR with weighted Quantile DMatrix. (#7975)
• Fix Python package source install. (#8036)
• Limit max_depth to 30 for GPU. (#8098)
• Fix compatibility with the latest cupy. (#8129)
• [dask] Deterministic rank assignment. (#8018)
• Fix loading DMatrix binary in distributed env. (#8149)

v1.6.1 (2022 May 9)

This is a patch release for bug fixes and Spark barrier mode support. The R package is unchanged.

Experimental support for categorical data

• Fix segfault when the number of samples is smaller than the number of categories. (https://github.com/dmlc/xgboost/pull/7853)
• Enable partition-based split for all model types. (https://github.com/dmlc/xgboost/pull/7857)

JVM packages

We replaced the old parallelism tracker with spark barrier mode to improve the robustness of the JVM package and fix the GPU training pipeline.

• Fix GPU training pipeline quantile synchronization. (#7823, #7834)
• Use barrier model in spark package. (https://github.com/dmlc/xgboost/pull/7836, https://github.com/dmlc/xgboost/pull/7840, https://github.com/dmlc/xgboost/pull/7845, https://github.com/dmlc/xgboost/pull/7846)
• Fix shared object loading on some platforms. (https://github.com/dmlc/xgboost/pull/7844)

Artifacts

You can verify the downloaded packages by running this on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

2633f15e7be402bad0660d270e0b9a84ad6fcfd1c690a5d454efd6d55b4e395b  ./xgboost.tar.gz

v1.6.0 (2022 Apr 16)

After a long period of development, XGBoost v1.6.0 is packed with many new features and improvements. We summarize them in the following sections starting with an introduction to some major new features, then moving on to language binding specific changes including new features and notable bug fixes for that binding.

Development of categorical data support

This version of XGBoost features new improvements and full coverage of experimental categorical data support in Python and C package with tree model. Both hist, approx and gpu_hist now support training with categorical data. Also, partition-based categorical split is introduced in this release. This split type is first available in LightGBM in the context of gradient boosting. The previous XGBoost release supported one-hot split where the splitting criteria is of form x \in {c}, i.e. the categorical feature x is tested against a single candidate. The new release allows for more expressive conditions: x \in S where the categorical feature x is tested against multiple candidates. Moreover, it is now possible to use any tree algorithms (hist, approx, gpu_hist) when creating categorical splits. For more information, please see our tutorial on categorical data, along with examples linked on that page. (#7380, #7708, #7695, #7330, #7307, #7322, #7705, #7652, #7592, #7666, #7576, #7569, #7529, #7575, #7393, #7465, #7385, #7371, #7745, #7810)

In the future, we will continue to improve categorical data support with new features and optimizations. Also, we are looking forward to bringing the feature beyond Python binding, contributions and feedback are welcomed! Lastly, as a result of experimental status, the behavior might be subject to change, especially the default value of related hyper-parameters.

Experimental support for multi-output model

XGBoost 1.6 features initial support for the multi-output model, which includes multi-output regression and multi-label classification. Along with this, the XGBoost classifier has proper support for base margin without to need for the user to flatten the input. In this initial support, XGBoost builds one model for each target similar to the sklearn meta estimator, for more details, please see our quick introduction.

(#7365, #7736, #7607, #7574, #7521, #7514, #7456, #7453, #7455, #7434, #7429, #7405, #7381)

External memory support

External memory support for both approx and hist tree method is considered feature complete in XGBoost 1.6. Building upon the iterator-based interface introduced in the previous version, now both hist and approx iterates over each batch of data during training and prediction. In previous versions, hist concatenates all the batches into an internal representation, which is removed in this version. As a result, users can expect higher scalability in terms of data size but might experience lower performance due to disk IO. (#7531, #7320, #7638, #7372)

Rewritten approx

The approx tree method is rewritten based on the existing hist tree method. The rewrite closes the feature gap between approx and hist and improves the performance. Now the behavior of approx should be more aligned with hist and gpu_hist. Here is a list of user-visible changes:

• Supports both max_leaves and max_depth.
• Supports grow_policy.
• Supports monotonic constraint.
• Supports feature weights.
• Use max_bin to replace sketch_eps.
• Supports categorical data.
• Faster performance for many of the datasets.
• Improved performance and robustness for distributed training.
• Supports prediction cache.
• Significantly better performance for external memory when depthwise policy is used.

New serialization format

Based on the existing JSON serialization format, we introduce UBJSON support as a more efficient alternative. Both formats will be available in the future and we plan to gradually phase out support for the old binary model format. Users can opt to use the different formats in the serialization function by providing the file extension json or ubj. Also, the save_raw function in all supported languages bindings gains a new parameter for exporting the model in different formats, available options are json, ubj, and deprecated, see document for the language binding you are using for details. Lastly, the default internal serialization format is set to UBJSON, which affects Python pickle and R RDS. (#7572, #7570, #7358, #7571, #7556, #7549, #7416)

General new features and improvements

Aside from the major new features mentioned above, some others are summarized here:

• Users can now access the build information of XGBoost binary in Python and C interface. (#7399, #7553)
• Auto-configuration of seed_per_iteration is removed, now distributed training should generate closer results to single node training when sampling is used. (#7009)
• A new parameter huber_slope is introduced for the Pseudo-Huber objective.
• During source build, XGBoost can choose cub in the system path automatically. (#7579)
• XGBoost now honors the CPU counts from CFS, which is usually set in docker environments. (#7654, #7704)
• The metric aucpr is rewritten for better performance and GPU support. (#7297, #7368)
• Metric calculation is now performed in double precision. (#7364)
• XGBoost no longer mutates the global OpenMP thread limit. (#7537, #7519, #7608, #7590, #7589, #7588, #7687)
• The default behavior of max_leave and max_depth is now unified (#7302, #7551).
• CUDA fat binary is now compressed. (#7601)
• Deterministic result for evaluation metric and linear model. In previous versions of XGBoost, evaluation results might differ slightly for each run due to parallel reduction for floating-point values, which is now addressed. (#7362, #7303, #7316, #7349)
• XGBoost now uses double for GPU Hist node sum, which improves the accuracy of gpu_hist. (#7507)

Performance improvements

Most of the performance improvements are integrated into other refactors during feature developments. The approx should see significant performance gain for many datasets as mentioned in the previous section, while the hist tree method also enjoys improved performance with the removal of the internal pruner along with some other refactoring. Lastly, gpu_hist no longer synchronizes the device during training. (#7737)

General bug fixes

This section lists bug fixes that are not specific to any language binding.

• The num_parallel_tree is now a model parameter instead of a training hyper-parameter, which fixes model IO with random forest. (#7751)
• Fixes in CMake script for exporting configuration. (#7730)
• XGBoost can now handle unsorted sparse input. This includes text file formats like libsvm and scipy sparse matrix where column index might not be sorted. (#7731)
• Fix tree param feature type, this affects inputs with the number of columns greater than the maximum value of int32. (#7565)
• Fix external memory with gpu_hist and subsampling. (#7481)
• Check the number of trees in inplace predict, this avoids a potential segfault when an incorrect value for iteration_range is provided. (#7409)
• Fix non-stable result in cox regression (#7756)

Changes in the Python package

Other than the changes in Dask, the XGBoost Python package gained some new features and improvements along with small bug fixes.

• Python 3.7 is required as the lowest Python version. (#7682)
• Pre-built binary wheel for Apple Silicon. (#7621, #7612, #7747) Apple Silicon users will now be able to run pip install xgboost to install XGBoost.
• MacOS users no longer need to install libomp from Homebrew, as the XGBoost wheel now bundles libomp.dylib library.
• There are new parameters for users to specify the custom metric with new behavior. XGBoost can now output transformed prediction values when a custom objective is not supplied. See our explanation in the tutorial for details.
• For the sklearn interface, following the estimator guideline from scikit-learn, all parameters in fit that are not related to input data are moved into the constructor and can be set by set_params. (#6751, #7420, #7375, #7369)
• Apache arrow format is now supported, which can bring better performance to users' pipeline (#7512)
• Pandas nullable types are now supported (#7760)
• A new function get_group is introduced for DMatrix to allow users to get the group information in the custom objective function. (#7564)
• More training parameters are exposed in the sklearn interface instead of relying on the **kwargs. (#7629)
• A new attribute feature_names_in_ is defined for all sklearn estimators like XGBRegressor to follow the convention of sklearn. (#7526)
• More work on Python type hint. (#7432, #7348, #7338, #7513, #7707)
• Support the latest pandas Index type. (#7595)
• Fix for Feature shape mismatch error on s390x platform (#7715)
• Fix using feature names for constraints with multiple groups (#7711)
• We clarified the behavior of the callback function when it contains mutable states. (#7685)
• Lastly, there are some code cleanups and maintenance work. (#7585, #7426, #7634, #7665, #7667, #7377, #7360, #7498, #7438, #7667, #7752, #7749, #7751)

Changes in the Dask interface

• Dask module now supports user-supplied host IP and port address of scheduler node. Please see introduction and API document for reference. (#7645, #7581)
• Internal DMatrix construction in dask now honers thread configuration. (#7337)
• A fix for nthread configuration using the Dask sklearn interface. (#7633)
• The Dask interface can now handle empty partitions. An empty partition is different from an empty worker, the latter refers to the case when a worker has no partition of an input dataset, while the former refers to some partitions on a worker that has zero sizes. (#7644, #7510)
• Scipy sparse matrix is supported as Dask array partition. (#7457)
• Dask interface is no longer considered experimental. (#7509)

Changes in the R package

This section summarizes the new features, improvements, and bug fixes to the R package.

• load.raw can optionally construct a booster as return. (#7686)
• Fix parsing decision stump, which affects both transforming text representation to data table and plotting. (#7689)
• Implement feature weights. (#7660)
• Some improvements for complying the CRAN release policy. (#7672, #7661, #7763)
• Support CSR data for predictions (#7615)
• Document update (#7263, #7606)
• New maintainer for the CRAN package (#7691, #7649)
• Handle non-standard installation of toolchain on macos (#7759)

Changes in JVM-packages

Some new features for JVM-packages are introduced for a more integrated GPU pipeline and better compatibility with musl-based Linux. Aside from this, we have a few notable bug fixes.

• User can specify the tracker IP address for training, which helps running XGBoost on restricted network environments. (#7808)
• Add support for detecting musl-based Linux (#7624)
• Add DeviceQuantileDMatrix to Scala binding (#7459)
• Add Rapids plugin support, now more of the JVM pipeline can be accelerated by RAPIDS (#7491, #7779, #7793, #7806)
• The setters for CPU and GPU are more aligned (#7692, #7798)
• Control logging for early stopping (#7326)
• Do not repartition when nWorker = 1 (#7676)
• Fix the prediction issue for multi:softmax (#7694)
• Fix for serialization of custom objective and eval (#7274)
• Update documentation about Python tracker (#7396)
• Remove jackson from dependency, which fixes CVE-2020-36518. (#7791)
• Some refactoring to the training pipeline for better compatibility between CPU and GPU. (#7440, #7401, #7789, #7784)
• Maintenance work. (#7550, #7335, #7641, #7523, #6792, #4676)

Deprecation

Other than the changes in the Python package and serialization, we removed some deprecated features in previous releases. Also, as mentioned in the previous section, we plan to phase out the old binary format in future releases.

• Remove old warning in 1.3 (#7279)
• Remove label encoder deprecated in 1.3. (#7357)
• Remove old callback deprecated in 1.3. (#7280)
• Pre-built binary will no longer support deprecated CUDA architectures including sm35 and sm50. Users can continue to use these platforms with source build. (#7767)

Documentation

This section lists some of the general changes to XGBoost's document, for language binding specific change please visit related sections.

• Document is overhauled to use the new RTD theme, along with integration of Python examples using Sphinx gallery. Also, we replaced most of the hard-coded URLs with sphinx references. (#7347, #7346, #7468, #7522, #7530)
• Small update along with fixes for broken links, typos, etc. (#7684, #7324, #7334, #7655, #7628, #7623, #7487, #7532, #7500, #7341, #7648, #7311)
• Update document for GPU. [skip ci] (#7403)
• Document the status of RTD hosting. (#7353)
• Update document for building from source. (#7664)
• Add note about CRAN release [skip ci] (#7395)

Maintenance

This is a summary of maintenance work that is not specific to any language binding.

• Add CMake option to use /MD runtime (#7277)
• Add clang-format configuration. (#7383)
• Code cleanups (#7539, #7536, #7466, #7499, #7533, #7735, #7722, #7668, #7304, #7293, #7321, #7356, #7345, #7387, #7577, #7548, #7469, #7680, #7433, #7398)
• Improved tests with better coverage and latest dependency (#7573, #7446, #7650, #7520, #7373, #7723, #7611, #7771)
• Improved automation of the release process. (#7278, #7332, #7470)
• Compiler workarounds (#7673)
• Change shebang used in CLI demo. (#7389)
• Update affiliation (#7289)

CI

Some fixes and update to XGBoost's CI infrastructure. (#7739, #7701, #7382, #7662, #7646, #7582, #7407, #7417, #7475, #7474, #7479, #7472, #7626)

Roadmap: https://github.com/dmlc/xgboost/issues/7726 Release note: https://github.com/dmlc/xgboost/pull/7746

This is a patch release for compatibility with latest dependencies and bug fixes.

• [dask] Fix asyncio with latest dask and distributed.
• [R] Fix single sample SHAP prediction.
• [Python] Update python classifier to indicate support for latest Python versions.
• [Python] Fix with latest mypy and pylint.
• Fix indexing type for bitfield, which may affect missing value and categorical data.
• Fix num_boosted_rounds for linear model.
• Fix early stopping with linear model.

This is a patch release for compatibility with the latest dependencies and bug fixes. Also, all GPU-compatible binaries are built with CUDA 11.0.

• [Python] Handle missing values in dataframe with category dtype. (#7331)

• [R] Fix R CRAN failures about prediction and some compiler warnings.

• [JVM packages] Fix compatibility with latest Spark (#7438, #7376)

• Support building with CTK11.5. (#7379)

• Check user input for iteration in inplace predict.

• Handle OMP_THREAD_LIMIT environment variable.

• [doc] Fix broken links. (#7341)

Artifacts

You can verify the downloaded packages by running this on your Unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

3a6cc7526c0dff1186f01b53dcbac5c58f12781988400e2d340dda61ef8d14ca  xgboost_r_gpu_linux_afb9dfd4210e8b8db8fe03380f83b404b1721443.tar.gz
6f74deb62776f1e2fd030e1fa08b93ba95b32ac69cc4096b4bcec3821dd0a480  xgboost_r_gpu_win64_afb9dfd4210e8b8db8fe03380f83b404b1721443.tar.gz
565dea0320ed4b6f807dbb92a8a57e86ec16db50eff9a3f405c651d1f53a259d  xgboost.tar.gz

This release comes with many exciting new features and optimizations, along with some bug fixes. We will describe the experimental categorical data support and the external memory interface independently. Package-specific new features will be listed in respective sections.

Development on categorical data support

In version 1.3, XGBoost introduced an experimental feature for handling categorical data natively, without one-hot encoding. XGBoost can fit categorical splits in decision trees. (Currently, the generated splits will be of form x \in {v}, where the input is compared to a single category value. A future version of XGBoost will generate splits that compare the input against a list of multiple category values.)

Most of the other features, including prediction, SHAP value computation, feature importance, and model plotting were revised to natively handle categorical splits. Also, all Python interfaces including native interface with and without quantized DMatrix, scikit-learn interface, and Dask interface now accept categorical data with a wide range of data structures support including numpy/cupy array and cuDF/pandas/modin dataframe. In practice, the following are required for enabling categorical data support during training:

• Use Python package.
• Use gpu_hist to train the model.
• Use JSON model file format for saving the model.

Once the model is trained, it can be used with most of the features that are available on the Python package. For a quick introduction, see https://xgboost.readthedocs.io/en/latest/tutorials/categorical.html

Related PRs: (#7011, #7001, #7042, #7041, #7047, #7043, #7036, #7054, #7053, #7065, #7213, #7228, #7220, #7221, #7231, #7306)

• Next steps

  • Revise the CPU training algorithm to handle categorical data natively and generate categorical splits
  • Extend the CPU and GPU algorithms to generate categorical splits of form x \in S where the input is compared with multiple category values. split. (#7081)

External memory

This release features a brand-new interface and implementation for external memory (also known as out-of-core training). (#6901, #7064, #7088, #7089, #7087, #7092, #7070, #7216). The new implementation leverages the data iterator interface, which is currently used to create DeviceQuantileDMatrix. For a quick introduction, see https://xgboost.readthedocs.io/en/latest/tutorials/external_memory.html#data-iterator . During the development of this new interface, lz4 compression is removed. (#7076). Please note that external memory support is still experimental and not ready for production use yet. All future development will focus on this new interface and users are advised to migrate. (You are using the old interface if you are using a URL suffix to use external memory.)

New features in Python package

• Support numpy array interface and all numeric types from numpy in DMatrix construction and inplace_predict (#6998, #7003). Now XGBoost no longer makes data copy when input is numpy array view.
• The early stopping callback in Python has a new min_delta parameter to control the stopping behavior (#7137)
• Python package now supports calculating feature scores for the linear model, which is also available on R package. (#7048)
• Python interface now supports configuring constraints using feature names instead of feature indices.
• Typehint support for more Python code including scikit-learn interface and rabit module. (#6799, #7240)
• Add tutorial for XGBoost-Ray (#6884)

New features in R package

• In 1.4 we have a new prediction function in the C API which is used by the Python package. This release revises the R package to use the new prediction function as well. A new parameter iteration_range for the predict function is available, which can be used for specifying the range of trees for running prediction. (#6819, #7126)
• R package now supports the nthread parameter in DMatrix construction. (#7127)

New features in JVM packages

• Support GPU dataframe and DeviceQuantileDMatrix (#7195). Constructing DMatrix with GPU data structures and the interface for quantized DMatrix were first introduced in the Python package and are now available in the xgboost4j package.
• JVM packages now support saving and getting early stopping attributes. (#7095) Here is a quick example in JAVA (#7252).

General new features

• We now have a pre-built binary package for R on Windows with GPU support. (#7185)
• CUDA compute capability 86 is now part of the default CMake build configuration with newly added support for CUDA 11.4. (#7131, #7182, #7254)
• XGBoost can be compiled using system CUB provided by CUDA 11.x installation. (#7232)

Optimizations

The performance for both hist and gpu_hist has been significantly improved in 1.5 with the following optimizations:

• GPU multi-class model training now supports prediction cache. (#6860)
• GPU histogram building is sped up and the overall training time is 2-3 times faster on large datasets (#7180, #7198). In addition, we removed the parameter deterministic_histogram and now the GPU algorithm is always deterministic.
• CPU hist has an optimized procedure for data sampling (#6922)
• More performance optimization in regression and binary classification objectives on CPU (#7206)
• Tree model dump is now performed in parallel (#7040)

Breaking changes

• n_gpus was deprecated in 1.0 release and is now removed.
• Feature grouping in CPU hist tree method is removed, which was disabled long ago. (#7018)
• C API for Quantile DMatrix is changed to be consistent with the new external memory implementation. (#7082)

Notable general bug fixes

• XGBoost no long changes global CUDA device ordinal when gpu_id is specified (#6891, #6987)
• Fix gamma negative likelihood evaluation metric. (#7275)
• Fix integer value of verbose_eal for xgboost.cv function in Python. (#7291)
• Remove extra sync in CPU hist for dense data, which can lead to incorrect tree node statistics. (#7120, #7128)
• Fix a bug in GPU hist when data size is larger than UINT32_MAX with missing values. (#7026)
• Fix a thread safety issue in prediction with the softmax objective. (#7104)
• Fix a thread safety issue in CPU SHAP value computation. (#7050) Please note that all prediction functions in Python are thread-safe.
• Fix model slicing. (#7149, #7078)
• Workaround a bug in old GCC which can lead to segfault during construction of DMatrix. (#7161)
• Fix histogram truncation in GPU hist, which can lead to slightly-off results. (#7181)
• Fix loading GPU linear model pickle files on CPU-only machine. (#7154)
• Check input value is duplicated when CPU quantile queue is full (#7091)
• Fix parameter loading with training continuation. (#7121)
• Fix CMake interface for exposing C library by specifying dependencies. (#7099)
• Callback and early stopping are explicitly disabled for the scikit-learn interface random forest estimator. (#7236)
• Fix compilation error on x86 (32-bit machine) (#6964)
• Fix CPU memory usage with extremely sparse datasets (#7255)
• Fix a bug in GPU multi-class AUC implementation with weighted data (#7300)

Python package

Other than the items mentioned in the previous sections, there are some Python-specific improvements.

• Change development release postfix to dev (#6988)
• Fix early stopping behavior with MAPE metric (#7061)
• Fixed incorrect feature mismatch error message (#6949)
• Add predictor to skl constructor. (#7000, #7159)
• Re-enable feature validation in predict proba. (#7177)
• scikit learn interface regression estimator now can pass the scikit-learn estimator check and is fully compatible with scikit-learn utilities. __sklearn_is_fitted__ is implemented as part of the changes (#7130, #7230)
• Conform the latest pylint. (#7071, #7241)
• Support latest panda range index in DMatrix construction. (#7074)
• Fix DMatrix construction from pandas series. (#7243)
• Fix typo and grammatical mistake in error message (#7134)
• [dask] disable work stealing explicitly for training tasks (#6794)
• [dask] Set dataframe index in predict. (#6944)
• [dask] Fix prediction on df with latest dask. (#6969)
• [dask] Fix dask predict on DaskDMatrix with iteration_range. (#7005)
• [dask] Disallow importing non-dask estimators from xgboost.dask (#7133)

R package

Improvements other than new features on R package:

• Optimization for updating R handles in-place (#6903)
• Removed the magrittr dependency. (#6855, #6906, #6928)
• The R package now hides all C++ symbols to avoid conflicts. (#7245)
• Other maintenance including code cleanups, document updates. (#6863, #6915, #6930, #6966, #6967)

JVM packages

Improvements other than new features on JVM packages:

• Constructors with implicit missing value are deprecated due to confusing behaviors. (#7225)
• Reduce scala-compiler, scalatest dependency scopes (#6730)
• Making the Java library loader emit helpful error messages on missing dependencies. (#6926)
• JVM packages now use the Python tracker in XGBoost instead of dmlc. The one in XGBoost is shared between JVM packages and Python Dask and enjoys better maintenance (#7132)
• Fix "key not found: train" error (#6842)
• Fix model loading from stream (#7067)

General document improvements

• Overhaul the installation documents. (#6877)
• A few demos are added for AFT with dask (#6853), callback with dask (#6995), inference in C (#7151), process_type. (#7135)
• Fix PDF format of document. (#7143)
• Clarify the behavior of use_rmm. (#6808)
• Clarify prediction function. (#6813)
• Improve tutorial on feature interactions (#7219)
• Add small example for dask sklearn interface. (#6970)
• Update Python intro. (#7235)
• Some fixes/updates (#6810, #6856, #6935, #6948, #6976, #7084, #7097, #7170, #7173, #7174, #7226, #6979, #6809, #6796, #6979)

Maintenance

• Some refactoring around CPU hist, which lead to better performance but are listed under general maintenance tasks:

  • Extract evaluate splits from CPU hist. (#7079)
  • Merge lossgude and depthwise strategies for CPU hist (#7007)
  • Simplify sparse and dense CPU hist kernels (#7029)
  • Extract histogram builder from CPU Hist. (#7152)

• Others

  • Fix gpu_id with custom objective. (#7015)
  • Fix typos in AUC. (#6795)
  • Use constexpr in dh::CopyIf. (#6828)
  • Update dmlc-core. (#6862)
  • Bump version to 1.5.0 snapshot in master. (#6875)
  • Relax shotgun test. (#6900)
  • Guard against index error in prediction. (#6982)
  • Hide symbols in CI build + hide symbols for C and CUDA (#6798)
  • Persist data in dask test. (#7077)
  • Fix typo in arguments of PartitionBuilder::Init (#7113)
  • Fix typo in src/common/hist.cc BuildHistKernel (#7116)
  • Use upstream URI in distributed quantile tests. (#7129)
  • Include cpack (#7160)
  • Remove synchronization in monitor. (#7164)
  • Remove unused code. (#7175)
  • Fix building on CUDA 11.0. (#7187)
  • Better error message for ncclUnhandledCudaError. (#7190)
  • Add noexcept to JSON objects. (#7205)
  • Improve wording for warning (#7248)
  • Fix typo in release script. [skip ci] (#7238)
  • Relax shotgun test. (#6918)
  • Relax test for decision stump in distributed environment. (#6919)
  • [dask] speed up tests (#7020)

CI

• [CI] Rotate access keys for uploading MacOS artifacts from Travis CI (#7253)
• Reduce Travis environment setup time. (#6912)
• Restore R cache on github action. (#6985)
• [CI] Remove stray build artifact to avoid error in artifact packaging (#6994)
• [CI] Move appveyor tests to action (#6986)
• Remove appveyor badge. [skip ci] (#7035)
• [CI] Configure RAPIDS, dask, modin (#7033)
• Test on s390x. (#7038)
• [CI] Upgrade to CMake 3.14 (#7060)
• [CI] Update R cache. (#7102)
• [CI] Pin libomp to 11.1.0 (#7107)
• [CI] Upgrade build image to CentOS 7 + GCC 8; require CUDA 10.1 and later (#7141)
• [dask] Work around segfault in prediction. (#7112)
• [dask] Remove the workaround for segfault. (#7146)
• [CI] Fix hanging Python setup in Windows CI (#7186)
• [CI] Clean up in beginning of each task in Win CI (#7189)
• Fix travis. (#7237)

Acknowledgement

• Contributors: Adam Pocock (@Craigacp), Jeff H (@JeffHCross), Johan Hansson (@JohanWork), Jose Manuel Llorens (@JoseLlorensRipolles), Benjamin Szőke (@Livius90), @ReeceGoding, @ShvetsKS, Robert Zabel (@ZabelTech), Ali (@ali5h), Andrew Ziem (@az0), Andy Adinets (@canonizer), @david-cortes, Daniel Saxton (@dsaxton), Emil Sadek (@esadek), @farfarawayzyt, Gil Forsyth (@gforsyth), @giladmaya, @graue70, Philip Hyunsu Cho (@hcho3), James Lamb (@jameslamb), José Morales (@jmoralez), Kai Fricke (@krfricke), Christian Lorentzen (@lorentzenchr), Mads R. B. Kristensen (@madsbk), Anton Kostin (@masguit42), Martin Petříček (@mpetricek-corp), @naveenkb, Taewoo Kim (@oOTWK), Viktor Szathmáry (@phraktle), Robert Maynard (@robertmaynard), TP Boudreau (@tpboudreau), Jiaming Yuan (@trivialfis), Paul Taylor (@trxcllnt), @vslaykovsky, Bobby Wang (@wbo4958),
• Reviewers: Nan Zhu (@CodingCat), Adam Pocock (@Craigacp), Jose Manuel Llorens (@JoseLlorensRipolles), Kodi Arfer (@Kodiologist), Benjamin Szőke (@Livius90), Mark Guryanov (@MarkGuryanov), Rory Mitchell (@RAMitchell), @ReeceGoding, @ShvetsKS, Egor Smirnov (@SmirnovEgorRu), Andrew Ziem (@az0), @candalfigomoro, Andy Adinets (@canonizer), Dante Gama Dessavre (@dantegd), @david-cortes, Daniel Saxton (@dsaxton), @farfarawayzyt, Gil Forsyth (@gforsyth), Harutaka Kawamura (@harupy), Philip Hyunsu Cho (@hcho3), @jakirkham, James Lamb (@jameslamb), José Morales (@jmoralez), James Bourbeau (@jrbourbeau), Christian Lorentzen (@lorentzenchr), Martin Petříček (@mpetricek-corp), Nikolay Petrov (@napetrov), @naveenkb, Viktor Szathmáry (@phraktle), Robin Teuwens (@rteuwens), Yuan Tang (@terrytangyuan), TP Boudreau (@tpboudreau), Jiaming Yuan (@trivialfis), @vkuzmin-uber, Bobby Wang (@wbo4958), William Hicks (@wphicks)

Artifacts

You can verify the downloaded packages by running this on your unix shell:

echo "<hash> <artifact>" | shasum -a 256 --check

2c63e8abd3e89795ac9371688daa31109a9514eebd9db06956ba5aa41d0c0e20  xgboost_r_gpu_linux_1.5.0.tar.gz
8b19f817dcb6b601b0abffa9cf943ee92c3e9a00f56fa3f4fcdfe98cd3777c04  xgboost_r_gpu_win64_1.5.0.tar.gz
25ee3adb9925d0529575c0f00a55ba42202a1cdb5fdd3fb6484b4088571326a5  xgboost.tar.gz

Roadmap: https://github.com/dmlc/xgboost/issues/6846 RC: https://github.com/dmlc/xgboost/issues/7260 Release note: https://github.com/dmlc/xgboost/pull/7271

This is a patch release for Python package with following fixes:

• Handle the latest version of cupy.ndarray in inplace_predict. https://github.com/dmlc/xgboost/pull/6933
• Ensure output array from predict_leaf is (n_samples, ) when there's only 1 tree. 1.4.0 outputs (n_samples, 1). https://github.com/dmlc/xgboost/pull/6889
• Fix empty dataset handling with multi-class AUC. https://github.com/dmlc/xgboost/pull/6947
• Handle object type from pandas in inplace_predict. https://github.com/dmlc/xgboost/pull/6927

You can verify the downloaded source code xgboost.tar.gz by running this on your unix shell:

echo "3ffd4a90cd03efde596e51cadf7f344c8b6c91aefd06cc92db349cd47056c05a *xgboost.tar.gz" | shasum -a 256 --check

This is a bug fix release.

• Fix GPU implementation of AUC on some large datasets. (#6866)

You can verify the downloaded source code xgboost.tar.gz by running this on your unix shell:

echo "f3a37e5ddac10786e46423db874b29af413eed49fd9baed85035bbfee6fc6635 *xgboost.tar.gz" | shasum -a 256 --check

Introduction of pre-built binary package for R, with GPU support

Starting with release 1.4.0, users now have the option of installing {xgboost} without having to build it from the source. This is particularly advantageous for users who want to take advantage of the GPU algorithm (gpu_hist), as previously they'd have to build {xgboost} from the source using CMake and NVCC. Now installing {xgboost} with GPU support is as easy as: R CMD INSTALL ./xgboost_r_gpu_linux.tar.gz. (#6827)

See the instructions at https://xgboost.readthedocs.io/en/latest/build.html

Improvements on prediction functions

XGBoost has many prediction types including shap value computation and inplace prediction. In 1.4 we overhauled the underlying prediction functions for C API and Python API with an unified interface. (#6777, #6693, #6653, #6662, #6648, #6668, #6804)

• Starting with 1.4, sklearn interface prediction will use inplace predict by default when input data is supported.
• Users can use inplace predict with dart booster and enable GPU acceleration just like gbtree.
• Also all prediction functions with tree models are now thread-safe. Inplace predict is improved with base_margin support.
• A new set of C predict functions are exposed in the public interface.
• A user-visible change is a newly added parameter called strict_shape. See https://xgboost.readthedocs.io/en/latest/prediction.html for more details.

Improvement on Dask interface

• Starting with 1.4, the Dask interface is considered to be feature-complete, which means all of the models found in the single node Python interface are now supported in Dask, including but not limited to ranking and random forest. Also, the prediction function is significantly faster and supports shap value computation.

  • Most of the parameters found in single node sklearn interface are supported by Dask interface. (#6471, #6591)
  • Implements learning to rank. On the Dask interface, we use the newly added support of query ID to enable group structure. (#6576)
  • The Dask interface has Python type hints support. (#6519)
  • All models can be safely pickled. (#6651)
  • Random forest estimators are now supported. (#6602)
  • Shap value computation is now supported. (#6575, #6645, #6614)
  • Evaluation result is printed on the scheduler process. (#6609)
  • DaskDMatrix (and device quantile dmatrix) now accepts all meta-information. (#6601)

• Prediction optimization. We enhanced and speeded up the prediction function for the Dask interface. See the latest Dask tutorial page in our document for an overview of how you can optimize it even further. (#6650, #6645, #6648, #6668)

• Bug fixes

  • If you are using the latest Dask and distributed where distributed.MultiLock is present, XGBoost supports training multiple models on the same cluster in parallel. (#6743)
  • A bug fix for when using dask.client to launch async task, XGBoost might use a different client object internally. (#6722)

• Other improvements on documents, blogs, tutorials, and demos. (#6389, #6366, #6687, #6699, #6532, #6501)

Python package

With changes from Dask and general improvement on prediction, we have made some enhancements on the general Python interface and IO for booster information. Starting from 1.4, booster feature names and types can be saved into the JSON model. Also some model attributes like best_iteration, best_score are restored upon model load. On sklearn interface, some attributes are now implemented as Python object property with better documents.

• Breaking change: All data parameters in prediction functions are renamed to X for better compliance to sklearn estimator interface guidelines.

• Breaking change: XGBoost used to generate some pseudo feature names with DMatrix when inputs like np.ndarray don't have column names. The procedure is removed to avoid conflict with other inputs. (#6605)

• Early stopping with training continuation is now supported. (#6506)

• Optional import for Dask and cuDF are now lazy. (#6522)

• As mentioned in the prediction improvement summary, the sklearn interface uses inplace prediction whenever possible. (#6718)

• Booster information like feature names and feature types are now saved into the JSON model file. (#6605)

• All DMatrix interfaces including DeviceQuantileDMatrix and counterparts in Dask interface (as mentioned in the Dask changes summary) now accept all the meta-information like group and qid in their constructor for better consistency. (#6601)

• Booster attributes are restored upon model load so users don't have to call attr manually. (#6593)

• On sklearn interface, all models accept base_margin for evaluation datasets. (#6591)

• Improvements over the setup script including smaller sdist size and faster installation if the C++ library is already built (#6611, #6694, #6565).

• Bug fixes for Python package:

  • Don't validate feature when number of rows is 0. (#6472)
  • Move metric configuration into booster. (#6504)
  • Calling XGBModel.fit() should clear the Booster by default (#6562)
  • Support _estimator_type. (#6582)
  • [dask, sklearn] Fix predict proba. (#6566, #6817)
  • Restore unknown data support. (#6595)
  • Fix learning rate scheduler with cv. (#6720)
  • Fixes small typo in sklearn documentation (#6717)
  • [python-package] Fix class Booster: feature_types = None (#6705)
  • Fix divide by 0 in feature importance when no split is found. (#6676)

JVM package

• [jvm-packages] fix early stopping doesn't work even without custom_eval setting (#6738)
• fix potential TaskFailedListener's callback won't be called (#6612)
• [jvm] Add ability to load booster direct from byte array (#6655)
• [jvm-packages] JVM library loader extensions (#6630)

R package

• R documentation: Make construction of DMatrix consistent.
• Fix R documentation for xgb.train. (#6764)

ROC-AUC

We re-implemented the ROC-AUC metric in XGBoost. The new implementation supports multi-class classification and has better support for learning to rank tasks that are not binary. Also, it has a better-defined average on distributed environments with additional handling for invalid datasets. (#6749, #6747, #6797)

Global configuration.

Starting from 1.4, XGBoost's Python, R and C interfaces support a new global configuration model where users can specify some global parameters. Currently, supported parameters are verbosity and use_rmm. The latter is experimental, see rmm plugin demo and related README file for details. (#6414, #6656)

Other New features.

• Better handling for input data types that support __array_interface__. For some data types including GPU inputs and scipy.sparse.csr_matrix, XGBoost employs __array_interface__ for processing the underlying data. Starting from 1.4, XGBoost can accept arbitrary array strides (which means column-major is supported) without making data copies, potentially reducing a significant amount of memory consumption. Also version 3 of __cuda_array_interface__ is now supported. (#6776, #6765, #6459, #6675)
• Improved parameter validation, now feeding XGBoost with parameters that contain whitespace will trigger an error. (#6769)
• For Python and R packages, file paths containing the home indicator ~ are supported.
• As mentioned in the Python changes summary, the JSON model can now save feature information of the trained booster. The JSON schema is updated accordingly. (#6605)
• Development of categorical data support is continued. Newly added weighted data support and dart booster support. (#6508, #6693)
• As mentioned in Dask change summary, ranking now supports the qid parameter for query groups. (#6576)
• DMatrix.slice can now consume a numpy array. (#6368)

Other breaking changes

• Aside from the feature name generation, there are 2 breaking changes:
  • Drop saving binary format for memory snapshot. (#6513, #6640)
  • Change default evaluation metric for binary:logitraw objective to logloss (#6647)

CPU Optimization

• Aside from the general changes on predict function, some optimizations are applied on CPU implementation. (#6683, #6550, #6696, #6700)
• Also performance for sampling initialization in hist is improved. (#6410)

Notable fixes in the core library

These fixes do not reside in particular language bindings:

• Fixes for gamma regression. This includes checking for invalid input values, fixes for gamma deviance metric, and better floating point guard for gamma negative log-likelihood metric. (#6778, #6537, #6761)
• Random forest with gpu_hist might generate low accuracy in previous versions. (#6755)
• Fix a bug in GPU sketching when data size exceeds limit of 32-bit integer. (#6826)
• Memory consumption fix for row-major adapters (#6779)
• Don't estimate sketch batch size when rmm is used. (#6807) (#6830)
• Fix in-place predict with missing value. (#6787)
• Re-introduce double buffer in UpdatePosition, to fix perf regression in gpu_hist (#6757)
• Pass correct split_type to GPU predictor (#6491)
• Fix DMatrix feature names/types IO. (#6507)
• Use view for SparsePage exclusively to avoid some data access races. (#6590)
• Check for invalid data. (#6742)
• Fix relocatable include in CMakeList (#6734) (#6737)
• Fix DMatrix slice with feature types. (#6689)

Other deprecation notices:

• This release will be the last release to support CUDA 10.0. (#6642)

• Starting in the next release, the Python package will require Pip 19.3+ due to the use of manylinux2014 tag. Also, CentOS 6, RHEL 6 and other old distributions will not be supported.

Known issue:

MacOS build of the JVM packages doesn't support multi-threading out of the box. To enable multi-threading with JVM packages, MacOS users will need to build the JVM packages from the source. See https://xgboost.readthedocs.io/en/latest/jvm/index.html#installation-from-source

Doc

• Dedicated page for tree_method parameter is added. (#6564, #6633)
• [doc] Add FLAML as a fast tuning tool for XGBoost (#6770)
• Add document for tests directory. [skip ci] (#6760)
• Fix doc string of config.py to use correct versionadded (#6458)
• Update demo for prediction. (#6789)
• [Doc] Document that AUCPR is for binary classification/ranking (#5899)
• Update the C API comments (#6457)
• Fix document. [skip ci] (#6669)

Maintenance: Testing, continuous integration

• Use CPU input for test_boost_from_prediction. (#6818)
• [CI] Upload xgboost4j.dll to S3 (#6781)
• Update dmlc-core submodule (#6745)
• [CI] Use manylinux2010_x86_64 container to vendor libgomp (#6485)
• Add conda-forge badge (#6502)
• Fix merge conflict. (#6512)
• [CI] Split up main.yml, add mypy. (#6515)
• [Breaking] Upgrade cuDF and RMM to 0.18 nightlies; require RMM 0.18+ for RMM plugin (#6510)
• "featue_map" typo changed to "feature_map" (#6540)
• Add script for generating release tarball. (#6544)
• Add credentials to .gitignore (#6559)
• Remove warnings in tests. (#6554)
• Update dmlc-core submodule and conform to new API (#6431)
• Suppress hypothesis health check for dask client. (#6589)
• Fix pylint. (#6714)
• [CI] Clear R package cache (#6746)
• Exclude dmlc test on github action. (#6625)
• Tests for regression metrics with weights. (#6729)
• Add helper script and doc for releasing pip package. (#6613)
• Support pylint 2.7.0 (#6726)
• Remove R cache in github action. (#6695)
• [CI] Do not mix up stashed executable built for ARM and x86_64 platforms (#6646)
• [CI] Add ARM64 test to Jenkins pipeline (#6643)
• Disable s390x and arm64 tests on travis for now. (#6641)
• Move sdist test to action. (#6635)
• [dask] Rework base margin test. (#6627)

Maintenance: Refactor code for legibility and maintainability

• Improve OpenMP exception handling (#6680)
• Improve string view to reduce string allocation. (#6644)
• Simplify Span checks. (#6685)
• Use generic dispatching routine for array interface. (#6672)

You can verify the downloaded source code xgboost.tar.gz by running this on your unix shell:

echo "ff77130a86aebd83a8b996c76768a867b0a6e5012cce89212afc3df4c4ee6b1c *xgboost.tar.gz" | shasum -a 256 --check

• Fix regression on best_ntree_limit. (#6616)

• Fix compatibility with newer scikit-learn. (https://github.com/dmlc/xgboost/pull/6555)
• Fix wrong best_ntree_limit in multi-class. (https://github.com/dmlc/xgboost/pull/6569)
• Ensure that Rabit can be compiled on Solaris (https://github.com/dmlc/xgboost/pull/6578)
• Fix best_ntree_limit for linear and dart. (https://github.com/dmlc/xgboost/pull/6579)
• Remove duplicated DMatrix creation in scikit-learn interface. (https://github.com/dmlc/xgboost/pull/6592)
• Fix evals_result in XGBRanker. (#https://github.com/dmlc/xgboost/pull/6594)

• Enable loading model from <1.0.0 trained with objective='binary:logitraw' (#6517)
• Fix handling of print period in EvaluationMonitor (#6499)
• Fix a bug in metric configuration after loading model. (#6504)
• Fix save_best early stopping option (#6523)
• Remove cupy.array_equal, since it's not compatible with cuPy 7.8 (#6528)

You can verify the downloaded source code xgboost.tar.gz by running this on your unix shell:

echo "fd51e844dd0291fd9e7129407be85aaeeda2309381a6e3fc104938b27fb09279 *xgboost.tar.gz" | shasum -a 256 --check

#6422

R package: xgboost_1.3.0.1.tar.gz

This patch release applies the following patches to 1.2.0 release:

• Hide C++ symbols from dmlc-core (#6188)

XGBoost4J-Spark now supports the GPU algorithm (#5171)

• Now XGBoost4J-Spark is able to leverage NVIDIA GPU hardware to speed up training.
• There is on-going work for accelerating the rest of the data pipeline with NVIDIA GPUs (#5950, #5972).

XGBoost now supports CUDA 11 (#5808)

• It is now possible to build XGBoost with CUDA 11. Note that we do not yet distribute pre-built binaries built with CUDA 11; all current distributions use CUDA 10.0.

Better guidance for persisting XGBoost models in an R environment (#5940, #5964)

• Users are strongly encouraged to use xgb.save() and xgb.save.raw() instead of saveRDS(). This is so that the persisted models can be accessed with future releases of XGBoost.
• The previous release (1.1.0) had problems loading models that were saved with saveRDS(). This release adds a compatibility layer to restore access to the old RDS files. Note that this is meant to be a temporary measure; users are advised to stop using saveRDS() and migrate to xgb.save() and xgb.save.raw().

New objectives and metrics

• The pseudo-Huber loss reg:pseudohubererror is added (#5647). The corresponding metric is mphe. Right now, the slope is hard-coded to 1.
• The Accelerated Failure Time objective for survival analysis (survival:aft) is now accelerated on GPUs (#5714, #5716). The survival metrics aft-nloglik and interval-regression-accuracy are also accelerated on GPUs.

Improved integration with scikit-learn

• Added n_features_in_ attribute to the scikit-learn interface to store the number of features used (#5780). This is useful for integrating with some scikit-learn features such as StackingClassifier. See this link for more details.
• XGBoostError now inherits ValueError, which conforms scikit-learn's exception requirement (#5696).

Improved integration with Dask

• The XGBoost Dask API now exposes an asynchronous interface (#5862). See the document for details.
• Zero-copy ingestion of GPU arrays via DaskDeviceQuantileDMatrix (#5623, #5799, #5800, #5803, #5837, #5874, #5901): Previously, the Dask interface had to make 2 data copies: one for concatenating the Dask partition/block into a single block and another for internal representation. To save memory, we introduce DaskDeviceQuantileDMatrix. As long as Dask partitions are resident in the GPU memory, DaskDeviceQuantileDMatrix is able to ingest them directly without making copies. This matrix type wraps DeviceQuantileDMatrix.
• The prediction function now returns GPU Series type if the input is from Dask-cuDF (#5710). This is to preserve the input data type.

Robust handling of external data types (#5689, #5893)

• As we support more and more external data types, the handling logic has proliferated all over the code base and became hard to keep track. It also became unclear how missing values and threads are handled. We refactored the Python package code to collect all data handling logic to a central location, and now we have an explicit list of of all supported data types.

Improvements in GPU-side data matrix (DeviceQuantileDMatrix)

• The GPU-side data matrix now implements its own quantile sketching logic, so that data don't have to be transported back to the main memory (#5700, #5747, #5760, #5846, #5870, #5898). The GK sketching algorithm is also now better documented.
  • Now we can load extremely sparse dataset like URL, although performance is still sub-optimal.
• The GPU-side data matrix now exposes an iterative interface (#5783), so that users are able to construct a matrix from a data iterator. See the Python demo.

New language binding: Swift (#5728)

• Visit https://github.com/kongzii/SwiftXGBoost for more details.

Robust model serialization with JSON (#5772, #5804, #5831, #5857, #5934)

• We continue efforts from the 1.0.0 release to adopt JSON as the format to save and load models robustly.
• JSON model IO is significantly faster and produces smaller model files.
• Round-trip reproducibility is guaranteed, via the introduction of an efficient float-to-string conversion algorithm known as the Ryū algorithm. The conversion is locale-independent, producing consistent numeric representation regardless of the locale setting of the user's machine.
• We fixed an issue in loading large JSON files to memory.
• It is now possible to load a JSON file from a remote source such as S3.

Performance improvements

• CPU hist tree method optimization
  • Skip missing lookup in hist row partitioning if data is dense. (#5644)
  • Specialize training procedures for CPU hist tree method on distributed environment. (#5557)
  • Add single point histogram for CPU hist. Previously gradient histogram for CPU hist is hard coded to be 64 bit, now users can specify the parameter single_precision_histogram to use 32 bit histogram instead for faster training performance. (#5624, #5811)
• GPU hist tree method optimization
  • Removed some unnecessary synchronizations and better memory allocation pattern. (#5707)
  • Optimize GPU Hist for wide dataset. Previously for wide dataset the atomic operation is performed on global memory, now it can run on shared memory for faster histogram building. But there's a known small regression on GeForce cards with dense data. (#5795, #5926, #5948, #5631)

API additions

• Support passing fmap to importance plot (#5719). Now importance plot can show actual names of features instead of default ones.
• Support 64bit seed. (#5643)
• A new C API XGBoosterGetNumFeature is added for getting number of features in booster (#5856).
• Feature names and feature types are now stored in C++ core and saved in binary DMatrix (#5858).

Breaking: The predict() method of DaskXGBClassifier now produces class predictions (#5986). Use predict_proba() to obtain probability predictions.

• Previously, DaskXGBClassifier.predict() produced probability predictions. This is inconsistent with the behavior of other scikit-learn classifiers, where predict() returns class predictions. We make a breaking change in 1.2.0 release so that DaskXGBClassifier.predict() now correctly produces class predictions and thus behave like other scikit-learn classifiers. Furthermore, we introduce the predict_proba() method for obtaining probability predictions, again to be in line with other scikit-learn classifiers.

Breaking: Custom evaluation metric now receives raw prediction (#5954)

• Previously, the custom evaluation metric received a transformed prediction result when used with a classifier. Now the custom metric will receive a raw (untransformed) prediction and will need to transform the prediction itself. See demo/guide-python/custom_softmax.py for an example.
• This change is to make the custom metric behave consistently with the custom objective, which already receives raw prediction (#5564).

Breaking: XGBoost4J-Spark now requires Spark 3.0 and Scala 2.12 (#5836, #5890)

• Starting with version 3.0, Spark can manage GPU resources and allocate them among executors.
• Spark 3.0 dropped support for Scala 2.11 and now only supports Scala 2.12. Thus, XGBoost4J-Spark also only supports Scala 2.12.

Breaking: XGBoost Python package now requires Python 3.6 and later (#5715)

• Python 3.6 has many useful features such as f-strings.

Breaking: XGBoost now adopts the C++14 standard (#5664)

• Make sure to use a sufficiently modern C++ compiler that supports C++14, such as Visual Studio 2017, GCC 5.0+, and Clang 3.4+.

Bug-fixes

• Fix a data race in the prediction function (#5853). As a byproduct, the prediction function now uses a thread-local data store and became thread-safe.
• Restore capability to run prediction when the test input has fewer features than the training data (#5955). This capability is necessary to support predicting with LIBSVM inputs. The previous release (1.1) had broken this capability, so we restore it in this version with better tests.
• Fix OpenMP build with CMake for R package, to support CMake 3.13 (#5895).
• Fix Windows 2016 build (#5902, #5918).
• Fix edge cases in scikit-learn interface with Pandas input by disabling feature validation. (#5953)
• [R] Enable weighted learning to rank (#5945)
• [R] Fix early stopping with custom objective (#5923)
• Fix NDK Build (#5886)
• Add missing explicit template specializations for greater portability (#5921)
• Handle empty rows in data iterators correctly (#5929). This bug affects file loader and JVM data frames.
• Fix IsDense (#5702)
• [jvm-packages] Fix wrong method name setAllowZeroForMissingValue (#5740)
• Fix shape inference for Dask predict (#5989)

Usability Improvements, Documentation

• [Doc] Document that CUDA 10.0 is required (#5872)
• Refactored command line interface (CLI). Now CLI is able to handle user errors and output basic document. (#5574)
• Better error handling in Python: use raise from syntax to preserve full stacktrace (#5787).
• The JSON model dump now has a formal schema (#5660, #5818). The benefit is to prevent dump_model() function from breaking. See this document to understand the difference between saving and dumping models.
• Add a reference to the GPU external memory paper (#5684)
• Document more objective parameters in the R package (#5682)
• Document the existence of pre-built binary wheels for MacOS (#5711)
• Remove max.depth in the R gblinear example. (#5753)
• Added conda environment file for building docs (#5773)
• Mention dask blog post in the doc, which introduces using Dask with GPU and some internal workings. (#5789)
• Fix rendering of Markdown docs (#5821)
• Document new objectives and metrics available on GPUs (#5909)
• Better message when no GPU is found. (#5594)
• Remove the use of silent parameter from R demos. (#5675)
• Don't use masked array in array interface. (#5730)
• Update affiliation of @terrytangyuan: Ant Financial -> Ant Group (#5827)
• Move dask tutorial closer other distributed tutorials (#5613)
• Update XGBoost + Dask overview documentation (#5961)
• Show n_estimators in the docstring of the scikit-learn interface (#6041)
• Fix a type in a doctring of the scikit-learn interface (#5980)

Maintenance: testing, continuous integration, build system

• [CI] Remove CUDA 9.0 from CI (#5674, #5745)
• Require CUDA 10.0+ in CMake build (#5718)
• [R] Remove dependency on gendef for Visual Studio builds (fixes #5608) (#5764). This enables building XGBoost with GPU support with R 4.x.
• [R-package] Reduce duplication in configure.ac (#5693)
• Bump com.esotericsoftware to 4.0.2 (#5690)
• Migrate some tests from AppVeyor to GitHub Actions to speed up the tests. (#5911, #5917, #5919, #5922, #5928)
• Reduce cost of the Jenkins CI server (#5884, #5904, #5892). We now enforce a daily budget via an automated monitor. We also dramatically reduced the workload for the Windows platform, since the cloud VM cost is vastly greater for Windows.
• [R] Set up automated R linter (#5944)
• [R] replace uses of T and F with TRUE and FALSE (#5778)
• Update Docker container 'CPU' (#5956)
• Simplify CMake build with modern CMake techniques (#5871)
• Use hypothesis package for testing (#5759, #5835, #5849).
• Define _CRT_SECURE_NO_WARNINGS to remove unneeded warnings in MSVC (#5434)
• Run all Python demos in CI, to ensure that they don't break (#5651)
• Enhance nvtx support (#5636). Now we can use unified timer between CPU and GPU. Also CMake is able to find nvtx automatically.
• Speed up python test. (#5752)
• Add helper for generating batches of data. (#5756)
• Add c-api-demo to .gitignore (#5855)
• Add option to enable all compiler warnings in GCC/Clang (#5897)
• Make Python model compatibility test runnable locally (#5941)
• Add cupy to Windows CI (#5797)
• [CI] Fix cuDF install; merge 'gpu' and 'cudf' test suite (#5814)
• Update rabit submodule (#5680, #5876)
• Force colored output for Ninja build. (#5959)
• [CI] Assign larger /dev/shm to NCCL (#5966)
• Add missing Pytest marks to AsyncIO unit test (#5968)
• [CI] Use latest cuDF and dask-cudf (#6048)
• Add CMake flag to log C API invocations, to aid debugging (#5925)
• Fix a unit test on CLI, to handle RC versions (#6050)
• [CI] Use mgpu machine to run gpu hist unit tests (#6050)
• [CI] Build GPU-enabled JAR artifact and deploy to xgboost-maven-repo (#6050)

Maintenance: Refactor code for legibility and maintainability

• Remove dead code in DMatrix initialization. (#5635)
• Catch dmlc error by ref. (#5678)
• Refactor the gpu_hist split evaluation in preparation for batched nodes enumeration. (#5610)
• Remove column major specialization. (#5755)
• Remove unused imports in Python (#5776)
• Avoid including c_api.h in header files. (#5782)
• Remove unweighted GK quantile, which is unused. (#5816)
• Add Python binding for rabit ops. (#5743)
• Implement Empty method for host device vector. (#5781)
• Remove print (#5867)
• Enforce tree order in JSON (#5974)

Acknowledgement

Contributors: Nan Zhu (@CodingCat), @LionOrCatThatIsTheQuestion, Dmitry Mottl (@Mottl), Rory Mitchell (@RAMitchell), @ShvetsKS, Alex Wozniakowski (@a-wozniakowski), Alexander Gugel (@alexanderGugel), @anttisaukko, @boxdot, Andy Adinets (@canonizer), Ram Rachum (@cool-RR), Elliot Hershberg (@elliothershberg), Jason E. Aten, Ph.D. (@glycerine), Philip Hyunsu Cho (@hcho3), @jameskrach, James Lamb (@jameslamb), James Bourbeau (@jrbourbeau), Peter Jung (@kongzii), Lorenz Walthert (@lorenzwalthert), Oleksandr Kuvshynov (@okuvshynov), Rong Ou (@rongou), Shaochen Shi (@shishaochen), Yuan Tang (@terrytangyuan), Jiaming Yuan (@trivialfis), Bobby Wang (@wbo4958), Zhang Zhang (@zhangzhang10)

Reviewers: Nan Zhu (@CodingCat), @LionOrCatThatIsTheQuestion, Hao Yang (@QuantHao), Rory Mitchell (@RAMitchell), @ShvetsKS, Egor Smirnov (@SmirnovEgorRu), Alex Wozniakowski (@a-wozniakowski), Amit Kumar (@aktech), Avinash Barnwal (@avinashbarnwal), @boxdot, Andy Adinets (@canonizer), Chandra Shekhar Reddy (@chandrureddy), Ram Rachum (@cool-RR), Cristiano Goncalves (@cristianogoncalves), Elliot Hershberg (@elliothershberg), Jason E. Aten, Ph.D. (@glycerine), Philip Hyunsu Cho (@hcho3), Tong He (@hetong007), James Lamb (@jameslamb), James Bourbeau (@jrbourbeau), Lee Drake (@leedrake5), DougM (@mengdong), Oleksandr Kuvshynov (@okuvshynov), RongOu (@rongou), Shaochen Shi (@shishaochen), Xu Xiao (@sperlingxx), Yuan Tang (@terrytangyuan), Theodore Vasiloudis (@thvasilo), Jiaming Yuan (@trivialfis), Bobby Wang (@wbo4958), Zhang Zhang (@zhangzhang10)

#5970

R package: xgboost_1.2.0.1.tar.gz (Manual: xgboost_1.2.0.1-manual.pdf)

#5970

R package: xgboost_1.2.0.1.tar.gz (Manual: xgboost_1.2.0.1-manual.pdf)

This patch release applies the following patches to 1.1.0 release:

• CPU performance improvement in the PyPI wheels (#5720)
• Fix loading old model. (#5724)
• Install pkg-config file (#5744)

Better performance on multi-core CPUs (#5244, #5334, #5522)

• Poor performance scaling of the hist algorithm for multi-core CPUs has been under investigation (#3810). #5244 concludes the ongoing effort to improve performance scaling on multi-CPUs, in particular Intel CPUs. Roadmap: #5104
• #5334 makes steps toward reducing memory consumption for the hist tree method on CPU.
• #5522 optimizes random number generation for data sampling.

Deterministic GPU algorithm for regression and classification (#5361)

• GPU algorithm for regression and classification tasks is now deterministic.
• Roadmap: #5023. Currently only single-GPU training is deterministic. Distributed training with multiple GPUs is not yet deterministic.

Improve external memory support on GPUs (#5093, #5365)

• Starting from 1.0.0 release, we added support for external memory on GPUs to enable training with larger datasets. Gradient-based sampling (#5093) speeds up the external memory algorithm by intelligently sampling a subset of the training data to copy into the GPU memory. Learn more about out-of-core GPU gradient boosting.
• GPU-side data sketching now works with data from external memory (#5365).

Parameter validation: detection of unused or incorrect parameters (#5477, #5569, #5508)

• Mis-spelled training parameter is a common user mistake. In previous versions of XGBoost, mis-spelled parameters were silently ignored. Starting with 1.0.0 release, XGBoost will produce a warning message if there is any unused training parameters. The 1.1.0 release makes parameter validation available to the scikit-learn interface (#5477) and the R binding (#5569).

Thread-safe, in-place prediction method (#5389, #5512)

• Previously, the prediction method was not thread-safe (#5339). This release adds a new API function inplace_predict() that is thread-safe. It is now possible to serve concurrent requests for prediction using a shared model object.
• It is now possible to compute prediction in-place for selected data formats (numpy.ndarray / scipy.sparse.csr_matrix / cupy.ndarray / cudf.DataFrame / pd.DataFrame) without creating a DMatrix object.

Addition of Accelerated Failure Time objective for survival analysis (#4763, #5473, #5486, #5552, #5553)

• Survival analysis (regression) models the time it takes for an event of interest to occur. The target label is potentially censored, i.e. the label is a range rather than a single number. We added a new objective survival:aft to support survival analysis. Also added is the new API to specify the ranged labels. Check out the tutorial and the demos.
• GPU support is work in progress (#5714).

Improved installation experience on Mac OSX (#5597, #5602, #5606, #5701)

• It only takes two commands to install the XGBoost Python package: brew install libomp followed by pip install xgboost. The installed XGBoost will use all CPU cores. Even better, starting with this release, we distribute pre-compiled binary wheels targeting Mac OSX. Now the install command pip install xgboost finishes instantly, as it no longer compiles the C++ source of XGBoost. The last three Mac versions (High Sierra, Mojave, Catalina) are supported.
• R package: the 1.1.0 release fixes the error Initializing libomp.dylib, but found libomp.dylib already initialized (#5701)

Ranking metrics are now accelerated on GPUs (#5380, #5387, #5398)

GPU-side data matrix to ingest data directly from other GPU libraries (#5420, #5465)

• Previously, data on GPU memory had to be copied back to the main memory before it could be used by XGBoost. Starting with 1.1.0 release, XGBoost provides a dedicated interface (DeviceQuantileDMatrix) so that it can ingest data from GPU memory directly. The result is that XGBoost interoperates better with GPU-accelerated data science libraries, such as cuDF, cuPy, and PyTorch.
• Set device in device dmatrix. (#5596)

Robust model serialization with JSON (#5123, #5217)

• We continue efforts from the 1.0.0 release to adopt JSON as the format to save and load models robustly. Refer to the release note for 1.0.0 to learn more.
• It is now possible to store internal configuration of the trained model (Booster) object in R as a JSON string (#5123, #5217).

Improved integration with Dask

• Pass through verbose parameter for dask fit (#5413)
• Use DMLC_TASK_ID. (#5415)
• Order the prediction result. (#5416)
• Honor nthreads from dask worker. (#5414)
• Enable grid searching with scikit-learn. (#5417)
• Check non-equal when setting threads. (#5421)
• Accept other inputs for prediction. (#5428)
• Fix missing value for scikit-learn interface. (#5435)

XGBoost4J-Spark: Check number of columns in the data iterator (#5202, #5303)

• Before, the native layer in XGBoost did not know the number of columns (features) ahead of time and had to guess the number of columns by counting the feature index when ingesting data. This method has a failure more in distributed setting: if the training data is highly sparse, some features may be completely missing in one or more worker partitions. Thus, one or more workers may deduce an incorrect data shape, leading to crashes or silently wrong models.
• Enforce correct data shape by passing the number of columns explicitly from the JVM layer into the native layer.

Major refactoring of the DMatrix class

• Continued from 1.0.0 release.
• Remove update prediction cache from predictors. (#5312)
• Predict on Ellpack. (#5327)
• Partial rewrite EllpackPage (#5352)
• Use ellpack for prediction only when sparsepage doesn't exist. (#5504)
• RFC: #4354, Roadmap: #5143

Breaking: XGBoost Python package now requires Pip 19.0 and higher (#5589)

• Your Linux machine may have an old version of Pip and may attempt to install a source package, leading to long installation time. This is because we are now using manylinux2010 tag in the binary wheel release. Ensure you have Pip 19.0 or newer by running python3 -m pip -V to check the version. Upgrade Pip with command

python3 -m pip install --upgrade pip

Upgrading to latest pip allows us to depend on newer versions of system libraries. TensorFlow also requires Pip 19.0+.

Breaking: GPU algorithm now requires CUDA 10.0 and higher (#5649)

• CUDA 10.0 is necessary to make the GPU algorithm deterministic (#5361).

Breaking: silent parameter is now removed (#5476)

• Please use verbosity instead.

Breaking: Set output_margin to True for custom objectives (#5564)

• Now both R and Python interface custom objectives get un-transformed (raw) prediction outputs.

Breaking: Makefile is now removed. We use CMake exclusively to build XGBoost (#5513)

• Exception: the R package uses Autotools, as the CRAN ecosystem did not yet adopt CMake widely.

Breaking: distcol updater is now removed (#5507)

• The distcol updater has been long broken, and currently we lack resources to implement a working implementation from scratch.

Deprecation notices

• Python 3.5. This release is the last release to support Python 3.5. The following release (1.2.0) will require Python 3.6.
• Scala 2.11. Currently XGBoost4J supports Scala 2.11. However, if a future release of XGBoost adopts Spark 3, it will not support Scala 2.11, as Spark 3 requires Scala 2.12+. We do not yet know which XGBoost release will adopt Spark 3.

Known limitations

• (Python package) When early stopping is activated with early_stopping_rounds at training time, the prediction method (xgb.predict()) behaves in a surprising way. If XGBoost runs for M rounds and chooses iteration N (N < M) as the best iteration, then the prediction method will use M trees by default. To use the best iteration (N trees), users will need to manually take the best iteration field bst.best_iteration and pass it as the ntree_limit argument to xgb.predict(). See #5209 and #4052 for additional context.
• GPU ranking objective is currently not deterministic (#5561).
• When training parameter reg_lambda is set to zero, some leaf nodes may be assigned a NaN value. (See discussion.) For now, please set reg_lambda to a nonzero value.

Community and Governance

• The XGBoost Project Management Committee (PMC) is pleased to announce a new committer: Egor Smirnov (@SmirnovEgorRu). He has led a major initiative to improve the performance of XGBoost on multi-core CPUs.

Bug-fixes

• Improved compatibility with scikit-learn (#5255, #5505, #5538)
• Remove f-string, since it's not supported by Python 3.5 (#5330). Note that Python 3.5 support is deprecated and schedule to be dropped in the upcoming release (1.2.0).
• Fix the pruner so that it doesn't prune the same branch twice (#5335)
• Enforce only major version in JSON model schema (#5336). Any major revision of the model schema would bump up the major version.
• Fix a small typo in sklearn.py that broke multiple eval metrics (#5341)
• Restore loading model from a memory buffer (#5360)
• Define lazy isinstance for Python compat (#5364)
• [R] fixed uses of class() (#5426)
• Force compressed buffer to be 4 bytes aligned, to keep cuda-memcheck happy (#5441)
• Remove warning for calling host function (std::max) on a GPU device (#5453)
• Fix uninitialized value bug in xgboost callback (#5463)
• Fix model dump in CLI (#5485)
• Fix out-of-bound array access in WQSummary::SetPrune() (#5493)
• Ensure that configured dmlc/build_config.h is picked up by Rabit and XGBoost, to fix build on Alpine (#5514)
• Fix a misspelled method, made in a git merge (#5509)
• Fix a bug in binary model serialization (#5532)
• Fix CLI model IO (#5535)
• Don't use uint for threads (#5542)
• Fix R interaction constraints to handle more than 100000 features (#5543)
• [jvm-packages] XGBoost Spark should deal with NaN when parsing evaluation output (#5546)
• GPU-side data sketching is now aware of query groups in learning-to-rank data (#5551)
• Fix DMatrix slicing for newly added fields (#5552)
• Fix configuration status with loading binary model (#5562)
• Fix build when OpenMP is disabled (#5566)
• R compatibility patches (#5577, #5600)
• gpu_hist performance fixes (#5558)
• Don't set seed on CLI interface (#5563)
• [R] When serializing model, preserve model attributes related to early stopping (#5573)
• Avoid rabit calls in learner configuration (#5581)
• Hide C++ symbols in libxgboost.so when building Python wheel (#5590). This fixes apache/incubator-tvm#4953.
• Fix compilation on Mac OSX High Sierra (10.13) (#5597)
• Fix build on big endian CPUs (#5617)
• Resolve crash due to use of vector<bool>::iterator (#5642)
• Validation JSON model dump using JSON schema (#5660)

Performance improvements

• Wide dataset quantile performance improvement (#5306)
• Reduce memory usage of GPU-side data sketching (#5407)
• Reduce span check overhead (#5464)
• Serialise booster after training to free up GPU memory (#5484)
• Use the maximum amount of GPU shared memory available to speed up the histogram kernel (#5491)
• Use non-synchronising scan in Thrust (#5560)
• Use cudaDeviceGetAttribute() instead of cudaGetDeviceProperties() for speed (#5570)

API changes

• Support importing data from a Pandas SparseArray (#5431)
• HostDeviceVector (vector shared between CPU and GPU memory) now exposes HostSpan interface, to enable access on the CPU side with bound check (#5459)
• Accept other gradient types for SplitEntry (#5467)

Usability Improvements, Documentation

• Add JVM_CHECK_CALL to prevent C++ exceptions from leaking into the JVM layer (#5199)
• Updated Windows build docs (#5283)
• Update affiliation of @hcho3 (#5292)
• Display Sponsor button, link to OpenCollective (#5325)
• Update docs for GPU external memory (#5332)
• Add link to GPU documentation (#5437)
• Small updates to GPU documentation (#5483)
• Edits on tutorial for XGBoost job on Kubernetes (#5487)
• Add reference to GPU external memory (#5490)
• Fix typos (#5346, #5371, #5384, #5399, #5482, #5515)
• Update Python doc (#5517)
• Add Neptune and Optuna to list of examples (#5528)
• Raise error if the number of data weights doesn't match the number of data sets (#5540)
• Add a note about GPU ranking (#5572)
• Clarify meaning of training parameter in the C API function XGBoosterPredict() (#5604)
• Better error handling for situations where existing trees cannot be modified (#5406, #5418). This feature is enabled when process_type is set to update.

Maintenance: testing, continuous integration, build system

• Add C++ test coverage for data sketching (#5251)
• Ignore gdb_history (#5257)
• Rewrite setup.py. (#5271, #5280)
• Use scikit-learn in extra dependencies (#5310)
• Add CMake option to build static library (#5397)
• [R] changed FindLibR to take advantage of CMake cache (#5427)
• [R] fixed inconsistency in R -e calls in FindLibR.cmake (#5438)
• Refactor tests with data generator (#5439)
• Resolve failing Travis CI (#5445)
• Update dmlc-core. (#5466)
• [CI] Use clang-tidy 10 (#5469)
• De-duplicate code for checking maximum number of nodes (#5497)
• [CI] Use Ubuntu 18.04 LTS in JVM CI, because 19.04 is EOL (#5537)
• [jvm-packages] [CI] Create a Maven repository to host SNAPSHOT JARs (#5533)
• [jvm-packages] [CI] Publish XGBoost4J JARs with Scala 2.11 and 2.12 (#5539)
• [CI] Use Vault repository to re-gain access to devtoolset-4 (#5589)

Maintenance: Refactor code for legibility and maintainability

• Move prediction cache to Learner (#5220, #5302)
• Remove SimpleCSRSource (#5315)
• Refactor SparsePageSource, delete cache files after use (#5321)
• Remove unnecessary DMatrix methods (#5324)
• Split up LearnerImpl (#5350)
• Move segment sorter to common (#5378)
• Move thread local entry into Learner (#5396)
• Split up test helpers header (#5455)
• Requires setting leaf stat when expanding tree (#5501)
• Purge device_helpers.cuh (#5534)
• Use thrust functions instead of custom functions (#5544)

Acknowledgement

Contributors: Nan Zhu (@CodingCat), Rory Mitchell (@RAMitchell), @ShvetsKS, Egor Smirnov (@SmirnovEgorRu), Andrew Kane (@ankane), Avinash Barnwal (@avinashbarnwal), Bart Broere (@bartbroere), Andy Adinets (@canonizer), Chen Qin (@chenqin), Daiki Katsuragawa (@daikikatsuragawa), David Díaz Vico (@daviddiazvico), Darius Kharazi (@dkharazi), Darby Payne (@dpayne), Jason E. Aten, Ph.D. (@glycerine), Philip Hyunsu Cho (@hcho3), James Lamb (@jameslamb), Jan Borchmann (@jborchma), Kamil A. Kaczmarek (@kamil-kaczmarek), Melissa Kohl (@mjkohl32), Nicolas Scozzaro (@nscozzaro), Paul Kaefer (@paulkaefer), Rong Ou (@rongou), Samrat Pandiri (@samratp), Sriram Chandramouli (@sriramch), Yuan Tang (@terrytangyuan), Jiaming Yuan (@trivialfis), Liang-Chi Hsieh (@viirya), Bobby Wang (@wbo4958), Zhang Zhang (@zhangzhang10)

Reviewers: Nan Zhu (@CodingCat), @LeZhengThu, Rory Mitchell (@RAMitchell), @ShvetsKS, Egor Smirnov (@SmirnovEgorRu), Steve Bronder (@SteveBronder), Nikita Titov (@StrikerRUS), Andrew Kane (@ankane), Avinash Barnwal (@avinashbarnwal), @brydag, Andy Adinets (@canonizer), Chandra Shekhar Reddy (@chandrureddy), Chen Qin (@chenqin), Codecov (@codecov-io), David Díaz Vico (@daviddiazvico), Darby Payne (@dpayne), Jason E. Aten, Ph.D. (@glycerine), Philip Hyunsu Cho (@hcho3), James Lamb (@jameslamb), @johnny-cat, Mu Li (@mli), Mate Soos (@msoos), @rnyak, Rong Ou (@rongou), Sriram Chandramouli (@sriramch), Toby Dylan Hocking (@tdhock), Yuan Tang (@terrytangyuan), Oleksandr Pryimak (@trams), Jiaming Yuan (@trivialfis), Liang-Chi Hsieh (@viirya), Bobby Wang (@wbo4958)