tokenizers
0.23.1
223M downloads/mo
#186 most downloaded on PyPI
huggingface/tokenizers
What this package is like to depend on
Last release 3 months ago
27 Apr 2026
Ships fairly regularly
a new release about every 5 weeks
No release notes found
nothing matched a version
2 versions withdrawn
withdrawn after publishing
7 years old
100 releases · first in 2019
9 releases in the last 12 months
see the full history below
Release timeline
100 releases · Nov 2019 to Apr 2026Releases
latest 60 of 100-
0.23.127 Apr 2026Nothing published for this version
-
0.23.1rc027 Apr 2026 pre-releaseNothing published for this version
-
0.23.0rc024 Apr 2026 pre-releaseNothing published for this version
-
0.22.205 Jan 2026Nothing published for this version
-
0.22.2rc002 Dec 2025 pre-releaseNothing published for this version
-
0.22.119 Sep 2025Nothing published for this version
-
0.22.1rc019 Sep 2025 pre-releaseNothing published for this version
-
0.22.029 Aug 2025Nothing published for this version
-
0.22.0rc029 Aug 2025 pre-releaseNothing published for this version
-
0.21.428 Jul 2025Nothing published for this version
-
0.21.224 Jun 2025Nothing published for this version
-
0.21.2rc024 Jun 2025 pre-releaseNothing published for this version
-
0.21.113 Mar 2025Nothing published for this version
-
0.21.1rc012 Mar 2025 pre-releaseNothing published for this version
-
0.21.027 Nov 2024Nothing published for this version
-
0.21.0rc027 Nov 2024 pre-releaseNothing published for this version
-
0.20.426 Nov 2024 withdrawnRemoving support for python3.7 and 3.8 is breaking and is move to v0.21.0
Nothing published for this version
-
0.20.4rc026 Nov 2024 pre-releaseNothing published for this version
-
0.20.305 Nov 2024Nothing published for this version
-
0.20.204 Nov 2024Nothing published for this version
-
0.20.110 Oct 2024Nothing published for this version
-
0.20.008 Aug 2024Nothing published for this version
-
0.19.117 Apr 2024Nothing published for this version
-
0.19.017 Apr 2024Nothing published for this version
-
0.15.212 Feb 2024Nothing published for this version
-
0.15.122 Jan 2024Nothing published for this version
-
0.15.014 Nov 2023Nothing published for this version
-
0.14.106 Oct 2023Nothing published for this version
-
0.14.007 Sep 2023Nothing published for this version
-
0.13.305 Apr 2023Nothing published for this version
-
0.13.207 Nov 2022Nothing published for this version
-
0.13.106 Oct 2022Nothing published for this version
-
0.13.021 Sep 2022Nothing published for this version
-
0.12.113 Apr 2022Nothing published for this version
-
0.12.031 Mar 2022 withdrawnBreaking change with unexpected effect
Nothing published for this version
-
0.11.628 Feb 2022Nothing published for this version
-
0.11.516 Feb 2022Nothing published for this version
-
0.11.417 Jan 2022Nothing published for this version
-
0.11.317 Jan 2022Nothing published for this version
-
0.11.204 Jan 2022Nothing published for this version
-
0.11.128 Dec 2021Nothing published for this version
-
0.11.023 Dec 2021Nothing published for this version
-
0.10.324 May 2021Nothing published for this version
-
0.10.205 Apr 2021Nothing published for this version
-
0.10.108 Feb 2021Nothing published for this version
-
0.10.012 Jan 2021Nothing published for this version
-
0.10.0rc108 Dec 2020 pre-releaseNothing published for this version
-
0.9.410 Nov 2020Nothing published for this version
-
0.9.326 Oct 2020Nothing published for this version
-
0.9.215 Oct 2020Nothing published for this version
-
0.9.113 Oct 2020Nothing published for this version
-
0.9.009 Oct 2020Nothing published for this version
-
0.9.0rc206 Oct 2020 pre-releaseNothing published for this version
-
0.9.0rc129 Sep 2020 pre-releaseNothing published for this version
-
0.9.0.dev424 Sep 2020 pre-releaseNothing published for this version
-
0.9.0.dev318 Sep 2020 pre-releaseNothing published for this version
-
0.9.0.dev214 Sep 2020 pre-releaseNothing published for this version
-
0.9.0.dev114 Sep 2020 pre-releaseNothing published for this version
-
0.9.0.dev021 Aug 2020 pre-releaseNothing published for this version
-
0.8.120 Jul 2020Nothing published for this version