NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #3645 most downloaded on PyPI
🦛 CHONK your texts with Chonkie ✨ - The no-nonsense chunking library
Last release 2 months ago
07 Jul 2026
Ships fairly regularly
a new release about every 2 weeks
Most releases are documented
notes for 36 of 52 stable releases
Nothing withdrawn
no release was ever pulled
2 years old
62 releases · first in 2024
One column per month.
Chonkie now integrates with LiteParse, making it easier to parse and chunk a wider range of document formats with a lightweight, dependency-friendly p
<p align="center"> <img width="25%" alt="chef x llamaindex" src="https://github.com/user-attachments/assets/e777312b-1123-4e6c-aecb-ea42afb80f68" /> </p>
Chonkie now integrates with LiteParse, making it easier to parse and chunk a wider range of document formats with a lightweight, dependency-friendly parser.
<p align="center"> <img width="50%" alt="image" src="https://github.com/user-attachments/assets/cde3f946-2ec7-49aa-a89d-d3df15169141" /> </p>
ComponentRegistryComponentRegistry is now publicly exposed, making it easier to inspect, extend, and build custom integrations on top of Chonkie's component system.
<p align="center"> <img width="50%" alt="image" src="https://github.com/user-attachments/assets/b33ddf7e-8273-422f-9dcf-65ca9b6d88b8" /> </p>
The documentation has been completely migrated and redesigned with improved navigation, clearer guides, and a better overall developer experience. It is now hosted on GitHub Pages.
<p align="center"> <img width="80%" alt="image" src="https://github.com/user-attachments/assets/b0c64b37-7f7c-4bce-abad-d012473f86bd" /> </p>
Full Changelog: https://github.com/feyninc/chonkie/compare/v1.6.8...v1.7.0
Support Gemini embedding dimensions in Catsu wrappers by @hannibal-lee in https://github.com/chonkie-inc/chonkie/pull/571
<p align="center"> <img width="50%" alt="lancedb handshake" src="https://github.com/user-attachments/assets/7c6ef4c1-2c4c-4cfa-8def-332767159343" /> </p>
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.7...v1.6.8
Support Gemini embedding dimensions in Catsu wrappers by @hannibal-lee in https://github.com/chonkie-inc/chonkie/pull/571
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.7...v1.6.8-alpha.1
Extract text from images with the new MistralOCR integration: `python from chonkie import MistralOCR ocr = MistralOCR() out = ocr("image.jpg") print(o
<img width="1360" height="768" alt="image" src="https://github.com/user-attachments/assets/638b6049-51c1-4b96-ae91-7b1061e6b93f" />
Extract text from images with the new MistralOCR integration:
from chonkie import MistralOCR
ocr = MistralOCR()
out = ocr("image.jpg")
print(out.content)
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.6...v1.6.7
🔥 CodeChunker update and huge performance speedup. bechmarks below using semble as a reference:
🔥 CodeChunker update and huge performance speedup. bechmarks below using semble as a reference:
| Metric | Baseline | Semble (uring the current version) | Delta | % Change |
|---|---|---|---|---|
| NDCG@10 (search quality) | 0.854 | 0.856 | +0.002 | +0.2% |
| p50 latency | 7.19 ms | 6.16 ms | -1.03 ms | -14.3% |
| p90 latency | 22.15 ms | 18.57 ms | -3.57 ms | -16.1% |
| p95 latency | 25.12 ms | 20.60 ms | -4.51 ms | -18.0% |
| p99 latency | 28.69 ms | 23.70 ms | -4.99 ms | -17.4% |
| Index time | 5,138 ms | 2,853 ms | -2,285 ms | -44.5% |
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.5...v1.6.6
Chonkie now supports agentic skills via  you can get started directly by running the following comma
Chonkie now supports agentic skills via
you can get started directly by running the following command 🤖
npx skills add chonkie-inc/skills
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.4...v1.6.5
Pandas Import Fix : Fixed import chonkie failing with ModuleNotFoundError: No module named 'pandas' when installed without the [table] extra. The pand
import chonkie failing with ModuleNotFoundError: No module named 'pandas' when installed without the [table] extra. The pandas import in utils/table_converter.py is now lazy-loaded, matching the existing pattern for optional dependencies. By @Pringled in https://github.com/chonkie-inc/chonkie/pull/566[!TIP] If you encountered
ModuleNotFoundError: No module named 'pandas'on v1.6.3, upgrading to v1.6.4 resolves the issue:pip install --upgrade chonkie
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.3...v1.6.4
> Known Bug: import chonkie fails with ModuleNotFoundError: No module named 'pandas' when installed without the [table] extra. This is caused by an un
<p align="center"> <img width="50%" alt="lancedb handshake" src="https://github.com/user-attachments/assets/3deca5d8-af13-4c01-b2c0-dc10afa300d2" />
</p>
[!CAUTION] Known Bug:
import chonkiefails withModuleNotFoundError: No module named 'pandas'when installed without the[table]extra. This is caused by an unconditional top-level pandas import inutils/table_converter.py. Please upgrade to v1.6.4 which fixes this issue.
filename to metadata for better traceability by @chonk-lain in https://github.com/chonkie-inc/chonkie/pull/554MarkdownDocument support for CodeChunker and fixed no-op behavior in TableChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/563_generate_default_id into BaseHandshake by @chimchim89 in https://github.com/chonkie-inc/chonkie/pull/455RecursiveChunker._split_text by @nightcityblade in https://github.com/chonkie-inc/chonkie/pull/537CONTRIBUTING.md by @swamy18 in https://github.com/chonkie-inc/chonkie/pull/465Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.2...v1.6.3
A new chunker has been added to your toolkit 🎉 you can now use the newly added TeraflopAI chunker freely using the code below ```py from chonkie impor
<p align="center"> <img width="50%" alt="image" src="https://github.com/user-attachments/assets/b70fab9e-dab4-497f-85fc-a0d5d7e5d309" /> </p>
A new chunker has been added to your toolkit 🎉
you can now use the newly added TeraflopAI chunker freely using the code below
from chonkie import TeraflopAIChunker
chunker = TeraflopAIChunker(api_key="your_api_key_here")
text = "Your text here"
chunker.chunk(text)
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.1...v1.6.2
This patch release focuses on fixing import issues and updating dependencies
This patch release focuses on fixing import issues and updating dependencies
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.6.0...v1.6.1
This release brings HTML table support, a self-hostable chunking API, native async capabilities, and a migration to ty for faster type-checking, along
<p align="center"> <img src="https://github.com/chonkie-inc/chonkie/blob/main/docs/assets/logo/chonkie_scientist.png?raw=true" height="200" alt="Chonkie logo" /> </p>
This release brings HTML table support, a self-hostable chunking API, native async capabilities, and a migration to ty for faster type-checking, alongside a range of internal refactors and quality improvements that make the library leaner and easier to extend.
Structured data in HTML documents has historically been a challenge for chunking pipelines. With 1.6.0, we're introducing first-class support for HTML tables via two new components TableChef and TableChunker.
TableChef handles the extraction and normalization of HTML tables into clean, structured representations that are ready for downstream processing.TableChunker builds on top of that to intelligently chunk tabular content, preserving row and column semantics rather than blindly splitting on token boundaries.Whether you're working with HTML or markdown, you can now process tables in your chunking pipeline.
<table> <thead> <tr> <th>Input Code</th> <th>Original HTML Table</th> <th>Chunked Output</th> </tr> </thead>
<tbody> <tr>
<td> <p align="center"> <img src="https://github.com/user-attachments/assets/251de10a-f97f-46cb-bddd-5f645c76e98e" height="400" alt="Chonkie table"/> </p>
</td>
<td>
<table> <thead> <tr><th>ID</th><th>Status</th></tr> </thead> <tbody> <tr><td>1</td><td>Active</td></tr> <tr><td>2</td><td>Pending</td></tr> <tr><td>3</td><td>Inactive</td></tr> <tr><td>4</td><td>Active</td></tr> </tbody> </table>
</td>
<td>
<table> <thead> <tr><th>ID</th><th>Status</th></tr> </thead> <tbody> <tr><td>1</td><td>Active</td></tr> <tr><td>2</td><td>Pending</td></tr> </tbody> </table>
<br>
<table> <thead> <tr><th>ID</th><th>Status</th></tr> </thead> <tbody> <tr><td>3</td><td>Inactive</td></tr> <tr><td>4</td><td>Active</td></tr> </tbody> </table>
</td>
</tr> </tbody> </table>
Chonkie can now run as a fully self-hosted REST API, making it easy to expose chunking capabilities as a service within your infrastructure. Built on FastAPI, the Chonkie OSS API can be spun up with a single CLI command and accessed from any machine on your network.
This is ideal for teams that want to centralize their chunking logic, integrate Chonkie into polyglot stacks, or simply avoid re-initializing models in every service that needs chunking.
To start the server, run:
chonkie serve
Once running, any client on your network can hit the endpoint on the consumer side, full documentation can be found in http://localhost:8000/docs
<p align="center">
<img alt="image" src="https://github.com/user-attachments/assets/d57a83e7-1aa3-4b9e-bcf2-db13522f5e92" width="50%" height="auto" />
<img alt="api" src="https://github.com/user-attachments/assets/12c7f130-4ac5-431e-a2be-5390203cd3f0" height="auto" width="50%" />
<p/>
Chonkie now supports asynchronous chunking out of the box. This has been a long-requested feature for users building async-native applications, whether that's async data pipelines, or anything else running on an event loop. You can now await chunking operations without blocking, making Chonkie a natural fit for high-throughput, I/O-bound workloads.
| Method | Async Equivalent | Description |
|---|---|---|
chunk(text) |
achunk(text) |
Chunk a single text |
chunk_batch(texts) |
achunk_batch(texts) |
Chunk a list of texts |
chunk_document(doc) |
achunk_document(doc) |
Chunk a Document object |
logger.warning for warnings by @akx in https://github.com/chonkie-inc/chonkie/pull/425Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.5.6...v1.6.0
The visulizer now has support for a list of strings as well, allowing it to extend beyond the scope of the current library `python from chonkie import
The visulizer now has support for a list of strings as well, allowing it to extend beyond the scope of the current library
from chonkie import Visualizer, RecursiveChunker
viz = Visualizer()
texts = ["hello, ","world"]
viz(texts)
<img width="100" height="40" alt="image" src="https://github.com/user-attachments/assets/8bdbe433-c97f-4874-b997-a3d70167f8d7" />
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.5.5...v1.5.6
A patch release focused on stability improvements and bug fixes. This release makes Chonkie more robust for users with partial dependencies and custom
A patch release focused on stability improvements and bug fixes. This release makes Chonkie more robust for users with partial dependencies and custom implementations.
OpenAI Lazy Import - Users without openai installed can now import Chonkie without errors. Previously, importing chonkie would fail with ModuleNotFoundError even when not using OpenAI features. (#481)
Multiprocessing Safe by Default - Custom chunkers now work out-of-the-box without pickle errors. The BaseChunker now defaults to sequential processing, with parallel processing available as an opt-in. (#471)
CodeChunker Leaf Node Handling - Fixed an edge case where leaf nodes in the syntax tree could produce empty output, causing valid code to be dropped during chunking. (#473)
SlumberChunker Safe Split - Improved split index handling in SlumberChunker for better reliability. (#479)
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.5.4...v1.5.5
checkout the docs for more details about CLI commands and usage
checkout the docs for more details about CLI commands and usage
<img width="1467" height="353" alt="image" src="https://github.com/user-attachments/assets/a585f7ee-e30c-4c22-8868-9a356bec7172" />
we have officially migrated from Cython towards rust 🦀
raise ... from ... when reraising by @akx in https://github.com/chonkie-inc/chonkie/pull/424Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.5.2...v1.5.4
Feat: Add FastChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/430
<p align="center"> <img width="500" height="500" alt="FastChonkie!" src="https://github.com/user-attachments/assets/94cc864b-4013-4f9d-9cb0-d71a3c091a00" /> </p>
FastChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/430__all__ to make pyright happy by @Kludex in https://github.com/chonkie-inc/chonkie/pull/429Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.5.1...v1.5.2
CI: also build wheels for Python 3.14 by @akx in https://github.com/chonkie-inc/chonkie/pull/406
._is_available() always a classmethod by @akx in https://github.com/chonkie-inc/chonkie/pull/415Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.5.0...v1.5.1
Fix: Deprecate Py3.9 and add support for Py3.13! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/320
Py3.9, and adds support for Py3.13.RowTokenizer to chunk tables based on row values, thanks to @chonk-lain.<p align="center"> <img src="https://github.com/user-attachments/assets/65fbfe3a-31d6-4459-b4bb-fe1908009e36" width="50%"> </p>
run() by @chonknick in https://github.com/chonkie-inc/chonkie/pull/391ByteTokenizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/397Py3.9 and add support for Py3.13! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/320Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.4.2...v1.5.0
feat: fixed azure openai AutoEmbeddings by @MODSetter in https://github.com/chonkie-inc/chonkie/pull/368
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.4.1...v1.4.2
Add support for MilvusHandshake! Now you can send your chunks Milvus through Chonkie~
<img width="960" height="540" alt="Chonkie Logo (54)" src="https://github.com/user-attachments/assets/251cc897-d8e0-4991-bd89-37d97d0624bd" />
MilvusHandshake! Now you can send your chunks Milvus through Chonkie~Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.4.0...v1.4.1
Fix: remove legacy chunkers for the next breaking change in chonkie by @chonknick in https://github.com/chonkie-inc/chonkie/pull/286
<img width="960" height="540" alt="Chonkie Logo (49)" src="https://github.com/user-attachments/assets/fab18c49-0139-463d-bc66-1a326561ec75" />
chonkie.Pipeline for composable chunking + retrieval workflowsTableChef and TableChunker for smart table-aware chunking, plus MarkdownChef.parse() for quick Markdown ingestionChromaHandshake, ElasticHandshake, and TurboPufferHandshake for one-line vector-store checksocean_breeze and midnight—for prettier chunk inspectionnumpy is now a core dependency—remove any pinned excludes in your envtokenizer_or_token_counter kwarg is gone; use tokenizer everywhere (single, consistent name)parse method to the MarkdownChef + Enhance testing for chefs by @chonknick in https://github.com/chonkie-inc/chonkie/pull/310Document and more... by @chonknick in https://github.com/chonkie-inc/chonkie/pull/308tokenizer_or_token_counter to tokenizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/311legacy chunkers for the next breaking change in chonkie by @chonknick in https://github.com/chonkie-inc/chonkie/pull/286TableChef to always return MarkdownTable + remove stale sdpm docs by @chonk-lain in https://github.com/chonkie-inc/chonkie/pull/332numpy as a default install in chonkie by @chonknick in https://github.com/chonkie-inc/chonkie/pull/333chonkie.Pipeline by @chonknick in https://github.com/chonkie-inc/chonkie/pull/319Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.3.1...v1.4.0
Feat: add pinecone source_tag by @chonk-lain in https://github.com/chonkie-inc/chonkie/pull/294
MarkdownDocument and MarkdownChef, rename text->content in Document by @chonknick in https://github.com/chonkie-inc/chonkie/pull/299MarkdownImage type + Handle hybrid chunking via Document by @chonknick in https://github.com/chonkie-inc/chonkie/pull/301MarkdownImages and filter empty text blocks in MarkdownChef by @chonknick in https://github.com/chonkie-inc/chonkie/pull/302TableChunker failing edge case and indexing by @chonknick in https://github.com/chonkie-inc/chonkie/pull/304chunk_document to the TableChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/305Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.3.0...v1.3.1
Added search methods to handshakes for quick retrieval checks! Now you can search things directly with QdrantHandshake, WeaviateHandshake, PineconeHan
<img width="960" height="540" alt="Chonkie Logo (45)" src="https://github.com/user-attachments/assets/e17a2d5d-0526-4ae1-9bd2-78f814ae798b" />
search methods to handshakes for quick retrieval checks! Now you can search things directly with QdrantHandshake, WeaviateHandshake, PineconeHandshake, MongoDBHandshake and PgvectorHandshake.file input for all the cloud chunkers for easier processing of the files directly!Chunk type, instead of SemanticChunk, SentenceChunk or RecursiveChunk, leader to stricter typing and more uniform structure.SemanticChunker now matches the chonkie's SemanticChunker!chonkie.cloud.SemanticChunker has new parameters & add file to all the chunkers by @chonknick in https://github.com/chonkie-inc/chonkie/pull/279chonkie favicon image by @chonknick in https://github.com/chonkie-inc/chonkie/pull/282friends directory from the codebase by @chonknick in https://github.com/chonkie-inc/chonkie/pull/287Chunk type only by @chonknick in https://github.com/chonkie-inc/chonkie/pull/285Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.2.1...v1.3.0
Removes --ffast-math from setup.py, since it's not supported on all platforms/linux distros!
--ffast-math from setup.py, since it's not supported on all platforms/linux distros!Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.2.0...v1.2.1
New SemanticChunker with advance peak detection, skip_window support, group embedding support and better thresholding! The new semantic chunker requir
<img width="712" height="499" alt="image" src="https://github.com/user-attachments/assets/0eb4c719-0048-4d41-9651-44c5fa7fb1be" />
SemanticChunker with advance peak detection, skip_window support, group embedding support and better thresholding! The new semantic chunker requires much less tuning for consistently sized chunks.# Import the chunker
from chonkie import SemanticChunker
# Initialize the chunker
chunker = SemanticChunker(chunk_size=512)
# define the text
text = ...
# CHONK!
chunk = chunker(text)
MongoDBHandshake!from chonkie import MongoDBHandshake, SemanticChunker
# Initialize the handshake
handshake = MongoDBHandshake(uri="mongodb://localhost:8080", db_name="my_documents")
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Write chunks to MongoDB
handshake.write(chunks)
PineconeHandshake!from chonkie import PineconeHandshake, SemanticChunker
# Initialize the handshake
handshake = PineconeHandshake(api_key=...)
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Write chunks to Pinecone
handshake.write(chunks)
WeaviateHandshake!from chonkie import WeaviateHandshake, SemanticChunker
# Initialize the handshake
handshake = WeaviateHandshake(url=url, api_key=...)
# Create some chunks
chunker = SemanticChunker()
chunks = chunker.chunk("Chonkie loves to chonk your texts!")
# Write chunks to Weaviate
handshake.write(chunks)
_truncate to the OpenAIEmbddings (#239) by @chonknick in https://github.com/chonkie-inc/chonkie/pull/247PineconeHandshake by @chonk-lain in https://github.com/chonkie-inc/chonkie/pull/243MongoDBHandshake by @chonk-lain in https://github.com/chonkie-inc/chonkie/pull/253SemanticChunker with peak finding, window similarities and more! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/251chonkie-inc/cookbook by @chonknick in https://github.com/chonkie-inc/chonkie/pull/258Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.1.2...v1.2.0
Added initial support for FileFetcher and TextChef classes — Classes to support fetching files and cleaning up basic text structures.
FileFetcher and TextChef classes — Classes to support fetching files and cleaning up basic text structures.jina-embedding-4 embedding model (thanks to @real-jiakai 🙏)AzureOpenAIEmbeddings and AzureOpenAIGenie (thanks to @TaylorN15 🙏)FileFetcher and TextChef classes — for better input support in chonkie by @chonknick in https://github.com/chonkie-inc/chonkie/pull/220jina-embeddings-v4 + Support for AzureOpenAIEmbeddings by @chonknick in https://github.com/chonkie-inc/chonkie/pull/226chonkie.cloud.auth and chonkie.cloud.refineries by @shreyash-chonkie in https://github.com/chonkie-inc/chonkie/pull/230v1.1.2 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/232Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.1.1...v1.1.2
Chonkie imports properly on windows without Microsoft C++ Build Tools — solving #212!
DatasetsPorter to save chunks as a dataset.v1.1.1 and add DatasetsPorter by @chonknick in https://github.com/chonkie-inc/chonkie/pull/217Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.1.0...v1.1.1
Feat: Bump version and add pre-built wheels for windows to avoid #212 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/216
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.1.0...v1.1.1a0
Update docs with Cloud SDKs by @shreyash-chonkie in https://github.com/chonkie-inc/chonkie/pull/189
character type to make the tokenizers optional by @chonknick in https://github.com/chonkie-inc/chonkie/pull/205chunks to simplify mypy static typing by @chonknick in https://github.com/chonkie-inc/chonkie/pull/206return_type='texts' to increase type consistency and resolve complex mypy issues by @chonknick in https://github.com/chonkie-inc/chonkie/pull/207__init__.py by @chonknick in https://github.com/chonkie-inc/chonkie/pull/209Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.10...v1.1.0
Fix: Add mypy stubs, py.typed and fix any remaining typing issues from mypy by @chonknick in https://github.com/chonkie-inc/chonkie/pull/186
py.typed and fix any remaining typing issues from mypy by @chonknick in https://github.com/chonkie-inc/chonkie/pull/186v1.0.10 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/187Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.9...v1.0.10
Fix pickling issue in BaseTokenizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/153
OverlapRefinery, Tokenizers and SlumberChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/155SPDMChunker, SemanticChunker, NeuralChunker and Visualizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/156CodeChunker + tests by @chonknick in https://github.com/chonkie-inc/chonkie/pull/157CodeChunker fails multiprocessing due to pickling issues; Run it sequentially by @chonknick in https://github.com/chonkie-inc/chonkie/pull/158mode="recursive" fails with level gets too low by @chonknick in https://github.com/chonkie-inc/chonkie/pull/161OverlapRefinery is caching too much; revert context_size calculation caching by @chonknick in https://github.com/chonkie-inc/chonkie/pull/162split and merge basic ops for chunking! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/163split and merge ops for performance boosts! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/164chonkie-docs repo to chonkie for easier management + maintainance by @chonknick in https://github.com/chonkie-inc/chonkie/pull/149JSONPorter by @chonknick in https://github.com/chonkie-inc/chonkie/pull/167PsycopgHandshake by @chonknick in https://github.com/chonkie-inc/chonkie/pull/171PsycopgHandshake for initial support of pgvector by @chonknick in https://github.com/chonkie-inc/chonkie/pull/172PyscopgHandshake to PgvectorHandshake via vecs by @chonknick in https://github.com/chonkie-inc/chonkie/pull/174SemanticChunker bug for all comparison by @chonknick in https://github.com/chonkie-inc/chonkie/pull/176experimental.CodeChunker — improved code chunking for few languages by @chonknick in https://github.com/chonkie-inc/chonkie/pull/178cython module build failing during CD — Add cython build info to pyproject.toml by @chonknick in https://github.com/chonkie-inc/chonkie/pull/180cibuildwheel to build and publish wheels to PyPI by @chonknick in https://github.com/chonkie-inc/chonkie/pull/182uv instead in the hopes of finally publishing the v1.0.9 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/183Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.8...v1.0.9
Use base_url with OpenAIEmbeddings to use OpenAI API compatible embedding services!
base_url with OpenAIEmbeddings to use OpenAI API compatible embedding services!AutoEmbeddings a URI string with the alias to choose different providers for a model simply and easily. Just do AutoEmbeddings.get_embedding("model2vec://minishlab/potion-base-8M") or equivalently for the sentence_transformers version do AutoEmbeddings.get_embedding("st://minishlab/potion-base-8M"). This would work with all the supported embeddings in Chonkie. As Chonkie grows, it would support various providers and these notations help you choose between them quite easily.chonkie.cloud chunkers with updated support for NeuralChunker and SlumberChunker.SlumberChunker with OpenRouter models by @chonknick in https://github.com/chonkie-inc/chonkie/pull/103Invalid Notebook error by @chonknick in https://github.com/chonkie-inc/chonkie/pull/104cloud.SDPMChunker + cloud.LateChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/107base_url to the OpenAIEmbeddings and **kwargs support by @chonknick in https://github.com/chonkie-inc/chonkie/pull/108RAGHub by @chonknick in https://github.com/chonkie-inc/chonkie/pull/111AutoEmbeddings model matching + URI support for providers by @chonknick in https://github.com/chonkie-inc/chonkie/pull/146URI support via provider://organization/model style strings in AutoEmbeddings by @chonknick in https://github.com/chonkie-inc/chonkie/pull/148cloud.NeuralChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/152NeuralChunker and SlumberChunker to the chonkie.cloud by @chonknick in https://github.com/chonkie-inc/chonkie/pull/151Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.7...v1.0.8
Added initial support for Handshakes and Porters with 1 new Porter (JSONPorter) as well as 3 new Handshakes (ChromaHandshake, QdrantHandshake and Turb
Handshakes and Porters with 1 new Porter (JSONPorter) as well as 3 new Handshakes (ChromaHandshake, QdrantHandshake and TurbopufferHandshake)! Extemely simple to use, you can just import, init and call on your List[Chunks] that you get out from your chunkers/refineries and query them for search!OpenAIGenie which allows you to use the OpenAI models with the SlumberChunker and many more! Any API that supports the OpenAI API format will be usable with the base_url changed. Try out various LLMs with the SlumberChunker to see which works best for you!VoyageAIEmbeddings (thanks to @Pratik960): Use VoyageAI models with EmbeddingsRefinery and SemanticChunkers for your ingestion~Visualizer: Dark mode and a retro tiktokenizer theme (thanks to @Udayk02)NeuralChunkerDOCS.md for easy answers about Chonkie with LLMs.tiktokenizer theme to the Visualizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/81VoyageAIEmbeddings by @chonknick in https://github.com/chonkie-inc/chonkie/pull/85NeuralChunker by adding support for multiple models, custom… by @chonknick in https://github.com/chonkie-inc/chonkie/pull/86BaseHandshake and BasePorter by @chonknick in https://github.com/chonkie-inc/chonkie/pull/91ChromaHandshake + QdrantHandshake + Document + Chomp + More by @chonknick in https://github.com/chonkie-inc/chonkie/pull/93OpenAIGenie + TurbopufferHandshake by @chonknick in https://github.com/chonkie-inc/chonkie/pull/95dark mode to chonkie's Visualizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/98Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.6...v1.0.7
Welcome Chonkie's very own agentic chunker, SlumberChunker! Requires the genie optional install to work. Genie is Chonkie's Generative Inference Engin
SlumberChunker! Requires the genie optional install to work. Genie is Chonkie's Generative Inference Engines, that allow any generative models or API to easily plug-in with Chonkie. Currently, the genie optional install installs dependencies for GeminiGenie — and will require a GEMINI_API_KEY to work properly.pip install "chonkie[genie]"
# Import
from chonkie import SlumberChunker
# Initialize
chunker = SlumberChunker(verbose=True) # set verbose to True, since it takes a while~
# CHONK!
chunker(text)
NeuralChunker! Requires the neural optional install to work. This uses a BERT-like model that's fine-tuned for chunking, making it really fast and high-quality. Second only to SlumberChunker in terms of chunk quality.pip install "chonkie[neural]"
# import
from chonkie import NeuralChunker
# initialize
chunker = NeuralChunker()
# CHONK!
chunks = chunker(text)
auto language detection for CodeChunker! Now you can just pass in code to it without having to specify the language before hand. It will detect the language by itself. While the latency for detection is minimal (sub-millisecond), it does affect performance, so please specify the language if you care.# Import
from chonkie import CodeChunker
# Initialize the "auto" CodeChunker
chunker = CodeChunker() # No need to specify, "auto" by default
# CHONK!
chunks = chunker(code)
Genies! Since this version features our first generative feature, the SlumberChunker, we added the Genie to work with it as well as future generative features. Genie's are chonkie's way to handle multiple APIs and model interfaces working together for chunking~ The first Genie to be added is the GeminiGenie — which works with Gemini models. Requires the genie optional install.pip install "chonkie[genie]"
# Import
from chonkie import GeminiGenie
# Init
genie = GeminiGenie(api_key=YOUR_API_KEY)
# generate
genie.generate("Hi!")
# generate JSON
genie.generate_json("Hi", JSON_SCHEMA)
CodeChunker via Magika by @chonknick in https://github.com/chonkie-inc/chonkie/pull/71SlumberChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/73auto language support for CodeChunker + Add initial support for SlumberChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/74SlumberChunker's splitting algorithm from RecursiveChunker to custom + Pickling issue in BaseTokenizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/78NeuralChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/77NeuralChunker + Bump version to v1.0.6 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/79Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.5...v1.0.6
Feat : update test workflow to run on prs even before merging by @not-lain in https://github.com/chonkie-inc/chonkie/pull/69
CodeChunker via Magika by @chonknick in https://github.com/chonkie-inc/chonkie/pull/71SlumberChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/73auto language support for CodeChunker + Add initial support for SlumberChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/74Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.5...v1.0.6a0
This is a quick patch release to include CodeChunker in the __init__.py for chonkie so it can be properly accessed via from chonkie import CodeChunker
CodeChunker in the __init__.py for chonkie so it can be properly accessed via from chonkie import CodeChunker.CodeChunker properly to the main chonkie.__init__ + Bump up version to v1.0.5 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/64Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.4...v1.0.5
Welcome our newest chunker to the family: CodeChunker! CodeChunker is specialized to handle code files and can gain structural understanding of the co
CodeChunker! CodeChunker is specialized to handle code files and can gain structural understanding of the code before chunking each portion separately~ Supports 100+ programming languages! Let's check out the usage~Firstly, install the code chunker dependencies via:
pip install "chonkie[code]"
and then simply run it like any other chunker~
# Import
from chonkie import CodeChunker
# Init
chunker = CodeChunker(language="python")
# Get some python code
code = ...
# CHONK!
chunks = chunker(code)
JinaAI embeddings with JinaEmbeddings — allowing for use with SemanticChunker and SDPMChunker!Install it via the following command:
pip install "chonkie[jina]"
and use it like this~
# Import
from chonkie import JinaEmbeddings
from chonkie import SemanticChunker
# Optionally import Visaulizer for visalizations
from chonkie import Visualizer()
# Init
viz = Visualizer()
embeddings = JinaEmbeddings()
chunker = SemanticChunker(embeddings)
# Get some text
text = ...
# CHONK!
chunks = chunker(text)
# Optional: Visualize!
viz(chunks)
OverlapRefinery which allows you to add overlap context to your chunks~ It's available in the default install and can be used with any chunker. Just chunk with a chunker and pass your chunks through the refinery!# import
from chonkie import RecursiveChunker, OverlapRefinery
# Init
chunker = RecursiveChunker()
refinery = OverlapRefinery("gpt2") # By default initializes with "character" tokenizer, can pass in "gpt2" to match the chunker
# Get some text
text = ...
# CHONK and Refine!
chunks = chunker(text)
chunks = refinery(chunks)
EmbeddingsRefinery which allows you to run the chunks through a embedding model and have the embeddings available for downstream loading in a vector database. Similar to the OverlapRefinery just pass in the chunks from a Chunker into a EmbeddingsRefinery object loaded with the appropriate embedding model and each Chunk will then be loaded with .embedding value which can be used downstream.CodeChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/53CodeChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/54No module named 'chonkie.cloud.chunkers' error by @chonknick in https://github.com/chonkie-inc/chonkie/pull/58chonkie.cloud path bug + update README.md to have integrations by @chonknick in https://github.com/chonkie-inc/chonkie/pull/60Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.3...v1.0.4
The new Chonkie Visualizer is here! You can now view your chunks, understand chunk quality and debug your chunker with visual feedback~ Use the print
Visualizer is here! You can now view your chunks, understand chunk quality and debug your chunker with visual feedback~ Use the print method to print rich text on your terminal or use the save method to save a highlighted html on your device! It's very simple to use, just pass in your chunks~from chonkie import Visualizer
viz = Visualizer()
# Print the chunks on the terminal with .print or directly call the Visualizer object too
viz.print(chunks)
# Save the HTML file
viz.save("chonkie.html", chunks)
<img width="1028" alt="image" src="https://github.com/user-attachments/assets/ea38959b-01bd-4be9-90de-6442542e98a0" />
Recipes which allow you to use multilingual chunking out of the box, as well as document specific chunking methods. Initial support starts with: en, hi, zh, jp and ko, while document type markdown is supported too. Use it via the from_recipe class method with any chunker that takes delimiters or RecursiveRules.from chonkie import RecursiveChunker
# Initialize the recursive chunker to chunk Markdown
chunker = RecursiveChunker.from_recipe("markdown", lang="en")
# Initialize the recursive chunker to chunk Hindi texts
chunker = RecursiveChunker.from_recipe(lang="hi")
RecursiveChunker, SentenceChunker, and WordTokenizerSentenceChunker to remove estimate + feedback by @chonknick in https://github.com/chonkie-inc/chonkie/pull/23RecursiveRules.from_dict() showing Key error: 'levels' does not exist because of .pop() by @chonknick in https://github.com/chonkie-inc/chonkie/pull/24.find for indexing in the RecursiveChunker for better efficiency by @Pratik960 in https://github.com/chonkie-inc/chonkie/pull/26Recipes and multilingual CHONKs! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/27RecursiveChunker whitespace splitting is not reconstructable (missing spaces) + (#26, #27) by @chonknick in https://github.com/chonkie-inc/chonkie/pull/28WordTokenizer.count_tokens should use .tokenize instead of .encode by @chonknick in https://github.com/chonkie-inc/chonkie/pull/36Visualizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/39Visualizer class for easy chunk visualizations by @chonknick in https://github.com/chonkie-inc/chonkie/pull/40chonkie.utils module to package list in pyproject.toml by @chonknick in https://github.com/chonkie-inc/chonkie/pull/41chonkie.cloud does not contain chunkers error + bump up version to v1.0.3 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/43Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.2...v1.0.3
Fix: Remove .find for indexing in the RecursiveChunker for better efficiency by @Pratik960 in https://github.com/chonkie-inc/chonkie/pull/26
.find for indexing in the RecursiveChunker for better efficiency by @Pratik960 in https://github.com/chonkie-inc/chonkie/pull/26Recipes and multilingual CHONKs! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/27RecursiveChunker whitespace splitting is not reconstructable (missing spaces) + (#26, #27) by @chonknick in https://github.com/chonkie-inc/chonkie/pull/28Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.3a0...v1.0.3a1
Fix: Refactor SentenceChunker to remove estimate + feedback by @chonknick in https://github.com/chonkie-inc/chonkie/pull/23
SentenceChunker to remove estimate + feedback by @chonknick in https://github.com/chonkie-inc/chonkie/pull/23RecursiveRules.from_dict() showing Key error: 'levels' does not exist because of .pop() by @chonknick in https://github.com/chonkie-inc/chonkie/pull/24Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.2...v1.0.3a0
Fix: numpy array not serializable for LateChunk; convert to list + bump version to v1.0.2a0 by @chonknick in https://github.com/chonkie-inc/chonkie/pu
LateChunk; convert to list + bump version to v1.0.2a0 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/19RecursiveChunker to use _recursive_chunk method by @chonknick in https://github.com/chonkie-inc/chonkie/pull/20Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.1...v1.0.2
Fix: numpy array not serializable for LateChunk; convert to list + bump version to v1.0.2a0 by @chonknick in https://github.com/chonkie-inc/chonkie/pu
LateChunk; convert to list + bump version to v1.0.2a0 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/19Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.1...v1.0.2a0
Update version to 1.0.1 in pyproject.toml and __init__.py to fix PyPI release issue by @chonknick in https://github.com/chonkie-inc/chonkie/pull/18
Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.0...v1.0.1
Fix: update numpy version to >=2.0.0 by @chonknick in https://github.com/chonkie-inc/chonkie/pull/12
RecursiveLevel + better error message in LateChunker + Add tests for LateChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/14Full Changelog: https://github.com/chonkie-inc/chonkie/compare/v1.0.0a0...v1.0.0a1
Documentation by @shreyash-chonkie in https://github.com/chonkie-inc/chonkie/pull/1
_CHARS_PER_TOKEN estimate update + other minor changes by @chonknick in https://github.com/chonkie-inc/chonkie/pull/51.0.0a0 + Fix minor bug in SDPMChunker by @chonknick in https://github.com/chonkie-inc/chonkie/pull/11Full Changelog: https://github.com/chonkie-inc/chonkie/commits/v1.0.0a0
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →