Typer removed click as a dependency in favour of vendoring it, but spaCy imports from Click, so the requirement needs to be added.

• Fix spacy download failing in environments where pip is not on PATH but is available as a Python module (e.g., some virtual environments and containers)

The v3.8.12 release didn't update the confection pin, which meant that if you did an upgrade-install models wouldn't load.

Use confection v1.3 and Thinc v8.3.13, which implement custom validation logic in place of Pydantic, allowing us to properly adopt Pydantic v2 and provide full Python 3.14 support.

Our dependency tree used Pydantic v1 in unusual ways, and relied on behaviours that Pydantic v2 reformed. In the time since Pydantic v2 was released there were a few attempts to migrate over to it, but the task has been complicated by the fact that the confection library has a fairly tangled implementation and I had reduced availability for open-source work in 2024 and 2025.

Specifically, our library confection provides the extensible configuration system we use in spaCy and Thinc. The config system allows you to refer to values that will be supplied by arbitrary functions, that e.g. define some neural network model or its sublayers. The functionality in confection is complicated because we aggressively prioritised user experience in the specification, even if it required increased implementation complexity.

Confection's original implementation built a dynamic Pydantic v1 schema for function-supplied values ("promises"). We validate the schema before calling any promises, and then validate the schema again after calling all the promises and substituting in their values. The variable-interpolation system adds further difficulties to the implementation, and we have to do it all subclassing the Python built-in configparser, which ties us to implementation choices I'd do differently if I had a clean slate.

Here's one summary of Pydantic v1-specific behaviours that the migration to v2 particularly difficult for us. This particular summary was produced during a session with Claude Code Opus 4.6, so nuances of it might be wrong. The full history of attempts at doing this spans over different refactors separated by a few months at a time, so I don't have a full record of all the things that I struggled with. It's possible some details of this summary are incorrect though.

The core problem we kept hitting: Pydantic v2 compiles validation schemas upfront and has much stricter immutability. The whole session has been a series of workarounds for this:

 1. Schema mutation — v1 let you mutate __fields__ in place; v2 needs model_rebuild() which loses forward ref namespaces, or create_model subclasses which don't propagate to parent schemas.
 2. model_dump vs dict — v2 converts dataclasses to dicts, breaking resolved objects. Needed a custom _model_to_dict helper.
 3. model_construct drops extras — v2 silently drops fields with extra="forbid", needed manual workarounds.
 4. Strict coercion — v2 coerces ndarray to List[Floats1d] via iteration, needed strict=True.
 5. Forward refs — Every schema with TYPE_CHECKING imports needs model_rebuild() with the right namespace, and that breaks when confection re-rebuilds later.
In order to adjust for behavioural differences like this, I'd refactored confection to build the different versions of the schema in multiple passes, instead of building all the representations together as we'd been doing. However this refactor itself had problems, further complicating the migration.

I've now bitten the bullet and rolled back the refactor I'd been attempting of confection, and instead replaced the Pydantic validation with custom logic. This allows Confection to remove Pydantic as a dependency entirely.~ Update: Actually I went back and got the refactor working. All much nicer now.

I've taken some lengths to explain this because migrating off a dependency after breaking changes can be a sensitive topic. I want to stress that the changes Pydantic made from v1 to v2 are very good, and I greatly appreciate them as a user of FastAPI in our services. It would be very bad for the ecosystem if Pydantic pinned themselves to exactly matching the behaviours they had in v1 just to avoid breaking support for the sort of thing we'd been doing. Instead users who were relying on those behaviours like us should just find some way to adapt --- either vendor the v1 version we need, or change our behaviours, or implement an alternative. I would have liked to do this sooner but we've ultimately gone with the third option.

Add wheels for Python 3.11, 3.12, 3.13 and 3.14 for Windows ARM. Windows ARM wheels for Python 3.10 and earlier are not available in numpy, so aren't provided.

Windows arm needs to be disabled at the ci level, so remove this skip…

… selector

Add wheels for Python 3.14

• Fix deprecation warnings from click imports
• Update requirements, including switch to typer-slim to reduce dependency footprint
• Drop support for Python 3.9 (end-of-life)

Other dependencies in spaCy's tree have also been updated to widen the numpy compatibility pin, which should reduce installation problems for some users.

In order to support Python 3.13, spaCy is now compiled with Cython 3. This brings a change to the way types are handled at runtime (Cython 3 uses the from __future__ import annotations semantics, which stores types as strings at runtime. This difference caused problems for components registered within Cython files, as we rely on building Pydantic models from factory function signatures to do validation.

To support Python 3.13 we therefore create a new module, spacy.pipeline.factories, which contains the factory function implementations. __getattr__ import shims have been added to the previous locations of these functions to prevent backwards incompatibilities.

As well as moving the factories, the new implementation avoids import-time side-effects, by moving the actual calls to the decorator inside a function, which is executed once when the Language class is initialised.

A matching change has been made to the catalogue registry decorators. A new module spacy.registrations has been created that performs all the catalogue registrations. Moving these registrations away from the functions prevents these decorators from running at import time. This change was not necessary for the Python 3.13 support, but it means we no longer rely on any import-time side-effects, which will allow us to improve spaCy's import time and therefore CLI execution time. The change also makes maintenance easier as it's easier to find the implementations of different registry functions (this may help library users as well).

Restores support for wheels for ARM platforms, while correctly noting compatibility range.

Fix bug in memory zones when non-transient strings were added to the StringStore inside a memory zone. This caused a bug in the morphological analyser that caused string not found errors when applied during a memory zone.

Optional memory management for persistent services

Support a new context manager method Language.memory_zone(), to allow long-running services to avoid growing memory usage from cached entries in the Vocab or StringStore. Once the memory zone block ends, spaCy will evict Vocab and StringStore entries that were added during the block, freeing up memory. Doc objects created inside a memory zone block should not be accessed outside the block.

The current implementation disables population of the tokenizer cache inside the memory zone, resulting in some performance impact. The performance difference will likely be negligible if you're running a full pipeline, but if you're only running the tokenizer, it'll be much slower. If this is a problem, you can mitigate it by warming the cache first, by processing the first few batches of text without creating a memory zone. Support for memory zones in the tokenizer will be added in a future update.

The Language.memory_zone() context manager also checks for a memory_zone() method on pipeline components, so that components can perform similar memory management if necessary. None of the built-in components currently require this.

If you component needs to add non-transient entries to the StringStore or Vocab, you can pass the allow_transient=False flag to the Vocab.add() or StringStore.add() components.

Example usage:

import spacy
import json
from pathlib import Path
from typing import Iterator
from collections import Counter
import typer
from spacy.util import minibatch


def texts(path: Path) -> Iterator[str]:
    with path.open("r", encoding="utf8") as file_:
        for line in file_:
            yield json.loads(line)["text"]

def main(jsonl_path: Path) -> None:
    nlp = spacy.load("en_core_web_sm")
    counts = Counter()
    batches = minibatch(texts(jsonl_path), 1000)
    for i, batch in enumerate(batches):
        print("Batch", i)
        with nlp.memory_zone():
            for doc in nlp.pipe(batch):
                for token in doc:
                    counts[token.text] += 1
    for word, count in counts.most_common(100):
        print(count, word)

if __name__ == "__main__":
    typer.run(main)

Numpy v2 compatibility

Numpy 2.0 isn't binary-compatible with numpy v1, so we need to build against one or the other. This release isolates the dependency change and has no other changes, to make things easier if the dependency change causes problems.

This dependency change was previously attempted in version 3.7.6, but dependencies within the v3.7 family of models resulted in some conflicts, and some packages depending on numpy v1 were incompatible with v3.7.6. I've therefore removed the 3.7.6 release and replaced it with this one, which increments the minor version.

Model packages no longer list spacy as a requirement

I've also made a change to the way models are packaged to make it easier to release more quickly. Previously spaCy models specified a versioned requirement on spacy itself. This meant that there was no way to increment the spaCy version and have it work with the existing models, because the models would specify they were only compatible with spacy>=3.7.0,<3.8.0. We have a compatibility table that allows spacy to see which models are compatible, but the models themselves can't know which future versions of spaCy they work with.

I've therefore added a flag --require-parent/--no-require-parent to the spacy package CLI, which controls where the parent package (e.g. spaCy) should be listed as a requirement of the model. --require-parent is the default for v3.8, but this will change to --no-require-parent by default in v4. I've set --no-require-parent for the v3.8 models, so that further changes can be published that don't impact the models, without retraining the models or forcing users to redownload them.

✨ New features and improvements

• Sanitize direct download for spacy download (#13313).
• Convert Cython properties to decorator syntax (#13390).
• Bump Weasel pin to allow v0.4.x (#13409).
• Improvements to the test suite (#13469, #13470).
• Bump Typer pin to allow v0.10.0 and above (#13471).
• Allow typing-extensions<5.0.0 for Python < 3.8 (#13516).

🔴 Bug fixes

• #13400: Fix use_gold_ents behaviour for EntityLinker.

📖 Documentation and examples

• Make the file name for code listings stick to the top (#13379).
• Update the documentation of MorphAnalysis (#13433).
• Typo fixes in the documentation (#13466).

👥 Contributors

@danieldk, @honnibal, @ines, @JoeSchiff, @nokados, @Paillat-dev, @rmitsch, @schorfma, @strickvl, @svlandeg, @ynx0

✨ New features and improvements

• Improve NumPy 2.0 compatibility (#13103).
• Added language extensions for Faroese and Norwegian Nynorsk (#13116).
• Add new TextCatReduce.v1 layer for text classification (#13181).
• Add new TextCatParametricAttention.v1 layer for text classification (#13201).
• Use build module for creating model packages by default (#13109).
• Add support for code loading to the benchmark speed command (#13247).
• Extend lexical attributes for English with more numericals (#13106).
• Warn about reloading dependencies after downloading models (#13081).

🔴 Bug fixes

• #13259, #13304, #13321: Correctness fixes for multiprocessing support in Language.pipe.
• #13187: Typing and documentation fixes for Doc.
• #13086: Update Tokenizer.explain for special cases with whitespace.
• #13068: Fix displaCy span stacking.
• #13149: Add spacy.TextCatBOW.v3 to use the fixed SparseLinear layer.

📖 Documentation and examples

• Many improvements and updates to the LLM documentation.
• Update trf_data examples and the transformer pipeline design section.

👥 Contributors

@adrianeboyd, @danieldk, @evornov, @honnibal, @ines, @lise-brinck, @ridge-kimani, @rmitsch, @shadeMe, @svlandeg

✨ New features and improvements

• Update __all__ fields (#13063).

🔴 Bug fixes

• #13035: Remove Pathy requirement.
• #13053: Restore spacy.cli.project API.
• #13057: Support Any comparisons for Token and Span.

📖 Documentation and examples

• Many updates for spacy-llm including Azure OpenAI, PaLM, and Mistral support.
• Various documentation corrections.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @rmitsch, @svlandeg

🔴 Bug fixes

• Revert lazy loading of CLI module for spacy.info to fix availability of spacy.cli following import spacy (#13040).

👥 Contributors

@adrianeboyd, @honnibal, @ines, @svlandeg

This release drops support for Python 3.6 and adds support for Python 3.12.

✨ New features and improvements

• Add support for Python 3.12 (#12979).
• Use the new library Weasel for spaCy projects functionality (#12769).
  • All spacy project commands should run as before, just now they're using Weasel under the hood.
  • ⚠️ Remote storage is not yet supported for Python 3.12. Use Python 3.11 or earlier for remote storage.
• Extend to Thinc v8.2 (#12897).
• Extend transformers extra to spacy-transformers v1.3 (#13025).
• Support registered vectors (#12492).
• Add --spans-key option for CLI evaluation with spacy benchmark accuracy (#12981).
• Load the CLI module lazily for spacy.info (#12962).
• Add type stubs for spacy.training.example (#12801).
• Warn for unsupported pattern keys in dependency matcher (#12928).
• Language.replace_listeners: Pass the replaced listener and the tok2vec pipe to the callback in order to support spacy-curated-transformers (#12785).
• Always use tqdm with disable=None to disable output in non-interactive environments (#12979).
• Language updates:
  • Add left and right pointing angle brackets as punctuation to ancient Greek (#12829).
  • Update example sentences for Turkish (#12895).
• Package setup updates:
  • Update NumPy build constraints for NumPy 1.25+ (#12839). For Python 3.9+, it is no longer necessary to set build constraints while building binary wheels.
  • Refactor Cython profiling in order to disable profiling for Python 3.12 in the package setup, since Cython does not currently support profiling for Python 3.12 (#12979).

📦 Trained pipelines updates

The transformer-based trf pipelines have been updated to use our new Curated Transformers library through the Thinc model wrappers and pipeline component from spaCy Curated Transformers.

⚠️ Backwards incompatibilities

• Drop support for Python 3.6.
• Drop mypy checks for Python 3.7.
• Remove ray extra.
• spacy project has a few backwards incompatibilities due to the transition to the standalone library Weasel, which is not as tightly coupled to spaCy. Weasel produces warnings when it detects older spaCy-specific settings in your environment or project config.
  • Support for the spacy_version configuration key has been dropped.
  • Support for the check_requirements configuration key has been dropped due to the deprecation of pkg_resources.
  • The SPACY_CONFIG_OVERRIDES environment variable is no longer checked. You can set configuration overrides using WEASEL_CONFIG_OVERRIDES.
  • Support for SPACY_PROJECT_USE_GIT_VERSION environment variable has been dropped.
  • Error codes are now Weasel-specific and do not follow spaCy error codes.

📖 Documentation and examples

• New and updated documentation for large language models and spaCy Curated Transformers.
• Various documentation corrections and updates.
• New additions to the spaCy Universe:
  • Hobbit spaCy: NLP for Middle Earth
  • rolegal: a spaCy Package for Noisy Romanian Legal Document Processing

👥 Contributors

@adrianeboyd, @bdura, @connorbrinton, @danieldk, @davidberenstein1957, @denizcodeyaa, @eltociear, @evornov, @honnibal, @ines, @jmyerston, @koaning, @magdaaniol, @pdhall99, @ringohoffman, @rmitsch, @senisioi, @shadeMe, @svlandeg, @vinbo8, @wjbmattingly

✨ New features and improvements

• Allow Pydantic v2 using transitional v1 support (#12888).
• Add find-function CLI for finding locations of registered functions (#12757).
• Add extra spacy[cuda12x] for cupy-cuda12x (#12890).
• Extend tests for init config and train CLI (#12173).
• Switch from distutils to setuptools/sysconfig (#12853).

🔴 Bug fixes

• #12817: Escape annotated HTML tags in displaCy span renderer.
• #12857: Display model's full base version string in incompatibility warning.
• #12882: Update <br> tags in displaCy.

📖 Documentation and examples

• Various documentation corrections and updates.
• New additions to spaCy Universe:
  • OdyCy
  • SaysWho

👥 Contributors

@adrianeboyd, @afriedman412, @arplusman, @bdura, @connorbrinton, @honnibal, @ines, @it176131, @pmbaumgartner, @rmitsch, @shadeMe, @svlandeg, @thomashacker, @victorialslocum, @x-tabdeveloping

✨ New features and improvements

• NEW: span_finder pipeline component to identify overlapping, unlabeled spans (#12507).
• Language updates:
  • Add initial support for Malay (#12602).
  • Update Latin defaults to support noun chunks, update lexical/tokenizer defaults and add example sentences (#12538).
• Add option to return scores separately keyed by component name with spacy evaluate --per-component, Language.evaluate(per_component=True) and Scorer.score(per_component=True) (#12540).
• Support custom token/lexeme attribute for vectors (#12625).
• Support spancat_singlelabel in spacy debug data CLI (#12749).
• Typing updates for PhraseMatcher and SpanGroup (#12642, #12714).

🔴 Bug fixes

• #12569: Require that all SpanGroup spans come from the current doc.

📦 Trained pipelines updates

We have added new pipelines for Slovenian that use the trainable lemmatizer and floret vectors.

• 🙏 Special thanks to @orglce for help with the new pipelines!

The English pipelines have been updated to improve handling of contractions with various apostrophes and to lemmatize "get" as a passive auxiliary.

The Danish pipeline da_core_news_trf has been updated to use vesteinn/DanskBERT with performance improvements across the board.

⚠️ Backwards incompatibilities

• SpanGroup spans are now required to be from the same doc. When initializing a SpanGroup, there is a new check to verify that all added spans refer to the current doc. Without this check, it was possible to run into string store or other errors.

📖 Documentation and examples

• Various documentation corrections and updates.
• New additions to spaCy Universe:
  • spaCy-SetFit
  • spacy-vscode
  • SpanMarker
  • vetiver

👥 Contributors

@adrianeboyd, @bdura, @danieldk, @davidberenstein1957, @diyclassics, @essenmitsosse, @honnibal, @ines, @isabelizimm, @jmyerston, @kadarakos, @KennethEnevoldsen, @khursani8, @ljvmiranda921, @rmitsch, @shadeMe, @svlandeg, @tomaarsen, @victorialslocum, @vin-ivar, @ZiadAmerr

✨ New features and improvements

• Extend Typer support to v0.9 (#12631).

🔴 Bug fixes

• #12701: Fix issues with component names and listeners for sourced components.
• #12623: Support overrides for registered functions in configs.

👥 Contributors

@adrianeboyd, @bdura, @honnibal, @ines, @svlandeg

✨ New features and improvements

• Huge speed improvements for spancat, in particular on GPU (~10x-30x faster) (#12577).
• Improve speed for child operators (>+, >-, >++, >--) for the dependency matcher (#12528).
• Improve loading speed for tokenizers with a large number of exceptions (#12553).
• Support doc.spans for displaCy output in spacy benchmark accuracy / spacy evaluate (#12575).
• Add MorphAnalysis.get(default=) argument for user-provided default values similar to dict (#12545).
• Only perform vectors checks during initialization if there are sourced components (#12607).

🔴 Bug fixes

• #12567: Remove #egg from download URLs due to future deprecation in pip.

📖 Documentation and examples

• Various documentation corrections and updates.
• New additions to spaCy Universe:
  • LatinCy
  • parsigs
  • spaCysee
  • spacy-wasm

👥 Contributors

@adrianeboyd, @andyjessen, @bdura, @davidberenstein1957, @diyclassics, @honnibal, @ines, @kadarakos, @KennethEnevoldsen, @ljvmiranda921, @moxley01, @royashcenazi, @svlandeg, @tanloong, @victorialslocum

✨ New features and improvements

• Add support for floret vectors in spacy pretrain (#12435).
• Save final model as model-last.bin for spacy pretrain (#12459).
• Support Span input for displacy.parse_deps (#12477).
• Extend support to CuPy 12.0 for cupy install extras.

🔴 Bug fixes

• #12398: Fix entity linker failure on sentence-crossing entities.
• #12405: Fix sentence indexing bug in Span.sents.
• #12469: Fix scores attribute for spancat_singlelabel.
• #12484: Fix Span.sents when the final sentence is the last token in a Doc.
• #12486: Fix pickle for the ngram suggester.
• #12493: Include Span.kb_id and Span.id strings in Doc and DocBin serialization.

📖 Documentation and examples

• Various documentation corrections and updates.
• New addition to spaCy Universe:
  • Sentimental Onix

👥 Contributors

@adrianeboyd, @BLKSerene, @honnibal, @ines, @kadarakos, @prajakta-1527, @rmitsch, @shadeMe, @sloev, @svlandeg, @thomashacker, @willfrey

💥 We'd love to hear more about your experience with spaCy! Take our survey here.

✨ New features and improvements

• NEW: spancat_singlelabel pipeline component for multi-class and non-overlapping span classification. The spancat_singlelabel component predicts at most one label for each suggested span and adds a new setting allow_overlap to restrict the output to non-overlapping spans (#11365).
• Extend to mypy v1.0 (#12245).
• Use transformer + CNN for efficient GPU textcat with spacy init config (#11900).
• Support trainable lemmatizer in spacy debug data (#11419).
• Add new operators to dependency matcher for left/right immediate child/parent nodes (>+, >-, <+, <-) (#12334).
• Add spacy.PlainTextCorpusReader.v1 for plain text input (#12122).
• Add alignment_mode and span_id to Span.char_span() (#12145, #12196).
• Use string formatting types in logging calls (#12215).

🔴 Bug fixes

• #12017: Improve speed for top_k>1 in trainable lemmatizer.
• #12048: Make test_cli_find_threshold() test more robust.
• #12227: Fix return type of registry.find().
• #12272: Fix speed regression for Matcher patterns with extension attributes.
• #12287: Add grc to languages with lexeme norms in spacy-lookups-data.
• #12320: Make generation of empty KnowledgeBase instances configurable.
• #12343: Fix error message for displacy auto_select_port.
• #12347: Fix length check for knowledge base in entity linker, add InMemoryLookupKB.is_empty.
• #12365: Fix types for Lexeme.orth and Lexeme.lower.
• #12366: Raise error for non-default vectors with PretrainVectors.
• #12368: Partially address pending deprecation of pkg_resources.
• Various improvements and fixes for the test suite (#12148, #12157, #12210, #12303, #12372).

📖 Documentation and examples

• Many website updates to improve accessibility.
• Various documentation corrections and updates.
• New projects:
  • Span labeling datasets
  • Comparing embedding layers in spaCy from the technical report Multi hash embeddings in spaCy

👥 Contributors

@adrianeboyd, @andyjessen, @danieldk, @essenmitsosse, @honnibal, @ines, @itssimon, @kadarakos, @kwhumphreys, @ljvmiranda921, @pmbaumgartner, @polm, @richardpaulhudson, @rmitsch, @shadeMe, @svlandeg, @tanloong, @thomashacker, @victorialslocum

✨ New features and improvements

• NEW: New apply CLI command to annotate new documents with a trained pipeline (#11376).
• NEW: New benchmark CLI command to benchmark pipelines. The new benchmark speed subcommand measures the speed of a pipeline, the benchmark accuracy subcommand is a new alias for evaluate (#11902).
• NEW: New find-threshold CLI command to identify an optimal threshold for classification models (#11280).
• NEW: New FUZZY Matcher operator for fuzzy matches based on Levenshtein edit distance. In addition, the FUZZY and REGEX operators are now supported in combination with IN/NOT_IN. (#11359).
• Language updates for Ancient Greek, Dutch, Russian, Slovenian and Ukrainian (#11345, #11162, #11426, #11753, #11811, #11997, more details below).
• Allow up to typer v0.7.x (#11720), mypy 0.990 (#11801) and typing_extensions v4.4.x (#12036).
• New spacy.ConsoleLogger.v3 with expanded progress tracking (#11972).
• Improved scoring behavior for textcat with spacy.textcat_scorer.v2 (#11696 and #11971) and spacy.textcat_multilabel_scorer.v2 (#11820).
• Improved customizability of the knowledge base used for entity linking, with the default implementation being the new InMemoryLookupKB (#11268).
• Optional before_update callback that is invoked at the start of each training step (#11739).
• Improve performance of SpanGroup (#11380).
• Improve UX around displacy.serve when the default port is in use (#11948).
• Patch a security vulnerability in extracting tar files (#11746).
• Add equality definition for vectors (#11806).
• Allow interpolation of variables in directory names in projects (#11235).
• Update default component configs to use the latest tok2vec version (#11618).

🔴 Bug fixes

• #11382: Fix lookup behavior for the French and Catalan lemmatizers.
• #11385: Ensure that downstream components can train properly on a frozen tok2vec or transformer layer.
• #11762: Support local file system remotes for projects.
• #11763: Raise an error when unsupported values are used for textcat.
• #11834: Ensure Vocab.to_disk respects the exclude setting for lookups and vectors.
• #12009: Fix a few typing issues for SpanGroup and Span objects.
• #12098: Correctly handle missing annotations in the edit tree lemmatizer.

⚠️ Backwards incompatibilities and model updates

The following changes may require you to update code that is using the relevant functionality:

• An error is now raised when unsupported values are given as input to train a textcat or textcat_multilabel model - ensure that values are 0.0 or 1.0 as explained in the docs.
• As KnowledgeBase is now an abstract class, you should call the constructor of the new InMemoryLookupKB instead when you want to use spaCy's default KB implementation. If you've written a custom KB that inherits from KnowledgeBase, you'll need to implement its abstract methods, or alternatively inherit from InMemoryLookupKB instead.

The following changes may influence the output of your language pipeline or trained models:

• Updates to language defaults:
  • Extended support for Slovenian (#11162).
  • Switch Russian and Ukrainian lemmatizers to pymorphy3 (#11345, #11811).
  • Support for editorial punctuation in Ancient Greek (#11426).
  • Update to Russian tokenizer exceptions (#11753).
  • Small fix in the list of Dutch stop words (#11997).
• Updates to model defaults:
  • Use the latest tok2vec defaults in all components (#11618).
  • Improve the default attributes used for the textcat and textcat_multilabel components (#11698).
  • Update the default scorer for textcat and textcat_multilabel to fix a bug related to threshold for textcat and to make it possible to score multiple textcat/textcat_multilabel components in a single pipeline with custom scorers. If no custom scorers are used, the cat_p/r/f scores will now only reflect the final component's labels and performance (#11696, #11820).
  • Correct the token_acc score to report the intended measure (# correct tokens / # predicted tokens, the same as in spaCy v2). The token_acc scores for v3.5 will be lower for the same performance because they were incorrectly inflated in v3.0-v3.4. The token_p/r/f scores should remain unchanged (#12073).

The following functionality will be changed in the near future - so it's best to start updating your scripts now to make them more generic:

• From v4 onwards, we'll rename the master branch to main.

📦 Trained pipelines updates

• The CNN pipelines add IS_SPACE as a tok2vec feature for tagger and morphologizer components to improve tagging of non-whitespace vs. whitespace tokens.
• The transformer pipelines require spacy-transformers v1.2, which uses the exact alignment from tokenizers for fast tokenizers instead of the heuristic alignment from spacy-alignments. For all trained pipelines except ja_core_news_trf, the alignments between spaCy tokens and transformer tokens may be slightly different. More details about the spacy-transformers changes in the v1.2.0 release notes.

📖 Documentation and examples

• We've ported our website from Gatsby to Next 🥳
• Updated the documentation on supported languages.
• Added a note about experimental M1 GPU support to the installation quickstart.
• Included documentation for the biluo_to_iob and iob_to_biluo functions.
• Fixed model links in the v3.4 usage documentation.
• Removed "new" tags of functionality from spaCy v2.x.
• Various small additions, spelling and typo fixes.
• spaCy Universe additions:
  • greCy: Providing Ancient Greek models
  • spacy-pythainlp: Add Thai support for spaCy
• New projects:
  • Accelerate NER with Speedster (experimental)

👥 Contributors

@aaronzipp, @adrianeboyd, @albertvillanova, @ArchiDevil, @cfuerbachersparks, @damian-romero, @danieldk, @darigovresearch, @DSLituiev, @essenmitsosse, @gremur, @honnibal, @ines, @jmyerston, @JosPolfliet, @kadarakos, @koaning, @kwhumphreys, @ljvmiranda921, @MarcoGorelli, @orglce, @pmbaumgartner, @polm, @richardpaulhudson, @rmitsch, @ryndaniels, @shadeMe, @svlandeg, @thomashacker, @TrellixVulnTeam, @wannaphong, @zhiiw, @zrpxx

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11934: Add strings when initializing from labels in EditTreeLemmatizer.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @danieldk, @honnibal, @ines, @polm, @svlandeg

✨ New features and improvements

• Extend Typer support to v0.7.x (#11720).

🔴 Bug fixes

• #11640: Handle docs with no entities in EntityLinker.
• #11688: Restore custom doc extension values in Doc.to_json() for attributes set by getters.
• #11706: Remove incorrect warning for pipeline_package.load().
• #11735: Improve spacy project requirements checks for unsupported specifiers and requirements lines.
• #11745: Revert modifications to spacy.load(disable=) that could enable currently disabled components.

👥 Contributors

@aaronzipp, @adrianeboyd, @honnibal, @ines, @polm, @rmitsch, @ryndaniels, @svlandeg, @thomashacker

✨ New features and improvements

• NEW: Luganda language support (#10847).
• NEW: Latin language support (#11349).
• NEW: spacy.ConsoleLogger.v2 optionally saves training logs to JSONL (#11214).
• NEW: New operators for the DependencyMatcher to include matching parents or children to the left or the right of the node (#10371).
• Prebuilt Python 3.11 wheels are now available for all spaCy dependencies distributed by @explosion.
• Support pydantic v1.10 and mypy 0.980+, drop mypy support for Python 3.6 (#11546, #11635).
• Support CuPy v11 and add extras for cuda11x and cuda-autodetect (using cupy-wheel) (#11279).
• Support custom attributes for tokens and spans in Doc.to_json() and Doc.from_json() (#11125).
• Make the enable and disable options for spacy.load() more consistent (#11459).
• Allow a single string argument for disable/enclude/exclude for spacy.load() (#11406).
• New --url flag for spacy info to print the direct download URL for a pipeline (#11175).
• Add a check for missing requirements in the spacy project CLI (#11226).
• Add a Levenshtein distance function (#11418).
• Improvements to the spacy debug data CLI for spancat data (#11504).
• Allow overriding spacy_version in spacy package metadata (#11552).
• Improve the error message when using the wrong command for spacy project assets (#11458).
• Ensure parent directories are created when storing the results of the spacy pretrain command (#11210).
• Extend support to newer versions of natto-py for the ko extra (#11222).

📦 Trained pipelines updates

This release includes updated English pipelines for spaCy v3.4 with improved NER performance. The updates in en_core_web_* v3.4.1 address issues related to training from data with partial named entity annotation, which led to lower NER recall in English pipeline versions v3.0.0–v3.4.0. In particular, entities that appear in the sections of the OntoNotes training data without NER annotation were not predicted consistently by the earlier pipeline versions, such as names and places that are frequent in the Biblical sections, e.g., "David" and "Egypt" (see #7493).

Use spacy download to update your English pipelines to the newest version. If you'd prefer to keep using an earlier version, you can specify the version directly with e.g. spacy download -d en_core_web_sm-3.4.0. You can check that you are using the new version (v3.4.1) with spacy validate:

NAME                     SPACY            VERSION
en_core_web_md           >=3.4.0,<3.5.0   3.4.1     ✔

🔴 Bug fixes

• #11275: Fix Dutch noun chunks to skip overlapping spans.
• #11276: Fix regex invalid escape sequences.
• #11312: Better handling of unexpected types in SetPredicate.
• #11460: Fix config validation failures caused by NVTX pipeline wrappers.
• #11506: Avoid unwanted side effects in Doc.__init__.
• #11540: Preserve missing entity annotation in augmenters.
• #11592: Fix issues with DVC commands.
• #11631: Fix initialization for pymorphy2_lookup lemmatizer mode for Russian and Ukrainian.

⚠️ Backwards incompatibilities

• If you're using a custom component that does not return a Doc type, an error will now be raised (#11424).
• If you're using a dot in a factory name, an error is raised as this is not supported (#11336).

📖 Documentation and examples

• Added documentation for the new experimental coref component.
• Added Ukrainian trained pipelines to the website.
• Added documentation for the spacy.models_and_pipes_with_nvtx_range.v1 callback.
• Fix English pipeline names in v3.4 release notes.
• Various fixes to the Example API documentation.
• Extensions and improvements to the displacy docs.
• Fix the example command for spacy project dvc.
• Update example code for spacy-wordnet.
• Improve API documentation around the initialize() function for pipeline components.
• Fix various typos and inconsistencies.
• spaCy universe additions:
  • concepCy: A spaCy wrapper for ConceptNet.
  • spaCy partial tagger: build a CRF tagger with a partially annotated dataset.
  • Zshot: Zero and Few shot named entity & relationships recognition.

👥 Contributors

@adrianeboyd, @bdura, @danieldk, @diyclassics, @DSLituiev, @GabrielePicco, @honnibal, @ines, @JulesBelveze, @kadarakos, @ljvmiranda921, @ninjalu, @pmbaumgartner, @polm, @radandreicristian, @richardpaulhudson, @rmitsch, @shadeMe, @stefawolf, @svlandeg, @thomashacker, @tobiusaolo, @tzussman , @yasufumy

🔴 Bug fixes

• Fix issue #11137: Fix compatibility with CuPy v9.x.

📖 Documentation and examples

• spaCy universe additions:
  • BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics.
  • English Interpretation Sentence Pattern: English interpretation for accurate translation from English to Japanese.

👥 Contributors

@adrianeboyd, @danieldk, @honnibal, @ines, @lll-lll-lll-lll, @Lucaterre, @MaartenGr, @mr-bjerre, @polm, @radenkovic

✨ New features and improvements

• Support for mypy 0.950+ and pydantic v1.9 (#10786).
• Prebuilt linux aarch64 wheels are now available for all spaCy dependencies distributed by @explosion.
• Min/max {n,m} operator for Matcher patterns (#10981).
• Language updates:
  • Improve tokenization for Cyrillic combining diacritics (#10837).
  • Improve English tokenizer exceptions for contractions with this/that/these/those (#10873).
• Improved speed of vector lookups (#10992).
• For the parser, use C saxpy/sgemm provided by the Ops implementation in order to use Accelerate through thinc-apple-ops (#10773).
• Improved speed of Example.get_aligned_parse and Example.get_aligned (#10952).
• Improved speed of StringStore lookups (#10938).
• Updated spacy project clone to try both main and master branches by default (#10843).
• Added confidence threshold for named entity linker (#11016).
• Improved handling of Typer optional default values for init_config_cli (#10788).
• Added cycle detection in parser projectivization methods (#10877).
• Added counts for NER labels in debug data (#10960).
• Support for adding NVTX ranges to TrainablePipe components (#10965).
• Support env variable SPACY_NUM_BUILD_JOBS to specify the number of build jobs to run in parallel with pip (#11073).

📦 Trained pipelines updates

We have added new pipelines for Croatian that use the trainable lemmatizer and floret vectors.

🙏 Special thanks to @gtoffoli for help with the new pipelines!

The English pipelines have new word vectors:

All CNN pipelines have been extended to add whitespace augmentation.

🔴 Bug fixes

• Fix issue #10960: Support hyphens in NER labels.
• Fix issue #10994: Fix horizontal spacing for spans in displaCy.
• Fix issue #11013: Check for any token with a vector in Doc.has_vector, distinguish 0-vectors and missing vectors in similarity warnings.
• Fix issue #11056: Don't use get_array_module in textcat.
• Fix issue #11092: Fix vertical alignment for spans in displaCy.

🚀 Notes about upgrading from v3.3

• Doc.has_vector now matches Token.has_vector and Span.has_vector: it returns True if at least one token in the doc has a vector rather than checking only whether the vocab contains vectors.

📖 Documentation and examples

• spaCy universe additions:
  • Aim-spacy: An Aim-based spaCy experiment tracker.
  • Asent: Fast, flexible and transparent sentiment analysis.
  • spaCy fishing: Named entity disambiguation and linking on Wikidata in spaCy with Entity-Fishing.
  • spacy-report: Generates interactive reports for spaCy models.

👥 Contributors

@adrianeboyd, @danieldk, @ericholscher, @gorarakelyan, @honnibal, @ines, @jademlc, @kadarakos, @KennethEnevoldsen, @koaning, @Lucaterre, @maxTarlov, @philipvollet, @pmbaumgartner, @polm, @richardpaulhudson, @rmitsch, @sadovnychyi, @shadeMe, @shen-qin, @single-fingal, @svlandeg, @victorialslocum, @Zackere

This bug fix release is primarily to address Pydantic incompatibility with typing_extensions>=4.6.0.

✨ New features and improvements

• Huge speed improvements for spancat, in particular on GPU (~10x-30x faster) (#12577).

🔴 Bug fixes

• Add typing_extensions requirement due to Pydantic incompatibility with typing_extensions>=4.6.0.
• Remove #egg from download URLs due to future deprecation in pip.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @kadarakos, @svlandeg

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #10911, #11194: Improve speed in precomputable_biaffine by avoiding concatenation.
• #11276, #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11934: Add strings when initializing from labels in EditTreeLemmatizer.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @danieldk, @honnibal, @ines, @polm, @svlandeg

✨ New features and improvements

• Add the SpanRuler component. This component saves a list of matched spans to Doc.spans[spans_key].
• Support for JSON serialization and deserialization of Doc objects.
• Add span analysis to debug data.
• Allow data assets to be made optional in a spaCy project.
• Prebuilt macOS ARM64 wheels are now available for all spaCy dependencies distributed by @Explosion.

🔴 Bug fixes

• Fix issue #9575: Fix Entity Linker with tokenization mismatches between gold and predicted Doc objects.
• Fix issue #10685: Fix serialization of SpanGroup objects that share the same name within one SpanGroups container.
• Fix issue #10718: Remove debug print statements in walk_head_nodes to avoid acquiring the GIL.
• Fix issue #10741: Make the StringStore.__getitem__ return type dependent on its parameter type.
• Fix issue #10734: Support removal of overlapping terms in PhraseMatcher.
• Fix issue #10772: Override SpanGroups.setdefault to also support Iterable[SpanGroup] as the default.
• Fix issue #10817: Ensure that the term ROOT is in the glossary.
• Fix issue #10830: Better errors for Doc.has_annotation and Matcher.
• Fix issue #10864: Avoid pickling Doc inputs passed to Language.pipe().
• Fix issue #10898: Fix schemas import in Doc.

⚠️ Backward incompatibilities

• Before this release, a validation bug allowed the configuration of a pipeline component to override the name of the pipeline itself through the name attribute. For example, the following pipeline component:

  [components.transformer]
  factory = "transformer"
  name = "custom_transformer_name"
  

  would be registered erroneously as custom_transformer_name. Such overrides are now ignored and a warning is emitted (#10779). From spaCy v3.3.1 onwards, this component will be registered as transformer.

👥 Contributors

@adrianeboyd, @danieldk, @freddyheppell, @honnibal, @ines, @kadarakos, @ldorigo, @ljvmiranda921, @maxTarlov, @pmbaumgartner, @polm, @pypae, @richardpaulhudson, @rmitsch, @shadeMe, @single-fingal, @svlandeg

✨ New features and improvements

• Improved speeds for many components, see speed benchmarks for trained pipelines:
  • Speed up parser and NER by using constant-time head lookups (#10048).
  • Support unnormalized softmax probabilities in spacy.Tagger.v2 to speed up inference for the tagger, morphologizer, senter and trainable lemmatizer (#10197).
  • Speed up parser projectivization functions (#10241).
  • Replace Ragged with faster AlignmentArray in Example for training (#10319).
  • Improve Matcher speed (#10659).
  • Improve serialization speed for empty Doc.spans (#10250).
• NEW: A trainable lemmatizer component that uses edit trees to transform tokens to lemmas. Add it to your config with spacy init config -p trainable_lemmatizer or using the quickstart.
• Language updates:
  • Initial support for Lower Sorbian and Upper Sorbian.
  • New noun chunks for Finnish.
  • Updated noun chunks for French, Italian and Spanish.
  • Additional updates for English, French, Italian, Japanese, Korean, Norwegian, Russian, Slovenian, Spanish, Turkish, Ukrainian and Vietnamese.
• Big endian support with thinc v8.0.14+ and thinc-bigendian-ops.
• Config comparisons with spacy debug diff-config.
• displaCy support for overlapping span annotation and multiple labeled arcs between the same tokens.
• SpanCategorizer.set_candidates for debugging span suggesters.
• The quickstart now supports adding spancat and trainable_lemmatizer components.

📦 Trained pipelines

v3.3 introduces trained pipelines for Finnish, Korean and Swedish which feature the trainable lemmatizer and floret vectors. Due to the use Bloom embeddings and subwords, the pipelines have compact vectors with no out-of-vocabulary words.

🙏 Special thanks to @aajanki, @thiippal (Finnish) and Elena Fano (Swedish) for their help with the new pipelines!

The new trainable lemmatizer is used for Danish, Dutch, Finnish, German, Greek, Italian, Korean, Lithuanian, Norwegian, Polish, Portuguese, Romanian and Swedish.

🔴 Bug fixes

• Fix issue #5447: Avoid overlapping arcs when using displaCy in manual mode.
• Fix issue #9443: Fix Scorer.score_cats for missing labels.
• Fix issue #9669: Fix entity linker batching.
• Fix issue #9903: Handle _ value for UPOS in CoNLL-U converter.
• Fix issue #9904: Fix textcat loss scaling.
• Fix issue #9956: Compare all Span attributes consistently.
• Fix issue #10073: Add "spans" to the output of doc.to_json.
• Fix issue #10086: Add tokenizer option to allow Matcher handling for all special cases.
• Fix issue #10189: Allow Example to align whitespace annotation.
• Fix issue #10302: Fix check for NER annotation in MISC in CoNLL-U converter.
• Fix issue #10324: Fix Tok2Vec for empty batches.
• Fix issue #10347: Update basic functionality for rehearse.
• Fix issue #10394: Fix Vectors.n_keys for floret vectors.
• Fix issue #10400: Use meta in util.load_model_from_config.
• Fix issue #10451: Fix Example.get_matching_ents.
• Fix issue #10460: Fix initial special cases for Tokenizer.explain.
• Fix issue #10521: Stream large assets on download in spaCy projects.
• Fix issue #10536: Handle unknown tags in KoreanTokenizer tag map.
• Fix issue #10551: Add automatic vector deduplication for init vectors.

🚀 Notes about upgrading from v3.2

• To see the speed improvements for the Tagger architecture, edit your configs to switch from spacy.Tagger.v1 to spacy.Tagger.v2 and then run init fill-config.
• Span comparisons involving ordering (<, <=, >, >=) now take all span attributes into account (start, end, label, and KB ID) so spans may be sorted in a slightly different order (#9956).
• Annotation on whitespace tokens is handled in the same way as annotation on non-whitespace tokens during training in order to allow custom whitespace annotation (#10189).
• Doc.from_docs now includes Doc.tensor by default and supports excludes with an exclude argument in the same format as Doc.to_bytes. The supported exclude fields are spans, tensor and user_data.

📖 Documentation and examples

• spaCy universe additions:
  • classy-classification: A Python library for classy few-shot and zero-shot classification within spaCy.
  • Concise Concepts: Concise Concepts uses few-shot NER based on word embedding similarity.
  • Crosslingual Coreference: Crosslingual coreference with an English coreference model plus crosslingual embeddings.
  • EDS-NLP: spaCy components to extract information from clinical notes written in French.
  • HuSpaCy: Industrial-strength Hungarian natural language processing.
  • Klayers: spaCy as a AWS Lambda Layer.
  • Named Entity Recognition (NER) using spaCy (video).
  • Scrubadub: Remove personally identifiable information from text using spaCy.
  • spacy-setfit-textcat: Experiments with SetFit & Few-Shot Classification.
  • tmtoolkit: Text mining and topic modeling toolkit.

👥 Contributors

@aajanki, @adrianeboyd, @apjanco, @bdura, @BramVanroy, @danieldk, @danmysak, @davidberenstein1957, @DuyguA, @fonfonx, @gremur, @HaakonME, @harmbuisman, @honnibal, @ines, @internaut, @jfainberg, @jnphilipp, @jsnfly, @kadarakos, @koaning, @ljvmiranda921, @martinjack, @mgrojo, @nrodnova, @ofirnk, @orglce, @pepemedigu, @philipvollet, @pmbaumgartner, @polm, @richardpaulhudson, @ryndaniels, @SamEdwardes, @Schero1994, @shadeMe, @single-fingal, @svlandeg, @thebugcreator, @thomashacker, @umaxfun, @y961996

This bug fix release is primarily to address Pydantic incompatibility with typing_extensions>=4.6.0.

✨ New features and improvements

• Huge speed improvements for spancat, in particular on GPU (~10x-30x faster) (#12577).

🔴 Bug fixes

• Add typing_extensions requirement due to Pydantic incompatibility with typing_extensions>=4.6.0.
• Remove #egg from download URLs due to future deprecation in pip.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @kadarakos, @svlandeg

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #10573: Remove Click pin following Typer updates.
• #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @polm, @svlandeg

🔴 Bug fixes

• Fix issue #10564: Restrict supported Click versions as a workaround for incompatibilities between Click v8.1.0 and Typer v0.4.0.

👥 Contributors

@adrianeboyd, @honnibal, @ines

🔴 Bug fixes

• Fix issue #10324: Fix Tok2Vec for empty batches.

👥 Contributors

@adrianeboyd, @honnibal, @ines

✨ New features and improvements

• Improved parser and ner speeds on long documents (see technical details in #10019).
• Support for spancat components in debug data.
• Support for ENT_IOB as a Matcher token pattern key.
• Extended and improved types for many classes.

🔴 Bug fixes

• Fix issue #9735: Make floret murmurhash endian-neutral.
• Fix issue #9738: Support string IOB values for ENT_IOB.
• Fix issue #9746: Updates to avoid "dictionary size changed during iteration" runtime errors.
• Fix issue #9960: Warn about entities that cross sentence boundaries in debug data.
• Fix issue #9979: Fix type for Lexeme.rank.
• Fix issue #10026: Check for 0-size assets in spacy project.
• Fix issue #10051: Consistently return scalars from similarity methods.
• Fix issue #10052: Fix spaces in Doc.from_docs() for empty docs.
• Fix issue #10079: Fix label detection in debug data for components with custom names.
• Fix issue #10109: Add types to Underscore and DependencyMatcher and improve types in Language, Matcher and PhraseMatcher.
• Fix issue #10130: Fix Tokenizer.explain when infixes appear as prefixes.
• Fix issue #10143: Use simple suggester in spancat initialization.
• Fix issue #10164: Support IS_SENT_END in Doc.has_annotation.
• Fix issue #10192: Detect invalid package names in spacy package.
• Fix issue #10223: Support mixed case in package names.
• Fix issue #10234: Fix type in PhraseMatcher.

📖 Documentation and examples

• Various documentation updates.
• New spaCy version tags in spaCy universe.
• New Dockerfile for repeatable website builds and easier local development.
• New additions to spaCy universe:
  • Augmenty: a text augmentation library
  • Healthsea: an end-to-end spaCy pipeline for exploring health supplement effects
  • spacy-wrap: wrap fine-tuned transformers in spaCy pipelines
  • spacypdfreader: easy PDF to text to spaCy text extraction
  • textnets: text analysis with networks

👥 Contributors

@adrianeboyd, @antonpibm, @ColleterVi, @danieldk, @DuyguA, @ezorita, @HaakonME, @honnibal, @ines, @jboynyc, @KennethEnevoldsen, @ljvmiranda921, @mrshu, @pmbaumgartner, @polm, @ramonziai, @richardpaulhudson, @ryndaniels, @svlandeg, @thiippal, @thomashacker, @yoavxyoav

✨ New features and improvements

• NEW: doc_cleaner component for removing doc.tensor,doc._._trf_data or other Doc attributes at the end of the pipeline to reduce size of output docs.
• NEW: ENT_ID and ENT_KB_ID to Matcher pattern attributes.
• Support kb_id for entities in displaCy from Doc input.
• Add Span.sents property for spans spanning over more than one sentence.
• Add EntityRuler.remove to remove patterns by id.
• Make the Tagger neg_prefix configurable.
• Use Language.pipe in Language.evaluate for more efficient processing.
• Test suite updates: move regression tests into core test modules with pytest markers for issue numbers, extend tests for languages with alpha support.

🔴 Bug fixes

• Fix issue #9638: Make JsonlCorpus path optional again.
• Fix issue #9654: Fix spancat for empty docs and zero suggestions.
• Fix issue #9658: Improve error message for incorrect .jsonl paths in EntityRuler.
• Fix issue #9674: Fix language-specific factory handling in package CLI.
• Fix issue #9694: Convert labels to strings for README in package CLI.
• Fix issue #9697: Exclude strings from source vector checks.
• Fix issue #9701: Allow Scorer.score_spans to handle predicted docs with missing annotation.
• Fix issue #9722: Initialize parser from reference parse rather than aligned example.
• Fix issue #9764: Set annotations more efficiently in tagger and morphologizer.

📖 Documentation and examples

• Various documentation updates: init_tok2vec after pretraining, batch contract for listeners.
• New additions to the spaCy universe:
  • eng-spacysentiment: Sentiment analysis for English.
  • Applied Language Technology course: NLP for newcomers using spaCy and Stanza.

👥 Contributors

@adrianeboyd, @danieldk, @DuyguA, @honnibal, @ines, @ljvmiranda921, @narayanacharya6, @nrodnova, @Pantalaymon, @polm, @richardpaulhudson, @svlandeg, @thiippal, @Vishnunkumar

✨ New features and improvements

• NEW: Registered scoring functions for each component in the config.
• NEW: nlp() and nlp.pipe() accept Doc input, which simplifies setting custom tokenization or extensions before processing.
• NEW: Support for floret vectors, which combine fastText subwords with Bloom embeddings for compact, full-coverage vectors.
• overwrite config settings for entity_linker, morphologizer, tagger, sentencizer and senter.
• extend config setting for morphologizer for whether existing feature types are preserved.
• Support for a wider range of language codes in spacy.blank() including IETF language tags, for example fra for French and zh-Hans for Chinese.
• New package spacy-loggers for additional loggers.
• New Irish lemmatizer.
• New Portuguese noun chunks and updated Spanish noun chunks.
• Language updates for Bulgarian, Catalan, Sinhala, Tagalog, Tigrinya and Vietnamese.
• Japanese reading and inflection from sudachipy are annotated as Token.morph features.
• Additional morph_micro_p/r/f scores for morphological features from Scorer.score_morph_per_feat().
• LIKE_URL attribute includes the tokenizer URL pattern.
• --n-save-epoch option for spacy pretrain.
• Trained pipelines:
  • New transformer pipeline for Japanese ja_core_news_trf, thanks to @hiroshi-matsuda-rit and the spaCy Japanese community!
  • Updates for Catalan data, tokenizer and lemmatizer, thanks to @cayorodriguez, Carme Armentano and @TeMU-BSC!
  • Transformer pipelines are trained using spacy-transformers v1.1, with improved IO and more options for model config and output.
  • Universal Dependencies corpora updated to v2.8.
  • Trailing space added as a tok2vec feature, improving the performance for many components, especially fine-grained tagging and sentence segmentation.
  • English attribute ruler patterns updated to improve Token.pos and Token.morph.

For more details, see the New in v3.2 usage guide.

🔴 Bug fixes

• Fix issue #8972: Fix pickling for Japanese, Korean and Vietnamese tokenizers.
• Fix issue #9032: Retain alignment between doc and context for Language.pipe(as_tuples=True) for multiprocessing with custom error handlers.
• Fix issue #9136: Ignore prefixes when applying suffix patterns in Tokenizer.
• Fix issue #9584: Use metaclass to subclass errors to allow better pickling.

⚠️ Backwards incompatibilities

• In the Tokenizer, prefixes are now removed before suffix matches are applied, which may lead to minor differences in the output. In particular, the default tokenization of °[cfk]. is now ° c . instead of ° c. for most languages.
• The tokenizer classes ChineseTokenizer, JapaneseTokenizer, KoreanTokenizer, ThaiTokenizer and VietnameseTokenizer require Vocab rather than Language in __init__.
• In DocBin, user data is now always serialized according to the store_user_data option, see #9190.

📖 Documentation and examples

• Demo projects for floret vectors:
  • pipelines/floret_vectors_demo: basic floret vector training and importing.
  • pipelines/floret_fi_core_demo: Finnish UD+NER vector and pipeline training, comparing standard vs. floret vectors.
  • pipelines/floret_ko_ud_demo: Korean UD vector and pipeline training, comparing standard vs. floret vectors.

👥 Contributors

@adrianeboyd, @Avi197, @baxtree, @BramVanroy, @cayorodriguez, @DuyguA, @fgaim, @honnibal, @ines, @Jette16, @jimregan, @polm, @rspeer, @rumeshmadhusanka, @svlandeg, @syrull, @thomashacker

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #10573: Remove Click pin following Typer updates.
• #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @polm, @svlandeg

🔴 Bug fixes

• Fix issue #10564: Restrict supported Click versions as a workaround for incompatibilities between Click v8.1.0 and Typer v0.4.0.

👥 Contributors

@adrianeboyd, @honnibal, @ines

🔴 Bug fixes

• Fix issue #9593: Use metaclass to subclass errors for easier pickling.
• Fix issue #9654: Fix spancat for empty docs and zero suggestions.
• Fix issue #9979: Fix type of Lexeme.rank.
• Fix issue #10324: Fix Tok2Vec for empty batches.

👥 Contributors

@adrianeboyd, @BramVanroy, @brucewlee, @danieldk, @honnibal, @ines, @ljvmiranda921, @polm, @svlandeg, @vgautam, @xxyzz

✨ New features and improvements

• NEW: Binary wheels for Python 3.10.
• NEW: Improve performance on Apple M1 with AppleOps: pip install spacy[apple].
• GPU profiling with spacy.models_with_nvtx_range.v1.
• Full mypy integration in the CI and many type fixes across the code base.
• Added custom Protocol classes in ty.py to define behavior of pipeline components.
• Support for entity linking visualization in displacy.
• Allow overriding vars in spacy project assets .
• Standalone train function to run the training from Python scripts just like the spacy train CLI.
• Support for spacy-transformers>=1.1.0 with improved IO.
• Support for thinc>=8.0.11 with improved gradient clipping.

🔴 Bug fixes

• Fix issue #5507: Improve UX for multiprocessing on GPU.
• Fix issue #9137: Fix serialization for KnowledgeBase.set_entities.
• Fix issue #9244: Fix vectors for 0-length spans.
• Fix issue #9247: Improve UX for the DocBin constructor.
• Fix Issue #9254: Allow unicode in a spacy project title.
• Fix issue #9263: Make added patterns consistent in the DependencyMatcher.
• Fix issue #9305: Restore tokenization timing during evaluation.
• Fix issue #9335: Sync vocab in vectors and sourced components.
• Fix issue #9387: Ensure lemmas are consistent for Catalan, Dutch, French, Russian and Ukrainian.
• Fix issue #9404: Create consistent default textcat and textcat_multilabel configurations.
• Fix issue #9437: Improve UX around Doc object creation.
• Fix issue #9465: Fix minor issues with convert CLI.
• Fix issue #9500: Include .pyi files in the distributed package.

📖 Documentation and examples

• Various updates to the documentation.
• New additions to the spaCy universe:
  • deplacy: CUI-based dependency visualizer
  • ipymarkup: Visualizations for NER and syntax trees
  • PhruzzMatcher: Find fuzzy matches
  • spacy-huggingface-hub: Push spaCy pipelines to the Hugging Face Hub
  • spaCyOpenTapioca: Entity Linking on Wikidata
  • spacy-clausie: Clause-based information extraction system
  • "Applied Natural Language Processing in the Enterprise": Book by Ankur A. Patel
  • "Introduction to spaCy 3": Free course by Dr. W.J.B. Mattingly

👥 Contributors

@adrianeboyd, @connorbrinton, @danieldk, @DuyguA, @honnibal, @ines, @Jette16, @ljvmiranda921, @mjvallone, @philipvollet, @polm, @rspeer, @ryndaniels, @shigapov, @svlandeg, @thomashacker

✨ New features and improvements

• The v3 of WandbLogger now supports optional run_name and entity parameters.
• Improved UX when providing invalid pos values for a Doc or Token.

🔴 Bug fixes

• Fix issue #9001: Pass alignments to Matcher callbacks.
• Fix issue #9009: Include component factories in third-party dependencies resolver.
• Fix issue #9012: Correct type of config in create_pipe.
• Fix issue #9014: Allow typer 0.4 to provide support for both Click 7 and Click 8.
• Fix issue #9033: Fix verbs list for French tokenizer exceptions.
• Fix issue #9059: Pass overrides to subcommands in spacy project workflows.
• Fix issue #9074: Improve UX around repo and path arguments in spacy project.
• Fix issue #9084: Fix inference of epoch_resume in spacy pretrain.
• Fix issue #9163: Handle spacy-legacy in spacy package dependency detection.
• Fix issue #9211: Include only runtime-relevant dependencies in spacy package.

📖 Documentation and examples

• Various updates to the documentation.
• Few additions and updates to the spaCy universe.
• Extended the developer documentation with information about the listener pattern, the StringStore and the Vocab.

👥 Contributors

@adrianeboyd, @davidefiocco, @davidstrouk, @filipematos95, @honnibal, @ines, @j-frei, @Joozty, @kwhumphreys, @mjhajharia, @mylibrar, @polm, @rspeer, @shigapov, @svlandeg, @thomashacker

✨ New features and improvements

• NEW: Provide scores for the SpanCategorizer predictions.
• NEW: Broader compatibility with type checkers thanks to .pyi stub files.
• NEW: Auto-detect package dependencies in spacy package.
• New INTERSECTS operator for the Matcher.
• More debugging info for spacy project push and pull commands.
• Allow passing in a precomputed array for speeding up multiple Span.as_doc calls.
• The default da transformer is now the same as the one from the trained pipelines (Maltehb/danish-bert-botxo).

🔴 Bug fixes

• Fix issue #8767: Fix offsets of empty and out-of-bounds spans.
• Fix issue #8774: Ensure debug data runs correctly with a custom tokenizer.
• Fix issue #8784: Fix incorrect ISSUBSET and ISSUPERSET in schema and docs.
• Fix issue #8796: Respect the no_skip value for spacy project run.
• Fix issue #8810: Make ConsoleLogger flush after each logging line.
• Fix issue #8819: Pass exclude when serializing the vocab.
• Fix issue #8830: Avoid adding sourced vectors hashes if not necessary.
• Fix issue #8970: Fix allow_overlap default for span categorizer scoring.
• Fix issue #8982: Add glossary entry for _SP.
• Fix issue #9007: Fix span categorizer training on nested entities.

📖 Documentation and examples

• New developer documentation covering spaCy's internals and code conventions.
• Added a documentation section on preparing training data in spaCy's binary format.
• Updated some error/log messages to be more informative.
• Various updates to the documentation.
• A few new additions to the spaCy universe.

👥 Contributors

@adrianeboyd, @bbieniek, @DuyguA, @ezorita, @HLasse, @honnibal, @ines, @kabirkhan, @kevinlu1248, @ldorigo, @Ledenel, @nsorros, @polm, @svlandeg, @swfarnsworth, @themrmax, @thomashacker

✨ New features and improvements

• Alpha tokenization support for Ancient Greek.
• Implementation of a noun_chunk iterator for Dutch.
• Support for black & flake8 as pre-commit hooks.
• New spacy.ngram_range_suggester.v1 for suggesting a range of n-gram sizes for the spancat component.

🔴 Bug fixes

• Fix issue #8638: Fix Azerbaijani initialization.
• Fix issue #8639: Use 0-vector for OOV lexemes.
• Fix issue #8640: Update lexeme ranks for loaded vectors.
• Fix issue #8651: Fix ru and uk multiprocessing (with spawn).
• Fix issue #8663: Preserve existing meta information with spacy package.
• Fix issue #8718: Ensure that replace_pipe takes disabled components into account.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @jmyerston, @julien-talkair, @KennethEnevoldsen, @mariosasko, @mylibrar, @polm, @rynoV, @svlandeg, @thomashacker, @yohasebe

✨ New features and improvements

• NEW: Trained pipelines for Catalan and a new transformer-based pipeline for Danish.
• NEW: Experimental SpanCategorizer component for labeling arbitrary and potentially overlapping spans of text.
• NEW: Use predicted annotations during training via the [training.annotating_components] config setting.
• Alpha tokenization support for Azerbaijani.
• Part-of-speech tag-based lemmatizers for Catalan and Italian.
• The TextCatCNN and TextCatBOW architectures are now resizable.
• Support updating the EntityRecognizer with known incorrect span annotations.
• Auto-generate a pretty README.md based on the meta in spacy package.

For more details, see the New in v3.1 usage guide.

📦 New trained pipelines

⚠️ Upgrading from v3.0

• Due to the use of configs with extensive versioning, v3.0 pipelines should be compatible with v3.1, however you may see slight differences in performance. Test your v3.0 pipeline with v3.1 against your test suite and if the performance is identical, extend the spacy_version in your model package meta to ">=3.0.0,<3.2.0". If you run into degraded performance, retrain your pipeline with v3.1.
• Use spacy init fill-config to update a v3.0 config for v3.1.
• When sourcing a pipeline component that requires static vectors, it is now required to include the source model's vectors in [initialize.vectors].
• Logger warnings have been converted to Python warnings. Use warnings.filterwarnings or the new helper method spacy.errors.filter_warning(action, error_msg='') to manage warnings.

For more information, see Notes on upgrading from v3.0.

🔴 Bug fixes

• Fix issue #7036: Use a context manager when reading model.
• Fix issue #7629: Fix scoring normalization.
• Fix issue #7799: Ensure spacy ray command works.
• Fix issue #7807: Show warning if entity ruler runs without patterns.
• Fix issue #7886: Fix unknown tokens percentage in debug data.
• Fix issue #7930: Make EntityLinker robust for nO=None.
• Fix issue #7925: Skip vector ngram backoff if minn is not set.
• Fix issue #7973: Fix debug model for transformers.
• Fix issue #7988: Preserve existing ENT_KB_ID in ner annotation.
• Fix issue #8004: Handle errors while multiprocessing.
• Fix issue #8009: Fix Doc.from_docs() for all empty docs.
• Fix issue #8012: Fix ensemble textcat with listener.
• Fix issue #8054: Add ENT_ID and NORM to DocBin strings.
• Fix issue #8055: Handle partial entities in Span.as_doc.
• Fix issue #8062: Make all Span attrs writable.
• Fix issue #8066: Update debug data for textcat.
• Fix issue #8069: Custom warning if DocBin is too large.
• Fix issue #8099: Update Vietnamese tokenizer.
• Fix issue #8113: Support to/from_bytes for KnowledgeBase and EntityLinker.
• Fix issue #8116: Fix offsets in Span.get_lca_matrix.
• Fix issue #8132: Remove unsupported attrs from attrs.IDS.
• Fix issue #8158: Ensure tolerance is passed on in spacy.batch_by_words.v1.
• Fix issue #8169: Fix bug from EntityRuler: ent_ids returns None for phrases.
• Fix issue #8208: Address missing config overrides post load of models.
• Fix issue #8212: Add all symbols in Unicode Currency Symbols to currency characters.
• Fix issue #8216: Don't add duplicate patterns in EntityRuler.
• Fix issue #8265: Address mypy errors.
• Fix issue #8335: Raise error if deps not provided with heads in Doc.
• Fix issue #8368: Preserve whitespace in Span.lemma_.
• Fix issue #8388: Don't clobber vectors when loading components from source models.
• Fix issue #8421: Fix non-deterministic deduplication in Greek lemmatizer.
• Fix issue #8426: Fix setting empty entities in Example.from_dict.
• Fix issue #8441: Add correct types for Language.pipe return values.
• Fix issue #8487: Fix span offsets and keys in Doc.from_docs.
• Fix issue #8559: Fix vectors check for sourced components.
• Fix issue #8584: Raise an error for textcat with <2 labels.

👥 Contributors

@aajanki, @adrianeboyd, @bodak, @bryant1410, @dhruvrnaik, @explosion-bot, @fhopp, @frascuchon, @graue70, @gtoffoli, @honnibal, @ines, @jacopofar, @jenojp, @jhroy, @jklaise, @juliensalinas, @kevinlu1248, @ldorigo, @mathcass, @meghanabhange, @michael-k, @narayanacharya6, @NirantK, @nsorros, @polm, @sevdimali, @svlandeg, @themrmax, @xadrianzetx, @yohasebe, @ZeeD

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @polm, @svlandeg