spacy

✨ New features and improvements

• New assemble CLI command for assembling a pipeline from a config without training.
• Add support for match alignments in the Matcher to align matched tokens with matcher patterns.
• Add support for training from streamed corpora.
• Add support for W&B data and model checkpoint logging and versioning in spacy.WandbLogger.v2.
• Extend Scorer.score_spans to support overlapping and unlabeled spans.
• Update debug data for new v3 components.
• Improve language data for Italian.
• Various improvements to error handling and UX.

🔴 Bug fixes

• Fix issue #7408: Add vocab kwarg to spacy.load.
• Fix issue #7419: Exclude user hooks in displacy conversion.
• Fix issue #7421: Update --code usage in CLI commands.
• Fix issue #7424: Preserve sent starts on retokenization without parse.
• Fix issue #7440: Fix pymorphy2 lookup lemmatizer.
• Fix issue #7471: Improve warnings related to listening components.
• Fix issue #7488: Fix upstream check in pretraining.
• Fix issue #7489: Support callbacks entry points.
• Fix issue #7497: Merge doc.spans in Doc.from_docs().
• Fix issue #7528: Preserve user data for DependencyMatcher on spans.
• Fix issue #7557: Fix __add__ method for PRFScore.
• Fix issue #7574: Fix conversion of custom extension data in Span.as_doc and Doc.from_docs.
• Fix issue #7620: Fix replace_listeners in configs.
• Fix issue #7626: Fix vectors data on GPU.
• Fix issue #7630: Update NEL for entities crossing sentence boundaries.
• Fix issue #7631: Fix parser sourcing in NER converter.
• Fix issue #7642: Fix handling of hyphen string value in config files.
• Fix issue #7655: Fix sent starts when converting from v2 JSON training format.
• Fix issue #7674: Fix handling of unknown tokens in StaticVectors.
• Fix issue #7690: Fix pickling of Lemmatizer.
• Fix issue #7749: Update Tokenizer.explain for special cases in v3.
• Fix issue #7755: Fix config parsing of ints/strings.
• Fix issue #7836: Fix tokenizer cache flushing.
• Fix issue #7847: Fix handling of boolean values in Example.from_dict for sent starts.

📖 Documentation and examples

• Add documentation for legacy functions and architectures.
• Add documentation for pretrained pipeline design.
• Add more details about pipe and multiprocessing.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @alvaroabascar, @armsp, @AyushExel, @BramVanroy, @broaddeep, @bryant1410, @bsweileh, @dpalmasan, @Findus23, @graue70, @jaidevd, @koaning, @langdonholmes, @m0canu1, @meghanabhange, @paoloq, @plison, @richardpaulhudson, @SamEdwardes, @Stannislav for the pull requests and contributions!

🔴 Bug fixes

• Fix related to issue #7075: Update thinc requirement for Jupyter notebook GPU warning

✨ New features and improvements

• Allow sourcing disabled components in config.
• Support Doc.spans in Example.from_dict.
• Improve transformer recommendations in quickstart widget and init config.
• Improve language data for Bulgarian.
• Various improvements to error handling and UX.

🔴 Bug fixes

• Fix issue #6952, #7285, #7289: Make tok2vec pretraining and pretrain command work as expected again.
• Fix issue #7062: Only evaluate named entities for NEL if there is a corresponding gold span.
• Fix issue #7065: Correctly handle sentence boundaries in Span.sent.
• Fix issue #7071: Fix conll converter option.
• Fix issue #7100: Re-add n_sents to entity linker and fix config handling and I/O.
• Fix issue #7122: Fix displaCy output in evaluate CLI.
• Fix issue #7127: Fix initialization of UkrainianLemmatizer.
• Fix issue #7176: Re-refactor Sentencizer to use Pipe API.
• Fix issue #7182: Allow SpanGroup import from spacy.tokens.
• Fix issue #7204: Adjust Cython compilation for setups with custom include paths.
• Fix issue #7222: Correct YAML formatting in quickstart recommendations for bg and bn.
• Fix issue #7225: Fix spans weakref in Doc.copy.
• Fix issue #7237: Fix is_cython_func for additional imported code.
• Fix issue #7250: Fix patience for identical scores.
• Fix issue #7329: Make spacy.orth_variants.v1 and spacy.lower_case.v1 augmenters work as expected.
• Fix issue #7352: Sort EntityRuler.labels alphabetically.

📖 Documentation and examples

• Add documentation for textcat_multilabel component.
• Extend documentation for Vocab.get_noun_chunks.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @MartinoMensio, @SergeyShk, @R1j1t, @palandlom, @dardoria, @Tocic, @clippered, @graue70, @koaning and @jankrepl for the pull requests and contributions!

🔴 Bug fixes

• Fix issue #7035, #7056: Fix parser transition bug that could lead to incorrect sentence fragments.
• Fix issue #7055: Preserve sourced components in init fill-config.

📖 Documentation and examples

• Update spaCy Universe.

👥 Contributors

Thanks @MartinoMensio for the pull request!

✨ New features and improvements

• NEW: Base support for Setswana.
• The PhraseMatcher can now also be run on Span objects.
• Support CLI overrides and environment variables in project.yml: a section env defines environment variable names that can be used in commands. The project run command now also supports CLI overrides, e.g. --vars.batch_size 128.
• Reduce memory load when reading all vectors from file during initialization.
• Update recommended transformers in training quickstart and init config CLI.

🔴 Bug fixes

• Fix issue #6826: Ensure the loss value is cast to a float.
• Fix issue #6891: Include noun_chunks when pickling Vocab.
• Fix issue #6908: Fix expected type for textcat labels.
• Fix issue #6924: Correctly pass vocab forward in spacy.blank.
• Fix issue #6950: Allow pickling Tok2Vec with listeners .
• Fix issue #6983: Ensure is_same_func works correctly for classes in component decorator.
• Fix issue #7019: Correctly handle non-float/int values in spacy evaluate printer.
• Fix issue #7029: Fix listener architecture with empty Doc in batch.

📖 Documentation and examples

• Improve installation instructions.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @peter-exos, @KoichiYasuoka, @tarskiandhutch, @reneoctavio, @melonwater211, @mapmeld and @Shumie82 for the pull requests and contributions.

🔴 Bug fixes

• Fix issue #6883: Fix bug in transformer training for Cannot get dimension 'nO' for model 'transformer': value unset.

📣 NEW: Want to make the transition from spaCy v2 to spaCy v3 as smooth as possible for you and your organization? We're now offering commercial migration support for your spaCy pipelines! We've put a lot of work into making it easy to upgrade your existing code and training workflows – but custom projects may always need some custom work, especially when it comes to taking advantage of the new capabilities. Details & application →

🚀 Quickstart

For the smoothest updating process, we recommend starting with a fresh virtual environment.

pip install -U spacy

• New in v3.0: New features, backwards incompatibilities and migration guide.
• Installation Quickstart: Install the new version, pipelines and add-ons for your specific setup.
• Training Quickstart: Generate a training config for your specific use case.
• Benchmarks: Results and accuracy comparisons.
• Projects & Project Templates: Get started by cloning a project template.

✨ New features and improvements

• Transformer-based pipelines with support for multi-task learning.
• Retrained model families for 18+ languages and 58 trained pipelines in total, including 5 transformer-based pipelines.
• Retrained pipelines for all supported languages, plus new core pipelines for Macedonian and Russian. Thanks to @borijang, @buriy and @kuk for the contributions!
• New training workflow and config system.
• Implement custom models using any machine learning framework, including PyTorch, TensorFlow and MXNet.
• spaCy Projects for managing end-to-end multi-step workflows from preprocessing to model deployment.
• Integrations with Data Version Control (DVC), Streamlit, Weights & Biases, Ray and more.
• Parallel training and distributed computing with Ray.
• New built-in pipeline components: SentenceRecognizer, Morphologizer, Lemmatizer, AttributeRuler and Transformer.
• New and improved pipeline component API and decorators for custom components.
• Source trained components from other pipelines in your training config.
• Pre-built and more efficient binary wheels for all trained pipeline packages.
• DependencyMatcher for matching patterns within the dependency parse using Semgrex operators.
• Support for greedy patterns in Matcher.
• New data structure SpanGroup for efficiently storing collections of potentially overlapping spans via the Doc.spans.
• Type hints and type-based data validation for custom registered functions.
• Various new methods, attributes and commands.

📺 Video introductions & tutorials

spaCy v3: State-of-the-art NLP from Prototype to Production	spaCy v3: Design concepts explained (behind the scenes)	spaCy v3: Custom trainable relation extraction component
<img src="https://user-images.githubusercontent.com/13643239/106374435-ddeb8f00-6383-11eb-99f4-c2f2570d5bb7.png" width="300" />	<img src="https://user-images.githubusercontent.com/13643239/106374436-df1cbc00-6383-11eb-905a-d5db2ca598d3.png" width="300" />	<img src="https://user-images.githubusercontent.com/13643239/106374432-d926db00-6383-11eb-8b3c-5173dd5cec7d.png" width="300" />

📦 Trained pipelines (58)

To download a trained pipeline, you can use the spacy download command. See the training documentation for details on how to train your own pipelines on your data.

Name	Language	POS	TAG	LAS	UAS	NER	Sent	Size
da_core_news_lg v3.0.0	Danish	0.97	0.97	0.78	0.82	0.82	0.88	547 MB	📖
da_core_news_md v3.0.0	Danish	0.96	0.96	0.78	0.82	0.81	0.86	47 MB	📖
da_core_news_sm v3.0.0	Danish	0.95	0.95	0.76	0.81	0.72	0.86	17 MB	📖
de_core_news_lg v3.0.0	German	0.98	0.98	0.91	0.93	0.85	0.95	546 MB	📖
de_core_news_md v3.0.0	German	0.98	0.98	0.91	0.93	0.84	0.95	47 MB	📖
de_core_news_sm v3.0.0	German	0.98	0.97	0.90	0.92	0.82	0.94	18 MB	📖
de_dep_news_trf v3.0.0	German	0.99	0.99	0.95	0.96	n/a	0.98	393 MB	📖
el_core_news_lg v3.0.0	Greek	0.97	0.94	0.85	0.88	0.80	1.00	544 MB	📖
el_core_news_md v3.0.0	Greek	0.96	0.93	0.84	0.87	0.79	1.00	42 MB	📖
el_core_news_sm v3.0.0	Greek	0.94	0.91	0.81	0.85	0.72	1.00	12 MB	📖
en_core_web_lg v3.0.0	English	n/a	0.97	0.90	0.92	0.86	0.89	742 MB	📖
en_core_web_md v3.0.0	English	n/a	0.97	0.90	0.92	0.85	0.89	44 MB	📖
en_core_web_sm v3.0.0	English	n/a	0.97	0.90	0.92	0.84	0.89	13 MB	📖
en_core_web_trf v3.0.0	English	n/a	0.98	0.94	0.95	0.90	0.89	438 MB	📖
es_core_news_lg v3.0.0	Spanish	0.99	0.98	0.88	0.91	0.90	1.00	547 MB	📖
es_core_news_md v3.0.0	Spanish	0.99	0.98	0.88	0.91	0.90	1.00	46 MB	📖
es_core_news_sm v3.0.0	Spanish	0.98	0.97	0.87	0.90	0.89	1.00	17 MB	📖
es_dep_news_trf v3.0.0	Spanish	0.99	0.98	0.93	0.95	n/a	0.97	395 MB	📖
fr_core_news_lg v3.0.0	French	0.98	0.95	0.86	0.90	0.82	0.88	546 MB	📖
fr_core_news_md v3.0.0	French	0.97	0.94	0.85	0.89	0.81	0.87	45 MB	📖
fr_core_news_sm v3.0.0	French	0.96	0.93	0.84	0.88	0.79	0.85	16 MB	📖
fr_dep_news_trf v3.0.0	French	0.99	0.96	0.92	0.94	n/a	0.94	381 MB	📖
it_core_news_lg v3.0.0	Italian	0.98	0.97	0.88	0.91	0.89	0.97	545 MB	📖
it_core_news_md v3.0.0	Italian	0.97	0.97	0.88	0.91	0.87	0.97	44 MB	📖
it_core_news_sm v3.0.0	Italian	0.97	0.97	0.86	0.90	0.85	0.97	16 MB	📖
ja_core_news_lg v3.0.0	Japanese	0.96	0.97	0.90	0.92	0.72	0.98	531 MB	📖
ja_core_news_md v3.0.0	Japanese	0.96	0.97	0.90	0.92	0.72	0.99	41 MB	📖
ja_core_news_sm v3.0.0	Japanese	0.96	0.97	0.90	0.92	0.64	0.99	12 MB	📖
lt_core_news_lg v3.0.0	Lithuanian	0.96	0.89	0.68	0.75	0.80	0.82	545 MB	📖
lt_core_news_md v3.0.0	Lithuanian	0.95	0.86	0.67	0.74	0.79	0.83	44 MB	📖
lt_core_news_sm v3.0.0	Lithuanian	0.91	0.82	0.59	0.68	0.74	0.79	15 MB	📖
mk_core_news_lg v3.0.0	Macedonian	0.93	n/a	0.51	0.68	0.76	0.73	312 MB	📖
mk_core_news_md v3.0.0	Macedonian	0.93	n/a	0.51	0.67	0.75	0.73	44 MB	📖
mk_core_news_sm v3.0.0	Macedonian	0.92	n/a	0.47	0.62	0.70	0.62	18 MB	📖
nb_core_news_lg v3.0.0	Norwegian	0.97	0.97	0.87	0.89	0.85	0.94	547 MB	📖
nb_core_news_md v3.0.0	Norwegian	0.97	0.97	0.87	0.90	0.85	0.93	44 MB	📖
nb_core_news_sm v3.0.0	Norwegian	0.97	0.97	0.85	0.88	0.77	0.93	15 MB	📖
nl_core_news_lg v3.0.0	Dutch	0.96	0.95	0.82	0.87	0.77	0.87	546 MB	📖
nl_core_news_md v3.0.0	Dutch	0.96	0.95	0.82	0.87	0.74	0.87	45 MB	📖
nl_core_news_sm v3.0.0	Dutch	0.95	0.93	0.80	0.85	0.72	0.86	16 MB	📖
pl_core_news_lg v3.0.0	Polish	0.97	0.98	0.84	0.89	0.85	0.99	584 MB	📖
pl_core_news_md v3.0.0	Polish	0.97	0.98	0.84	0.89	0.84	0.98	84 MB	📖
pl_core_news_sm v3.0.0	Polish	0.95	0.98	0.79	0.86	0.80	0.98	55 MB	📖
pt_core_news_lg v3.0.0	Portuguese	0.97	0.90	0.86	0.90	0.91	0.95	551 MB	📖
pt_core_news_md v3.0.0	Portuguese	0.97	0.90	0.86	0.90	0.90	0.95	49 MB	📖
pt_core_news_sm v3.0.0	Portuguese	0.97	0.89	0.85	0.89	0.88	0.92	21 MB	📖
ro_core_news_lg v3.0.0	Romanian	0.96	0.97	0.84	0.89	0.77	0.96	546 MB	📖
ro_core_news_md v3.0.0	Romanian	0.96	0.97	0.85	0.89	0.76	0.96	44 MB	📖
ro_core_news_sm v3.0.0	Romanian	0.96	0.96	0.82	0.87	0.72	0.97	15 MB	📖
ru_core_news_lg v3.0.0	Russian	0.99	0.99	0.95	0.96	0.95	1.00	491 MB	📖
ru_core_news_md v3.0.0	Russian	0.99	0.99	0.95	0.96	0.94	1.00	41 MB	📖
ru_core_news_sm v3.0.0	Russian	0.99	0.99	0.95	0.96	0.95	1.00	16 MB	📖
xx_ent_wiki_sm v3.0.0	MultiLanguage	n/a	n/a	n/a	n/a	0.82	n/a	14 MB	📖
xx_sent_ud_sm v3.0.0	MultiLanguage	n/a	n/a	n/a	n/a	n/a	0.86	10 MB	📖
zh_core_web_lg v3.0.0	Chinese	n/a	0.90	0.66	0.71	0.71	0.75	577 MB	📖
zh_core_web_md v3.0.0	Chinese	n/a	0.90	0.65	0.70	0.70	0.76	76 MB	📖
zh_core_web_sm v3.0.0	Chinese	n/a	0.90	0.64	0.70	0.69	0.75	47 MB	📖
zh_core_web_trf v3.0.0	Chinese	n/a	0.92	0.73	0.77	0.75	0.65	398 MB	📖

💬 TAG: Part-of-speech tags (fine-grained tags, i.e. Token.tag_) POS: Part-of-speech tags (coarse-grained tags, i.e. Token.pos_) UAS: Unlabelled dependencies (parser). LAS: Labelled dependencies (parser). NER: Named entities (F-score). Sent: Sentence segmentation. Size: Model file size (zipped archive).

⚠️ Backwards incompatibilities

For more info on how to migrate from spaCy v2.x, see the detailed migration guide.

API changes

• Pipeline package symlinks, the link command and shortcut names are now deprecated. There can be many different trained pipelines and not just one "English model", so you should always use the full package name like en_core_web_sm explicitly.
• A pipeline's meta.json is now only used to provide meta information like the package name, author, license and labels. It's not used to construct the processing pipeline anymore. This is all defined in the config.cfg, which also includes all settings used to train the pipeline.
• The train, pretrain and debug data commands now only take a config.cfg.
• Language.add_pipe now takes the string name of the component factory instead of the component function.
• Custom pipeline components now need to be decorated with the @Language.component or @Language.factory decorator.
• The Language.update, Language.evaluate and TrainablePipe.update methods now all take batches of Example objects instead of Doc and GoldParse objects, or raw text and a dictionary of annotations.
• The begin_training methods have been renamed to initialize and now take a function that returns a sequence of Example objects to initialize the model instead of a list of tuples.
• Matcher.add and PhraseMatcher.add now only accept a list of patterns as the second argument (instead of a variable number of arguments). The on_match callback becomes an optional keyword argument.
• The Doc flags like Doc.is_parsed or Doc.is_tagged have been replaced by Doc.has_annotation.
• The spacy.gold module has been renamed to spacy.training.
• The PRON_LEMMA symbol and -PRON- as an indicator for pronoun lemmas has been removed.
• The TAG_MAP and MORPH_RULES in the language data have been replaced by the more flexible AttributeRuler.
• The Lemmatizer is now a standalone pipeline component and doesn't provide lemmas by default or switch automatically between lookup and rule-based lemmas. You can now add it to your pipeline explicitly and set its mode on initialization.
• Various keyword arguments across functions and methods are now explicitly declared as keyword-only arguments. Those arguments are documented accordingly across the API reference.

Removed or renamed API

Removed	Replacement
Language.disable_pipes	Language.select_pipes, Language.disable_pipe, Language.enable_pipe
Language.begin_training, Pipe.begin_training, ...	Language.initialize, Pipe.initialize, ...
Doc.is_tagged, Doc.is_parsed, ...	Doc.has_annotation
GoldParse	Example
GoldCorpus	Corpus
KnowledgeBase.load_bulk, KnowledgeBase.dump	KnowledgeBase.from_disk, KnowledgeBase.to_disk
Matcher.pipe, PhraseMatcher.pipe	not needed
gold.offsets_from_biluo_tags, gold.spans_from_biluo_tags, gold.biluo_tags_from_offsets	training.biluo_tags_to_offsets, training.biluo_tags_to_spans, training.offsets_to_biluo_tags
spacy init-model	spacy init vectors
spacy debug-data	spacy debug data
spacy profile	spacy debug profile
spacy link, util.set_data_path, util.get_data_path	not needed, symlinks are deprecated

The following deprecated methods, attributes and arguments were removed in v3.0. Most of them have been deprecated for a while and many would previously raise errors. Many of them were also mostly internals. If you've been working with more recent versions of spaCy v2.x, it's unlikely that your code relied on them.

Removed	Replacement
Doc.tokens_from_list	Doc.__init__
Doc.merge, Span.merge	Doc.retokenize
Token.string, Span.string, Span.upper, Span.lower	Span.text, Token.text
Language.tagger, Language.parser, Language.entity	Language.get_pipe
keyword-arguments like vocab=False on to_disk, from_disk, to_bytes, from_bytes	exclude=["vocab"]
n_threads argument on Tokenizer, Matcher, PhraseMatcher	n_process
verbose argument on Language.evaluate	logging (DEBUG)
SentenceSegmenter hook, SimilarityHook	user hooks, Sentencizer, SentenceRecognizer

👥 Contributors

This release is brought to you by @honnibal, @ines, @svlandeg and @adrianeboyd. Thanks to @AMArostegui, @BramVanroy, @Cristianasp, @DeNeutoy, @DuyguA, @Jan-711, @KKsharma99, @KeshavG-lb, @KoichiYasuoka, @MartinoMensio, @Nuccy90, @PluieElectrique, @SamEdwardes, @Stannislav, @abchapman93, @alexcombessie, @alvaroabascar, @baranitharan2020, @bittlingmayer, @bjascob, @borijang, @borijang, @bratao, @bryant1410, @buriy, @chopeen, @danielvasic, @delzac, @dhruvrnaik, @erip, @florijanstamenkovic, @forest1988, @gandersen101, @garethsparks, @graue70, @guadiromero, @hertelm, @hiroshi-matsuda-rit, @holubvl3, @idoshr, @jabortell, @jbesomi, @jenojp, @jganseman, @jgutix, @jmargeta, @jumasheff, @kuk, @leyendecker, @lizhe2004, @lorenanda, @mahnerak, @mikeizbicki, @myavrum, @nipunsadvilkar, @oculusrepairo, @ophelielacroix, @rahul1990gupta, @rameshhpathak, @rasyidf, @revuel, @richardliaw, @robertsipek, @snsten, @solarmist, @tamuhey, @thomasbird, @tiangolo, @tilusnet, @timgates42, @vha14, @walterhenry, @wannaphong, @werew, @yosiasz and @zaibacu for the pull requests and contributions!

This release addresses future compatibility with NumPy v1.24+.

🔴 Bug fixes

• #11940: Update for compatibility with NumPy v1.24+ integer conversions.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @svlandeg

✨ New features and improvements

• Updates and binary wheels for Python 3.10 and 3.11.

👥 Contributors

@adrianeboyd, @honnibal, @ines

🔴 Bug fixes

• Fix issue #8286: Fix spacy download.

✨ New features and improvements

• Add base support for Amharic.
• Add noun chunk iterator for Danish.
• Updates to French, Portuguese and Romanian stop words.

🔴 Bug fixes

• Fix issue #6705: Fix deserialization of null token_match and url_match for the tokenizer.
• Fix issue #6712: Prevent overlapping noun chunks for Spanish.
• Fix issue #6745: Fix minibatch iterator when size iterator is finished.
• Fix issue #6759: Skip 0-length matches in the Matcher.
• Fix issue #6771: Support IS_SENT_START in the PhraseMatcher.
• Fix issue #6772: Fix Span.text for empty spans.
• Fix issue #6820: Improve Doc.char_span alignment_mode handling.
• Fix issue #6857: Remove --no-cache-dir when downloading models.
• Fix issue #8115: Fix offsets in Span.get_lca_matrix.

👥 Contributors

Thanks to @alexcombessie, @AMArostegui, @bryant1410, @Cristianasp, @garethsparks, @jenojp, @jganseman, @jumasheff, @lorenanda, @ophelielacroix, @thomasbird, @timgates42, @tupui and @yosiasz for the pull requests and contributions.

✨ New features and improvements

• Modify blis and numpy build dependencies to simplify source installations.
• Support cupy v8+ in combination with thinc v7.4.5.

🔴 Bug fixes

• Fix issue #6443: Only set NORM on token in retokenizer.
• Fix issue #6453: Add SPACY as a Matcher attribute.
• Fix issue #6512: Add nlp.max_length check to nlp.pipe through nlp.make_doc.
• Fix issue #6515: Add missing .pipe methods to Chinese, Japanese, Korean and Thai tokenizers.
• Fix issue #6518: Fix subsequent pipe detection in EntityRuler.
• Fix issue #6523: Remove non-working --use-chars from train CLI.

👥 Contributors

Thanks to @KoichiYasuoka for the pull requests and contributions.

🔴 Bug fixes

• Fix issue #6446: Restore cleanup_beam method.

📖 Documentation and examples

• Update rule-based matching docs

👥 Contributors

Thanks to @jabortell for the pull requests and contributions.

✨ New features and improvements

• NEW: Add alpha support for Macedonian and Sanskrit.
• Update language data for Croatian, Czech, English, Hebrew, Hindi, Indonesian, Swedish, Thai and Turkish.
• Add support for aarch64 and ppc64le on linux with binary packages available on conda-forge.

🔴 Bug fixes

• Fix issue #5610: Make sure sys.argv exists.
• Fix issue #5643: Add ent_id_ to strings serialized with Doc.
• Fix issue #5727: Clarify warning for misaligned BILUO tags.
• Fix issue #5768: Improve tag map initialization and updating.
• Fix issue #5794: Improve warnings around normalization tables.
• Fix issue #5796: Update invalid tag maps.
• Fix issue #5799: Remove hard-coded GPU ID from pretrain.
• Fix issue #5802: Mark Japanese documents as tagged.
• Fix issue #5823: Fix typo in unit tests.
• Fix issue #5838: Fix EntityRenderer to support break lines (after last entity).
• Fix issue #5843: Prefer earlier spans in EntityRuler.
• Fix issue #5849: Allow Doc.char_span to snap to token boundaries.
• Fix issue #5853: Fix span boundary handling in Spanish noun chunks.
• Fix issue #5861: Add Span index boundary checks.
• Fix issue #5904: Fix typos in comments.
• Fix issue #5910: Update default sentencizer characters for Armenian, Greek and Arabic.
• Fix issue #6014: Fix off-by-one error for best iteration calculation.
• Fix issue #6112: Fix overlapping German noun chunks.
• Fix issue #6148: Identify final Matcher pattern node by quantifier.
• Fix issue #6164: Reorder so tag map is replaced only if a custom file is provided.
• Fix issue #6218: Reproducibility for TextCategorizer and Tok2Vec.
• Fix issue #6219: Add re-enabled pipe names back to the meta before serializing.
• Fix issue #6300: Fix on_match callback and exclude empty match lists from results for DependencyMatcher.
• Fix issue #6347: Memory leak issues with beam_parse (requires thinc>=7.4.3).
• Fix issue #6373: Bugfix textcat reproducibility on GPU (requires thinc>=7.4.3).
• Fix issue #6405: Add all vectors to vocab before pruning.
• Fix issue #6413: Use int8_t instead of char in Matcher.

👥 Contributors

Thanks to @abchapman93, @baranitharan2020, @bittlingmayer, @bjascob, @borijang, @BramVanroy, @chopeen, @danielvasic, @delzac, @DuyguA, @erip, @florijanstamenkovic, @graue70, @hiroshi-matsuda-rit, @holubvl3, @idoshr, @jgutix, @KKsharma99, @leyendecker, @lizhe2004, @MartinoMensio, @nipunsadvilkar, @Nuccy90, @oculusrepairo, @rahul1990gupta, @rasyidf, @robertsipek, @SamEdwardes, @snsten, @solarmist, @Stannislav, @tamuhey, @tilusnet, @vha14, @wannaphong, @zaibacu for the pull requests and contributions.

✨ New features and improvements

• Improve Korean tokenizer speed.
• Add experimental character-based pretraining.

🔴 Bug fixes

• Fix issue #5728: Fix French lemmatizer.
• Fix issue #5729: Fix lemmatizer for python 2.7.
• Fix issue #5751: Fix meta serialization in train CLI.

👥 Contributors

Thanks to @graue70, @mikeizbicki, @jbesomi, @gandersen101 and @DeNeutoy for the pull requests and contributions.

✨ New features and improvements

• NEW: Add alpha support for Nepali.
• Refactor Japanese tokenizer and include additional custom tokenizer features.
• Update Armenian language data.
• Include spacy git commit in package and model meta for reference.

🔴 Bug fixes

• Fix issue #5620: Skip vocab in component config overrides.
• Fix issue #5634: Fix polarity of Token.is_oov and Lexeme.is_oov.
• Fix issue #5643: Add strings and ENT_KB_ID to Doc serialization.
• Fix issue #5648: Disregard special tag _SP in check for new tag map.
• Fix issue #5658 : Move lemmatizer is_base_form to language settings.

👥 Contributors

Thanks to @myavrum, @mahnerak, @rameshhpathak, @hiroshi-matsuda-rit, @PluieElectrique, @hertelm and @alvaroabascar for the pull requests and contributions.

⚠️ This version of spaCy requires downloading new models. You can use the spacy validate command to find out which models need updating, and print update instructions. If you've been training your own models, you'll need to retrain them with the new version.

✨ New features and improvements

• NEW: Pretrained model families for Chinese, Danish, Japanese, Polish and Romanian, as well as larger models with vectors for Dutch, German, French, Italian, Greek, Lithuanian, Portuguese and Spanish. 29 new models and 46 model packages in total!
• NEW: 2-4× faster loading times for models with vectors and 2× smaller packages.
• NEW: Alpha support for Armenian, Gujarati and Malayalam.
• NEW: Lookup lemmatization for Polish.
• NEW: Allow Matcher to match on both Doc and Span objects.
• NEW: Add Token.is_sent_end property.
• Improve language data for Danish, Dutch, French, German, Italian, Lithuanian, Norwegian, Romanian and Spanish to better match UD corpora.
• Update language data for Danish, Kannada, Korean, Persian, Swedish and Urdu.
• Add support for pkuseg alongside jieba for Chinese.
• Switch from fugashi to sudachipy for Japanese.
• Improve punctuation used in sentencizer.
• Switch to new and more consistent alignment method in gold.align.
• Reduce stored lexemes data and move non-derivable features to spacy-lookups-data.

🔴 Bug fixes

• Fix issue #5056: Introduce support for matching Span objects.
• Fix issue #5086: Remove Vectors.from_glove.
• Fix issue #5131: Improve data processing in named entity linking scripts.
• Fix issue #5137: Fix passing of component configuration to component.
• Fix issue #5144: Fix sentence comparison in test util.
• Fix issue #5166: Fix handling of exclusive_classes in textcat ensemble.
• Fix issue #5170: Set rank for new vector in Vocab.set_vector.
• Fix issue #5181: Prevent None values in gold fields.
• Fix issue #5191: Fix GoldParse initialization when the number of tokens has changed.
• Fix issue #5193: Correctly pin cupy-cuda extra dependencies.
• Fix issue #5200: Fix minor bugs in train CLI.
• Fix issue #5216: Modify Vectors.resize to work with cupy.
• Fix issue #5228: Raise error for inplace resize with new vector dimension.
• Fix issue #5230: Fix unittest warnings when saving a model.
• Fix issue #5257: Use inline flags in token_match patterns.
• Fix issue #5278, #5359: Add missing __init__.py files to language data tests.
• Fix issue #5281: Fix comparison predicate handling for !=.
• Fix issue #5287: Normalize TokenC.sent_start values for Matcher.
• Fix issue #5292: Fix typo in option name --n-save_every.
• Fix issue #5303: Use max(uint64) for OOV lexeme rank.
• Fix issue #5311: Fix alignment of cards on landing page.
• Fix issue #5320: Fix most_similar for vectors with unused rows.
• Fix issue #5344: Prevent pip from installing spaCy on Python 3.4.
• Fix issue #5356: Fix bug in Span.similarity that could trigger TypeError.
• Fix issue #5361: Fix problems with lower and whitespace in variants.
• Fix issue #5373: Improve exceptions for 'd (would/had) in English.
• Fix issue #5387: Fix logic in train CLI timing eval on CPU/GPU.
• Fix issue #5393, #5458: Fix check for overlapping spans in noun chunks.
• Fix issue #5429: Modify array type to accommodate OOV_RANK.
• Fix issue #5430: Check that row is within bounds when adding vector.
• Fix issue #5435: Use Token.sent_start for Span.sent.
• Fix issue #5436: Fix ErrorsWithCodes().__class__ return value.
• Fix issue #5450: Disallow merging 0-length spans.

⚠️ Backwards incompatibilities

• This version of spaCy requires downloading new models. You can use the spacy validate command to find out which models need updating, and print update instructions.
• If you're training new models, you'll want to install the package spacy-lookups-data, which now includes both the lemmatization tables (as in v2.2) and the normalization tables (new in v2.3). If you're using pretrained models, nothing changes, because the relevant tables are included in the model packages.
• Due to the updated Universal Dependencies training data, the fine-grained part-of-speech tags will change for many provided language models. The coarse-grained part-of-speech tagset remains the same, but the mapping from particular fine-grained to coarse-grained tags may show minor differences.
• For French, Italian, Portuguese and Spanish, the fine-grained part-of-speech tagsets contain new merged tags related to contracted forms, such as ADP_DET for French "au", which maps to UPOS ADP based on the head "à". This increases the accuracy of the models by improving the alignment between spaCy's tokenization and Universal Dependencies multi-word tokens used for contractions.
• spaCy's custom warnings have been replaced with native Python warnings. Instead of setting SPACY_WARNING_IGNORE, use the warnings filters to manage warnings.

📖 Documentation and examples

• Fix various typos and inconsistencies.
• Add new projects to the spaCy Universe.
• Move bin/wiki_entity_linking scripts for Wikipedia to projects repo.

🔥 ICYMI: We recently updated the free and interactive spaCy course to include translations for German (with German NLP examples), Spanish (with Spanish NLP examples) and Japanese, as well as videos for English and German. Translations for Chinese (with Chinese NLP examples), French (with French NLP examples) and Russian coming soon!

📦 Model packages (43)

Model	Language	Version	Vectors
zh_core_web_sm	Chinese	2.3.0	𐄂
zh_core_web_md	Chinese	2.3.0	✓
zh_core_web_lg	Chinese	2.3.0	✓
da_core_news_sm	Danish	2.3.0	𐄂
da_core_news_md	Danish	2.3.0	✓
da_core_news_lg	Danish	2.3.0	✓
nl_core_news_sm	Dutch	2.3.0	𐄂
nl_core_news_md	Dutch	2.3.0	✓
nl_core_news_lg	Dutch	2.3.0	✓
en_core_web_sm	English	2.3.0	𐄂
en_core_web_md	English	2.3.0	✓
en_core_web_lg	English	2.3.0	✓
fr_core_news_sm	French	2.3.0	𐄂
fr_core_news_md	French	2.3.0	✓
fr_core_news_lg	French	2.3.0	✓
de_core_news_sm	German	2.3.0	𐄂
de_core_news_md	German	2.3.0	✓
de_core_news_lg	German	2.3.0	✓
el_core_news_sm	Greek	2.3.0	𐄂
el_core_news_md	Greek	2.3.0	✓
el_core_news_lg	Greek	2.3.0	✓
it_core_news_sm	Italian	2.3.0	𐄂
it_core_news_md	Italian	2.3.0	✓
it_core_news_lg	Italian	2.3.0	✓
ja_core_news_sm	Japanese	2.3.0	𐄂
ja_core_news_md	Japanese	2.3.0	✓
ja_core_news_lg	Japanese	2.3.0	✓
lt_core_news_sm	Lithuanian	2.3.0	𐄂
lt_core_news_md	Lithuanian	2.3.0	✓
lt_core_news_lg	Lithuanian	2.3.0	✓
nb_core_news_sm	Norwegian Bokmål	2.3.0	𐄂
nb_core_news_md	Norwegian Bokmål	2.3.0	✓
nb_core_news_lg	Norwegian Bokmål	2.3.0	✓
pl_core_news_sm	Polish	2.3.0	𐄂
pl_core_news_md	Polish	2.3.0	✓
pl_core_news_lg	Polish	2.3.0	✓
pt_core_news_sm	Portuguese	2.3.0	𐄂
pt_core_news_md	Portuguese	2.3.0	✓
pt_core_news_lg	Portuguese	2.3.0	✓
ro_core_news_sm	Romanian	2.3.0	𐄂
ro_core_news_md	Romanian	2.3.0	✓
ro_core_news_lg	Romanian	2.3.0	✓
es_core_news_sm	Spanish	2.3.0	𐄂
es_core_news_md	Spanish	2.3.0	✓
es_core_news_lg	Spanish	2.3.0	✓
xx_ent_wiki_sm	Multi-language	2.3.0	𐄂

👥 Contributors

Thanks to @mabraham, @sloev, @pinealan, @pmbaumgartner, @Baciccin, @nlptechbook, @guerda, @Tiljander, @nikhilsaldanha, @tommilligan, @Jacse, @leicmi, @YohannesDatasci, @mirfan899, @koaning, @umarbutler, @chopeen, @paoloq, @thomasthiebaud, @sebastienharinck, @elben10, @laszabine, @Mlawrence95, @sabiqueqb, @punitvara, @michael-k, @louisguitton, @vondersam, @thoppe, @vishnupriyavr, @ilivans and @osori for the pull requests and contributions.

🙏 Special thanks to everyone who helped us develop and test the new models: @lixiepeng, @lingvisa and @howl-anderson (Chinese), @hvingelby (Danish), @hiroshi-matsuda-rit and @polm (Japanese), @ryszardtuora (Polish) and @avramandrei and @dumitrescustefan (Romanian).

✨ New features and improvements

• NEW: Add Span.char_span method.
• NEW: Base language support for Yoruba and Basque.
• NEW: Add --tag-map-path argument to debug-data and train commands.
• NEW Add add_lemma option to displacy dependency visualizer.
• Add IDX as an attribute available via Doc.to_array.
• Improve speed of adding large number of patterns to EntityRuler.
• Replace python-mecab3 with fugashi for Japanese.
• Improve language data for Norwegian, Luxembourgish, Finnish, Slovak, Romanian, Greek and German.

🔴 Bug fixes

• Fix issue #3979, #4819, #4871: Add tok2vec parameters to train command.
• Fix issue #4009: Fix use of pretrained vectors in text classifier.
• Fix issue #4342: Improve CLI training with base model.
• Fix issue #4432: Add destructors for states in TransitionSystem.
• Fix issue #4440: Require HEAD for is_parsed in Doc.from_array.
• Fix issue #4615: Update SHAPE docs and examples.
• Fix issue #4665: Allow HEAD field in CoNLL-U format to be an underscore.
• Fix issue #4673: Ensure correct array module is used when returning a vector via Vocab.
• Fix issue #4674: Make set_entities in the KnowledgeBase more robust.
• Fix issue #4677: Add missing tags to tag maps for el, es and pt.
• Fix issue #4688: Iterate over lr_edges until Doc.sents are correct.
• Fix issue #4703, #4823: Facilitate large training files.
• Fix issue #4707: Auto-exclude disabled when calling from_disk during load.
• Fix issue #4717: Fix int value handling in Matcher.
• Fix issue #4719: Add message when cli train script throws exception.
• Fix issue #4723: Update EntityLinker example.
• Fix issue #4725: Take care of global vectors in multiprocessing.
• Fix issue #4770: Include Doc.cats in serialization of Doc and DocBin.
• Fix issue #4772: Fix bug in EntityLinker.predict.
• Fix issue #4777: Fix link to user hooks in documentation.
• Fix issue #4829: Update build dependencies in pyproject.toml.
• Fix issue #4830: Warn for punctuation in entities when training with noise.
• Fix issue #4833: Make example scripts work with transformer starter models.
• Fix issue #4849: Fix serialization of ENT_ID.
• Fix issue #4862: Fix and improve URL pattern.
• Fix issue #4868: Include .pyx and .pxd files in the distribution.
• Fix issue #4876: Add friendlier error to entity linking example script.
• Fix issue #4903: Fix handling of custom underscore attributes during multiprocessing.
• Fix issue #4924: Fix handling of empty docs or golds in Language.evaluate.
• Fix issue #4934: Prevent updating component config if the Model was already defined.
• Fix issue #4935: Fix Sentencizer.pipe for empty Doc.
• Fix issue #4961: Remove old docs section links.
• Fix issue #4965: Sync Span.__eq__ and Span.__hash__.
• Fix issue #4975: Adjust srsly pin.
• Fix issue #5048: Fix behavior of get_doc test utility.
• Fix issue #5073: Normalize IS_SENT_START to SENT_START for Matcher.
• Fix issue #5075: Make it impossible to create invalid heads with Doc.from_array.
• Fix issue #5082: Correctly set vector of merged span in merge_entities.
• Fix issue #5115: Ensure paths in Tokenizer.to_disk and Tokenizer.from_disk.
• Fix issue #5117: Clarify behavior of Doc.is_ flags for empty Docs.

📖 Documentation and examples

• Fix various typos and inconsistencies.
• Add new projects to the spaCy Universe.

👥 Contributors

Thanks to @polm, @mmaybeno, @jarib, @questoph, @aajanki, @mr-bjerre, @Tclack88, @thiagola92, @tamuhey, @Olamyy, @AlJohri, @iechevarria, @iurshina, @lineality, @pbadeer, @BramVanroy, @kabirkhan, @ceteri, @omri374, @maknotavailable, @onlyanegg, @drndos, @ju-sh, @nlptechbook, @chkoar, @Jan-711, @MisterKeefe, @bryant1410, @mirfan899, @dhpollack and @mabraham for the pull requests and contributions!

✨ New features and improvements

• NEW: Tokenizer.explain method to see which rule or pattern was matched.

  tok_exp = nlp.tokenizer.explain("(don't)")
  assert [t[0] for t in tok_exp] == ["PREFIX", "SPECIAL-1", "SPECIAL-2", "SUFFIX"]
  assert [t[1] for t in tok_exp] == ["(", "do", "n't", ")"]
  

• NEW: Official Python 3.8 wheels for spaCy and its dependencies.
• Base language support for Korean.
• Add Scorer.las_per_type (labelled depdencency scores per label).
• Rework Chinese language initialization and tokenization
• Improve language data for Luxembourgish.

🔴 Bug fixes

• Fix issue #4573, #4645: Improve tokenizer usage docs.
• Fix issue #4575: Add error in debug-data if no dev docs are available.
• Fix issue #4582: Make as_tuples=True in Language.pipe work with multiprocessing.
• Fix issue #4590: Correctly call on_match in DependencyMatcher.
• Fix issue #4593: Build wheels for Python 3.8.
• Fix issue #4604: Fix realloc in Retokenizer.split.
• Fix issue #4656: Fix conllu2json converter when -n > 1.
• Fix issue #4662: Fix Language.evaluate for components without .pipe method.
• Fix issue #4670: Ensure EntityRuler is deserialized correctly from disk.
• Fix issue #4680: Raise error if non-string labels are added to Tagger or TextCategorizer.
• Fix issue #4691: Make Vectors.find return keys in correct order.

📖 Documentation and examples

• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @yash1994, @walterhenry, @prilopes, @f11r, @questoph, @erip, @richardpaulhudson and @GuiGel for the pull requests and contributions.

✨ New features and improvements

• NEW: Support multiprocessing in nlp.pipe via the n_process argument (Python 3 only).
• Base language support for Luxembourgish.
• Add noun chunks iterator for Swedish.
• Retrained models for Greek, Norwegian Bokmål and Lithuanian that now correctly support parser-based sentence segmentation.
• Repackaged models for Greek and German with improved lookup tables via spacy-lookups-data.
• Add warning in debug-data for low sentences per doc ratio.
• Improve checks and errors related to ill-formed IOB input in convert and debug-data CLI.
• Support training dict format as JSONL.
• Make EntityRuler ID resolution 2× faster and support "id" in patterns to set Token.ent_id.
• Improve rendering of named entity spans in displacy for RTL languages.
• Update Thinc to ditch thinc_gpu_ops for simpler GPU install.
• Support Mish activation in spacy pretrain.
• Add forwards-compatible support for new Language.disable_pipes API, which will become the default in the future. The method can now also take a list of component names as its first argument (instead of a variable number of arguments).

  - disabled = nlp.disable_pipes("tagger", "parser")
  + disabled = nlp.disable_pipes(["tagger", "parser"])
  

• Add forwards-compatible support for new Matcher.add and PhraseMatcher.add API, which will become the default in the future. The patterns are now the second argument and a list (instead of a variable number of arguments). The on_match callback becomes an optional keyword argument.

  patterns = [[{"TEXT": "Google"}, {"TEXT": "Now"}], [{"TEXT": "GoogleNow"}]]
  - matcher.add("GoogleNow", None, *patterns)
  + matcher.add("GoogleNow", patterns)
  - matcher.add("GoogleNow", on_match, *patterns)
  + matcher.add("GoogleNow", patterns, on_match=on_match)
  

• Add new and improved tokenization alignment in gold.align behind a feature flag. The new alignment may produce backwards-incompatible results, so it won't be enabled by default before v3.0.

  import spacy.gold
  spacy.gold.USE_NEW_ALIGN = True
  

🔴 Bug fixes

• Fix issue #1303: Support multiprocessing in nlp.pipe.
• Fix issue #1745: Ditch thinc_gpu_ops for simpler GPU install.
• Fix issue #2411: Update Thinc to fix compilation on cygwin.
• Fix issue #3412: Prevent division by zero in Vectors.most_similar.
• Fix issue #3618: Fix memory leak for long-running parsing processes.
• Fix issue #4241: Update Greek lookups in spacy-lookups-data.
• Fix issue #4269: Extend unicode character block for Sinhala.
• Fix issue #4362: Improve URL_PATTERN and handling in tokenizer.
• Fix issue #4373: Make PhraseMatcher.vocab consistent with Matcher.vocab.
• Fix issue #4377: Clarify serialization of extension attributes.
• Fix issue #4382: Improve usage of pkg_resources and handling of entry points.
• Fix issue #4386: Consider batch_size when sorting similar vectors.
• Fix issue #4389: Fix ner_jsonl2json converter.
• Fix issue #4397: Ensure on_match callback is executed in PhraseMatcher.
• Fix issue #4401, #4408: Fix sentence segmentation in Greek, Norwegian and Lithuanian models.
• Fix issue #4402: Fix issue with how training data was passed through the pipeline.
• Fix issue #4406: Correct spelling in lemmatizer API docs.
• Fix issue #4418, #4438: Improve knowledge base and Wikidata parsing.
• Fix issue #4435: Fix PhraseMatcher.remove for overlapping patterns.
• Fix issue #4443: Fix bug in Vectors.most_similar.
• Fix issue #4452: Fix gold.docs_to_json documentation.
• Fix issue #4463: Add missing cats to GoldParse.from_annot_tuples in Scorer.
• Fix issue #4470: Suppress convert output if writing to stdout.
• Fix issue #4475: Correct mistake in docs example.
• Fix issue #4485: Update tag maps and docs for English and German.
• Fix issue #4493: Update information in spaCy Universe.
• Fix issue #4496: Improve docs of PhraseMatcher.add arguments.
• Fix issue #4506: Ensure Vectors.most_similar returns 1.0 for identical vectors.
• Fix issue #4509: Fix None iteration error in entity linking script.
• Fix issue #4524: Fix typo in Parser sample construction of GoldParse.
• Fix issue #4528: Fix serialization of extension attribute values in DocBin.
• Fix issue #4529: Ensure GoldParse is initialized correctly with misaligned tokens.
• Fix issue #4538: Backport memory leak fix to v2.1.x branch and release v2.1.9.

⚠️ Backwards incompatibilities

• The unused attributes lemma_rules, lemma_index, lemma_exc and lemma_lookup of the Language.Defaults have now been removed to prevent confusion (e.g. if users add rules that then have no effect). The only place lemmatization tables are stored and can be modified at runtime is via nlp.vocab.lookups.

  - nlp.Defaults.lemma_lookup["spaCies"] = "spaCy"
  + lemma_lookup = nlp.vocab.lookups.get_table("lemma_lookup")
  + lemma_lookup["spaCies"] = "spaCy"
  

📖 Documentation and examples

• Fix various typos and inconsistencies.
• Add more projects to the spaCy Universe.

👥 Contributors

Thanks to @tamuhey, @PeterGilles, @akornilo, @danielkingai2, @ghollah, @pberba, @gustavengstrom, @ju-sh, @kabirkhan, @ZhuoruLin, @nipunsadvilkar and @neelkamath for the pull requests and contributions.

✨ New features and improvements

• Make Vectors.most_similar return the top most similar vectors instead of only one.

🔴 Bug fixes

• Fix issue #4365: Fix tag map in Dutch model.
• Fix issue #4368: Fix initialization of DocBin with attributes.

📖 Documentation and examples

• Add API docs for Vectors.most_similar.

👥 Contributors

Thanks to @bintay and @svlandeg for the pull requests and contributuons.

⚠️ This version of spaCy requires downloading new models. You can use the spacy validate command to find out which models need updating, and print update instructions. If you've been training your own models, you'll need to retrain them with the new version.

✨ New features and improvements

• NEW: Pretrained core models for Norwegian (MIT) and Lithuanian (CC BY-SA).
• NEW: Better pre-trained Dutch NER using custom labelled UD corpus instead of WikiNER.
• NEW: Make spaCy roughly 5-10× smaller on disk (depending on your platform) by compressing and moving lookups to a separate package.
• NEW: EntityLinker and KnowledgeBase API to train and access entity linking models, plus scripts to train your own Wikidata models.
• NEW: 10× faster PhraseMatcher and improved phrase matching algorithm.
• NEW: DocBin class to efficiently serialize collections of Doc objects.
• NEW: Train text classification models on the command line with spacy train and get textcat results via the Scorer.
• NEW: debug-data command to validate your training and development data, get useful stats, and find problems like invalid entity annotations, cyclic dependencies, low data labels and more.
• NEW: Efficient Lookups class using Bloom filters that allows storing, accessing and serializing large dictionaries via vocab.lookups.
• Data augmentation in spacy train via the --orth-variant-level flag, which defines the percentage of occurrences of some tokens subject to replacement during training.
• Add nlp.pipe_labels (labels assigned by pipeline components) and include "labels" in nlp.meta.
• Support spacy_displacy_colors entry point to allow packages to add entity colors to displacy.
• Allow template config option in displacy to customize entity HTML template.
• Improve match pattern validation and handling of unsupported attributes.
• Add lookup lemmatization data for Croatian and Serbian.
• Update and improve language data for Chinese, Croatian, Thai, Romanian, Hindi and English.

🔴 Bug fixes

• Fix issue #3258: Reduce package size on disk by moving and compressing large dictionaries.
• Fix issue #3540: Update lemma and vector information after splitting a token.
• Fix issue #3687: Automatically skip duplicates in Doc.retokenize.
• Fix issue #3830: Retrain German model and fix subtok errors.
• Fix issue #3850: Allow customizing entity HTML template in displaCy.
• Fix issue #3879, #3951, #4154: Fix bug in Matcher retry loop that'd cause problems with ? operator.
• Fix issue #3917: Raise error for negative token indices in displacy.
• Fix issue #3922: Add PhraseMatcher.remove method.
• Fix issue #3959, #4133: Make sure both pos and tag are correctly serialized.
• Fix issue #3972: Ensure PhraseMatcher returns multiple matches for identical rules.
• Fix issue #4020: Raise error for overlapping entities in biluo_tags_from_offsets.
• Fix issue #4051: Ensure retokenizer sets POS tags correctly on merge.
• Fix issue #4070: Improve token pattern checking without validation.
• Fix issue #4096: Add checks for cycles in debug-data.
• Fix issue #4100: Improve docs on phrase pattern attributes.
• Fix issue #4102: Correct mistakes in English lookup lemmatizer data.
• Fix issue #4104: Make visualized NER examples in docs more clear.
• Fix issue #4107: Automatically set span root attributes on merging.
• Fix issue #4111, #4170: Improve NER/IOB converters.
• Fix issue #4120: Correctly handle ? operator at the end of pattern.
• Fix issue #4123: Provide more details in cycle error message E069.
• Fix issue #4138: Correctly open .html files as UTF-8 in evaluate command.
• Fix issue #4139: Make emoticon data a raw string.
• Fix issue #4148: Add missing API docs for force flag on set_extension.
• Fix issue #4155: Correct language code for Serbian.
• Fix issue #4165: Add more attributes to matcher validation schema.
• Fix issue #4190: Fix caching issue that'd cause tokenizer to not be deserialized correctly.
• Fix issue #4200: Work around tqdm bug that'd remove text color from terminal output.
• Fix issue #4229: Fix handling of pre-set entities.
• Fix issue #4238: Flush tokenizer cache when affixes, token_match, or special cases are modified.
• Fix issue #4242: Make .pos/.tag distinction more clear in the docs.
• Fix issue #4245: Fix bug that occurred when processing empty string in Korean.
• Fix issue #4262: Fix handling of spaces in Japanese.
• Fix issue #4269: Tokenize punctuation correctly in Kannada, Tamil, and Telugu and add unicode characters to default sentencizer config.
• Fix issue #4270: Fix --vectors-loc documentation.
• Fix issue #4302: Remove duplicate Parser.tok2vec property.
• Fix issue #4303: Correctly support as_tuples and return_matches in Matcher.pipe.
• Fix issue #4307: Ensure that pre-set entities are preserved and allow overwriting unset tokens.
• Fix issue #4308: Fix bug that could cause PhraseMatcher with very large lists to miss matches.
• Fix issue #4348: Ensure training doesn't crash with empty batches.

⚠️ Backwards incompatibilities

• This version of spaCy requires downloading new models. You can use the spacy validate command to find out which models need updating, and print update instructions.
• The lemmatization tables have been moved to their own package, spacy-lookups-data, which is not installed by default. If you're using pre-trained models, nothing changes, because the tables are now included in the model packages. If you want to use the lemmatizer for other languages that don't yet have pre-trained models (e.g. Turkish or Croatian) or start off with a blank model that contains lookup data (e.g. spacy.blank("en")), you'll need to explicitly install spaCy plus data via pip install spacy[lookups]. The data will be registered automatically via entry points.
• Lemmatization tables (rules, exceptions, index and lookups) are now part of the Vocab and serialized with it. This means that serialized objects (nlp, pipeline components, vocab) will now include additional data, and models written to disk will include additional files.
• The Lemmatizer class is now initialized with an instance of Lookups containing the rules and tables, instead of dicts as separate arguments. This makes it easier to share data tables and modify them at runtime. This is mostly internals, but if you've been implementing a custom Lemmatizer, you'll need to update your code.
• If you've been training your own models, you'll need to retrain them with the new version.
• The Dutch model has been trained on a new NER corpus (custom labelled UD instead of WikiNER), so their predictions may be very different compared to the previous version. The results should be significantly better and more generalizable, though.
• The spacy download command does not set the --no-deps pip argument anymore by default, meaning that model package dependencies (if available) will now be also downloaded and installed. If spaCy (which is also a model dependency) is not installed in the current environment, e.g. if a user has built from source, --no-deps is added back automatically to prevent spaCy from being downloaded and installed again from pip.
• The built-in biluo_tags_from_offsets converter is now stricter and will raise an error if entities are overlapping (instead of silently skipping them). If your data contains invalid entity annotations, make sure to clean it and resolve conflicts. You can now also use the new debug-data command to find problems in your data.
• Pipeline components can now overwrite IOB tags of tokens that are not yet part of an entity. Once a token has an ent_iob value set, it won't be reset to an "unset" state and will always have at least O assigned. list(doc.ents) now actually keeps the annotations on the token level consistent, instead of resetting O to an empty string.
• The default punctuation in the Sentencizer has been extended and now includes more characters common in various languages. This also means that the results it produces may change, depending on your text. If you want the previous behaviour with limited characters, set punct_chars=[".", "!", "?"] on initialization.
• The PhraseMatcher algorithm was rewritten from scratch and it's now 10× faster. The rewrite also resolved a few subtle bugs with very large terminology lists. So if you were matching large lists, you may see slightly different results – however, the results should now be fully correct. See #4309 for details on this change.
• The Serbian language class (introduced in v2.1.8) incorrectly used the language code rs instead of sr. This has now been fixed, so Serbian is now available via spacy.lang.sr.
• The "sources" in the meta.json have changed from a list of strings to a list of dicts. This is mostly internals, but if your code used nlp.meta["sources"], you might have to update it.

📈 Benchmarks

Model	Language	Version	UAS	LAS	POS	NER F	Vec	Size
en_core_web_sm	English	2.2.0	91.61	89.71	97.03	85.07	𐄂	11 MB
en_core_web_md	English	2.2.0	91.65	89.77	97.14	86.10	✓	91 MB
en_core_web_lg	English	2.2.0	91.98	90.16	97.21	86.30	✓	789 MB
de_core_news_sm	German	2.2.0	90.75	88.63	96.29	83.11	𐄂	14 MB
de_core_news_md	German	2.2.0	91.26	89.36	96.44	83.42	✓	214 MB
es_core_news_sm	Spanish	2.2.0	90.20	87.05	96.79	89.45	𐄂	15 MB
es_core_news_md	Spanish	2.2.0	90.89	87.94	97.03	89.86	✓	74 MB
pt_core_news_sm	Portuguese	2.2.0	89.53	86.07	79.96	87.97	𐄂	20 MB
fr_core_news_sm	French	2.2.0	87.27	84.28	94.38	82.77	𐄂	14 MB
fr_core_news_md	French	2.2.0	88.82	86.07	95.15	82.82	✓	84 MB
it_core_news_sm	Italian	2.2.0	90.79	86.94	96.06	86.29	𐄂	13 MB
nl_core_news_sm	Dutch	2.2.0	76.79	69.53	90.10	68.79	𐄂	14 MB
el_core_news_sm	Greek	2.2.0	84.40	80.98	94.41	71.88	𐄂	10 MB
el_core_news_md	Greek	2.2.0	87.96	84.88	96.38	77.59	✓	126 MB
nb_core_news_sm	Norwegian	2.2.0	89.02	86.49	95.72	83.99	𐄂	12 MB
lt_core_news_sm	Lithuanian	2.2.0	59.87	48.00	74.02	76.58	𐄂	12 MB
xx_ent_wiki_sm	Multi	2.2.0	-	-	-	79.88	𐄂	3 MB

💬 UAS: Unlabelled dependencies (parser). LAS: Labelled dependencies (parser). POS: Part-of-speech tags (fine-grained tags, i.e. Token.tag_). NER F: Named entities (F-score). Vec: Model contains word vectors. Size: Model file size (zipped archive).

📖 Documentation and examples

• Add "label scheme" section to all models in the models directory that lists the labels assigned by the different components.
• Extend the sources listed in the meta.json of pre-trained models with more details on the training corpora and include more information in the models directory.
• Add more examples of matching regular expressions.
• Add instructions for training an entity linking model.
• Add API docs for new debug-data, EntityLinker, KnowledgeBase and Lookups.
• Add new projects to the spaCy Universe.
• Add example for interactive model visualizer with Streamlit.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @ICLRandD, @phiedulxp, @ajrader, @RyanZHe, @jenojp, @yanaiela, @isaric, @mrdbourke, @avramandrei, @Pavle992, @chkoar, @wannaphongcom, @BreakBB, @b1uec0in, @mihaigliga21, @tamuhey, @euand, @Hazoom, @SeanBE, @esemeniuc, @zqianem, @ajkl, @jaydeepborkar, @EarlGreyT and @er-raoniz for the pull requests and contributions.

Special thanks to our spaCy team @svlandeg and @adrianeboyd for the bug fixes and new features, @polm for the Bloom filters implementation and data compression and @yvespeirsman, @lemontheme, @jarib, @miktoki and @rokasramas for the help and resources for the new models.

This is a small maintenance update that backports a bug fix for a memory leak that'd occur in long-running parsing processes. It's intended for users who can't or don't yet want to upgrade to spaCy v2.2 (e.g. because it requires retraining all the models). If you're able to upgrade, you shouldn't use this version and instead install the latest v2.2.

🔴 Bug fixes

• Fix issue #3618: Fix memory leak for long-running parsing processes.
• Fix issue #4538: Backport memory leak fix to v2.1.x branch.

✨ New features and improvements

• NEW: Alpha tokenization support for Serbian
• Improve language data for Urdu.
• Support installing and loading model packages in the same session.

🔴 Bug fixes

• Fix issue #4002: Make PhraseMatcher work as expected for NORM attribute.
• Fix issue #4063: Improve docs on Matcher attributes.
• Fix issue #4068: Make Korean work as expected on Python 2.7.
• Fix issue #4069: Add validate option to EntityRuler.
• Fix issue #4074: Raise error if annotation dict in simple training style has unexpected keys.
• Fix issue #4081: Fix typo in pyproject.toml.
• Fix handling of keyword arguments in Language.evaluate.

📖 Documentation and examples

• Improve Matcher attribute docs.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @akornilo, @mirfan899, @veer-bains, @seppeljordan, @Pavle992, @svlandeg, @jenojp and @adrianeboyd for the pull requests and contributions.

✨ New features and improvements

• Add Token.tensor and Span.tensor attributes.
• Support simple training format of (text, annotations) instead of only (doc, gold) for nlp.evaluate.
• Add support for "lang_factory" setting in model meta.json (see #4031).
• Also support "requirements" in meta.json to define packages for setup's install_requires.
• Improve Pipe base class methods and make them less presumptuous.
• Improve Danish and Korean tokenization.
• Improve error messages when deserializing model fails.

🔴 Bug fixes

• Fix issue #3669, #3962: Fix dependency copy in Span.as_doc that could cause segfault.
• Fix issue #3968: Fix bug in per-entity scores.
• Fix issue #4000: Improve entity linking API.
• Fix issue #4022: Fix error when Korean text contains special characters.
• Fix issue #4030: Handle edge case when calling TextCategorizer.predict with empty Doc.
• Fix issue #4045: Correct Span.sent docs.
• Fix issue #4048: Fix init-model command if there's no vocab.
• Fix issue #4052: Improve per-type scoring of NER.
• Fix issue #4054: Ensure the lang of nlp and nlp.vocab stay consistent.
• Fix bugs in Token.similarity and Span.similarity when called via hook.

📖 Documentation and examples

• Add documentation for gold.align helper.
• Add more explicit section on processing text.
• Improve documentation on disabling pipeline components.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @sorenlind, @pmbaumgartner, @svlandeg, @FallakAsad, @BreakBB, @adrianeboyd, @polm, @b1uec0in, @mdaudali and @ejarkm for the pull requests and contributions.

🔴 Bug fixes

• Fix issue #3958: Fix order of symbols that caused tag maps to be out-of-sync.

✨ New features and improvements

• NEW: Base language data for Marathi and Korean (via mecab-ko, mecab-ko-dic and natto-py).
• Improve language data for Lithuanian, Spanish, Kannada, French, Norwegian and Hindi.
• Add evaluation metrics per entity type.
• Add resume logic to spacy pretrain.
• Add optional id property to EntityRuler patterns.
• Better introspection and IDE automcomplete for custom extension attributes.
• Make Doc.is_sentenced always return True for single-token docs.

🔴 Bug fixes

• Fix issue #3490: Add evaluation metrics per entity type to Scorer.
• Fix issue #3526: Serialize EntityRuler settings correctly.
• Fix issue #3558: Improve E024 error message for incorrect GoldParse.
• Fix issue #3611: Fix bug when setting ngram parameter in text classifier.
• Fix issue #3625: Improve default punctuation rules for Hindi.
• Fix issue #3707: Improve introspection of custom attributes.
• Fix issue #3737: Check if component is callable in Language.replace_pipe.
• Fix issue #3743: Fix documentation of lex_id.
• Fix issue #3749: Change vector training script to work with latest Gensim.
• Fix issue #3762, #3934: Make Doc.is_sentenced default to True for single-token Docs.
• Fix issue #3802: Fix typo in docs example.
• Fix issue #3811: Fix type of --seed option in spacy pretrain.
• Fix issue #3822: Allow passing PhraseMatcher arguments to EntityRuler.
• Fix issue #3839: Ensure the Matcher returns correct match IDs when used with operators.
• Fix issue #3840: Improve error messages in spacy pretrain.
• Fix issue #3853: Rename vectors if multiple models are loaded to prevent clashes.
• Fix issue #3859: Update pretrain to prevent unintended overwriting of weight files.
• Fix issue #3862: Fix matcher callback example.
• Fix issue #3868: Add "v.s." to English tokenizer exceptions.
• Fix issue #3869: Make Doc.count_by work as expected.
• Fix issue #3880: Fix unflatten padding in Thinc when last element is empty.
• Fix issue #3882: Exclude user_data when copying doc in displaCy.
• Fix issue #3892: Update Tokenizer initialization docs.
• Fix issue #3912: Make text classifier raise more friendly errors.

📖 Documentation and examples

• Add documentation for Scorer, Language.evaluate and gold.docs_to_json.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @BreakBB, @ujwal-narayan, @estr4ng7d, @maknotavailable, @ramananbalakrishnan, @nipunsadvilkar, @NirantK, @munozbravo, @intrafindBreno, @Azagh3l, @jarib, @tokestermw, @polm, @skrcode, @kabirkhan, @demongolem, @elbaulp, @clarus, @BramVanroy, @rokasramas, @askhogan, @khellan, @kognate, @cedar101 and @yash1994 for the pull requests and contributions.

✨ New features and improvements

• NEW: util.filter_spans helper to filter duplicates and overlaps from a list of Span objects.
• Improve language data for Thai, Japanese, Indonesian and Dutch.
• Add --n-save-every to spacy pretrain and rename --nr-iter to --n-iter for consistency.
• Add --return-scores flag to spacy evaluate to return a dict.
• Add --n-early-stopping option to spacy train to define maximum number of iterations without dev accuracy improvements.

🔴 Bug fixes

• Fix issue #3307: Fix symlink creation to show error on Windows.
• Fix issue #3473: Fix GPU training for text classification.
• Fix issue #3475: Change favicon.
• Fix issue #3482: Add Estonian base support to documentation.
• Fix issue #3484: Ensure lemmatization is always consistent between sessions.
• Fix issue #3521: Add variations of contractions to English stop words.
• Fix issue #3523: Make spacy convert correctly default to json.
• Fix issue #3525, #3551, #3572: Fix problem that'd cause lemmas to not be lowercase.
• Fix issue #3531: Don't make "settings" or "title" required in displaCy data.
• Fix issue #3533: Remove non-existent example from docs.
• Fix issue #3546: Make sure path in GoldParse.__del__ is a string.
• Fix issue #3549: Ensure match pattern error isn't raised on empty errors list.
• Fix issue #3561: Fix DependencyParser.predict docs.
• Fix issue #3598: Allow jupyter=False to override Jupyter mode in displacy.
• Fix issue #3620: Fix bug in .iob converter.
• Fix issue #3628: Relax jsonschema pin.
• Fix issue #3667: Fix offset bug in loading pre-trained word2vec.
• Fix issue #3679: Update glossary to include missing labels in spacy.explain.
• Fix issue #3680: Re-add missing universe README.
• Fix issue #3681: Rewrite information extraction example to use Doc.retokenize.
• Fix issue #3692: Fix return value in Language.update docs.
• Fix issue #3694: Make "text" in spacy pretrain optional when "tokens" is provided.
• Fix issue #3701: Improve Token.prob and Lexeme.prob docs.
• Fix issue #3708: Fix error in regex matcher examples.
• Fix issue #3713: Call rmtree and copytree with strings in spacy train.
• Fix issue #3720: Add version tag to --base-model argument in spacy train docs.

📖 Documentation and examples

• Add free interactive spaCy course.
• Fix various typos and inconsistencies.
• Add new projects to the spaCy universe.

👥 Contributors

Thanks to @svlandeg, @wannaphongcom, @Bharat123rox, @DuyguA, @SamuelLKane, @graus, @HiromuHota, @jeannefukumaru, @ivigamberdiev, @socool, @yvespeirsman, @lemontheme, @Dobita21, @w4nderlust, @pierremonico, @bryant1410, @celikomer, @xssChauhan, @kowaalczyk, @BreakBB, @fizban99, @tokestermw, @bjascob, @pickfire, @yaph, @amitness, @henry860916, @d5555, @BramVanroy, @F0rge1cE, @richardpaulhudson, @ldorigo, @aaronkub and @devforfu for the pull requests and contributions.

✨ New features and improvements

• Allow customizing punctuation characters in sentencizer and make it serializable.
• Add new "bow" architecture for TextCategorizer, to do faster bag-of-words text classification.

🔴 Bug fixes

• Fix issue #3433, #3458: Fix mismatch of classes in parser after serialization.
• Fix issue #3464: Fix training loop in train_textcat.py example.
• Fix issue #3468: Make sentencizer set Token.is_sent_start correctly.
• Fix bug in the "ensemble" TextClassifier architecture that prevented the unigram bag-of-words submodel from working properly.

👥 Contributors

Thanks to @chkoar for the pull request!

🔴 Bug fixes

• Fix issue #3356: Fix handling of unicode ranges in regular expressions on Python 2.
• Fix issue #3432: Update wasabi to better handle non-UTF-8 terminals.
• Fix issue #3445: Update docs on label argument in Span.__init__.
• Fix issue #3455: Bring English tag_map in line with UD Treebank.

📖 Documentation and examples

• Add --init-tok2vec argument to train_textcat.py example.
• Fix various typos and inconsistencies.

✨ New features and improvements

• Raise error if user is running a narrow unicode build.
• Move ud_train, ud_evaluate and other UD scripts from CLI to /bin in repo only.
• Improve accuracy of spacy pretrain by implementing cosine loss.

🔴 Bug fixes

• Fix issue #3421: Update docs and raise error for narrow unicode builds.
• Fix issue #3427: Correct mistake in French lemmatizer.
• Fix issue #3431: Make Doc.vector and Doc.vector_norm work as expected on GPU.
• Fix issue #3437: Fix installation problem on GPU.
• Fix issue #3439, #3446: Don't include UD scripts in spacy.cli.

👥 Contributors

Thanks to @mhham and @Bharat123Rox for the pull requests!

⚠️ This version of spaCy requires downloading new models. You can use the spacy validate command to find out which models need updating, and print update instructions. If you've been training your own models, you'll need to retrain them with the new version.

✨ New features and improvements

Tagger, Parser, NER and Text Categorizer

• NEW: Experimental ULMFit/BERT/Elmo-like pretraining (see #2931) via the new spacy pretrain command. This pre-trains the CNN using BERT's cloze task. A new trick we're calling Language Modelling with Approximate Outputs is used to apply the pre-training to smaller models. The pre-training outputs CNN and embedding weights that can be used in spacy train, using the new -t2v argument.
• NEW: Allow parser to do joint word segmentation and parsing. If you pass in data where the tokenizer over-segments, the parser now learns to merge the tokens.
• Make parser, tagger and NER faster, through better hyperparameters.
• Add simpler, GPU-friendly option to TextCategorizer, and allow setting exclusive_classes and architecture arguments on initialization.
• Add EntityRecognizer.labels property.
• Remove document length limit during training, by implementing faster Levenshtein alignment.
• Use Thinc v7.0, which defaults to single-thread with fast blis kernel for matrix multiplication. Parallelisation should be performed at the task level, e.g. by running more containers.

Models & Language Data

• NEW: 2-3 times faster tokenization across all languages at the same accuracy!
• NEW: Small accuracy improvements for parsing, tagging and NER for 6+ languages.
• NEW: The English and German models are now available under the MIT license.
• NEW: Statistical models for Greek.
• NEW: Alpha support for Tamil, Ukrainian and Kannada, and base language classes for Afrikaans, Bulgarian, Czech, Icelandic, Lithuanian, Latvian, Slovak, Slovenian and Albanian.
• Improve loading time of French by ~30%.
• Add Vocab.writing_system (populated via the language data) to expose settings like writing direction.

CLI

• NEW: pretrain command for ULMFit/BERT/Elmo-like pretraining (see #2931).
• NEW: New ud-train command, to train and evaluate using the CoNLL 2017 shared task data.
• Check if model is already installed before downloading it via spacy download.
• Pass additional arguments of download command to pip to customise installation.
• Improve train command by letting GoldCorpus stream data, instead of loading into memory.
• Improve init-model command, including support for lexical attributes and word-vectors, using a variety of formats. This replaces the spacy vocab command, which is now deprecated.
• Add support for multi-task objectives to train command.
• Add support for data-augmentation to train command.

Other

• NEW: Enhanced pattern API for rule-based Matcher (see #1971).
• NEW: Doc.retokenize context manager for merging and splitting tokens more efficiently.
• NEW: Add support for custom pipeline component factories via entry points (#2348).
• NEW: Implement fastText vectors with subword features.
• NEW: Built-in rule-based NER component to add entities based on match patterns (see #2513).
• NEW: Allow PhraseMatcher to match on token attributes other than ORTH, e.g. LOWER (for case-insensitive matching) or even POS or TAG.
• NEW: Replace ujson, msgpack, msgpack-numpy, pickle, cloudpickle and dill with our own package srsly to centralise dependencies and allow binary wheels.
• NEW: Doc.to_json() method which outputs data in spaCy's training format. This will be the only place where the format is hard-coded (see #2932).
• NEW: Built-in EntityRuler component to make it easier to build rule-based NER and combinations of statistical and rule-based systems.
• NEW: gold.spans_from_biluo_tags helper that returns Span objects, e.g. to overwrite the doc.ents.
• Add warnings if .similarity method is called with empty vectors or without word vectors.
• Improve rule-based Matcher and add return_matches keyword argument to Matcher.pipe to yield (doc, matches) tuples instead of only Doc objects, and as_tuples to add context to the Doc objects.
• Make stop words via Token.is_stop and Lexeme.is_stop case-insensitive.
• Accept "TEXT" as an alternative to "ORTH" in Matcher patterns.
• Use black for auto-formatting .py source and optimse codebase using flake8. You can now run flake8 spacy and it should return no errors or warnings. See CONTRIBUTING.md for details.

🔴 Bug fixes

• Fix issue #795: Fix behaviour of Token.conjuncts.
• Fix issue #1487: Add Doc.retokenize() context manager.
• Fix issue #1537: Make Span.as_doc return a copy, not a view.
• Fix issue #1574: Make sure stop words are available in medium and large English models.
• Fix issue #1585: Prevent parser from predicting unseen classes.
• Fix issue #1642: Replace regex with re and speed up tokenization.
• Fix issue #1665: Correct typos in symbol Animacy_inan and add Animacy_nhum.
• Fix issue #1748, #1798, #2756, #2934: Add simpler GPU-friendly option to TextCategorizer.
• Fix issue #1773: Prevent tokenizer exceptions from setting POS but not TAG.
• Fix issue #1782, #2343: Fix training on GPU.
• Fix issue #1816: Allow custom Language subclasses via entry points.
• Fix issue #1865: Correct licensing of it_core_news_sm model.
• Fix issue #1889: Make stop words case-insensitive.
• Fix issue #1903: Add relcl dependency label to symbols.
• Fix issue #1963: Resize Doc.tensor when merging spans.
• Fix issue #1971: Update Matcher engine to support regex, extension attributes and rich comparison.
• Fix issue #2014: Make Token.pos_ writeable.
• Fix issue #2091: Fix displacy support for RTL languages.
• Fix issue #2203, #3268: Prevent bad interaction of lemmatizer and tokenizer exceptions.
• Fix issue #2329: Correct TextCategorizer and GoldParse API docs.
• Fix issue #2369: Respect pre-defined warning filters.
• Fix issue #2390: Support setting lexical attributes during retokenization.
• Fix issue #2396: Fix Doc.get_lca_matrix.
• Fix issue #2464, #3009: Fix behaviour of Matcher's ? quantifier.
• Fix issue #2482: Fix serialization when parser model is empty.
• Fix issue #2512, #2153: Fix issue with deserialization into non-empty vocab.
• Fix issue #2603: Improve handling of missing NER tags.
• Fix issue #2644: Add table explaining training metrics to docs.
• Fix issue #2648: Fix KeyError in Vectors.most_similar.
• Fix issue #2671, #2675: Fix incorrect match ID on some patterns.
• Fix issue #2693: Only use 'sentencizer' as built-in sentence boundary component name.
• Fix issue #2728: Fix HTML escaping in displacy NER visualization and correct API docs.
• Fix issue #2740: Add ability to pass additional arguments to pipeline components.
• Fix issue #2754, #3028: Make NORM a Token attribute instead of a Lexeme attribute to allow setting context-specific norms in tokenizer exceptions.
• Fix issue #2769: Fix issue that'd cause segmentation fault when calling EntityRecognizer.add_label.
• Fix issue #2772: Fix bug in sentence starts for non-projective parses.
• Fix issue #2779: Fix handling of pre-set entities.
• Fix issue #2782: Make like_num work with prefixed numbers.
• Fix issue #2833: Raise better error if Token or Span are pickled.
• Fix issue #2838: Add Retokenizer.split method to split one token into several.
• Fix issue #2869: Make doc[0].is_sent_start == True.
• Fix issue #2870: Make it illegal for the entity recognizer to predict whitespace tokens as B, L or U.
• Fix issue #2871: Fix vectors for reserved words.
• Fix issue #2901: Fix issue with first call of nlp in Japanese (MeCab).
• Fix issue #2924: Make IDs of displaCy arcs more unique to avoid clashes.
• Fix issue #3012: Fix clobber of Doc.is_tagged in Doc.from_array.
• Fix issue #3027: Allow Span to take unicode value for label argument.
• Fix issue #3036: Support mutable default arguments in extension attributes.
• Fix issue #3048: Raise better errors for uninitialized pipeline components.
• Fix issue #3064: Allow single string attributes in Doc.to_array.
• Fix issue #3093, #3067: Set vectors.name correctly when exporting model via CLI.
• Fix issue #3112: Make sure entity types are added correctly on GPU.
• Fix issue #3191: Fix pickling of Japanese.
• Fix issue #3122: Correct docs of Token.subtree and Span.subtree.
• Fix issue #3128: Improve error handling in converters.
• Fix issue #3248: Fix PhraseMatcher pickling and make __len__ consistent.
• Fix issue #3274: Make Token.sent work as expected without the parser.
• Fix issue #3277: Add en/em dash to tokenizer prefixes and suffixes.
• Fix issue #3346: Expose Japanese stop words in language class.
• Fix issue #3357: Update displaCy examples in docs to correctly show Token.pos_.
• Fix issue #3345: Fix NER when preset entities cross-sentence boundaries.
• Fix issue #3348: Don't use numpy directly for similarity.
• Fix issue #3366: Improve converters, training data formats and docs.
• Fix issue #3369: Fix #egg fragments in direct downloads.
• Fix issue #3382: Make Doc.from_array consistent with Doc.to_array.
• Fix issue #3398: Don't set extension attributes in language classes.
• Fix issue #3373: Merge and improve conllu converters.
• Fix serialization of custom tokenizer if not all functions are defined.
• Fix bugs in beam-search training objective.
• Fix problems with model pickling.

⚠️ Backwards incompatibilities

• This version of spaCy requires downloading new models. You can use the spacy validate command to find out which models need updating, and print update instructions.
• If you've been training your own models, you'll need to retrain them with the new version.
• Due to difficulties linking our new blis for faster platform-independent matrix multiplication, v2.1.x currently doesn't work on Python 2.7 on Windows. We expect this to be corrected in the future.
• While the Matcher API is fully backwards compatible, its algorithm has changed to fix a number of bugs and performance issues. This means that the Matcher in v2.1.x may produce different results compared to the Matcher in v2.0.x.

• The deprecated Doc.merge and Span.merge methods still work, but you may notice that they now run slower when merging many objects in a row. That's because the merging engine was rewritten to be more reliable and to support more efficient merging in bulk. To take advantage of this, you should rewrite your logic to use the Doc.retokenize context manager and perform as many merges as possible together in the with block.

- doc[1:5].merge()
- doc[6:8].merge()
+ with doc.retokenize() as retokenizer:
+     retokenizer.merge(doc[1:5])
+     retokenizer.merge(doc[6:8])

• The serialization methods to_disk, from_disk, to_bytes and from_bytes now support a single exclude argument to provide a list of string names to exclude. The docs have been updated to list the available serialization fields for each class. The disable argument on the Language serialization methods has been renamed to exclude for consistency.

- nlp.to_disk("/path", disable=["parser", "ner"])
+ nlp.to_disk("/path", exclude=["parser", "ner"])
- data = nlp.tokenizer.to_bytes(vocab=False)
+ data = nlp.tokenizer.to_bytes(exclude=["vocab"])

• The .pos value for several common English words has changed, due to corrections to long-standing mistakes in the English tag map (see #593, #3311).
• For better compatibility with the Universal Dependencies data, the lemmatizer now preserves capitalization, e.g. for proper nouns (see #3256).
• The keyword argument n_threads on the .pipe methods is now deprecated, as the v2.x models cannot release the global interpreter lock. (Future versions may introduce a n_process argument for parallel inference via multiprocessing.)
• The Doc.print_tree method is not deprecated in favour of a unified Doc.to_json method, which outputs data in the same format as the expected JSON training data.
• The built-in rule-based sentence boundary detector is now only called 'sentencizer' – the name 'sbd' is deprecated.

- sentence_splitter = nlp.create_pipe('sbd')
+ sentence_splitter = nlp.create_pipe('sentencizer')

• The is_sent_start attribute of the first token in a Doc now correctly defaults to True. It previously defaulted to None.
• The spacy train command now lets you specify a comma-separated list of pipeline component names, instead of separate flags like --no-parser to disable components. This is more flexible and also handles custom components out-of-the-box.

- $ spacy train en /output train_data.json dev_data.json --no-parser
+ $ spacy train en /output train_data.json dev_data.json --pipeline tagger,ner

• The spacy init-model command now uses a --jsonl-loc argument to pass in a a newline-delimited JSON (JSONL) file containing one lexical entry per line instead of a separate --freqs-loc and --clusters-loc.

- $ spacy init-model en ./model --freqs-loc ./freqs.txt --clusters-loc ./clusters.txt
+ $ spacy init-model en ./model --jsonl-loc ./vocab.jsonl

• Also note that some of the model licenses have changed: it_core_news_sm is now correctly licensed under CC BY-NC-SA 3.0, and all English and German models are now published under the MIT license.

📈 Benchmarks

Model	Language	Version	UAS	LAS	POS	NER F	Vec	Size
en_core_web_sm	English	2.1.0	91.5	89.7	96.8	85.9	𐄂	10 MB
en_core_web_md	English	2.1.0	91.8	90.0	96.9	86.6	✓	90 MB
en_core_web_lg	English	2.1.0	91.8	90.1	97.0	86.6	✓	788 MB
de_core_news_sm	German	2.1.0	90.7	88.6	96.3	83.1	𐄂	10 MB
de_core_news_md	German	2.1.0	91.2	89.4	96.6	83.8	✓	210 MB
es_core_news_sm	Spanish	2.1.0	90.4	87.3	96.9	89.5	𐄂	10 MB
es_core_news_md	Spanish	2.1.0	91.0	88.2	97.2	89.7	✓	69 MB
pt_core_news_sm	Portuguese	2.1.0	89.1	85.9	80.4	88.9	𐄂	12 MB
fr_core_news_sm	French	2.1.0	87.6	84.7	94.5	82.6	𐄂	14 MB
fr_core_news_md	French	2.1.0	89.1	86.4	95.3	83.1	✓	82 MB
it_core_news_sm	Italian	2.1.0	91.0	87.3	95.8	86.1	𐄂	10 MB
nl_core_news_sm	Dutch	2.1.0	83.7	77.6	91.6	87.0	𐄂	10 MB
el_core_news_sm	Greek	2.1.0	84.4	80.6	94.6	71.6	𐄂	10 MB
el_core_news_md	Greek	2.1.0	88.3	85.0	96.6	81.1	✓	126 MB
xx_ent_wiki_sm	Multi	2.1.0	-	-	-	81.3	𐄂	3 MB

💬 UAS: Unlabelled dependencies (parser). LAS: Labelled dependencies (parser). POS: Part-of-speech tags (fine-grained tags, i.e. Token.tag_). NER F: Named entities (F-score). Vec: Model contains word vectors. Size: Model file size (zipped archive).

📖 Documentation and examples

Although it looks pretty much the same, we've rebuilt the entire documentation using Gatsby and MDX. It's now an even faster progressive web app and allows us to write all content entirely in Markdown, without having to compromise on easy-to-use custom UI components. We're hoping that the Markdown source will make it even easier to contribute to the documentation. For more details, check out the styleguide and source.

While converting the pages to Markdown, we've also fixed a bunch of typos, improved the existing pages and added some new content:

• Usage Guide: Rule-based Matching. How to use the Matcher, PhraseMatcher and the new EntityRuler, and write powerful components to combine statistical models and rules.
• Usage Guide: Saving and Loading. Everything you need to know about serialization, and how to save and load pipeline components, package your spaCy models as Python modules and use entry points.
• Usage Guide: Merging and Splitting. How to retokenize a Doc using the new retokenize context manager and merge spans into single tokens and split single tokens into multiple.
• Universe: Videos and Podcasts
• API: EntityRuler
• API: SentenceSegmenter
• API: Pipeline functions

👥 Contributors

Thanks to @DuyguA, @giannisdaras, @mgogoulos, @louridas, @skrcode, @gavrieltal, @svlandeg, @jarib, @alvaroabascar, @kbulygin, @moreymat, @mirfan899, @ozcankasal, @willprice, @alvations, @amperinet, @retnuh, @Loghijiaha, @DeNeutoy, @gavrieltal, @boena, @BramVanroy, @pganssle, @foufaster, @adrianeboyd, @maknotavailable, @pierremonico, @lauraBaakman, @juliamakogon, @Gizzio, @Abhijit-2592, @akki2825, @grivaz, @roshni-b, @mpuig, @mikelibg, @danielkingai2, @adrienball and @Poluglottos for the pull requests and contributions.

✨ New features and improvements

• NEW: Alpha tokenization support for Catalan.
• Improve French tokenization.
• Fix regex pin to harmonise dependencies with conda.
• Fix msgpack pin.
• Update tests for pytest 4.0.

🔴 Bug fixes

• Fix issue #2933: Correct mistake in is_ascii documentation.
• Fix issue #2976: Fix bug where Vocab.prune_vectors did not use batch_size.
• Fix issue #2986: Correctly document when Span.ents was added.
• Fix issue #2995, #2996: Fix msgpack pin.

📖 Documentation and examples

• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @mpuig, @ALSchwalm, @bpben, @svlandeg and @wxv for the pull requests and contributions.

✨ New features and improvements

• Make max_length of input text inclusive.
• Raise error when setting overlapping entities as doc.ents.
• Improve French lemmatization and check if a word is in one of the regular lists specific to each part-of-speech tag.

🔴 Bug fixes

• Fix issue #1581, #1969, #1986: Fix out-of-bounds access in NER training that'd cause segmentation fault.
• Fix issue #2924: Prevent problem where displacy arcs would receive the same IDs in Jupyter notebooks, causing weirdly positioned arc labels.
• Fix issue #2948: Fix problem with symlink creation on Windows.

📖 Documentation and examples

• Fix various typos and inconsistencies.
• Update spaCy Universe with new projects.
• Add example script showing a fix-up rule for whitespace entities like '\n'.

👥 Contributors

Thanks to @digest0r, @BramVanroy, @grivaz, @wannaphongcom, @mikelibg, @danielhers, @frascuchon, @mauryaland and @cicorias for the pull requests and contributions.

🔴 Bug fixes

• Fix msgpack-numpy pin, which could affect serialization on Python 2.7.

✨ New features and improvements

• Improve version compatibility to support wheels for all spaCy dependencies maintained by us: thinc, cymem, preshed and murmurhash.
• Support GPU installation by specifying spacy[cuda], spacy[cuda90], spacy[cuda91], spacy[cuda92] or spacy[cuda10], which will install cupy and thinc_gpu_ops.
• Add spacy.prefer_gpu() and spacy.require_gpu() functions.

📖 Documentation and examples

• Update GPU installation and usage docs.

✨ New features and improvements

• NEW: Pre-built wheels and up to 10 times faster installation! This release starts the journey towards pre-built wheels for all of spaCy's dependencies. Once that's completed, you won't even need a local compiler anymore to install the library. For more details on our wheels process, see explosion/wheelwright.
• NEW: Alpha support for Telugu and Sinhala.
• NEW: Rule-based lemmatization for Greek and French.
• Port over Chinese support (#1210) from v1.x.
• Improve language data for Persian, Greek, Swedish, Bengali, Polish, Portuguese, Indonesian, French, German and Russian.
• Add Span.ents property for consistency with Doc.ents.
• Add --verbose option to spacy train to output more details for debugging.

🔴 Bug fixes

• Fix issue #653: Introduce bulk merge function.
• Fix issue #1445, #1917, #2209, #2362, #2371, #2383, #2501, #2743, #2758: Fix Keras examples.
• Fix issue #2261, #2800: Fix bug that could cause a crash with too many entity types.
• Fix issue #2540: Improve French stop words.
• Fix issue #2582, #2640, #2645, #2657, #2705, #2784, #2815, #2841, #2845: Fix typos and inconsistencies in documentation.
• Fix issue #2593: Prevent numpy warning.
• Fix issue #2706: Add missing label FAC to spacy.explain glossary.
• Fix issue #2709: Pass default option when calling getoption() in conftest.py.

📖 Documentation and examples

• Improve Keras examples.
• Update training examples to use minibatching.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @DimaBryuhanov, @kororo, @AndriyMulyar, @katarkor, @giannisdaras, @bphi, @vikaskyadav, @sammous, @EmilStenstrom, @howl-anderson, @ohenrik, @aashishg, @aryaprabhudesai, @steve-prod, @njsmith, @aniruddha-adhikary, @pzelasko, @mbkupfer, @sainathadapa, @tyburam, @grivaz, @filipecaixeta, @aongko, @free-variation, @mauryaland, @pmj642, @keshan, @darindf, @charlax, @phojnacki, @skrcode, @jacopofar, @Cinnamy and @JKhakpour for the pull requests and contributions!