NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #2063 most downloaded on PyPI
Python framework for fast Vector Space Modelling
Last release 11 months ago
18 Oct 2025
Ships fairly regularly
a new release about every 7 months
Nearly every release is documented
notes for 59 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
17 years old
80 releases · first in 2010
Removed all code, methods, attributes and functions marked as deprecated in Gensim 3.8.3.
⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.
Gensim 4.0 is a major release with lots of performance & robustness improvements, and a new website.
Massively optimized popular algorithms the community has grown to love: fastText, word2vec, doc2vec, phrases:
a. Efficiency
| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|---|---|---|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / 1.26 GB / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | 1.2h / 0.33 GB / 1762k words/s |
In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. (4.0 benchmarks)
b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)
c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.
These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.
Dropped a bunch of externally contributed modules and wrappers: summarization, pivoted TFIDF, Mallet…
Code quality was not up to our standards. Also there was no one to maintain these modules, answer user questions, support them.
So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them, please fork & publish into your own repo. They can live happily outside of Gensim.
Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.
A new Gensim website – finally! 🙃
So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.
This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting concrete NLP & document similarity use-cases.
max_final_vocab parameter in fastText constructor, by @mpenkovalpha parameter in LDA model, by @xh2save_facebook_model failure after update-vocab & other initialization streamlining, by @gojomoxml.etree.cElementTree, by @hugovksimilarities.index to the more appropriate similarities.annoy, by @piskvorkynum_words to topn in dtm_coherence, by @MeganStodelon_batch_begin and on_batch_end callbacks, by @mpenkovpattern dependency, by @mpenkovgensim.viz subpackage, by @mpenkovOne column per quarter.
Prepare for removal of deprecated lxml.etree.cElementTree (PR #2777, __@tirkarthi__)
This is primarily a bugfix release to bring back Py2.7 compatibility to gensim 3.8.
lxml.etree.cElementTree (PR #2777, @tirkarthi)Remove
gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utils"deprecated" functions and attributes
smart_open version for compatibility with Py2.7Remove
gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utils"deprecated" functions and attributes
Remove
gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsFix smart_open deprecation warning globally (__itayB__, #2530)
gensim.downloader to run offline, by introducing a local file cache (mpenkov, #2545)gensim.downloader target directory configurable (mpenkov, #2456)smart_open deprecation warning globally (itayB, #2530)topn=0 versus topn=None bug in most_similar, accept topn of any integer type (Witiko, #2497)CHANGELOG.md (mpenkov, #2482)gensim.similarities.termsim module (Witiko, #2485)Support section in README (piskvorky, #2542)Remove
gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utils"deprecated" functions and attributes
WordEmbeddingsKeyedVectors.most_similar (Witiko, #2461)matutils.unitvec always return float norm when requested (Witiko, #2419)Doc2Vec.docvecs comment (gojomo, #2472)Remove
gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsThe gensim.models.FastText.load_fasttext_format function (deprecated) now loads the entire model contained in the .bin file, including the shallow neu…
gensim.models.fasttext.load_facebook_model function: load full model (slower, more CPU/memory intensive, supports training continuation)
>>> from gensim.test.utils import datapath
>>>
>>> cap_path = datapath("crime-and-punishment.bin")
>>> fb_model = load_facebook_model(cap_path)
>>>
>>> 'landlord' in fb_model.wv.vocab # Word is out of vocabulary
False
>>> oov_term = fb_model.wv['landlord']
>>>
>>> 'landlady' in fb_model.wv.vocab # Word is in the vocabulary
True
>>> iv_term = fb_model.wv['landlady']
>>>
>>> new_sent = [['lord', 'of', 'the', 'rings'], ['lord', 'of', 'the', 'flies']]
>>> fb_model.build_vocab(new_sent, update=True)
>>> fb_model.train(sentences=new_sent, total_examples=len(new_sent), epochs=5)
gensim.models.fasttext.load_facebook_vectors function: load embeddings only (faster, less CPU/memory usage, does not support training continuation)
>>> fbkv = load_facebook_vectors(cap_path)
>>>
>>> 'landlord' in fbkv.vocab # Word is out of vocabulary
False
>>> oov_vector = fbkv['landlord']
>>>
>>> 'landlady' in fbkv.vocab # Word is in the vocabulary
True
>>> iv_vector = fbkv['landlady']
To achieve consistency with the reference implementation from Facebook,
a FastText model will now always report any word, out-of-vocabulary or
not, as being in the model, and always return some vector for any word
looked-up. Specifically:
'any_word' in ft_model will always return True. Previously, it
returned True only if the full word was in the vocabulary. (To test if a
full word is in the known vocabulary, you can consult the wv.vocab
property: 'any_word' in ft_model.wv.vocab will return False if the full
word wasn't learned during model training.)ft_model['any_word'] will always return a vector. Previously, it
raised KeyError for OOV words when the model had no vectors
for any ngrams of the word.The gensim.models.FastText.load_fasttext_format function (deprecated) now loads the entire model contained in the .bin file, including the shallow neural network that enables training continuation.
Loading this NN requires more CPU and RAM than previously required.
Since this function is deprecated, consider using one of its alternatives (see below).
Furthermore, you must now pass the full path to the file to load, including the file extension. Previously, if you specified a model path that ends with anything other than .bin, the code automatically appended .bin to the path before loading the model. This behavior was confusing, so we removed it.
Remove
gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utils"deprecated" functions and attributes
FastText.load_fasttext_model (@mpenkov, #2340)Doc2Vec.infer_vector (@tobycheese, #2347)LdaSeqModel (@horpto, #2360)process_result_queue from cycle in LdaMulticore (@horpto, #2358)LdaModel.do_mstep (@horpto, #2344)FastTextKeyedVectors using KeyedVectors (missing attribute compatible_hash) (@menshikh-iv, #2349)WordEmbeddingsKeyedVectors.most_similar (@Witiko, #2356)flake8==3.7.1 (@horpto, #2365)FastText documentation (@mpenkov, #2353)Any*Vec docstrings (@tobycheese, #2345)poincare documentation to indicate the relation format (@AMR-KELEG, #2357)Remove
gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsFix deprecation warning np.sum(generator) (__@rsdel2007__, #2296)
Fast Online NMF (@anotherbugmaster, #2007)
Benchmark wiki-english-20171001
| Model | Perplexity | Coherence | L2 norm | Train time (minutes) |
|---|---|---|---|---|
| LDA | 4727.07 | -2.514 | 7.372 | 138 |
| NMF | 975.74 | -2.814 | 7.265 | 73 |
| NMF (with regularization) | 985.57 | -2.436 | 7.269 | 441 |
Simple to use (same interface as LdaModel)
from gensim.models.nmf import Nmf
from gensim.corpora import Dictionary
import gensim.downloader as api
text8 = api.load('text8')
dictionary = Dictionary(text8)
dictionary.filter_extremes()
corpus = [
dictionary.doc2bow(doc) for doc in text8
]
nmf = Nmf(
corpus=corpus,
num_topics=5,
id2word=dictionary,
chunksize=2000,
passes=5,
random_state=42,
)
nmf.show_topics()
"""
[(0, '0.007*"km" + 0.006*"est" + 0.006*"islands" + 0.004*"league" + 0.004*"rate" + 0.004*"female" + 0.004*"economy" + 0.003*"male" + 0.003*"team" + 0.003*"elections"'),
(1, '0.006*"actor" + 0.006*"player" + 0.004*"bwv" + 0.004*"writer" + 0.004*"actress" + 0.004*"singer" + 0.003*"emperor" + 0.003*"jewish" + 0.003*"italian" + 0.003*"prize"'),
(2, '0.036*"college" + 0.007*"institute" + 0.004*"jewish" + 0.004*"universidad" + 0.003*"engineering" + 0.003*"colleges" + 0.003*"connecticut" + 0.003*"technical" + 0.003*"jews" + 0.003*"universities"'),
(3, '0.016*"import" + 0.008*"insubstantial" + 0.007*"y" + 0.006*"soviet" + 0.004*"energy" + 0.004*"info" + 0.003*"duplicate" + 0.003*"function" + 0.003*"z" + 0.003*"jargon"'),
(4, '0.005*"software" + 0.004*"games" + 0.004*"windows" + 0.003*"microsoft" + 0.003*"films" + 0.003*"apple" + 0.003*"video" + 0.002*"album" + 0.002*"fiction" + 0.002*"characters"')]
"""
See also:
Massive improvement of FastText compatibilities (@mpenkov, #2313)
from gensim.models import FastText
# 'cc.ru.300.bin' - Russian Facebook FT model trained on Common Crawl
# Can be downloaded from https://s3-us-west-1.amazonaws.com/fasttext-vectors/word-vectors-v2/cc.ru.300.bin.gz
model = FastText.load_fasttext_format("cc.ru.300.bin")
# Fixed hash-function allow to produce same output as FB FastText & works correctly for non-latin languages (for example, Russian)
assert "мяу" in m.wv.vocab # 'мяу' - vocab word
model.wv.most_similar("мяу")
"""
[('Мяу', 0.6820122003555298),
('МЯУ', 0.6373013257980347),
('мяу-мяу', 0.593108594417572),
('кис-кис', 0.5899622440338135),
('гав', 0.5866007804870605),
('Кис-кис', 0.5798211097717285),
('Кис-кис-кис', 0.5742273330688477),
('Мяу-мяу', 0.5699705481529236),
('хрю-хрю', 0.5508339405059814),
('ав-ав', 0.5479759573936462)]
"""
assert "котогород" not in m.wv.vocab # 'котогород' - out-of-vocab word
model.wv.most_similar("котогород", topn=3)
"""
[('автогород', 0.5463314652442932),
('ТагилНовокузнецкНовомосковскНовороссийскНовосибирскНовотроицкНовочеркасскНовошахтинскНовый',
0.5423436164855957),
('областьНовосибирскБарабинскБердскБолотноеИскитимКарасукКаргатКуйбышевКупиноОбьТатарскТогучинЧерепаново',
0.5377570390701294)]
"""
# Now we load full model, for this reason, we can continue an training
from gensim.test.utils import datapath
from smart_open import smart_open
with smart_open(datapath("crime-and-punishment.txt"), encoding="utf-8") as infile: # russian text
corpus = [line.strip().split() for line in infile]
model.train(corpus, total_examples=len(corpus), epochs=5)
Similarity search improvements (@Witiko, #2016)
Add similarity search using the Levenshtein distance in gensim.similarities.LevenshteinSimilarityIndex
Performance optimizations to gensim.similarities.SoftCosineSimilarity (full benchmark)
| dictionary size | corpus size | speed |
|---|---|---|
| 1000 | 100 | 1.0× |
| 1000 | 1000 | 53.4× |
| 1000 | 100000 | 156784.8× |
| 100000 | 100 | 3.8× |
| 100000 | 1000 | 405.8× |
| 100000 | 100000 | 66262.0× |
See updated soft-cosine tutorial for more information and usage examples
Add python3.7 support (@menshikh-iv, #2211)
Phraser memory usage (drop frequencies) (@jenishah, #2208)ldamodel.update_dir_prior (@horpto, #2274)KeyedVector.wmdistance (@horpto, #2326)remove_unreachable_nodes in gensim.summarization (@horpto, #2263)mz_entropy from gensim.summarization (@horpto, #2267)filter_extremes methods in Dictionary and HashDictionary (@horpto, #2303)KeyedVectors.relative_cosine_similarity (@rsdel2007, #2307)random_seed to LdaMallet (@Zohaggie & @menshikh-iv, #2153)common_terms parameter to sklearn_api.PhrasesTransformer (@pmlk, #2074)corpora.Dictionary based on special tokens (@Froskekongen, #2200)six usage (xrange, map, zip) (@horpto, #2264)line2doc methods of LowCorpus and MalletCorpus (@horpto, #2269)PYTHONHASHSEED) (@menshikh-iv, #2196)__getitem__ code duplication in gensim.models.phrases (@jenishah, #2206)flake8-rst for docstring code examples (@kataev, #2192)py26 stuff (@menshikh-iv, #2214)itertools.chain instead of sum to concatenate lists (@Stigjb, #2212)utils.get_max_id (@horpto, #2254)np.sum(generator) (@rsdel2007, #2296)BM25 (@horpto, #2275)metadata=True for make_wikicorpus script by default (@Xinyi2016, #2245)Phrases (@rsdel2007, #2331)open() by smart_open() in gensim.models.fasttext._load_fasttext_format (@rsdel2007, #2335)*Vec corpusfile-based training (@bm371613, #2239)malletmodel2ldamodel conversion (@horpto, #2288)LdaModel (@horpto, #2308)SvmLightCorpus.serialize if labels instance of numpy.ndarray (@aquatiko, #2243)plotly>=3.0.0 (@jenishah, #2226)keep_n behavior for Dictionary.filter_extremes (@johann-petrak, #2232)sphinx==1.8.1 (last r (@menshikh-iv, #None)np.issubdtype warnings (@marioyc, #2210)-c from gensim.downloader description (@horpto, #2262)viz.line() instead of viz.updatetrace()) (@allenyllee, #2252)gensim.downloader & fix rendering of code examples (@menshikh-iv, #2327)gensim.models (@rsdel2007, #2323)Doc2Vec documentation: how tags are assigned in corpus_file mode (@persiyanov, #2320)gensim/models/keyedvectors.py (@rsdel2007, #2290)Phrases (@jenishah, #2242)KeyedVectors.evaluate_word_* (@Stigjb, #2205)KeyedVector.evaluate_word_analogies (@Stigjb, #2207)WmdSimilarity documentation (@jagmoreira, #2217)fify -> fifty in gensim.parsing.preprocessing.STOPWORDS (@coderwassananmol, #2220)alpha="auto" from LdaMulticore (not supported yet) (@johann-petrak, #2225)tutorials.md (@rsdel2007, #2302)Remove
gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsReplace deprecated parameters with new in docstring of gensim.models.Doc2Vec (__@xuhdev__, #2165)
File-based training for *2Vec models (@persiyanov, #2127 & #2078 & #2048)
New training mode for *2Vec models (word2vec, doc2vec, fasttext) that allows model training to scale linearly with the number of cores (full GIL elimination). The result of our Google Summer of Code 2018 project by Dmitry Persiyanov.
Benchmark on the full English Wikipedia, Intel(R) Xeon(R) CPU @ 2.30GHz 32 cores (GCE cloud), MKL BLAS:
| Model | Queue-based version [sec] | File-based version [sec] | speed up | Accuracy (queue-based) | Accuracy (file-based) |
|---|---|---|---|---|---|
| Word2Vec | 9230 | 2437 | 3.79x | 0.754 (± 0.003) | 0.750 (± 0.001) |
| Doc2Vec | 18264 | 2889 | 6.32x | 0.721 (± 0.002) | 0.683 (± 0.003) |
| FastText | 16361 | 10625 | 1.54x | 0.642 (± 0.002) | 0.660 (± 0.001) |
Usage:
import gensim.downloader as api
from multiprocessing import cpu_count
from gensim.utils import save_as_line_sentence
from gensim.test.utils import get_tmpfile
from gensim.models import Word2Vec, Doc2Vec, FastText
# Convert any corpus to the needed format: 1 document per line, words delimited by " "
corpus = api.load("text8")
corpus_fname = get_tmpfile("text8-file-sentence.txt")
save_as_line_sentence(corpus, corpus_fname)
# Choose num of cores that you want to use (let's use all, models scale linearly now!)
num_cores = cpu_count()
# Train models using all cores
w2v_model = Word2Vec(corpus_file=corpus_fname, workers=num_cores)
d2v_model = Doc2Vec(corpus_file=corpus_fname, workers=num_cores)
ft_model = FastText(corpus_file=corpus_fname, workers=num_cores)
FastText (@mcemilg, #2178)BM25 (@Shiki-H, #2146)name_only option for downloader api (@aneesh-joshi, #2143)word2vec2tensor script compatible with python3 (@vsocrates, #2147)Wikicorpus (@mattilyra, #2089)similarity_matrix support non-contiguous dictionaries (@Witiko, #2047)AuthorTopicModel (@philipphager, #2122)AuthorTopicModel (@probinso, #2133)keywords issue with short input (@LShostenko, #2154)min_count handling in phrases detection using npmi_scorer (@lopusz, #2072)Phraser log message (@robguinness, #2151)np.integer -> np.int in AuthorTopicModel (@menshikh-iv, #2145)prune_at parameter description for gensim.corpora.Dictionary (@yxonic, #2128)default -> auto prior parameter in documentation for lda-related models (@Laubeee, #2156)gensim.models.translation_matrix (@nzw0301, #2164)gensim.models.Word2Vec (@nzw0301, #2161)gensim.models.Doc2Vec (@xuhdev, #2165)Phrases (@RunHorst, #2148)Remove
gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsFix deprecated parameters in D2VTransformer and W2VTransformer(__@MritunjayMohitesh__, #1945)
This release comprises a glorious 38 pull requests from 28 contributors. Most of the effort went into improving the documentation—hence the release code name "Docs 💬"!
Apart from the massive overhaul of all Gensim documentation (including docstring style and examples—you asked for it), we also managed to sneak in some new functionality and a number of bug fixes. As usual, see the notes below for a complete list, with links to pull requests for more details.
Huge thanks to all contributors! Nobody loves working on documentation. 3.5.0 is a result of several months of laborious, unglamorous, and sometimes invisible work. Enjoy!
*2vec models (@steremma & @piskvorky & @menshikh-iv, #1944, #2087)gensim.models.phrases (@CLearERR & @menshikh-iv, #1950)gensim.models.AuthorTopicModel (@souravsingh & @menshikh-iv, #1907)gensim.similarities.docsim (@CLearERR & @menshikh-iv, #2030)IndexedCorpus (@darindf, #2033)gensim.models.coherencemodel (@CLearERR & @menshikh-iv, #1933)gensim.sklearn_api (@steremma & @menshikh-iv, #1895)gensim.models.KeyedVectors.similarity_matrix (@Witiko, #1971)smart_open() instead of open() in notebooks (@sharanry, #1812)add_entity method to KeyedVectors to allow adding word vectors manually (@persiyanov, #1957)AuthorTopicModel (@Stamenov, #1766)evaluate_word_analogies (will replace accuracy) method to KeyedVectors (@akutuzov, #1935)TfidfModel (@markroxor, #1780)max_final_vocab in lieu of min_count in Word2Vec(@aneesh-joshi, #1915)dtype argument for chunkize_serial in LdaModel (@darindf, #2027)Phrases.analyze_sentence (@JonathanHourany, #2070)ns_exponent parameter to control the negative sampling distribution for *2vec models (@fernandocamargoti, #2093)Doc2Vec.infer_vector + notebook cleanup (@gojomo, #2103)Doc2Vec.infer_vector (@umangv, #2063)word2vec and doc2vec models saved using old Gensim versions (@manneshiva, #2012)SoftCosineSimilarity.get_similarities on corpora ssues/1955) (@Witiko, #1972)matutils.unitvec according to input dtype (@o-P-o, #1992)gensim.corpora.WikiCorpus (@steremma, #2042)Similarity.query_shards in multiprocessing case (@bohea, #2044)df == "n" (@PeteBleackley, #2021)_is_single from Phrases for case when corpus is a NumPy array (@rmalouf, #1987)EuclideanKeyedVectors.similarity_matrix (@Witiko, #1984)D2VTransformer and W2VTransformer(@MritunjayMohitesh, #1945)Doc2Vec.infer_vector after loading old Doc2Vec (gensim<=3.2)(@manneshiva, #1974)load_word2vec_format (@DennisChen0307, #1968)keras==2.1.5) (@menshikh-iv, #1963)Remove
gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsFix deprecation warning from inspect.getargspec. Fix #1878 (__@aneesh-joshi__, #1887)
Massive optimizations of gensim.models.LdaModel: much faster training, using Cython. (@arlenk, #1767)
Training benchmark :boom:
| dataset | old LDA [sec] | optimized LDA [sec] | speed up |
|---|---|---|---|
| nytimes | 3473 | 1975 | 1.76x |
| enron | 774 | 437 | 1.77x |
This change affects all models that depend on LdaModel, such as LdaMulticore, LdaSeqModel, AuthorTopicModel.
Huge speed-ups to corpus I/O with MmCorpus (Cython) (@arlenk, #1825)
File reading benchmark
| dataset | file compressed? | old MmReader [sec] | optimized MmReader [sec] | speed up |
|---|---|---|---|---|
| enron | no | 22.3 | 2.6 | 8.7x |
| yes | 37.3 | 14.4 | 2.6x | |
| nytimes | no | 419.3 | 49.2 | 8.5x |
| yes | 686.2 | 275.1 | 2.5x | |
| text8 | no | 25.4 | 2.5 | 10.1x |
| yes | 41.9 | 17.0 | 2.5x |
Overall, a 2.5x speedup for compressed .mm.gz input and 8.5x :fire::fire::fire: for uncompressed plaintext .mm.
Performance and memory optimization to gensim.models.FastText :rocket: (@jbaiter, #1916)
Benchmark (first 500,000 articles from English Wikipedia)
| Metric | old FastText | optimized FastText | improvement |
|---|---|---|---|
| Training time (1 epoch) | 4823.4s (80.38 minutes) | 1873.6s (31.22 minutes) | 2.57x |
| Training time (full) | 1h 26min 13s | 36min 43s | 2.35x |
| Training words/sec | 72,781 | 187,366 | 2.57x |
| Training peak memory | 5.2 GB | 3.7 GB | 1.4x |
Overall, a 2.5x speedup & memory usage reduced by 30%.
Implemented Soft Cosine Measure (@Witiko, #1827)
New method for assessing document similarity, a nice faster alternative to WMD, Word Mover's Distance
Benchmark
| Technique | MAP score | Duration |
|---|---|---|
| softcossim | 45.99 | 1.24 sec |
| wmd-relax | 44.48 | 12.22 sec |
| cossim | 44.22 | 4.39 sec |
| wmd-gensim | 44.08 | 98.29 sec |
Soft Cosine notebook with detailed description, examples & benchmarks
Related papers:
python -m gensim.scripts.package_info --info. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903)license field to setup.py, allowing the use of tools like pip-licenses (@nils-werner, #1909)gensim.corpora.UciCorpus.save_corpus (@darindf, #1875)wv property to KeyedVectors for backward compatibility. Fix #1882 (@manneshiva, #1884)inspect.getargspec. Fix #1878 (@aneesh-joshi, #1887)LabeledSentence to gensim.models.doc2vec for backward compatibility. Fix #1886 (@manneshiva, #1891)Phrases (when using model[tokens] twice). Fix #1401 (@sj29-innovate, #1853)D2VTransformer.fit_transform. Fix #1834 (@Utkarsh-Mishra-CIC, #1845)datatype parameter for KeyedVectors.load_word2vec_format. Fix #1682 (@pushpankar, #1819)doc2vec-lee notebook (@TheFlash10, #1918)gensim.corpora.MmCorpus. Fix #1869 (@sj29-innovate, #1911)test_similarities.py, no more FP fails. (@menshikh-iv, #1928)WordEmbeddingsKeyedVectors.evaluate_word_pairs. (@akutuzov, #1934)gensim.models.Word2Vec (@nzw0301, #1870)doc2vec-lee notebook (@numericlee, #1870)README.md directly in repository. Fix #1849 (@ibrahimsharaf, #1861)CONTRIBUTING.md (@aneesh-joshi, #1880)sg parameter for gensim.models.word2vec (@mdcclv, #1919)gensim.similarities.docsim and MmCorpus-related. (@CLearERR & @menshikh-iv, #1910)gensim.test.utils (@yurkai & @menshikh-iv, #1904)gensim.scripts. Partial fix #1665 (@yurkai & @menshikh-iv, #1792)gensim.corpora. Partial fix #1671 (@CLearERR & @menshikh-iv, #1835)gensim.models.wrappers (@kakshay21 & @menshikh-iv, #1859)gensim.interfaces (@yurkai & @menshikh-iv, #1913)Remove
gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utils:warning: Deprecations (will be removed in the next major release)
:star2: New features:
Re-designed all "*2vec" implementations (@manneshiva, #1777)
Word2Vec, Doc2Vec, FastText, etc ..., making it easier to add new models in the future and re-use codeImprove gensim.scripts.segment_wiki by retaining interwiki links. Fix #1712
(@steremma, PR #1839)
Optionally extract interlinks from Wikipedia pages (use the --include-interlinks option). This will output one additional JSON dict for each article:
{
"interlinks": {
"article title 1": "interlink text 1",
"article title 2": "interlink text 2",
...
}
}
Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump:
python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz
segment_wiki output:import json
from smart_open import smart_open
with smart_open("enwiki-latest.jsonl.gz") as infile:
for doc in infile:
doc = json.loads(doc)
src_node = doc['title']
dst_nodes = doc['interlinks'].keys()
print(u"Source node: {}".format(src_node))
print(u"Destination nodes: {}".format(u", ".join(dst_nodes)))
break
"""
OUTPUT:
Source node: Anarchism
Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, Andalusia, Renzo Novatore, trade union, autonomist marxism, dictatorship of the proletariat, Mujeres Libres, Voltairine de Cleyre, Post-anarchism, participatory economics, Confederación Nacional del Trabajo, Syncretic politics, direct democracy, Jean-Jacques Rousseau, Green anarchism, Surrealism, labour unions, A. S. Neill, christian anarchist, Bonnot Gang, Anti-capitalism, Anarchism in Brazil, simple living, enlightened self-interest, Confédération générale du travail, class conflict, International Workers' Day, Hébertists, Gerrard Winstanley, Francoism, anarcho-pacifist, Andrej Grubacic, individualist anarchist and social anarchist thinkers., April Carter, private property, penal colonies, Libertarian socialism, Camillo Berneri, Christian anarchism, transhumanism, Lucifer, the Light-Bearer, Edna St. Vincent Millay, unschooling, Leo Tolstoy, M. E. Lazarus, Spanish Anarchists, Buddhist anarchism, ideology, William McKinley, anarcho-primitivism, Francesc Pi i Margall, :Category:Anarchism by country, International Workers Association, Anarcho-capitalism, Lois Waisbrooker, wikt:Solidarity, Baja California, social revolution, Unione Sindacale Italiana, Lev Chernyi, Alex Comfort, Sonnenburg, Leon Czolgosz, Volin, utopian, Argentine Libertarian Federation, Nudism, Left-wing market anarchism, insurrection, definitional concerns in anarchist theory, infinitive, affinity group, World Trade Organization Ministerial Conference of 1999 protest activity, class struggle, nonviolence, John Zerzan, poststructuralist, Noam Chomsky, Second Fitna, Julian Beck, Philadelphes, League of Peace and Freedom, Fédération Anarchiste, Kronstadt rebellion, Cold War, André Breton, Silvio Gesell, libertarian anarchism, voluntary association, anti-globalisation movement, birth control, L. Susan Brown, anarcho-naturism, personal property, Roundhead, Harold Barclay, The Joy of Sex, Council communism, Lucía Sánchez Saornil, tyrannicide, Neopaganism, lois scélérates, Johann Most, Anarchist Catalonia, Albert Camus, Protests of 1968, Alexander II of Russia, Spain's economy, Federazione Anarchica Italiana, Cuba, German Revolution of 1918–1919, stirner, Property is theft, Situationist International, law and economics
Add support for SMART notation for TfidfModel. Fix #1785 (@markroxor, #1791)
TfidfModel to allow different weighting and normalization schemesfrom gensim.corpora import Dictionary
from gensim.models import TfidfModel
import gensim.downloader as api
data = api.load("text8")
dct = Dictionary(data)
corpus = [dct.doc2bow(line) for line in data]
# Train Tfidf model using the SMART notation, smartirs="ntc" where
# 'n' - natural term frequency
# 't' - idf document frequency
# 'c' - cosine normalization
#
# More information about possible values available in documentation or https://nlp.stanford.edu/IR-book/html/htmledition/document-and-query-weighting-schemes-1.html
model = TfidfModel(corpus, id2word=dct, smartirs="ntc")
vectorized_corpus = list(model[corpus])
Add CircleCI for building Gensim documentation. Fix #1807 (@menshikh-iv, #1822)
:red_circle: Bug fixes:
get_my_ip. Fix #1771 (@darindf, #1772)gensim.summarization.bm25. Fix #1828 (@sj29-innovate, #1833)TranslationMatrix (@robotcator, #1838)gensim.models.CoherenceModel in gensim.models.callbacks (@Alexjmsherman, #1823)FastText.train. Fix #1818 (@sj29-innovate, #1837)score_function from LexicalEntailmentEvaluation. Fix #1858 (@hachibaka, #1863):books: Tutorial and doc improvements:
gensim.summarization (@yurkai & @menshikh-iv, #1709)gensim.similarities.index. Partial fix #1666 (@menshikh-iv, #1681)gensim.models.translation_matrix (@KokuKUSIAKU & @menshikh-iv, #1806)gensim.models.rpmodel (@jazzmuesli & @menshikh-iv, #1802)gensim.utils (@kakshay21 & @menshikh-iv, #1797)gensim.matutils (@Cheukting & @menshikh-iv, #1804)gensim.models.logentropy_model (@minggli & @menshikh-iv, #1803)gensim.models.normmodel (@AustenLamacraft & @menshikh-iv, #1805)gensim.topic_coherence. Fix #1669 (@CLearERR & @menshikh-iv, #1714)gensim.corpora.dictionary and gensim.corpora.hashdictionary. Partial fix #1671 (@CLearERR & @menshikh-iv, #1814)gensim.corpora. Partial fix #1671 (@anotherbugmaster & @menshikh-iv, #1729)model_to_dict one-liner to word2vec notebook. Fix #1269 (@kakshay21, #1776)sg parameter for gensim.models.FastText (@akutuzov, #1801)doc2vec-IMDB. Fix #1788 (@apoorvaeternity, #1796)bz2 + MmCorpus examples from tutorials (@menshikh-iv, #1867):+1: Improvements:
:warning: Deprecations (will be removed in the next major release)
Remove
gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsNew fast multithreaded implementation of FastText, natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation. ```py…
:star2: New features:
New download API for corpora and pre-trained models (@chaitaliSaini & @menshikh-iv, #1705 & #1632 & #1492)
import gensim.downloader as api
for article in api.load("wiki-english-20171001"):
print(article)
import gensim.downloader as api
model = api.load("glove-twitter-25")
model.most_similar("engineer")
# [('specialist', 0.957542896270752),
# ('developer', 0.9548177123069763),
# ('administrator', 0.9432312846183777),
# ('consultant', 0.93915855884552),
# ('technician', 0.9368376135826111),
# ('analyst', 0.9342101216316223),
# ('architect', 0.9257484674453735),
# ('engineering', 0.9159940481185913),
# ('systems', 0.9123805165290833),
# ('consulting', 0.9112802147865295)]
New model: Poincaré embeddings (@jayantj, #1696 & #1700 & #1757 & #1734)
from gensim.models.poincare import PoincareRelations, PoincareModel
from gensim.test.utils import datapath
data = PoincareRelations(datapath('poincare_hypernyms.tsv'))
model = PoincareModel(data)
model.kv.most_similar("cat.n.01")
# [('kangaroo.n.01', 0.010581353439700418),
# ('gib.n.02', 0.011171531439892076),
# ('striped_skunk.n.01', 0.012025106076442395),
# ('metatherian.n.01', 0.01246679759214648),
# ('mammal.n.01', 0.013281303506525968),
# ('marsupial.n.01', 0.013941330203709653)]
Optimized FastText (@manneshiva, #1742)
import gensim.downloader as api
from gensim.models import FastText
model = FastText(api.load("text8"))
model.most_similar("cat")
# [('catnip', 0.8538144826889038),
# ('catwalk', 0.8136177062988281),
# ('catchy', 0.7828493118286133),
# ('caf', 0.7826495170593262),
# ('bobcat', 0.7745151519775391),
# ('tomcat', 0.7732658386230469),
# ('moat', 0.7728310823440552),
# ('caye', 0.7666271328926086),
# ('catv', 0.7651021480560303),
# ('caveat', 0.7643581628799438)]
Binary pre-compiled wheels for Windows, OSX and Linux (@menshikh-iv, MacPython/gensim-wheels/#7)
Added DeprecationWarnings to deprecated methods and parameters, with a clear schedule for removal.
:+1: Improvements:
scan_vocab speed, build_vocab_from_freq method (@jodevak, #1695)segment_wiki script (@piskvorky, #1707)dtype support for LdaModel. Partially fix #1576 (@xelez, #1656)doc2idx method for gensim.corpora.Dictionary. Fix #1634 (@roopalgarg, #1720)gensim.downloader.info (@menshikh-iv, #1736)gensim.corpora.Dictionary (@formi23, #1715)tox.ini, setup.cfg, README.md (@menshikh-iv, #1741)logsumexp for LdaModel (@arlenk, #1745):red_circle: Bug fixes:
gensim.summarization.bm25. Fix #1718 (@souravsingh, #1726)FastText wrapper. Fix #1642 (@chinmayapancholi13, #1723)gensim.sklearn_api bug with documents_columns parameter. Fix #1676 (@chinmayapancholi13, #1704)num_words to topn in LdaMallet.show_topics. Fix #1747 (@apoorvaeternity, #1749)os.rename from gensim.downloader when 'src' and 'dst' on different partitions (@anotherbugmaster, #1733)DeprecationWarning from logsumexp (@dreamgonfly, #1703)Phrases.load. Fix #1751 (@alexgarel, #1758)load_word2vec_format from FastText. Fix #1743 (@manneshiva, #1755)Dockerfile. Fix #1762 (@rbahumi, #1764)segment_wiki (@horpto, #1763)segment_wiki (@horpto, #1750)FastText. Fix #1752 (@manneshiva, #1756)dtype of model.wv.syn0_vocab on updating vocab for FastText. Fix #1759 (@manneshiva, #1760)FastText.build_vocab. Fix #1765 (@manneshiva, #1768)DeprecationWarning for all outdated stuff. Fix #1753 (@menshikh-iv, #1769)dtype in LdaModel (@menshikh-iv, #1770):books: Tutorial and doc improvements:
segment_wiki (@piskvorky, #1708)gensim.summarization.summarize. Fix #1575 (@fbarrios, #1702)gensim.parsing. Fix #1664 (@CLearERR, #1684):warning: Deprecations (will be removed in the next major release)
Remove
gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wikiMove
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsFix DeprecationWarnings generated by deprecated assertEquals. Partial fix #1519 (__@poornagurram__, #1658)
:star2: New features:
Massive optimizations to LSI model training (@isamaru, #1620 & #1622)
LSI model allows use of single precision (float32), to consume 40% less memory while being 40% faster.
LSI model can now also accept CSC matrix as input, for further memory and speed boost.
Overall, if your entire corpus fits in RAM: 3x faster LSI training (SVD) in 4x less memory!
# just an example; the corpus stream is up to you
streaming_corpus = gensim.corpora.MmCorpus("my_tfidf_corpus.mm.gz")
# convert your corpus to a CSC sparse matrix (assumes the entire corpus fits in RAM)
in_memory_csc_matrix = gensim.matutils.corpus2csc(streaming_corpus, dtype=np.float32)
# then pass the CSC to LsiModel directly
model = LsiModel(corpus=in_memory_csc_matrix, num_topics=500, dtype=np.float32)
Even if you continue to use streaming corpora (your training dataset is too large for RAM), you should see significantly faster processing times and a lower memory footprint. In our experiments with a very large LSI model, we saw a drop from 29 GB peak RAM and 38 minutes (before) to 19 GB peak RAM and 26 minutes (now):
model = LsiModel(corpus=streaming_corpus, num_topics=500, dtype=np.float32)
Add common terms to Phrases. Fix #1258 (@alexgarel, #1568)
Phrases allows to use common terms in bigrams. Before, if you are searching to reveal ngrams like car_with_driver and car_without_driver, you can either remove stop words before processing, but you will only find car_driver, or you won't find any of those forms (because they have three words, but also because high frequency of with will avoid them to be scored correctly), inspired by ES common grams token filter.
phr_old = Phrases(corpus)
phr_new = Phrases(corpus, common_terms=stopwords.words('en'))
print(phr_old[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with", "driver"]
print(phr_new[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with_driver"]
New segment_wiki.py script (@menshikh-iv, #1483 & #1694)
CLI script for processing a raw Wikipedia dump (the xml.bz2 format provided by MediaWiki) to extract its articles in a plain text format. It extracts each article's title, section names and section content and saves them as json-line:
python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 | gzip > enwiki-latest-pages-articles.json.gz
Processing the entire English Wikipedia dump (13.5 GB, link here) takes about 2.5 hours (i7-6700HQ, SSD).
The output format is one article per line, serialized into JSON:
for line in smart_open('enwiki-latest-pages-articles.json.gz'): # read the file we just created
article = json.loads(line)
print("Article title: %s" % article['title'])
for section_title, section_text in zip(article['section_titles'], article['section_texts']):
print("Section title: %s" % section_title)
print("Section text: %s" % section_text)
:+1: Improvements:
SlicedCorpus.__len__ (@horpto, #1679)word_vec return immutable vector. Fix #1651 (@CLearERR, #1662)build_vocab_from_freq to Word2Vec, speedup scan_vocab (@jodevak, #1599)most_similar_to_given method for KeyedVectors (@TheMathMajor, #1582)__getitem__ method to Sparse2Corpus to allow direct queries (@isamaru, #1621):red_circle: Bug fixes:
test_filename_filtering test (@nehaljwani, #1647):books: Tutorial and doc improvements:
:warning: Deprecation part (will come into force in the next major release)
Remove
gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wikiMove
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsAlso, we'll create experimental subpackage for unstable models. Specific lists will be available in the next major release.
Add unsupervised FastText to Gensim (@chinmayapancholi13, #1525)
:star2: New features:
:+1: Improvements:
:red_circle: Bug fixes:
:books: Tutorial and doc improvements:
Add Dockerfile for gensim with external wrappers (@parulsethi, #1368)
:star2: New features:
:+1: Improvements:
:red_circle: Bug fixes:
:books: Tutorial and doc improvements:
Fix backward incompatibility for LdaModel (@chinmayapancholi13, #1327)
:star2: New features:
:+1: Improvements:
:red_circle: Bug fixes:
:books: Tutorial and doc improvements:
Docs word2vec docstring improvement, deprecation labels (@shubhvachher, #1274)
:star2: New features:
:+1: Improvements:
logger.warn by logger.warning (@chinmayapancholi13, #1295):red_circle: Bug fixes:
:books: Tutorial and doc improvements:
size to Word2Vec Notebook (@jbcoe, #1305)Allow training if model is not modified by "_minimize_model". Add deprecation warning. (@chinmayapancholi13,#1207)
Breaking changes:
Any direct calls to method train() of Word2Vec/Doc2Vec now require an explicit epochs parameter and explicit estimate of corpus size. The most usual way to call train is vec_model.train(sentences, total_examples=self.corpus_count, epochs=self.iter)
See the method documentation for more information.
New features:
Improvements:
Bug fixes:
Fix word2vec reset_from bug in v1.0.1 Fix #1230. (@Kreiswolke,#1234)
Distributed LDA: checking the length of docs instead of the boolean value, plus int index conversion (@saparina ,#1191)
syn0_lockf initialised with zero in intersect_word2vec_format() (@KiddoZhu,#1267)
Fix wordrank max_iter_dump calculation. Fix #1216 (@ajkl,#1217)
Make SgNegative test use sg (@shubhvachher ,#1252)
pep8/pycodestyle fixes for hanging indents in Summarization module (@SamriddhiJain ,#1202)
WordRank and Mallet wrappers single vs double quote issue in windows.(@prakhar2b,#1208)
Fix #824 : no corpus in init, but trim_rule in init (@prakhar2b ,#1186)
Hardcode version number. Fix #1138. ( @tmylk, #1138)
Tutorial and doc improvements:
Color dictionary according to topic notebook update (@bhargavvader, #1164)
Fix hdp show_topic/s docstring ( @parulsethi, #1264)
Add docstrings for word2vec.py forwarding functions ( @shubhvachher, #1251)
updated description for worker_loop function used in score function ( @chinmayapancholi13 , #1206)
Rebuild cumulative table on load. Fix #1180. (@tmylk,#1181)
Two methods and several attributes in word2vec class have been deprecated. The methods are load_word2vec_format and save_word2vec_format. The attribut…
1.0.0, 2017-02-24
Deprecated methods:
In order to share word vector querying code between different training algos(Word2Vec, Fastext, WordRank, VarEmbed) we have separated storage and querying of word vectors into a separate class KeyedVectors.
Two methods and several attributes in word2vec class have been deprecated. The methods are load_word2vec_format and save_word2vec_format. The attributes are syn0norm, syn0, vocab, index2word . They have been moved to KeyedVectors class.
After upgrading to this release you might get exceptions about deprecated methods or missing attributes.
DeprecationWarning: Deprecated. Use model.wv.save_word2vec_format instead.
AttributeError: 'Word2Vec' object has no attribute 'vocab'
To remove the exceptions, you should use
KeyedVectors.load_word2vec_format instead of Word2Vec.load_word2vec_format
word2vec_model.wv.save_word2vec_format instead of word2vec_model.save_word2vec_format
model.wv.syn0norm instead of model.syn0norm
model.wv.syn0 instead of model.syn0
model.wv.vocab instead of model.vocab
model.wv.index2word instead of model.index2word
Changelog of this release:
New features:
Deprecated features:
load_word2vec_format and save_word2vec_format out of Word2Vec class to KeyedVectors (@tmylk,#1107)syn0norm, syn0, vocab, index2word from Word2Vec class to KeyedVectors (@tmylk,#1147)Improvements:
Tutorial and doc improvements:
After upgrading to this release you might see deprecation warnings like this:
After upgrading to this release you might see deprecation warnings like this:
WARNING:gensim.models.word2vec:direct access to syn0norm will not be supported in future gensim releases, please use model.wv.syn0norm
These warnings are correct and you are encouraged to change your Word2vec/Doc2vec code to use the new model.wv.syn0norm and model.wv.vocab fields instead of old direct access like model.syn0norm and model.vocab. The direct access will be deprecated in Feb 2017.
Specifically, you should use
model.wv.syn0norm instead of model.syn0norm
model.wv.syn0 instead of model.syn0
model.wv.vocab instead of model.vocab
model.wv.index2word instead of model.index2word
The reason for this deprecation is to separate word vectors from word2vec training. There are now new ways to get word vectors that don't involve training word2vec. We are adding capabilities to use word vectors trained in GloVe, FastText, WordRank, Tensorflow and Deeplearning4j word2vec. In order to have cleaner code and standard APIs for all word embeddings we extracted a KeyedVectors class and a word-vectors wv variable into the models.
0.13.4, 2016-12-22
Changelog:
update_eta) was simplified some. It also no longer logs eta when updated, because it is too large for that.eta='asymmetric' now should raise an error._breaking change in HdpTopicFormatter.show___topics_
0.13.3, 2016-10-20
wordtopics has changed to word_topics in ldamallet, and fixed issue #764. (@bhargavvader, #771)
0.13.2, 2016-08-19
c_uci, c_npmi measures. LdaMallet, LdaVowpalWabbit support. Add topics parameter to coherencemodel. Can now provide tokenized topics to calculate coherence value. Faster backtracking. (@dsquareindia, #750, #793)limit, datatype for load_word2vec_format(); lockf for intersect_word2vec_format (@gojomo, #817)use_lowercase option in word2vec accuracy to case_insensitive to account for case variations in training vocabulary (@jayantj, #804Initial release of Topic Coherence C_v and U_mass. More work will be done here but external API will remain the same.
Initial release of Topic Coherence C_v and U_mass. More work will be done here but external API will remain the same.
Tutorials migrated from website to ipynb (@j9chan, #721), (@jesford, #733, #725, 716) New doc2vec intro tutorial (@seanlaw, #730) Gensim Quick Start T
0.12.5, 2016
Tutorials migrated from website to ipynb (@j9chan, #721), (@jesford, #733, #725, 716) New doc2vec intro tutorial (@seanlaw, #730) Gensim Quick Start Tutorial (@andrewjlm, #727) Add export_phrases(sentences) to model Phrases (hanabi1224 #588) SparseMatrixSimilarity returns a sparse matrix if maintain_sparsity is True (@davechallis, #590) added functionality for Topics of Words in document - i.e, dynamic topics. (@bhargavvader, #704) also included tutorial which explains new functionalities, and document word-topic coloring. Made normalization an explicit transformation. Added 'l1' norm support (@squareindia, #649) added term-topics API for most probable topic for word in vocab. (@bhargavvader, #706) build_vocab takes progress_per parameter for smaller output (@zer0n, #624) Control whether to use lowercase for computing word2vec accuracy. (@alantian, #607) Easy import of GloVe vectors using Gensim (Manas Ranjan Kar, #625) Allow easy port of GloVe vectors into Gensim Standalone script with command line arguments, compatible with Python>=2.6 Usage: python -m gensim.scripts.glove2word2vec -i glove_vectors.txt -o output_word2vec_compatible.txt Add similar_by_word() and similar_by_vector() to word2vec (@isohyt, #381) Convenience method for similarity of two out of training sentences to doc2vec (@ellolo, #707) Dynamic Topic Modelling Tutorial updated with Dynamic Influence Model (@bhargavvader, #689) Added function to filter 'n' most frequent words from the dictionary (@abhinavchawla, #718) Raise warnings if vocab is single character elements and if alpha is increased in word2vec/doc2vec (@dsquareindia, #705) Tests for wikidump (@jonmcoe, #723) Mallet wrapper sparse format support (@RishabGoel, #664) Doc2vec pre-processing script translated from bash to Python (@andrewjlm, #720) Added Distance Metrics to matutils.pt (@bhargavvader, #656)
LdaModel and LdaMulticore produce a large number of DeprecationWarnings from .inference() because the term ids in each chunk returned from utils.group…
init_sims() call for performance improvements when normalized vectors are not needed.norm_only parameter (API change). Call init_sims(replace=True) after the load_word2vec_format() call for the old norm_only=True behavior.model.docvecs[key] now raises KeyError for unknown keys (Gordon Mohr, #520)DocvecsArray.index_to_doctag so most_similar() returns string doctags (Gordon Mohr, #560)pattern library in utils.lemmatize (Jan Zikes, #461)
utils.HAS_PATTERN flag moved to utils.has_pattern()TestWord2VecModel.test_cbow_hs() against random failures (Gordon Mohr, #531)default_timer() indicate no elapsed time (Gordon Mohr, #518)This is a breaking change that affects users of the LsiModel, LdaModel, and LdaMulticore that may be reliant on the old tuple layout of (probability,…
0.12.3rc1, 05/11/2015
show_topics method should return a list of
(topic_number, topic) tuples, where topic is a list of
(word, probability) tuples.LsiModel, LdaModel,
and LdaMulticore that may be reliant on the old tuple layout of
(probability, word).index2doctag list is renamed/reinterpreted as offset2doctagoffset2doctag entries map to doctag_syn0 indexes after last plain-int doctag (if any)offset2doctag may be interpreted same as index2doctag.)tutorial on text summarization (Ólavur Mortensen, #436)
ignore list can be passed in (Matti Lyra, #331)improvements to testing, switch to Travis CI containers
complete API, performance, memory overhaul of doc2vec (Gordon Mohr, #356, #373, #380, #384)
Nothing published for this version
added streamed phrases = collocation detection (Miguel Cabrera, #258)
new parallelized, LdaMulticore implementation (Jan Zikes, #232)
word2vec: new n_similarity method for comparing two sets of words (François Scharffe, #219)
full Python 3 support (targeting 3.3+, #196)
LdaMallet support for printing/showing topics
save/load automatically single out large arrays + allow mmap
use travis-ci for continuous integration
python3 port by Parikshit Samant: https://github.com/samantp/gensimPy3
initial version of word2vec, a neural network deep learning algo
added HashDictionary (by Homer Strong)
improved performance of sharding (similarity queries)
better support for Pandas series input (thx to JT Bates)
pattern python package)fixed Similarity sharding bug (issue #65, thx to Paul Rudin)
improved accuracy of SVD (Latent Semantic Analysis) (thx to Mark Tygert)
tox for testingtransactional similarity server: see docs/simserver.html
changed all variable and function names to comply with PEP8 (numTopics->num_topics): BREAKS BACKWARD COMPATIBILITY!
added corpora.IndexedCorpus, a base class for corpus serializers (thx to Dieter Plaetinck). This allows corpus formats that inherit from it (MmCorpus,
corpora.IndexedCorpus, a base class for corpus serializers (thx to Dieter Plaetinck). This allows corpus formats that inherit from it (MmCorpus, SvmLightCorpus, BleiCorpus etc.) to retrieve individual documents by their id in O(1), e.g. corpus[14] returns document #14.corpora.textcorpus, models.logentropy_model, lots of unit tests etc.)lda[bow] transformation (was returning gamma distribution instead of theta). LDA model generation was not affected, only transforming new vectors.new LDA implementation after Hoffman et al.: Online Learning for Latent Dirichlet Allocation
added workaround for a bug in numpy: pickling a fortran-order array (e.g. LSA model) and then loading it back and using it results in segfault (thx to
further optimization to LSA; this is the version used in my NIPS workshop paper
sped up Latent Dirichlet ~10x (through scipy.weave, optional)
added stochastic SVD decomposition (faster than the current one-pass LSI algo, but needs two passes over the input corpus)
added workaround for a numpy bug where SVD sometimes fails to converge for no good reason
fixed a bug in LSA that occurred when the number of features was smaller than the number of topics (thx to Richard Berendsen)
optimized vocabulary generation in gensim.corpora.dictionary (faster and less memory-intense)
added option for online LSI training (yay!). the transformation can now be used after any amount of training, and training can be continued at any tim
finished all tutorials, stable version
Your coding agent can read these notes before it upgrades. Set up the MCP server →