NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2433 most downloaded on PyPI
Python framework for fast Vector Space Modelling
Latest stable
4.4.0
Last release 1 years ago
18 Oct 2025
Ships fairly regularly
a new release about every 7 months
Nearly every release is documented
notes for 59 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
17 years old
80 releases · first in 2010
Drag across the ruler to compare, or tap a release to open it.
:warning: Deprecations (will be removed in the next major release)
:star2: New features:
Re-designed all "*2vec" implementations (@manneshiva, #1777)
Word2Vec, Doc2Vec, FastText, etc ..., making it easier to add new models in the future and re-use codeImprove gensim.scripts.segment_wiki by retaining interwiki links. Fix #1712
(@steremma, PR #1839)
Optionally extract interlinks from Wikipedia pages (use the --include-interlinks option). This will output one additional JSON dict for each article:
{
"interlinks": {
"article title 1": "interlink text 1",
"article title 2": "interlink text 2",
...
}
}
Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump:
python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz
segment_wiki output:import json
from smart_open import smart_open
with smart_open("enwiki-latest.jsonl.gz") as infile:
for doc in infile:
doc = json.loads(doc)
src_node = doc['title']
dst_nodes = doc['interlinks'].keys()
print(u"Source node: {}".format(src_node))
print(u"Destination nodes: {}".format(u", ".join(dst_nodes)))
break
"""
OUTPUT:
Source node: Anarchism
Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, Andalusia, Renzo Novatore, trade union, autonomist marxism, dictatorship of the proletariat, Mujeres Libres, Voltairine de Cleyre, Post-anarchism, participatory economics, Confederación Nacional del Trabajo, Syncretic politics, direct democracy, Jean-Jacques Rousseau, Green anarchism, Surrealism, labour unions, A. S. Neill, christian anarchist, Bonnot Gang, Anti-capitalism, Anarchism in Brazil, simple living, enlightened self-interest, Confédération générale du travail, class conflict, International Workers' Day, Hébertists, Gerrard Winstanley, Francoism, anarcho-pacifist, Andrej Grubacic, individualist anarchist and social anarchist thinkers., April Carter, private property, penal colonies, Libertarian socialism, Camillo Berneri, Christian anarchism, transhumanism, Lucifer, the Light-Bearer, Edna St. Vincent Millay, unschooling, Leo Tolstoy, M. E. Lazarus, Spanish Anarchists, Buddhist anarchism, ideology, William McKinley, anarcho-primitivism, Francesc Pi i Margall, :Category:Anarchism by country, International Workers Association, Anarcho-capitalism, Lois Waisbrooker, wikt:Solidarity, Baja California, social revolution, Unione Sindacale Italiana, Lev Chernyi, Alex Comfort, Sonnenburg, Leon Czolgosz, Volin, utopian, Argentine Libertarian Federation, Nudism, Left-wing market anarchism, insurrection, definitional concerns in anarchist theory, infinitive, affinity group, World Trade Organization Ministerial Conference of 1999 protest activity, class struggle, nonviolence, John Zerzan, poststructuralist, Noam Chomsky, Second Fitna, Julian Beck, Philadelphes, League of Peace and Freedom, Fédération Anarchiste, Kronstadt rebellion, Cold War, André Breton, Silvio Gesell, libertarian anarchism, voluntary association, anti-globalisation movement, birth control, L. Susan Brown, anarcho-naturism, personal property, Roundhead, Harold Barclay, The Joy of Sex, Council communism, Lucía Sánchez Saornil, tyrannicide, Neopaganism, lois scélérates, Johann Most, Anarchist Catalonia, Albert Camus, Protests of 1968, Alexander II of Russia, Spain's economy, Federazione Anarchica Italiana, Cuba, German Revolution of 1918–1919, stirner, Property is theft, Situationist International, law and economics
Add support for SMART notation for TfidfModel. Fix #1785 (@markroxor, #1791)
TfidfModel to allow different weighting and normalization schemesfrom gensim.corpora import Dictionary
from gensim.models import TfidfModel
import gensim.downloader as api
data = api.load("text8")
dct = Dictionary(data)
corpus = [dct.doc2bow(line) for line in data]
# Train Tfidf model using the SMART notation, smartirs="ntc" where
# 'n' - natural term frequency
# 't' - idf document frequency
# 'c' - cosine normalization
#
# More information about possible values available in documentation or https://nlp.stanford.edu/IR-book/html/htmledition/document-and-query-weighting-schemes-1.html
model = TfidfModel(corpus, id2word=dct, smartirs="ntc")
vectorized_corpus = list(model[corpus])
Add CircleCI for building Gensim documentation. Fix #1807 (@menshikh-iv, #1822)
:red_circle: Bug fixes:
get_my_ip. Fix #1771 (@darindf, #1772)gensim.summarization.bm25. Fix #1828 (@sj29-innovate, #1833)TranslationMatrix (@robotcator, #1838)gensim.models.CoherenceModel in gensim.models.callbacks (@Alexjmsherman, #1823)FastText.train. Fix #1818 (@sj29-innovate, #1837)score_function from LexicalEntailmentEvaluation. Fix #1858 (@hachibaka, #1863):books: Tutorial and doc improvements:
gensim.summarization (@yurkai & @menshikh-iv, #1709)gensim.similarities.index. Partial fix #1666 (@menshikh-iv, #1681)gensim.models.translation_matrix (@KokuKUSIAKU & @menshikh-iv, #1806)gensim.models.rpmodel (@jazzmuesli & @menshikh-iv, #1802)gensim.utils (@kakshay21 & @menshikh-iv, #1797)gensim.matutils (@Cheukting & @menshikh-iv, #1804)gensim.models.logentropy_model (@minggli & @menshikh-iv, #1803)gensim.models.normmodel (@AustenLamacraft & @menshikh-iv, #1805)gensim.topic_coherence. Fix #1669 (@CLearERR & @menshikh-iv, #1714)gensim.corpora.dictionary and gensim.corpora.hashdictionary. Partial fix #1671 (@CLearERR & @menshikh-iv, #1814)gensim.corpora. Partial fix #1671 (@anotherbugmaster & @menshikh-iv, #1729)model_to_dict one-liner to word2vec notebook. Fix #1269 (@kakshay21, #1776)sg parameter for gensim.models.FastText (@akutuzov, #1801)doc2vec-IMDB. Fix #1788 (@apoorvaeternity, #1796)bz2 + MmCorpus examples from tutorials (@menshikh-iv, #1867):+1: Improvements:
:warning: Deprecations (will be removed in the next major release)
Remove
gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)Move
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsYour coding agent can read these notes before it upgrades. Set up the MCP server →
New fast multithreaded implementation of FastText, natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation. ```py…
:star2: New features:
New download API for corpora and pre-trained models (@chaitaliSaini & @menshikh-iv, #1705 & #1632 & #1492)
import gensim.downloader as api
for article in api.load("wiki-english-20171001"):
print(article)
import gensim.downloader as api
model = api.load("glove-twitter-25")
model.most_similar("engineer")
# [('specialist', 0.957542896270752),
# ('developer', 0.9548177123069763),
# ('administrator', 0.9432312846183777),
# ('consultant', 0.93915855884552),
# ('technician', 0.9368376135826111),
# ('analyst', 0.9342101216316223),
# ('architect', 0.9257484674453735),
# ('engineering', 0.9159940481185913),
# ('systems', 0.9123805165290833),
# ('consulting', 0.9112802147865295)]
New model: Poincaré embeddings (@jayantj, #1696 & #1700 & #1757 & #1734)
from gensim.models.poincare import PoincareRelations, PoincareModel
from gensim.test.utils import datapath
data = PoincareRelations(datapath('poincare_hypernyms.tsv'))
model = PoincareModel(data)
model.kv.most_similar("cat.n.01")
# [('kangaroo.n.01', 0.010581353439700418),
# ('gib.n.02', 0.011171531439892076),
# ('striped_skunk.n.01', 0.012025106076442395),
# ('metatherian.n.01', 0.01246679759214648),
# ('mammal.n.01', 0.013281303506525968),
# ('marsupial.n.01', 0.013941330203709653)]
Optimized FastText (@manneshiva, #1742)
import gensim.downloader as api
from gensim.models import FastText
model = FastText(api.load("text8"))
model.most_similar("cat")
# [('catnip', 0.8538144826889038),
# ('catwalk', 0.8136177062988281),
# ('catchy', 0.7828493118286133),
# ('caf', 0.7826495170593262),
# ('bobcat', 0.7745151519775391),
# ('tomcat', 0.7732658386230469),
# ('moat', 0.7728310823440552),
# ('caye', 0.7666271328926086),
# ('catv', 0.7651021480560303),
# ('caveat', 0.7643581628799438)]
Binary pre-compiled wheels for Windows, OSX and Linux (@menshikh-iv, MacPython/gensim-wheels/#7)
Added DeprecationWarnings to deprecated methods and parameters, with a clear schedule for removal.
:+1: Improvements:
scan_vocab speed, build_vocab_from_freq method (@jodevak, #1695)segment_wiki script (@piskvorky, #1707)dtype support for LdaModel. Partially fix #1576 (@xelez, #1656)doc2idx method for gensim.corpora.Dictionary. Fix #1634 (@roopalgarg, #1720)gensim.downloader.info (@menshikh-iv, #1736)gensim.corpora.Dictionary (@formi23, #1715)tox.ini, setup.cfg, README.md (@menshikh-iv, #1741)logsumexp for LdaModel (@arlenk, #1745):red_circle: Bug fixes:
gensim.summarization.bm25. Fix #1718 (@souravsingh, #1726)FastText wrapper. Fix #1642 (@chinmayapancholi13, #1723)gensim.sklearn_api bug with documents_columns parameter. Fix #1676 (@chinmayapancholi13, #1704)num_words to topn in LdaMallet.show_topics. Fix #1747 (@apoorvaeternity, #1749)os.rename from gensim.downloader when 'src' and 'dst' on different partitions (@anotherbugmaster, #1733)DeprecationWarning from logsumexp (@dreamgonfly, #1703)Phrases.load. Fix #1751 (@alexgarel, #1758)load_word2vec_format from FastText. Fix #1743 (@manneshiva, #1755)Dockerfile. Fix #1762 (@rbahumi, #1764)segment_wiki (@horpto, #1763)segment_wiki (@horpto, #1750)FastText. Fix #1752 (@manneshiva, #1756)dtype of model.wv.syn0_vocab on updating vocab for FastText. Fix #1759 (@manneshiva, #1760)FastText.build_vocab. Fix #1765 (@manneshiva, #1768)DeprecationWarning for all outdated stuff. Fix #1753 (@menshikh-iv, #1769)dtype in LdaModel (@menshikh-iv, #1770):books: Tutorial and doc improvements:
segment_wiki (@piskvorky, #1708)gensim.summarization.summarize. Fix #1575 (@fbarrios, #1702)gensim.parsing. Fix #1664 (@CLearERR, #1684):warning: Deprecations (will be removed in the next major release)
Remove
gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wikiMove
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsFix DeprecationWarnings generated by deprecated assertEquals. Partial fix #1519 (__@poornagurram__, #1658)
:star2: New features:
Massive optimizations to LSI model training (@isamaru, #1620 & #1622)
LSI model allows use of single precision (float32), to consume 40% less memory while being 40% faster.
LSI model can now also accept CSC matrix as input, for further memory and speed boost.
Overall, if your entire corpus fits in RAM: 3x faster LSI training (SVD) in 4x less memory!
# just an example; the corpus stream is up to you
streaming_corpus = gensim.corpora.MmCorpus("my_tfidf_corpus.mm.gz")
# convert your corpus to a CSC sparse matrix (assumes the entire corpus fits in RAM)
in_memory_csc_matrix = gensim.matutils.corpus2csc(streaming_corpus, dtype=np.float32)
# then pass the CSC to LsiModel directly
model = LsiModel(corpus=in_memory_csc_matrix, num_topics=500, dtype=np.float32)
Even if you continue to use streaming corpora (your training dataset is too large for RAM), you should see significantly faster processing times and a lower memory footprint. In our experiments with a very large LSI model, we saw a drop from 29 GB peak RAM and 38 minutes (before) to 19 GB peak RAM and 26 minutes (now):
model = LsiModel(corpus=streaming_corpus, num_topics=500, dtype=np.float32)
Add common terms to Phrases. Fix #1258 (@alexgarel, #1568)
Phrases allows to use common terms in bigrams. Before, if you are searching to reveal ngrams like car_with_driver and car_without_driver, you can either remove stop words before processing, but you will only find car_driver, or you won't find any of those forms (because they have three words, but also because high frequency of with will avoid them to be scored correctly), inspired by ES common grams token filter.
phr_old = Phrases(corpus)
phr_new = Phrases(corpus, common_terms=stopwords.words('en'))
print(phr_old[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with", "driver"]
print(phr_new[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with_driver"]
New segment_wiki.py script (@menshikh-iv, #1483 & #1694)
CLI script for processing a raw Wikipedia dump (the xml.bz2 format provided by MediaWiki) to extract its articles in a plain text format. It extracts each article's title, section names and section content and saves them as json-line:
python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 | gzip > enwiki-latest-pages-articles.json.gz
Processing the entire English Wikipedia dump (13.5 GB, link here) takes about 2.5 hours (i7-6700HQ, SSD).
The output format is one article per line, serialized into JSON:
for line in smart_open('enwiki-latest-pages-articles.json.gz'): # read the file we just created
article = json.loads(line)
print("Article title: %s" % article['title'])
for section_title, section_text in zip(article['section_titles'], article['section_texts']):
print("Section title: %s" % section_title)
print("Section text: %s" % section_text)
:+1: Improvements:
SlicedCorpus.__len__ (@horpto, #1679)word_vec return immutable vector. Fix #1651 (@CLearERR, #1662)build_vocab_from_freq to Word2Vec, speedup scan_vocab (@jodevak, #1599)most_similar_to_given method for KeyedVectors (@TheMathMajor, #1582)__getitem__ method to Sparse2Corpus to allow direct queries (@isamaru, #1621):red_circle: Bug fixes:
test_filename_filtering test (@nehaljwani, #1647):books: Tutorial and doc improvements:
:warning: Deprecation part (will come into force in the next major release)
Remove
gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wikiMove
gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.pygensim.summarization ➡ gensim.models.summarizationgensim.topic_coherence ➡ gensim.models._coherencegensim.utils ➡ gensim.utils.utils (old imports will continue to work)gensim.parsing.* ➡ gensim.utils.text_utilsAlso, we'll create experimental subpackage for unstable models. Specific lists will be available in the next major release.
Add unsupervised FastText to Gensim (@chinmayapancholi13, #1525)
:star2: New features:
:+1: Improvements:
:red_circle: Bug fixes:
:books: Tutorial and doc improvements:
Add Dockerfile for gensim with external wrappers (@parulsethi, #1368)
:star2: New features:
:+1: Improvements:
:red_circle: Bug fixes:
:books: Tutorial and doc improvements:
Fix backward incompatibility for LdaModel (@chinmayapancholi13, #1327)
:star2: New features:
:+1: Improvements:
:red_circle: Bug fixes:
:books: Tutorial and doc improvements:
Docs word2vec docstring improvement, deprecation labels (@shubhvachher, #1274)
:star2: New features:
:+1: Improvements:
logger.warn by logger.warning (@chinmayapancholi13, #1295):red_circle: Bug fixes:
:books: Tutorial and doc improvements:
size to Word2Vec Notebook (@jbcoe, #1305)Allow training if model is not modified by "_minimize_model". Add deprecation warning. (@chinmayapancholi13,#1207)
Breaking changes:
Any direct calls to method train() of Word2Vec/Doc2Vec now require an explicit epochs parameter and explicit estimate of corpus size. The most usual way to call train is vec_model.train(sentences, total_examples=self.corpus_count, epochs=self.iter)
See the method documentation for more information.
New features:
Improvements:
Bug fixes:
Fix word2vec reset_from bug in v1.0.1 Fix #1230. (@Kreiswolke,#1234)
Distributed LDA: checking the length of docs instead of the boolean value, plus int index conversion (@saparina ,#1191)
syn0_lockf initialised with zero in intersect_word2vec_format() (@KiddoZhu,#1267)
Fix wordrank max_iter_dump calculation. Fix #1216 (@ajkl,#1217)
Make SgNegative test use sg (@shubhvachher ,#1252)
pep8/pycodestyle fixes for hanging indents in Summarization module (@SamriddhiJain ,#1202)
WordRank and Mallet wrappers single vs double quote issue in windows.(@prakhar2b,#1208)
Fix #824 : no corpus in init, but trim_rule in init (@prakhar2b ,#1186)
Hardcode version number. Fix #1138. ( @tmylk, #1138)
Tutorial and doc improvements:
Color dictionary according to topic notebook update (@bhargavvader, #1164)
Fix hdp show_topic/s docstring ( @parulsethi, #1264)
Add docstrings for word2vec.py forwarding functions ( @shubhvachher, #1251)
updated description for worker_loop function used in score function ( @chinmayapancholi13 , #1206)
Rebuild cumulative table on load. Fix #1180. (@tmylk,#1181)
Two methods and several attributes in word2vec class have been deprecated. The methods are load_word2vec_format and save_word2vec_format. The attribut…
1.0.0, 2017-02-24
Deprecated methods:
In order to share word vector querying code between different training algos(Word2Vec, Fastext, WordRank, VarEmbed) we have separated storage and querying of word vectors into a separate class KeyedVectors.
Two methods and several attributes in word2vec class have been deprecated. The methods are load_word2vec_format and save_word2vec_format. The attributes are syn0norm, syn0, vocab, index2word . They have been moved to KeyedVectors class.
After upgrading to this release you might get exceptions about deprecated methods or missing attributes.
DeprecationWarning: Deprecated. Use model.wv.save_word2vec_format instead.
AttributeError: 'Word2Vec' object has no attribute 'vocab'
To remove the exceptions, you should use
KeyedVectors.load_word2vec_format instead of Word2Vec.load_word2vec_format
word2vec_model.wv.save_word2vec_format instead of word2vec_model.save_word2vec_format
model.wv.syn0norm instead of model.syn0norm
model.wv.syn0 instead of model.syn0
model.wv.vocab instead of model.vocab
model.wv.index2word instead of model.index2word
Changelog of this release:
New features:
Deprecated features:
load_word2vec_format and save_word2vec_format out of Word2Vec class to KeyedVectors (@tmylk,#1107)syn0norm, syn0, vocab, index2word from Word2Vec class to KeyedVectors (@tmylk,#1147)Improvements:
Tutorial and doc improvements:
After upgrading to this release you might see deprecation warnings like this:
After upgrading to this release you might see deprecation warnings like this:
WARNING:gensim.models.word2vec:direct access to syn0norm will not be supported in future gensim releases, please use model.wv.syn0norm
These warnings are correct and you are encouraged to change your Word2vec/Doc2vec code to use the new model.wv.syn0norm and model.wv.vocab fields instead of old direct access like model.syn0norm and model.vocab. The direct access will be deprecated in Feb 2017.
Specifically, you should use
model.wv.syn0norm instead of model.syn0norm
model.wv.syn0 instead of model.syn0
model.wv.vocab instead of model.vocab
model.wv.index2word instead of model.index2word
The reason for this deprecation is to separate word vectors from word2vec training. There are now new ways to get word vectors that don't involve training word2vec. We are adding capabilities to use word vectors trained in GloVe, FastText, WordRank, Tensorflow and Deeplearning4j word2vec. In order to have cleaner code and standard APIs for all word embeddings we extracted a KeyedVectors class and a word-vectors wv variable into the models.
0.13.4, 2016-12-22
Changelog:
update_eta) was simplified some. It also no longer logs eta when updated, because it is too large for that.eta='asymmetric' now should raise an error._breaking change in HdpTopicFormatter.show___topics_
0.13.3, 2016-10-20
wordtopics has changed to word_topics in ldamallet, and fixed issue #764. (@bhargavvader, #771)
0.13.2, 2016-08-19
c_uci, c_npmi measures. LdaMallet, LdaVowpalWabbit support. Add topics parameter to coherencemodel. Can now provide tokenized topics to calculate coherence value. Faster backtracking. (@dsquareindia, #750, #793)limit, datatype for load_word2vec_format(); lockf for intersect_word2vec_format (@gojomo, #817)use_lowercase option in word2vec accuracy to case_insensitive to account for case variations in training vocabulary (@jayantj, #804Initial release of Topic Coherence C_v and U_mass. More work will be done here but external API will remain the same.
Initial release of Topic Coherence C_v and U_mass. More work will be done here but external API will remain the same.
Tutorials migrated from website to ipynb (@j9chan, #721), (@jesford, #733, #725, 716) New doc2vec intro tutorial (@seanlaw, #730) Gensim Quick Start T
0.12.5, 2016
Tutorials migrated from website to ipynb (@j9chan, #721), (@jesford, #733, #725, 716) New doc2vec intro tutorial (@seanlaw, #730) Gensim Quick Start Tutorial (@andrewjlm, #727) Add export_phrases(sentences) to model Phrases (hanabi1224 #588) SparseMatrixSimilarity returns a sparse matrix if maintain_sparsity is True (@davechallis, #590) added functionality for Topics of Words in document - i.e, dynamic topics. (@bhargavvader, #704) also included tutorial which explains new functionalities, and document word-topic coloring. Made normalization an explicit transformation. Added 'l1' norm support (@squareindia, #649) added term-topics API for most probable topic for word in vocab. (@bhargavvader, #706) build_vocab takes progress_per parameter for smaller output (@zer0n, #624) Control whether to use lowercase for computing word2vec accuracy. (@alantian, #607) Easy import of GloVe vectors using Gensim (Manas Ranjan Kar, #625) Allow easy port of GloVe vectors into Gensim Standalone script with command line arguments, compatible with Python>=2.6 Usage: python -m gensim.scripts.glove2word2vec -i glove_vectors.txt -o output_word2vec_compatible.txt Add similar_by_word() and similar_by_vector() to word2vec (@isohyt, #381) Convenience method for similarity of two out of training sentences to doc2vec (@ellolo, #707) Dynamic Topic Modelling Tutorial updated with Dynamic Influence Model (@bhargavvader, #689) Added function to filter 'n' most frequent words from the dictionary (@abhinavchawla, #718) Raise warnings if vocab is single character elements and if alpha is increased in word2vec/doc2vec (@dsquareindia, #705) Tests for wikidump (@jonmcoe, #723) Mallet wrapper sparse format support (@RishabGoel, #664) Doc2vec pre-processing script translated from bash to Python (@andrewjlm, #720) Added Distance Metrics to matutils.pt (@bhargavvader, #656)
LdaModel and LdaMulticore produce a large number of DeprecationWarnings from .inference() because the term ids in each chunk returned from utils.group…
init_sims() call for performance improvements when normalized vectors are not needed.norm_only parameter (API change). Call init_sims(replace=True) after the load_word2vec_format() call for the old norm_only=True behavior.model.docvecs[key] now raises KeyError for unknown keys (Gordon Mohr, #520)DocvecsArray.index_to_doctag so most_similar() returns string doctags (Gordon Mohr, #560)pattern library in utils.lemmatize (Jan Zikes, #461)
utils.HAS_PATTERN flag moved to utils.has_pattern()TestWord2VecModel.test_cbow_hs() against random failures (Gordon Mohr, #531)default_timer() indicate no elapsed time (Gordon Mohr, #518)This is a breaking change that affects users of the LsiModel, LdaModel, and LdaMulticore that may be reliant on the old tuple layout of (probability,…
0.12.3rc1, 05/11/2015
show_topics method should return a list of
(topic_number, topic) tuples, where topic is a list of
(word, probability) tuples.LsiModel, LdaModel,
and LdaMulticore that may be reliant on the old tuple layout of
(probability, word).index2doctag list is renamed/reinterpreted as offset2doctagoffset2doctag entries map to doctag_syn0 indexes after last plain-int doctag (if any)offset2doctag may be interpreted same as index2doctag.)tutorial on text summarization (Ólavur Mortensen, #436)
ignore list can be passed in (Matti Lyra, #331)improvements to testing, switch to Travis CI containers
complete API, performance, memory overhaul of doc2vec (Gordon Mohr, #356, #373, #380, #384)
Nothing published for this version
added streamed phrases = collocation detection (Miguel Cabrera, #258)
new parallelized, LdaMulticore implementation (Jan Zikes, #232)
word2vec: new n_similarity method for comparing two sets of words (François Scharffe, #219)
full Python 3 support (targeting 3.3+, #196)
LdaMallet support for printing/showing topics
save/load automatically single out large arrays + allow mmap
use travis-ci for continuous integration
python3 port by Parikshit Samant: https://github.com/samantp/gensimPy3
initial version of word2vec, a neural network deep learning algo
added HashDictionary (by Homer Strong)
improved performance of sharding (similarity queries)
better support for Pandas series input (thx to JT Bates)
pattern python package)fixed Similarity sharding bug (issue #65, thx to Paul Rudin)
improved accuracy of SVD (Latent Semantic Analysis) (thx to Mark Tygert)
tox for testingtransactional similarity server: see docs/simserver.html
changed all variable and function names to comply with PEP8 (numTopics->num_topics): BREAKS BACKWARD COMPATIBILITY!
added corpora.IndexedCorpus, a base class for corpus serializers (thx to Dieter Plaetinck). This allows corpus formats that inherit from it (MmCorpus,
corpora.IndexedCorpus, a base class for corpus serializers (thx to Dieter Plaetinck). This allows corpus formats that inherit from it (MmCorpus, SvmLightCorpus, BleiCorpus etc.) to retrieve individual documents by their id in O(1), e.g. corpus[14] returns document #14.corpora.textcorpus, models.logentropy_model, lots of unit tests etc.)lda[bow] transformation (was returning gamma distribution instead of theta). LDA model generation was not affected, only transforming new vectors.new LDA implementation after Hoffman et al.: Online Learning for Latent Dirichlet Allocation
added workaround for a bug in numpy: pickling a fortran-order array (e.g. LSA model) and then loading it back and using it results in segfault (thx to
further optimization to LSA; this is the version used in my NIPS workshop paper
sped up Latent Dirichlet ~10x (through scipy.weave, optional)
added stochastic SVD decomposition (faster than the current one-pass LSI algo, but needs two passes over the input corpus)
added workaround for a numpy bug where SVD sometimes fails to converge for no good reason
fixed a bug in LSA that occurred when the number of features was smaller than the number of topics (thx to Richard Berendsen)
optimized vocabulary generation in gensim.corpora.dictionary (faster and less memory-intense)
added option for online LSI training (yay!). the transformation can now be used after any amount of training, and training can be continued at any tim
finished all tutorials, stable version
* tutorial on transformations
added Random Projections (aka Random Indexing), as another transformation model.
further memory optimizations in SVD (LSI)
added missing test files to MANIFEST.in
added gensim reference to Wikipedia articles (SVD, LSI, LDA, TFIDF, ...)
similarity queries got their own package
removed dependency on python2.5 (theoretically, gensim now runs on 2.6 and 2.7 as well).
support for python setup.py test
python setup.py testNothing published for this version
* First version
One column per quarter.