PackageTrack

PyPI

newspaper4k

0.9.6AndyTheFactory/newspaper4k ↗

Simplified python article discovery & extraction.

Release timeline

9 releases since 2023
20232026

One column per month.

Releases

  1. 0.9.619 Jul 2026
    Release notes
    • Added native PyInstaller support
    • Fixed article configuration isolation
    • Improved extraction and duplicate detection
    • Added lxml 6.x support
    • Expanded documentation and test coverage

    What's Changed

    Full Changelog: [0.9.5...0.9.6](https://github.com/AndyTheFactory/newspaper4k/compare/0.9.5...0.9.6)

    Open source →
  2. 0.9.528 Feb 2026
    Release notes
    • lang: Add ISO 639-3 language code support for Kurdish (ckb, kmr)
    • tests: add robots tests
    • feat: added robots.txt check with hook in do_request
    • feat: add hooks to get_html
    • parse: prioritize datePublished over dateCreated in JSON-LD extraction
    • docs: Readme improvements
    • feat: add nltk as an optional dependency for leaner deployments
    • docs: added additional documentation for GoogleNews and Cloudscraper integration
    • rework: type annotations removed deprecated types (python 3.10+)

    What's Changed

    • lang: Add ISO 639-3 language code support for Kurdish (ckb, kmr) by Muzaffer Cikay in #691
    • tests: add robots tests by Andrei in 030e50d
    • feat: added robots.txt check with hook in do_request by Andrei in 62dece9
    • feat: add hooks to get_html by Andrei in 708cc10
    • parse: prioritize datePublished over dateCreated in JSON-LD extraction by Pontus Svensson in cdadb9e
    • docs: Readme improvements by Andrei in 18ca21c
    • feat: add nltk as an optional dependency for leaner deployments by Andrei in e073459
    • docs: added additional documentation for GoogleNews and Cloudscraper integration by Andrei in aceb853
    • rework: type annotations removed deprecated types (python 3.10+) by Andrei in bd82a41

    Bugs Fixed

    • skip null entries in JSON-LD arrays during extraction (fix #692) by ghxm in 77d6ccc
    • ArticleException f-string not interpolating status_code (fix #684) by Andrei in 7caa2a5
    • accept relative paths for categories (PR #667) by BRNMan in #667
    • use w3lib to detect webpage encoding by Andrei in 3bd4f00

    New Contributors

    • Muzaffer Cikay made their first contribution in #691

    Full Changelog: 0.9.4...0.9.5

    Open source →
  3. 0.9.4.118 Nov 2025
    Release notes
    • feat: add support for python 3.14
    • rework: minor typing changes
    • tests: increase test coverage

    What's Changed

    New Contributors

    Full Changelog: 0.9.4...0.9.4.1

    Open source →
    Additional notes

    New Features

    • feat: add support for python 3.14
    • rework: minor typing changes
    • tests: increase test coverage
    • lang: add Kurdish Kurmanji stopwords (#677) (by cikay)
    • docs: update supported languages (#676) (by cikay)
    • docs: bump sphinx version (#680) (by Andrei)
    • docs: Docs 0.9.4 (#681) (by Andrei)
    Open source →
  4. 0.9.415 Nov 2025
    Release notes

    New Features

    Bumped min Python version to 3.10. Version 3.8 and 3.9 are no longer supported, but might still work.

    • misc: switch to uv from poetry
    • parse: add brotli compression
    • install: dependency versions pin
    • tests: split tests into unit, integration and e2e. Only unit tests are ran on each PR. Integration and e2e tests are ran locally when developing.
    • tests: added coverage report generation. Coverage uploaded to coveralls.io
    Open source →
    Additional notes

    New Features

    Bumped min Python version to 3.10. Version 3.8 and 3.9 are no longer supported, but might still work.

    • misc: switch to uv from poetry(2345076) (by Andrei)
    • parse: add brotli compression(6ff72bd) (by Andrei)
    • install: dependency versions pin(10cae21) (by Andrei)
    • typing: update type hint for data parameter to allow None(80279d1) (by Andrei)
    • tests: split tests into unit, integration and e2e. Only unit tests are ran on each PR. Integration and e2e tests are ran locally when developing.
    • tests: added coverage report generation. Coverage uploaded to coveralls.io

    Refactor

    • rework: :art: reformat with ruff, new line-width 120, sort imports(cb560f7) (by Andrei)
    • doc: :tada: Better explanation of min_word_count, min_sent_count configuration(7ed25e9) (by Andrei)
    • chore: typing-extensions, lxml compatibility (#639)(c5e4170) (by Chris)

    Bugs fixed:

    • parse: :boom: fix: repair Google News URL decoding and update network request handling(059b45c) (by Andrei)
    • misc: erroneous debug statements (#663)(2ab8208) (by Michael Braun)
    • docs: :memo: correct spelling of 'memoize_articles' to 'memorize_articles' in user guide(3039463) (by Andrei)
    • lang: :bug: correct iso-code for nepali language (#624)(88fc5a7) (by Andrei)
    • requests: Fixed issue [BUG] Responses with no headers break some of the internal code #635(802ae11) (by Andrei)
    Open source →
  5. 0.9.3.118 Mar 2024
    Release notes2 sources agree

    Some fixes with regards to python >= 3.11 dependencies. Numpy version was incompatible with colab. Now it is fixed.

    Also, there was a typo in the Nepali language code - it was "np" instead of "ne". This is now fixed.

    Open source →
  6. 0.9.318 Mar 2024
    Release notes

    Massive improvements in multi-language capabilities. Added over 40 new languages and completely reworked the language module. Much easier to add new languages now. Additionally, added support for Google News as a source. You can now search and parse news based on keywords, topic, location or website.
    Integrated cloudscraper as an optional dependency. If installed, it will us cloudscraper as a layer over requests. Cloudscraper tries to bypass cloudflair protection.
    We now have use two evaluation datasets - the one from scrapinghub and one created by us drom the top 200 most popular websites. This will help keeping track of future improvements and to have a clear view of the impact of the changes.

    We see a steady improvement from version 0.9.0 up to 0.9.3. The evaluation results are available in the documentation. The evaluation dataset is also available in the following repository: Article Extraction Dataset

    • You can now install languages that need special packages as optional dependencies
    • Google News full integrated in the scraping process.
    • You can now pickle sources and articles - easier to save and recover scraping
    • Bumped minimum python version support to Python 3.8
    Open source →
    Additional notes

    Massive improvements in multi-language capabilities. Added over 40 new languages and completely reworked the language module. Much easier to add new languages now. Additionally, added support for Google News as a source. You can now search and parse news based on keywords, topic, location or website. Integrated cloudscraper as an optional dependency. If installed, it will us cloudscraper as a layer over requests. Cloudscraper tries to bypass cloudflair protection. We now have use two evaluation datasets - the one from scrapinghub and one created by us drom the top 200 most popular websites. This will help keeping track of future improvements and to have a clear view of the impact of the changes.

    We see a steady improvement from version 0.9.0 up to 0.9.3. The evaluation results are available in the documentation. The evaluation dataset is also available in the following repository: Article Extraction Dataset

    New Features

    • lang: :zap: Rework of tokenizer. Additionally implemented new (easier) way of adding languages to the packet(0833859) (by Andrei)
    • lang: :rocket: added support for another 13 languages(fd41af5) (by Andrei)
    • lang: :memo: Added stopwords for af, br, ca,eo, eu, ga, gl, gu, ha, hy, ku, ms, so, st, tl, ur, yo, zu from https://github.com/stopwords-iso(bba7a99) (by Andrei)
    • lang: :memo: Added Burmese language(13670c3) (by Andrei)
    • lang: :memo: Added Slovak language support(4ff82a8) (by Andrei)
    • lang: :memo: Added Czech Language support(afcdc27) (by Andrei)
    • lang: :memo: Added Latvian language support(89f3152) (by Andrei)
    • lang: :memo: Added Telugu Language support(f0f8133) (by Andrei)
    • lang: :memo: Added Marathi language support(ef40042) (by Andrei)
    • lang: :memo: Added Georgian language support(afca45b) (by Andrei)
    • lang: :memo: Added Tamil language support(0bd48ec) (by Andrei)
    • lang: :memo: Added Bengali language support(7a08fc2) (by Andrei)
    • parse: :sparkles: added filter that limits the source.build to a specific category. use source.build(url,only_in_path=True) to scrape only stories that are in the starting url path(665f6fe) (by Andrei)
    • parse: :fire: Source object is now pickleable(af3f80f) (by Andrei)
    • parse: :fire: article is now pickleable(f564524) (by Andrei)
    • sources: :sparkles: New integration of Google news using GNews module. You can now use GoogleNewsSource to search and parse news based on keywords, topic, location or website(33c3409) (by Andrei)
    • sources: :sparkles: new option when building sources. You can limit the article parsing to the source home page only. Other categories or feeds are then ignored(6b8c23e) (by Andrei)
    • misc: :chart_with_upwards_trend: added cloudscraper as optional dependency. If installed, it will us cloudscraper as a layer over requests. Cloudscraper tries to bypass cloudflair protection(720bfe4) (by Andrei)
    • misc: better typing support and type hinting Author: Tom Parker-Shemilt <palfrey@***.net>
    • misc: Simplify favicon return Author: Tom Parker-Shemilt <palfrey@***.net>
    • misc: Basic mypy support Author: Tom Parker-Shemilt <palfrey@***.net>
    • core: added language dependencies, cloudscrape and gnews as optional(cd921a3) (by Andrei)
    • doc: 📝 adding evaluation results
    • doc: 🚀 Documentation Update. Added Examples, documented new features
    • doc: 🔥 Added typing and docstrings to most of the code

    Refactor

    • lang: moving all language related files in languages folder
    • lang: added valid_languages function that returns available languages
    • misc: ⚡ removed ParsingCandidate, RawHelper, URLHelper classes. Removed link_hash from article (was never used)
    • parse: article.link_hash is no longer available
    • parse: ✨ Tidying up the gravity scoring process. No changes in the final score result
    • parse: 🚀 compute word statistics for a node taking children nodes into account
    • core: Minimum Python now 3.8; Also test 3.10/11/12 Author: Tom Parker-Shemilt <palfrey@***.net>
    • core: run gh actions on PR's. Author: Tom Parker-Shemilt <palfrey@***.net>
    • core: Set SETUPTOOLS_USE_DISTUTILS. setuptools as per numpy recommendations. Upgrade numpy and pandas for >= 3.9.Author: Tom Parker-Shemilt <palfrey@***.net>
    • core: Upgrade regex, virtualenv to avoid breaking pre-commit, distutils for everyone. Author: Tom Parker-Shemilt <palfrey@***.net>
    • parse: 💥 deprecated text_cleaned, clean_doc. Removed clean_top_node, article.clean_top_node is removed. Failures if it was accessed

    Bugs fixed:

    • lang: :zap: better is_highlink_density for non-latin languages(a3b6250) (by Andrei)
    • parse: :bug: fixed an issue with non latin high density detection(17a2dad) (by Andrei)
    • parse: :bug: better feed discovery in Source objects(7a3abe9) (by Andrei)
    • parse: :fire: better binary content detection(7ad77cf) (by Andrei)
    • parse: :zap: Better title parsing. Added language specific regex for article titles(d5e8b2b) (by Andrei)
    • parse: :zap: get feeds fixed, it was not parsing the main page for possible feeds(2f7b698) (by Andrei)
    • parse: :fire: better article paragraph detection(0096999) (by Andrei)
    • parse: :zap: added figure as a tag to be removed before text generation(5a226e0) (by Andrei)
    • parse: :zap: Bug with autodetecting website language. If no language supplied, the detected language was not used(07076cb) (by Andrei)
    • misc: :sparkles: tydiing up some code in urls.py(3bb4ca9) (by Andrei)
    • misc: :ambulance: python-setup github action version bump(5bb581e) (by Andrei)
    • misc: :art: mypy stubs for gnews and cloudscraper + small typing fixes(2644f7a) (by Andrei)
    • cli: json output in stdout missing (by Andrei)
    • types: :art: added stubs for gnews(86d7128) (by Andrei)
    Open source →
  7. 0.9.214 Jan 2024
    Release notes
    • You can now us the module as a command line interface (CLI). Usage: python -m newspaper --url https://www.test.com. More information in the documentation.
    • I have added an evaluation script against a dataset from scrapinghub. This will help keeping track of future improvements.
    • Better handling of multithreaded requests. The previous version had a bug that could lead to a deadlock. I implemented ThreadPoolExecutor from the concurrent.futures module, which is more stable. The previously news_pool was replaced with a fetch_news() function.
    • Caching is now much more flexible. You can disable it completely or for one request.
    • You can now use newspaper.article() function for convenience. It will create, download and parse an article in one step. It takes all the parameters of the Article class.
    • protected sites by cloudflare are better detected and raise an exception. The reason will be in the exception message.
    Open source →
    Additional notes

    Some major changes in document parsing. In previous versions the chance that parts of the article body were missing was high. In addition, in some cases the order of the paragraphs was not correct. This release should fix these issues.

    Highlighted features:

    • You can now us the module as a command line interface (CLI). Usage: python -m newspaper --url https://www.test.com. More information in the documentation.
    • I have added an evaluation script against a dataset from scrapinghub. This will help keeping track of future improvements.
    • Better handling of multithreaded requests. The previous version had a bug that could lead to a deadlock. I implemented ThreadPoolExecutor from the concurrent.futures module, which is more stable. The previously news_pool was replaced with a fetch_news() function.
    • Caching is now much more flexible. You can disable it completely or for one request.
    • You can now use newspaper.article() function for convenience. It will create, download and parse an article in one step. It takes all the parameters of the Article class.
    • protected sites by cloudflare are better detected and raise an exception. The reason will be in the exception message.

    New feature:

    • category: :sparkles: improved category link parsing / category link detection(41677b0) (by Andrei)
    • category: :zap: Added option to disable the category_url cache for Source objects. Refactored the cache_disk decorator(670aad9) (by Andrei)
    • cli: :sparkles: added command line interface (CLI) for the module. Usage: python -m newspaper --url https://www.test.com(f46b443) (by Andrei)
    • cli: added output format "text"(31b9079) (by Andrei)
    • core Article.download() and Article.parse() now returns self. Calls can be chained(3be1e47) (by Andrei)
    • lang: :art: automatically load nltk punkt if not present (d0fcdd8) (by Andrei)
    • nlp added the keyword scores as a dictionary attribute in Articles. Additionally, config.MAX_KEYWORDS is really taken into consideration when computing article keywords(f51a04f) (by Andrei)
    • parse: :rocket: improvements in the article body extraction. some sections that were ignored are now added to the extracted text.(1af12d2) (by Andrei)
    • parse: :sparkles: better parametrization of top_node detection. magic constants moved out of the score computation(6485c40) (by Andrei)
    • parse: :triangular_flag_on_post: added some Author detection tags (Issue #347)(4aebf29) (by Andrei)
    • parse: added fine-grained score for top node article attribute booster(0d41fc7) (by Andrei)
    • parse: Added twitch as a video provider (Issue #349, #348)(f4d8f0f) (by Andrei)
    • parse: minor improvement on top node detection(95d5cfa) (by Andrei)
    • parse: parsing rules improvements suggested by @aleksandar-devedzic in issue #577(8677dbe) (by Andrei)
    • requests: :bookmark: Added redirection history from the request calls in Article.download(8ca3d40) (by Andrei)
    • requests: :chart_with_upwards_trend: added a binary file detection. Files that are known binary content-types or have in the first 1000 bytes more than 40% non-ascii characters will raise an exception in article.download.(e7a60dd) (by Andrei)
    • tests: :sparkles: added evaluation script to test against the dataset from https://github.com/scrapinghub/article-extraction-benchmark/(737c226) (by Andrei)

    Bugs fixed:

    • bug: :lipstick: instead of memorize_articles the option / function / parameter was memoize_articles(aaef712) (by Andrei)

    • bug: MEMO_DIR is now Path object. addition with str forgotten from refactoring(0b98e71) (by Andrei)

    • depend: removed feedfinder2 as dependency. was not used(c230aca) (by Andrei)

    • doc: some minor documentation changes(764742a) (by Andrei)

    • lang added additional stopwords for "fa". Issue #398(3453538) (by Andrei)

    • lang: :speech_balloon: fixed serbian stopwords. added chirilic version (Issue #389)(dfcb760) (by Andrei)

    • parse itemprop containing but not equal to articleBody(510be0e) (by Andrei)

    • parse: :art: removed some additional advertising snippets(bd30d48) (by Andrei)

    • parse: :chart_with_upwards_trend: removed possible image caption remains from cleaned article text (Issue #44)(7298140) (by Andrei)

    • parse: :globe_with_meridians: image parsing and movie parsing improvements. get links from additional attributes such as "data-src".(c02bb23) (by Andrei)

    • parse: :memo: exclude some tags from get_text. Tags such as script, option can add garbage to the text output(f0e1965) (by Andrei)

    • parse: :memo: Improved newline geeneration based on block level tags. <br>'s are better taken into account.(22327d8) (by Andrei)

    • parse: added youtu.be to video sources(bf516a1) (by Andrei)

    • parse: additional fixes for caption(3e7fdcc) (by Andrei)

    • refactor: deprecated non pythonic configuration attributes (all caps vs lower caps). for the moment both approaches work(691e12f) (by Andrei)

    • sec: bump nltk and requests min version(553ef27) (by Andrei)

    • sources: :bug: fixed a problem with some type of articlelinks.(9a5c0e2) (by Andrei)

    Open source →
  8. 0.9.18 Nov 2023
    Release notes

    New feature:

    • version bump(f7107be)
    • tests: Add test case for(592f6f6)
    • parse: added possibility to follow "read more" links in articles(0720de1)
    • Allow to pass any requests parameter to the Article constructor. You can now pass verify=False in order to ignore certificate errors (issue #462)(5ff5d27)
    • parse: extended data parsing of json-ld metadata (issue #518)(fc413af)
    • tests: added script to create test cases(9df8c16)
    • parse: added tag for date detection issue #835(41152eb)
    • parse: added og:regDate to known date tags(dc35e29)
    • tests: convert unittest to pytest(45c4e8d)

    Bugs fixed:

    • typing annotation for set python 3.8(895343f)
    • parse: improve meta tag content for articles and pubdate(37bb0b7)
    • parse: 📝 improved author detection. improved video links detection(23c547f)
    • parse: ensured that clean_doc/doc to clean_top_node are on the same DOM. And doc/top_node on the same DOM.(6874d05)
    • small changes, replace os.path with pathlib(5598d95)
    • parse: use one file of stopwords for english, the one in the standard folder #503(6bdf813)
    • parse: better author parsing based on issue #493(f93a9c2)
    • parse: make the url date parsing stricter. Issue #514(0cc1e83)
    • parse: replace \n with space in sentence split (Issue #506)(3ccb87c)
    • parsing: catch url errors resulting resulting from parsed image links(9140a04)
    • correct python versions in pipeline(7e671df)
    • gitignore update(8855f00)
    Open source →
    Additional notes

    New feature:

    • version bump(f7107be) (by Andrei)
    • tests: Add test case for(592f6f6) (by Andrei)
    • parse: added possibility to follow "read more" links in articles(0720de1) (by Andrei)
    • core: Allow to pass any requests parameter to the Article constructor. You can now pass verify=False in order to ignore certificate errors (issue #462)(5ff5d27) (by Andrei)
    • lang Macedonian file raises an error(cadea6a) (by Murat Çorlu)
    • parse: extended data parsing of json-ld metadata (issue #518)(fc413af) (by Andrei)
    • tests: added script to create test cases(9df8c16) (by Andrei)
    • parse: added tag for date detection issue #835(41152eb) (by Andrei)
    • parse: added og:regDate to known date tags(dc35e29) (by Andrei)
    • tests: convert unittest to pytest(45c4e8d) (by Andrei)
    • doc add autodoc for readthedocs (22e9dca) (by Andrei)
    • doc: Added docstring to Article, Source and Configuration.(8e54946) (by Andrei)
    • doc: some clarifications in the documentation(e8126d5) (by Andrei)
    • doc: some template changes(0261054, bfbac2c) (by Andrei)

    Bugs fixed:

    • corec: typing annotation for set python 3.8(895343f) (by Andrei)
    • parse: improve meta tag content for articles and pubdate(37bb0b7) (by Andrei)
    • parse: :memo: improved author detection. improved video links detection(23c547f) (by Andrei)
    • parse: ensured that clean_doc/doc to clean_top_node are on the same DOM. And doc/top_node on the same DOM.(6874d05) (by Andrei)
    • core: small changes, replace os.path with pathlib(5598d95) (by Andrei)
    • parse: use one file of stopwords for english, the one in the standard folder #503(6bdf813) (by Andrei)
    • parse: better author parsing based on issue #493(f93a9c2) (by Andrei)
    • parse: make the url date parsing stricter. Issue #514(0cc1e83) (by Andrei)
    • parse: replace \n with space in sentence split (Issue #506)(3ccb87c) (by Andrei)
    • parsing: catch url errors resulting resulting from parsed image links(9140a04) (by Andrei)
    • repo: correct python versions in pipeline(7e671df) (by Andrei)
    • repo: gitignore update(8855f00) (by Andrei)
    Open source →
  9. 0.9.029 Oct 2023
    Release notes

    First release after the fork. This release is based on the 0.1.7 release of the original newspaper3k project. I jumped versions such that it is clear that this is a fork and not the original project.

    New feature:

    • tests: starting moving tests to pytest(f294a01) (by Andrei)
    • parser: add yoast schema parse for date extraction(39a5cff) (by Andrei)

    Bugs fixed:

    • docs: update README.md(d5f9209) (by Andrei)
    • parse: feed_url parsing, issue #915(ec2d474) (by Andrei)
    • parse: better content detection. added <article> and <div> tag as candidate for content parent_node(447a429) (by Andrei)
    • core: close pickle files - PR #938(d7608da) (by Andrei)
    • parse: improved publication date extraction(4d137eb) (by Andrei)
    • core: some linter errors, whitespaces and spelling(79553f6) (by Andrei) <!-- package description limit --> ################################### These are the original newspaper3k release notes ################################### ########################################################################################################################
    Open source →