fivefilters/readability.php

What's Changed

• Lazy Article properties + metadataOnly option (4.1.0) by @fivefilters in #52

Full Changelog: v4.0.0...v4.1.0

Added

• metadataOnly option (PHP-specific): extract only the title and metadata, skipping content extraction entirely. parse() then skips every stage that mutates the document (noscript image unwrapping, script removal, document prep, and the article grab itself), so a passed-in \Dom\HTMLDocument is guaranteed to be left unmodified. The returned Article has the content-derived properties set to null, as when no content is found (hasContent() returns false)

Changed

• Article's content-derived properties (content, textContent, length, images) are now computed lazily from contentElement on first access (via property hooks) and cached, so callers that only use contentElement, hasContent() or the metadata no longer pay for serializing the article HTML and text on every parse. Reads are unchanged (same names, same types, same values); hasContent() now checks contentElement. json_encode() and var_dump() output is preserved via jsonSerialize()/__debugInfo() (note that both trigger the full serialization). Two edges to note: the Article constructor no longer takes content/textContent/length/images (it derives them from contentElement), and the lazy properties, being virtual, no longer appear in get_object_vars() or (array) casts

What's Changed

Readability.php 4.0 is a ground-up rewrite: a fresh port of Mozilla's Readability.js v0.6.0 built on PHP 8.4's new DOM API and HTML parser. This is the first stable release of the 4.0 series and includes everything from v4.0.0-beta.1.

Highlights

• PHP 8.4+ with native WHATWG parsing — parsing and serialization use Dom\HTMLDocument (the spec-compliant Lexbor parser bundled with PHP), replacing HTML5-PHP and the legacy libxml path. No more entity workarounds (substituteEntities, summonCthulhu and friends are gone).
• Parity with Readability.js 0.6.0 — including lang and publishedTime output, maxElemsToParse, classesToPreserve, allowedVideoRegex, linkDensityModifier and debug options, and Readerable::isProbablyReaderable().
• New result API — parse() returns a readonly Article value object (title, content, textContent, excerpt, byline, siteName, dir, lang, publishedTime, image, images, …). Options are passed as named constructor arguments: new Readability(fixRelativeURLs: true).
• Real WHATWG URL resolution — relative URLs are fixed using PHP 8.5's native Uri\WhatWg\Url when available, with rowbot/url as the fallback on PHP 8.4, matching the new URL() behavior Readability.js relies on.
• Tested against Mozilla's own corpus — the test suite runs Mozilla's 130 test pages verbatim, with their structural DOM comparison ported to PHP.
• Broader dependency compatibility — constraints widened to psr/log ^1.0 || ^2.0 || ^3.0 and rowbot/url ^3.1.7 || ^4.0 so the library co-installs cleanly alongside a wider range of applications, with CI now also testing the lowest allowed versions (#49).

Upgrading from 3.x

The API has changed — see UPGRADE.md for the full migration guide, and the CHANGELOG for the complete list of changes.

Upgrading from v4.0.0-beta.1

The only changes since beta.1 are the widened dependency constraints and lowest-deps CI coverage noted above — no code changes.

Full Changelog: v3.3.3...v4.0.0

First stable release of the 4.0 series — a ground-up rewrite. This release includes all changes from v4.0.0-beta.1 below (see UPGRADE.md for the 3.x → 4.0 migration guide), plus the following changes since beta.1:

Changed

• Widened dependency constraints so the library co-installs with a broader range of applications: psr/log now accepts ^1.0 || ^2.0 || ^3.0 (the library only consumes the interface, so every major works) and rowbot/url now accepts ^3.1.7 || ^4.0 (older majors avoid rowbot/url 4.x's hard requirements on psr/log ^3 and newer brick/math, both common sources of version conflicts). Note: rowbot/url 3.x emits PHP 8.4 implicit-nullable deprecation notices internally; it is only loaded on PHP 8.4 (PHP 8.5+ uses the native Uri\WhatWg\Url), and Composer still picks 4.x unless another dependency forces an older version
• CI now also runs the test suite against the lowest allowed dependency versions (composer update --prefer-lowest) to keep the widened constraints honest

What's Changed

• v4.0: Rewrite on PHP 8.4's native DOM API, parity with Readability.js 0.6.0 by @fivefilters in #41
• Test corpus is Mozilla's own 130 test pages, verbatim
• See the CHANGELOG for full details and UPGRADE.md for the 3.x → 4.0 migration guide.

Full Changelog: v3.3.3...v4.0.0-beta.1

Ground-up port from the latest Readability.js (v0.6.0) using Claude's Fable model. Uses PHP 8.4's new DOM API and native parser. See UPGRADE.md for the full 3.x → 4.0 migration guide.

Changed

• Requires PHP >= 8.4; parsing and serialization use Dom\HTMLDocument (the WHATWG-spec Lexbor parser bundled with PHP), replacing HTML5-PHP and the legacy libxml path
• parse() now returns a readonly Article value object (title, content, textContent, length, excerpt, byline, siteName, dir, lang, publishedTime, image, images, contentElement). When no article content is found (where Readability.js returns null), the Article still carries the extracted title and metadata with null content — check Article::hasContent(). ParseException is thrown only for empty input or documents over maxElemsToParse
• Options are passed directly to the Readability constructor as named arguments, like the options object in Readability.js (new Readability(fixRelativeURLs: true)), and are all optional; a readonly Configuration object taking the same named arguments can be passed instead. maxTopCandidates renamed to nbTopCandidates (matching Readability.js)
• Article output is wrapped in <div id="readability-page-1" class="page">, as in Readability.js
• Byline is always extracted into Article::$byline; the articleByline option becomes keepInlineByline, which only controls whether an inline byline stays in the content (default removes it, as in Readability.js)
• Image extraction moved onto the result object: getImage()/getImages() become $article->image / $article->images
• PSR-3 logging: pass a LoggerInterface as the logger option instead of setLogger()
• Relative URL resolution now uses a real WHATWG URL parser — PHP 8.5's native Uri\WhatWg\Url when available, rowbot/url on PHP 8.4 — matching the new URL() behavior Readability.js relies on; replaces league/uri
• Test corpus replaced with Mozilla's 130 test pages verbatim; content comparison ports Mozilla's structural DOM comparison

Added

• Parity with Readability.js 0.6.0: lang and publishedTime output; maxElemsToParse, classesToPreserve, allowedVideoRegex, linkDensityModifier and debug options; aria-modal dialog removal; ad/loading-indicator stripping; parsely/article:author/itemprop metadata sources; JSON-LD @graph, @context-object and array handling; Unicode comma scoring; updated regexes (mathjax, bilibili, en/em-dash title separators)
• Readerable::isProbablyReaderable(), a port of Readability-readerable.js
• parse() accepts an already-parsed Dom\HTMLDocument as well as an HTML string
• Cross-check harness (test/tools/) that diffs this port's output against Readability.js over the whole corpus
• Static analysis with Psalm (composer analyse), run in CI alongside the test suite

Removed

• HTML5-PHP dependency; ext-xml requirement
• Options that existed as libxml workarounds: parser, substituteEntities, normalizeEntities, summonCthulhu
• The custom DOM subclass layer (src/Nodes/) and its workarounds (attribute-based state, shifting-aware iteration)
• The Docker-based local test setup (docker-compose.yml, Makefile, docker/); tests and static analysis run directly on PHP 8.4/8.5, locally and in CI

Fix type error - extends type support to add DOMProcessingInstruction in more method signatures (reported in
#39 by @reinierkors)

Full Changelog: v3.3.2...v3.3.3

• Fix type error - extends type support to add DOMProcessingInstruction in more method signatures (reported by @reinierkors)

Fix type error - extends type support to include DOMCdataSection and DOMProcessingInstruction in various method signatures (reported in
#37 and #38 by @mikiescolarmrf and @Grotax)

Full Changelog: v3.3.1...v3.3.2

• Fix type error - extends type support to include DOMCdataSection and DOMProcessingInstruction in various method signatures (reported by @mikiescolarmrf and @Grotax)

What's Changed

• fix: allow DOMProcessingInstruction by @blizzz in #36, and #33 #35

Full Changelog: v3.3.0...v3.3.1

• Fix DOMProcessingInstruction errors

• Fixed PHP 8.4 deprecation warning #31 (reported by @pich)
• Migrated type declarations from PHPDoc blocks to native PHP 8 property and method types
• Empty class attributes now removed when keepClasses is disabled
• Replaced legacy DOM operations with native PHP 8 methods:
  • isWhitespaceInElementContent() for whitespace detection
  • firstElementChild and previousElementSibling for DOM traversal
• Updated Docker test environment to support PHP 8.1-8.4

Full Changelog: v3.2.0...v3.3.0

• Fixed PHP 8.4 deprecation warning (reported by @pich)
• Migrated type declarations from PHPDoc blocks to native PHP 8 property and method types
• Empty class attributes now removed when keepClasses is disabled
• Replaced legacy DOM operations with native PHP 8 methods:
  • isWhitespaceInElementContent() for whitespace detection
  • firstElementChild and previousElementSibling for DOM traversal
• Updated Docker test environment to support PHP 8.1-8.4

• Update dependencies to newer versions (League/URI version 7) to improve compatibility with projects already relying on those versions
• Minimum PHP version set to 8.1 (required by League/URI 7)
• Update Docker tests to use PHP 8.1, 8.2 and 8.3

Full Changelog: v3.1.7...v3.2.0

• Update dependencies to newer versions (League/URI version 7), to make it compatible with projects already relying on those versions
• Minimum PHP version set to 8.1 (required by League/URI 7)
• Update Docker tests to use PHP 8.1, 8.2 and 8.3

• Fixes URL syntax errors when bad URLs are encountered when rewriting relative URLs - reported by @marcelklehr
• Fixes PHP 8 deprecation notice when base URLs (used for rewriting relative URLs) don't have a path component - thanks to @blat and @Markus-GS

Full Changelog: v3.1.6...v3.1.7

• Fixes URL syntax errors when bad URLs are encountered when rewriting relative URLs - reported by @marcelklehr
• Fixes PHP 8 deprecation notice when base URLs (used for rewriting relative URLs) don't have a path component - thanks to @blat and @Markus-GS

What's Changed

• Avoid re-parsing source HTML when making multiple attempts to identify content in parse()

Full Changelog: v3.1.5...v3.1.6

• Avoid re-parsing source HTML when making multiple attempts to identify content in parse()

• Allow psr/log version 2.x and 3.x - thanks to @piotrek-r and @ArondeParon

• Fixes improper use of null coalescing operator - reported by @thedf

• Fixes issue where exception was thrown when resolving an invalid relative URL (when setFixRelativeURLs(true)) - reported by @jeffbotw

• Fixes issue "Warning: Undefined array key 2" reported by @castroCrea
• Fixes issue "Notice: Trying to get property '' of non-object" reported by @thedf

• Exclude tests folder when using composer

• Minimum PHP version 7.4 (composer.json updated)
• Updated the Docker file to support versions of PHP from 7.4 to 8.1
• Updated the Docker file to allow you to run PHP with libxml 2.9.10, 2.9.13, 2.9.14
• Test with PHP 8.1

• Implemented changes made to Readability.js up to 26 August 2021, with the exception of a piece of code which doesn't produce the same results in PHP for us compard to the JS version.
• Default parser is now HTML5-PHP, which handles HTML better than libxml
• Replaced the expected HTML files in the tests folder to reflect HTML5-PHP's serialisation
• Updated the Docker file to support versions of PHP from 7.3 to 8.0 (previously it was 7.0 to 7.3)
• Updated the Docker file to allow you to run PHP with libxml 2.9.4, 2.9.5, 2.9.10, and 2.9.12
• Fatal error bug fix (thanks Balazsp)

• Avoid overwriting extracted metadata with similarly named keys (like og:image and og:image:width)
• Imported new getSiteName() feature from JS version as of 21 Dec 2018
• Added getFirstElementChild function to NodeTrait + test case (Issue #83)
• Reworked the test suit to use TestPage objects and give more hints about what failed
• Removed getWordThreshold and setWordThreshold configuration functions
• Added NodeUtility::filterTextNodes and deprecated NodeTrait getChildren()
• Added new DOMNodeList fake class that mimics the original DOMNodeList class but allows to add new nodes to the list
• Added new Dockerfiles that pulls different versions of PHP and libxml. Now we are supporting 4 versions of PHP and 6 versions of libxml!

• Fixed small issue that prevented the main image from showing up in the results

• [BREAKING CHANGE] Bumped the minimum supported version of PHP to 7.0
• Clean <aside> tags during prepArticle().
• Merged PR #58: Fix notice non-object on $parentOfTopCandidate for tumblr.com
• Fixed issue #63: Division by zero
• Housekeeping:
  • Removed $parseSuccessful flag that wasn't needed anymore
• Rename wordThreshold to charThreshold and throw deprecation notices. WordThreshold will be removed in version 3.0.
• Added "-ad-" as unlikely candidate
• Added Docker containers with PHP 7.0, 7.1, and 7.2 and makefile to trigger the tests.
• Imported new code from the JS version as of 19 Nov 2018, which includes the following changes:
  • Move phrasing contents into paragraphs
  • Improved the title detection
  • Remove single cell tables
  • Improved the detection of video related elements
  • New test cases
  • Various minor fixes

• Merged PR#49 (Missing object when calling ->getContent())
• Imported all changes from Readability.js as of 2 March 2018 (8525c6a):
  • Check for <base> elements before converting URLs to absolute.
  • Clean <link> tags on prepArticle()
  • Attempt to return at least some text if all the algorithm runs fail (Check PR #423 on JS version)
  • Add new test cases for the previous changes
  • And all other changes reflected in this diff

• Switched from assertEquals to assertSame on unit testing to avoid weak comparisons.
• Added a safe check to avoid sending the DOMDocument as a node when scanning for node ancestors.
• Fix issue #45: Small mistake in documentation
• Fix issue #46: Added data-src as a image source path
• Fixed bug when extracting all the image of the article (Was extracting images from the original DOM instead of the parsed one)
• Added the ->getDOMDocument() getter to retrieve the fully parsed DOMDocument
• Merged PR #48 that allows passing an array as configuration (@topotru)

• Added 'data-orig' as an URL source for images
• Removed 'modal' as a negative property from classes
• Added option to inject a logger
• Removed all references to the data-readability tags that don't apply anymore to the new structure
• Merged PR #38 (Missing DOMEntityReference)

• Node encapsulation is gone. Pre v1 all nodes where encapsulated in a Readability class, which created lots of trouble with dependencies, responsibilities, and properties. Now all the encapsulation is gone: all the DOMNodes inside the Readability class are extensions of the original DOM classes, which allows the system to take advantage of the functions and properties of DOMDocument.
• HTMLParser is gone, Readability is the new main class. Switched things a bit for this release. Pre v1 you had to create an HTMLParser class to parse the HTML. Now you have to create a Readability class, feed it the text, and check the result.
• No more dumb arrays as a result. If you want to get the title, content, images, or anything else you'll have to use the getters of the Readability class.
• Environment class is gone. Now you have to create a configuration class and use setters to set your configuration options.
• Exceptions. Make sure you wrap your Readability class in a try catch block, because if it fails to parse your HTML, it will throw a ParseException.
• Minimum PHP version bumped to 5.6.

• Trim titles when detecting hierarchical separators to avoid false negatives on strings with spaces.
• Fix issue when converting divs to p nodes and never rating them (issue #29)
• Fix "Unsupported operand types" (PR #31)
• Fix division by zero when no title was found (issue #32)
• New function to retrieve all images at once (PR #30)
• Get the title from the <title> tag before searching on the <meta> tags

• Merged PR #24. Fixes notice when trying to extract og:image
• Up to date to commit eb221c5 (2017-10-16), which includes the following changes:
  • New tags added to the unlikelyCandidates regex
  • Detection and removal of hierarchical separators in titles
  • Added more tags to clean after parsing the article (button, textarea, select, etc.)
  • New way to detect empty nodes (including a edge case where a node with a &nsbp; was detected as a node with content)
  • Better approach to find a top candidate (specially when a top candidate is the only child of a parent node, which allows a more accurate joining of sibling elements)
  • Detect text direction (ltr or rtl)
  • Detect and mark data tables to avoid removing them during final clean up
  • Major fixes when scanning and deleting nodes (no need to traverse backwards anymore)
  • Node cleaning via regex matches
  • Clean table attributes during final clean up.
• Added license

Next release after this one will be v1 and will be a major refactor around Readability and HTMLParser methods and responsibilities.

• Added a safecheck for really nasty HTML
• Added summonCthulhu option, to remove all script tags via regex

• Added normalizeEntities flag to convert UTF-8 characters to its HTML Entity equivalent. Fixes bugs on htmls with mixed encoding.
• Added more information to the readme.md file
• New way to create a backup DOM: not creating a backup. In the previous version, the system cloned the $this->dom object to keep it as a backup in order to restart the algorithm with other flags, if needed. This seemed to work until I realized that sometimes the backup changes even if we are not touching it. Seems that the dom and backupdom objects are linked and some changes on the dom object reach the bakcupdom object. The new approach consists in deleting the backupdom object and recreating from scratch the dom object. Of course this has a performance impact, but seems to be quite low.

100% complete port of Readability.js!

• Every test unit passes
• Readability.php produces the same exact output as Readability.js
• I'm happy :)

Fixed

• Lots of bugs
• Merged PR by DavidFricker to avoid exceptions while grabbing the document content

Added

• substituteEntities flag, to avoid replacing especial characters with HTML entities. There's nothing we can do about  , that entity is replaced by libxml and there's no way to disable it.
• Named data sets so it's easier to detect which test case is failing.

Removed

• Couple of test cases that involved broken JS. There's nothing we can do about JS spilling onto the text.

We are getting closer to be a 100% complete port of Readability.js!

• Added prepArticle to remove junk after selecting the top candidates.
• Added a function to restore score after selecting top candidates. This basically works by scanning the data-readability tag and restoring the score to the contentScore variable. This is an horrible hack and should be removed once we ditch the Element interface of html-to-markdown and start extending the DOMDocument object.
• Switched all strlen functions to mb_strlen
• Fixed lots of bugs and pretty sure that introduced a bunch of new ones.

• Last version I'm using master as the main development branch. All unreleased changes and main development will happen in the develop branch.

• Initial release