pymupdf4llm

setup.py: add _build.py file to wheel/install containing git info.

pdf4llm/setup.py: increment version to 1.27.2.3.

CHANGES.md: add version number.

pymupdf4llm/: fix naming bug and update version to 1.27.2.1.

For change details please consult this file.

For change details please consult this file.

For a description of changes see this file.

For a summary of changes see file https://github.com/pymupdf/pymupdf4llm/blob/main/CHANGES.md

Fixes:

• 323 - page_chunks=True parameter was ignored in PyMuPDF-Layout mode

Other Changes:

• Methods to_markdown() / to_text() now both support Page chunk output via parameter page_chunks=True.

Fixes:

• Forum - List index out of range ...

Other Changes:

Fixes:

• 341 - Broken markdown parsing for new line directly followed by 'o'...

Other Changes:

• New parameter table_format in method to_text() (PyMuPDF-Layout only). This allows selecting the appearance of tables in plain text outputs. The possible values are defined in the list tabulate.tabulate_formats. Default is "grid".
• Installing PyMuPDF4LLM now supports including all optional dependencies in the pip command: pip install --upgrade pymupdf4llm[ocr,layout]. This will install pymupdf4llm, pymupdf, and pymupdf-layout. The "ocr" parameter - when needed - installs opencv-python for automatic OCR support in PyMuPDF-Layout mode. Combine this with parameters --upgrade, --force-reinstall or --no-cache-dir as necessary.
• Major rework of the heuristics that determine whether a page should be OCR'd.

Fixes:

• 335 - KeyError "has_ocr_text"

Other Changes:

Fixes:

• 332 - TypeError("to_markdown() got an unexpected keyword argument 'header'")

Other Changes:

• Output (backend) methods now accept a new parameter ocr_dpi=400 which sets the OCR resolution for full-page OCR.
• The OCR detection heuristics is more fine-grained and now detects more OCR situations.
• Resolved multiple performance issues, specifically cases with overwhelmingly many images and extremely large StructTreeRoot objects in PDF.
• Reflected layout-specific API changes in the legacy code and will now raise NotImplementedError exceptions when layout-only features are used.
• Information messages during document parsing are now written to stdout collectively at the end of the phase. This applies to announcing page OCR decisions specifically.
• Support parameter page_separators as in the legacy mode.

Fixes:

• 320 - [Bug] ValueError: min() iterable argument is empty ...
• 319 - [Bug] ValueError: min() arg is an empty sequence

Other Changes:

• OCR invocation now differentiates between full-page OCR and text-only OCR: If the page does contain text but the percentage of unreadable characters exceeds a certain threshold (90%), we only OCR text span boundary boxes and replace span text with OCR'ed text where necessary.

This version introduces full support of the PyMuPDF-Layout package. This entails a radically new approach for detecting the layout of document pages using the AI-based features of the layout package.

Improvements include:

• Greatly improved table detection
• Support of list item hierachy levels
• Detection of page headers and footers
• Improved detection of text paragraphs, titles and section headers
• New output options beyond Markdown: plain text and JSON
• Automatically detect whether a page needs OCR and invoke Tesseract if both, Tesseract is installed and OpenCV (package opencv-python) is available. Invocation criteria include absence of readable text, full-page coverage with images, presence of many character-sized vector graphics.

The PyMuPDF-Layout package is not open-source and has its own license, which is different from PyMuPDF4LLM. It also is dependent on a number of other, fairly large packages like onnxruntime, numpy, sympy and OpenCV, which each in turn have their own dependencies.

We therefore keep the use of the layout feature optional. To activate PyMuPDF-Layout support the following import statement must be included before importing PyMuPDF4LLM itself:

import pymupdf.layout
import pymupdf4llm

Thereafter, PyMuPDF's namespace is available. The known method pymupdf4llm.to_markdown() automatically works with AI-based empowerment. In addition, two new methods become available:

• pymupdf4llm.to_text() - which works much like markdown output but produces plain text.
• pymupdf4llm.to_json() - which outputs the document's metadata and the selected pages in JSON format.

Fixes:

Other Changes:

• If show_progress=True, Python package tqdm is automatically used when available to display a progress bar. If tqdm is not installed, our own text-based progress bar is used.

Fixes:

• 296 - [Bug] A specific diagram recognized as significant ...
• 294 - Unable to extract images from Page
• 272 - Disappeared page breaks

Other Changes:

• Added new parameter to to_markdown: page_separators=False. If True and page_chunks=False a line like --- end of page=nnn --- is appended to each pages markdown text. The page number is 0-based. Intended for debugging purposes.

Fixes:

• 289 - Content Duplication with the latest version
• 275 - Text with background missing from output
• 262 - Markdown error parsing

Other Changes:

• The table module in package PyMuPDF has been modified: Its method to_markdown() will now output markdown-styled cell text. Previously, table cells were extracted as plain text only.

• The class TocHeaders is now a top-level import and can now be directly used.

• Method to_markdown has a new parameter detect_bg_color=True (default) which guesses the page's background color. If a background is detected, fill-only vectors having this color are ignored. False will always consider "fill" vectors in vector graphics detection.

• Text written with a Type 3 font will now always be considered. Previously, this text was always treated as invisible and was hence suppressed.

• The package now contains the license file GNU Affero GPL 3.0 to ease distribution (see LICENSE). It also clarifies that PyMuPDF4LLM is dual licensed under GNU AGPL 3.0 and individual commercial licenses.

• There is a new file versions_file.py which contains version information. This is used to ensure the presence of a minimum PyMuPDF version at import time.

Fixes:

• 282 - Content Duplication with the latest version
• 281 - Latest version of pymupdf4llm.to_markdown returns empty text for some PDFs.
• 280 - Cannot extract text when ignore_images=False, can extract otherwise.
• 278 - Title words are fragmented
• 249 - Title duplication problem in markdown format
• 202 - BAD RECT ISSUE

Other Changes:

• The table module in package PyMuDDF has been: Its method to_markdown() will now output markdown-styled cell text. Previously, table cells were extracted as plain text only.

• The class TocHeaders is now a top-level import and can now be directly used.

• Text written with a Type 3 font will now always be considered. Previously, this text was always treated as invisible and was hence suppressed.

Fixes:

• Fixing "UnboundLocalError"

Other Changes:

Fixes:

• 265 - Code error correction
• 263 - Table Strategy = None raises error
• 261 - wrong markdown in latest pymupdf versions

Other Changes:

• Highspeed vector graphics count: if graphics_limit is specified, drawings are no longer extracted for counting purposes.

Fixes:

• 251 - Images a little larger than the page size are being ignored
• 255 - Single-row/column tables are skipped
• 258 - Pymupdf4llm to_markdown crashes on some documents

Other Changes:

• Added class TocHeaders as an alternative way for identifying headers.

Fixes:

• 116 - Handling Graphical Images & Superscripts

Other Changes:

Fixes:

• 171 - Text rects overlap with tables and images that should be excluded.
• 189 - The position of the extracted image is incorrect
• 238 - When text is laid out around the picture, text extraction is missing.

Other Changes:

• Added new parameter ignore_images: (bool) optional. True will not consider images in any way. May be useful for pages where a plethora of images prevents meaningful layout analysis. Typical examples are PowerPoint slides and derived / similar pages.

• Added new parameter ignore_graphics: (bool), optional. True will not consider graphics except for table detection. May be useful for pages where a plethora of vector graphics prevents meaningful layout analysis. Typical examples are PowerPoint slides and derived / similar pages.

• Added new parameter to class IdentifyHeaders: Use max_levels (integer <= 6) to limit the generation of header tag levels. e.g. headers = pymupdf4llm.IdentifyHeaders(doc, max_level=3) ensures that only up to 3 header levels will ever be generated. Any text with a font size less than the value of ### will be body text. In this case, the markdown generation itself would be coded as md = pymupdf4llm.to_markdown(doc, hdr_info=headers, ...).

• Changed parameter table_strategy: When specifying None, no effort to detecting tables will be made. This can be useful when tables are of no interest or known to not exist in a given file. This will speed up processing significantly. Be prepared to see more changes and extensions here.

Fixes:

The following list includes fixes made in version 0.0.18 already.

• 158 - Very long titles when converting to markdown.
• 155 - Inconsistent image extraction from image-only PDFs
• 161 - force_text param ignored.
• 162 - to_markdown isn't outputting all the pages but get_text is.
• 173 - First column of table is repeated before the actual table.
• 187 - Unsolicited Text Particles
• 188 - Takes lot of time to convert into markdown.
• 191 - Extraction of text stops in the middle while working fine with PyMuPDF.
• 212 - In pymupdf4llm, if a page has multiple images, only 1 image per-page is extracted.
• 213 - Many ���� after converting when using pymupdf4llm
• 215 - Spending too much time on identifying text bboxes
• 218 - IndexError in get_raw_lines when processing PDFs with formulas
• 225 - Text with background missing from output.
• 229 - Duplicated Table Content on pymuPDF4LLM.

Other Changes:

• Added new parameter filename: (str), optional. Overwrites or sets the filename for saved images. Useful when the document is opened from memory.

• Added new parameter use_glyphs: (bool), optional. Request to use the glyph number (if possible) of a character if the font has no back-translation to the original Unicode value. The default is False which causes � symbols to be rendered in these cases.

• Added strike-out support: We now detect and render striked-out text.

• Improved background color detection: We have introduced a simple background color detection mechanism: If a page shows an identical color in all four corners, we assume this to be the background color. Text and vector graphics with this color will be ignored as invisible.

• Improved invisible text detection: Text with an alpha value of 0 is now ignored.

• Improved fake-bold detection: Text mimicking bold appearance is now treated like standard bold text in most cases.

• Header handling changes:

  • Detection now happens based on the largest font size of the line.
  • Uniformly rendered: All spans of a header line will now be rendered with the same appearance.

• Changed handling of parameter graphics_limit: We previously ignored a page completely if the vector graphics count exceeded the limit. We now only ignore vector graphics if their count outside table boundary boxes is too large. This should only suppress vector graphics on the page, while keeping images, text and table content extractable.

• Changed the margins default to 0. The previous default (0, 50, 0, 50) ignored 50 points at the top and bottom of pages. This has turned out to cause confusion in too many cases.

Fixes:

• 147 - Error when page contains nothing but a table.
• 81 - Issues with bullet points in PDFs.
• 78 - multi column pdf file text extraction.

Fixes:

• 138 - Table is not extracted and some text order was wrong.
• 135 - Problem with multiple columns in simple text.
• 134 - Exclude images based on size threshold parameter.
• 132 - Optionally embed images as base64 string.
• 128 - Enhanced image embedding format.

Improvements

• New parameter embed_images (bool) embeds images and vector graphics in the markdown text as base64-encoded strings. Ignores write_images and image_path parameters.
• New parameter image_size_limit which is a float between 0 and 1, default is 0.05 (5%). Causes images to be ignored if their width or height values are smaller than the corresponding fraction of the page's width or height.
• The algorithm has been improved which determins the sequence of the text rectangles on multi-column pages.
• Change of the header identification algorithm: If more than six header levels are required for a document, then all text with a font size larger than body text is assumed to be a header of level 6 (i.e. HTML "h6" = "###### ").

Fixes

• 112 - Invalid bandwriter header dimensions/setup.

Improvements

• New parameter ignore_code suppresses special formatting of text in mono-spaced fonts.
• New parameter extract_words enforces page_chunks=True and adds a "words" list to each page dictionary.

Fixes

• 90 - 'Quad' object has no attribute 'tl'.
• 88 - Bug in is_significant function.

Improvements

• Extended the list of known bullet point characters.

Fixes

• 73 - bug in to_markdown internal function.
• 74 - minimum area for images & vector graphics.
• 75 - Poor Markdown Generation for Particular PDF.
• 76 - suggestion on useful api parameters.

Improvements

• Improved recognition of "insignificant" vector graphics. Graphics like text highlights or borders will be ignored.
• The format of saved images can now be controlled via new parameter image_format.
• Images can be stored in a specific folder via the new parameter image_path.
• Images are not stored if contained in another image on same page.
• Images are not stored if too small: if width or height are less than 5% of corresponding page dimension.
• All text is always written. If write_images=True, text on images / graphics can be suppressed by setting force_text=False.

Fixes

• 71 - Unexpected results in pymupdf4llm but pymupdf works.
• 68 - Issue with text extraction near footer of page.

Improvements

• Improved identification of scattered text span particles. This should address most issues with out-of-sequence situations.
• We now correctly process rotated pages (see issue 68).

Fixes

• 65 - Fix typo in pymupdf_rag.py.

Fixes

• 54 - Mistakes in orchestrating sentences. Additional fix: text extraction no longer uses the TEXT_DEHYPHENATE flag bit.

Improvements

• Improved the algorithm dealing with vector graphics. Vector graphics are now more reliably classified as irrelevant: We now detect when "strokes" only exist in the neighborhood of the graphics boundary box border itself. This is quite often the case for code snippets.

Fixes

• 55 - Bug in helpers/multi_column.py - IndexError: list index out of range.
• 54 - Mistakes in orchestrating sentences.
• 52 - Chunking of text files.
• Partial fix for 41 / 40 - Improved page column detection, but still no silver bullet for overly complex page layouts.

Improvements

• New parameter dpi to specify the resolution of images.
• New parameters page_width / page_height for easily processing reflowable documents (Text, Office, e-books).
• New parameter graphics_limit to avoid spending runtimes for value-less content.
• New parameter table_strategy to directly control the table detection strategy.