📢 Spark NLP 6.4.2: BM25 Retrieval, SaT Sentence Detection, and Model Conversion Workflows

Spark NLP 6.4.2 is a feature and reliability release focused on strengthening retrieval, sentence segmentation, model conversion, and release infrastructure. This release introduces BM25Approach and BM25Model for Okapi BM25 lexical retrieval, adds SentenceDetectorSaTModel for transformer-based sentence boundary detection with SaT / Segment any Text models, and expands model conversion examples with new Hugging Face to Spark NLP notebooks for ONNX, OpenVINO, and GGUF workflows.

In addition, this release improves production robustness by properly closing resource streams and file systems during local resource copying, which avoids leak warnings that can affect dependent pipelines. It also updates jsl-llamacpp to 2.0.3.

🔥 Highlights

• BM25Approach + BM25Model: New Okapi BM25 lexical retrieval annotators implemented as a fit-once, query-many Estimator/Model pair. They complement semantic retrieval components such as DocumentSimilarityRanker with a fast, corpus-statistics-based lexical ranking option.
• SentenceDetectorSaTModel: New sentence detector based on SaT / Segment any Text transformer models. It uses ONNX-backed XLM-R SentencePiece models to produce sentence spans from per-token boundary probabilities and supports long documents through overlapping token windows.
• Model conversion notebooks: New end-to-end notebooks demonstrate Hugging Face to Spark NLP conversion flows for ONNX, OpenVINO, and GGUF models.

🚀 New Features & Enhancements

BM25 Lexical Retrieval

Spark NLP 6.4.2 introduces BM25 lexical document retrieval through a two-stage Spark ML design. This implements SPARKNLP-1145 and adds a purely lexical ranking option that complements semantic retrieval components such as DocumentSimilarityRanker.

• BM25Approach scans the tokenized corpus during fit() and learns corpus-level statistics: document count, per-term document frequency, average document length, and inverse document frequency.
• BM25Model reuses those learned statistics at transform time to score each document against a query.

This design is necessary because BM25 scores depend on corpus-level statistics that are only known after fitting over the full dataset. Once fitted, the model can be reused across many queries without rescanning the corpus.

Key capabilities:

• Emits BM25_RANKINGS annotations.
• Adds bm25_score, num_query_terms_matched, query, and doc_len metadata.
• Supports range-validated k1, b, minDocFreq, and caseSensitive parameters.
• Uses the non-negative Lucene / Elasticsearch-style IDF variant.
• Supports setQuery(...) for raw query strings.
• Supports setQueryTokens(...) for pre-analyzed query terms when the corpus pipeline includes normalization, stemming, lemmatization, or other token transformations.
• Makes caseSensitive read-only on fitted BM25Model to avoid corrupting scores after the IDF vocabulary has been built.
• Includes Scala and Python tests for ranking, save/load, exact-score verification, query reuse, parameter propagation, analyzer symmetry, and invalid parameter handling.

from sparknlp.base import DocumentAssembler
from sparknlp.annotator import Tokenizer, StopWordsCleaner, BM25Approach
from pyspark.ml import Pipeline
from pyspark.sql.functions import col, explode

corpus = spark.createDataFrame([
    (1, "Apples are a great source of dietary fiber and vitamin C."),
    (2, "Machine learning uses neural networks and statistical models."),
    (3, "Vitamin C deficiency can affect the immune system."),
], ["id", "text"])

document = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer() \
    .setInputCols(["document"]) \
    .setOutputCol("token")

stop_words = StopWordsCleaner() \
    .setInputCols(["token"]) \
    .setOutputCol("clean_token") \
    .setCaseSensitive(False)

bm25 = BM25Approach() \
    .setInputCols(["clean_token"]) \
    .setOutputCol("bm25_rankings") \
    .setK1(1.2) \
    .setB(0.75) \
    .setMinDocFreq(1) \
    .setCaseSensitive(False)

pipeline = Pipeline(stages=[document, tokenizer, stop_words, bm25])
model = pipeline.fit(corpus)

# Re-query the fitted model without recomputing corpus statistics.
model.stages[-1].setQuery("vitamin C health")
result = model.transform(corpus)

result.select(col("id"), explode(col("bm25_rankings")).alias("ranking")) \
    .select(
        col("id"),
        col("ranking.metadata")["bm25_score"].cast("double").alias("bm25_score"),
        col("ranking.metadata")["num_query_terms_matched"].cast("int").alias("terms_matched"),
    ) \
    .orderBy(col("bm25_score").desc()) \
    .show(truncate=False)

For pipelines that transform tokens before BM25 training, use setQueryTokens(...) with query tokens produced by the same analysis pipeline. This avoids query/document analyzer asymmetry and ensures query terms match the learned IDF vocabulary.

See the new BM25 Retrieval notebook for a complete example.

SentenceDetectorSaTModel

This release adds SentenceDetectorSaTModel, a new transformer-based sentence segmentation annotator built around SaT / Segment any Text models.

SentenceDetectorSaTModel predicts sentence boundaries from per-token probabilities using an ONNX-backed XLM-R SentencePiece model. It supports documents longer than a single model window by slicing text into overlapping sub-word token windows, merging the boundary probabilities, and projecting them back to character spans.

Key capabilities:

• Input annotation type: DOCUMENT.
• Output annotation type: DOCUMENT.
• Supports SaT ONNX models such as segment-any-text/sat-12l-sm and segment-any-text/sat-12l.
• Default pretrained model: sat_12l_sm with language xx.
• Supports long text via blockSize and stride.
• Supports batched ONNX inference via satBatchSize.
• Supports overlap weighting strategies: hat and uniform.
• Supports whitespace trimming and sentence-row explosion behavior.
• Supports length-constrained segmentation with minSentenceLength and maxSentenceLength.
• Supports local model loading through loadSavedModel(...) when the model folder contains model.onnx and assets/sentencepiece.bpe.model.

from sparknlp.base import DocumentAssembler
from sparknlp.annotator import SentenceDetectorSaTModel
from pyspark.ml import Pipeline

assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

sentence_detector = SentenceDetectorSaTModel.pretrained() \
    .setInputCols(["document"]) \
    .setOutputCol("sentence")

pipeline = Pipeline().setStages([assembler, sentence_detector])
data = spark.createDataFrame([["This is a sentence. This is another one."]]).toDF("text")

result = pipeline.fit(data).transform(data)
result.selectExpr("explode(sentence.result) as sentence").show(truncate=False)

A new example notebook demonstrates Hugging Face ONNX usage with SentenceDetectorSaTModel:

• HuggingFace ONNX in Spark NLP SentenceDetectorSaT

Hugging Face to Spark NLP Model Conversion Notebooks

Spark NLP 6.4.2 adds three end-to-end model-conversion notebooks under examples/python/model-conversion/:

• HuggingFace_to_SparkNLP_ONNX.ipynb
• HuggingFace_to_SparkNLP_OpenVINO.ipynb
• HuggingFace_to_SparkNLP_GGUF.ipynb

These notebooks make it easier to validate and document model import paths across the main runtime families used by Spark NLP: ONNX, OpenVINO, and GGUF.

Runtime, Build, and Maintenance Updates

This release also includes several infrastructure and maintenance improvements:

• Resource cleanup: ResourceHelper now closes copied resource streams and associated file systems, avoiding stream/file-system leak warnings that could affect dependent pipelines even when the underlying issue appears as a warning rather than a hard error.
• jsl-llamacpp upgrade: The jsl-llamacpp dependency is bumped from 2.0.0 to 2.0.3.
• CI hardening: GitHub Actions now disables sbt/setup-sbt@v1 disk-cache behavior in Spark 3.3, 3.4, and 3.5 build jobs to avoid unrelated hashFiles(...) cache hashing failures before the Spark NLP build starts.
• Issue template routing: GitHub issue templates now route new bug reports, documentation improvements, and feature requests to the current maintainer by default, improving visibility and triage ownership.

🐛 Bug Fixes

• Fixed resource stream cleanup in ResourceHelper by closing the copied resource and the associated file system.
• Fixed an empty similarity/__init__.py issue in Python API wiring.
• Hardened GitHub Actions Spark build jobs by disabling sbt/setup-sbt disk cache.

✅ Tests and Validation

Validation added or updated in this release includes:

• Scala BM25TestSpec coverage for statistics learning, ranking, query reuse, save/load, exact-score verification, non-default parameters, minDocFreq pruning, analyzer symmetry, and parameter-range rejection.
• Python bm25_test.py coverage for fit/query/reuse, save/load, parameter propagation, exact score checks, setQueryTokens(...), and read-only caseSensitive behavior.
• The BM25 retrieval notebook was validated in Google Colab against a built Spark NLP JAR.
• Scala SentenceDetectorSaTSpec coverage for the new SaT sentence detector.
• Python sentence_detector_sat_test.py coverage for the Python wrapper.
• Generated Scala and Python API docs refreshed for the new annotators.

❤️ Community Support

• Slack real-time discussion with the Spark NLP community and team
• GitHub issue tracking, feature requests, and contributions
• Discussions community ideas and showcases
• Medium latest Spark NLP articles and tutorials
• YouTube educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.4.2

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.2

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.2

Maven

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.4.2</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.4.2.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.4.2.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.4.2.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.4.2.jar

What's Changed

• Properly close stream and avoid warning #14774 by @albertoandreottiATgmail
• [SPARKNLP-1384] Add Hugging Face to Spark NLP model conversion notebooks for ONNX, OpenVINO, and GGUF #14775 by @AbdullahMubeenAnwar
• [SPARKNLP-1145] Add BM25 lexical retrieval annotator, including BM25Approach, BM25Model, Scala/Python APIs, docs, tests, and example notebook #14776 by @AbdullahMubeenAnwar
• [SPARKNLP-1395] Update GitHub issue routing, disable fragile sbt disk-cache behavior, and bump jsl-llamacpp to 2.0.3 #14777 by @danilojsl
• Introduce SentenceDetectorSaTModel sentence segmentation annotator with Scala/Python APIs, tests, and ONNX example notebook #14782 by @ahmedlone127
• Spark NLP 6.4.2 release aggregation, version bump, generated Scala API docs, and generated Python API docs #14778 by @danilojsl

Full Changelog: 6.4.1...6.4.2

📢 Spark NLP 6.4.1: Late Chunking, Multimodal Embeddings, and Smarter Pretrained Workflows

Spark NLP 6.4.1 is a feature-rich follow-up release that expands retrieval and multimodal capabilities while improving pretrained model loading and production robustness. This release introduces two powerful new annotators: LateChunkEmbeddings for context-aware chunk embeddings and BiEncoderMultimodalEmbeddings for dual-encoder text-image retrieval workflows.

In addition, VectorDBConnector now supports multimodal image indexing, pretrained model resolution becomes smarter with preferred engine selection, and document ingestion workflows gain important reliability fixes for HTML readers and cloud-backed pretrained caches.

For a deeper walkthrough of the major additions in this release, see our Medium post: Spark NLP 6.4.1: Context-Aware Retrieval, Engine Control, and Multimodal Indexing.

🔥 Highlights

• LateChunkEmbeddings: A new annotator that applies the late chunking technique so chunk embeddings are computed from token embeddings generated over the full document context rather than embedding each chunk independently.
• BiEncoderMultimodalEmbeddings: A new multimodal dual-encoder annotator for generating aligned text and image embeddings for retrieval, search, and indexing workflows.
• Multimodal VectorDBConnector: VectorDBConnector now supports indexing image embeddings in addition to text embeddings, enabling multimodal vector database pipelines.
• Smarter .pretrained() resolution: Pretrained model loading now supports a preferred engine parameter and resolves duplicate model names more reliably by filtering first by annotator class.
• Improved production robustness: Fixes for cloud-backed pretrained caches and HTML reader header handling make Spark NLP more reliable in serverless and cloud-storage-based workflows.

🚀 New Features & Enhancements

LateChunkEmbeddings

Spark NLP 6.4.1 introduces LateChunkEmbeddings, a new annotator based on the Late Chunking approach. Unlike traditional chunk embedding approaches that embed each chunk in isolation, this annotator assumes the upstream embedding model has already encoded the full document in a single pass. It then pools token embeddings corresponding to each chunk span, allowing every chunk embedding to retain broader document context.

This is especially valuable for:

• Retrieval-augmented generation (RAG)
• Semantic search over long documents
• Chunk retrieval where local spans depend on earlier context

A follow-up improvement in this same release adds sentenceAwareFiltering, which is enabled by default. This restricts pooling to token embeddings within the chunk’s sentence boundary, reducing noise from overlapping tokens across adjacent sentences.

late_chunk = LateChunkEmbeddings() \
    .setInputCols(["document", "chunk", "token", "embeddings"]) \
    .setOutputCol("chunk_embeddings")

See the LateChunkEmbeddings example notebook for a full walkthrough and practical usage examples. You can also read our Medium article for a broader explanation of why late chunking improves retrieval quality over naive chunk-first embedding pipelines.

BiEncoderMultimodalEmbeddings

This release adds BiEncoderMultimodalEmbeddings, a new annotator for multimodal dual-encoder workflows. It accepts aligned DOCUMENT and IMAGE annotations and emits two embedding outputs:

• <outputCol>_doc_embeddings
• <outputCol>_image_embeddings

This enables use cases such as:

• text-to-image retrieval
• image-to-text retrieval
• multimodal document search
• image/document semantic matching
• multimodal indexing pipelines

The first supported implementation targets dual-encoder ONNX exports in the Ops-MM / Qwen2VL-style architecture.

mm = BiEncoderMultimodalEmbeddings.pretrained() \
    .setInputCols(["vision_pair_doc", "vision_pair_image"]) \
    .setOutputCol("mm")

To see this in action, check out the BiEncoderMultimodalEmbeddings + Pinecone RAG notebook, which demonstrates multimodal retrieval and indexing in a complete end-to-end workflow. Our Medium post also covers how these multimodal embeddings fit into retrieval and vector search pipelines.

Multimodal Support in VectorDBConnector

VectorDBConnector now supports multimodal image indexing through a new modalityMode parameter.

Supported modes:

• text (default): expects DOCUMENT + SENTENCE_EMBEDDINGS
• image: expects IMAGE + SENTENCE_EMBEDDINGS

In image mode, Spark NLP augments metadata with image-specific fields such as origin, width, height, and channel count, and generates deterministic vector IDs derived from the image origin path for stable re-indexing.

This makes it possible to build end-to-end pipelines that extract images, compute embeddings, and store them directly in vector databases such as Pinecone.

vectorDB = VectorDBConnector() \
    .setInputCols(["image_assembler", "image_embeddings"]) \
    .setOutputCol("vectordb_result") \
    .setProvider("pinecone") \
    .setIndexName("my-multimodal-index") \
    .setModalityMode("image")

For a complete multimodal retrieval example, see the BiEncoderMultimodalEmbeddings + Pinecone RAG notebook, which demonstrates how multimodal embeddings and vector indexing work together in an end-to-end RAG pipeline.

Smarter Pretrained Model Resolution

Spark NLP 6.4.1 improves .pretrained() behavior in two important ways:

• Pretrained models are now filtered by annotator class first, avoiding ambiguity when different annotators share the same model name.
• .pretrained() now accepts an engine parameter, allowing users to prefer a specific engine when multiple versions of a model are available.

If no engine is specified, Spark NLP uses this priority order:

• ONNX
• TensorFlow
• OpenVINO

This makes model loading more predictable and helps users better control runtime behavior across supported backends.

Cloud Pretrained Cache Reliability

This release fixes several issues affecting pretrained resource caching when cache_pretrained points to cloud-backed storage such as S3, GCS, or Azure Blob Storage.

Improvements include:

• More reliable extraction of large pretrained resources into cloud caches
• A completion marker to prevent incomplete cache directories from being reused
• Correct handling of Azure wasbs:// cache paths through Hadoop FS
• Explicit propagation of spark.hadoop.* params into Hadoop configuration

Additionally, sparknlp.start() now includes a skip_sparknlp_maven option for developer workflows that need to run Spark NLP using a custom local JAR.

HTMLReader Improvements

HTMLReader receives important reliability and metadata improvements in this release:

• Fixes failures caused by header serialization mismatches between Java and Scala map types
• Adds paragraph-level metadata fields:
  • paragraph_index
  • paragraph_y
  • page_y

These additions improve downstream layout-aware document understanding and help preserve ordering and spatial context in HTML ingestion workflows.

LLMEntityExtractor Improvements

This release also improves LLMEntityExtractor usability and robustness:

• Updated example notebook to reflect parameter name changes
• Improved getFewShotExamples handling for multiple input cases, particularly in notebook and Colab-style environments

Model Card Tag Cleanup

A metadata cleanup pass also ensures Markdown model cards use engine tags more consistently by aligning engine tags with the declared engine field where available.

🐛 Bug Fixes

• Fixed pretrained engine resolution edge cases in ResourceDownloader
• Fixed cloud-cache extraction and incomplete cache reuse for pretrained resources
• Fixed Azure wasbs:// cache handling
• Fixed HTML reader header serialization issues
• Improved paragraph metadata emission in HTMLReader
• Improved LLMEntityExtractor few-shot example handling
• Updated example notebooks and documentation consistency

❤️ Community Support

• Slack real-time discussion with the Spark NLP community and team
• GitHub issue tracking, feature requests, and contributions
• Discussions community ideas and showcases
• Medium latest Spark NLP articles and tutorials
• YouTube educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.4.1

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.1

Maven

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.4.1</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.4.1</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.4.1</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.4.1</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.4.1.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.4.1.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.4.1.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.4.1.jar

What's Changed

• Adding preferred engine to pretrained and minor fixes #14689 by @ahmedlone127
• Fix tags #14695 by @AbdullahMubeenAnwar
• Update llama_cpp_in_Spark_NLP_LLMEntityExtractor.ipynb #14756 by @AbdullahMubeenAnwar
• Add multimodal image support to VectorDBConnector #14760 by @AbdullahMubeenAnwar
• Update GetFewShotExamples handling in LLMEntityExtractor #14762 by @ahmedlone127
• [SPARKNLP-1369] Fix cloud pretrained cache handling for large models #14763 by @danilojsl
• Add LateChunkEmbeddings annotator #14764 by @AbdullahMubeenAnwar
• [SPARKNLP-1359] Implement BiEncoderMultimodalEmbeddings #14767 by @danilojsl
• Fix pretrained engine parameter behavior #14769 by @ahmedlone127
• [SPARKNLP-1378] HTMLReader Default Headers Error #14770 by @danilojsl
• [SPARKNLP-1385] Add sentenceAwareFiltering param #14772 by @danilojsl

Full Changelog: 6.4.0...6.4.1

📢 Spark NLP 6.4.0: LLM-Powered Entity Extraction, Expanded Document Readers, and Pipeline Robustness

Spark NLP 6.4.0 is a feature-rich release focused on bringing LLM-powered extraction into standard NLP pipelines, dramatically expanding the document reader ecosystem, and hardening production ingestion workflows. This release introduces LLMEntityExtractor, a new annotator that leverages on-device GGUF LLMs to extract named entities from text using flexible few-shot prompting, no fine-tuning required. We also introduce DocumentTitleSplitter, a semantic chunker for reader-based pipelines, and extend the document reader stack with native support for four new file formats: ODT, EPUB, TSV, and RTF. Finally, reader pipelines gain significant production robustness improvements including graceful error handling for HTML and image fetch failures.

🔥 Highlights

• LLMEntityExtractor: Use any GGUF-based LLM to extract named entities from text with configurable few-shot examples, no model fine-tuning needed. Returns results as standard CHUNK annotations compatible with the rest of the Spark NLP ecosystem.
• DocumentTitleSplitter: A new semantic annotator that splits element-level reader output into coherent chunks using title boundaries, designed to work alongside Reader2Doc in element mode across Word, PDF, HTML, PowerPoint, ODT, and Markdown files.
• Expanded Reader Format Support: ODT, EPUB, TSV, and RTF files are now first-class citizens in the Spark NLP reader stack, joinng the existing support for Word, PDF, HTML, PowerPoint, Markdown, CSV, and more.
• Resilient Reader Pipelines: HTMLReader and Reader2Image now degrade gracefully on unreachable URLs and broken remote images rather than failing the entire Spark job critical for large-scale batch processing and ETL workflows.
• Databricks Unity Catalog Volume support: Documentation and instructions added for loading pretrained models from Databricks Unity Catalog Volumes.

🚀 New Features & Enhancements

LLMEntityExtractor

Teams that need flexible entity extraction without model fine-tuning can now use LLMEntityExtractor to leverage on-device GGUF models (via AutoGGUFModel) for NER-style extraction. The annotator uses structured prompts (optionally enhanced with few-shot examples) to identify and extract entities from input text, returning them as standard CHUNK annotations.

Key capabilities:

• Works with any GGUF-format LLM supported by AutoGGUFModel
• Supports configurable few-shot examples via FewShotExample objects
• Reasoning budget is set to 0 by default to minimize latency
• Includes partial JSON recovery for robustness against malformed LLM outputs
• Fully integrated into Python and Scala APIs, including .pretrained() and .loadSavedModel()

llm_extractor = LLMEntityExtractor.pretrained() \
    .setInputCols(["document"]) \
    .setOutputCol("entities")

See the LLMEntityExtractor example notebook for full usage including few-shot configuration.

DocumentTitleSplitter

Reader pipelines that use Reader2Doc in element mode produce structured output with elementType metadata (e.g., Title, NarrativeText). DocumentTitleSplitter closes a key gap by turning this element sequence into semantic, title-bounded chunks that downstream annotators like sentence embedders and LLMs can consume coherently.

Key parameters:

• setSplitOnPageBreak(bool): Also enforce chunk boundaries on page breaks
• setJoinDelimiter(str): Delimiter used when joining narrative text within a chunk
• setMaxChunkLength(int): Overflow splitting for oversized title sections

Supported document types include Word, PDF, HTML, PowerPoint, ODT, and Markdown any format for which Reader2Doc extracts title metadata.

splitter = DocumentTitleSplitter() \
    .setInputCols(["document"]) \
    .setOutputCol("title_chunks")

See the DocumentTitleSplitter example notebook for full usage.

Expanded Document Reader Support

ODT (OpenDocument Text)

A new dedicated ODTReader brings full parity with WordReader for LibreOffice and OpenOffice documents. It supports:

• Narrative text and heading-based hierarchy
• Tables
• Embedded images
• Page-break metadata

EPUB

A new EpubReader enables direct ingestion of e-book content with full structural preservation:

• Opens EPUB archives and resolves the package manifest and reading order
• Parses each chapter's XHTML content using the existing HTML reader behavior
• Resolves embedded images from within the archive no external file references required

TSV (Tab-Separated Values)

Tab-separated files are now supported natively alongside the existing CSV reader, using the same ingestion APIs without any preprocessing or conversion step.

RTF (Rich Text Format)

A new RTFReader adds native ingestion of legacy .rtf files common in enterprise document repositories and archived templates directly into Reader2Doc, and ReaderAssembler pipelines.

Resilient HTML and Image Fetch Handling

Large-scale document ingestion pipelines no longer need to worry about a single unreachable URL or broken remote image crashing the entire Spark job. Two targeted improvements address this:

• HTMLReader: Unreachable URLs and timeouts are now handled gracefully by returning a structured fallback artifact instead of failing the task. Configurable via ignoreUrlErrors (default: true).
• Reader2Image: Remote image fetch failures now produce image-level error artifacts when setIgnoreExceptions(true) (the default), instead of aborting the task.

Fail-fast behavior remains available for strict workloads:

# Image strict mode
Reader2Image().setIgnoreExceptions(False)

Python API improvement: Reader2Doc and Reader2Table are now re-exported directly from sparknlp.reader, simplifying imports:

# Before
from sparknlp.reader.reader2doc import Reader2Doc

# Now
from sparknlp.reader import Reader2Doc, Reader2Table

Databricks Unity Catalog Volume Support

Documentation has been added with instructions for loading and caching pretrained Spark NLP models using Databricks Unity Catalog Volumes, enabling teams running on Databricks to take advantage of centralized model storage within the Unity Catalog ecosystem.

❤️ Community Support

• Slack real-time discussion with the Spark NLP community and team
• GitHub issue tracking, feature requests, and contributions
• Discussions community ideas and showcases
• Medium latest Spark NLP articles and tutorials
• YouTube educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.4.0

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.4.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.4.0

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.4.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.4.0

Maven

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.4.0</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.4.0</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.4.0</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.4.0</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.4.0.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.4.0.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.4.0.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.4.0.jar

What's Changed

• [SPARKNLP-1279] Introduce LLMEntityExtractor annotator #14735 by @ahmedlone127
• [SPARKNLP-1340] Adding instructions for Databricks Unity Catalog Volume #14746 by @danilojsl
• [SPARKNLP-1342] Adding support for ODT files to readers #14747 by @danilojsl
• [SPARKNLP-1342] Adding support for TSV files to readers #14748 by @danilojsl
• [SPARKNLP-1346] Adding support for Epub files to readers #14749 by @danilojsl
• [SPARKNLP-1347] Adding Support for Rich Text Format #14750 by @danilojsl
• [SPARKNLP-1344] Adding DocumentTitleSplitter annotator #14751 by @danilojsl
• [SPARKNLP-1328] Handle HTML/image fetch failures #14752 by @danilojsl

Full Changelog: 6.3.3...6.4.0

📢 Spark NLP 6.3.3: ModernBERT Embeddings, Vector DB Integration, and Layout-Aware Document Processing

Spark NLP 6.3.3 is a feature-packed release aimed at practitioners building modern NLP and multimodal document pipelines. This release introduces ModernBertEmbeddings for dramatically faster and more memory-efficient text embeddings, VectorDBConnector to close the gap between embedding pipelines and vector search infrastructure.

Moreover, we introduce a new suite of document-understanding annotators: LayoutAlignerForVision and LayoutAlignerForText which together enable coherent end-to-end multimodal pipelines over complex documents like PDFs and PowerPoint files. To gain an in-depth walkthrough on how to build use your own pipelines and documents, please see our Medium blog post at Efficient Document Ingestion with Layout Aware Annotators: A Case Study on Mixed-Type Documents.

In addition, a new MultiColumnAssembler can merge multiple annotation column into one and LightPipeline also gains metadata column support for more powerful batch inference workflows.

🔥 Highlights

• ModernBertEmbeddings: A state-of-the-art encoder that is 8x faster and uses 5x less memory than traditional BERT, with native support for sequences up to 8192 tokens which is ideal for long-document NLP tasks.
• VectorDBConnector: Bridges Spark NLP embedding pipelines with external vector databases (initially Pinecone), so teams building semantic search and RAG systems no longer need custom glue code to store embeddings.
• LayoutAlignerForVision and LayoutAlignerForText: New annotators for multimodal document pipelines that spatially align text and images from complex documents, giving downstream Vision-Language Models (VLMs) the coherent context they need to produce better results.
• MultiColumnAssembler: Closes a common pipeline gap when ReaderAssembler splits document content across multiple columns (text, table, image captions). This annotator merges them back into a single column that downstream annotators like AutoGGUFVisionModel expect.
• Enhanced LightPipeline with metadata column support for richer, context-aware inference workflows.

🚀 New Features & Enhancements

ModernBertEmbeddings

Teams working with long documents, code, or large-scale embedding workloads will benefit from ModernBertEmbeddings, which brings the latest generation of bidirectional encoder models to Spark NLP. Based on the paper Smarter, Better, Faster, Longer, ModernBERT was trained on 2 trillion tokens with a native sequence length of up to 8,192 tokens (eight times the limit of classic BERT) enabling faster, cheaper embeddings for longer sequences without truncation.

• Default pretrained model: "modernbert-base" (English)
• 768-dimensional token-level WORD_EMBEDDINGS output

embeddings = ModernBertEmbeddings.pretrained() \
    .setInputCols(["document", "token"]) \
    .setOutputCol("modernbert_embeddings")

See the ModernBertEmbeddings notebook for extended examples, including how to import custom HuggingFace ModernBERT models via ONNX.

VectorDBConnector

For teams building semantic search, retrieval-augmented generation (RAG), or similarity-based recommendation systems, manually bridging Spark NLP with a vector database has historically required custom integration code. VectorDBConnector eliminates this gap by letting you store embeddings from any Spark NLP embedding annotator directly into a vector database as part of the pipeline. It initially supports Pinecone, with more providers planned.

vectorDB = VectorDBConnector() \
    .setInputCols(["document", "sentence_embeddings"]) \
    .setOutputCol("vectordb_result") \
    .setProvider("pinecone") \
    .setIndexName("my-semantic-index") \
    .setNamespace("production") \
    .setIdColumn("doc_id") \
    .setMetadataColumns(["text", "category"]) \
    .setBatchSize(100)

The Pinecone API key is configured via spark.jsl.settings.vectordb.api_key. See the VectorDBConnector Pinecone Demo notebook for a full walkthrough.

LayoutAlignerForVision and LayoutAlignerForText

When processing rich documents like PDFs or PowerPoint presentations, text and images are spatially interleaved. For example, A chart sits next to the paragraph it illustrates, a diagram is surrounded by its explanation. Without layout awareness, VLMs operating on extracted content lose this spatial context entirely. LayoutAlignerForVision and LayoutAlignerForText solve this problem for teams building multimodal document intelligence pipelines.

LayoutAlignerForVision takes document chunks and images extracted by ReaderAssembler and aligns each image with its spatially nearby text paragraphs based on actual page coordinates. It produces three output columns <outputCol>_doc, <outputCol>_image, and <outputCol>_prompt, ready to be fed directly into a VLM (e.g. AutoGGUFVisionModel) for captioning or question answering.

Key parameters:

• setMaxDistance(int): Maximum vertical distance (px) for image-paragraph alignment
• setIncludeContextWindow(bool): Include neighboring paragraphs as context for floating images
• setAddNeighborText(bool): Include aligned text in the prompt output
• setImageCaptionBasePrompt(str): Customize the captioning prompt sent to downstream VLMs
• setNeighborTextCharsWindow(int): Include surrounding text characters as prompt context
• setExplodeDocs(bool): Emit one output row per aligned doc/image pair

LayoutAlignerForText takes the VLM-generated image captions produced after LayoutAlignerForVision and weaves them back into the document's text flow, replacing raw image placeholders with meaningful captions and re-computing begin/end offsets so the resulting document is coherent for downstream NLP tasks.

Key parameters:

• setJoinDelimiter(str) – Delimiter used to join rebuilt text segments
• setExplodeElements(bool) – Emit one output row per aligned text element

For more extended examples and walkthroughs, see refer to the notebook Spark NLP LayoutAligners for Document Understanding and our Medium blog post Efficient Document Ingestion with Layout Aware Annotators: A Case Study on Mixed-Type Documents.

MultiColumnAssembler

When using ReaderAssembler to process documents such as PDFs or PPTX files, content is extracted into separate typed columns: document_text, document_table, and image-related outputs. However, many downstream annotators expect a single input column. Previously, bridging this split required custom Spark transformations. MultiColumnAssembler fills this gap directly within Spark NLP pipelines.

It merges any number of DOCUMENT-type annotation columns into a single output column, preserving all annotation metadata and adding a source_column key to track provenance. Annotations can optionally be sorted by their begin offset using setSortByBegin(True).

multiColumnAssembler = MultiColumnAssembler() \
    .setInputCols(["document_text", "document_table"]) \
    .setOutputCol("merged_document")

Key parameters:

• setInputCols([...]) – List of DOCUMENT-type annotation columns to merge
• setOutputAsAnnotatorType(str) – Override the output annotator type (default: "document")
• setSortByBegin(bool) – Sort merged annotations by begin position (default: False)

Note: Columns using the AnnotationImage schema (i.e., IMAGE-typed columns from ReaderAssembler) are not supported.

See the Merging Annotation Columns notebook for a full walkthrough.

LightPipeline Metadata Support

Users running inference with LightPipeline on data that carries additional context — such as document source, language, or category — previously had no way to pass that context through alongside the text. LightPipeline now supports passing metadata columns alongside text inputs in both annotate() and fullAnnotate(), enabling richer, context-aware inference for applications like routing, filtering, and conditional processing.

New supported call signatures:

• fullAnnotate(text: str, metadata: dict[str, list[str]])
• fullAnnotate(texts: list[str], metadata: list[dict])
• fullAnnotate(texts: list[str], metadata: dict[str, list[str]]) (columnar format)
• Same patterns apply to annotate()

Metadata can be passed as a keyword argument or as a positional trailing argument:

result = light_pipeline.fullAnnotate(
    "U.N. official Ekeus heads for Baghdad.",
    metadata={"source": ["news_article"]}
)

This feature is also surfaced through PretrainedPipeline.annotate() and PretrainedPipeline.fullAnnotate().

🐛 Bug Fixes

• Apache POI upgraded to 5.4.1: The Apache POI dependency used by document readers has been upgraded from 4.1.2 to 5.4.1 (poi-ooxml-full) to avoid deprecated dependencies.

❤️ Community Support

• Slack real-time discussion with the Spark NLP community and team
• GitHub issue tracking, feature requests, and contributions
• Discussions community ideas and showcases
• Medium latest Spark NLP articles and tutorials
• YouTube educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.3.3

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.3

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.3

Maven

Supported on Apache Spark 3.x.

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.3.3</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.3.3</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.3.3</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.3.3</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.3.3.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.3.3.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.3.3.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.3.3.jar

What's Changed

• [SPARKNLP-1334] Updating POI dependency for readers #14727 by @danilojsl
• [SPARKNLP-1335] Implement Layout Aligner annotators #14737 by @danilojsl
• [SPARKNLP-1336] Enhancements to LightPipeline #14734 by @danilojsl
• [SPARKNLP-1287] vector db connector annotator #14729 by @ahmedlone127
• [SPARKNLP-1231] implement modern bert embeddings #14736 by @ahmedlone127
• [SPARKNLP-1341] Add MultiColumnAssembler #14743 by @AbdullahMubeenAnwar

Full Changelog: 6.3.2...6.3.3

📢 Spark NLP 6.3.2: Scala 2.13 Support, Layout-Aware Images, and Enhanced LightPipeline Tracking

Spark NLP 6.3.2 is a foundational release that introduces official support for Scala 2.13, alongside important improvements in document layout understanding and lightweight inference workflows.
This release improves long-term model portability through JSON-based serialization, enriches document image extraction with spatial metadata, and enhances LightPipeline with document ID tracking and output filtering.

🔥 Highlights

• Official Scala 2.13 support
• Layout-aware image extraction with spatial coordinates added to Reader2Image for HTML, DOCX, and PPTX documents.
• Enhanced LightPipeline with document ID propagation and output column filtering for better batch inference workflows.

🚀 New Features & Enhancements

Scala 2.13 Support

Spark NLP now supports Scala 2.13 with this release! This will enable you to run your Spark NLP pipelines on Spark versions that run on Scala 2.13, such as used by Databricks and Dataproc. See our Installation Instructions for Scala 2.13 on how to use it with our project.

There are some things you have to consider when using the Scala 2.13 version

1. You need to adjust your dependency from spark-nlp_2.12 to spark-nlp_2.13.
2. If you install PySpark from PyPi, then the session will be Scala 2.12 by default. If you need to start a Scala 2.13 instance, you can set the SPARK_HOME environment variable to a Spark Scala 2.13 installation, or install PySpark from the official Spark archives.
3. If you want to load DependencyParserModel or TextMatcherModel from Scala 2.12 into Scala 2.13, you will need to manually export them again with the latest version. See the notebook

Layout-Aware Image Metadata in Reader2Image

The Reader2Image annotator now extracts spatial image coordinates from rich document formats, adding layout awareness to image annotations.

• Supported formats:
  • HTML
  • Word (DOCX)
  • PowerPoint (PPTX)
• New metadata fields:
  • x, y, width, height
• Coordinates are included alongside existing metadata such as image format, type, and DOM position

This enables:

• Layout-aware document and multimodal pipelines
• Visual reconstruction of documents
• More accurate association of images with surrounding text content

Document ID Support in LightPipeline

LightPipeline now supports passing document IDs together with text inputs, improving traceability in batch and production inference scenarios.

Key capabilities:

• New overloads:
  • fullAnnotate(ids, texts)
  • annotate(ids, texts)
• Document IDs are propagated as annotation metadata (doc_id)
• New output_cols parameter to restrict returned annotation types

Benefits:

• Reliable document-to-result mapping
• Easier debugging and downstream integration
• Reduced memory usage through selective outputs

Existing LightPipeline usage remains unchanged and backward compatible.

🐛 Bug Fixes

• Fix out of memory error when copying big models to a cloud storage

❤️ Community Support

• Slack – real-time discussion with the Spark NLP community and team
• GitHub – issue tracking, feature requests, and contributions
• Discussions – community ideas and showcases
• Medium – latest Spark NLP articles and tutorials
• YouTube – educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.3.2

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.2

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.2

Maven

Supported on on Apache Spark 3.x.

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.3.2.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.3.2.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.3.2.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.3.2.jar

What's Changed

• [SPARKNLP-1136] JSON Serialization for Features #14722 by @DevinTDHa
• [SPARKNLP-1329] Adding image coordinates to metadata for Reader2Image #14725 by @danilojsl
• [SPARKNLP-1333] Adding ids input for LightPipeline #14726 by @danilojsl

Full Changelog: 6.3.1...6.3.2

📢 Spark NLP 6.3.1: LLM Backend Upgrade and Document Processing Improvements

Spark NLP 6.3.1 focuses on strengthening distributed local LLM inference by upgrading the jsl-llamacpp backend to a newer llama.cpp release, while also delivering important improvements in document structure handling and metadata consistency.

This enables you to use the latest LLMs and embeddings compatible with llama.cpp and perform advanced ingestion of tables and images.

🔥 Highlights

• Upgraded jsl-llamacpp backend to llama.cpp tag b7247, bringing upstream performance improvements, stability fixes, and expanded model compatibility for local LLM inference.
• Improved Reader2X annotator capabilities with structural position metadata for tables and images and integration with AutoGGUFVisionModel
  • For an exhaustive overview on how to use Spark NLP for unstructured document ingestion, see our blog post Evaluating Document AI Frameworks: Spark NLP vs Unstructured for Large-Scale Text Processing

🚀 New Features & Enhancements

LLM Backend Upgrade (llama.cpp)

The jsl-llamacpp backend has been upgraded to llama.cpp tag b7247, applying upstream fixes and enabling the use of the latest LLMs. These benefit distributed LLM workloads in Spark NLP and affects the annotators AutoGGUFModel, AutoGGUFEmbeddings, AutoGGUFVisionModel, AutoGGUFReranker:

• Performance and memory improvements, bug fixes from upstream llama.cpp for offline LLM inference within Spark NLP pipelines
• Better support for newer GGUF/GGML model variants. This means you can now load models such as gpt-oss, Qwen3 and embeddinggemma.

Structural Metadata for Document Readers

Previously, our document parsers (HTMLReader, XMLReader, WordReader, PowerPointReader, ExcelReader) relied heavily on positional or page-based coordinates for layout metadata.
However, non-PDF formats such as HTML, XML, DOC(X), PPT(X), and XLS(X) do not have fixed pages
To ensure deterministic element referencing and structural traceability across all document types, we needed to adopt a unified DOM-like metadata model.

This change standardizes metadata extraction so every element can be uniquely identified and re-located within its source document, independent of visual layout.

These additions enable layout-aware downstream processing and more precise filtering especially for HTML and rich document formats.

Reader2Image Integration with AutoGGUFVisionModel

Previously, you could use Reader2Image to ingest images from various file formats into Spark NLP. However, processing was limited to Spark NLP native VLM implementations (such as Qwen2VLTransformer).

Reader2Image now supports interoperability with our llama.cpp backend with AutoGGUFVisionModel by introducing flexible handling of encoded vs. decoded image bytes and optional prompt output.

• Added a new boolean parameter useEncodedImageBytes to control whether the image result stores:
  • true: Encoded (compressed) file bytes for models like AutoGGUFVisionModel
  • false: Decoded pixel matrix for models such as Qwen2VLTransformer
• outputPromptColumn parameter to optionally output a separate prompt column containing text prompts as Spark NLP Annotations. This is the required format for AutoGGUFVisionModel.

Platform Setup Documentation

Added official documentation and instructions for setting up and running Spark NLP on Microsoft Fabric, simplifying configuration and improving developer onboarding on the platform. You can see them at Spark NLP - Installation

🐛 Bug Fixes

• Sentence metadata is now consistently included in DocumentAssembler outputs when using LightPipeline.
• Fixed an issue where resetting the cache in ResourceDownloader could fail under certain conditions.
• Fixed a document parsing bug where some HTML elements (such as section titles or diagnosis entries) could appear multiple times in the parsed output.
• Improved robustness when loading ONNX BertEmbeddings models with non-standard output tensor names.

❤️ Community Support

• Slack – real-time discussion with the Spark NLP community and team
• GitHub – issue tracking, feature requests, and contributions
• Discussions – community ideas and showcases
• Medium – latest Spark NLP articles and tutorials
• YouTube – educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.3.1

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.1

Maven

Supported on on Apache Spark 3.x.

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.3.1.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.3.1.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.3.1.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.3.1.jar

What's Changed

• #14705 by @danilojsl
• #14710 by @danilojsl
• #14711 by @AbdullahMubeenAnwar
• #14712 by @mehmetbutgul
• #14713 by @danilojsl
• #14714 by @mehmetbutgul
• #14719 by @DevinTDHa

Full Changelog: 6.3.0...6.3.1

New Features & Enhancements

• [SPARKNLP-1318] Reader2Image Integration with AutoGGUFVisionModel #14705

Bug Fixes

None

=======

📢 Spark NLP 6.2.3: Further Improvements for NerDL

Spark NLP 6.2.3 introduces targeted improvements to training performance and stability of NerDLApproach and bug fixes for CamemBertForTokenClassification.

NerDLApproach now uses new internal data-loading behavior, and improving training speed and preventing out-of-memory errors.

🔥 Highlights

Enhanced NerDLApproach training performance through threaded data loading and optimized partitioning.

🚀 New Features & Enhancements

NerDLApproach Training Optimizations

Significant performance improvements for training of NerDLApproach:

Threaded Data Loading: When enabling the memory optimizer (setEnableMemoryOptimizer(true)), data can now be pre-fetched through a threaded data loader. By default, it is disabled but can be tuned by using:

.setPrefetchBatches(int)

By tuning this parameter (for example 20 batches), you can get training time reductions of about 10%.

Optimized Partitioning Strategy: NerDLApproach now applies optimized dataframe partitioning when using the memory optimizer (setEnableMemoryOptimizer(true)) by default, improving parallelization efficiency during training and preventing out-of-memory errors.

For manual tuning of the input data frames, this behavior can be disabled with:

.setOptimizePartitioning(false)

🐛 Bug Fixes

• CamemBertForTokenClassification: Fixed an issue with expected input types during inference.

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.3

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.3

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.3

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.3</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.3.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.3.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.3.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.3.jar

What's Changed

• #14701 by @ahmedlone127
• #14699 by @DevinTDHa

Full Changelog: 6.2.2...6.2.3

📢 Spark NLP 6.2.2: Bugfix Release

Spark NLP 6.2.2 brings bug fixes to WordEmbeddings and NerDLApproach logging.

🐛 Bug Fixes

• WordEmbeddings: Fixed a bug where WordEmbeddings would duplicate input tokens in the output
• NerDLApproach: Fixed a bug during logging, that would show inaccurate training/validation counts.

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.2

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.2

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.2

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.2</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.2.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.2.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.2.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.2.jar

What's Changed

• #14693 by @ahmedlone127

Full Changelog: 6.2.0...6.2.2

📢 Spark NLP 6.2.1: Enhanced hierarchical document processing and training optimizations

Spark NLP 6.2.1 brings significant improvements to document ingestion with expanded hierarchical support, XML processing enhancements, and optimizations for NerDL training. This release builds on the foundation of 6.2.0, continuing to focus on structure-awareness, flexibility, and performance for production NLP pipelines.

🔥 Highlights

• Hierarchical Document Processing: Extended support for PDF, Word, and Markdown with parent-child element relationships
• NerDLApproach Training Optimizations: Reduced memory footprint and improved training performance with BERT based embeddings
• Improved Document Output Format: Single document annotations by default for more intuitive behavior with large documents
• Enhanced XML Reading: Attribute extraction and improved tag handling in Reader2Doc

🚀 New Features & Enhancements

Hierarchical Support for Multiple Document Formats

Building on the HTMLReader hierarchical features introduced in 6.2.0, this release extends structured element tracking to additional document formats:

• Reader2Doc now supports hierarchical processing for PDF, Microsoft Word, and Markdown files

• Each extracted element includes:

  • element_id: Unique UUID identifier per element
  • parent_id: References the parent element's ID for logical document structure

• Enables tree-like navigation and contextual understanding of document hierarchy:

  Chapter 1
   ├── Narrative Text A
   ├── Narrative Text B
  Chapter 2
   ├── Paragraph C
  

• Supports advanced use cases including hierarchical retrieval, graph-based indexing, and multi-level document analysis

• Metadata propagation ensures downstream annotators maintain structural relationships

NerDLApproach Training Optimizations

Significant performance improvements for training of NerDLApproach:

• Reduced Memory Usage with BERT based embeddings: Optimized output embeddings allocations, lowering peak memory footprint during training
• Automatic Dataset Caching: When using setEnableMemoryOptimizer(true) with maxEpoch > 1, input datasets are automatically cached to improve training speed
• Graph Metadata Reuse: NerDLGraphChecker now populates TensorFlow graph metadata that NerDLApproach can reuse, reducing redundant computations during training initialization

With all these improvements you can expect up half the memory consumption and training time on RAM constrained environments (when using setEnableMemoryOptimizer(true)). For larger distributed datasets, the effect will be more pronounced.

XML Reader and Reader2Doc Enhancements

• Single Document Output by Default: Reader2Doc now creates single document annotations per file by default, providing more expected behavior when processing large documents

  • Lines are joined by newline character \n by default, configurable via new setJoinString(string) parameter for custom separators
  • Automatically includes specified attribute values in document output

• Improved Tag Handling: XML reader now ignores empty tags without text content, reducing noise in parsed output

• Enhanced content type handling for application/xml documents

• XML Tag Attribute Extraction: New setExtractTagAttributes(attributes: list[str]) parameter enables extraction of XML attribute values. Example:

  <bookstore>
      <book category="children">
          <title lang="en">Harry Potter</title>
          <author>J K. Rowling</author>
          <year>2005</year>
          <price>29.99</price>
      </book>
      <book category="web">
          <title lang="en">Learning XML</title>
          <author>Erik T. Ray</author>
          <year>2003</year>
          <price>39.95</price>
      </book>
  </bookstore>

  We can extract category and lang values with the Reader2Doc Config

  reader2doc = Reader2Doc() \
      .setContentType("application/xml") \
      .setContentPath("../src/test/resources/reader/xml/test.xml") \
      .setOutputCol("document") \
      .setExtractTagAttributes(["category", "lang"])

  Resulting in

  children
  en
  Harry Potter
  J K. Rowling
  2005
  29.99
  web
  en
  Learning XML
  Erik T. Ray
  2003
  39.95
  

🐛 Bug Fixes

• Colab Environment Setup: Added Java installation to Colab setup script for improved out-of-the-box compatibility

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.1

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.1

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.1</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.1.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.1.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.1.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.1.jar

What's Changed

• #14681 by @danilojsl
• #14682 by @AbdullahMubeenAnwar
• #14685 by @DevinTDHa
• #14691 by @DevinTDHa

Full Changelog: 6.2.0...6.2.1

📢 Spark NLP 6.2.0: A new stage for unstructured document ingestion and processing at scale

Spark NLP 6.2.0 introduces key upgrades across entity extraction, document normalization, HTML reading, and GGUF-based models. To recap, since the releases of Spark NLP 6.1 you can:

• Infer quantized cutting-edge LLMs and VLMs such as Gemma 3, Phi-4, Llama 3.1, Qwen 2.5
• Rerank documents using llama.cpp with AutoGGUFReranker
• Ingest unstructured documents of diverse formats
  • Reader2Doc: streamlines the process of loading and integrating diverse file formats (PDFs, Word, Excel, PowerPoint, HTML, Text, Email, Markdown) directly into Spark NLP pipelines with a unified and flexible interface.
  • Reader2Table: streamlines tabular data extraction from multiple document formats with seamless pipeline integration.
  • Reader2Image: extract structured image content from various document types

Spark NLP release 6.2.0 further focuses on automation, structure-awareness, and resource efficiency, making pipelines easier to configure, manage, and extend.

🔥 Highlights

• Auto Modes for EntityRuler and DocumentNormalizer: automatic regex and text-cleaning presets for faster setup.
• Hierarchical Element Tracking in HTMLReader: adds element and parent identifiers for structure-aware document processing.
• Resource Management for AutoGGUF Annotators: improved control and cleanup of llama.cpp-based models.

🚀 New Features & Enhancements

EntityRulerModel and DocumentNormalizer Auto Modes

EntityRulerModel

• Added autoMode parameter to enable predefined regex entity groups ("network_entities", "communication_entities", "media_entities", "email_entities", "all_entities").
• Added extractEntities parameter to filter entities within auto modes.
• Automatically applies case-insensitive regex presets and falls back to manual mode if not specified.
• Retains full backward compatibility with JSON or RocksDB-based rules.

DocumentNormalizer

• Added presetPattern and autoMode parameters to apply built-in text cleaning patterns.
• New modes include "light_clean", "document_clean", "social_clean", "html_clean", and "full_auto".
• Enables quick application of multiple cleaning operations without manual configuration.

Together, these additions significantly reduce boilerplate setup for common text extraction and normalization workflows.

Hierarchical Element Identification in HTMLReader

• Introduced element_id and parent_id metadata fields for each parsed HTML element.
• Enables explicit structural relationships (e.g., title → paragraph → link) for hierarchical retrieval and contextual reasoning.
• Supports graph-based indexing, hybrid search, and multi-level document analysis.
• Metadata propagation improvements ensure Sentence Detector outputs also retain upstream hierarchy information.

AutoGGUF Annotator Enhancements

For AutoGGUFModel, AutoGGUFVision, AutoGGUFEmbeddings, AutoGGUFReranker

• Added close() method to explicitly release llama.cpp model resources, preventing memory retention in long-running sessions.
• Introduced setRemoveThinkingTag(tag: String) parameter to remove internal <think>...</think> sections from model outputs.
  • Regex pattern: (?s)<$tag>.+?</$tag>
  • Simplifies downstream processing for chat and reasoning models.

🐛 Bug Fixes

• RobertaEmbeddings Warmup Test - fixed token sequence bug where unknown tokens caused initialization errors.

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.0

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.0

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.0

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.0</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.0.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.0.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.0.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.0.jar

What's Changed

• https://github.com/JohnSnowLabs/spark-nlp/pull/14671 by @DevinTDHa
• https://github.com/JohnSnowLabs/spark-nlp/pull/14672 by @DevinTDHa
• https://github.com/JohnSnowLabs/spark-nlp/pull/14674 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14675 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14677 by @ahmedlone127
• https://github.com/JohnSnowLabs/spark-nlp/pull/14673 by @AbdullahMubeenAnwar

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.1.5...6.2.0

📢 Spark NLP 6.1.5: Smarter Readers and More Resilient Pipelines

Spark NLP 6.1.5 focuses on improving data ingestion reliability and pipeline flexibility. This release enhances reader components with better fault tolerance, broader input support, and introduces a new ReaderAssembler annotator for streamlined integration. Several key fixes also improve model loading and stability in distributed environments.

🔥 Highlights

• New ReaderAssembler Annotator: Unify multiple reader annotators into one configurable component for simpler and cleaner ingestion pipelines.

🚀 New Features & Enhancements

Reader Pipeline Enhancements

• ReaderAssembler Annotator A new meta-annotator that unifies Reader2X components (e.g., Reader2Doc, Reader2Image, Reader2Table) under a single interface.

  • Automatically selects the right reader(s) based on configuration.
  • Supports declarative assembly of reading stages.
  • Provides parameters for reader selection, fallback rules, and error handling. This simplifies pipeline construction and improves maintainability for multi-format ingestion workflows. (Link to notebook)

• Support for String Input Columns in Readers (SPARKNLP-1291) Spark NLP readers only supported inputs via file paths. That means if you already had a DataFrame with text content (say from another pipeline or a preliminary load), you had to write it to disk just to let the reader ingest it. This adds friction and overhead, especially in streaming or in-memory pipelines.

  With this change, you can:

  • Feed raw text stored in a DataFrame column directly into Spark NLP readers — zero I/O overhead when not needed.
  • Simplify workflows and pipelines (no need for temporary file staging just to “read” back data).
  • Improve performance and resource usage in scenarios where input is already available as strings (e.g. generated, preprocessed, or coming from another system).
  • Make the reader APIs more flexible and general-purpose.

• Fault-Tolerant XML Reader The XML reader now skips malformed XML fragments (e.g., mismatched tags, missing closures, invalid characters) instead of failing the job. Enhanced error handling ensures more resilient ingestion of imperfect real-world data.

🐛 Bug Fixes

• GGUF Model Loading Duplication Fixed an issue in FeaturesFallbackReader that caused duplicate loading or missing model files when calling .pretrained() on GGUF-based annotators such as AutoGGUFModel and rerankers, especially in Databricks environments.

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

💻 Installation

Python

pip install spark-nlp==6.1.5

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.5

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.5

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.5

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.5

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.5.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.5.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.5.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.5.jar

What's Changed

• https://github.com/JohnSnowLabs/spark-nlp/pull/14665 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14666 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14668 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14667 by @C-K-Loan, @DevinTDHa

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.1.4...6.1.5

📢 Spark NLP 6.1.4: Advancing Multimodal Workflows with Reader2Image

We are excited to announce the release of Spark NLP 6.1.4!
This version introduces a powerful new annotator, Reader2Image, which extends Spark NLP’s universal ingestion capabilities to embedded images across a wide range of document formats. With this release, Spark NLP users can now seamlessly integrate text and image processing in the same pipeline, unlocking new opportunities for vision-language modeling (VLM), multimodal search, and document understanding.

🔥 Highlights

• New Reader2Image Annotator: Extract and structure image content directly from documents like PDFs, Word, PowerPoint, Excel, HTML, Markdown, and Email files.
• Multimodal Pipeline Expansion: Build workflows that combine text, tables, and now images for comprehensive document AI applications.
• Consistent Structured Output: Access image metadata (filename, dimensions, channels, mode) alongside binary image data in Spark DataFrames, fully compatible with other visual annotators.

🚀 New Features & Enhancements

Document Ingestion

• Reader2Image Annotator
  A new multimodal annotator designed to parse image content embedded in structured documents. Supported formats include:

  • PDFs
  • Word (.doc/.docx)
  • Excel (.xls/.xlsx)
  • PowerPoint (.ppt/.pptx)
  • HTML & Markdown (.md)
  • Email files (.eml, .msg)

  Output Fields:

  • File name
  • Image dimensions (height, width)
  • Number of channels
  • Mode
  • Binary image data
  • Metadata

  This enables seamless integration with vision-language models (VLMs), multimodal embeddings, and downstream Spark NLP annotators, all within the same distributed pipeline.

🐛 Bug Fixes

• None

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

⚙️ Installation

Python

pip install spark-nlp==6.1.4

Spark Packages

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.4

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.4

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.4

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.4

Maven

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.4</version>
</dependency>

• GPU: spark-nlp-gpu_2.12:6.1.4
• Apple Silicon: spark-nlp-silicon_2.12:6.1.4
• AArch64: spark-nlp-aarch64_2.12:6.1.4

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.4.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.4.jar
• M1/M2: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.4.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.4.jar

📊 What’s Changed

• [SPARKNLP-1261] Introducing Reader2Image Annotator (#14658) by @danilojsl

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.1.3...6.1.4

What's Changed

• SPARKNLP-1261 Introducing Reader2Image Annotator by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14658

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.1.3...6.1.4

📢 Spark NLP 6.1.3: NerDL Graph Checker, Reader2Doc Enhancements, Ranking Finisher

We are pleased to announce Spark NLP 6.1.3, introducing a new graph validation annotator for NER training, enhancements to Reader2Doc for flexible document handling, and a new ranking finisher for AutoGGUFReranker outputs. This release focuses on improving training robustness, document processing flexibility, and retrieval ranking capabilities.

🔥 Highlights

• New NerDLGraphChecker annotator to validate NER training graphs before training starts.
• Reader2Doc enhancements with options for consolidated output and filtering.
• New AutoGGUFRerankerFinisher for ranking, filtering, and normalizing reranker outputs.

🚀 New Features & Enhancements

Named Entity Recognition (NER)

NerDLGraphChecker: A new annotator that validates whether a suitable NerDL graph is available for a given training dataset before embeddings or training start. This helps avoid wasted computation in custom training scenarios. (Link to notebook)

• Must be placed before embedding or NerDLApproach annotators.
• Requires token and label columns in the dataset.
• Automatically extracts embedding dimensions from the pipeline to validate graph compatibility.

Document Processing

Reader2Doc Enhancements: New configuration options provide more control over output formatting:

• outputAsDocument: Concatenates all sentences into a single document.
• excludeNonText: Filters out non-textual elements (e.g., tables, images) from the document.

Ranking & Retrieval

AutoGGUFRerankerFinisher: A finisher for processing AutoGGUFReranker outputs, adding advanced ranking and filtering capabilities (Link to notebook):

• Top-k document selection.
• Score threshold filtering.
• Min-max score normalization (0–1 range).
• Sorting by relevance score.
• Rank assignment in metadata while preserving document structure.

🐛 Bug Fixes

None.

❤️ Community Support

• Slack Live discussion with the Spark NLP community and team
• GitHub Bug reports, feature requests, and contributions
• Discussions Share ideas and engage with other community members
• Medium Spark NLP technical articles
• JohnSnowLabs Medium Official blog
• YouTube Spark NLP tutorials and demos

Installation

Python

pip install spark-nlp==6.1.3

Spark Packages

spark-nlp on Apache Spark 3.0.x–3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.3

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.3

Maven

spark-nlp:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.3.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.3.jar
• M1: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.3.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.3.jar

What's Changed

• https://github.com/JohnSnowLabs/spark-nlp/pull/14657
• https://github.com/JohnSnowLabs/spark-nlp/pull/14656
• https://github.com/JohnSnowLabs/spark-nlp/pull/14653

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.1.2...6.1.3

📢 Spark NLP 6.1.2: AutoGGUFReranker and AutoGGUF improvements

We are excited to announce Spark NLP 6.1.2, enhancing AutoGGUF model support and introduces a brand new reranking annotator based on llama.cpp LLMs. This release also brings fixes for AutoGGUFVision model and improvements for CUDA compatibility of AutoGGUF models.

🔥 Highlights

New AutoGGUFReranker annotator for advanced LLM-based reranking in information retrieval and retrieval-augmented generation (RAG) pipelines.

🚀 New Features & Enhancements

Large Language Models (LLMs)

• AutoGGUFReranker A new annotator for reranking candidate results using AutoGGUF-based LLM embeddings. This enables more accurate ranking in retrieval pipelines, benefiting applications such as search, RAG, and question answering. (Link to notebook)

🐛 Bug Fixes

• Fixed Python initialization errors in AutoGGUFVisionModel.
• Using save for AutoGGUF models now supports more file protocols.
• Ensured better GPU support for AutoGGUF annotators on a broader range of CUDA devices.

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

Installation

Python

pip install spark-nlp==6.1.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.2

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.2</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.2</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.2.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.2.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.2.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.2.jar

What's Changed

• https://github.com/JohnSnowLabs/spark-nlp/pull/14649 by @prabod
• https://github.com/JohnSnowLabs/spark-nlp/pull/14650 by @DevinTDHa

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.1.1...6.1.2

📢 Spark NLP 6.1.1: Enhanced LLM Performance and Expanded Data Ingestion Capabilities

We are thrilled to announce Spark NLP 6.1.1, a focused release that delivers significant performance improvements and enhanced functionality for large language models and universal data ingestion. This release continues our commitment to providing state-of-the-art AI capabilities within the native Spark ecosystem, with optimized inference performance and expanded multimodal support.

🔥 Highlights

• Performance Boost for llama.cpp models: Inference optimizations in AutoGGUFModel and AutoGGUFEmbeddings deliver improvements for large language model workflows on GPU.
• Multimodal Vision Models Restored: The AutoGGUFVisionModel annotator is back with full functionality and latest SOTA VLMs, enabling sophisticated vision-language processing capabilities.
• Enhanced Table Processing: New Reader2Table annotator streamlines tabular data extraction from multiple document formats with seamless pipeline integration.
• Upgraded openVINO backend: We upgraded our openVINO backend to 2025.02 and added hyperthreading configuration options to maximize performance on multi-core systems.

🚀 New Features & Enhancements

Large Language Models (LLMs)

• Optimized AutoGGUFModel Performance: We improved the inference of llama.cpp models and achieved a 10% performance increase for AutoGGUFModel on GPU.
• Restored AutoGGUFVisionModel: The multimodal vision model annotator is fully operational again, enabling powerful vision-language processing capabilities. Users can now process images alongside text for comprehensive multimodal AI applications while using the latest SOTA vision-language models.
• Enhanced Model Compatibility: AutoGGUFModel can now seamlessly load the language model components from pretrained AutoGGUFVisionModel instances, providing greater flexibility in model deployment and usage. (Link to notebook)
• Robust Model Loading: Pretrained AutoGGUF-based annotators now load despite the inclusion of deprecated parameters, ensuring broader compatibility.
• Updated Default Models: All AutoGGUF annotators now use more recent and capable pretrained models:

Document Ingestion

• Reader2Table Annotator: This powerful new annotator provides a streamlined interface for extracting and processing tabular data from various document formats (Link to notebook). It offers:
  • Unified API for interacting with Spark NLP readers
  • Enhanced flexibility through reader-specific configurations
  • Improved maintainability and scalability for data loading workflows
  • Support for multiple formats including HTML, Word (.doc/.docx), Excel (.xls/.xlsx), PowerPoint (.ppt/.pptx), Markdown (.md), and CSV (.csv)

Performance Optimizations

• OpenVINO Upgrade: We upgrade the backend to 2025.02 and added comprehensive hyperthreading configuration options for the OpenVINO backend, enabling users to optimize performance on multi-core systems by fine-tuning thread allocation and CPU utilization.

🐛 Bug Fixes

None

❤️ Community Support

• Slack: For live discussion with the Spark NLP community and the team.
• GitHub: Bug reports, feature requests, and contributions.
• Discussions: Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium: Spark NLP articles.
• JohnSnowLabs official Medium
• YouTube: Spark NLP video tutorials.

Installation

Python

pip install spark-nlp==6.1.1

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.1

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.1</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.1</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.1</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.1.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.1.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.1.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.1.jar

What's Changed

• https://github.com/JohnSnowLabs/spark-nlp/pull/14641 by @prabod
• https://github.com/JohnSnowLabs/spark-nlp/pull/14640 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14644 by @DevinTDHA and @C-K-Loan

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.1.0...6.1.1

📢 Spark NLP 6.1.0: State-of-the-art LLM Capabilities and Advancing Universal Ingestion

We are excited to announce Spark NLP 6.1.0, another milestone for building scalable, distributed AI pipelines! This major release significantly enhances our capabilities for state-of-the-art multimodal and large language models and universal data ingestion. Upgrade Spark NLP to 6.1.0 to improve both usability and performance across ingestion, inference, and multimodal processing pipelines, all within the native Spark ecosystem.

🔥 Highlights

• Upgraded llama.cpp Integration: We've updated our llama.cpp backend to tag b5932 which supports inference with the latest generation of LLMs.
• Unified Document Ingestion with Reader2Doc: Introducing a new annotator that streamlines the process of loading and integrating diverse file formats (PDFs, Word, Excel, PowerPoint, HTML, Text, Email, Markdown) directly into Spark NLP pipelines with a unified and flexible interface.
• Support for Phi-4: Spark NLP now natively supports the Phi-4 model, allowing users to leverage its advanced reasoning capabilities.

🚀 New Features & Enhancements

Large Language Models (LLMs)

• llama.cpp Upgrade: Our llama.cpp backend has been upgraded to version b5932. This update enables native inference for the newest LLMs, such as Gemma 3 and Phi-4, ensuring broader model compatibility and improved performance.
  • NOTE: We are still in the process of upgrading our multimodal AutoGGUFVisionModel annotator to the latest backend. This means that this annotator will not be available in this version. As a workaround, please use version 6.0.5 of Spark NLP.
• Phi-4 Model Support: Spark NLP now integrates the Phi-4 model, an advanced open model trained on a blend of synthetic data, filtered public domain content, and academic Q&A datasets. This integration enables sophisticated reasoning capabilities directly within Spark NLP. (Link to notebook)

Document Ingestion

• Reader2Doc Annotator: This new annotator provides a simplified, unified interface for integrating various Spark NLP readers. It supports a wide range of formats, including PDFs, plain text, HTML, Word (.doc/.docx), Excel (.xls/.xlsx), PowerPoint (.ppt/.pptx), email files (.eml, .msg), and Markdown (.md).
• Using this annotator, you can read all these different formats into Spark NLP documents, making them directly accessible in all your Spark NLP pipelines. This significantly reduces boilerplate code and enhances flexibility in data loading workflows, making it easier to scale and switch between data sources.

Let's use a code example to see how easy it is to use:

reader2doc = Reader2Doc() \
    .setContentType("application/pdf") \
    .setContentPath("./pdf-files") \
    .setOutputCol("document")

# other NLP stages in `nlp_stages`

pipeline = Pipeline(stages=[reader2doc] + nlp_stages)
model = pipeline.fit(empty_df)
result_df = model.transform(empty_df)

Check out our full example notebook to see it in action.

🐛 Bug Fixes

• HuggingFace OpenVINO Notebook for Qwen2VL: Addressed and fixed issues in the notebook related to the OpenVINO conversion of the Qwen2VL model, ensuring smoother functionality.

❤️ Community Support

• Slack: For live discussion with the Spark NLP community and the team.
• GitHub: Bug reports, feature requests, and contributions.
• Discussions: Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium: Spark NLP articles.
• JohnSnowLabs official Medium
• YouTube: Spark NLP video tutorials.

Installation

Python

pip install spark-nlp==6.1.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.0

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.0</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.0.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.0.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.0.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.0.jar

What's Changed

• Update HuggingFace_OpenVINO_in_Spark_NLP_Qwen2VL.ipynb #14631 by @AbdullahMubeenAnwar
• Sparknlp 1189 Introducing Phi4 #14606 by @prabod
• SPARKNLP-1259 Introducing Reader2Doc Annotator #14632 by @danilojsl
• [SPARKNLP-1194] Upgrade jsl-llamacpp to newest version #14633 by @DevinTDHa
• Add telemetry to github actions [skip-test] #14568 by @KshitizGIT

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.0.5...6.1.0

📢 Spark NLP 6.0.5: Enhanced Microsoft Fabric Integration & Markdown Processing

We're thrilled to announce the release of Spark NLP 6.0.5! This version introduces a new Markdown Reader, enabling direct processing of Markdown files into structured Spark DataFrames for more diverse NLP workflows. We have also enhanced Microsoft Fabric integration, allowing for seamless model downloads from Lakehouse containers.

🔥 Highlights

• New Markdown Reader: Introduce the new MarkdownReader for effortlessly parsing Markdown files into structured Spark DataFrames, paving the way for advanced content analysis and NLP on Markdown content.
• Enhanced Microsoft Fabric Support: Download models directly from Microsoft Fabric Lakehouse containers, streamlining your NLP deployments in the Fabric environment.

🚀 New Features & Enhancements

• New MarkdownReader Annotator: Introducing the MarkdownReader, a powerful new feature that allows you to read and parse Markdown files directly into a structured Spark DataFrame. This enables efficient processing and analysis of Markdown content for various NLP applications. We recommend using this reader automatically in our Partition annotator. (Link to notebook)

  partitioner = Partition(content_type = "text/markdown"").partition(md_directory)
  

• Microsoft Fabric Integration: Spark NLP now supports downloading models from Microsoft Fabric Lakehouse containers, providing a more integrated and efficient workflow for users leveraging Microsoft Fabric. This enhancement ensures smoother model access and deployment within the Fabric ecosystem. For example, you can define the path to our pretrained models in Spark like so:

  from pyspark import SparkConf
  
  conf = SparkConf()
  conf.set("spark.jsl.settings.pretrained.cache_folder", "abfss://my_workspace@onelake.dfs.fabric.microsoft.com/lakehouse_folder.Lakehouse/Files/my_models")
  

🐛 Bug Fixes

We performed crucial maintenance updates to all of our example notebooks, ensuring that they are reproducible and properly displayed in GitHub.

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

⚙️ Installation

Python

#PyPI
pip install spark-nlp==6.0.5

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.5

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.5

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.5

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.5

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.0.5</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.0.5</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.0.5</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.0.5</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.0.5.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.0.5.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.0.5.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.0.5.jar

What's Changed

• https://github.com/JohnSnowLabs/spark-nlp/pull/14617 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14618 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14620 by @AbdullahMubeenAnwar

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.0.4...6.0.5

📢 Spark NLP 6.0.4: MiniLMEmbeddings, DataFrame Optimization, and Enhanced PDF Processing

We are excited to announce the release of Spark NLP 6.0.4! This version brings advancements in text embeddings with the introduction of the MiniLM family, Spark DataFrame optimizations, and enhanced PDF document parsing. Upgrade to 6.0.4 to leverage these cutting-edge features and expand your NLP capabilities at scale.

Stay updated with our latest examples and tutorials by visiting our Medium - Spark NLP blog!

🔥 Highlights

• Introducing MiniLMEmbeddings: Support for the efficient and powerful MiniLMEmbeddings models, providing state-of-the-art text representations.
• New DataFrameOptimizer: A new DataFrameOptimizer transformer to streamline and optimize Spark DataFrame operations, offering configurable repartitioning, caching, and persistence options.
• Advanced PDF Reader Features: Enhancements to the PDF Reader with extractCoordinates for spatial metadata, normalizeLigatures for improved text consistency, and a new exception column for enhanced fault tolerance.

🚀 New Features & Enhancements

Advanced Text Embeddings

This release introduces a new family of efficient text embedding models:

• MiniLMEmbeddings: Support for the MiniLMEmbeddings annotator, enabling the use of MiniLM models for generating highly efficient and effective sentence embeddings. These models are designed to provide strong performance while being significantly smaller and faster than larger alternatives, making them ideal for a wide range of NLP tasks requiring compact and powerful text representations. (Link to notebook)

Spark DataFrame Optimization

• DataFrameOptimizer: Introducing the new DataFrameOptimizer transformer, designed to enhance the performance and manageability of Spark DataFrames within your NLP pipelines. (Link to notebook)
  • Configurable Repartitioning: Allows for automatic repartitioning of DataFrames, ensuring optimal data distribution for downstream processing.
  • Optional Caching: Supports DataFrame caching (doCache) to significantly speed up iterative computations.
  • Persistent Output: Adds robust support for persisting DataFrames to disk in various formats (csv, json, parquet) with custom writer options via outputOptions.
  • Schema Preservation: Efficiently preserves the original DataFrame schema, making it a seamless utility for complex Spark NLP pipelines.

Enhanced PDF Document Processing

The PDF Reader and PdfToText transformer have been significantly improved for more comprehensive and fault-tolerant document parsing. (Link to notebook)

• Spatial Metadata Extraction (extractCoordinates): A new configurable parameter extractCoordinates in PdfToText and the PDF Reader. When enabled, this outputs detailed spatial metadata (text position and dimensions) for each character in the PDF.
• Ligature Normalization (normalizeLigatures): When extractCoordinates is enabled, the normalizeLigatures option ensures that ligature characters (e.g., fi, fl, œ) are automatically normalized to their decomposed forms (fi, fl, oe).
• Fault Tolerance with Exception Column: A new exception output column has been introduced to capture and log any processing errors encountered while handling individual PDF documents.

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

⚙️ Installation

Python

#PyPI
pip install spark-nlp==6.0.4

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.4

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.4

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.4

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.4

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.0.4</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.0.4</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.0.4</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.0.4</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.0.4.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.0.4.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.0.4.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.0.4.jar

What's Changed

• Sparknlp 282 Introducing MiniLMEmbeddings #14610 by @prabod
• [SPARKNLP-1086] Introducing DataFrameOptimizer #14607 by @danilojsl
• [SPARKNLP-1161] Adding features to PDF Reader #14596 by @danilojsl

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.0.3...6.0.4

📢 Spark NLP 6.0.3: Multimodal E5-V Embeddings and Enhanced Document Partitioning

We are excited to announce the release of Spark NLP 6.0.3! This version introduces significant advancements in multimodal capabilities and further refines document processing workflows. Upgrade to 6.0.3 to leverage these cutting-edge features and expand your NLP and vision task capabilities at scale.

🔥 Highlights

• Introducing E5-V Universal Multimodal Embeddings: Support for E5VEmbeddings, enabling universal multimodal embeddings with Multimodal Large Language Models (MLLMs). It can express semantic similarly between texts, images, or a combination of both.
• Enhanced Document Partitioning: Improvements to the Partition and PartitionTransformer annotators with new character and title-based chunking strategies.
• New XML Reader: Added sparknlp.read().xml() and integrated XML support into the Partition annotator for streamlined XML document processing.

🚀 New Features & Enhancements

E5-V Multimodal Embeddings

This release further boosts Spark NLP's multimodal processing power with the integration of E5-V.

• E5VEmbeddings is designed to adapt MLLMs for achieving universal multimodal embeddings. It leverages MLLMs with prompts to effectively bridge the modality gap between different types of inputs, demonstrating strong performance in multimodal embeddings even without fine-tuning. (Link to notebook)

Enhanced Unstructured Document Processing

The Partition and PartitionTransformer components now include additional chunking strategies and enhancements, which divides content into meaningful units based on the document's structure or number of characters.

• New Chunking Strategies (Link to notebook)
  • Character Number Strategy (maxCharacters): Split documents by number of characters.
  • Title-Based Chunking Strategy (byTitle): Split documents by titles in the documents. Additional settings:
  • Soft Chunking Limit (newAfterNChars): Allows for early section breaks before reaching the maxCharacters threshold.
  • Contextual Overlap (overlapAll): Adds trailing context from the previous chunk to the next, improving semantic continuity.
• Enhancements
  • Page Boundary Splitting: Respects pageNumber metadata and starts a new section when a page changes.
  • Title Inclusion Behavior: Ensures titles are embedded within the following content rather than forming isolated chunks.
  • New XML Reader: This release introduces a new feature that enables reading and parsing XML files into a structured Spark DataFrame. (Link to notebook)
    • Added sparknlp.read().xml(): This method accepts file paths of XML content.
    • Use in Partition: XML content can now be processed using the Partition annotator by setting content_type = "application/xml".

🐛 Bug Fixes

• @thec0dewriter fixed a typo in our excel reader notebook (https://github.com/JohnSnowLabs/spark-nlp/pull/14591) Thanks a lot!

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

⚙️ Installation

Python

#PyPI
pip install spark-nlp==6.0.3

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.3

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.3

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.0.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.0.3</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.0.3</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.0.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.0.3.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.0.3.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.0.3.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.0.3.jar

What's Changed

• [SPARKNLP-1138] Adding semantic chunking to partition #14593 by @danilojsl
• [SPARKNLP-1163] Adding title chunking strategy #14594 by @danilojsl
• SparkNLP 1143 - Introducing e5-v universal embeddings with multimodal large language models #14597 by @prabod
• Fix reference copy pasted from Excel reader #14591 by @thec0dewriter
• [SPARKNLP-1119] Adding XML reader #14598 by @danilojsl

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.0.2...6.0.3

📢 Spark NLP 6.0.2: Advancing Multimodal Capabilities and Streamlining Document Processing

We are thrilled to announce the release of Spark NLP 6.0.2! This version introduces powerful new multimodal models and significantly enhances document processing workflows. Upgrade to 6.0.2 to leverage these cutting-edge features and expand your NLP and vision task capabilities at scale.

Stay updated with our latest examples and tutorials by visiting our Medium - Spark NLP blog!

🔥 Highlights

• Introducing InternVL: Support for the state-of-the-art InternVLForMultiModal model, enabling advanced visual question answering with InternVL 2, 2.5, and 3 series models.
• Introducing Florence-2: Integration of Florence-2 in Florance2Transformer, a sophisticated vision foundation model for diverse prompt-based vision and vision-language tasks like captioning, object detection, and segmentation.
• New Document Partitioning Feature: Added the Partition and PartitionTransformer annotator for a unified and configurable interface with Spark NLP readers, simplifying unstructured data loading.

🚀 New Features & Enhancements

Advanced Multimodal Model Integrations

This release significantly boosts Spark NLP's multimodal processing power with the integration of two new visual language models:

• InternVL: InternVLForMultiModal is a powerful multimodal large language model is specifically designed for visual question answering. This annotator is versatile, supporting the InternVL 2, 2.5, and 3 families of models, allowing users to tackle complex visual-linguistic tasks. (Link to notebook)
• Florence-2: Introducing Florance2Transformer, an advanced vision foundation model. Florence-2 utilizes a prompt-based approach, enabling it to perform a wide array of vision and vision-language tasks. Users can leverage simple text prompts to execute tasks such as image captioning, object detection, and image segmentation with high accuracy. (Link to notebook)

Enhanced Unstructured Document Processing

• Partitioning Documents: This release introduces the new Partition and PartitionTransformer annotator.
  • Partition provides a unified interface for extracting structured content from various document formats into Spark DataFrames. It supports input from files, URLs, in-memory strings, or byte arrays and handles formats such as text, HTML, Word, Excel, PowerPoint, emails, and PDFs. It automatically selects the appropriate reader based on file extension or MIME type and allows customization via parameters. (Link to notebook)
  • The PartitionTransformer annotator allows you to use the Partition feature more smoothly within existing Spark NLP workflows, enabling seamless reuse of your pipelines. PartitionTransformer can be used for extracting structured content from various document types using Spark NLP readers. It supports reading from files, URLs, in-memory strings, or byte arrays, and returns parsed output as a structured Spark DataFrame. (Link to notebook)
• Key Improvements:
  • Simplifies integration with Spark NLP readers through a unified interface.
  • Adds flexibility by enabling more reader-specific configurations.
  • Enhances the maintainability and scalability of data loading workflows.

🐛 Bug Fixes

• Adjusted python type annotations for the AutoGGUFModel (#14576)

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

⚙️ Installation

Python

#PyPI
pip install spark-nlp==6.0.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.2

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.0.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.0.2</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.0.2</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.0.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.0.2.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.0.2.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.0.2.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.0.2.jar

What's Changed

• SparkNLP - 1123 Introducing InternVL #14578 by @prabod
• Documentation for SparkNLP Readers and Partition class #14581 by @paulamib123
• Sparknlp-1174 Adding Partitioning Documents Feature #14579 by @danilojsl
• SparkNLP 1131 - Introducing Florance-2 #14585 by @prabod

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.0.1...6.0.2

📢 Spark NLP 6.0.1: Introducing New State-of-the-Art Vision-Language Models and Enhanced Document Processing

We are pleased to announce the release of Spark NLP 6.0.1, bringing exciting new vision features and continued enhancements. Expand your NLP capabilities at scale for a wide range of tasks by upgrading to 6.0.1 and leverage these powerful new additions and improvements!

We also have been adding blog posts covering various examples for our newest features. Check them out at Medium - Spark NLP!

🔥 Highlights

• Added support for several new State-of-the-Art vision language models (VLM) including Gemma 3, PaliGemma, PaliGemma2, and SmolVLM.
• Introduced new parameter options for the PDF Reader for enhanced document ingestion control.

🚀 New Features & Enhancements

New VLM Implementations

This release adds support for several cutting-edge VLMs, significantly expanding the range of tasks you can tackle with Spark NLP:

• Gemma 3: The latest version of Google's lightweight, state-of-the-art open models. (link to notebook)
• PaliGemma and PaliGemma 2: Integration of the original PaliGemma vision-language model by Gogle. This annotator can also read PaliGemma2 models. (link to notebook)
• SmolVLM: small, fast, memory-efficient, and fully open-source 2B VLM (link to notebook)

PDF Reader Enhancements

The PDF Reader now includes additional parameters and options, providing users with more flexible and controlled ingestion of PDF documents, improving handling of various PDF structures. (link to notebook)

You can now

• Add splitPage parameter to identify the correct number of pages
• Add onlyPageNum parameter to display only the number of pages of the document
• Add textStripper parameter used for output layout and formatting
• Add sort parameter to enable or disable sorting lines

🐛 Bug Fixes

This release also includes fixes for several issues:

• Fixed a python error in RoBERtaMultipleChoice, preventing these types of annotators to be loaded in Python
• Fixed various typos and issues in our Jupyter notebook examples

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

⚙️ Installation

Python

#PyPI

pip install spark-nlp==6.0.1

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.1

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.0.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.0.1</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.0.1</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.0.1</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.0.1.jar
• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.0.1.jar
• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.0.1.jar
• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.0.1.jar

What's Changed

• [SPARKNLP-1164] Updating python 3.7 to 3.8 for jobs spark34 and spark33 by @danilojsl https://github.com/JohnSnowLabs/spark-nlp/pull/14565
• [SPARKNLP-1177] Solving typo in Python for RoBertaForMultipleChoice by @danilojsl https://github.com/JohnSnowLabs/spark-nlp/pull/14567
• Sparknlp 1115 Introducing SmolVLM by @prabod https://github.com/JohnSnowLabs/spark-nlp/pull/14552
• Fixing typos in notebooks by @ahmedlone127 https://github.com/JohnSnowLabs/spark-nlp/pull/14570
• SparkNLP 1121 Introducing PaliGemma by @prabod https://github.com/JohnSnowLabs/spark-nlp/pull/14551
• SparkNLP 1124 Introducing Gemma 3 by @prabod https://github.com/JohnSnowLabs/spark-nlp/pull/14556
• Sparknlp-1158 Adding Parameters Options to PDF Reader by @danilojsl https://github.com/JohnSnowLabs/spark-nlp/pull/14562

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.0.0...6.0.1

:loudspeaker: Spark NLP 6.0.0: A New Era for Universal Ingestion and Multimodal LLM Processing at Scale

From raw documents to multimodal insights at enterprise scale

With Spark NLP 6.0.0, we are setting a new standard for building scalable, distributed AI pipelines. This release transforms Spark NLP from a pure NLP library into the de facto platform for distributed LLM ingestion and multimodal batch processing.

This release introduces native ingestion for enterprise file types including PDFs, Excel spreadsheets, PowerPoint decks, and raw text logs, with automatic structure extraction, semantic segmentation, and metadata preservation — all in scalable, zero-code Spark pipelines.

At the same time, Spark NLP now natively supports Vision-Language Models (VLMs), loading quantized multimodal models like LLAVA, Phi Vision, DeepSeek Janus, and Llama 3.2 Vision directly via Llama.cpp, ONNX, and OpenVINO runtimes with no external inference servers, no API bottlenecks.

With 6.0.0, Spark NLP offers a complete, distributed architecture for universal data ingestion, multimodal understanding, and LLM batch inference at scale — enabling retrieval-augmented generation (RAG), document understanding, compliance audits, enterprise search, and multimodal analytics — all within the native Spark ecosystem.

One unified framework. Text, vision, documents — at Spark scale. Zero boilerplate. Maximum performance.

:star2: Spotlight Feature: AutoGGUFVisionModel — Native Multimodal Inference with Llama.cpp

Spark NLP 6.0.0 introduces the new AutoGGUFVisionModel, enabling native multimodal inference for quantized GGUF models directly within Spark pipelines. Powered by Llama.cpp, this annotator makes it effortless to run Vision-Language Models (VLMs) like LLAVA-1.5-7B Q4_0, Qwen2 VL, and others fully on-premises, at scale, with no external servers or APIs required.

With Spark NLP 6.0.0, Llama.cpp vision models are now first-class citizens inside DataFrames, delivering multimodal inference at scale with native Spark performance.

Why it matters

For the first time, Spark NLP supports pure vision-text workflows, allowing you to pass raw images and captions directly into LLMs that can describe, summarize, or reason over visual inputs.
This unlocks batch multimodal processing across massive datasets with Spark’s native scalability — perfect for product catalogs, compliance audits, document analysis, and more.

How it works

• Accepts raw image bytes (not Spark's OpenCV format) for true end-to-end multimodal inference.
• Provides a convenient helper function ImageAssembler.loadImagesAsBytes to prepare image datasets effortlessly.
• Supports all Llama.cpp runtime parameters like context length (nCtx), top-k/top-p sampling, temperature, and repeat penalties, allowing fine control over completions.

Example usage

documentAssembler = DocumentAssembler() \
    .setInputCol("caption") \
    .setOutputCol("caption_document")

imageAssembler = ImageAssembler() \
    .setInputCol("image") \
    .setOutputCol("image_assembler")

data = ImageAssembler \
    .loadImagesAsBytes(spark, "src/test/resources/image/") \
    .withColumn("caption", lit("Caption this image."))

model = AutoGGUFVisionModel.pretrained() \
    .setInputCols(["caption_document", "image_assembler"]) \
    .setOutputCol("completions") \
    .setBatchSize(4) \
    .setNPredict(40) \
    .setTopK(40) \
    .setTopP(0.95) \
    .setTemperature(0.05)

pipeline = Pipeline().setStages([documentAssembler, imageAssembler, model])
results = pipeline.fit(data).transform(data)
results.selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "completions.result").show(truncate=False)

📚 A full notebook walkthrough is available here.

:fire: New Features & Enhancements

PDF Reader

Font-aware PDF ingestion is now available with automatic page segmentation, encrypted file support, and token-level coordinate extraction, ideal for legal discovery and document Q&A.

Excel Reader

Spark NLP can now ingest .xls and .xlsx files directly into Spark DataFrames with automatic schema detection, multiple sheet support, and rich-text extraction for LLM pipelines.

PowerPoint Reader

Spark NLP introduces a native reader for .ppt and .pptx files. Capture slides, speaker notes, themes, and alt text at the document level for downstream summarization and retrieval.

Extractor and Cleaner Annotators

New Extractor and Cleaner annotators allow you to pull structured data (emails, IP addresses, dates) from text or clean noisy text artifacts like bullets, dashes, and non-ASCII characters at scale.

Text Reader

A high-performance TextReader is now available to load .txt, .csv, .log and similar files. It automatically detects encoding and line endings for massive ingestion jobs.

AutoGGUFVisionModel for Multimodal Llama.cpp Inference

Spark NLP now supports vision-language models in GGUF format using the new AutoGGUFVisionModel annotator. Run models like LLAVA-1.5-7B Q4_0 or Qwen2 VL entirely within Spark using Llama.cpp, enabling native multimodal batch inference without servers.

DeepSeek Janus Multimodal Model

The DeepSeek Janus model, tuned for instruction-following across text and images, is now fully integrated and available via a simple pretrained call.

Qwen-2 Vision-Language Model Catalog

Support for Alibaba’s Qwen-2 VL series (0.5B to 7B parameters) is now available. Use Qwen-2 checkpoints for OCR, product search, and multimodal retrieval tasks with unified APIs.

Native Multimodal Support with Phi-3.5 Vision

The new Phi3Vision annotator brings Microsoft’s Phi-3.5 multimodal model into Spark NLP. Process images and prompts together to generate grounded captions or visual Q&A results, all with a model footprint of less than 1 GB.

LLAVA 1.5 Vision-Language Transformer

Spark NLP now supports LLAVA 1.5 (7B) natively for screenshot Q&A, chart reading, and UI testing tasks. Build fully distributed multimodal inference pipelines without external services or dependencies.

Native Cohere Command-R Models

Cohere’s multilingual Command-R models (up to 35B parameters) are now fully integrated. Perform reasoning, RAG, and summarization tasks with no REST API latency and no token limits.

OLMo Family Support

Spark NLP now supports the full OLMo suite of open-weight language models (7B, 1.7B, and more) directly in Scala and Python. OLMo models come with full training transparency, Dolma-sized vocabularies, and reproducible experiment logs, making them ideal for academic research and benchmarking.

Multiple-Choice Heads for LLMs

New lightweight multiple-choice heads are now available for ALBERT, DistilBERT, RoBERTa, and XLM-RoBERTa models. These are perfect for building auto-grading systems, educational quizzes, and choice ranking pipelines.

• AlbertForMultipleChoice
• DistilBertForMultipleChoice
• RoBertaForMultipleChoice
• XlmRoBertaForMultipleChoice

VisionEncoderDecoder Improvements

The Scala API for VisionEncoderDecoder has been fully refactored to expose .generate() parameters like batch size and maximum tokens, aligning it one-to-one with the Python API.

🐛 Bug Fixes

Better GGUF Error Reporting

When a GGUF file is missing tensors or uses unsupported quantization, Spark NLP now provides clear and actionable error messages, including guidance on how to fix or convert the model.

Fixed MXBAI Typo

A small typo related to the MXBAI integration was corrected to ensure consistency across annotator names and pretrained model references.

VisionEncoderDecoder Alignment

The Scala VisionEncoderDecoder wrapper has been updated to fully match the Python API. It now exposes parameters like batch size and maximum tokens, fixing discrepancies that could occur in cross-language pipelines.

Minor Naming Improvements

Variable naming inconsistencies have been cleaned up throughout the codebase to ensure a more uniform and predictable developer experience.

📝 Models

We have added more than 110,000 new models and pipelines. The complete list of all 88,000+ models & pipelines in 230+ languages is available on our Models Hub.

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==6.0.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.0.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.0.0

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.0.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.0.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.0.0</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.0.0</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.0.0</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.0.0</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.0.0.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.0.0.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.0.0.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.0.0.jar

What's Changed

• Update create_search_index.yml by @agsfer in https://github.com/JohnSnowLabs/spark-nlp/pull/14526
• SPARKNLP-1006: Introducing OLMo by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14242
• Sparknlp 1060 implement phi 3.5 vision by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14444
• SparkNLP 1033: Introducing LLAVA by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14450
• SparkNLP 1032: Introducing CoHere by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14457
• SparkNLP 1077- Introducing Qwen2 - VL by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14474
• updating python and scala model names by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14488
• [SPARKNLP-1102] Adding support to read Excel files by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14489
• [SPARKNLP-1103] Adding support to read power point files by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14491
• [SPARKNLP-1105] Introducing AlbertForMultipleChoice Transformer by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14492
• [SPARKNLP-1106] Introducing DistilBertForMultipleChoice Transformer by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14493
• [SPARKNLP-1107] Introducing RoBertaForMultipleChoice by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14495
• [SPARKNLP-1108] Introducing XlmRoBertaForMultipleChoice Transformer by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14497
• [SPARKNLP-1098] Adding PDF reader support by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14499
• Sparknlp 1078 Introducing llama 3.2 vision models by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14502
• [SPARKNLP-1079] AutoGGUFVisionModel by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14505
• fixing typo in MXBAI notebook by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14510
• SPARKNLP-1109 Adding Extractor to Sparknlp by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14519
• [SPARKNLP-1113] Adding Text Reader by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14524
• SparkNLP 1088 - Introducing Deepseek Janus by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14532
• Improved Error Handling for AutoGGUF models by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14533
• Update VisionEncoderDecoder.scala by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14553
• fixing name by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14554
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14557
• Release/600 release candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14534

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.5.3...6.0.0

:loudspeaker: Spark NLP: Enhanced Embeddings, Fixed Attention Masks, and Performance Optimizations

Introduction

We’re excited to introduce the latest release of Spark NLP 5.5.3, featuring critical enhancements and bug fixes for several of our Text Embeddings annotators. These improvements ensure even more reliable and efficient performance for your NLP workflows.

But that’s not all—we’re also celebrating a major milestone: crossing 100,000 truly free and open models on our Models Hub! This achievement underscores our commitment to making state-of-the-art NLP accessible to everyone, forever.

Upgrade today to take advantage of these enhancements, and thank you for being part of the Spark NLP community. Your support and contributions continue to drive innovation forward!

🔥 Highlights

• Enhanced BGE Embeddings with configurable pooling strategies
• Fixed attention mask padding across multiple embedding models
• Major performance optimizations for transformer models
• Improved model default configurations and traits

🚀 New Features

Enhanced BGE Embeddings

Previously, BGE embeddings used a fixed pooling strategy that didn't match all model variants, resulting in suboptimal performance for some models (cosine similarity around 0.97 compared to the original implementation). Different BGE models are trained with different pooling strategies - some use CLS token pooling while others use attention-based average pooling.

• Added new useCLSToken parameter to control embedding pooling strategy
• Changed default pretrained model from "bge_base" to "bge_small_en_v1.5"

val embeddings = BGEEmbeddings.pretrained("bge_small_en_v1.5")
  .setUseCLSToken(true)  // Use CLS token pooling (default)
  .setInputCols("document")
  .setOutputCol("embeddings")

🛠 Improvements & Bug Fixes

Attention Mask Fixes

Fixed incorrect padding in attention mask calculations for multiple models:

• MPNet
• BGE
• E5
• Mxbai
• Nomic
• SnowFlake
• UAE

This fix ensures consistent results between native implementations and ONNX versions.

Other Fixes

• Fixed Llama3 download issues in Python
• Optimized OpenVINO and ONNX inference paths
• Enhanced code cleanup and standardization

🔄 Breaking Changes

BGE Embeddings Updates

1. Default Model Change:

   • Old default: "bge_base"
   • New default: "bge_small_en_v1.5"
   • Action required: Explicitly specify "bge_base" if needed

2. Pooling Strategy:

   • New useCLSToken parameter defaults to True
   • May affect embedding calculations
   • Action required: Verify existing implementations and set parameter explicitly if needed

💡 Usage Examples

Specifying BGE Model Version

// Using new default
val embeddingsNew = BGEEmbeddings.pretrained()

// Using previous default explicitly
val embeddingsOld = BGEEmbeddings.pretrained("bge_base")

Configuring Pooling Strategy

// Using CLS token pooling
val embeddingsCLS = BGEEmbeddings.pretrained()
  .setUseCLSToken(true)

// Using attention-based average pooling
val embeddingsAvg = BGEEmbeddings.pretrained()
  .setUseCLSToken(false)

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.5.3

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.3

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.3

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.5.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.5.3</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.5.3</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.5.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.5.3.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.5.3.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.5.3.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.5.3.jar

What's Changed

• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14490
• LLama3 download fix by @C-K-Loan in https://github.com/JohnSnowLabs/spark-nlp/pull/14508
• Bugfix: wrong attention mask calculation resulted in wrong embeddings by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14496
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14513
• Release/553 release candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14511

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.5.2...5.5.3

:loudspeaker: Spark NLP 5.5.2

We’re thrilled to introduce the latest enhancements and new features in this release of Spark NLP! These additions bring more powerful model inference capabilities, seamless data ingestion methods, and greater flexibility for scaling your NLP workflows.

Upgrade today to take advantage of these new capabilities and improvements. As always, we look forward to your feedback and contributions, and thank you for being part of the Spark NLP community!

:fire: New Features & Enhancements

🚀 Major New Features

• OpenVINO Support for Transformers (#14408) Many popular transformer-based annotators now leverage OpenVINO for faster inference on Intel hardware. Enjoy speedier pipelines across a wide array of models—such as DeBerta, DistilBert, RoBerta, XlmRoBerta, Albert, and more—enabling efficient, production-grade NLP at scale.

• BLIPForQuestionAnswering Transformer (#14422) Introducing BLIPForQuestionAnswering, a new image-based question-answering transformer. Simply provide an image and a question, and BLIP will deliver contextually relevant answers. Perfect for use cases in image analysis, e-commerce, and beyond.

• AutoGGUFEmbeddings Annotator (#14433) Seamlessly integrate AutoGGUFModels into your NLP pipeline. The new AutoGGUFEmbeddings annotator provides dense vector embeddings, making it easier than ever to incorporate advanced sentence embeddings into your workflows. We’ve included an end-to-end notebook to help you get started right away.

📜 New Data Ingestions

• Parsing HTML to DataFrames (#14449) Need to analyze web content at scale? Use sparknlp.read().html() to parse local or remote HTML files into structured Spark DataFrames. This new feature makes web-scale data analysis and downstream NLP tasks more accessible and scalable.

• Email Content to DataFrames (#14455) Leverage sparknlp.read().email() to transform email content into organized DataFrames. Analyze communications, extract insights, and enrich your NLP pipelines with minimal effort. (Requires #14449 to be merged first.)

• Microsoft Word Document Parsing (#14476) Turn .docx and .doc files into structured Spark DataFrames for streamlined integration into your NLP projects. From enterprise documents to reports, this feature simplifies data preparation and analysis at scale.

🐛 Bug Fixes

• Microsoft Fabric Integration (#14467) We’ve added support for Microsoft Fabric to store and retrieve word embeddings efficiently. Leverage your existing infrastructure to scale Spark NLP solutions more effectively.

• cuDNN Upgrade Instructions for Databricks (#14451) Easily upgrade cuDNN on Databricks to accelerate ONNX model inference on GPU, and take advantage of updated installation instructions for a cleaner setup.

• Metadata Preservation in ChunkEmbeddings (#14462) ChunkEmbeddings now retain original metadata, ensuring richer context and more meaningful insights in your downstream tasks.

• Default Names and Languages for New Annotators (#14469) We’ve standardized default names and languages in our seq2seq annotators for better clarity, consistency, and ease of use.

📦 Dependencies

Updated:

• Jsoup has been updated from 1.18.1 to 1.18.2 to ensure compatibility and maintain security and performance standards.

New Additions for Email and Document Parsing:

• Jakarta Mail (jakarta.mail:jakarta.mail-api:2.1.3): Added to support parsing and processing email content.

• Angus Mail (org.eclipse.angus:angus-mail:2.0.3): Complementary mail handling library integrated for more robust email parsing capabilities.

• Apache POI (org.apache.poi:poi-ooxml:4.1.2 & org.apache.poi:poi-scratchpad:4.1.2): Introduced for parsing Word documents (.docx and .doc) into structured DataFrames, enabling seamless integration of document-based data into Spark NLP workflows.

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.5.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.2

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.5.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.5.2</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.5.2</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.5.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.5.2.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.5.2.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.5.2.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.5.2.jar

What's Changed

• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14458
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14470
• adding openvino support to all ClassificationForXXX annotators by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14408
• [SPARKNLP-1068] Introducing BLIPForQuestionAnswering transformer by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14422
• [SPARKNLP-1091] AutoGGUFModel embeddings support by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14433
• Apache Spark vulnerable Fix by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14441
• [SPARKNLP-1092] Adding support to read HTML files by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14449
• [SPARKNLP-1095] Add installation instructions for ONNX GPU on Databricks by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14451
• [SPARKNLP-1093] Adding support to read Email files by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14455
• Small typos by @svlandeg in https://github.com/JohnSnowLabs/spark-nlp/pull/14459
• Addition chunk metadata to ChunkEmbeddings output by @mehmetbutgul in https://github.com/JohnSnowLabs/spark-nlp/pull/14462
• [SPARKNLP-1096] Adding support to Microsoft Fabric for WordEmbeddings by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14467
• Default name updates by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14469
• SPARKNLP-1094 Adding Support to Read Word Files by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14476
• ignore html as linguist-vendored by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14481
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14482
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14485
• Spark NLP 5.5.2 Release Candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14473

New Contributors

• @svlandeg made their first contribution in https://github.com/JohnSnowLabs/spark-nlp/pull/14459

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.5.1...5.5.2

:fire: Enhancements & Bug Fixes

• BertForMultipleChoice Transformer Added. Enhanced BERT’s capabilities to handle multiple-choice tasks such as standardized test questions and survey or quiz automation.
• PromptAssembler Annotator Introduced. Introduced a new annotator that constructs prompts for LLMs using a chat template and a sequence of messages. Accepts an array of tuples with roles (“system”, “user”, “assistant”) and message texts. Utilizes llama.cpp as a backend for template parsing, supporting basic template applications. Example Notebook

promptAssembler = (
    PromptAssembler()
    .setInputCol("messages")
    .setOutputCol("prompt")
    .setChatTemplate(template)
)

• Integrated New Tasks and Documentation. Added support and documentation for the following tasks:
  • Automatic Speech Recognition
  • Dependency Parsing
  • Image Captioning
  • Image Classification
  • Landing Page
  • Question Answering
  • Summarization
  • Table Question Answering
  • Text Classification
  • Text Generation
  • Text Preprocessing
  • Token Classification
  • Translation
  • Zero-Shot Classification
  • Zero-Shot Image Classification
• Resolved Pretrained Model Loading Issue on DBFS Systems.
• Fixed a bug where pretrained models were not found when running AutoGGUFModel pipelines on Databricks due to incorrect path handling of gguf files.

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.5.1

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x: (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.1

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.1

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.5.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.5.1</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.5.1</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.5.1</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.5.1.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.5.1.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.5.1.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.5.1.jar

What's Changed

• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14418
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14420
• Add a new llama_cpp engine by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14436
• tasks-docs-integration by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14428
• Introducing BertForMultipleChoice transformer by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14435
• Fix pretrained models not being found on dbfs systems by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14438
• [SPARKNLP-1067] PromptAssembler by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14439
• Release/551 release candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14437

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.5.0...5.5.1

:loudspeaker: Spark NLP 5.5.0: Unlocking New Horizons with Llama.cpp Integration and More!

We're thrilled to announce the release of Spark NLP 5.5.0, a groundbreaking update that pushes the boundaries of natural language processing! This release is packed with exciting new features, optimizations, and integrations that will transform your NLP workflows. At the heart of this update is our game-changing integration with Llama.cpp, but that's just the beginning of what's in store!

:star2: Spotlight Feature: Llama.cpp Integration

Introducing Llama.cpp Integration: A New Era of Efficient Language Models!

We're proud to present the centerpiece of Spark NLP 5.5.0: the integration of Llama.cpp! This revolutionary addition brings unparalleled efficiency and performance to large language models within the Spark NLP ecosystem.

• Optimized Performance: Llama.cpp's C/C++ implementation allows for blazing-fast inference on CPUs, making large language models more accessible than ever.
• Reduced Memory Footprint: Enjoy the power of advanced language models with significantly lower RAM requirements.
• Quantization Support: Take advantage of various quantization options to further optimize model size and speed without sacrificing quality.
• Seamless Integration: Easily incorporate Llama.cpp models into your existing Spark NLP pipelines with our new AutoGGUFModel annotator.

This integration opens up new possibilities for deploying state-of-the-art language models in resource-constrained environments, making advanced NLP capabilities available to a wider range of applications and users.

We extend our heartfelt thanks to all contributors who made this release possible. Your innovative ideas, code contributions, and feedback continue to drive Spark NLP forward. Our Models Hub now contains over 83,000+ free and truly open-source models & pipelines. 🎉

:fire: New Features & Enhancements

Introducing QWEN2Transformer

We have added the QWEN2Transformer annotator, supporting the Qwen-2 model architecture known for its efficiency and performance in various NLP tasks like text generation and summarization.

View Pull Request

Introducing MiniCPM

The MiniCPM annotator is now available, providing support for the MiniCPM model designed for efficient language modeling with smaller parameter sizes without compromising performance.

View Pull Request

Introducing NLLB (No Language Left Behind)

We are excited to include the NLLB annotator, supporting No Language Left Behind models aimed at providing high-quality machine translation capabilities for a wide range of languages, especially low-resource languages.

View Pull Request

Implementing Nomic Embeddings

Introducing support for Nomic Embeddings, which provide robust semantic representations for downstream tasks like clustering and classification.

View Pull Request

Snowflake Integration

We have implemented integration with Snowflake, allowing seamless data transfer and processing between Spark NLP and Snowflake data warehouses.

View Pull Request

Introducing CamemBertForZeroShotClassification

The CamemBertForZeroShotClassification annotator is now available, enabling zero-shot classification capabilities using the CamemBERT model, optimized for French language processing.

View Pull Request

Implementing MxBai Embeddings

We have added support for MxBaiEmbeddings, providing embeddings from the MxBai model designed for multilingual text representation.

View Pull Request

ONNX Support for Vision Annotators

We have extended ONNX support to our vision annotators, allowing for optimized and accelerated inference for image-related NLP tasks.

View Pull Request

OpenVINO and ONNX Support for Additional Annotators

Building upon our commitment to performance optimization, we have added OpenVINO and ONNX support to several additional annotators, ensuring you can leverage hardware acceleration across a broader range of models.

View Pull Request

Introducing AlbertForZeroShotClassification

We are excited to introduce the AlbertForZeroShotClassification annotator, bringing zero-shot classification capabilities using the ALBERT model known for its parameter efficiency and strong performance.

View Pull Request

Introducing Phi-3

We have integrated Phi-3 models into Spark NLP, providing enhanced performance with high-efficiency quantization, supporting INT4 and INT8 quantization for CPUs via OpenVINO.

View Pull Request

Introducing StarCoder2 for Causal Language Modeling

The StarCoder2 model is now supported for causal language modeling tasks, enabling advanced code generation and understanding capabilities.

View Pull Request

Introducing LLAMA 3

Continuing our support for the latest in language modeling, we have introduced support for LLAMA 3, bringing the latest advancements in the LLaMA model series to Spark NLP.

View Pull Request

🐛 Bug Fixes

• OpenVINO Installation Instructions: Updated the installation instructions for OpenVINO to ensure a smoother setup process.

View Pull Request

• Fixed Default Auto GGUF Pretrained Model: Addressed issues with the default auto GGUF pretrained model in the Llama.cpp integration.

View Pull Requests, View Pull Request

• Updated Models Hub: Improved the Models Hub for better accessibility and search functionality.

View Pull Requests, View Pull Request, View Pull Request

• Artifact Creation Optimization: Switched to using 7zip instead of vimtor/action-zip for creating artifacts to enhance compatibility and performance.

View Pull Request

📦 Dependencies

• Published New OpenVINO Artifacts: Built and published new OpenVINO artifacts for both CPU and GPU to enhance performance and compatibility.

• Upgraded ONNX Runtime: Updated onnxruntime to the latest version for improved stability and performance on both CPU and GPU.

📝 Models

We have added more than 50,000 new models and pipelines. The complete list of all 83,000+ models & pipelines in 230+ languages is available on our Models Hub.

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.5.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.0

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.5.0</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.5.0</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.5.0</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.5.0</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.5.0.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.5.0.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.5.0.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.5.0.jar

What's Changed

• SparkNLP 997 Introducing QWEN2Transformer by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14188
• SparkNLP 1004 - Introducing MiniCPM by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14205
• SparkNLP 1018 - Introducing NLLB by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14209
• SparkNLP 1005 implement nomic embeddings by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14217
• implementing SnowFlake by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14353
• Introducing CamemBertForZeroShotClassification annotator by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14354
• Implementing Mxbai Embeddings by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14355
• Introducing onnx support to vision annotators by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14356
• Introducing onnx and OpenVino support to Missing Annotators by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14359
• [SPARKNLP-855] Introducing AlbertForZeroShotClassification by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14361
• SparkNLP introducing Phi-3 by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14373
• OpenVINO install instructions by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14382
• SPARKNLP 1034 implement starcoder2 for causal lm by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14358
• SPARKNLP Introducing LLAMA 3 by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14379
• 550 rc export notebooks by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14393
• [SPARKNLP-1027] llama.cpp integration by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14364
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14383
• Update create_search_index.yml by @pabla in https://github.com/JohnSnowLabs/spark-nlp/pull/14406
• Adding openvino support to missing annotators by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14390
• [SPARKNLP-1027] Change Default AutoGGUF pretrained model by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14411
• [SPARKNLP-1027] Fix issue with pretrained model by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14413
• Use 7zip instead of vimtor/action-zip for creating artifacts by @pabla in https://github.com/JohnSnowLabs/spark-nlp/pull/14414
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14416
• release/550-release-candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14389

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.4.2...5.5.0

:fire: Enhancements & Bug Fixes

• Added demo notebook for Image Classification Annotators https://github.com/JohnSnowLabs/spark-nlp/pull/14360
• Added aggressiveMatching parameter to DateMatcher and MultiDateMatcher annotators https://github.com/JohnSnowLabs/spark-nlp/pull/14365
• Added aggressiveMatching parameter to DocumentSimilarityRanker annotator https://github.com/JohnSnowLabs/spark-nlp/pull/14370

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.4.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x: (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.4.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.4.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.4.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.4.2

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.4.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.4.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.4.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.4.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.4.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.4.2</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.4.2</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.4.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.4.2.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.4.2.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.4.2.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.4.2.jar

What's Changed

• fixies in docs by @agsfer in https://github.com/JohnSnowLabs/spark-nlp/pull/14357
• release/542-release-candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14381

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.4.1...5.4.2

:fire: New Features & Enhancements

• Added support for loading duplicate models in Spark NLP, allowing multiple models from the same annotator to be loaded simultaneously.
• Updated the README for better coherence and added new pages to the website.
• Added support for a stop IDs list to halt text generation in Phi, Mistral, and Llama annotators.

:bug: Bug Fixes

• Fixed the default model names for Phi2 and Mistral AI annotators.

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.4.1

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x: (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.4.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.4.1

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.4.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.4.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.4.1

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.4.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.4.1</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.4.1</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.4.1</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.4.1.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.4.1.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.4.1.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.4.1.jar

What's Changed

• Fixing default names for Phi2 and MistralAI by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14338
• [SPARKNLP-1052] Adding random suffix to avoid duplication in spark files by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14340
• [SPARKNLP-1015] Restructuring Readme and Documentation by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14341
• Added custom stop token id support by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14344
• Update 2023-03-01-t5_flan_base_xx.md by @dcecchini in https://github.com/JohnSnowLabs/spark-nlp/pull/14345
• Spark NLP 5.4.1 by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14350

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.4.0...5.4.1

:loudspeaker: It's All About LLMs!

We're excited to share some amazing updates in the latest Spark NLP release of Spark NLP 🚀 5.4.0! This update is packed with new features and improvements that are set to transform natural language processing. One of the highlights is the integration of OpenVINO Runtime, which significantly boosts performance and efficiency across Intel hardware. You can now enjoy up to a 40% increase in performance compared to TensorFlow, with support for various model formats like ONNX, PaddlePaddle, TensorFlow, and TensorFlow Lite.

We've also added some powerful new annotators: BertEmbeddings, RoBertaEmbeddings, and XlmRoBertaEmbeddings. These are specially fine-tuned to take full advantage of the OpenVINO toolkit, offering better model accuracy and speed.

Another big change is in how we distribute models. We've moved from Broadcast to addFile for model distribution, which makes it easier to scale and manage large language models (LLMs) in cloud environments. This is especially helpful for models with over 7 billion parameters.

In addition, we've introduced the Mistral and Phi-2 architectures, optimized for high-efficiency quantization. There are also practical improvements to core components, like enhanced pooling for BERT-based models and updates to the OpenAIEmbeddings annotator for better performance and integration.

We want to thank our community for their valuable feedback, feature requests, and contributions. Our Models Hub now contains over 37,000+ free and truly open-source models & pipelines. 🎉

Spark NLP :heart: OpenVINO

:fire: New Features & Enhancements

NEW Integration: OpenVINO Runtime for Spark NLP 🚀: We're thrilled to announce the integration of OpenVINO Runtime, enhancing Spark NLP with high-performance inference capabilities. OpenVINO Runtime supports direct reading of models in ONNX, PaddlePaddle, TensorFlow, and TensorFlow Lite formats, enabling out-of-the-box optimizations and superior performance on supported Intel hardware.

Enhanced Model Support and Performance Gains: The integration allows Spark NLP to utilize the OpenVINO Runtime API for Java, facilitating the loading and execution of models across various formats including ONNX, PaddlePaddle, TensorFlow, TensorFlow Lite, and OpenVINO IR. Impressively, benchmarks show up to a 40% performance improvement over TensorFlow with no additional tuning required. Additionally, users can harness the full optimization and quantization capabilities of the OpenVINO toolkit via the Model Conversion API.

Enabled Annotators: This update brings OpenVINO compatibility to a range of Spark NLP annotators, including BertEmbeddings, RoBertaEmbeddings, XlmRoBertaEmbeddings, T5Transformer, E5Embeddings, LLAMA2, Mistral, Phi2, and M2M100.

Acknowledgements: This significant enhancement was accomplished during Google Summer of Code 2023. Special thanks to Rajat Krishna (@rajatkrishna) and the entire OpenVINO team for their invaluable support and collaboration. https://github.com/JohnSnowLabs/spark-nlp/pull/14200

<img width="613" alt="bert-large-cased-bs4" src="https://github.com/JohnSnowLabs/spark-nlp/assets/5762953/3595b668-556a-47a7-97b6-a506fb340b89"> <img width="612" alt="roberta-large-bs4" src="https://github.com/JohnSnowLabs/spark-nlp/assets/5762953/1003d687-928a-46e6-96aa-11216a8d8d2d">

• New Mistral Integration: We are excited to introduce the Mistral integration, featuring models fine-tuned on the MistralForCasualLM architecture. This addition enhances performance and efficiency by supporting quantization in INT4 and INT8 for CPUs via OpenVINO. https://github.com/JohnSnowLabs/spark-nlp/pull/14318

<img width="1622" alt="image" src="https://github.com/JohnSnowLabs/spark-nlp/assets/5762953/d2720f5d-a9a0-4c87-8ce9-3e6caea706c2">

Performance of Mistral 7B and different Llama models on a wide range of benchmarks. For all metrics, all models were re-evaluated with our evaluation pipeline for accurate comparison. Mistral 7B significantly outperforms Llama 2 13B on all metrics, and is on par with Llama 34B (since Llama 2 34B was not released, we report results on Llama 34B). It is also vastly superior in code and reasoning benchmarks. https://mistral.ai/news/announcing-mistral-7b/

Continuing our commitment to user-friendly and scalable solutions, the integration of the Mistral architecture has been designed to be straightforward and easily adoptable, ensuring that users can leverage these enhancements without complexity:

doc_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

mistral = MistralTransformer \
            .pretrained() \
            .setMaxOutputLength(50) \
            .setDoSample(False) \
            .setInputCols(["document"]) \
            .setOutputCol("mistral_generation")

• New Phi-2 Integrations: Introducing Phi-2, featuring models fine-tuned using the PhiForCausalLM architecture. This update enhances OpenVINO's capabilities, enabling quantization in INT4 and INT8 for CPUs to optimize both performance and efficiency. https://github.com/JohnSnowLabs/spark-nlp/pull/14318

Continuing our commitment to user-friendly and scalable solutions, the integration of the Phi architecture has been designed to be straightforward and easily adoptable, ensuring that users can leverage these enhancements without complexity:

doc_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

phi2 = Phi2Transformer \
        .pretrained() \
        .setMaxOutputLength(50) \
        .setDoSample(False) \
        .setInputCols(["document"]) \
        .setOutputCol("phi2_generation")

• NEW: Enhanced LLM Distribution: We've optimized the scalability of large language models (LLMs) in the cloud by transitioning from Broadcast to addFile for deep learning distribution across any cluster. This change addresses the challenges of handling modern LLMs—some boasting over 7 billion parameters—by improving memory management and overcoming serialization limits previously encountered with Java Bytes and Apache Spark's Broadcast method. This update significantly boosts Spark NLP's ability to process LLMs efficiently, underscoring our dedication to delivering scalable NLP solutions.https://github.com/JohnSnowLabs/spark-nlp/pull/14236

• NEW: MPNetForTokenClassification Annotator: Introducing the MPNetForTokenClassification annotator in Spark NLP 🚀. This annotator efficiently loads MPNet models equipped with a token classification head (a linear layer atop the hidden-states output), ideal for Named-Entity Recognition (NER) tasks. It supports models trained or fine-tuned in ONNX format using MPNetForTokenClassification for PyTorch or TFCamembertForTokenClassification for TensorFlow from HuggingFace 🤗. [View Pull Request](https://github.com/JohnSnowLabs/spark-nlp/pull/14322

• Enhanced Pooling for BERT, RoBERTa, and XLM-RoBERTa: We've added support for average pooling in BertSentenceEmbeddings, RoBertaSentenceEmbeddings, and XLMRoBertaEmbeddings annotators. This feature is especially useful when the [CLS] token is not fine-tuned for sentence embeddings via average pooling. View Pull Request

• Refined OpenAIEmbeddings: Upgraded to support escape characters to prevent JSON content issues, changed the output annotator type from DOCUMENT to SENTENCE_EMBEDDINGS (note: this affects backward compatibility), enhanced output embeddings with metadata from the document column, introduced a Python unit test class, and added a new submodule for reliable saving/loading of the annotator. View Pull Request

• New OpenVINO Notebooks: Released notebooks for exporting HuggingFace models using Optimum Intel and importing into Spark NLP. This update includes notebooks for BertEmbeddings, E5Embeddings, LLAMA2Transformer, RoBertaEmbeddings, XlmRoBertaEmbeddings, and T5Transformer. View Pull Request

:bug: Bug Fixes

• Resolved Connection Timeout Issue: Fixed the Timeout waiting for connection from pool error that occurred when downloading multiple models simultaneously. View Pull Request
• Corrected Llama-2 Decoder Position ID: Addressed an issue where the Llama-2 decoder received an incorrect next position ID. View Pull Request
• Stabilized BertForZeroShotClassification: Fixed crashes in sentence-wise pipelines by implementing a method to pad all required arrays within a batch to the same length. View Pull Request
• Updated Transformers Dependency: Resolved the import issue with keras.engine by updating the transformers version to 4.34.1. View Pull Request
• ONNX Model Version Compatibility: Fixed Unsupported model IR version: 10, max supported IR version: 9 by setting the ONNX version to onnx==1.14.0. View Pull Request
• Resolved Breeze Compatibility Issue: Addressed java.lang.NoSuchMethodError by ensuring compatibility with Spark 3.4 and updating documentation accordingly. View Pull Request
• Updated Libraries in Notebooks: Updated transformers and TensorFlow versions across all notebooks. View Pull Request
• Fixed Division by Zero Error in SwinForImageClassification Notebook: Addressed an error that occurred when updating the TensorFlow version. View Pull Request
• Fixed Missing spp File in XLMRoberta Annotator: Corrected a bug causing a missing spp file in the XLMRobertaForXXX annotator. View Pull Request
• Enhanced XLNet Embeddings Signature: Updated TensorFlow signature in XLNet embeddings source code to support custom inputs while maintaining backward compatibility. View Pull Request
• Added ModelHub Cards for M2M100 and Llama-2: Included missing modelhub cards to enhance model accessibility. View Pull Request
• Optimized Caching for Streamlit Demos: Implemented caching to enhance performance across all Streamlit demonstrations. View Pull Request
• Introduced UAEEmbeddings Notebook: Added a new example notebook for UAEEmbeddings. View Pull Request

:bug: Dependencies

• Published New OpenVINO Artifacts: Built and published new OpenVINO artifacts for both CPU and GPU to enhance performance and compatibility. View Changes
• Updated ONNX Runtime: Upgraded onnxruntime to version 1.18.0 for enhanced stability and performance on both CPU and GPU.
• Upgraded Azure Libraries: Updated azure-identity to 1.12.2 and azure-storage-blob to 12.26.0 to improve security and integration with Azure services.

:floppy_disk: Models

The complete list of all 37000+ models & pipelines in 230+ languages is available on Models Hub

:notebook: New Notebooks

• You can visit Import Transformers in Spark NLP
• You can visit Spark NLP Examples for 100+ examples

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.4.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.4.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.4.0

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.4.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.4.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.4.0</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.4.0</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.4.0</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.4.0</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.4.0.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.4.0.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.4.0.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.4.0.jar

Pull Requests:

• https://github.com/JohnSnowLabs/spark-nlp/pull/14233
• https://github.com/JohnSnowLabs/spark-nlp/pull/14255
• https://github.com/JohnSnowLabs/spark-nlp/pull/14236
• https://github.com/JohnSnowLabs/spark-nlp/pull/14200
• https://github.com/JohnSnowLabs/spark-nlp/pull/14249
• https://github.com/JohnSnowLabs/spark-nlp/pull/14253
• https://github.com/JohnSnowLabs/spark-nlp/pull/14276
• https://github.com/JohnSnowLabs/spark-nlp/pull/14238
• https://github.com/JohnSnowLabs/spark-nlp/pull/14234
• https://github.com/JohnSnowLabs/spark-nlp/pull/14232
• https://github.com/JohnSnowLabs/spark-nlp/pull/14309
• https://github.com/JohnSnowLabs/spark-nlp/pull/14308
• https://github.com/JohnSnowLabs/spark-nlp/pull/14322
• https://github.com/JohnSnowLabs/spark-nlp/pull/14313
• https://github.com/JohnSnowLabs/spark-nlp/pull/14318
• https://github.com/JohnSnowLabs/spark-nlp/pull/14319
• https://github.com/JohnSnowLabs/spark-nlp/pull/14328
• https://github.com/JohnSnowLabs/spark-nlp/pull/14333
• https://github.com/JohnSnowLabs/spark-nlp/pull/14331
• https://github.com/JohnSnowLabs/spark-nlp/pull/14330
• https://github.com/JohnSnowLabs/spark-nlp/pull/14324
• https://github.com/JohnSnowLabs/spark-nlp/pull/14334

What's Changed

• Update 2024-02-11-bge_m3_xx.md by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14244
• Closed the S3 connection by @mehmetbutgul in https://github.com/JohnSnowLabs/spark-nlp/pull/14233
• openVINO Dependencies by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14255
• [SPARKNLP-1037] Adding addFile changes to to replace broadcast in all ONNX based annotators by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14236
• Integrating OpenVINO Runtime in Spark NLP by @rajatkrishna in https://github.com/JohnSnowLabs/spark-nlp/pull/14200
• Fixing colab notebook bugs by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14249
• adding model hub cards + updating readme + small typo fix on M2M100Te… by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14253
• bert for zero shot classification crashes on sentence basis by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14276
• Sparknlp 1035 test all notebooks to import tensor flow models to spark nlp by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14238
• SPARKNLP-1036: Onnx Example notebooks by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14234
• Adding caching to streamlit demos by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14232
• Fixies by @agsfer in https://github.com/JohnSnowLabs/spark-nlp/pull/14307
• Add openvino GPU dependency by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14309
• LLAMA2 OpenVINO Position ID Fix by @rajatkrishna in https://github.com/JohnSnowLabs/spark-nlp/pull/14308
• Sparknlp 1016 implement mp net for token classification by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14322
• Uploading OpenVINO example notebooks by @rajatkrishna in https://github.com/JohnSnowLabs/spark-nlp/pull/14313
• SparkNLP - 995 Introducing MistralAI LLMs by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14318
• SparkNLP 1043 integrate new casual lm annotators to use open vino by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14319
• Fixed LLAMA generation bug by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14320
• Add Pooling Average to Broken XXXForSentenceEmbedding annotators by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14328
• Fix models link on FAQ by @dcecchini in https://github.com/JohnSnowLabs/spark-nlp/pull/14333
• adding onnx support and average pooling by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14330
• uploading UAEEmbeddings notebook by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14324
• Refactor OpenAIEmbeddings by @mehmetbutgul in https://github.com/JohnSnowLabs/spark-nlp/pull/14334
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14335
• 540 Release Candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14247

New Contributors

• @rajatkrishna made their first contribution in https://github.com/JohnSnowLabs/spark-nlp/pull/14200

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.3.3...5.4.0

:fire: New Features & Enhancements

• NEW: Introducing UAEEmbeddings for sentence embeddings using Universal AnglE Embedding, aimed at improving semantic textual similarity tasks.

UAE is a novel angle-optimized text embedding model, designed to improve semantic textual similarity tasks, which are crucial for Large Language Model (LLM) applications. By introducing angle optimization in a complex space, AnglE effectively mitigates saturation of the cosine similarity function. https://arxiv.org/pdf/2309.12871.pdf

🔥 The universal English sentence embedding WhereIsAI/UAE-Large-V1 achieves SOTA on the MTEB Leaderboard with an average score of 64.64!

• Introduce critical enhancements and optimizations to the processing of the CoNLL-U format for Dependency Parsers training, including enhanced multiword token handling and improved handling of missing uPos values
• Implement cache mechanism for metadata.json, enhancing efficiency by avoiding unnecessary downloads
• Add example notebook for DocumentCharacterTextSplitter
• Add example notebook for DeBertaForZeroShotClassification
• Add example notebooks for BGEEmbeddings and MPNetEmbeddings
• Add example notebook for MPNetForQuestionAnswering
• Add example notebook for MPNetForSequenceClassification

:bug: Bug Fixes

• Address a bug with serializing ONNX models that lack a .onnx_data file, ensuring better reliability in model serialization processes
• Delete redundant Multilingual_Translation_with_M2M100.ipynb notebook entries
• Fix Colab link for the M2M100 notebook

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.3.3

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x: (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.3

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.3

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.3.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.3.3</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.3.3</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.3.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.3.3.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.3.3.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.3.3.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.3.3.jar

What's Changed

• Uploading missing notebooks from Spark NLP v 5.1.4 by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14196
• SPARKNLP-962: UAEEmbeddings by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14199
• Cache mechanism implementation for metadata.json by @mehmetbutgul in https://github.com/JohnSnowLabs/spark-nlp/pull/14224
• [SPARKNLP-1031] Solves Dependency Parsers training issue by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14225
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14228
• release/533-release-candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14227
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14230

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.3.2...5.3.3

🐛 Bug Fixes & Enhancements

• Over 40 new interactive Streamlit demos https://github.com/JohnSnowLabs/spark-nlp/pull/14175
• Fix and add notebooks to import models from Hugging Face https://github.com/JohnSnowLabs/spark-nlp/pull/14192
• Add ONNX and TensorFlow notebooks
• Fix XlnetForSeqeunceClassification and added XlnetForTokenClassificaiton
• Rename DistilBertForZeroShotClassification
• Add missing notebooks
• Update documentation for sparknlp.start() https://github.com/JohnSnowLabs/spark-nlp/pull/14206
• Add MPNetEmbeddings to annotator https://github.com/JohnSnowLabs/spark-nlp/pull/14202
• Fix XLMRoBertaForQuestionAnswering, XLMRoBertaForTokenClassification, and XLMRoBertaForSequenceClassification: Reverted the change in tfFile naming that was causing exceptions while loading and saving the models https://github.com/JohnSnowLabs/spark-nlp/pull/14204

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.3.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x: (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.2

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.3.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.3.2</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.3.2</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.3.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.3.2.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.3.2.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.3.2.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.3.2.jar

What's Changed

• Add Spark NLP 5.3 to models.json [skip test] by @pabla in https://github.com/JohnSnowLabs/spark-nlp/pull/14208
• Sparknlp 1010 adding missing notebooks to onnx and tensor flow transformers by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14192
• Add MPNetEmbeddings to annotator by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14202
• Fix XLMRoBertaForXX : revert the change in tfFile naming by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14204
• SPARKNLP-1026: Fix documentation for sparknlp.start() by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14206
• uploading example/demos by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14175
• release/523-release-candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14211
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14212

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.3.1...5.3.2

🐛 Bug Fixes

• Fix M2M100 not working on the second run (closing ONNX session by mistake) https://github.com/JohnSnowLabs/spark-nlp/commit/75d398e5184e91e92e05f6add4e538cb4ce4ceb3
• Fix ONNX models failing in clusters like Databricks https://github.com/JohnSnowLabs/spark-nlp/commit/88774541d63d593cbc2db8be6ad0ed369b6efd93
• Fix ZeroShotNerClassification issue with NerConverter https://github.com/JohnSnowLabs/spark-nlp/pull/14186
• Adding Colab notebook for M2M100 #14191

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.3.1

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x: (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.1

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.1

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.3.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.3.1</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.3.1</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.3.1</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.3.1.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.3.1.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.3.1.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.3.1.jar

What's Changed

• [SPARKNLP-994] Solves ZeroShotNerClassification Issue by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14186
• adding colab notebook for M2M100 by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14191
• Release/531 release candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14190

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.3.0...5.3.1

🎉 Celebrating 91 Million Downloads on PyPI - A Spark NLP Milestone! 🚀

We're thrilled to announce the release of Spark NLP 5.3.0, a monumental update that brings cutting-edge advancements and enhancements to the forefront of Natural Language Processing (NLP). This release underscores our commitment to providing the NLP community with state-of-the-art tools and models, furthering our mission to democratize NLP technologies.

This release also addresses critical bug fixes, enhancing the stability and reliability of Spark NLP. Fixes include Spark NLP configuration adjustments, score calculation corrections, input validation, notebook improvements, and serialization issues.

We invite the community to explore these new features and enhancements, and we look forward to seeing the innovative applications that Spark NLP 5.3.0 will enable. 🌟

:fire: New Features & Enhancements

• Llama-2 Integration: We're introducing Llama-2 along with models fine-tuned on this architecture, marking our first foray into CasualLM annotators in ONNX. This groundbreaking addition supports quantization in INT4 and INT8 for CPUs, optimizing performance and efficiency.

<img width="1990" alt="image" src="https://github.com/JohnSnowLabs/spark-nlp/assets/5762953/660957eb-f153-492b-879c-b0a680a2dbbc">

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs. - https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/

We have made LLAMA2Transformer annotator compatible with ONNX exports and quantizations:

• 16 bit (CUDA only)
• 8 bit (CPU or CUDA)
• 4 bit (CPU or CIDA)

As always, we made this feature super easy and scalable:

doc_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("documents")

llama2 = LLAMA2Transformer \
    .pretrained() \
    .setMaxOutputLength(50) \
    .setDoSample(False) \
    .setInputCols(["documents"]) \
    .setOutputCol("generation")

We will continue improving this annotator and import more models in the future

• Multilingual Translation with M2M100: The M2M100 model sets a new benchmark for multilingual translation, supporting direct translation across 9,900 language pairs from 100 languages. This feature represents a significant leap in breaking down language barriers in global communication.

Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. However, much of this work is English-Centric by training only on data which was translated from or to English. While this is supported by large sources of training data, it does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT. We open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model. - https://arxiv.org/pdf/2010.11125.pdf

m2m100 = M2M100Transformer.pretrained() \
    .setInputCols(["documents"]) \
    .setMaxOutputLength(50) \
    .setOutputCol("generation") \
    .setSrcLang("zh") \
    .setTgtLang("en")

• Document Similarity and Retrieval: We've implemented a retrieval feature in our DocumentSimilarity annotator, offering an efficient and scalable solution for ranking documents based on similarity, ideal for retrieval-augmented generation (RAG) applications.

query = "Florence in Italy, is among the most beautiful cities in Europe."

doc_similarity_ranker = DocumentSimilarityRankerApproach()\
    .setInputCols("sentence_embeddings")\
    .setOutputCol("doc_similarity_rankings")\
    .setSimilarityMethod("brp")\ # brp for BucketedRandomProjectionLSH and mh for MinHashLSH
    .setNumberOfNeighbours(3)\
    .setVisibleDistances(True)\
    .setIdentityRanking(True)\
    .asRetriever(query)

• NEW: Introducing MPNetForSequenceClassification annotator for sequence classification tasks. This annotator is based on the MPNet architecture, enhances our capabilities in sequence classification tasks, offering more precise and context-aware processing.
• NEW: Introducing MPNetForQuestionAnswering annotator for question answering tasks. This annotator is based on the MPNet architecture, enhances our capabilities in question answering tasks, offering more precise and context-aware processing.
• NEW: Introducing a new DeBertaForZeroShotClassification annotator, leveraging the DeBERTa architecture, introduces sophisticated zero-shot classification capabilities, enabling the classification of text into predefined classes without direct example training.
• NEW: Add support for in-memory use of WordEmbeddingsModel annotator in serverless clusters. We initially introduced the in-memory feature for this annotator for users inside Kubernetes clusters without any HDFS. However, today it runs without any issue locally, on Google Colab, Kaggle, Databricks, AWS EMR, GCP, and AWS Glue.
• Add ONNX support for BertForZeroShotClassification annotator
• Introduce new Whisper Large and Distil models.
• Support new Databricks Runtimes of 14.2, 14.3, 14.2 ML, 14.3 ML, 14.2 GPU, and 14.3 GPU.
• Support new EMR versions 6.15.0 and 7.0.0.
• Add a notebook to fine-tune a BERT for Sentence Embeddings in Hugging Face and import it into Spark NLP.
• Add a notebook to import BERT for Zero-Shot classification from Hugging Face.
• Add a notebook to import DeBERTa for Zero-Shot classification from Hugging Face.
• Update EntityRuler documentation.
• Improve SBT project and resolve warnings (almost!).
• Update ONNX Runtime to 1.17.0 to enjoy the following features in upcoming releases:
  • Support for CUDA 12.1
  • Enhanced security for Linux binaries to comply with BinSkim, added Windows ARM64X source build support, removed Windows ARM32 binaries, and introduced AMD GPU packages.
  • Optimized graph inlining, added custom logger support at the session level, and introduced new logging and tracing features for session and execution provider options.
  • Added 4bit quantization support for NVIDIA GPU and ARM64.

:bug: Bug Fixes

• Fix Spark NLP Configuration to set cluster_tmp_dir on Databricks' DBFS via spark.jsl.settings.storage.cluster_tmp_dir https://github.com/JohnSnowLabs/spark-nlp/issues/14129
• Fix score calculation in RoBertaForQuestionAnswering annotator https://github.com/JohnSnowLabs/spark-nlp/pull/14147
• Fix optional input col validations https://github.com/JohnSnowLabs/spark-nlp/pull/14153
• Fix notebooks for importing DeBERTa classifiers https://github.com/JohnSnowLabs/spark-nlp/pull/14154
• Fix GPT2 deserialization over the cluster (Databricks) https://github.com/JohnSnowLabs/spark-nlp/pull/14177

ℹ️ Known Issues

• Llama-2, M2M100, and Whisper Large do not work in a cluster. We are working on how best share these large models over a cluster and will provide a fix in the future releases
• Previously some ONNX models did not work on CUDA 12.x as we have reported this problem - We have not tested this yet, but it should be resolved in onnxruntime 1.17.0 in Spark NLP 5.3.0

:floppy_disk: Models

The complete list of all 37000+ models & pipelines in 230+ languages is available on Models Hub

:notebook: New Notebooks

• You can visit Import Transformers in Spark NLP
• You can visit Spark NLP Examples for 100+ examples

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.3.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.0

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.3.0</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.3.0</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.3.0</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.3.0</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.3.0.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.3.0.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.3.0.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.3.0.jar

Pull Requests:

• https://github.com/JohnSnowLabs/spark-nlp/pull/14132
• https://github.com/JohnSnowLabs/spark-nlp/pull/14147
• https://github.com/JohnSnowLabs/spark-nlp/pull/14148
• https://github.com/JohnSnowLabs/spark-nlp/pull/14149
• https://github.com/JohnSnowLabs/spark-nlp/pull/14158
• https://github.com/JohnSnowLabs/spark-nlp/pull/14151
• https://github.com/JohnSnowLabs/spark-nlp/pull/14152
• https://github.com/JohnSnowLabs/spark-nlp/pull/14153
• https://github.com/JohnSnowLabs/spark-nlp/pull/14154
• https://github.com/JohnSnowLabs/spark-nlp/pull/14155
• https://github.com/JohnSnowLabs/spark-nlp/pull/14165
• https://github.com/JohnSnowLabs/spark-nlp/pull/14169
• https://github.com/JohnSnowLabs/spark-nlp/pull/14167
• https://github.com/JohnSnowLabs/spark-nlp/pull/14168
• https://github.com/JohnSnowLabs/spark-nlp/pull/14177
• https://github.com/JohnSnowLabs/spark-nlp/pull/14172
• https://github.com/JohnSnowLabs/spark-nlp/pull/14156

What's Changed

• Update 2023-02-08-zero_shot_ner_roberta_en.md by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14161
• [Issue#14129] Fix for spark.jsl.settings.storage.cluster_tmp_dir configuration by @jiamaozheng in https://github.com/JohnSnowLabs/spark-nlp/pull/14132
• SPARKNLP-942: MPNet Classifiers by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14147
• adding import notebook + changing default model + adding onnx support by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14158
• Sparknlp 876: Introducing LLAMA2 by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14148
• Doc sim rank as retriever by @wolliq in https://github.com/JohnSnowLabs/spark-nlp/pull/14149
• 812 implement de berta for zero shot classification annotator by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14151
• SPARKNLP-886: Add Fine tuned sentence bert notebook by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14152
• [SPARKNLP-986] Fixing optional input col validations by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14153
• [SPARKNLP-984] Fixing Deberta notebooks URIs by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14154
• SparkNLP 933: Introducing M2M100 : multilingual translation model by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14155
• SPARKNLP-985: Make Whisper compatible with onnx_data files by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14165
• Fixed a bug with models that has 'onnx_data' file not working in dbfs/hdfs by @prabod in https://github.com/JohnSnowLabs/spark-nlp/pull/14169
• [SPARKNLP-940] Adding changes to correctly copy cluster index storage… by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14167
• [SPARKNLP-988] Updating EntityRuler documentation by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14168
• SPARKNLP-1000: Fix No Operation named [init_all_tables] for GPT2 by @DevinTDHa in https://github.com/JohnSnowLabs/spark-nlp/pull/14177
• fixes python documentation by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14172
• fixed all sbt warnings by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14156
• Replace hard exception with soft logs by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14179
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14183
• release/530-release-candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14164

New Contributors

• @jiamaozheng made their first contribution in https://github.com/JohnSnowLabs/spark-nlp/pull/14132

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.2.3...5.3.0

:loudspeaker: Overview

Spark NLP 5.2.3 🚀 comes with an array of exciting features and optimizations. We're thrilled to announce support for ONNX Runtime in XLMRoBertaForTokenClassification, XLMRoBertaForSequenceClassification, and XLMRoBertaForQuestionAnswering annotators. This release also showcases a significant refinement in the use of AWS SDK in Spark NLP, shifting from aws-java-sdk-bundle to aws-java-sdk-s3, resulting in a substantial ~320MB reduction in library size and a 20% increase in startup speed, new notebooks to import external models from Hugging Face, over 400+ new LLM models, and more!

We're pleased to announce that our Models Hub now boasts 36,000+ free and truly open-source models & pipelines 🎉. Our deepest gratitude goes out to our community for their invaluable feedback, feature suggestions, and contributions.

:fire: New Features & Enhancements

• NEW: Introducing support for ONNX Runtime in XLMRoBertaForTokenClassification annotator
• NEW: Introducing support for ONNX Runtime in XLMRoBertaForSequenceClassification annotator
• NEW: Introducing support for ONNX Runtime in XLMRoBertaForQuestionAnswering annotator
• Refactored the use of AWS SDK in Spark NLP, transitioning from the aws-java-sdk-bundle to the aws-java-sdk-s3 dependency. This change has resulted in a 318MB reduction in the library's overall size and has enhanced the Spark NLP startup time by 20%. For instance, using sparknlp.start() in Google Colab is now 14 to 20 seconds faster. Special thanks to @c3-avidmych for requesting this feature.
• Add new notebooks to import DeBertaForQuestionAnswering, DebertaForSequenceClassification, and DeBertaForTokenClassification models from HuggingFace
• Add a new DocumentTokenSplitter notebook
• Add a new training NER notebook by using DeBerta Embeddings
• Add a new training text classification notebook by using INSTRUCTOR Embeddings
• Update RoBertaForTokenClassification notebook
• Update RoBertaForSequenceClassification notebook
• Update OpenAICompletion notebook with new gpt-3.5-turbo-instruct model

🐛 Bug Fixes

• Fix BGEEmbeddings not downloading in Python

ℹ️ Known Issues

• ONNX models crash when they are used in Colab's T4 GPU runtime https://github.com/JohnSnowLabs/spark-nlp/issues/14109

:notebook: New Notebooks

:book: Documentation

• Import models from TF Hub & HuggingFace
• Spark NLP Notebooks
• Models Hub with new models
• Spark NLP Articles
• Spark NLP in Action
• Spark NLP Documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs

❤️ Community support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==5.2.3

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x: (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.2.3

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.2.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.2.3

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>5.2.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>5.2.3</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>5.2.3</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>5.2.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-5.2.3.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-5.2.3.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-5.2.3.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x/3.4.x/3.5.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-5.2.3.jar

What's Changed

• HuggingFace_ONNX_in_Spark_NLP_RoBertaForSequenceClassification updated by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14122
• HuggingFace_ONNX_in_Spark_NLP_RoBertaForTokenClassification updated by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14123
• adding notebooks for onnx Deberta Import from Huggingface by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14126
• Sparknlp 967 add onnx support to xlm roberta classifiers by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14130
• adding BGEEmbeddings to resource downloader by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14133
• adding missing notebooks by @ahmedlone127 in https://github.com/JohnSnowLabs/spark-nlp/pull/14135
• Uploading and fixing example notebooks to spark-nlp by @AbdullahMubeenAnwar in https://github.com/JohnSnowLabs/spark-nlp/pull/14137
• [SPARKNLP-978] Refactoring to use aws-java-sdk-s3 library by @danilojsl in https://github.com/JohnSnowLabs/spark-nlp/pull/14136
• Models hub by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14141
• Release/523 release candidate by @maziyarpanahi in https://github.com/JohnSnowLabs/spark-nlp/pull/14140

New Contributors

• @AbdullahMubeenAnwar made their first contribution in https://github.com/JohnSnowLabs/spark-nlp/pull/14122

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/5.2.2...5.2.3