• fix: Master release bug fixes

• Handle retired Azure Maps Spatial API (CheckPointInPolygon throws on transform)
• Remove AzMapsPointInPolygonSuite test class and add fuzzing exemptions
• Update Language API version from 2022-10-01-preview to 2024-11-01
• Fix PEP 440 Python wheel version normalization (lowercase + local version)
• Fix conda environment activation on Linux (use conda run)

See: https://azure.microsoft.com/en-us/updates/v2/azure-maps-creator-services-retirement-on-30-september-2025

• test: Exclude geospatial notebooks from E2E tests

Azure Maps Spatial API was retired on 9/30/2025. The notebooks
'Quickstart - Flooding Risk.ipynb' and 'Geospatial Services.ipynb'
use CheckPointInPolygon which calls this retired API.

Exclude these from DatabricksCPUTests to prevent E2E test failures.

<a name="SynapseML v1.1.0"></a> <img width="100%" src="https://mmlspark.blob.core.windows.net/graphics/emails/email_header_synapseml.jpg" alt="SynapseML: Simple and distributed machine learning" href="https://github.com/Azure/mmlspark">

We are excited to announce the release of SynapseML v1.1 marking a host of powerful new features introduced since the initial v1.0 release. SynapseML is an open-source library that aims to streamline the development of massively scalable machine learning pipelines. It unifies several existing ML Frameworks and new Microsoft algorithms in a single, scalable API that is usable across Python, R, Scala, and Java. SynapseML is usable from any Apache Spark platform with first class enterprise support on Microsoft Fabric.

Highlights

<img width="400px" src="https://mmlspark.blob.core.windows.net/graphics/releases/Fabric.png">	<img width="400px" src="https://mmlspark.blob.core.windows.net/graphics/releases/OpenAI.png">	<img width="400px" src="https://mmlspark.blob.core.windows.net/graphics/releases/onelake.png">
Microsoft Fabric	AI Functions	OneLake
Build and operationalize distributed ML with SynapseML in Fabric	Apply Pandas and Spark LLM transformations with one line of code	Automatically derive AI insights for unstructured data in OneLake
Build Your First Model	Explore AI Functions	Learn More

<img src="https://mmlspark.blob.core.windows.net/graphics/releases/HuggingFace.png">	<img src="https://mmlspark.blob.core.windows.net/graphics/releases/AIFoundry.png">
Hugging Face	Azure AI Foundry
Use open source models hosted on Hugging Face	Run Azure AI Foundry models in your notebook
Try an Example	View Notebook

More Hightlights

Spark 3.5 Support – In this version we transitioned to Spark 3.5 as our main Spark platform.

OpenAI Ecosystem – Comprehensive improvements including global parameter defaults, GPT-4 enablement, custom endpoints/headers, GPU-accelerated embeddings with KNN, and fine-grained control over model parameters (top_p, seed, responseFormat, temperature).

ML Innovation – HuggingFaceCausalLM transformer for distributed language model evaluation, custom embedder support, and synthetic difference-in-differences causal inference module.

Platform features – Spark Native OneLake support; MSI for Azure Storage; OpenAITranslate transformer.

AI Functions in Data Wrangler on Fabric – AI Functions built into Data Wrangler in Fabric allow you to apply LLM-powered operations to your dataframe without writing a single line of code.

New Features

Documentation 📚

• AI Functions
• AI Powered Transforms in OneLake
• Azure OpenAI for Big Data in SynapseML
• AI Functions in Data Wrangler
• AI Foundry
• Hugging Face

AI Functions ⚡

• Added support for AI Functions in Pandas (#1579613, #1585011, #1596611, #1509195, #1501185, #1494610, #1494951)
• Added support for AI Functions in PySpark (#1460790, #1572928, #1599735, #1439858, #1463533)
• Added support for async AI Functions execution (#1529058, #1523727)
• JSON response support & improved language validation. (#1551823, #1566189)
• Seed param for reproducible chat/completions (API 2024-10-21). (#1551883)
• Fuzzy case-insensitive matching for Classify. (#1515064)
• Add AI Functions Operations to Data Wrangler (#1590130, #1638257, #1718967, #1725101, #1730446)

Azure OpenAI 🌸

• Enhanced Model Parameters – Added top_p, seed, responseFormat, temperature, and subscription key support (#2410, #2329, #2324)
• GPT-4 Enablement – Full GPT-4 support in OpenAIPrompt (#2248)
• Custom Endpoints & Headers – Support for custom URL endpoints and HTTP headers (#2232)
• GPU-Accelerated Embeddings – OpenAI embeddings with GPU-based KNN pipeline (#2157)
• Embedding Dimensions Control – Configurable dimensions parameter for OpenAIEmbedding (#2215)
• Global Parameter Defaults – Centralized OpenAI parameter management with Python wrapper support (#2318, #2327)
• Updated OpenAI API version to 2024 (#2190)
• Updated OpenAIDefaults implementation (#2415)
• OpenAIPrompt bug fixes and improvements (#2334)
• Added responseFormat parameter to Chat Completion (#2329)
• Optimized getOptionalParams in HasOpenAITextParams (#2315)

OneLake 🌊

• Add Spark Native OneLake support (#1190687)

Machine Learning 🕸️

• HuggingFaceCausalLM – Transformer for evaluating language models on Spark clusters (#2301)
• Custom Embedder – Extensible custom embedding transformer support (#2236)
• Synthetic DiD – Synthetic difference-in-differences module for causal inference (#2095)

Azure AI Foundry 🔨

• AIFoundryChatCompletion – New transformer for Azure AI Foundry chat models (#2398)
• AI Foundry + OpenAI Prompt – Unified interface for OpenAI and Foundry deployments (#2404)

General ✨

• Add Spark 3.5 Support – Added full Spark 3.5 compatibility with new build variants (#2052)
• Python 3.11 Baseline – Upgraded to Python 3.11 as minimum version (#2193)
• Fabric Billing Integration – Enhanced Fabric Cognitive Service token for billing support (#2291)
• Fabric WSPL FQDN Selection – Configurable Fabric workspace FQDN endpoints (#2376)
• Added Bool input support for ONNX models (#2130)
• Switched to MSI-based tokens (#2221)
• Updated Azure KeyVault task version (#2313)
• Updated build system service principals (#2181)
• Fabric & Azure Integration (#2160, #2403, #2335, #2175, #2225, #2222, #2173)

Azure AI Services & Search 🧠

• Azure Search GeoJSON Support – AzureSearchWriter now handles GeoJSON data types (#2422)
• Async Language Summarization – Cognitive Services Language Service with asynchronous summarization support (#2342)
• Auto-convert DateType/TimestampType to ISO-8601 format (#2381)
• Added support for scoring profiles in index parsing (#2383)
• Fixed model checking logic (#2379)

Additional Updates

Bug Fixes 🐞

• Early-exit behavior for async first-group errors; validated max_concurrency; add Conf getters/setters. (#1566664)
• Early exit async + missing NaN for Series. (#1595139)
• Public-preview: appropriate client init errors. (#1579646)
• Handle BadRequestError for content_filter input flag. (#1552020)
• AIRateLimitError param passthrough; clearer AIFuncRateLimitError messages. (#1545892, #1530448)
• Prevent double delete in Spark notebook test cleanup. (#1569780)
• Ensure value_counts works with numeric outputs. (#1585011)
• Fix for _ResultDtype.name type. (#1586122)
• Fix failed Scala tests. (#1395337)
• Fixed KeyError in Python environments (#2264, #2368)
• Fixed pyCodeGenImpl tag handling (#2194)
• Fixed telemetry-properties header handling (#2375)
• Fixed GPT review action (#2112)
• Updated default Anomaly Detector version to non-preview (#2280)
• Repaired failing speech tests (#2179)
• Fixed TranslatorSuite unit test break (#2111)
• Fixed LightGBM model loading when placed in Spark Pipeline with custom transformers (#2357)
• Fixed companionModelClassName generic type variable issue (#2195)
• Fixed Java class loader usage (#2135)
• Fixed LangChain crash on OpenAI versions >1.0.0 (#2307)
• Improved OpenAI prompt behavior on RAI (Responsible AI) errors (#2279)
• Enhanced error handling in networking layer (#2412)
• Fixed error handling for LangChain transformer (#2137)
• Fixed token acquisition on system context (#2378)
• Made setCustomHeaders compatible with PySpark (#2247)
• Added trailing slash normalization for URL sets (#2364)
• Switched to MWC token for tenant setting checks (#2300)
• Stopped Double Releases – Streamlined release process (#2419)

Installation & Setup 💾

• Fixed install instructions (#2136)
• Updated CONTRIBUTING.md (#2138)
• Updated Developer Setup to remove WinUtils and include ScalaTest configuration (#2244)
• Removed Spark 3.2 instructions (no longer supported)
• Fixed README install instructions

API & Service Documentation 📃

• Updated Anomaly Detector docs (#2103)
• Updated Isolation Forest documentation (#2210)
• Updated multivariate anomaly detection Fabric doc (#2191)
• Added analyze text document examples (#2127)
• Updated to use new AnalyzeText API in docs (#2126)
• Updated find_secret on Fabric documentation (#2132)
• Pointed Cognitive APIs documentation to Azure AI (#2119)
• Clarified default dataTransferMode is streaming, not bulk (#2377)
• Added audiobook paper to README
• Raised error with documentation link for find_secret (#2180)

Contributor Spotlight

We are excited to highlight the contributions of the following SynapseML contributors:

<img src="https://mmlspark.blob.core.windows.net/graphics/people/rana-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />	<img src="https://mmlspark.blob.core.windows.net/graphics/people/farrukh-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />	<img src="https://mmlspark.blob.core.windows.net/graphics/people/tom-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />
Rana Singh	Farrukh Masud	Tom Finley
Rana is the Senior Engineering Manager for SynapseML and was instrumental in improving the prompt engineering that powers AI Functions. Working alongside Tom, he helped build the feature from the ground up, ensuring high-quality and reliable AI-powered transformations. His attention to detail in refining prompts has made AI Functions more accurate and his leadership has been essential to the initiative’s success.	Farrukh is a Principal Engineer on the Code-First AI team and a prolific contributor this release. He was key in lighting up AI-powered transforms in OneLake, allowing users to apply AI transformations directly through shortcuts, dramatically expanding the reach of cognitive services across Fabric. Farrukh's contributions to Fabric integrations continue to expand the possibilities for AI-powered data workflows.	Tom is a Principal Engineer on the Code-First AI team and was pivotal in the API design for AI Functions. His thoughtful decision-making shaped AI Functions that are both powerful and intuitive. Working closely with Rana, he helped architect the feature from its earliest stages, making key choices that have shaped how users interact with AI Functions. Tom's design sensibility and technical expertise have been foundational to the feature's success.
<img src="https://mmlspark.blob.core.windows.net/graphics/people/jessica-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />	<img src="https://mmlspark.blob.core.windows.net/graphics/people/wendong-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />	<img src="https://mmlspark.blob.core.windows.net/graphics/people/samhitha-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />
Jessica Wang	Wendong Li	Samhitha Mamindla
Jessica is a Software Engineer on the SynapseML team and architected the AI Foundry integration. She has been a consistent and reliable contributor across multiple releases, building robust features and working directly with customers to understand their needs. Her integrations with AI Foundry and Hugging Face have been invaluable in helping SynapseML bridge the gap between closed and open source communities. We're excited for her continued impact on the team.	Wendong is a Software Engineer who recently joined the SynapseML team and has already emerged as a rising star. He single-handedly built multimodal AI Functions, enabling seamless transformations across text, images, and other data types. These are significant contributions for someone so new to the team, demonstrating both technical prowess and the ability to deliver complex features independently. We're eager to see what Wendong builds next.	Samhitha is a Software Engineer II who recently joined the SynapseML team and has quickly become pivotal in evaluating the quality and reliability of AI Functions. Samhitha works through logging infrastructure to ensure we can track, measure, and improve the performance of these features. Her meticulous approach to monitoring has been essential for continuously improving the user experience.
<img src="https://mmlspark.blob.core.windows.net/graphics/people/virginia-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />	<img src="https://mmlspark.blob.core.windows.net/graphics/people/elias-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />	<img src="https://mmlspark.blob.core.windows.net/graphics/people/shyam-circle.png" width="200" height="200" style="border-radius:50%; object-fit:cover;" />
Virginia Roman	Elias Yousefi	Shyam Sai
Virginia is a Senior Product Manager on the SynapseML team and leads AI Functions in Fabric. She presents and shares AI Functions with customers, gathering critical feedback that shapes the roadmap. Virginia also champions the integration of AI capabilities into Data Wrangler, making AI-powered data transformations accessible to a broader audience of data professionals. Virginia's product leadership is invaluabel, and we're lucky to have her as a collaborator.	Elias is a Senior Engineer on the Code-First AI team and is building core service infrastructure for AI Functions. He is driving the next generation of integrations with AI Functions, though many of these exciting developments are still under wraps. We look forward to sharing more of his contributions as they roll out publicly.	Shyam was a key contributor during his time with SynapseML and is now Co-Founder at Frizzle. Shyam brought AI Functions to PySpark, giving Spark users access to the same capabilities already available in Pandas and expanding the feature's reach to a broader developer community. Though Shyam has moved on to his next venture at Frizzle, his contributions and presence will be missed by the SynapseML team.

Acknowledgements ❤️

We would like to acknowledge the developers and contributors, both internal and external, who helped create this version of SynapseML

Mark Hamilton @mhamilton723	Markus Weimer	Avrilia Floratou
Eric Dettinger @sandshadow	Virginia Roman	Eren Orbey @orbey
Rana Singh @ranadeepsingh	Farrukh Masud @FarrukhMasud	Tom Finley @TomFinley
Brendan Walsh @BrendanWalsh	Jessica Wang @JessicaXYWang	Wendong Li @levscaut
Elias Yousefi @elyousef	Shyam Sai @sss04	Shirley Huang
Chongyu Wang	Ruoyu Jing	Bo Zhou
Paul Wang @pwang347	Kyle Cutler @kycutler	Dan Vilnoiu
Scott Votaw @svotaw	Mark Niehaus @niehaus59	Aydan Aksoylar @aydan-at-microsoft
Sheryl Zhao @sherylZhaoCode	Markus Cozowicz @eisber	Sailesh Baidya @saileshbaidya
Keerthi Yanda @KeerthiYandaOS	Kyle Rush @k-rush	Aadharsh Kannan @AKannanMSFT
Serena Ruan @serena-ruan	Cruise Li @mslhrotk @lhrotk	Jason Wang @memoryz
Haizhou (Dylan) Wang @dylanw-oss	Sarah Shy @sarahshy	Kashyap Patel @ms-kashyap
Puneet Pruthi @ppruthi	Ilya Matiach @imatiach-msft	Amir Jafari @amhjf
Nellie Gustafsson	Bogdan Crivat	Justyna Lucznik @juluczni
Richard Wydrowski @richwyd	Tania Arya @taniaarya	Adithya Mukund @adithyamukund
Roman Batoukov @RomanBat	Alexandra Savelieva @alsavelv	Jessica Wolk @msplants
Luis França @luisffranca	Paul Koch @paulbkoch	Rich Caruana
Martha Laguna @martthalch @marthalc	Jeff Zheng	Sicong Yang
Peixian Gong	Ruixin Xu	Chris Hoder
Derek Legenzoff	Misha Desai	Beverly Kodhek
Louise Han @jr-MS	Raj Rikhy	Brice Chung
Marcos Campos	Mike Estee	Kim Manis
Mitrabhanu Mohanty	Anand Raman	Sudarshan Raghunathan @drdarshan
William T. Freeman	Gregory B. Newby	John Moyer
Vidip Acharya	Ashit Gosalia	Miguel Fierro @miguelgfierro
Ismaël Mejía @iemejia	Kartavya Neema @kartavyaneema	Daniel Ciborowski @dciborow
Mark Tabladillo @marktab	Guilherme Beltramini @gcbeltramini	Akshaya Annavajhala (AK)
James Verbus @jverbus	Mopé Akande @msakande	Ikko Eltociear Ashimine @eltociear
Alexander Spiridonov @vonodiripsa	Hiroshi Yoshioka @hyoshioka0128	Frank Solomon @fbsolo-ms1
Leonard Herold @LeonardHd	David Smith @dsmith111	Denniz Svens @DennizSvens
João Moura @operte	Sean Marihugh	ONNX Team
Azure Global	Vowpal Wabbit Team	LightGBM Team
MSFT Garage Team	MSR Outreach Team	Speech SDK Team
MLflow Team	Azure Docs Team

Learn More

<img src="https://mmlspark.blob.core.windows.net/graphics/releases/SynapseMLSite.png">	<img src="https://mmlspark.blob.core.windows.net/graphics/releases/AIPoweredAnalytics.png">	<img src="https://mmlspark.blob.core.windows.net/graphics/releases/Introduction.png">
Visit our website for the latest docs, demos, and examples	See how to use AI Functions in Fabric Notebooks	Learn the basics of SynapseML
<img src="https://mmlspark.blob.core.windows.net/graphics/releases/FabConThumb.png">	<img src="https://mmlspark.blob.core.windows.net/graphics/releases/AIFunctionsDataWrangler.png">	<img src="https://mmlspark.blob.core.windows.net/graphics/releases/bookbot.png">
Read our highlights from Fabric Community Conference	Try Out AI Functions in Data Wrangler	Read our Latest Paper on Large Scale Audiobook Generation

Full Changelog: https://github.com/microsoft/SynapseML/compare/v1.0.0...v1.1.0

Changes:

• 123ead27e4ce8d60f24b7862f617c98c79589e14 chore: bump to v1.0.14 (#2418)

• 1ceb3b1ea46f584c0e01fbf6c1610ecfda49ebd1 chore: update OpenAIDefaults (#2415)

• 26220da27855026f20b3f7f622e988d16208237b feat: add aifoundry to openai prompt (#2404)

• aac2ed6b56d7b73b48d1793bf563fcfb98da325e fix: fix error handling in networking layer (#2412)

• 7c2f22a0a3761fc7c2c91d11f5a532a52385b248 feat: Add top_p and seed params to OpenAIDefaults (#2410)

• a9133aa3293bde93675847d98ff59796183a7c23 noop commit to Stop library releases in build

• a394061eade46b7ba8374a9140fdae9f5412d77c chore: Bump library to version to v1.0.13 (#2405)

• 85a66877da13de36b2ff324322c77bb25b99b5ed chore: fix propagation of fabric telemetry (#2403)

• 298c7ed29c3fec7b8daf4be190295f466e9bb5ee chore: Add devcontainer configuration for vscode and Copilot (#2400)

• 873884d4e4bf86ddda86489575970803980c76e9 feat: Add AIFoundaryChatCompletion (#2398) <details><summary><b>See More</b></summary>

• 83ebb5a868591f493a25b27ebb544828a98fd93f Update README.md

This list of changes was auto generated.</details>

Changes:

• a394061eade46b7ba8374a9140fdae9f5412d77c chore: Bump library to version to v1.0.13 (#2405)
• 85a66877da13de36b2ff324322c77bb25b99b5ed chore: fix propagation of fabric telemetry (#2403)
• 298c7ed29c3fec7b8daf4be190295f466e9bb5ee chore: Add devcontainer configuration for vscode and Copilot (#2400)
• 873884d4e4bf86ddda86489575970803980c76e9 feat: Add AIFoundaryChatCompletion (#2398)
• 83ebb5a868591f493a25b27ebb544828a98fd93f Update README.md

This list of changes was auto generated.

Changes:

• 9c911488b74340d022f919ce14a946c8ac75b0b4 chore: bump to v1.0.12 (#2397)

• e71fed83706792005415dcbe7693b8b8bcad6f6a chore: fix vw benchmarks (#2396)

• 2b23d7c3954a9240fc0a94f9861956832bac657c chore: split deepLearning tests (#2387)

• 1639b14db9fc4df2f3b9c4b088739c060a53c562 fix: testing telemetry-properties header (#2375)

• 9e0fde448d2600e6383be008720d11fab6fb1658 chore: Fixing logic for polling in LRO tasks (#2389)

• b672aa4d5c3395867c39602c240ab56b3eb6c26b chore: Updating codecov pipeline step (#2391)

• 84d7d6537259daf0a05388cd4126fef0c2d960ec feat: Support Choosing Fabric WSPL FQDN (#2376)

• ffa03839eecf3bcc97b47938c89a2cdcb56b2811 docs: default dataTransferMode is streaming, not bulk (#2377)

• 04d9dc425888e61cf588d7e0a019c52cfd2e865a fix: auto-convert DateType/TimestampType to ISO8601 in Azure Search (#2381)

• 5bbbbc7fe08b8a2699a913e8b49252ff32b09e11 chore: fix failing openai tests (#2385) <details><summary><b>See More</b></summary>

• 9af855ece5c15f65042247b4a7f24d58613a1042 fix: support scoring profiles in Azure Search index parsing (#2383)

• 326988de73cfc0355a7cb5bd6c472309fdcc0007 chore: update sbt version to allow for amd local builds (#2384)

• 3865e7146a70f7d9d014ecf3ee02acbb1038dc1f fix: fix model checking logic (#2379)

• 1eec70d7b0833545f992b63937307367a6bcee17 fix: fix bug where token cannot be acquired on system context (#2378)

• 141039b4510797d089106209966516cf48283471 fix: add hf causal LM python tests, fix build (#2374)

• 6c95bf0339140e7a5f6fe56131c063f8da680f74 fix: add case for Python only envs (#2368)

• c1cef65bfe7de5d19c56e666652384f71e68ab85 chore: limit adb concurrency (#2370)

This list of changes was auto generated.</details>

Changes:

• 58b945f1fefefa5f46ea437e30a5b31816c27225 chore: fix yml (#2373)

• 2ec8db408d2f773b18b4feac24670d814da5af63 chore:fix release (#2372)

• 33b5a394d8bffa3d8a433fd1b847d1a9a2b29d66 chore: bump to v1.0.11 (#2371)

• e12b0a1cab91f9ecb90f0f8935b3f7116926bd8a fix: Adding trailing / to URLs set (#2364)

• 53b76a826e680a0931cc8207500a86b4df5987a7 feat: Add HuggingFaceCausalLM Transformer for evaluating language models on cluster (#2301)

• 25b49cd8a6df54733b12082c0203e9c48f04d5a4 chore: free up space in build (#2365)

• 26baa095d847d9fb853b2de94e558be9157c730b fix: Cannot Load LightGBM Model When Placed in a Spark Pipeline with Custom Transformers (#2357)

• b7971eb183b684bd799aeb4fc11e98ec37ee2c9e chore: Making unit tests green again (#2360)

• 57d15ecefa56cb7360083b7de8361691bd676da3 chore: Removing unnecessary logs (#2359)

• 6c9c9ce4da52596dd253553af222afd5f8f17520 Add correct workspace ID for spark jobs (#2358) <details><summary><b>See More</b></summary>

• a4dec0898dd71994d5f538cee518ad88c520ef89 fix: update openai compeletion doc, fix failed OpenAI test (#2348) [ #2351, #2353, #2354 ]

• 7878fec6b505beba2e6be7858e98648b653753d1 chore: streamline cache fix (#2354)

• 8787c829c00aabfa2bde05a9e462012afdf2d683 chore: free up space on build machines prior to caching (#2353)

• b2f4080f617b2817f2270899218a47a6be1198d0 Make sure library isnt released twice

• 0240895a1aa9713dfc18488c844842afab501c2d chore: bump synapseml to v1.0.10 (#2351)

• 1b5df70393b43a61129135426732c87251211c00 feat: Adding capability use Cognitive Service Language Service asynchronously for Summarization (#2342)

• bab6aed67d65d4f0a7960df0a75b2e5f321cdda4 chore: fix github release yml (#2339)

This list of changes was auto generated.</details>

SynapseML v1.0.9

Features

• SubscriptionKey and Temperature: Added parameters for enhanced configurability. ([#2324], authored by sss04)
• Global Parameters: Introduced global parameters for broader flexibility. ([#2318], authored by sss04)

Bugfixes

• Certified Events: Fixed certified events emitting issue. ([#2335], authored by mhamilton723)
• OpenAIPrompt: Fixed various bugs. ([#2334], authored by sss04)
• LangChain Error Handling: Resolved crashes with openai > 1.0.0. ([#2307], authored by lhrotk and mslhrotk)

Build

• SynapseML: Bumped to version 1.0.9. ([#2338], authored by mhamilton723)
• Sklearn Datasets: Removed dependency. ([#2337], authored by mhamilton723)
• NLTK Version: Pinned an older version of nltk. ([#2309], authored by FarrukhMasud and FMasudMsft)

Fixes

• Azure Maps: Fixed failing tests. ([#2325], authored by mhamilton723)

Improvements

• OpenAIDefaults: Wrapped for Python for easier usage. ([#2327], authored by sss04)
• Response Format: Added responseFormat parameter in OpenAI Chat Completion Request. ([#2329], authored by FarrukhMasud and FMasudMsft)
• Post-Processing: Improved settings for better intelligence. ([#2325], authored by FarrukhMasud and FMasudMsft)
• Optional Params Optimization: Optimized getOptionalParams in HasOpenAITextParams. ([#2315], authored by FarrukhMasud and FMasudMsft)

Documentation

• v1.0.8 Release: Published updated documentation. ([#2308], authored by mhamilton723)

Maintenance

• Azure Key Vault: Updated task version. ([#2313], authored by FarrukhMasud and FMasudMsft)
• Failing Test: Disabled a failing test. ([#2311], authored by FarrukhMasud and FMasudMsft)

<a name="v1.0.7"></a>

v1.0.7

Maintenance 🔧

• bump synapseml to 1.0.7 (#2298)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• bd5b4013a14eed84d4ecf6b87ec6f6bc66925eaa chore: bump synapseml to 1.0.7 (#2298)
• 9905ae80a2805e37aeeb2a0c1e811caf4a38b557 chore: bump synapseml to 1.0.6 (#2295)
• 8e1c6f881daca538913a966f7aa02bdf3959822a feat: Change Fabric Cog Service Token to Support Billing (#2291)
• 6854b5f475db6fb6cb52bbe93cd700ee1ca612e8 chore: update pipeline branches
• c52758c55970b52ef0215ff130662eebdecc395c Update pipeline.yaml
• f3953bc403ce25470b38ec2530581c1be5ff7789 fix: update default anomaly detector version to compatible non-preview (#2280)
• 478e8a24812195d1cef32035ee865dc72e7d7bf9 fix: fix openai prompt behavior on RAI errors (#2279)

This list of changes was auto generated.

<a name="v1.0.6"></a>

v1.0.6

Bug Fixes 🐞

• update default anomaly detector version to compatible non-preview (#2280)
• fix openai prompt behavior on RAI errors (#2279)

Features 🌈

• Change Fabric Cog Service Token to Support Billing (#2291)

Maintenance 🔧

• bump synapseml to 1.0.6 (#2295)
• update pipeline branches

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• 9905ae80a2805e37aeeb2a0c1e811caf4a38b557 chore: bump synapseml to 1.0.6 (#2295)
• 8e1c6f881daca538913a966f7aa02bdf3959822a feat: Change Fabric Cog Service Token to Support Billing (#2291)
• 6854b5f475db6fb6cb52bbe93cd700ee1ca612e8 chore: update pipeline branches
• c52758c55970b52ef0215ff130662eebdecc395c Update pipeline.yaml
• f3953bc403ce25470b38ec2530581c1be5ff7789 fix: update default anomaly detector version to compatible non-preview (#2280)
• 478e8a24812195d1cef32035ee865dc72e7d7bf9 fix: fix openai prompt behavior on RAI errors (#2279)

This list of changes was auto generated.

<a name="v1.0.5"></a>

v1.0.5

Bug Fixes 🐞

• tiny bug fix to avoid KeyError (#2264)
• Make setCustomHeaders compatible with Pyspark (#2247)

Build 🏭

• bump braces from 3.0.2 to 3.0.3 in /website (#2239)
• bump ws from 7.5.9 to 7.5.10 in /website (#2241)

Continuous Integration 💜

• set pull-requests read permission (#2217)

Documentation 📘

• Update Developer Setup to remove WinUtils step and include ScalaTest Configuration update. (#2244)
• update isolation forest doc (#2210)
• update multivariate anomaly detection - fabric doc edit (#2191)

Features 🌈

• Add custom embedder (#2236)
• Enable GPT-4 in OpenAIPrompt (#2248)
• Adding Custom Url Endpoints and Headers (#2232)

Maintenance 🔧

• bump synapseml version to v1.0.5 (#2273)
• make debugging getterSetter tests easier (#2272)
• fix typo in conda env stage name
• fix conda caching in builds (#2261)
• improve test reliability (#2260)
• Allow logging without inheritance (#2257)
• Add feature name to log_verb (#2255)
• give tests more time (#2254)
• improve local python development (#2252)
• fix remaining build errors (#2250)
• fix databricks tests and MAD test errors (#2249)
• fix synapse tests and forms (#2245)
• fix errors in build pipeline (#2243)
• stop conda from using --force (#2238)
• update generate fabric doc (#2214)
• update resource location (#2225)
• update esrp names
• remove names from ESRP install
• switch to msi based tokens (#2221)
• fix clean acr (#2222)
• fix esrp to use new auth (#2218)
• add support for dimensions parameter to OpenAIEmbedding (#2215)
• remove hack for materializing keys (#2211)
• break up build steps to improve stability (#2208)
• add conda env caching (#2207)
• remove storage keys and dotnet (#2205)
• fix synapse notebook demo (#2204)
• add more build retries (#2203)
• add more retries to build steps (#2201)
• fix website version list

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• 392f60112bdd965cab3190d57fbfcdb31083f7c2 chore: bump synapseml version to v1.0.5 (#2273)

• 5c9d30407726be0fc9710a50c61f2af609471b31 chore: make debugging getterSetter tests easier (#2272)

• eb309feccb2179896335ded54baa3c33efc86665 fix: tiny bug fix to avoid KeyError (#2264)

• 72f99c852523bde612bd9a1b77d12862f66c52ac Update pipeline.yaml

• 8a631711f01c7c202d0b8708bafb2d5ec9bdd007 Update conda.yml

• c7ebfb290da6fc2ea664c53d2c06568698b7f438 Update conda.yml

• 8dc849773d0138bc4c5142dd6ac58d221ec675d5 chore: fix typo in conda env stage name

• 7956d93df4404fa3855e2137e6137f7357e4f2a9 chore: fix conda caching in builds (#2261)

• 84ed59d859b70977fa10b61a1cce98fc661dc0dd chore: improve test reliability (#2260)

• e3c878425ab00e6f5fbaa9b8c0ddfe72b61161ff chore: Allow logging without inheritance (#2257) <details><summary><b>See More</b></summary>

• 7c23d835d713d5e02798cd6a8b49db6389355541 Enable custom headers in OpenAIPrompt (#2259)

• 6da5f57401d7032ebd4fdccdba3e3ca5fbac6c83 feat: Add custom embedder (#2236)

• 12d8113711231ea45b49189ce260fdfc3a59c718 chore: Add feature name to log_verb (#2255)

• 46f16e085ff8866bcbc7b6615709e8d903bd2e0c chore: give tests more time (#2254)

• d453ba2a8c0e4fa83d5387ac0a7d7025abbe84bd build: bump braces from 3.0.2 to 3.0.3 in /website (#2239)

• 5a191b591f7e5c61b4b1fe76bd4c98bef25e4c46 chore: improve local python development (#2252)

• 8fb3e0ae93b5881b9d0e0479f0c91545f04c2eb6 build: bump ws from 7.5.9 to 7.5.10 in /website (#2241)

• 33180ef9a93f0da31a1b8fbdb6b892135665e79e chore: fix remaining build errors (#2250)

• a7029851ee51d1bf7534c764e5552b7638d8480b chore: fix databricks tests and MAD test errors (#2249)

• c033077a3a20f5779fd7efc7b2beab98937efb34 feat: Enable GPT-4 in OpenAIPrompt (#2248)

• b19b991b4fd76e5588b2e4902a7d8909c7778977 chore: fix synapse tests and forms (#2245)

• a5df69c8a7aaf7af48aaff5d7a34265d2b3472b3 fix: Make setCustomHeaders compatible with Pyspark (#2247)

• c94d84a144494fb61952d348bcd8a3614d7316a3 feat: Adding Custom Url Endpoints and Headers (#2232)

• 4c7f11ffa76e91318a471a85cf26b958292c222d docs: Update Developer Setup to remove WinUtils step and include ScalaTest Configuration update. (#2244)

• 5b2746b0901c802eb10dd5045765eb9595b18fac chore: fix errors in build pipeline (#2243)

• 440f18e8ad5dbeef7045eeb9e8fa1b38d531ce9f fix platform detection logic (#2234)

• 91e552c1bc4f38bd286724c9a9cc7ecbdc8d2cd0 chore: stop conda from using --force (#2238)

• a8820fee67975f14dbaf5ec2df94645028cfa8b1 chore: update generate fabric doc (#2214)

• 5c3734280dd364743ab47f61fcb54914e35951b5 chore: update resource location (#2225)

• a31b3f0d844a4fa7c5644d76198a2df2672d9962 chore: update esrp names

• 04897e059502d9c705894f51d190599df4ea65ca chore: remove names from ESRP install

• 7b54e8900be2156b410704a8b2e31971b9468835 chore: switch to msi based tokens (#2221)

• a72b94bc62b88ea5833d6e9fe1663a68027962f7 Update clean-acr.yml

• 80a5cb84811cba01f074261152eff0f58dbfe97b chore: fix clean acr (#2222)

• 42371e444bcc5bdf5f8010c256e8d177a13c52fd Added files for onebranch builds

• 93829b1708e3480b041e7fa2729bc7f1a0608675 Update clean-acr.yml

• d234d79ab2bec95095f1743b453c036e38d58f1c chore: fix esrp to use new auth (#2218)

• d0a21610c58693d158fcc853432a4283472f0ec6 chore: add support for dimensions parameter to OpenAIEmbedding (#2215)

• edb8c5406c4340a4466f01f7372c9b891549dcc7 ci: set pull-requests read permission (#2217)

• e10a91e192fcdad430174e9cfc6e10e7135e8f9d Update OpenAI.ipynb (Typo "Azure Open AI"→"Azure OpenAI") (#2216)

• 1ed7dd22fd0837061ed6835b03813e11dc013a64 chore: remove hack for materializing keys (#2211)

• 786e59aee653a963e7cf7284420142ba7238ddc2 docs: update isolation forest doc (#2210)

• 44873989ac217a5816ad599749942a8dddcf1c07 chore: break up build steps to improve stability (#2208)

• 07b2451e3b3ad123f5708911e359bdedae5602fd chore: add conda env caching (#2207)

• d789b02b9edfd095a1b0af5ba9177d06e195b1da chore: remove storage keys and dotnet (#2205)

• 139b13bda19b7c364c5cc076123690f4aa054adb chore: fix synapse notebook demo (#2204)

• 1c6970213524510966cf373ac51528a169c558c0 chore: add more build retries (#2203)

• 605faa40229220c5cfb9b58a7b0e988ca90b5917 chore: add more retries to build steps (#2201)

• 23fe098916823a381124d021b0307ca1de937a56 docs: update multivariate anomaly detection - fabric doc edit (#2191)

• 5adc9ba2daf177e312febef791d8b05b170d9df2 chore: fix website version list

This list of changes was auto generated.</details>

<a name="v1.0.4"></a>

v1.0.4

Bug Fixes 🐞

• companionModelClassName no longer returns generic type variable (#2195)
• Fix tag for pyCodeGenImpl (#2194)

Build 🏭

• bump azure/login from 1 to 2 (#2176)
• bump azure/CLI from 1 to 2 (#2178)

Maintenance 🔧

• Bump version to 1.0.4 (#2200)
• fix flaky HyperOpt NB (#2198)
• Bump python version to 3.11 (#2193)
• exclude non-executable docs from automated tests (#2197)
• add retry logic to build steps (#2192)
• update openai api version to 2024 (#2190)
• fix 1.0.1 version tags (#2189)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.

<a name="v1.0.3"></a>

[v1.0.3]

Bug Fixes 🐞

• repair failing speech tests (#2179)
• update openai completion (#2142)

Build 🏭

• bump github/codeql-action from 2 to 3 (#2148)
• bump actions/setup-python from 4 to 5 (#2146)
• bump peter-evans/create-or-update-comment from 3 to 4 (#2162)
• bump actions/upload-artifact from 3.1.3 to 4.3.1 (#2165)

Features 🌈

• OpenAI embeddings with GPU based KNN (#2157)
• Synthetic difference in differences (#2095)

Maintenance 🔧

• Update to version 1.0.3 (#2183)
• update build system service principals (#2181)
• raise error with documentation link - find_secret (#2180)
• check Fabric Tenant (#2175)
• rotate outdates SAS url in speech tests (#2173)
• Support Token Provider Mode (#2160)
• Bump isolation forest to 3.0.4 (#2168)
• Add script to generate pypi mfa qr (#2150)
• Add Unified Logging Base Class for Python (#2159)
• fix failing tests (#2153)

<a name="SynapseML v1.0.2"></a>

[SynapseML v1.0.2]

Bug Fixes 🐞

• Add the error handling for Langchain transformer (#2137)
• use java class loader (#2135)
• Support to Bool input for Onnx models (#2130)

Build 🏭

• bump amannn/action-semantic-pull-request from 5.3.0 to 5.4.0 (#2125)

Doc

• update find_secret on Fabric and doc (#2132)

Documentation 📘

• update CONTRIBUTING.md (#2138)
• fix install instructions (#2136)
• fix readme install
• add audiobook paper to README
• add analyze text document (#2127)
• use the new AnalyzeText API in docs(#2126)
• removing spark 3.2 instructions

Maintenance 🔧

• bump to v1.0.2 (#2140)
• change udf vec2array to pyspark.ml.functions.vector_to_array (#2131)
• fix failing notebooks (#2134)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• 522661ae3834f3a54c0ad746350b225663c51d41 chore: bump to v1.0.2 (#2140)

• 2a01c8e68ae58f281eab2afc5a3f69aa28dd2fc7 doc: update find_secret on Fabric and doc (#2132)

• 23222c08403bcc067c402b95f36e9da89e62b94a fix: Add the error handling for Langchain transformer (#2137)

• f3ae1465f5564afe69cf6697ac4e98937a9e0ed4 fix: use java class loader (#2135)

• fc3a9992675ff42e5d2a45566abce692ed3fd9b9 docs: update CONTRIBUTING.md (#2138)

• 9b20829010ff2818b623e4fb06aa7481f82ab2f9 docs: fix install instructions (#2136)

• c10f46ea3d7ede110d219806428932b486a8bbcc docs: fix readme install

• 28cd6db0f02c85a20dabf461e3f7a333de900b8a chore: change udf vec2array to pyspark.ml.functions.vector_to_array (#2131)

• 46a1ef816aa12292ad101ef16296bdd5aded557a docs: add audiobook paper to README

• 5e9bae1c442d5f9ea78274b219e18d690f5fe12f build: bump amannn/action-semantic-pull-request from 5.3.0 to 5.4.0 (#2125) <details><summary><b>See More</b></summary>

• 241062fac15ea96815d597f38acbc03984ef185c docs: add analyze text document (#2127)

• 4623219956d4629b74bf76f7b382252d30dbd187 review docs (#2128)

• 9195deef8b3c260983934010bc7f60efa93e6817 fix: Support to Bool input for Onnx models (#2130)

• 4c4fc8aa5d9e080ee13b1026a000edb15a4d6485 chore: fix failing notebooks (#2134)

• 90ded807fc28f8b6b6cdf25250b908543432c61e docs: use the new AnalyzeText API in docs(#2126)

• 5cd78c9f610bc14a429f49e8bfe32ca72e5cfe37 Improve LightGBM Network logs (#2124)

• a187cd063e4c7e6ac7f913187d41c373d66ab5f2 docs: removing spark 3.2 instructions

This list of changes was auto generated.</details>

<a name="SynapseML v1.0.1"></a>

v1.0.1

Documentation 📘

• pointing cognitive apis to azure ai (#2119)
• bump readme to spark 3.4

Maintenance 🔧

• bump to v1.0.1 (#2123)
• add back in exclusions (#2122)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• cb4fd82835e6193ac4c4283f21faa1ed4e69660c chore: bump to v1.0.1 (#2123)

• 91e8c8525df06110345ea774fc4417812af4ec49 chore: add back in exclusions (#2122)

• d240cbb1f4a6916fb46a622d3a33089e9001dae1 docs: pointing cognitive apis to azure ai (#2119)

• 77be64100870889b563c759a079d86c6bca23ce1 docs: bump readme to spark 3.4

• ef435a2917bc383a251e574c6d88cf909b1336e3 chore: bump to v1.0.0 (#2120)

• c2fdb05f44d6c705c954dc80e2c7c0f33b96a71b chore: Adding Spark34 support (#2052) (#2116)

• 903dc6b94e5ae617b995d94490dfafc8ff2ca4aa docs: move cognitive namespace to services namespace (#2118)

• fd00b8700441ef47205950b72c7bcbe84b0f5b36 chore: refactor cognitive package to services (#2117)

• b0caf2e5ff920094f2d7f80bc2dd8145009c4863 build: bump @babel/traverse from 7.18.9 to 7.23.2 in /website (#2098)

• c12afc51b0b68c8a3aa7188955b3795dcfc0a1c8 chore: bump speech sdk version (#2107) <details><summary><b>See More</b></summary>

• 1af71ed4d40ca52e14774623651c6fc2c784615f docs: update anomaly detector docs (#2103)

• 377df2f57d485f91bdef14139c658c14003c3576 build: bump ossf/scorecard-action from 2.3.0 to 2.3.1 (#2108)

• cd43ee7c73268a0545c612599fb598398923d0d7 fix: unit test break in TranslatorSuite (#2111)

• cc77eda925ceeda0de4354daa7b3624a6a26f84a chore: removing gpt-review (#2113)

• 70dc523114768eea12ff0648c03fcfc3785f69de fix: gpt-review action (#2112)

This list of changes was auto generated.</details>

<a name="SynapseML v1.0.0"></a>

<img width="100%" src="https://mmlspark.blob.core.windows.net/graphics/emails/email_header_synapseml.jpg" alt="SynapseML: Simple and distributed machine learning" href="https://github.com/Azure/mmlspark">

We are excited to announce the release and general availability of SynapseML v1.0 following seven years of continuous development. SynapseML is an open-source library that aims to streamline the development of massively scalable machine learning pipelines. It unifies several existing ML Frameworks and new Microsoft algorithms in a single, scalable API that is usable across Python, R, Scala, and Java. SynapseML is usable from any Apache Spark platform and is now generally available with enterprise support on Microsoft Fabric.

Highlights

<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/langchain.jpg">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/AzureCogSearch.svg">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/powerbi.svg">
Distributed Langchain	Vector Search Indices	Semantic Link
Deploy your LLM apps on millions of documents	Quickly create semantic and multi-modal search engines	Work with PowerBI datasets natively from Microsoft Fabric
View Notebook	Try an Example	Learn More

<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/azure_ai_services2.svg">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/EconML-Logo-MSFT-colorXL.png">
Keyless AI Services	Orthogonal Forests
Use built-in AI services without keys in Microsoft Fabric	Discover and measure heterogeneous causal effects
Learn More	Try an Example

New Features

General ✨

• Add support for spark 3.4.1 (#2052) (#2116)
• Enterprise support on Microsoft Fabric

Open AI and Langchain 🦜

• Add the LangchainTransformer for orchestrating LLMs at scale (#1925, #2036)
• Add ChatGPT through the OpenAIChatCompletion transformer (#1887)
• Add Langchain notebook (#2002, #2013)
• Add OpenAI document Q+A notebook (#2029, #2033)
• Add custom chatbot creation to form recognition demo (#1888)

Azure AI Services 🧠

• Add Support for Azure Cognitive Search Vector Indices (#2041)
• Add keyless Azure AI services on Microsoft Fabric (#2070, #1859)
• Support new form recognizer APIs (#1882)
• Support streaming multivariate anomaly detection (#1893)
• Add prerequisites page for setting up OpenAI and Azure AI services (#2008)

Deep Learning 🕸

• ONNX models support variable size inputs (#1851)
• Add distributed training overview (#1879)

Causal Learning 📈

• Add OrthogonalForestDML for causal learning with heterogeneous effects (#1873)
• Add Heterogeneous Effect Quickstart
• Support custom reference distribution in DistributionBalanceMeasures to detect data drift (#1885)
• Add statistical significance reporting for causal learners using getPValue (#1863)

LightGBM 🌳

• LightGBM streaming mode is now default (#2088)
• LightGBM supports passing Reference datasets to speed repeated execution (#1977)
• Add LightGBM streaming mode docs (#1992)

Additional Updates

Bug Fixes 🐞

• Improve LGBM exception and logging (#2037)
• AI Services and other HTTP Clients no longer retry 4XX codes other than 429 (#2005)
• Make geospatial services robust to 404s thrown by the service (#2007)
• Fix bug #1869, where DoubleML .setFitIntercept should default to true (#1876)
• Fix Multivariate Anomaly error handling (#1991)
• Fix import error when using AI services on Azure Machine Learning clusters (#1951)
• Fix default values of aadToken & url on Fabric (#1918)
• Fix ONNX model shape inference on batches with shape [-1] (#1906)
• Add getPValue to python API of DoubleML (#1909)
• Add diagnosticsInfo in Multivariate Anomaly detection response (#1892)
• Fix Double ML timeout on large datasets (#1903)
• Retry OnnxHub calls to improve test reliability (#1889)
• Remove case matching for erased generic types (#1880)
• Remove extraneous Foo type from Python codegen (#1867)
• Update OpenAIEmbedding Schema to account for internalServiceType
• Update Maven package to include correct GitHub path (#2073)

Documentation 📘

• Automatically create Azure docs from notebooks (#1976, #1911,#2023, #2021, #2043)
• Improve OpenAI Docs (#1938, #1937, #1999)
• Improved LightGBM docs (#2003)
• Improve Vowpal Wabbit Docs (#1971, #1972, #1970, #1969, #1968, #2072)
• General notebook quality improvements (#1979, #1932)
• Improve Causal learning docs (#1905)
• Remove old notebooks and demos (#1934)
• Fix R-setup.md docs (#1946)
• Fix broken links across website and repo (#2079, #2076, #2042, #2032, #2027, #2026, #2025,#2022, #1864, #2035, #2049)
• Fix Recommenders repo URL (#2086)
• Fix website developer API link (#1877)
• Migrate from Cognitive Services to Azure AI services (#2119, #2118)
• Update anomaly detector docs due to service deprecation (#2103)
• Fix docker link (#2019)
• Fix installation instructions (#2000, #1961, #1921)

Maintenance 🔧

• Rename Cognitive Services to AI Services and move cognitive.* APIs to services.* (#2117)
• Upgrade the Azure AI Speech sdk version to fix proxy issues and segfaults (#2107)
• Improve telemetry and logging across library (#2047, #2099, #2097, #2045, #1917, #2109)
• Onboard to ESRP release process (#2083)
• Publish binaries to ADO Feeds (#1995)
• Upload example notebooks to storage account on every build (#2001)
• Allow publishing of custom versions (#1998)
• Add fabric to the find_secret API (#1948)
• Ensure nightly build runs every night
• Scrub Shared Access Signatures from logs (#1939)
• Clean azure search indexes during tests (#1901)
• Maintain tests (#2122, #1994, #2092, #2077, #2071, #1982, #1981, #1927, #1959, #1861, #1896, #2111)
• Maintain Build System (#2024, #1983, #1963), #1944, #1916, #1915, #1866, #1949, #1908, #1984, #1954, #1954, #1904, #2102)
• Added and removed GPT PR Reviews (#2113, #1957, #2112, #2069)
• Add .trunk to .gitignore (#2078)
• Add .bloop to .gitignore (#1897)
• Bump SynapseML Versions (#2123),#2120,#2110,#2085,#2084,#2011,#1933)
• Fix website security issues (#2098, #1874, #1870, #2012)
• Keep GH Actions up to date (#2108, #2091, #2082, #2067, #2065, #2030, #1993, #1962, #1960, #1907, #1898, #1878)
• Stop running CodeQL for markdown-only changes (#1865)
• Normalize line-endings across repository (#1883)

Contributor Spotlight

We are excited to highlight the contributions of the following SynapseML contributors:

<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/aydan.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/sheryl.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/markus.jpg">
Aydan Aksoylar	Sheryl Zhao	Markus Cozowicz
Aydan is a Senior Applied AI Engineer and a first-time contributor to SynapseML. Aydan recently joined Azure Data but quickly led the efforts to add the new integration with Azure Cognitive Search's Vector Indices. This feature allows users to quickly create flexible semantic search engines powered by rich models like GPT4. Aydan went above and beyond on thie project and also contributed a Document Question and Answering with PDFs quickstart to showcase how to use these new features.	Sheryl is Principal Applied Scientist on the SynapseML team and a first-time contributor to SynapseML. Sheryl worked hard to devise an elegant connection between the LangChain and SynapseML to enable deploying chains on large datasets. She also designed and built a lovely quickstart to showcase how to build a distributed axiv reader with only a few lines of code.	Markus is a Principal Applied Scientist on the SynapseML team and a SynapseML veteran developer. Markus has contributed algorithms running the gamut from reinforcement learning and LLMs to anomaly detectors. This release, Markus contributed an ambitious and full-featured integration between SparkSQL and PowerBI data models. This allows users to explore their existing PowerBI datasets and measures with the full generality of PySpark or (Scala) Spark. This dramatically expands the automation possibilities within Microsoft Fabric. Markus never ceases to out-do his prior contributions and we are excited to see what he has in store next.
<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/amir.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/aadharsh.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/brendan.jpg">
Amir Jafari	Aadharsh Kannan	Brendan Walsh
Amir Jafari is Senior Product Manager on the SynapseML team and has recently taken over the role of the official SynapseML PM. Amir's passion to advance the library was instrumental in driving us to v1.0. He is fiercely productive and has a knack for simplifying and improving the SynapseML user experience. Additionally, Amir isn’t afraid to roll up his sleeves and contribute notebooks and blogs. He drove several efforts to create new quickstarts and documentation for a variety of SynapseML features.	Aadharsh is a Vice President and Head of Economics and Data Science at Western Digital. Aadharsh is also a new SynapseML contributor whose first contribution significantly generalized our causal inference stack to support fast estimation of heterogeneous causal treatment effects with Orthogonal Random Forests. This was a nontrivial and mathematically intensive contribution, and we are grateful for Aadharsh's expertise and persistence in getting this through our build system.	Brendan is a Senior Engineer on the SynapseML and a talented developer. Brendan's contributions range from core improvements to the SynapseML build and documentation generation system, to spearheading customer engagements and onboarding AI services. Most recently, Brendan used SynapseML to create and donate thousands of audiobooks to the open source in partnership with Project Gutenberg. This effort was considered one of TIME's top 200 inventions of 2023. You can learn more about Brendan’s awesome technical philanthropy efforts at https://aka.ms/audiobook.
<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/Jessica.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/serena_color.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/cruise.jpg">
Jessica Wang	Serena Ruan	Cruise Li
Jessica is Software Engineer who recently joined the SynapseML team. Already, Jessica has grown into the role of the SynapseML benevolent “doc”tator. This release Jessica has worked hard to ensure that the SynapseML notebooks work across a wide variety of Spark platforms and are easy and simple to get started with. This work requires knowledge of the entire library’s surface area, and we are thankful Jessica has worked so hard to learn this breadth of content. Furthermore, Jessica was also instrumental in building our Azure Doc auto-generation system to ensure all docs are tested as part of our CI build.	Serena is a Software Engineer at Databricks, a MLFlow maintainer, and a prolific SynapseML contributor. Serena's impact can be felt throughout almost every aspect of the library, and she is personally responsible for the new Form Recognizer V3 update, new streaming anomaly detection APIs, distributed deep network training, and many more features. Additionally, Serena laid the foundations of keyless authentication on Fabric, and pioneered our integration with MLFlow.	Cruise is a Software Engineer II on the SynapseML team in Bejing. Cruise has been instrumental in building and testing the keyless Azure AI services on Microsoft Fabric. With this contribution, Fabric users can configure their workspaces to use OpenAI, Langchain, and a variety of other AI services without the hassle of managing keys or authentication. Cruise has also worked hard to ensure AAD authentication works with Azure AI services and has helped the effort to standardize logging and telemetry across SynapseML and its sister projects.

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external, who helped create this version of SynapseML

Markus Weimer @markusweimer, Eric Dettinger @sandshadow, Scott Votaw @svotaw, Mark Niehaus @niehaus59, Aydan Aksoylar @aydan-at-microsoft, Sheryl Zhao @sherylZhaoCode, Markus Cozowicz @eisber, Brendan Walsh @BrendanWalsh, Jessica Wang @JessicaXYWang, Tom Finley @TomFinley, Sailesh Baidya @saileshbaidya, Keerthi Yanda @KeerthiYandaOS, Kyle Rush @k-rush, Aadharsh Kannan @AKannanMSFT, Serena Ruan @serena-ruan, Cruise Li @mslhrotk @lhrotk, Jason Wang @memoryz, Haizhou (Dylan) Wang @dylanw-oss, Sarah Shy @sarahshy, Kashyap Patel @ms-kashyap, Puneet Pruthi @ppruthi, Ilya Matiach @imatiach-msft, Amir Jafari @amhjf, Nellie Gustafsson, Bogdan Crivat, Justyna Lucznik @juluczni, Richard Wydrowski @richwyd, Tania Arya @taniaarya, Adithya Mukund @adithyamukund, Roman Batoukov @RomanBat, Alexandra Savelieva @alsavelv, Jessica Wolk @msplants Luis França @luisffranca Paul Koch @paulbkoch Rich Caruana, Avrilia Floratou, Martha Laguna @martthalch @marthalc, Jeff Zheng, Sciong Yang, Peixian Gong, Ruixin Xu, Chris Hoder, Derek Legenzoff, Misha Desai, Eren Orbey, Beverly Kodhek, Louise Han @jr-MS, Raj Rikhy, Brice Chung, Marcos Campos, Mike Estee, Kim Manis, Mitrabhanu Mohanty, Anand Raman, Sudarshan Raghunathan @drdarshan, William T. Freeman, John Moyer, Vidip Acharya, Ashit Gosalia, Miguel Fierro @miguelgfierro, Ismaël Mejía @iemejia, Kartavya Neema @kartavyaneema, Daniel Ciborowski @dciborow, Mark Tabladillo @marktab Guilherme Beltramini @gcbeltramini Akshaya Annavajhala (AK), James Verbus @jverbus, Mopé Akande @msakande, Frank Solomon @fbsolo-ms1, ONNX Team, Azure Global, Vowpal Wabbit Team, LightGBM Team, MSFT Garage Team, MSR Outreach Team, Speech SDK Team, MLflow Team, Azure Docs Team

Learn More

<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/synapseml_website.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/pg_tile.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/Thumbnail%20-%20Mark%20-%20Intro%20to%20SynapseML.png">
Visit our website for the latest docs, demos, and examples	Learn about our effort to create thousands of free audiobooks	Learn the basics of SynapseML
<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/F-103D-WgAIXx8I.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/SynapseML%20Part%204%20Thumbnail.png">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/pg_poster.jpg">
Read our full list of SynapseML Ignite Announcements	Apply OpenAI language models to your large datasets	Read our Paper on Custom Voice Audiobook Creation

<a name="SynapseML v0.11.4"></a>

v0.11.4

Bug Fixes 🐞

• Update Maven package to include correct github path (#2073)
• handle long run ids in adb tests (#2102)
• fix small bug in timing logic (#2099)

Build 🏭

• bump ossf/scorecard-action from 2.2.0 to 2.3.0 (#2091)
• bump amannn/action-semantic-pull-request from 5.2.0 to 5.3.0 (#2082)

Documentation 📘

• Update new URL of Recommenders (#2086)

Features 🌈

• add certified event feature categories (#2109)
• Make streaming the default LightGBM data transfer mode (#2088)

Maintenance 🔧

• bump to version 0.11.4 (#2110)
• Update new URL of Recommenders (#2086)
• factor accesstoken runtime-dependency wrapper out (#2097)
• fix cognitive service tests (#2092)
• Scala client for certified events (#2045)
• Update README.md to avoid re-releasing (#2085)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

<a name="v0.11.3"></a>

v0.11.3

Bug Fixes 🐞

• updated gpt-review to version 0.9.5 to fix break (#2069)
• fixed broken link to developer readme (#2049)

Build 🏭

• bump actions/upload-artifact from 3.1.2 to 3.1.3 (#2067)
• bump actions/checkout from 3 to 4 (#2065)

Features 🌈

• support using mwc token for fabric openai transformer (#2070)
• add Azure Cognitive Search vector store (#2041)

Maintenance 🔧

• bump version to v0.11.3(#2084)
• add Esrp release (#2083)
• Add more redirect links to website (#2079)
• refactor telemetry system (#2047)
• Add .trunk to gitignore (#2078)
• fix mlflow error in build (#2077)
• Add missing redirects to website (#2076)
• turn off barrier exec mode in VW generic sample (#2072)
• fix some of the failing build issues (#2071)
• fix daily midnight build chronjob

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• 6ac187db38feff9bf455c9ef2b2e6f1b8983b5e7 chore: bump version to v0.11.3(#2084)

• acfc39101cae8f5260da3c4f3dada9dd42901dc0 chore: add Esrp release (#2083)

• eb05959be6901ee21ac2e140f77d238ec98e96a4 fix geospatial tests (#2081)

• b8942a79ef569b027e0ef12b6ccf8a0f58696d87 chore: Add more redirect links to website (#2079)

• b0bac727365dde27de7744969b2cfc94067d71aa chore: refactor telemetry system (#2047)

• f7a0af54eec436c6dc3d4a534e8efcaa17370c29 build: bump actions/upload-artifact from 3.1.2 to 3.1.3 (#2067)

• bef5518a4065f8c26e6837f8e86be06e93d17a87 feat: support using mwc token for fabric openai transformer (#2070)

• e3c48f8210b6efebefc7e68c832afb732872f9ac chore: Add .trunk to gitignore (#2078)

• f2a4047e82f5e060431b5be49a028a4988e1a6e3 chore: fix mlflow error in build (#2077)

• 5a5e84876f6a8fbcf59f080b9be2b3483cbbacb9 chore: Add missing redirects to website (#2076) <details><summary><b>See More</b></summary>

• 17fddae285e9d16d902daab09beced9048b732b8 chore: turn off barrier exec mode in VW generic sample (#2072)

• dd7ba7c34c67033335ba1b7aee3459729dec4ad8 chore: fix some of the failing build issues (#2071)

• 3d1a7fc4c4a3e7177f4863c930e4595859bf5413 fix: updated gpt-review to version 0.9.5 to fix break (#2069)

• d494f6e5814eb39233a3eb4f8056eba962bb28f8 feat: add Azure Cognitive Search vector store (#2041)

• 111823d168519c030bc42e4e83250211b7227b3e build: bump actions/checkout from 3 to 4 (#2065)

• 7e9b0d1e3c6fa5e79ed141315f85d4da9b141b7d fix: fixed broken link to developer readme (#2049)

• ebecbe04ee9e1de7b7e9fb4f5955e19852e49373 chore: fix daily midnight build chronjob

• 0836e40efd9c48424e91aa10c8aa3fbf0de39f31 Fix problem with empty partition assigned to validation data (#2059)

• c6d58829435d8e2ab2cdc1cb1b6a9e0c5d4771b2 chore: remove secret scanner (#2048)

• 8f794c896dad5b790356671ac56590fbb4f36eaf feat: Support langchain transformer on fabric (#2036)

• 149c634005a935aef90226bc823c541934245568 docs: add badges to readme

• f6328b5dbe2a0721e12fdacd12a975a9c89c2494 fix: improve docgen (#2043)

• 33807625edcf678461c2916086a63fe91cf045a6 docs: fix small error in docgen docs

• 9eff35f5465269a16f2e936fdc069c6107536805 docs: initial POC of Jessica's fabric doc generator (#2023)

• fa497f09b58a462a4ca47c14cbe4bfd12231f1b2 docs: fix broken links (#2042)

• cde68347a44b6556ed4e6f6ab57bf6b4968cc6b2 fix: Improve LGBM exception and logging (#2037)

• db6386c6d6a133eb55fbec640012ad82bb526c94 fix: Fix ONNX link (#2035)

• 8be8fe3e61837dbe96a28189fd436e0b684f8ed2 docs: fix variable formatting for QandA nb (#2033)

• 072c9c9631d2c8c498677356dd2c152efd3cf51c chore: remove exclusions from pipeline.yml

• 495c9a9f297a31f05645324014e1ae738336d7eb chore: remove build exclusions from pipeline.yaml

• 0d8e613bc6e79f81795ab2b6f7cb9931cf66a864 build: bump actions/checkout from 2 to 3 (#2030)

• 4841ddb70cae5e6b8cf76d77bd2fed9d486b5471 docs: add QandA notebook. (#2029)

• d9b5d03e60f4e40323cace5a947732f43fa4454c docs: fix broken link (#2032)

• df2712afeb211003bd13e622239c9ae09c15615a docs: fix broken links (#2027)

• 81a36bd2457ccca5a526071c1cc7d06cafa7f807 docs: continue fixing broken links (#2026)

• 1c4c5b388edfa9cdffbf2a91fcb893a82e02c561 docs: fix broken links (#2025)

• 1f3a2bf9afcc45153b576eeb86c5d4ccaa92226e docs: add dead link checker (#2022)

• c4d077ad7f97bf5d08f1d329d6e866010c3889d0 chore: bump databricks e2e timeout (#2024)

• 09269384bbabf04d2532534b455263b172d32e9c docs: Refactor docs and docgen framework (#2021)

• ea60f56baf27d30e34e8f9fb7936a0e9227a098e docs: fix docker link (#2019)

• 094c180c3e884cfe9564abbc81986c3ccfb1737b docs: update notebooks - bring back fabric reviewers changes. (#1979)

• 46a56e1c8f9d1ed5e750892dc439964f817755fe docs: add prerequisites - openai and cognitive services resources (#2008)

• 8f79525f047ec19d6baad62a0e7f96cecb94892b build: bump semver from 5.7.1 to 5.7.2 in /website (#2012)

• 25c458223a8369baaaf67313a4d67803d640121b fix: modified the search engine in the demo notebook to bing (#2013)

• f7e8d80a96f12beb3fd1be0bb58590ef3646587c docs: remove pre-commit from docs

• 0c1c84b3bf8fcc3cf4dd079115020200dc94c61f chore: no-op PR to avoid double publishing

This list of changes was auto generated.</details>

<a name="v0.11.2"></a>

v0.11.2

Bug Fixes 🐞

• make geospatial services robust to 404 thrown by the service (#2007)
• dont retry 4XX codes other than 429 (#2005)
• improve tolerance for error handling (#1991)
• Some tidying and build fixes (#1984)
• fix import error on using the cognitive services on AML spark clusters (#1951)
• Update retry arguments in GeospatialServices - Flooding Risk.ipynb (#1954)
• Update retry arguments in GeospatialServices - Flooding Risk.ipynb (#1954)
• add retries for torchvision download (#1949)

Build 🏭

• upload notebooks to storage on every build (#2001)
• Publishing to ADO Feed (#1995)
• bump ossf/scorecard-action from 2.1.3 to 2.2.0 (#1993)
• Update documentprojection to support manifest files and custom formatters/endpoints (#1976)
• bump microsoft/gpt-review from 0.9.3 to 0.9.4 (#1962)
• bump microsoft/gpt-review from 0.9.2 to 0.9.3 (#1960)
• Added module for generating docs from notebooks (#1911)

Documentation 📘

• Improved LightGBM docs (#2003)
• Add Langchain notebook demo (#2002)
• update website synapse info and fabric installation (#2000)
• update OpenAI notebook for acrolinx (#1999)
• Add new LightGBM streaming docs (#1992)
• Update Vowpal Wabbit - Multi-class classification.ipynb (#1971)
• Update Vowpal Wabbit - Overview.ipynb (#1972)
• Update Vowpal Wabbit - Contextual Bandits.ipynb (#1970)
• Update Vowpal Wabbit - Classification using VW-native Format.ipynb (#1969)
• Update Vowpal Wabbit - Classification using SparkML Vector.ipynb (#1968)
• update spark33 installation instruction readme (#1961)
• remove cell output - sentiment analysis quickstart (#1932)
• improve and organize openai docs (#1937)

Features 🌈

• Reference dataset (#1977)
• Add Langchain Transformation (#1925)
• OrthogonalForestDML (#1873)

Maintenance 🔧

• bump to v0.11.2 (#2011)
• make it so custom versions are possible (#1998)
• fix spark_install for R tests (#1994)
• retry conda install and add a timeout (#1983)
• exclude failing explanation dashboard notebook (#1982)
• fix broken tests (#1981)
• add timeouts for R tests (#1963)
• add internal to find_secret (#1948)
• fix flakiness in spark installation (#1944)
• update R-setup.md docs (#1946)
• Add Azure Open AI based PR Summarization (#1957)
• fix gpu notebook protobuf version (#1959)
• A base scrubber and a "Shared Access Signature" Scrubber (#1939)
• remove old notebooks from website (#1934)
• no-op commit to ensure no double-releasing of library

Performance Improvements 🚀

• Update OpenAI Embedding with latest embedding model (#1938)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

<a name="v0.11.1"></a>

SynapseML v0.11.1

Bug Fixes 🐞

• set default values for aadToken & url for internal Synapse (#1918)
• ONNX model shape inference cannot handle batch with shape [-1] (#1906)
• forgot to add getPValue to python side (#1909)
• generate random dir for each test (#1908)
• add back diagnosticsInfo for MVAD (#1892)
• DML run get timeout if big dataset has more feature columns (Workaround Synapse Spark optimizer issue) (#1903)
• fix date parsing in FaceSuite test (#1896)
• fix Build pipeline (#1904)
• Retry OnnxHub call to improve test reliability (#1889)
• Normalize line-endings (#1883)
• Remove case matching for erased generic type (#1880)
• fix bug #1869, DML .setFitIntercept should be set to true (#1876)
• Remove extraneous "Foo" type from Py codegen (#1867)
• Allow variable size in ONNX inputs (#1851)
• Abstain from CodeQL for markdown-only changes (#1865)
• fix style
• update OpenAIEmbedding internalServiceType

Build 🏭

• bump peter-evans/create-or-update-comment from 2 to 3 (#1907)
• bump ossf/scorecard-action from 2.1.2 to 2.1.3 (#1898)
• bump amannn/action-semantic-pull-request from 5.1.0 to 5.2.0 (#1878)
• bump @sideway/formula from 3.0.0 to 3.0.1 in /website (#1874)
• bump webpack from 5.75.0 to 5.76.1 in /website (#1870)

Documentation 📘

• Fix installation instruction in the webpage for the build.sbt file (#1921)
• note discrete treatment data type (#1905)
• add custom chatbot creation to form demo (#1888)
• add overview page for simple DNN and fix some typos (#1879)
• Fix a typo in installation docs
• fix link issue in CONTRIBUTING.md (#1864)
• fix a few issues in cognitive service demo (#1861)

Features 🌈

• add streaming API for MVAD (#1893)
• [DistributionBalanceMeasure] Add implementation + unit tests for custom reference distribution (#1885)
• Add ChatGPT through the OpenAIChatCompletion transformer (#1887)
• support new api version of form recognizer (#1882)
• Add a new function to DMLModel, getPValue (#1863)
• update default internal endpoint for cog services (#1859)

Maintenance 🔧

• bump to v0.11.1 (#1933)
• Adding telemetry for the dataset metadata. This one is specially for … (#1917)
• fix r tests (#1927)
• fix build issues (#1916)
• disable test until Synapse is fixed (#1915)
• add .bloop to .gitignore (#1897)
• clean up old/missed search indexes in SearchWriterSuite (#1901)
• Add utility to clean azure search indexes
• update website docs to point to correct developer API docs (#1877)
• Update pipeline.yaml for Azure Pipelines (#1866)
• make sure nightly build has new commit

Changes:

• 866261c212441a92c4c5dfa14d0f16ce71be510f chore: bump to v0.11.1 (#1933)

• 3c097027eeba8896724d979ae50d50f432934ef6 chore: Adding telemetry for the dataset metadata. This one is specially for … (#1917)

• 0d0d10c7cdedca17bc7cb85d039c5c42ae954721 feat: add streaming API for MVAD (#1893)

• 1b71c1dadef393ce8173144230e1165a1fc651e4 chore: fix r tests (#1927)

• 0df97ad230e6ce7f2f90132b1117d1e39d0f1cb7 chore: fix build issues (#1916)

• 78695fb03b56e4eb8b179cccd91dd655fefda2f8 Update Regression - Vowpal Wabbit vs. LightGBM vs. Linear Regressor.ipynb (#1922)

• 87d5bc5391e5c4a2b04ae86ebf987a2c5c8dc10d docs: Fix installation instruction in the webpage for the build.sbt file (#1921)

• 8320b2baa84e5963ea5539b3d8aafd8fcebb2ec3 fix: set default values for aadToken & url for internal Synapse (#1918)

• 4912ae49c51fe7c335b4ad634d3496e03b0c23f7 chore: disable test until Synapse is fixed (#1915)

• 469445b7880b336c605fdb5f6bec570989134d27 fix: ONNX model shape inference cannot handle batch with shape [-1] (#1906) <details><summary><b>See More</b></summary>

• 3fa001e129c914b7f315a015434a4bb4462d8836 build: bump peter-evans/create-or-update-comment from 2 to 3 (#1907)

• f51327e236d09483247b64ee99c7de40c5f245c0 Update LightGBM version to 3.3.5 (#1910)

• b1e584ecd7e2b600043e218856bf5e5b3110a888 fix: forgot to add getPValue to python side (#1909)

• a09a6f775e3a21875eace9c6c710d88a533a014c docs: note discrete treatment data type (#1905)

• 0fa3f2a6647a16053b970f7e2e240bee9fc5436a fix: generate random dir for each test (#1908)

• 736c3172dea00c7116989959dd37bb8bb68e8d0a fix: add back diagnosticsInfo for MVAD (#1892)

• 13afff6ba89ec7951fd9d47739214ef6e4e57d52 fix: DML run get timeout if big dataset has more feature columns (Workaround Synapse Spark optimizer issue) (#1903)

• 7546e7fe9f4c14f396f3e99806df4b4e783105d7 build: bump ossf/scorecard-action from 2.1.2 to 2.1.3 (#1898)

• f227f02496c51f500a5d82bbe7ee1a9a8e2e9acb fix: fix date parsing in FaceSuite test (#1896)

• 0f02626eec3a23b5e5d21b880589b95f0045bb39 fix: fix Build pipeline (#1904)

• ce9fe41031e790b694a63056decb4642dfe249b9 chore: add .bloop to .gitignore (#1897)

• 7ffa970f56b31794b59765cd5f4fe4fdd82db483 chore: clean up old/missed search indexes in SearchWriterSuite (#1901)

• 9a6cf0358a45d16ff20d458ffc6dcdcb88ccf5d9 chore: Add utility to clean azure search indexes

• 52919ce40042ad19a9ca6834d49776d6c88ff595 fix: Retry OnnxHub call to improve test reliability (#1889)

• 979c62911f1d79fe19e337855a8bd0cf57a77390 feat: [DistributionBalanceMeasure] Add implementation + unit tests for custom reference distribution (#1885)

• 412620a88ceac095cd34f8a73ad90df5b3da6f82 docs: add custom chatbot creation to form demo (#1888)

• 9f634a62070d52775ef9df86afd7d04d4f6e9c7f feat: Add ChatGPT through the OpenAIChatCompletion transformer (#1887)

• 76570894b4108ab6e5d44ce8f790ee49485636b6 fix: Normalize line-endings (#1883)

• c1567920acc05d54dab96c8c30c55dc89eb2225a feat: support new api version of form recognizer (#1882)

• ed842a5f84e101862df5d85c7d9de1eacf763b85 docs: add overview page for simple DNN and fix some typos (#1879)

• 87e1c78611345d048bac3337cfa849f0dca7eb77 fix: Remove case matching for erased generic type (#1880)

• cd72bc921ebbb4e624142e5245d6241f0e382bc3 build: bump amannn/action-semantic-pull-request from 5.1.0 to 5.2.0 (#1878)

• 564d04756a514d69ed43edd338f17e84e605a8a1 fix: fix bug #1869, DML .setFitIntercept should be set to true (#1876)

• 392dbbf3583037d46f7b03e5cdf59323b5bead42 chore: update website docs to point to correct developer API docs (#1877)

• 129abdebb384457fe07d48c591f96278b517c255 build: bump @sideway/formula from 3.0.0 to 3.0.1 in /website (#1874)

• 4d1c560d57bd25786516124924a7eb2bdaecd5a6 build: bump webpack from 5.75.0 to 5.76.1 in /website (#1870)

• 62c79d84d9999a618a596a03bab12dfad05114ce docs: Fix a typo in installation docs

• 1f63dab87586607f774d6240718a6b59f1b8546f feat: Add a new function to DMLModel, getPValue (#1863)

• 83f8260df14b15cd3260e087706aee494a1824e2 fix: Remove extraneous "Foo" type from Py codegen (#1867)

• a5bec4577b305d19131e439a80bdaba8d46b1110 fix: Allow variable size in ONNX inputs (#1851)

• 23c9b0ac7f49e2eb6c537b57490fd485b5b0e029 chore: Update pipeline.yaml for Azure Pipelines (#1866)

• dedcbdac6ba10c9782106f04900e35678821e957 docs: fix link issue in CONTRIBUTING.md (#1864)

• a7f31d552cc3d42896455d4f84810306928836d2 fix: Abstain from CodeQL for markdown-only changes (#1865)

• a5f38b1732c089330457dd3d303f0608cf466a8f Update DoubleMLEstimator test CI verification (#1862)

• a44f917d211acc1593632a710cd476e93419ea82 fix: fix style

• cc931aff41fb733473e49a533e91862f65b4e428 fix: update OpenAIEmbedding internalServiceType

• 424d586e2782593da988d3d917ca81d3ab1465bc feat: update default internal endpoint for cog services (#1859)

• e4a0e2c381fd4b0638a47981e874046f74ae6f74 docs: fix a few issues in cognitive service demo (#1861)

• 8a216cedea99f40d71a88ef9e8f3b0f6ea815abf chore: make sure nightly build has new commit

This list of changes was auto generated.</details>

<a name="v0.11.0"></a>

<img width="100%" src="https://mmlspark.blob.core.windows.net/graphics/emails/email_header_synapseml.jpg" alt="SynapseML: Simple and distributed machine learning" href="https://github.com/Azure/mmlspark">

Building production ready distributed machine learning pipelines can be a challenge for even the most seasoned researcher or engineer. We are excited to announce the release of SynapseML v0.11.0 (Previously MMLSpark), an open-source library that aims to simplify the creation of massively scalable machine learning pipelines. SynapseML unifies several existing ML Frameworks and new MSFT algorithms in a single, scalable API that’s usable across Python, R, Scala, Java, .NET, C#, and F#.

Highlights

<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/OpenAI_Logo.svg">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/releases/hugging_face_and_pytorch.png">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/LightGBM_logo_black_text.svg">
ChatGPT and GPT-4 at Scale	Simple Deep Learning	LightGBM v2
Intelligent chat and embeddings. Simplified Prompting APIs.	Train custom image and text classifiers with ease	Higher performance, >10x lower memory footprint, same API
View Notebook	Learn More	Try an example

<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/Open_Neural_Network_Exchange_logo.svg">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/EconML-Logo-MSFT-colorXL.png">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/vw_horizontal.png">
ONNX Model Hub	Causal Learning	Vowpal Wabbit v2
Embed >150 state of the art deep networks into your pipelines	Discover and measure causal treatment effects	New second generation integration
Learn More	View Docs	Explore Samples

New Features

General ✨

• R Support is no longer Beta! (#1586)
• Support for Spark 3.2.3

Open AI 🤖

• Add OpenAI Prompt Template support (#1843)
• Add Azure OpenAI embedding support (#1832)
• Add Azure Active Directory authentication for OpenAI (#1829)
• Add Null-value handling for OpenAI models (#1854)

Deep Learning 🕸

• Remove CNTK functionality and replace with ONNX (#1593)
• Add the DeepTextClassifier a simple API for fine tuning a wide array of Hugging Face 🤗 text transformers using PyTorch Lightning (#1591)
• Add the DeepVisionClassifier a simple API for deep transfer learning and fine-tuning of a variety of vision backbones (#1518)

Azure Cognitive Services for Big Data 🧠

• Add SpeakerEmotionInference transformer to generate emotion annotation tags for emotive reading in SpeechToText (#1691)
• Add new AnalyzeText API (#1760)
• Support Azure Active Directory (AAD) authentication for the cognitive services (#1778, #1797)
• Move different cognitive services into sub packages (#1746)
• Add audiobook generation example (#1852)
• Add a notebook for advanced cognitive service usage (#1825)
• Upgrade MVAD to v1.1 (#1788)
• Remove MVAD's dependence on hardwired credentials and azure SDKs (#1629)
• Add word-level timing to SpeechToTextSDK and ConversationTranscription (#1801)
• Add the descriptionExcludes parameter to AnalyzeImage (#1590)

Causal Learning 📈

• Add the causal DoubleMLEstimator for learning causal treatment effects from data (#1715)
• Add a DoubleMLEstimator document and sample notebook (#1730)
• Fix DML regression bug, should remove both treatment and outcome columns as feature columns (#1820)
• Add TreatmentCol type checking (#1816)
• Update test to validate ATE value should be positive (#1821)
• Fix issue with missing causal test coverage (#1799)

LightGBM 🌳

• Add LightGBM streaming execution mode for more reliable performance with orders of magnitude less memory. (#1580)
• Add maxNumClasses param to LightGBMClassifier for multi-class (#1841)
• Added the passThroughArgs feature which allows users to set low level LGBM parameters before they are wrapped in SparkML (#1749)

Vowpal Wabbit 🐇

• Vowpal Wabbit v2 (#1579):
  • Support Vowpal Wabbit input format using VowpalWabbitGeneric model
  • Support additional algorithms & label types (multi-class, cost sensitive one against all): sample notebook
  • Progressive validation (aka 1-step ahead) using VowaplWabbitGenericProgressive
  • New Contextual Bandit Offline Policy Evaluation Notebook
  • Data parallel training independent of cluster size

Additional Updates

Bug Fixes 🐞

• Support grayscale images in toNDArray (#1592)
• Adjust learning rate in VW example notebook (#1853)
• Correct copy/paste error in acr cleanup (#1838)
• Fix synapse test config, and isolation forest notebook (#1833)
• Add spark config to fix ArrayStoreException (#1757)
• Fix breeze NoSuchMethodError (#1807)
• Fix modelVersion param in TextAnalytics (#1756)
• Make logging infrastructure consistent and add logging checks (#1755)
• Fix website sidebars and vulnerabilities in packages (#1753)
• Remove Vowpal Wabbit exclusion, add Interpretability exclusion (#1708)
• Update isolation forest notebook (#1696)
• Remove error on invalid columns in DropColumns (#1695)
• Fix PyArrow failure in deeplearning test (#1689)
• Fix linked service setters on cog service base class (#1685)
• KernelSHAP throws error when the key type in the ZipMap output is LongType (#1656)
• Fix flaky translate tests (#1643)
• Fix speechToTextSuite serialization Fuzzing failure (#1626)
• Fix translator endpoint and update all endpoints for gov regions (#1623)
• Finder runtime issues (#1598)
• Clean up cluster if Databricks tests pass (#1599)
• Fix deep-learning test flakiness (#1600)
• Update DotnetTestBase assembly version (#1601)
• Fix flaky forms test (#1584)
• Fix namespace import for Experimental (#1780)

Build 🏭

• Automate cleanup of Azure Container Registry images (#1787, #1751, #1735, #1814)
• Add bot to remove stale issues (#1602)
• Return values from TaskKeys (#1775)
• Remove unnecessary SbtPlugin settings (#1771)
• Simplify E2E test pipeline with a test matrix
• Add welcome message to new PRs/Issues (#1573, #1583)
• Add workflow to label new and reopened issues (#1571)
• Add a secret scanner (#1724)
• Add workflow to open GitHub issues after a comment (#1676)
• Add workflow to remove awaiting-response issue label on comment (#1674)
• Publish test jars so downstream projects can depend on test configurations and utilities
• Making build secrets optional and cached to remove 1 min latency on sbt commands (#1726)
• Add nightly build to catch flakes early (#1774)
• Automatically delete accumulated models in build (#1758, #1729, #1759)
• Add Dependabot for updating GitHub actions (#1608)
• Update build pipeline to ubuntu 20.04 (#1624)

Documentation 📘

• Add a hyperparameter tuning sample with HyperOpt (#1828)
• Add docs for new LightGBM executionMode parameter (#1779)
• Add additional ONNX docs for model hub and slicing (#1781)
• Improve OpenAI notebook (#1596)
• Add dotnet installation & examples (#1567, #1570)
• Update deep vision docs (#1752)
• Add custom search engine creation video to website (#1581)
• Replace Boston housing dataset with California housing dataset (#1856)
• Improve overview section in README
• Fix typo in old versions of Interpretability - Explanation Dashboard.md (#1846)
• Add versioned docs (#1858, #1566)
• Fix Synapse installation instructions for Spark 3.2 (#1815)
• Update required spark and python version on website doc (#1812)
• Fix latex rendering issue in Data Balance Analysis (#1796)
• Fix Acrolinx issues (#1792, #1793, #1808, #1794)
• Pin binder to latest released version
• Improve python env creation instructions in developer readme (#1693)
• Remove unused docs and fix links
• Improve example notebooks
• fix command to launch Jupyter notebook (#1649)
• Add documentation for MLFlow logging and loading (#1641)
• Update spark version in Readme
• Fix .NET logo on website (#1604)
• Update v0.10.0 installation guidance (#1578)

Maintenance 🔧

• Bump spark to 3.2.3 (#1744)
• Update Scalatest and Scalactic dependencies (#1706)
• Keep GitHub actions up to date (#1839, #1766, #1737, #1680, #1688, #1777, #1773, #1770, #1786, #1610, #1611, #1612)
• Keep website up to date (#1826, #1809, #1785, #1719, #1709, #1609, #1740)
• Update container registry and service principal connections
• Maintain Synapse tests (#1835, #1823, #1810, #1803, #1658)
• Fix codecov.yaml
• Fix conda env creation flakiness in build (#1748)
• Bump SynapseML Version (#1857, #1738, #1628)
• Improve code style (#1736, #1716, #1790)
• Pin az and python versions in build (#1705)
• Fix ado issue linker configurations (#1704)
• Add synapse-internal to platform detector function (#1651)
• Update OpenAI service to official deployment (#1619)
• Pin binder version (#1607)

Deprecations and Removals 🗑️

• Deprecate old TextAnalytics APIs (#1627)
• Remove old TextAnalytics APIs (#1622)
• Remove deprecated LIME APIs (#1620)
• Deprecate CNTK classes and ModelDownloader (#1712)
• Delete CNTK and related utils (#1743)
• Move ImageFeaturizer to onnx namespace (#1711)

Testing 💚

• Add Additional E2E testing infrastructure (#1727, #1769)
• Improve ONNX test reliability (#1713)
• Stabilize flaky tests (#1576, #1842)
• Remove Synapse E2E test exclusions (#1757, #1699, #1798, #1698, #1837)
• Add automated tests for getters and setters and improve test coverage (#1631)

Contributor Spotlight

We are excited to highlight the contributions of the following SynapseML contributors:

<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/scott.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/serena_color.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/dylan.jpg">
Scott Votaw	Serena Ruan	Haizhou (Dylan) Wang
Scott Votaw is a Principal Engineer on the SynapseML team has solved some of SynapseML’s toughest challenges in record time. In this release, Scott contributed both the new LightGBM streaming execution mode, and fully replaced our deep learning stack with the ONNX Runtime. These efforts were massive lifts including huge changes to the LightGBM native libraries and complex dependency management jujitsu respectively. Scott brings his love for the craft to every project he works on so keep your eyes peeled for more amazing feats of engineering from him in future releases.	Serena is a Software Engineer II on the SynapseML team and operates on a separate plane of existence than the rest of us mere mortals. Following up on prior major contributions like .NET support, form recognition, translation, and creating the SynapseML Website, Serena contributed the Simple Deep Learning package for this release. This package makes it easy to train modern deep text and vision networks from Hugging Face and torchvision on Spark clusters. Serena seeks only the most difficult engineering challenges and her contributions have laid the groundwork for many more deep-learning based algorithms in SynapseML.	Haizhou (Dylan) is a Senior Software engineer in the CSX Data team and a first-time contributor to the SynapseML library. Dylan contributed the new SynapseML causal learning package for the v0.11 release. This package helps users discover the effectiveness of things like medical treatments or economic policies even without controlled experiments. With his elegant contributions, Dylan has laid the foundation for more causal collaborations with the EconML library.
<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/markus.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/brendan.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/Jessica.jpg">
Markus Cozowicz	Brendan Walsh	Jessica Wang
Markus is a Principal Applied Scientist who (just!) joined the SynapseML team. Despite only recently coming on board officially, Markus has long been a prolific contributor to the library and built the Vowpal Wabbit and Isolation Forest integrations. In this release, Markus contributed the second generation of the Vowpal Wabbit integration, improving its generality and applicability. He also expanded the OpenAI integration to support embeddings and simplified prompt templating. Our team is incredibly lucky to have such a consistent and thoughtful collaborator.	Brendan is a Senior Engineer on the SynapseML team who recently joined after a long tenure on the Cognitive Services team where he developed their containerized cognitive service effort and co-authored the SynapseML publication on large-scale microservices. Brendan used this expertise to onboard Emotion Detection for text to speech models. He then went on to use this new emotive reading capability to create and donate thousands of audiobooks to the open source. You can learn more about Brendan’s awesome technical philanthropy efforts at https://aka.ms/audiobook.	Jessica is Software Engineer who recently joined the SynapseML team. Already, Jessica has grown into the role of the SynapseML benevolent “doc”tator. This release Jessica has worked hard to ensure that the SynapseML notebooks work across a wide variety of Spark platforms and are easy and simple to get started with. This work requires knowledge of the entire library’s surface area, and we are thankful Jessica has worked so hard to learn this breadth of content. If you have been following notebook examples from https://aka.ms/spark you have Jessica to thank!
<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/kyle.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/avrilla.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/emails/jason.jpg">
Kyle Rush	Avrilia Floratou	Jason Wang
Kyle is a Senior Software Engineer on the SynapseML team with a penchant for architecture and a streak of taking on big responsibility behind the scenes. Kyle has been instrumental in expanding our testing infrastructure to new platforms so that the lights stay on even as the number of contributions increases. This often requires nontrivial code and delicate cross-team collaboration, and Kyle has both the engineering might and the charismatic finesse to make sure these systems can be spun up successfully.	Avrilia is Principal Scientist Manager in the Grey Systems Lab, first-time SynapseML contributor, and a delightful collaborator. This release, Avrilia contributed the first prototype of the simplified OpenAI prompting transformer. This contribution makes it easy to ask ChatGPT and other LLMs questions about large datasets and to create new LLM-derived columns in databases. You can learn more about her work through the OpenAI Docs and prompting demo	Jason Wang is a Principal Software Engineering on the CSX Data team and has a long history of not only contributing huge features to SynapseML, but actively maintaining his contributions. This release, Jason’s work on the ONNX model hub protocol enables quick access to over >150 pretrained deep networks from the Java and Scala ecosystems. Jason has also been instrumental in fixing the most difficult and arduous bugs, some even stemming from the core Spark runtime. Finally, we deeply appreciate Jason’s leadership in the community: he consistently encourages and helps others contribute, and his impact extends far beyond his own personal contributions.

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external, who helped create this version of SynapseML

Eric Dettinger, Markus Weimer, Serena Ruan @serena-ruan, Scott Votaw @svotaw, Haizhou (Dylan) Wang @dylanw-oss, Puneet Pruthi @ppruthi, Markus Cozowicz @eisber, Brendan Walsh @BrendanWalsh, Jessica Wang @JessicaXYWang, Kyle Rush @k-rush, Avrilia Floratou, Jason Wang @memoryz, Mark Niehaus @niehaus59, Keerthi Yanda @KeerthiYandaOS, Ilya Matiach @imatiach-msft, Kashyap Patel @ms-kashyap, Martha Laguna @martthalch @marthalc, Sarah Shy @sarahshy, @ocworld, @adityakode, @nightscape, Alexandra Savelieva @alsavelv, Tom Finley, Jeff Zheng, James Verbus @jverbus, Chris Hoder, Misha Desai, Nellie Gustafsson, Eren Orbey, Beverly Kodhek, Louise Han @jr-MS, Raj Rikhy, Marcos Campos, Mike Estee, Brice Chung, Justyna Lucznik, Kim Manis, Mitrabhanu Mohanty, Bogdan Crivet, Anand Raman, William T. Freeman, Akshaya Annavajhala (AK), Guolin Ke, Spark.NET Team, ONNX Team, Azure Global, Vowpal Wabbit Team, LightGBM Team, MSFT Garage Team, MSR Outreach Team, Speech SDK Team, MLflow Team

Learn More

<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/synapseml_website.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/ProjectGutenberg_Mar13_w800.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/Thumbnail%20-%20Mark%20-%20Intro%20to%20SynapseML.png">
Visit our website for the latest docs, demos, and examples	Watch our AI Show episode on creating and donating thousands of audiobooks	Learn the basics of SynapseML
<img width="500" src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/463899i8693C747E0C251E1/image-size/large?v=v2&px=999">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/SynapseML%20Part%204%20Thumbnail.png">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/Data%20Quality%20OpenAI%20thumbnail.png">
Read the v0.11 Update blog	Apply OpenAI language models to your large datasets	Learn how to check data quality with GPT-3

Changes:

• 7b23764a93e73eab9c8903cb5f37e2438caedce0 docs: make v0.11.0 docs (#1858)

• 795bbecf5d1c803c549e78a0a940f5d7ef607127 docs: Add audiobook generation example (#1852)

• 8f9e970dd5df96db28a64b45687bf7e17adb8dd8 chore: bump version to 0.11.0 (#1857)

• 09648a32308f055d0cecf63eda2f82a2d2914d4c docs: replace Boston housing dataset with California housing dataset (#1856)

• 2b2419f1458634208b7127c557dc98717ce0b455 fix: OpenAI completion & prompting null handling (#1854)

• 66f30e30dfbcb8295ab29fe27b6e2501e87d5ac8 fix: adjust learning rate in VW example notebook (#1853)

• d64563f8efcb55a855d48ddea066ee361722ba5a chore: Upgrading ONNX version to fix assembly bug (#1849)

• a64b6e09e9c7e96f62ae8fd9a72663f66474d1a9 chore: fix mcr connection

• d23dd46e87cf0688c46c1ebedae1a4a7048186a2 chore: update build service principal

• 6876d53a95d71dc5c1fe66650d3f7b72b839fca1 chore: Update the build pipeline <details><summary><b>See More</b></summary>

• c1aa5a542e26d84912cf93b8780d62d8952cad01 docs: Fix type in old version of Interpretability - Explanation Dashboard.md (#1846)

• 48f9c4c46e2a80d81639bcfcb1fb374e038e4de0 feat: OpenAI Prompt Template support (#1843)

• fbbb4336d12fe1d3124295aa3c42f16d522d133e chore: fix form recognition tests (#1842)

• b90425c41cbff7451c1bece19b59b254660c5b7b build: bump amannn/action-semantic-pull-request from 5.0.2 to 5.1.0 (#1839)

• 31c4ea34bea59d8456d29f2791117ee3ef1a5627 fix: correct copy/paste error in acr cleanup (#1838)

• c99796f955692f041a707e729216be05eeae9e02 feat: add maxNumClasses param to LightGBMClassifier for multi-class (#1841)

• eb0bbe335ffdb00e58dd89947f0542e9a602ba31 fix: modify synapse test config, modify isolation forest notebook for testing (#1833)

• 8af5112d33f5de24f97f5d8f59fd9d403e412bd6 docs: remove hyperOpt exclusion - mlflow on synapse (#1837)

• d66796d7c743d63e9e0b2bd1b45874f448b61aff build: bump http-cache-semantics from 4.1.0 to 4.1.1 in /website (#1826)

• 7038e1da31f9ef48f119a7805110ea190155f97c feat: add Azure OpenAI embedding support (#1832)

• 6c6d89b54a595718a149e8accc9de3b311fadb22 chore: turn off failing synapse tests (#1835)

• eb355817285564bd062eb6deab9755146ba00157 docs: add hyperopt sample (#1828)

• 2262a9bf5d3db651f9d87cd2a3849e1154d600ad feat: add aad auth for openAI (#1829)

• a7e20ce39d479a680da5811f6fa04975e0b407c0 docs: Add a notebook for advanced cognitive service usage (#1825)

• 3eed94c205629764119ec1ba04b93f426c201cd8 test: Update test to validate ATE value should be positive with the test data (#1821)

• 8aa4ae17dc9e6e45fa53047f9827edc5beb2b8ac chore: re-enable E2E tests for Synapse-Extension (#1823)

• 1dcb5881235994c42ad88aafdfffebeb9ae29ea4 docs: update spark3.2 installation on Synapse (#1815)

• e36643f1a71e39b0c6bfe6c53935b857453dcd61 fix: Fix DML regression bug, should remove both treatment and outcome columns as feature columns (#1820)

• 4ef8e30167a304ecd2a58dbf4a443d91e062d8ad fix: Add spark config to fix ArrayStoreException in Synapse - Add back HyperparameterTuning nbs to test pipeline (#1757)

• eb403728f9162565fd3ec124012ed671b92447e0 fix: Add TreatmentCol Type check at the very beginning (#1816)

• af0a218bcc9b03d9b677ba7b6978a09fc64ba46b docs: update required spark and python version on website doc (#1812)

• e0c5364df052e6e896e01e5a3e971876771a7170 build: bump ua-parser-js from 0.7.31 to 0.7.33 in /website (#1809)

• e212f5df2f3c1da89409f3214a4d5c82dc1f135f chore: add retry to commands (#1814)

• bd1e0a61f3fc0dc109702e613d21eadd04c56565 fix: breeze NoSuchMethodError (#1807)

• 13ff3467fd0778761463ed3ebfa652e07dc5ad1c chore: Disable synapse-extension tests, add params to pipeline (#1810)

• 4c8d2e972f1bfb33efc6aced2b8a9a542de119ec doc: apply diffs from website/docs to website/versioned_docs (#1808)

• 7d8d6fd634bee190520853ebf69f6e62b5ac15b7 replicating the unit test data (#1806)

• fb47138d7e881ecb654d5d039893e63bb13abce9 docs: DoubleMLEstimator document and sample notebook (#1730)

• 8ba77e4eea742c4728a662ba68a660db6c974054 build: Return values from TaskKeys (#1775)

• 94ed68512d41434af28f26097dbe2ebef5c1643c chore: disable Interpretability tests (#1803)

• 54e7ac6143fc25ed2300e85fddb4e3ce21922e84 feat: add setting for getting word level timing information from SpeechToText (#1801)

• d8d523c6ba603d62744b06608265bb5bfbdad8df feat: annual Vowpal Wabbit improvements (#1579)

• 01e31dc69102694cbd05be1757c69b20efc13651 test: fix issue with missing causal tests (#1799)

• 9d92349ac6805321838eee0a9e6ca8407cdbf16f test: remove interpretability exclusion (#1798)

• d308dc4497abba715f245f4e1c3cb130968d26d2 refactor: enhancement to aad token (#1797)

• 333dedbdc8342ed2e73df29821c5a8912f3e58fd feat: upgrade MVAD to v1.1 (#1788)

• 54a749638c3383fc0609bed86af088d5d960223a chore: fix typo in chron build def

• c653ed77a13d7d99238c52ceb6aa4c4a6290ea3a docs: Clean latex - Data Balance Analysis (#1796)

• 76e7b73954cc1dc0da9e6c7022a42ed8a873329a chore: linx fixes for README and features (#1794)

• fc3a7a6a1485b2b3229a76648c2836d6c4fe4f53 chore: Re-enable e2e tests, add cron schedule to build definition (#1774)

• a95fad403184ae4fa4b886c28b2ea072b4344373 docs: Add docs for LightGBM execution mode (#1779)

• 44bcbf1fdcf5be829a46688ddc7e9cba06f98f13 improve documentation - bug, typo, correctness (#1791)

• 77da4b40822a5e40afdbe5a6956666ffc9a0068f chore: acrolinx fixes for reference, mlflow and getting_started in 0.10.2 (#1793)

• 8cc6a16a0e30cf6d8e58cb64f209c738931b9adf docs: mollify acrolinx (#1792)

• 974e36aaad0714a3e9ca83caa6a1130fb51f249a docs: Added more up-to-date ONNX docs (#1781)

• dc57deaf43010c91a181abe42776d43426f2e0cc feat: add aad authentication support for cognitive services (#1778)

• dd1563fc3004143c120de3b58d77d4f66338b8a2 build: bump json5 from 2.2.1 to 2.2.3 in /website (#1785)

• 3d8c84d7ffbe246a1e7651f75c0347b07fc5becd chore: fix style (#1790)

• 53788bd94cf0cc4badaaa91651b02221d297cf69 fix: small tweaks to clean_acr (#1787)

• 851efdc98de7c61480912c14bf192ad222c41b97 build: bump actions/upload-artifact from 3.1.1 to 3.1.2 (#1786)

• 9978c3b760b88839f0d5f574c159d7ed96323994 build: bump ossf/scorecard-action from 2.1.1 to 2.1.2 (#1777)

• 421e3fe3c09f7871db80f588654ef587d47adcea Fix: fix annamespace import for Experimental (#1780)

• 8dc4a582bb3e3f9c477583e850b76ddd4254973d build: bump ossf/scorecard-action from 2.1.0 to 2.1.1 (#1773)

• 61435147a64a52680152b3e55275a139a8b6b787 build: Remove unnecessary SbtPlugin settings (#1771)

• de21adaa9408df0ec566711f6ccd9fb1bc3ac903 chore: disable synapse-internal tests

• d0a9f20df0f753c93ad459cd401197165b64be96 feat: Causal DoubleMLEstimator (#8) (#1715)

• 7ab63a14ac1a9f14174b16b1be88ea99a94b5db4 fix: Update synapse-extension test environment, enable cleanup of old arti… (#1769)

• be02bf78b4ab3b789f6cc773cf71653acc8e0933 build: bump ossf/scorecard-action from 2.0.6 to 2.1.0 (#1770)

• 0bf4772ec35b0cd9b91d7da0c97d2fb1ad193cd1 build: bump actions/upload-artifact from 3.1.0 to 3.1.1 (#1766)

• bb6e37b956c62378b13cebeed0b63b2916e4a98c Update codeql.yml (#1765)

• 8a30fd482b9f564ff96277d712f1ec856a1c00cc Create codacy.yml

• d9810b157d68f22bda50b4dbfd271514e49a812a Create codeql.yml

• 630a442c1fcf7556e3514fd91dd8bf6d88a502ab Create scorecards.yml

• 7785cb5e15fe17b51d3610d95f045de5e0a72d26 feat: add new AnalyzeText API (#1760)

• cf14041be1e17e030af18fa1eb2afc96623e3e65 chore: delete old models before tests rather than after (#1759)

• 9e32a99b40151136d4af0c2bbb1ffba8865f45a1 fix: fix failing SpeakerEmotionInferenceSuite

• 4a2595461fb1d68609ec74ce4f7d33e58095452e fix: delete too many anomaly models (#1758)

• adad80d4454dd4fab4aacc93fa34e2aefb5ff75b chore: remove cntk and downloader tests from build

• 046689d9666c334aa0e11a603882afc62c8d1775 chore: fix codecov.yaml

• 3a3be327d0022d3181ac22ea90c9b6b2b5beae92 feat: Add LightGBM streaming execution mode (#1580)

• b205cc47b0e43ddcd1a947246bd568fbbe897630 fix: fix modelVersion param in TextAnalytics (#1756)

• b797d6c2ec9cea652ba01fcc03ca91baed413b76 fix: make logging infrastructure consistent and add logging checks (#1755)

• 557470bcec78acba3adcd9524337be628446311a fix: fix website sidebars and vulnerabilities in packages (#1753)

• 9c98609a23bb82f4bfeb2bcabf0b7c9840061691 docs: update deepvision docs on website (#1752)

• 629da631a263e46fbccd98d2026c73291087a46b refactor: move different cognitive services into sub packages (#1746)

• 37f2e90dd948419a7b31ec1bcbafb079f7fdb9ea chore: fix clean acr (#1751)

• c6cc0a88a14fbbce6296200d1d0b258cd4220c5f fix: Add docs for passThroughArgs (#1749)

• b6ef511ab10ca8101d7637cbd912b0efdabfb9fa docs: Pinning binder to latest released version

• 2a89e13870b136ffcfa5262956878bcfe8c6e578 feat: Delete CNTKand related utils (#1743)

• 98add7a4b2c260660d4b92dabd9d5a8ee4bd67d4 chore: fix conda env creation (#1748)

• 558f5d887302cb9b2af48d3a373002bba292eeb9 chore: bump spark to 3.2.3 (#1744)

• 70843d5c13d995742c277ae13b015b85c348e0a8 chore: bump docusaurus (#1740)

• 2d06b94683282770f126176e2a7ab9be1dd5db80 build: bump loader-utils from 2.0.3 to 2.0.4 in /website (#1719)

• aa69541cd290301eedcd7f0e386498d22ea99b0d docs: removing beta tag from R

This list of changes was auto generated.</details>

<a name="v0.10.2"></a>

v0.10.2

Bug Fixes 🐞

• remove Vowpal Wabbit exclusion, add Interpretability exclusion (#1708)
• remove synapse E2E testing exclusion - cyber ml (#1699)
• update isolation forest notebook (#1696)
• don't throw on invalid columns in DropColumns (#1695)
• fix pyarrow failure in deeplearning test (#1689)
• fix linked service on cog service base (#1685)
• fix Uplift Modelling style
• KernelSHAP throws error when the key type in the ZipMap output is LongType (#1656)
• fix flaky translate tests (#1643)
• update ubuntu to 20.04 in pipeline (#1624)

Build 🏭

• bump actions/checkout from 2 to 3 (#1737)
• bump loader-utils from 2.0.2 to 2.0.3 in /website (#1709)
• bump amannn/action-semantic-pull-request from 5.0.1 to 5.0.2 (#1688)
• bump amannn/action-semantic-pull-request from 4 to 5.0.1 (#1680)

Documentation 📘

• update developer readme instruction on python env creation (#1693)
• fix multiple typos and update error hintings in ai-samples-timeseries notebook (#1663)
• improve error msg to make it clearer for users and fix typos (#1662)
• simplify data downloading and add mlflow to uplift modelling (#1659)
• move magic command forward since it restarts interpreter
• remove unused docs and fix links
• improve example notebooks
• add aisample uplift modelling (#1640)
• fix command to launch jupyter notebook (#1649)
• add mlflow in ai samples time series forecasting (#1645)
• add mlflow logging and loading (#1641)
• update spark version in Readme
• improve readme overview
• add aisample on text classification (#1617)

Features 🌈

• add simple deep learning text classifier (#1591)
• Add SpeakerEmotionInference transformer for generating SSML t… (#1691)
• Deprecate CNTK objects (#1712)
• Remove CNTK functionality and replace with ONNX (#1593)
• R test generation (#1586)

Maintenance 🔧

• bump version to 0.10.2 (#1738)
• fix style (#1736)
• automate clean-acr with github action workflow (#1735)
• autodelete old models (#1729)
• Making secrets optional and cached (#1726)
• add secret scanning infrastructure (#1724)
• Move new ImageFeaturizer to onnx namespace (#1711)
• ScalaStyle fixes (#1716)
• update scalatest and scalactic (#1706)
• remove synapse test exclusions (#1698)
• pin az and python versions (#1705)
• fix ado integration (#1704)
• remove notebooks (#1703)
• fix reopen comment action
• fix reopen on comment workflow
• fix typo in issue reopen yaml
• re open github issues after a comment (#1676)
• clean up github workflows and add issue label remover (#1674)
• turn off failing synapse tests temporarily (#1658)
• added synapse-internal to platform detector function (#1651)
• publish test jars
• improve test coverage (#1631)
• Remove MVAD's dependence on hardwired credentials and azure SDKs (#1629)
• clean up TextAnalytics cog service APIs (#1622)

Testing 💚

• Additional E2E testing infrastructure (#1727)
• Improve ONNXtests reliability (#1713)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• cd1d2ea65ffcd0f89bf1fee231c430560508bcce chore: bump version to 0.10.2 (#1738)

• fd78889112b8d48927ddac1b660f62400bb1ba12 build: bump actions/checkout from 2 to 3 (#1737)

• c806ba79d17afa6edf53c9ea55e563f6842a6825 chore: fix style (#1736)

• e6b5a90352b7456333df92dc9f7755b6cb8f300b feat: add simple deep learning text classifier (#1591)

• 1de2d558996fed8ff12a312a52d53c39c3322fcb chore: automate clean-acr with github action workflow (#1735)

• 952d1bd3e0a4b7755d8aa3d069ff90ff626c7b17 clarify date comparisons when deleting old models/groups (#1733)

• 6ea02bd81ef09647119f4c36583f3a523dd7c795 chore: autodelete old models (#1729)

• 8b02e1d31751eef8b01b87ee590f1b3169e6371a chore: Making secrets optional and cached (#1726)

• c62c6ad441c18d98354099dbd39448d4b0734c58 test: Additional E2E testing infrastructure (#1727)

• aeb2ff7ecde6180fc546e2596d8121c71be49505 feat: Add SpeakerEmotionInference transformer for generating SSML t… (#1691) <details><summary><b>See More</b></summary>

• 0b96cc5bd5b26b677f2bd1ff7371c631c3984ee4 chore: add secret scanning infrastructure (#1724)

• 2a7a67ba373d3dd69c7ac4cd2f4118d1980eac06 feat: Deprecate CNTK objects (#1712)

• e38e3ad30c6bb10cbafa4351974aea8c2b8ebf37 chore: Move new ImageFeaturizer to onnx namespace (#1711)

• 0ff6802377328cb8875f7c60305da77074fa1771 test: Improve ONNXtests reliability (#1713)

• fe4c5d27a8d35aa90c5a70383b028de652247d48 chore: ScalaStyle fixes (#1716)

• 050b541e8b74c09d63abfdb2ad05d7582bd06f29 build: bump loader-utils from 2.0.2 to 2.0.3 in /website (#1709)

• f2e88fdea7c1010118913eecf0457b5daf881d25 feat: Remove CNTK functionality and replace with ONNX (#1593)

• abdfe19e79ca0533c168366aca96b8082a44a8db fix: remove Vowpal Wabbit exclusion, add Interpretability exclusion (#1708)

• 6a1f994812234ba861bcae0c96fb11f185eec261 chore: update scalatest and scalactic (#1706)

• 144674fdb9537e6dbad2817dee95d8c26eaf3fa5 chore: remove synapse test exclusions (#1698)

• 32c654b83781c6e028fdd23be47e083d1decb8e7 chore: pin az and python versions (#1705)

• c8fba2831d34338b5c82079a77b77963ceab32b6 chore: fix ado integration (#1704)

• 92d409574376a1570ab49cc82a60c7efbc9fd1de chore: remove notebooks (#1703)

• a9537809d877be4c2eec83b9a55c46a942026d41 fix: remove synapse E2E testing exclusion - cyber ml (#1699)

• b257c70562b312e03ca8fb566d71582601725800 fix: update isolation forest notebook (#1696)

• 9120b056920b54647b75c2c640a8e9e87c919969 using predictionCol for isolation forest (#1686) [ #1060 ]

• 448f6b7ca81d0e806e06410a5035bd5edff2ad6e Remove trident.mlflow APIs. (#1687)

• f4af33f719844d419083e6204bb58ee14b6de133 fix: don't throw on invalid columns in DropColumns (#1695)

• c531bbbfc93ccee3a3cc167060411941d3635e1b docs: update developer readme instruction on python env creation (#1693)

• 467e651dd814213bbfe4c13e5bc1b6dac7fd86ee build: bump amannn/action-semantic-pull-request from 5.0.1 to 5.0.2 (#1688)

• 302831ffd8cec84f0e24de6eef4c193d3ed0966a fix: fix pyarrow failure in deeplearning test (#1689)

• e857511e21e471829048650955823ecfe8e1e89d fix: fix linked service on cog service base (#1685)

• f29318a274610dda543ee1422bdbd74cdb6a752a build: bump amannn/action-semantic-pull-request from 4 to 5.0.1 (#1680)

• 50ac0c8aa7149637396700b8ccf16a422eb732ed Update reopen-issue-on-comment.yml

• c9278b5c1c2225c6b0f48bcd21996a1a584cd1f9 chore: fix reopen comment action

• b3a9ba9ca84e7af257664f81858301c9af03fc61 chore: fix reopen on comment workflow

• 9fe273b8665d9c70b5f0375a9b8e603d65143874 chore: fix typo in issue reopen yaml

• a7c50de2e905b55d64be9fdc413f012fdaecfb27 chore: re open github issues after a comment (#1676)

• 8914750ac804fe9072b327e4cecfc6c382cd52f2 chore: clean up github workflows and add issue label remover (#1674)

• 965231a98c7bc32151dc93f6c1a399c4b6ba4c76 docs: fix multiple typos and update error hintings in ai-samples-timeseries notebook (#1663)

• 4fa7249966386fdc86edb99ea1ea665ef1643c94 docs: improve error msg to make it clearer for users and fix typos (#1662)

• dd9e5d24a570f5735c2e048b35d0fb06aa887000 fix: fix Uplift Modelling style

• 5a52aef842eaeb36796fc4fc825e9198c371229f docs: simplify data downloading and add mlflow to uplift modelling (#1659)

• 95f451ab3f1b13d635c69b797f153610c2902ea4 chore: turn off failing synapse tests temporarily (#1658)

• 76d73826de969c36a3e71f884bf0dd7258beb7f0 fix: KernelSHAP throws error when the key type in the ZipMap output is LongType (#1656)

• e703ad4605e387e711de4e0ee3d9919d57e46674 chore: added synapse-internal to platform detector function (#1651)

• ca358e369a20fdbbfc43339cb0fb09d481cfe16a docs: move magic command forward since it restarts interpreter

• 3a160b395dae8d7d3af528b4d6180dc4d7737dd6 docs: remove unused docs and fix links

• d5a499720a4cd7a536ac755fbe1af2f355a27bc1 docs: improve example notebooks

• a7d097a7057d4363a5de4d1b9173867df60522b2 chore: publish test jars

• b7c8cf10b7b70fd90f45fe0e81a4bf7d8b58b4eb docs: add aisample uplift modelling (#1640)

• c8750ce83a6884ddb7503c361c53c6d67fab86e8 docs: fix command to launch jupyter notebook (#1649)

• 8d552746951ab4190e82dda1d4f04699fc46c69f docs: add mlflow in ai samples time series forecasting (#1645)

• d751a52b7e61d460bac15b01d5f2680274cbde77 fix: fix flaky translate tests (#1643)

• 59a922b4c7aa73f4a1f540b9f17ecc8f46c55a86 docs: add mlflow logging and loading (#1641)

• 4115d4f0f2ea5210b9eafd777ff7dc6f4567a7fb Create .acrolinx-config.edn

• 64fecca3f51ec6df6753edde0fba23ab87127a3e docs: update spark version in Readme

• 32037ecf357917d4270d7a2e7deec7074be91c4b docs: improve readme overview

• 289bd974275a7df6c0dfc79fb9a20156a14c3c7e remove extra packages installation in pythontests (#1633)

• 4878686cf58696bc276f810912f7a8667a2bcff0 feat: R test generation (#1586)

• 1381db524e9a48ba2d0463d7c1bfb8b057c9fc61 chore: improve test coverage (#1631)

• e700fd146a3c19aefac442b5f91f2b27600da938 chore: Remove MVAD's dependence on hardwired credentials and azure SDKs (#1629)

• d5ee8e747aeb0edc42a9c1e6b448503717bb1b1c fix: update ubuntu to 20.04 in pipeline (#1624)

• dbbe6814c6f82793f549cf798bce16a06b4abcc6 chore: clean up TextAnalytics cog service APIs (#1622)

• d98ac02c989492307e924f286cd5a7f3be767241 docs: add aisample on text classification (#1617)

This list of changes was auto generated.</details>

SynapseML v0.10.1

Bug Fixes 🐞

• fix speechToTextSuite serializationFuzzing failure (#1626)
• fix translator endpoint and update all endpoints for gov regions (#1623)
• binder runtime issues (#1598)
• clean up cluster if databricks tests pass (#1599)
• fix deep-learning test flakiness (#1600)
• update dotnetTestBase assembly version (#1601)
• fix flaky forms test (#1584)

Build 🏭

• bump EnricoMi/publish-unit-test-result-action from 1 to 2 (#1609)
• bump actions/setup-node from 2 to 3 (#1610)
• bump actions/setup-python from 2.3.2 to 4.2.0 (#1611)
• bump actions/setup-java from 2 to 3 (#1612)
• simplify e2e test pipeline with test matrix

Documentation 📘

• add aisample notebooks into community folder (#1606)
• add aisample time series forecasting (#1614)
• fix .NET logo on website (#1604)
• improve OpenAI notebook (#1596)
• pin mybinder to v0.10.0 to avoid thrashing
• add demo into videos on website (#1581)
• update installation guidance of v0.10.0 (#1578)
• add more .net samples (#1570)
• add dotnet installation & example doc (#1567)
• Update issue template

Features 🌈

• add stale bot for issues (#1602)
• Support grayscale images in toNDArray (#1592)
• Add the descriptionExcludes parameter to AnalyzeImage (#1590)
• Added the DeepVisionClassifier a simple API for deep transfer learning and fine-tuning of a variety of vision backbones (#1518)

Maintenance 🔧

• bump to v0.10.1 (#1628)
• deprecate old Text analytics APIs to prepare for refactoring (#1627)
• remove deprecated lime APIs (#1620)
• update openai service to the official deployment, and disable test due to outage (#1619)
• Auto update GitHub actions with dependabot (#1608)
• hotfix binder badge
• pin binder version for users (#1607)
• Bump spark to 3.2.2
• bump spark version
• Format welcome message with emojis (#1583)
• Add welcome message to new PRs/Issues (#1573)
• Add GH workflow to label new/reopened issues (#1571)
• update website (#1566)

Testing 💚

• stabilize unit tests (#1576)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• 0f54bc65e720ac89f1d4c04502bc2cb5c6310db7 chore: bump to v0.10.1 (#1628)

• 3d0f3f466afde7f69d91cdf27a97e629ac6dad91 chore: deprecate old Text analytics APIs to prepare for refactor (#1627)

• 2052e13b82f6ccc94848359f8070f24d5f06de6c chore: remove deprecated lime APIs (#1620)

• 09213b010e658833148026d242268e2eb0482b17 fix: fix speechToTextSuite serializationFuzzing failure (#1626)

• 9f78bf0074ebf3668fb7e1d5d18a681ec236f988 fix: fix translator endpoint and update all endpoints for gov regions (#1623)

• 7e90d190bd3f96869fe176c4eefe2bc417fe34fe docs: add aisample notebooks into community folder (#1606)

• ac40e5af5d2a7bd1f6c0a25b12b2e07e6ff92c2e chore: update openai service to official, and disable test due to outage (#1619)

• f54f7f68a4cb6596af5975d4caa20ea0ead798b2 docs: add aisample time series forecasting (#1614)

• 7b4b0e1c066e1ec7c3bff719e12b991d8193a25e build: bump EnricoMi/publish-unit-test-result-action from 1 to 2 (#1609)

• 43b0d1714954b1d160d1e0a11c64a52226d6825a build: bump actions/setup-node from 2 to 3 (#1610) <details><summary><b>See More</b></summary>

• c48a07a97493269799571c0df8a06654c414c247 build: bump actions/setup-python from 2.3.2 to 4.2.0 (#1611)

• b1a331c3c61fd89c147e2b0be65b9b23056eba9b build: bump actions/setup-java from 2 to 3 (#1612)

• 78e40cb37cab655d6d94b4df8690d1302154d019 chore: Auto update github actions with dependabot (#1608)

• 69d2d202439187f862bee59cac93b99e19ce0a4d chore: hotfix binder badge

• 93d7ccf7a782d89ac157d6e1c87ea3f55d11b886 chore: pin binder version for users (#1607)

• c7a61ecd57f9962590be3a075f586578a7fa3e13 fix: binder runtime issues (#1598)

• c960c06b8534e6b0013f4b2107d262fd4be62472 docs: fix .NET logo on website (#1604)

• 28a35b43ea7685e2e70ffc84c1bd39c6f7866176 fix: clean up cluster if databricks tests pass (#1599)

• 5a28740881a7298d783ed922b7746b7fc3d7c77b fix: fix deep-learning test flakiness (#1600)

• adf1a61d19a4493a39b063dc2abddc16a8b1bbe6 fix: update dotnetTestBase assembly version (#1601)

• c659b330342cdde38340b2c488d4b9bc8b2df58b feat: add stale bot for issues (#1602)

• 05a420257c25167c300a9a7c6e13f5674e4fba9a docs: improve OpenAI notebook (#1596)

• e019756ae7534cc1cdf81a8b24f8224b92855bdc feat: Support gray scale images in toNDArray (#1592)

• 51beaa0e462d5f7edc5f32242e0a7b8cc91b3ab5 feat: Add the descriptionExcludes parameter to AnalyzeImage (#1590)

• b9ac22a544a1398cd77e00dd10f0796f729eaf4c docs: pin mybinder to v0.10.0 to avoid thrashing

• 1808a0f452ffab9ee24063b1e6c16ef5ed06f95d chore: Bump spark to 3.2.2

• 8e7d4533e7f2c54da35438eb6561e47ee9269197 build: simplify e2e test pipeline with test matrix

• 8e34c7ba56687c2d92f601c5bbf1475cbad68584 chore: bump spark version

• 44c8ed5239dd7b7f43295865ff5b0caa87d40ab6 feat: Added the DeepVisionClassifier a simple API for deep transfer learning and fine-tuning of a variety of vision backbones (#1518)

• e4f0883740b970430cbbb5781431206b788caa49 fix: fix flaky forms test (#1584)

• 7da5f49d3161c2a2809f3dce5d117d9ec7903eb5 chore: Format welcome message with emojis (#1583)

• 0e6bb3557aff7314fd791bd40d8dccaaed7c5093 Serena/update issue template (#1582)

• a6a271860889dcc0b81bb8c5915bc35f31b866f3 docs: add demo into videos on website (#1581)

• 7c34fc4332443bff1d4e0f8c7a696f1c94d71977 test: stabilize unit tests (#1576)

• 49f3a58f9853421f832b6c50bf459d92af075459 chore: Add welcome message to new PRs/Issues (#1573)

• 4868e8bfed15da4d40cad1a910a272bc43bafc92 Add back LightGBM library initialization in booster (#1575)

• d427b8842a56a88dbbe1d533a4df083c41adb07f docs: update installation guidance of v0.10.0 (#1578)

• 55a60c9c017278881de70aa92bf516cf0e5fa552 docs: add more .net samples (#1570)

• 39fe2d8b987e0bee0823320d456d87da48b7a45d chore: Add GH workflow to label new/reopened issues (#1571)

• 0febe3cb5df1838bb5daaa138fbc74ee904a69ff docs: add dotnet installation & example doc (#1567)

• db95a1046584c158a3819325c2229d6501b48330 chore: update website (#1566)

This list of changes was auto generated.</details>

<img width="100%" src="https://mmlspark.blob.core.windows.net/graphics/emails/email_header_synapseml.jpg" alt="SynapseML" href="https://github.com/Azure/mmlspark">

Building production ready distributed machine learning pipelines can be a challenge for even the most seasoned researcher or engineer. We are excited to announce the release of SynapseML v0.10.0 (Previously MMLSpark), an open-source library that aims to simplify the creation of massively scalable machine learning pipelines. SynapseML unifies several existing ML Frameworks and new MSFT algorithms in a single, scalable API that’s usable across Python, R, Scala, Java, .NET, C#, and F#.

Highlights

<img width="600" src="https://mmlspark.blob.core.windows.net/graphics/emails/OpenAI_Logo.svg">	<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/Microsoft_.NET_logo.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/MLflow-logo-final-black.png">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/binder.svg">
OpenAI Language Models	.NET, C#, and F# Support	Full MLFlow Support	Live Demos in Browser
Embed 175-billion parameter models into your databases with ease	Use or train any SynapseML model from .NET	Quick and easy MLOps, model management, and autologging	Explore the SynapseML library with zero setup
Learn More	Getting Started Guide	Explore the Docs	Run in Browser

New Features

General ✨

• SynapseML now supports .NET, C#, F#, and other .NET ecosystem languages in addition to Scala, Python, and R. Please see our Setup Guide and LightGBM from .NET example for more details. (#1539, #1156, #1443)
• SynapseML is now usable from your browser with zero setup using Binder. Quickly explore our demos in Binder. (#1487, #1493)

Azure Cognitive Services for Big Data 🧠

• Added OpenAI GPT-3 Sentence Completion Transformer. Use this feature to embed 175-billion parameter language models into distributed pipelines and databases to solve a variety of general purpose NLP tasks across natural language and code. (#1495, #1541)
• Added an example of Sentence Completion with GPT-3 (#1564)
• Added support for Form Recognizer V3.0 (#1269)
• Improved MVAD usability with async training and better data validation (#1477)
• Upgraded the univariate anomaly detection version to v1.1-preview (#1440)
• Added a multivariate anomaly detection sample notebook (#1365)
• Added a Text to Speech example to cognitive service overview (#1350)
• Added opinion mining to TextSentiment Models (#1449)
• Fixed Azure Maps schemas (#1553)
• Removed modelID param validators in FormRecognizerV3 (#1551)
• Fixed form recognizer and form ontology learner issues (#1506)
• Fixed setServiceName python method in OpenAI (#1498)
• Fixed error in Text Analytics Analyze schema
• Improved error handling for MVAD (#1448, #1391)
• Removed unused concurrency parameter for MVAD (#1383)
• Improved robustness of flood risk notebook by adding polling (#1427)

Responsible AI at Scale 😇

• Added partial dependence plots (PDP) to allow for understanding how independent variables affect a model's prediction (#1426)
• Updated ICE/PDP documentation with PDP-based feature importance and additional examples (#1441, #1352)
• Added a notebook for ICE and PDP feature explainers (#1318)
• Updated data balance documentation to better describe how it can be used to ensure model fairness (#1540)

MLFlow 🔃

• Added documentation for MLFlow autologging (#1508)
• Added documentation on the SynapseML-MLFlow integration (#1428)

LightGBM on Spark 🌳

• Added the ability to pass in generic argument strings to LightGBM enabling many complex parameterizations (#1444)
• Added seed parameters to LightGBM (#1387)
• Added a method to get LightGBM native model string directly (#1515)
• Fixed issue with validation data creation during useSingleDataset mode (#1527)
• Fixed multiclass training with initial scores (#1526)
• Fixed saving LightGBM model iterations with early stopping (#1497)
• Fixed issue where chunk size parameter was incorrectly specified during data copy (#1490)
• Fixed issue where when empty partition is chosen as the main worker in singleDatasetMode (#1458)
• Fixed bug with data repartitioning in LightGBMRanker (#1368)
• Fixed outdated docs for useSingleDatasetMode (#1562)
• Refactored LightGBM class structure to improve logging and debugging (#1557)

Vowpal Wabbit 🐇

• Fixed issues with the saveNativeModel for the VWRegressionModel #1364 (#1366)
• Fixed issues with building quadratic interaction terms (#1460)

Isolation Forests 🌲

• Added an Isolation Forest Multivariate Anomaly Detection sample notebook (#1483)

Additional Updates

Maintenance 🔧

• Removed unused debugging code (#1546)
• Remove Synapse test exclusion for Explanation Dashboard notebook (#1531)
• Made python style checks verbose (#1532)
• Fixed library checking while installing library on Databricks cluster (#1488)
• Upgraded and fix Dockerfiles (#1472)
• Added Developer Docker Image build to pipeline (#1480)
• Fixed ADO area path in Issue Linker (#1464)
• Fix master version badge display
• Improved Databricks error reporting
• Updated azure cli to stop build errors
• Fixed SSL handshake flakiness
• Added itsdangerous as a dependency to ADB tests (#1412)
• Turned on debug for pr to work item workflow
• Pointed pr linker to official implementation
• Changed GitHub action trigger from pull_request_target to pull_request (#1413)
• Fixed issue where Unit Tests were not executing (#1409)
• Added Azure DevOps PR linker (#1394)
• Updated GH PAT name (#1389)
• Re-enable Synapse E2E Tests (#1517)
• Updated SynapseE2E Tests to Spark 3.2 (#1362)
• Fix ADO issue/pr linking (#1463)
• Cleaned up extra MVAD models and improved network resiliency (#1457)
• Updated azure blob client version (#1563)
• Fixed docker security vulnerability (#1561)
• Streamlined scalastyle hook (#1530)
• Updated CODEOWNERS (#1523)
• Updated OpenAI resource info (#1525)
• Fixed semantic PR checking (#1503)
• Updated docker images to remain compliant (#1500)
• Added component governance explicitly to build so timeout variable works (#1489)
• Fixed path for notebook test files in gitignore (#1485)
• Increased component governance timeout (#1482)
• Added conda caching to build
• Stopped build from failing after 1 hour
• Fixed flaking MVAD test
• Refactored build pipeline definitions
• Split Synapse tests into multiple test (#1377)
• Moved from ADO Pipelines to GitHub Workflows (#1406)

Website Improvements 💻

• Fixed MathJax expressions rendering (#1343)
• Fixed google analytics gtags (#1434)
• Corrected placement of BingSiteAuth.xml config (#1445, #1439)
• Fixed website security and upgrade docusaurus (#1545)
• Moveed Geospatial Services to its own folder (#1345)
• Bumped minimist from 1.2.5 to 1.2.6 in /website (#1455)
• Bumped node-forge from 1.2.1 to 1.3.0 in /website (#1451)
• Bumped prismjs from 1.25.0 to 1.27.0 in /website (#1430)
• Bumped follow-redirects from 1.14.7 to 1.14.8 in /website (#1402)
• Bumped nanoid from 3.1.23 to 3.2.0 in /website (#1355)
• Bumped shelljs from 0.8.4 to 0.8.5 in /website (#1347)
• Bumped follow-redirects from 1.14.1 to 1.14.7 in /website (#1348)
• Bumped cross-fetch from 3.1.4 to 3.1.5 in /website (#1496)
• Bumped async from 2.6.3 to 2.6.4 in /website (#1481)
• Pinned onnxmltools to a specific version (#1524)

Bug Fixes 🐞

• Fixed twitter sentiment detection notebook (#1544)
• Fixed issue with DataConversion serialization (#1505)
• Fixed typos in TestBase (#1501)
• Fixed issue in GridSpace python API (#1470)
• Fixed reflective class loading in IntelliJ (#1456)
• Removed verbose ComputeModelStatistics output and convert scoredLabelsCol to DoubleType (#1361)
• Fixed flaking in geospatial notebooks

Code Style 🎶

• Improved style checks using pre-commit (#1538, #1528, #1535)
• Formatted code and notebooks with Black style checker (#1522, #1520)

Documentation 📘

• Tabularized badges for readability (#1486)
• Added a PR template (#1418)
• Improved installation readme (#1369, #1422)
• Added a Security readme (#1511)
• Updated the Azure Synapse readme (#1372)
• Remove reference to custom maven resolver
• Added pointer to docs on synapse pool configuration
• Fixed typos in readme (#1516)

Contributor Spotlight

We are excited to highlight the contributions of the following SynapseML contributors:

<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/serena_color.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/ric.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/puneet.jpg">
Serena Ruan	Ric Serradas	Puneet Pruthi
Serena is a Software Engineer II on the Synapse team in Beijing and a force of nature. In this release, Serena has continued her prolific contribution steak by adding language support for .NET, C#, and F# and integrating SynapseML with MLFlow. Additionally, Serena has contributed several features to the MLFlow and Spark.NET open-source communities so that these systems can work better for every user. These contributions are just some of the many amazing things Serena has accomplished during this release, and her devotion and craft are pivotal to the ecosystem.	Ric is a Senior Engineering Manager on the OneNote team with a shining personality and drive to collaborate. In just a few weeks Ric hit the ground running by setting up an automated link between GitHub and Azure DevOps, building the first working version of SynapseE2E tests, and re-writing our entire build in GH Actions. Furthermore, Ric worked tirelessly through nights and weekends to land his contributions.	Puneet is a Senior Engineer on the SynapseML team with a knack for engineering systems and dockerization. Puneet's contributions to the library include architecting the new binder integration, driving our Synapse E2E tests to completion, and improving SynapseML’ s infrastructure around community engagement. Puneet is constantly thinking of ways to improve the community and we value his effort.
<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/mark_n.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/keerthi.jpg">	<img width="200px" src="https://mmlspark.blob.core.windows.net/graphics/people/yagna.jpg">
Mark Niehaus	Keerthi Yanda	Yagna Oruganti
Mark is a Senior Software Engineer on the SynapseML team with a deep knowledge of the .NET ecosystem and infrastructure development. In this release, Mark architected SynapseML’ s .NET binding blob publishing strategy, drove the OpenAI GPT-3 bindings to completion, and wrote a detailed GPT-3 walkthrough. Mark completed these projects while supporting the Time Series Insights service, speaking to his ability to keep multiple plates spinning at a time.	Keerthi is a Software Engineer II on the SynapseML team. Despite joining Microsoft just a few months ago, Keerthi has quickly learned the SynapseML ropes to take command of our integration with the Azure Synapse platform. Huge kudos to her for braving long build times, and daunting error messages to make sure SynapseML works out of the box on Synapse Analytics clusters.	Yagna is a Senior Data and Applied Scientist on the Industry AI team with a talent for building solutions that integrate many community tools to solve customer challenges. Yagna's first contribution to SynapseML was a masterpiece of a demo showing how to use Isolation Forests, MLFlow, Tabular SHAP, and the interpret-ml explanation dashboard in a single anomaly detection example.

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external, who helped create this version of SynapseML

Serena Ruan @serena-ruan, Eric Dettinger, Scott Votaw @svotaw, Puneet Pruthi @ppruthi, Ric Serradas @riserrad, Mark Niehaus @niehaus59, Kyle Rush @k-rush, Keerthi Yanda @KeerthiYandaOS, Yagna Oruganti @YagnaDeepika, Jason Wang @memoryz, Ilya Matiach @imatiach-msft, Yazeed Alaudah @yalaudah, Elena Zherdeva @ezherdeva, Kashyap Patel @ms-kashyap, Martha Laguna @martthalch @marthalc, Alex Li @liyzcj, Maria Guirguis @maguir, Alexandra Savelieva @alsavelv, @netang, Sudhindra Kovalam @SudhindraKovalam, Markus Cozowicz @eisber, Tom Finley, Markus Weimer, Jeff Zheng, James Verbus @jverbus, Chris Hoder, Misha Desai, Nellie Gustafsson, Eren Orbey, Beverly Kodhek, Louise Han @jr-MS, Justyna Lucznik, Kim Manis, Mitrabhanu Mohanty, Bogdan Crivat, Anand Raman, William T. Freeman, James Montemagno, Luis Quintanilla, Dennis Kennedy, Ryan Hurey, Jarno Ensio, Brian Mouncer, Steve Suh @suhsteve, Akshaya Annavajhala (AK), Guolin Ke, Tara Grumm, Niharika Dutta @Niharikadutta, Andrew Fogarty, Juanyong Duan, Weichen Xu @WeichenXu123, Spark.NET Team, ONNX Team, Azure Global, Vowpal Wabbit Team, LightGBM Team, MSFT Garage Team, MSR Outreach Team, Speech SDK Team, MLflow Team

Learn More

<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/synapseml_website.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/philosphy.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/dotnet_blog_card.jpg">
Visit our website for the latest docs, demos, and examples	Read more about SynapseML's GA release in the Microsoft Research Blog	Learn more about our .NET bindings and code generation system.
<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/synapseml_demo_series.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/large_scale_paper.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/openai_blog_card.jpg">
Watch a demonstration of SynapseML to create a multilingual search engine.	Read our Paper from IEEE Big Data '21	Explore our integration with the Azure OpenAI Service

<img width="100%" src="https://mmlspark.blob.core.windows.net/graphics/emails/email_header_synapseml.jpg" alt="SynapseML" href="https://github.com/Azure/mmlspark">

Building production ready distributed machine learning pipelines can be a challenge for even the most seasoned researcher or engineer. We are excited to announce the release of SynapseML v0.9.5 (Previously MMLSpark), an open-source library that aims to simplify the creation of massively scalable machine learning pipelines. SynapseML unifies several existing ML Frameworks and new MSFT algorithms in a single, scalable API that’s usable across Python, R, Scala, and Java.

Highlights

<img width="400" src="https://mmlspark.blob.core.windows.net/graphics/emails/azure_maps.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/anomaly_detector.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/scales.svg">	<img width="600" src="https://mmlspark.blob.core.windows.net/graphics/emails/tts.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/healcare.svg">
Geospatial Intelligence	Multivariate Anomaly Detection	Responsible AI at Scale	Text To Speech	Healthcare Analytics
Large-scale map and geocoding operations	Build custom time series anomaly detection systems	Distributed Conditional Expectation and Partial Dependence Analysis	East-to-use Neural Text to Speech for large datasets	Quickly understand entities and relationships in corpora of medical text.

New Features

Geospatial Intelligence 🗺️

• Added support for distributed geospatial queries backed by the Azure Maps API
• Added the geospatial usage overview (#1339)
• Explore how to use the geospatial intelligence services to analyze flood risks. (#1339)
• Added the AddressGeocoder transformer to map informal addresses to standardized adresses with latitude and longitude (#1294)
• Added the ReverseGeocoder transformer to map latitude and longitude measurements to standardized addresses. (#1339)
• Added the CheckPointInPolygon, to detect if latitude and longitude queries lie inside regions of interest (#1339)

Azure Cognitive Services for Big Data 🧠

• Added the Healthcare Analytics Transformer for extracting medical information, entities, and relationships for text. [Example Usage] (#1329)
• Added the FitMultivariateAnomaly estimator for training custom anomaly detection models on DataFrames of multivariate time series data (#1272)
• Added example notebook for Multivariate Anomaly Detector
• See how to train a custom Multivariate Anomaly detector in the Estimators reference docs (#1323)
• Added simplified Text Analytics transformers that support auto-batching (#1329)
• Added the TextToSpeech Transformer for transforming Dataframes of text to audio files with neural voice synthesis (#1320)
• Added the TextAnalyze transformer to support executing multiple text analytics workloads within a single API call (#1267, #1312)

Responsible AI at Scale 😇

• Added Individual Conditional Expectation explanations and Partial Dependence Plots with the ICETransformer. This tool gives detailed explanations of how features in opaque-box models affect the model prediction. (#1284)
• Learn about how to use the ICETransformer through an example with the Adult Census dataset

MLFlow 🔃

• Add MLFlow support for saving and loading SynapseML models (#1277)

LightGBM on Spark 🌳

• Improved LightGBM training performance 4x-10x by setting num_threads to be cores-1 (#1282)
• Added the predict_disable_shape_check in LightGBM (#1273)
• Reduced temporary file bloat by creating the LightGBM native temp directory lazily (#1326)
• Added logging for number of columns and rows when creating datasets, set useSingleDatasetMode=True by default (#1222)

Infrastructure 🏭

• SynapseML now installable from Maven Central!
• SynapseML now supports spark v3.2.x

Additional Updates

Bug Fixes 🐞

• Allowed FlattenBatch to propagate non-array values (#1286)
• Fixed flaky tests (#1342)
• Fixed website bugs and migrated docSearch (#1331)
• Fixed issue where IsolationForestModel does not properly exchange params with the inner model (#1330)
• Corrected the objective param when using fobj (#1292)
• Fixed issue where broadcasted sum in breeze 1.0 breaks in Spark 3.2.0 (#1299)
• Hotfixes for R test runners (#1283)
• fix installation instruction (#1268)
• Removing broadcast hint (#1255)
• fix install instructions (#1259)

Build 🏭

• bump algoliasearch-helper from 3.6.1 to 3.6.2 in /website (#1270)
• remove some deps that cause sec issues (#1264)

Documentation 📘

• Fixed broken link to CyberML notebook (#1322)
• Added website announcement bar (#1263)
• Updated and improve readme (#1262)
• Removed references to runme in contributing.md
• Supported Math expressions in website markdown (#1278)
• Corrected Synapse typo in website (#1335)

Maintenance 🔧

• Stopped lightGBM tests from timing out (#1315)
• Fixed r test flakiness (#1314)
• Updated VerifyLightGBMClassifier.scala (#1313)
• Update speech SDK test results
• Add in missing tests in build (#1300)
• Fix flaky build steps (#1298)
• Fix website telemetry (#1261)
• Add website telemetry (#1260)
• Added missing test classes to pipeline

Contributor Spotlight

We are excited to highlight the contributions of the following SynapseML contributors:

<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/serena.jpg">	<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/ilya%20(2).jpg">	<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/sudhindra.jpg">
Serena Ruan	Ilya Matiach	Sudhindra Kovalam
Serena is an engineer on the Azure Synapse team in Beijing. In this release, Serena has continued her unbelievable speed of contributions with support for Multivariate Anomaly Detection, MLFlow, and installation from Maven Central. These contributions are just a few of the many projects Serena has contributed since she joined just a few months ago!	Ilya is a prolific engineer on the Azure Machine Learning Boston team working on responsible AI. Ilya contributed LightGBM on Spark and worked tirelessly to improve and support this feature. Ilya has been an active contributor to the SynapseML project for 5 years and has built many of the tools in the library.	Sudhindra is an engineer on the Microsoft Maps team and has contributed intelligent geospatial APIs to SynapseML v0.9.5. Sudhindra developed new ways to automate generation of Spark code from swagger files allowing him to contribute a large suite of features rapidly.
<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/elena_2.jpg">	<img width="200" src="https://mmlspark.blob.core.windows.net/graphics/people/ta_interns%20(2).png">	<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/stuart%20(2).jpg">
Elena Zherdeva	The Text Analytics Explorer Interns	Stuart Leeks
Elena is an engineer on the CSX Data team working on building scalable responsible AI tools. In Elena's first contribution to SynapseML she added Individual Conditional Expectation plots at scale. She also contributed a detailed sample notebook that does a fantastic job of explaining key concepts in Responsible AI.	Samantha Konigsberg (top left), Preeti Pidatala (top right), and Victoria Johnston (bottom) were summer explorer interns on the text analytics team. They collaborated together to build new simplified API's for the text analytics service using the Java SDK layer. One of these contributions was the new Healthcare Analytics API in Spark. This was intern's first Scala project, making this contribution all the more impressive!	Stuart is Engineer on the Commercial Software Engineering. Stuart not only uses SynapseML to power customer engagements, but also directly contributes features needed to make his customers succeed. Stuart contributed support for the new Analyze Text API which allows users to perform multiple intelligent text tasks with a single API call. Stuart also added features to SynapseML’s Mini-batchers to improve their generality.

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML

Jason Wang @memoryz , Serena Ruan @serena-ruan, Ilya Matiach @imatiach-msft , Stuart Leeks @stuartleeks, Sudhindra Kovalam @SudhindraKovalam, Elena Zherdeva @ezherdeva, Preeti Pidatala @preetipidatala, Samantha Konigsberg @skonigs, Victoria Johnston @victoriajmicrosoft, Markus Cozowicz @eisber, Yazeed Alaudah @yalaudah, Suhas Mehta @suhas92, Kashyap Patel @ms-kashyap, Wenqing Xu @xuwq1993, Markus Weimer, Jeff Zheng, James Verbus @jverbus, Misha Desai, Nellie Gustafsson, Ruixin Xu, Eric Dettinger, Martha Laguna, Louise Han @jr-MS, Rashid Monin, Ali Emami, Clemens Schotte, Edward Un, Johannes Kebeck, Han Li, Assaf Israel @assafi, Tom Finley, Tomas Talius, Mitrabhanu Mohanty, Anand Raman, William T. Freeman, Ryan Hurey, Jarno Ensio, Brian Mouncer, Sharath Chandra, Beverly Kodhek, Nisheet Jain, Akshaya Annavajhala (AK), Euan Garden, Lev Novik, Guolin Ke, Tara Grumm, Ismaël Mejía, Keunhyun Oh, @martin0258, @sinnfashen, Dung Nguyen @nhymxu, @elswork, ONNX Team, Azure Global, Vowpal Wabbit Team, Light GBM Team, MSFT Garage Team, MSR Outreach Team, Speech SDK Team

Learn More

<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/synapseml_website.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/philosphy.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/General%20Availablity%20Card-01.jpg">
Visit our new website for the latest docs, demos, and examples	Read more about SynapseML's GA release in the Microsoft Research Blog	SynapseML is now generally available on Azure Synapse! Get started here.
<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/Multivariate%20Detector%20Card-01.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/large_scale_paper.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/Explainable%20Boosting%20Card-01.jpg">
Learn more about Multivariate Anomaly Detection in SynapseML	Read our Paper from IEEE Big Data '21	Sign up for the Private Preview of Explainable Boosting Machines in SynapseML

<img width="100%" src="https://mmlspark.blob.core.windows.net/graphics/emails/email_header_synapseml.jpg" alt="SynapseML" href="https://github.com/Azure/mmlspark">

Building production ready distributed machine learning pipelines can be a challenge for even the most seasoned researcher or engineer. We are excited to announce the release of SynapseML (Previously MMLSpark), an open-source library that aims to simplify the creation of massively scalable machine learning pipelines. SynapseML unifies several existing ML Frameworks and new MSFT algorithms in a single, scalable API that’s usable across Python, R, Scala, and Java.

Highlights

<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/synapse_recolor.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/onnxai-ar21_crop.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/scales.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/forms_and_translate.svg">	<img width="600" src="https://mmlspark.blob.core.windows.net/graphics/emails/vw-blue-dark-orange.svg">
General Availability on Synapse	ONNX on Spark	Responsible AI	Form Recognition and Translation	Reinforcement Learning
We are ready to help you productionalize on Azure Synapse Analytics	Distributed and hardware accelerated model inference on Spark	Understand opaque-box models, measure dataset biases, Explainable Boosting Machines	Parse PDFs and translate dataframes between over 100 languages	Contextual Bandit Reinforcement Learning with Vowpal Wabbit

New Features

General ✨

• Renamed and rebranded! Microsoft ML for Apache Spark is now SynapseML
• New modular library sub-packages for standalone install of each major set of features
• Support Spark 3.1.2 and Scala 2.12
• Support pip install synapseml for python bindings

ONNX on Spark 🕸

• ONNX model inference on Spark (#1152)
• Add documention and notebooks for ONNXModel evaluation (#1164)

Cognitive Services for Big Data🧠

• Added Multilingual Translation APIs (#1108) (Tutorial)
• Added FormRecognition APIs (Invoice, IDs, BusinessCards, Layouts, Custom Models) (#1099) (Tutorial)
• Added the FormOntologyLearner to extract meaningful "ontologies" of objects from collections of forms
• Add notebook to Create a Multilingual Search Engine from Forms
• Updated Text Analytics API to V3.1 (#1193)
• Add redactedText to PIIV3 (#1247)
• Added Personally Identifying Information (PII) identification
• Added Read API
• Added Conversation Transcription API
• Cognitive service now support data exfiltration protected (DEP) VNET allowing for individualized security solutions on Synapse Analytics (Learn More)
• Added support for the m4a codec in Speech to Text models
• Added predictive maintenance notebook
• Added Cognitive Service overview notebook
• Added support for linked service authentication in Synapse Analytics
• Simple no-code support in in Synapse Analytics

Responsible AI at Scale 😇

• Added Additive Shapley Explanations (SHAP) for understanding the predictions of opaque-box models (#1077)
• New API for Locally Interpretable Model-Agnostic Explanations (LIME), now supports background distributions text models, and has the same API as SHAP (#1077)
• Added Measure transformers for Data Balance Analysis (#1218)
• Add more notebook samples for documentation (#1043)
• Documentation and notebooks for Interpretability on Spark
• Introduce Responsible AI section on website (Interpretability + DataBalanceAnalysis) (#1241)
• Adding document and notebook for Data Balance Analysis (#1226)
• Explainable Boosting Machines for performant and interpretable ML (Private preview on Synapse Analytics only)

Vowpal Wabbit 🐇

• Added ContextualBandit reinforcement learning (#896)
• Added Vowpal Wabbit Overview Notebook

LightGBM 🌳

• Added matrix type parameter and improve logic to automatically infer dataset sparsity (#1052)
• Added several parameters related to dart boosting type (#1045)
• Added chunk size parameter for copying java data to native (#1041)
• Added number of threads parameter (#1055)
• Added custom objective function to LightGBM learners (#1054)
• Added singleton dataset mode for faster performance and reduced memory usage (#1066)
• Add num iteration and start iteration parameters to LightGBM model (#1024)
• Added the average precision metric (#1034)
• Added overview notebook for LightGBM
• Moved to new streaming API for dense data to reduce memory usage
• Tuned chinking code for faster performance

Build and Infrastructure Improvements 🏭

• New Docusaurus website generation system
• E2E Tests on Synapse Analytics (#1014)
• Split library into separately installable subprojects (#1073)
• Added a unified logging and telemetry system (#1019)
• Modernized R wrapper generation
• New Automated Python test generation (#998)
• New extensible code generation system
• New two-tiered security for build secrets
• Update ubuntu version to 18.04
• Automated back-up ACR images

Additional Updates

Bug Fixes 🐞

• Enable backwards compatibility for mmlspark python namespace imports (#1244)
• Fix publishing to maven and pypi (#1242)
• Fix broken link to notebook in Data Balance Analysis doc (#1240)
• min_data_in_leaf missing from dataset parameters in lightgbm (#1239)
• Fix performance issue in interpretability notebooks (#1238)
• Fixed cognitive service errors (#1176)
• Fixed flaky tests
• Rename NERPii to PII
• Fixed cog service test flakes
• Fixed setLinkedService issues in Synapse (#1177)
• Improved LGBM error message for invalid slot names (#1160)
• Fixed generated python code (#1121)
• Updated notebookUtils class path (#1118)
• Fixed LIME NaN weight output (#1117, #1112)
• Fixed Guava version issue in Azure Synapse and Databricks (#1103)
• Fixed flakiness in spark session stopping
• Fixed result parsing for forms
• Fixed explainers returning wrong results when targetClassesCol is specified
• Fixed CNTKModel issue due to catalyst bug on databricks (#1076)
• Fixed null handling in bing image response (#1067)
• Avoided strange issue with databricks json parser
• Fixed dependency exclusions and build secret querying
• Fixed issue in tabular lime sampler (#1058)
• Updated Bing search URLs (#1048)
• Refactored python wrappers to use common class (#758)
• Updated java params patch (#1027)
• Added missing returns in new python lightGBM model methods
• Stop R binding generation from failing silently
• Fixed conversation transcription participant column functionality
• Reduce verbosity to prevent RPC disassociated errors
• Fixed performance slip in Featurize
• Added timeout logic for speech to text
• Added ffmpeg time limit enforcing for flaky streams (#1001)
• Fixed upload python whl file to blob(#1000)
• Cleaned up python tests (#994)
• Fixed read schemas (#988)
• Made HTTP default concurrent timeout infinite
• Made HTTP rate limiting retry indefinitely
• Recommender Patch for Spark 3 Update (#982)
• Fix typo in text sentiment schema
• Changed ints to longs for offset and duration in STT
• Fixed processing sparse vector size
• Fixed Double User agent setting bug
• Fixed build warnings (#1080)
• Fixed build for new intellij
• Fixed livy dependency resolution
• Fixed pom for sbt dependencies (#1202)
• Fixed bug in testGen parallelism
• Auto-update packages in docker
• remove unused code
• Fix codecov logging of wrapper generation (#1098)
• Fix badge publishing
• Remove issue in scalastyle file for new IJ

Documentation 📘

• Add explicit pointer to HDI install
• fix typo (#990)
• Bump python install to top to make it clearer
• Add example CyberML notebook (#958)
• Add CyberML link to README.md (#989)

New Contributor Spotlight

We are excited to welcome several new developers to the SynapseML project.

<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/serena.jpg">	<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/jason.jpg">	<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/wenqing.jpg">
Serena Ruan	Jason Wang	Wenqing Xu
Serena is an Engineer on the Azure Synapse team in Beijing. Within her first months working on SynapseML, Serena contributed Forms and Translator cognitive services, a unified logging and telemetry system, notebooks and documentation for every transformer and estimator, and a new docusaurus-based website.	Jason is a Principal Engineer on Microsoft's DSP team and is focused on large-scale responsible AI. Jason started his contribution streak with a new API for model explainability that unifies both SHAP and LIME. Jason has also contributed ONNX on Spark which dramatically broadens the scope of models that can be used in SynapseML.	Wenqing is a software engineer on the Azure Synapse team in Beijing. Wenqing has been instrumental in preparing SynapseML for General Availability. In particular, Wenqing added support for linked service authentication of cognitive services, extended E2E testing to Synapse Analytics, and added the PII identification service.
<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/kashyap.jpg">	<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/rohit.jpg">	<img width="200" style="border-radius:50%" src="https://mmlspark.blob.core.windows.net/graphics/people/jack.jpg">
Kashyap Patel	Rohit Agrawal	Jack Gerrits
Kashyap is an Engineer on Microsoft's DSP team working on improving the fairness of machine learning models. Kashyap contributed tools for assessing dataset bias without requiring a labelled dataset or model.	Rohit is a Senior Engineer on Microsoft's Cognitive Service team working on large-scale orchestration of intelligent services. Rohit modernized our Text Analytics Stack by updating to v3.0 and laid the groundwork for E2E testing on Synapse Analytics.	Jack is a Senior Engineer on the decision service and reinforcement learning team at Microsoft Research NYC. Jack contributed support for contextual bandit reinforcement learning with Vowpal Wabbit.

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML

Jason Wang, Serena Ruan, Ilya Matiach, Jack Gerrits, Kashyap Patel, Wenqing Xu, Markus Weimer, Jeff Zheng, Nellie Gustafsson, Ruixin Xu, Martha Laguna, Markus Cozowicz, Rohit Agrawal, Daniel Ciborowski, Jako Tinkus, Tom Finley, Tomas Talius, Mitrabhanu Mohanty, Roy Levin, Anand Raman, William T. Freeman, Ryan Hurey, Sharath Chandra, Beverly Kodhek, Assaf Israel, Nisheet Jain, Ryan Hurey, Miguel Fierro, Dotan Patrich, Akshaya Annavajhala (AK), Euan Garden, Lev Novik, Guolin Ke, Tara Grumm, Keunhyun Oh, Vanunts Arsenii, Alexandr Severinov, David Lacalle Castillo, Ryosuke Horiuchi, Ashish Solanki, Matthieu Maitre, ONNX Team, Azure Global, Vowpal Wabbit Team, Light GBM Team, MSFT Garage Team, MSR Outreach Team, Speech SDK Team

Learn More

<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/website.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/philosphy.svg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/synapse_recolor.svg">
Visit our new website for the latest docs, demos, and examples	Read more about SynapseML in the Microsoft Research Blog	Get started with SynapseML on Azure Synapse Analytics
<img width="500" src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/322907iC428A7A3896E5862/image-size/large?v=v2&px=999">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/paper_2.jpg">	<img width="500" src="https://mmlspark.blob.core.windows.net/graphics/emails/webinar.jpg">
Read the Synapse Analytics Ignite Announcements	Read our Paper from IEEE Big Data '21	Watch our ODSC Webinar on working with AI services at scale