PyPI · #2521 most downloaded on PyPI

kfp-server-api

2.17.02.6M downloads/mokubeflow/pipelines ↗New releases in your daily digest, changelog included.

Kubeflow Pipelines API

Last release 2 months ago
09 Jul 2026
Ships fairly regularly
a new release about every 2 months
Most releases are documented
notes for 44 of 49 stable releases
Nothing withdrawn
no release was ever pulled
7 years old
91 releases · first in 2019

Release timeline

91 releases since 2019

20192020202120222023202420252026

Releases

2.0.0a313 Jul 2022pre-releaseBreaking
Upgrade Notes with notices and breaking changes
To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.7 above) by running:

python3 -m pip install kfp-server-api==2.0.0-alpha.3 --upgrade

Refer to:

Upgrade Notes with notices and breaking changes

Change Log

NOTE, kfp python SDK is NOT included and released separately.
Open source →
Additional notes
⚠ BREAKING CHANGES

sdk: make CLI output consistent, readable, and usable (#7739)

Features

Upgrade argo-workflow to v3.3.8 (#8009) (8bee292)

api: add runtime_config.parameter_values with string typed values (#7734) (5b64733)

backend: Add ExecutionSpec Interface (#7531) (8cf022c)

backend: Expand ExecutionSpec Interface (#7766) (06c9310)

backend: update minio-go to support irsa (#7946) (7923ba3)

backend: use cert-manager for cache server cert (#7843) (784f9fa)

component: Support BigQuery drop model job components (affbc09)

component: Support BigQuery ML advanced weights job component (f6b560a)

component: Support BigQuery ML centroids job components (45cfb91)

component: Support BigQuery ML reconstruction loss and trial info job components (d2f33c8)

component: Support BigQuery ML weights job component (4183344)

components: Add display name to import model evaluation component (9b12f6e)

components: Outputting system.artifact and google.BQTable as a replacement for the google.VertexBatchPredictionJob artifact. (18b3ab8)

conformance: adapt remaining tests to KF env (#7670) (bcbdbd0)

conformance: containerize KFP conformance test. (#7738) (3867496)

frontend: Add ability to filter by visualization type (#7906) (f5a1264)

frontend: Add loading and error states for HTML and Markdown displays (#7989) (6cbc10b)

frontend: Add V2 Run Comparison page (#7793) (f822eea)

frontend: Create metrics tabs (#7905) (e683290)

frontend: Create sections and run list for KFPv2 Run Comparison page (#7882) (146dae7)

frontend: Create two-level dropdown (#7933) (832858b)

frontend: Display HTML and Markdown files (#7981) (cfe3278)

frontend: Display multi-level dropdown on KFPv2 Run Comparison page (#7943) (53d8f7e)

frontend: Display two-panel layout and confusion matrices (#7966) (617ad83)

frontend: NewRunV2 page and NewRunParametersV2 page. (#7769) (e246ffb)

frontend: Request MLMD information for KFPv2 Run Comparison (#7897) (3bcdeb6)

frontend: Update compare page and banner states based on run versions and count (#7844) (468d780)

google-cloud: add explanation_metadata_artifact input arg to custom Batch Prediction component. (ad364d9)

google-cloud: Add GCPResources to ImportModelEvaluation component. (ba82969)

google-cloud: Add new components 'evaluation_data_sampler' and 'evaluation_data_splitter'. (d6f9265)

google-cloud: Change evaluation preprocessing component output type. (059afb0)

google-cloud: Modify GCPC evaluation templates. (176d3ff)

google-cloud: Release eval changes with v0.3 (d2fd41b)

google-cloud: Release notes for 1.0.12 and 1.0.13 (e42d9d2)

google-cloud: Update AutoML pipelines to reference next GCPC package release (1.0.8). (c849871)

IR: add is_optional to ParameterSpec proto (#7704) (f960a52)

IR: adds IteratorPolicy to PipelineTaskSpec proto for support of parallelism setting on ParallelFor (#7804) (a86ae9b)

kfp: implement config file support for registry client (#7908) (4f01b7e)

sdk: Add default registry context (#7948) (aab7fda)

sdk: Add function to sdk client for terminating run (#7835) (9ffcb2e)

sdk: add retry policy support to kfp v2 (#7867) (850a750)

sdk: add support for IfPresentPlaceholder and ConcatPlaceholder strings (#7795) (de0b824)

sdk: compile JSON with formatting (#7712) (2570922)

sdk: enable compilation of primitive components (#7580) (62972ec)

sdk: enable component compilation via cli (#7649) (c6125ff)

sdk: enable component compilation via component decorator (#7554) (6935a47)

sdk: enable read in component using IR (#7689) (cbc2ac5)

sdk: Implement Registry client (#7597) (25e4c58)

sdk: make CLI output consistent, readable, and usable (#7739) (b0db428)

sdk: make client return values consistent (#7659) (1ae6e11)

sdk: only use IR when saving component (#7690) (a6ec449)

sdk: throw informative exception on uncalled @pipeline decorator (#7913) (ecb55bd)

sdk: validate pipeline resource name before submission (#7713) (101d243)

sdk/client: implements overriding caching options at submission (#7912) (b95c5ab)

Bug Fixes

Update proxy agent image to fix CVE-2022-1292 (#8019) (5724849)

components: fix launch_crd.py to be compatible with kubernetes python SDK API. Fixes #7984 (#7985) (c7bf68e)

Update GCP marketplace deployer base image (#8018) (1d871fe)

Update mysql image to fix CVE-2022-1292 (#8017) (3bbc1d9)

backend: "Started at", "Finished at" and "Duration" display error. (#7876) (d11b097)

backend: change downloaded IR from JSON to YAML. Fixes #7673 (#7768) (c7b56d6)

backend: fixes healthz response by adding json content type. Fixes #7525 (#7532) (bd38cb5)

backend: Upgrade grpc-gateway patch version to enable correct swagger config (#7788) (cf1b873)

cache: Change kfp cache cert names to match cert-manager (#7538) (2a82a0f)

components: Pin pip version to fix apache-beam[gcp] installation issue (#8016) (bb83581)

components: Preserve empty features in explanation_spec (4c70132)

components/google-cloud: custom job util should add the nfs_mounts as a sub node of worker_pool_spec. (6296c18)

deps: update dependency npm to v8 [security] (#7885) (624f197)

frontend: Change "Layer" to unclickable. (#7737) (b7cf1fb)

frontend: Fix run comparison filter (#7833) (f41b545)

frontend: Fix Run Comparison refresh button (#7872) (85c7901)

frontend: Move all V2 flag check to V2_ALPHA (#7773) (6cc33f5)

ir: fix kfp-pipeline-spec proto imports (#7754) (436d7af)

manifests: Update snapshot to 2.0.0-alpha.2 (#7852) (d099864)

sdk: Fix corner cases and implement validation (#7763) (eaa8ec7)

sdk: fix extract docstring in load component (#7921) (a020c9b)

sdk: fix ifpresent and concat placeholder compile (#7930) (f877a1f)

sdk: fix load v1 if present placeholder (#7765) (0cf8173)

sdk: fix placeholder read/write logic (#7928) (ad458d5)

test: chmod +x test_load_all_components.sh (#7865) (95d0f7f)

test: Make exit handler test install KFP SDK from source (#7785) (d25bbc8)

Other Pull Requests

Fix the problem that AutoML Tabular pipeline could fail when using large number of features. (e.g., > 500 features). (9f6689e)

Move new bq components from experimental to v1 (527d9e1)

remove kms key name from the drop model interface. (660e847)

Internal change (6372420)

feat(frontend) Support LIST, STRUCT type in RuntimeConfig parameters (#7991) (2d81e7b)

chore(components/pytorch):kserve migration (#7615) (49c3587)

add registry json to package data (#7987) (7253203)

feat(frontend) Convert string-type RuntimeConfig parameters to real-type (currently support num, bool, str) (#7919) (b01000a)

Fix AutoML Tabular pipeline always runs evaluation. (f2d3d4d)

feat(components):Add pipeline metadata to import evaluation component (1fe3641)

Use FTE image directly to launch FTE component (e9221ec)

Add automl_tabular_pipeline pipeline for Tabular Workflow. Pin Tabular Workflow images to 20220629_0725_RC00. (5373af5)

Make generate analyze/transform data and split materialized data as components (9d3ac26)

For built-in algorithms, the transform_config input is expected to be a GCS file path. The file contains transformations to apply. (736f7fe)

weight_column_name -> weight_column and target_column_name -> target_column for Tables v1 and skip_architecture_search pipelines (a4a7462)

Make calculation logic in SDK helper function run inside a component for Tables v1 and skip_architecture_search pipelines (9b23034)

Use feature transform engine docker image for related components (7dc6448)

add algorithm to pipeline, also switch the default alg to be AMI. (4ca9993)

Update skip_architecture_search pipeline (a8aa75d)

google-cloud-pipeline-components-1.0.11 release notes (aedd748)

Added dataflow_service_account to specify custom service account to run dataflow jobs for stats_and_example_gen and transform components. (745abc2)

Update Tabular workflow to reference 1.0.10 launcher image. (419c8fa)

Change JSON to primitive types for Tables v1, built-in algorithm and internal pipelines (daf40bb)

move generating feature ranking to utils to be available in sdk (50ea394)

The added field transformations_path enables users to specify transformations in a file and provide a path to the file as an input. (5e39259)

add algorithm parameter to be configurable by users. (c5df3e8)

Add a study_spec_parameters_override parameter to Tables v1 and stage_1 component pipelines and update handling logic (7f886db)

Add explanation metadata to upload model component for L2L pipelines (5898a19)

Use 'unmanaged_container_model' instead of 'model' in infra validator component for all pipelines (768d7a9)

Remove use of input_directionary_to_parameter in v1 Tables and component-specific pipelines only and read from executor_input instead (958a181)

Feature transform engine config planner to generate training schema & instance baseline (b0a7ba5)

Decouple transform config planner from metadata (fcb8f5f)

Make Tables ensemble also output explanation_metadata artifact. (ed8e7e9)

Internal change (26fb337)

Internal change (5b74b21)

Merge pull request #7776 from evanseabrook:fix/empty-explanation-features (e3fc7cd)

Fix AutoML Tables pipeline and builtin pipelines on VPC-SC environment. (023680d)

Make stats-gen component to support running with example-gen only mode (f85dbe6)

Remove trailing whitespace (cf0a2ca)

KFP component for ml.detect_anomalies (d05110e)

Adding ML.GLOBAL_EXPLAIN KFP BQ Component. (b1c803b)

Add distill + evaluation pipeline for Tables (83e618e)

KFP component for ml.forecast (79b3335)

KFP component for ml.explain_forecast (44d9575)

Add ML.ARIMA_EVALUATE in component.yaml (dd8ccd2)

Adding ML.Recommend KFP BQ component. (56c9111)

Add ML.ARIMA_COEFFICIENTS in component.yaml (7af13b6)

Adding ML.FEATURE_IMPORTANCE KFP BQ Component. (b067fc6)

Adding ML.PRINCIPAL_COMPONENTS and ML.PRINCIPAL_COMPONENT_INFO KFP BQ component. (e7dbbcb)

Adding ML. ROC_CURVE KFP BQ Component. (640ef6f)

Merge distill_skip_evaluation and skip_evaluation pipelines with default pipeline using dsl.Condition (e95b9f5)

Adding ML.FEATURE_INFO KFP BQ Component. (cdd5760)

Adding ML.CONFUSION_MATRIX KFP BQ Component. (8b9fe02)

Wide and Deep and Tabnet models both now use the Feature Transform Engine pipeline instead of the Transform component. (1365f83)

fix(frontend) Regenerate the Typescript client library from swagger to match the backend changes in #7788 (#7847) (2c065d3)

add additional experiments in distillation pipeline. (db9ee59)

Adding ML.TRAINING_INFO KFP and ML.EXPLAIN_PREDICT BQ Component. (a6d0b86)

FTE now outputs training schema. (be317d0)

FTE transform config passed as path to config file instead of directly as string to FTE. (b83327f)

Add model eval component to built-in algorithm default pipelines (8829400)

add README with project description (#7782) (2c27475)

Renamed "Feature Transform Engine Transform Configuration" component to "Transform Configuration Planner" for clarity. (e38df3a)

feat[deployment]: add startup probe to pipeline API deployment (#7741) (dd98b1e)

Update default machine type to c2-standard-16 for built-in algorithms Custom and HyperparameterTuning Jobs (93ea81b)

Removed default location setting from AutoML components and documentation. (0932905)

Add window config to ARIMA pipeline. (f77ebc8)

Stats Gen and Feature Transform Engine pipeline integration. (1e105d3)

Uses BigQuery batch queries in ARIMA pipeline after first 50 queries. (9e7de60)

Typo fix (99b278f)

Allow ARIMA pipeline to overwrite output tables. (6efb7e5)

Replace custom copy_table component with BQ first-party query component. (59c8c6f)

support vpc in feature selection (a446ec6)

Fix failure when running distillation pipeline due to unexpected additional_experiments pipeline parameter. (076b050)

fix cli upload pipeline version (#7722) (49cdb42)

Create Python-based component to set study_spec_parameters for Wide & Deep HyperparameterTuningJobs (4d4b88b)

Create Python-based component to set study_spec_parameters for TabNet HyperparameterTuningJobs (8fa1235)

Add default Wide & Deep study_spec_parameters configs and add helper function to utils.py to get parameters (638a38e)

Internal testing change (669374b)

Update google-cloud-pipeline-components version and release notes (10d9fe3)

Updates Workbench Executor component for Pipelines to support kernel_spec. (b40f450)

Add default TabNet study_spec_parameters configs for different dataset sizes and search space modes and helper function to get the parameters (f877b6e)

Add init file to container/experimental directory to recognize as a python module. (8c20a2c)

add E2E test cases for classification type feature attribution pipeline. (0eb9d3b)

Minor changes to the feature transform engine and transform configuration component specs to support their integration. (ab2a84e)

2.0.0a26 May 2022pre-releaseBreaking

Upgrade Notes with notices and breaking changes

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api==2.0.0-alpha.2--upgrade

Refer to:

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

⚠ BREAKING CHANGES

sdk: promote v2 compiler command and deprecate v1 command (#7508)

Features

api: Update IR with RetryPolicy (#7581) (27d5404)
components: Add AutoML Tables default pipeline with evaluation (1265649)
components: Create new import model evaluation component (00eb97c)
components: Improve AutoML Tables default pipeline with evaluation component (b98a2ab)
components: Take problem type as input rather than schema URI (d1c0c75)
components: Tests for import model evaluation (cdddaef)
components/google-cloud: Add support for env variables in Custom_Job component. (5080288)
components/google-cloud: Support parametrized input for reserved_ip_range and other Vertex Training parameters in custom job utility. (be682e4)
conformance: adapt pipeline API tests to KF environment (#7647) (694a274)
conformance: adapt pipeline version API tests to run in KF (#7653) (913277a)
conformance: adapt run api tests to KF env (#7658) (8b11551)
conformance: Enable experiment API tests in KF mode. (#7596) (1265113)
frontend: Automatically fetch MLMD for Run status update in V2. (#7636) (d1a12cb)
frontend: Enable KFPv2 UI by default by creating a new feature key #7610 (#7624) (5fb63c7)
frontend: Support default feature flags update in localStorage. Fix #7379 (#7537) (7f4e9a4)
google-cloud: Add internal flag to ImportModelEvaluation component (e7985b8)
google-cloud: Add new pipeline template for Vertex Evaluation (8b81223)
sdk: Accept newer typing-extensions versions (#7632) (1118f48)
sdk: add .list_pipeline_versions and .unarchive_experiment methods to Client (#7563) (2494447)
sdk: add archive, unarchive, and delete run methods and tidy docstrings (#7562) (2636727)
sdk: add autocomplete and version options to kfp cli (#7567) (fbfeadd)
sdk: add client methods to cli (#7606) (027ac32)
sdk: add noun aliasing to cli (#7569) (e077d01)
sdk: improve cli help text (#7618) (ec00813)
sdk: port cli code from v1 to v2 (#7547) (d46fafe)
sdk: promote v2 compiler command and deprecate v1 command (#7508) (4eeb420)
sdk: rename CLI methods to 'create' (#7607) (652ec3b)
sdk: use custom basemodel and remove pydantic (#7639) (5da3826)
sdk: use dsl noun group for compile CLI commands (#7603) (e12ac39)

Bug Fixes

backend: fix simple loop bug (#7578) (1180e10)
components: Add retries to queries hitting BQ write quota. (c19facc)
frontend: Convert PipelineSpec format from json to yaml #7444 (#7591) (ec32711)
frontend: Update sample pipelines in mock-backend to use yaml format (#7570) (cdbfbe8)
frontend: Use yaml mode for PipelineSpec editor view (#7637) (7d6edee)
sdk: some cli fixes (#7668) (18f0842)

Other Pull Requests

Update test data for evaluation classification feature attribution pipeline. (d822f6a)
Internal change (6ceea6f)
Pipeline level exit handler and v2 test (#7642) (98a58c7)
generate feature selection tuning pipeline and test utils. (c6aaaef)
Internal change. (afe07f1)
Update TabNet and Wide & Deep parameter descriptions (b30fcee)
add deprecation warning to dsl-compile (#7628) (6cecfd9)
Update automl first-party pipeline jsons. (bc73718)
internal change. (2038336)
Add additional experiment flag in AutoML Tables related pipelines. (0dea8ba)
Fix the problem that skip_architecture_search pipeline is using the wrong compiled json. (9e4bd6a)
use click for components commands (#7559) (2db431b)
use click for dsl-compile command (#7560) (a554844)
Add feature transform engine component definition. (94a4459)
Update evaluation yaml to v0.2 (d452261)
Expose eval_frequency_secs and eval_steps as separate inputs in the built-in algorithm HPT component (88e4066)
Add e2e tests for ARIMA pipeline and release to GCPC experimental dir. (5a8202e)
Add template JSON pipeline spec for running evaluation on a managed GCP Vertex model. (0c91279)
Update documentation for Dataproc Serverless components v1.0. (c64879f)
Use if:cond:then when specifying image name in built-in algorithm hyperparameter tuning job component and add separate hyperparameter tuning job default pipelines for TabNet and Wide & Deep (493ce18)
Add gcp_resources in the eval component output (f97a8aa)
Add downsampled_test_split_json to example_and_stats_gen component. It is a JSON object that contains the downsampled test split information. (7884f46)
Dataproc Serverless components v1.0 launch. (6949964)
add typestub libraries (#7451) (d74d7b9)

Open source →

2.0.0a016 Mar 2022pre-releaseBreaking

Upgrade Notes with notices and breaking changes

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api==2.0.0-alpha.0 --upgrade

Refer to:

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

⚠ BREAKING CHANGES

sdk: rename commands, arguments parameters to command, args (#7391)
sdk: throw error for output_component_file (#7390)
sdk: move v2 to main namespace (#7376)
sdk: move v1 to deprecated folder (#7291)
sdk: drop support for python 3.6 (#7303)
Use Argo Emissary Executor instead of Docker by default. Partial #5718 (#7137)
sdk.v2: Block task dependency referencing tasks inside a sibling condition or loop group. (#7050)

Features

backend: add new endpoint that get pipeline by name (#7004) (81438f3)
backend: Forbid unarchive runs that belog to archived experiment (#7147) (96a6a3b)
backend: Upgrade go client for MLMD to 1.5.0. Partial #7029 (#7101) (ba59a64)
backend: Upgrade MLMD to 1.5.0 (#6996) (7157764)
chore: add labels and error to the gcp_resources.proto (bc56df6)
chore: add labels and error to the gcp_resources.proto (3da295e)
component: add BQ export model component (28c666f)
component: add generic custom training job component (a74eba1)
component: adding bq create model component (fef8c03)
component: adding bq evaluate model component (27d2ab5)
component: adding bq predict model component (90e759c)
component: adding component yaml and tests for bq query job component (accacf6)
component: Adds Vertex Pipelines notification email component. (e2687ce)
component: remove training_filter_split, validation_filter_split, test_filter_split as they are not working (ff99a98)
component: rename bq query job remote runner to bq remote runner for more bq components. (57cc5c3)
components: Add a KServe component (#6881) (72185ff)
components: Add GetWorkerPoolSpecsOp and update sample notebook to include training new model from tuning results (30308ea)
components: Add IsMetricBeyondThresholdOp util function for HP tuning job; add tests for all util components (37b3397)
components: add nfs mount support for create_custom_training_job_op_from_component (0e316b6)
components: Add util components for getting best trial/hyperparameters; delete HP Tuning Job lightweight python version to avoid potential confusion (a62bdbe)
components: add utility library for google artifacts, and use the library for FPCs. Also fix BQML export model output param error. (a17e1d3)
components: cleanup the deprecated fields in v1 model upload component. (9738726)
components: create gcpc v1. (5baa487)
components: Expose reserved_ip_ranges to custom job component (89c741d)
components: Implement cancellation propagation (61d9210)
components: Implement delete_endpoint (1d3eaf7)
components: Implement delete_model (284b46e)
components: Implement undeploy_model component (3bc17be)
components: Update batch prediction component to use artifact.uri instead of artifact.metadata['artifactUri'] (7d5690a)
components: update model upload component to accept the unmanaged model artifact (f341b67)
components/google-cloud: Accept UnmanagedContainerModel artifact in Batch Prediction component (e39eed1)
components/google-cloud: Commit experimental Vertex Evaluation component. (4cf9e5b)
components/google-cloud: Update ground_truth_column and update public container for Vertex Evaluation component. (b892888)
doc: Update kubeflow-e2e-mnist notebook to use KServe (#7355) (d93b390)
frontend: Upgrade MLMD client to 1.5.0. Partial #7029 (#7102) (ef8e092)
frontend: V2 create run from UI (#7001) (a1bef77)
google-cloud: Add 'generate_feature_attribution' and executor_output to Evaluation component inputs. (25508b6)
google-cloud: Fix bug when ExplanationMetadata.InputMetadata field is provided the batch prediction job component (b81b00c)
google-cloud: Update Evaluation Component to let the dataflow service account be an optional input (10df58c)
sdk: Add enable_job method to client (#7239) (e076434)
sdk: add metadata field for importer (#7112) (5a29d7f)
sdk: alias v2 imports and provide DeprecationWarning (#7398) (5fb126a)
sdk: client to support KFP v2 API. (#7411) (b741926)
sdk: drop support for python 3.6 (#7303) (54d5789)
sdk: Improve CLI experience for archiving experiments, managing recurring runs and listing resources (#6934) (ddbfcde)
sdk: move v1 to deprecated folder (#7291) (b693909)
sdk: move v2 to main namespace (#7376) (8bb0d0e)
sdk: rename commands, arguments parameters to command, args (#7391) (82d17ff)
sdk: Support getting pipeline status in exit handler. (#7309) (ab805f8)
sdk: throw error for output_component_file (#7390) (d695ec5)
adds secrets to ml-pipeline-ui-artifact deployments in user profiles (#5864) (c6c826d)
Use Argo Emissary Executor instead of Docker by default. Partial #5718 (#7137) (bea751c)
v2: control flow - conditional (#7088) (048218b)
v2: iterator and sub-DAG basic support (#6985) (713558c)

Bug Fixes

backend: AIP-5634: Add in LRU cache and temp file rotation for metadata-writer. Relates to #4347 (#7199) (3f4d686)
backend: Fix v1 caching to read Argo template from container env. Fixes #7266 (#7267) (8061da3)
backend: make cache-deployer generate CSR using kubelet-serving signerName (#7273) (5ba9653)
backend: missing permissions for scheduled workflows (#7155) (c09c366)
backend: Read env var for default service account in multi-user mode. Fix #7336 (#7337) (2afd52f)
backend: set correct permissions for local directory (#7212) (6b7adfa)
backend: Update API RuntimeConfig to use protobuf.Value. Fixes #7119 (#7231) (3e734ed)
backend: Update composite-controller.yaml to fix https://github.com/kubeflow/manifests/issues/2133 (#7311) (b79064f)
components: fix gcp component build script (#7423) (d719485)
components: Fix gcp component that causes error running xgboost_training_cm sample. Fixes #7363 (#7364) (6965dba)
components: fix legacy gcp dataproc component (#7424) (2b29a1e)
components: remove rsync usage in gcp container build (#7422) (dc3ca09)
components/google-cloud: Fix Help string for dataflow python_module_path (9828007)
docs: Update release doc with env var assignment. (#7170) (6fac617)
frontend: Batch upgrade for fundamental dependency blocks. (Node, CRA, tailwind and more) Fix #7148 (#7144) (48e1e29)
frontend: fixes link to Vertex AI workbench in GettingStarted page. (#7227) (145bc23)
frontend: Fulfill name/state/workspace for MLMD Executions and Artifacts list. (#6989) (181dd92)
manifests: Fix typo in default executor name (#7159) (9ba1a4f)
manifests: Remove redundant secret value in profile controller (#7087) (af9d058)
manifests: Upgrade rbac to v1 for multi-user mode (#7097) (632e71e)
manifests: Upgrade third_party CRDs to v1 for k8s 1.22. Partial #6014 (#7098) (f3a1b65)
manifests: Use number for cloudsql liveness port (#7252) (9cc892e)
Profile Controller: Disable indent when json dumps (#6982) (a0a8f1d)
samples: Replace deprecated tfx kubeflow example (#7342) (0c0e4b6)
samples: Update tensorboard_minio.py to load component from a permanent link (#7211) (e74c7bd)
samples: Use TFX library version for parameterized_tfx_oss. Fix #6974 (#6993) (3cfff3d)
sdk: Add in filter to list_pipeline_versions SDK method (#7223) (808ff5d)
sdk: add verify_ssl to kfp client, fixes #4569 (#7174) (060588e)
sdk: bumps up absl version to 1.x for compatibility with TF 2.8+. Fixes #7171 (#7172) (22a8fdf)
sdk: fix bug in checking values in _param_values. Fixes #6799 (#6965) (a71fce0)
sdk: Fix create_runtime_artifact not covering all types. (#7168) (83d54d3)
sdk: fix input/outputspec and positional arguments (#6980) (85d7433)
sdk: require google-api-core>=1.31.5, >=2.3.2 (#7377) (7f67130)
sdk: Respect namespace if set on client (#7056) (fc2d05f)
sdk.v2: fix boolean error for inputs (#7041) (1a0bef7)
sdk.v2: Fix importer not using correct output artifact type (#7235) (1acb90f)
give components/build_image.sh execution permission (#7222) (c86df21)
sdk: Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest (#7141) (577902c)
sdk.v2: Block task dependency referencing tasks inside a sibling condition or loop group. (#7050) (6dfaeeb)
sdk.v2: fix regression on nested loops (#6990) (c1b6715)

Other Pull Requests

Fix unit tests for forecasting components. (1f73a59)
Convert forecasting component types from String to JsonArray or JsonObject. (e406436)
Remove ForecastingTrainingWithExperimentsOp component. (1163510)
Add AutoML Tabular TabNet trainer component and default pipeline (4ba76b9)
Add AutoML Tables Built-in Algorithm HyperparameterTuning Job components and Wide & Deep HyperparameterTuning Job pipeline to GCPC SDK. (a7fa2fc)
Use unmanaged_container_model for model_upload for AutoML Tables pipelines (4cf515a)
Propagate pipeline cancellation. (0f244f4)
Update evaluation component documentations (ac730c6)
add prediction type to feature selection component. (854ab49)
Fix v1.wait_gcp_resources. (7ad9491)
In gapic library by default there is no retry configured, causing polling batch prediction job status to fail due to transient errors like ServiceUnavailable. (5eb08a2)
Lower max retries on HTTP errors. (f3ceb60)
feat(component):add enable web access to create_custom_training_job_op_from_component (4e1f19e)
deletes old FPC tracking file (0f104d3)
Renamed FPC components that do not match naming conventions (3941bc7)
Fix parameters and align documentation with existing components. (a9f18fe)
Add AutoML Tables Wide & Deep trainer component and pipeline to GCPC SDK. (765670c)
Fix comment typos in Dataproc component specs. (d26b315)
Update the output comment in eval yaml. (5f988fd)
Add Dataproc Batch components. (499007b)
Remove encryption key in input for create model. kms key should be a part in create model options. (ce0bc47)
Add AutoML Tables prebuilt pipeline with distillation to GCPC SDK. (ef6e01c)
Internal change (86c777d)
Update export model input from string to artifact and update its e2e test (79add47)
Update yaml to change model input from string to artifact. (436fe4f)
Add AutoML Tables prebuilt pipeline with feature selection to GCPC SDK. (2007776)
Add dataflow_subnetwork option to AutoML Tabular's components and pipelines. (9d9fcbb)
Add AutoML Tables components and pipeline to GCPC SDK. (973ea37)
Convert AutoML and Dataset components to static YAML components. (e039c49)
escape table name, model name by backquote. (0d64c34)
Fix the model eval and add more logs for debugability. (9760dd9)
update the status check when poll the query job. (7057bc6)
Creates a working list of FPC names with images (2e14672)
In BQ Predict/Eval components, enclose model name with `` for corner case that model name may container - in it. (d3a1f54)
Update version and README.md for v0.2.1 release (978a93b)
chore(component):update the doc (931cfcb)
1. Refactor the long if - action list in launcher to an action dictionary for error proof purpose. 2. Raise error if job type unsupported This is important to debug wrong job types as container logs show nothings in this case. 3. clear all lint warnings in existing code. (7713c03)
Add a new input - location for forecasting validation and preprocessing components. (275d642)
Fixed a bug in remote runner that checking string is None is not enough. The default input is empty string which may fail some of the assumptions. Therefore, adding is None and len(str) checks. (d916ed3)
Fixed a typo in predict model yaml file (f2df056)
feat(component):update label to dictionary (7b28539)
fix(deployment): the viewer controller does not work because of missing permissions (#7031) (ad5a767)

Open source →

1.8.016 Feb 2022

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.7 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

Open source →

Additional notes

⚠ BREAKING CHANGES

Use Argo Emissary Executor instead of Docker by default. Partial #5718 (#7137)
sdk.v2: Block task dependency referencing tasks inside a sibling condition or loop group. (#7050)
sdk: Deprecate V2 compatible mode in v1 compiler (#6958)
sdk.v2: Merge v2 experimental change back to v2 namespace (#6890)

Features

Use Argo Emissary Executor instead of Docker by default. Partial #5718 (#7137) (bea751c)
backend: Upgrade go client for MLMD to 1.5.0. Partial #7029 (#7101) (ba59a64)
backend: Upgrade MLMD to 1.4.0 (#6910) (b3d15cd)
backend: Upgrade MLMD to 1.5.0 (#6996) (7157764)
chore: add labels and error to the gcp_resources.proto (3da295e)
component: add BQ export model component (28c666f)
component: add code for creating and monitoring bq job (9cb0e71)
component: add generic custom training job component (a74eba1)
component: adding bq create model component (fef8c03)
component: adding bq evaluate model component (27d2ab5)
component: adding bq predict model component (90e759c)
component: adding component yaml and tests for bq query job component (accacf6)
component: fix typo in bp component (9062749)
component: rename bq query job remote runner to bq remote runner for more bq components. (57cc5c3)
component: rename VertexBatchPredictJob -> VertexBatchPredictionJob (#6757) (ca34902)
component/google-cloud: Add component yaml for Dataflow python component. (5902597)
component/google-cloud: Add sample notebook for HP Tuning Job (98677b2)
component/google-cloud: Add sample notebook for HP Tuning Job (#6729) (16c8928)
components: Add a KServe component (#6881) (72185ff)
components: Add GetWorkerPoolSpecsOp and update sample notebook to include training new model from tuning results (30308ea)
components: Add IsMetricBeyondThresholdOp util function for HP tuning job; add tests for all util components (37b3397)
components: Add util components for getting best trial/hyperparameters; delete HP Tuning Job lightweight python version to avoid potential confusion (a62bdbe)
components: not wait for dataflow job to finish in python component (eeb0b9c)
components/google-cloud: Accept UnmanagedContainerModel artifact in Batch Prediction component (e39eed1)
components/google-cloud: Add distributed training sample for custom job wrapper (#6724) (ecbbdf4)
components/google-cloud: Commit experimental Vertex Evaluation component. (4cf9e5b)
components/google-cloud: Update ground_truth_column and update public container for Vertex Evaluation component. (b892888)
frontend: Allow setting s3 region for artifacts (#6409) (af51bd1)
frontend: Disable delete button if no pipeline version is not selected (#6973) (6de35db)
frontend: RunDetail -> Detail tab for KFPv2. Fix #6710 (#6717) (40df465)
frontend: SubDAG styling for KFPv2. Fix #6419 (#6744) (31ff9ad)
frontend: Update unknown and invalid execution node style (#6795) (2ffa212)
frontend: Upgrade MLMD client to 1.5.0. Partial #7029 (#7102) (ef8e092)
frontend: Upload V2 Pipelines. Fix PipelineSpec editor prettified view. (#6909) (feebc6b)
frontend: V2 create run from UI (#7001) (a1bef77)
frontend: View pipeline from run ID in KFPv2. Fix #6758 (#6759) (b7a455d)
google-cloud: Add 'generate_feature_attribution' and executor_output to Evaluation component inputs. (25508b6)
google-cloud: Update Evaluation Component to let the dataflow service account be an optional input (10df58c)
manifests: Add liveness Probe for cloud sql proxy. Fix #4973 (#6904) (3bdf8c0)
sdk: Add filters to python client. (#6748) (37d3234)
sdk: add load_component_from_* (#6822) (f1bb852)
sdk: Add load_component_from_spec. Fixes #5708 #3748 (#6690) (3e6c776)
sdk: add metadata field for importer (#7112) (5a29d7f)
sdk: add set_env_variable for Pipeline task (#6919) (cd20373)
sdk: Add version description optionally to Py SDK. Part of #6256 (#6472) (5dfae1d)
sdk: Deprecate V2 compatible mode in v1 compiler (#6958) (8aee621)
sdk: Extend SDK client with disable/delete run methods (#6754) (ba0bb60)
sdk: ImporterSpec v2 (#6917) (c4d8dfd)
sdk: Improve CLI experience for archiving experiments, managing recurring runs and listing resources (#6934) (ddbfcde)
sdk: Pipeline task settings (#6746) (78dfac7)
sdk: Use google.protobuf.Value in v2 for passing parameters. (#6804) (0be57c3)
sdk.v2: Implement v2 experimental compiler. (#6803) (02b96f0)
v2: control flow - conditional (#7088) (048218b)
adds secrets to ml-pipeline-ui-artifact deployments in user profiles (#5864) (c6c826d)
sdk.v2: Merge v2 experimental change back to v2 namespace (#6890) (7921946)
v2: iterator and sub-DAG basic support (#6985) (713558c)
allow the default-editor to edit argo workflows and fix bug #6649 (#6622) (83ce21f)
sdk.v2: Implement experimental [@component](https://github.com/component) component (#6825) (ea2e5be)
v2: V2 create run api (#6689) (2e94575)
upgrade argo to v3.1.14 (#6809) (4cf34e2)
api: Update pipeline_spec.proto to add pipeline_job_uuid and pipeline_job_name in PipelineTaskFinalStatus (#6557) (6252b09)
api: Updates pipeline_spec.proto to use google.protobuf.Value (#6402) (b8c450f)
backend: Adding prometheus annotations for ml-pipeline service (#6572) (7875b68)
backend: support uploading v2 pipeline spec. Fixes #6169 (#6513) (49de8a9)
component: add new artifact types for gcp resources (#6627) (286a495)
component: use resource name for input artifact (#6698) (89e4c1c)
components: Add and update existing documentations for GCPC components (#6525) (99d1823)
components: Add model upload container component (#6609) (8a363ef)
components: Add static YAML for batch predict, create endpoint, upload and deploy model (#6466) (66a1840)
components: Adds Notebooks Executor API in the experimental components (#6630) (cd37c2c)
components: Copy the GCP components to contrib folder (#6421) (c783705)
components: Implement HyperparameterTuningJob component (#6235) (d9e463f)
components: move dataflow/bq component to google-cloud (#6422) (d864db1)
components: Refresh the create endpoint component to follow container interface (#6647) (8b67505)
components: Update batch prediction component (#6663) (fc40e0a)
components: Update model deploy and model export components (#6654) (bc19d92)
components/google-cloud: Add a new forecasting training component that uses the experimental flags (#6591) (77585f0)
components/google-cloud: add a new python based component that connects the forecasting preprocessing component with the training component (#6568) (cd4f191)
components/google-cloud: add dataflow wait component (#6484) (166c71c)
components/google-cloud: Add description for prep data for train component. (#6597) (6ed4899)
components/google-cloud: add gcp resources proto (#6478) (572643f)
components/google-cloud: add preprocessing_bigquery_dataset to the inputs of preprocessing component (#6461) (c127996)
components/google-cloud: add README.md file for forecasting components. (#6452) (a72c8b7)
components/google-cloud: Add support for labels in custom_job wrapper (#6579) (400ed0c)
components/google-cloud: Create remote runner for batch prediction jobs (#6580) (be47be1)
components/google-cloud: Expose all Custom Job parameters via Custom Job wrapper (#6539) (680e917)
components/google-cloud: Move flex template python files from piper (#6617) (7c939bd)
components/google-cloud: Support Proto Plus serialization (#6464) (610a1f2)
components/google-cloud: Write proper GCP_RESOURCE parameter to the custom_job output (#6558) (83ecb97)
deployment: update and secure metacontroller (#6537) (5c5f210)
frontend: Add styling for Artifact Node. Partial #6280 (#6418) (964a62e)
frontend: Add to UI pipeline version description. Part of #6256 (#6393) (19b3de4)
frontend: Create an Expand button to open SubDAG. Partial #6301 (#6490) (a358124)
frontend: Execution,Artifact side panel for KFPv2 Run Detail. Fix #6740 Fix #6739 Partial #6676 (#6741) (d2c234a)
frontend: Runtime DAG in RunDetailsV2. Fix #6673 (#6694) (74c7773)
frontend: Show hyperlink from Execution to Pipeline Run detail page and Original Execution cache. Fix #5977 (#6556) (3ba2a18)
frontend: Show Pipeline Spec in Pipeline Detail page. Fix #6279 (#6524) (5c2b529)
frontend: SidePanel for static pipeline template. Fix #6399, Fix #6610, Fix #6611 (#6613) (5c89d51)
frontend: Stateful v2 Node styling. Fix #6280 (#6519) (f6ee12a)
frontend: SubDag Node and mock samples. Partial #6419. Fix #6302. Fix #6398 (#6420) (d03277f)
frontend: Support Pipeline summary card and pulling version template for V2. Fix #6633 (#6634) (e0f4f7c)
frontend: V2 Run Details - Toolbar, Page title, Experiments. Fix #6708, partial #6707 (#6709) (a441369)
sdk: Add BaseModel to component_spec data classes (#6372) (65174cd)
sdk: Add IfPresentPlaceholder and ConcatPlaceholder for dsl v2 (#6639) (96a2609)
sdk: add v2 placeholder variables (#6693) (a2cdf74)
sdk: ComponentStore - URI template support. Fixes #4708 (#5434) (daccc5a)
sdk: Configure pipeline run service account. Fixes #6424 (#6455) (139acc8)
sdk: detail option to kfp run get. Fixes #6315 (#6404) (728915f)
sdk: Enable containerizing v2 Python components (#6417) (fe66377)
sdk: Enable re-use of PVC with VolumeOp (#6582) (24331a9)
sdk: load v1 and v2 component spec (#6497) (8100c5e)
sdk: Local runner supports additional docker options. Fixes #6598 (#6599) (36e627d)
sdk: Update cloudpickle to 2.0.0 (fixes #6699) (#6703) (bcb0d96)
sdk.v2: Define PipelineParameterChannel and PipelineArtifactChannel. (#6470) (2176fee)
sdk.v2: Support set_display_name in v2. (#6471) (cfefc6d)
sdk.v2: Support container environment variable in v2. (#6515) (29226e9)
v2: added importer support (#6415) (4438929)
v2: Support v2 caching (#6569) (951b968)
upgrade MLMD to 1.2.0. Fix #6436 (#6437) (e035a88)

Bug Fixes

backend: Decompress workflow node statuses if necessary. Fixes #6547 (#6548) (ccc5dc9)
backend: Fix connection lifetime default and variable names. (#6509) (e58aff7)
backend: Fix v1 caching to read Argo template from container env. Fixes #7266 (#7267) (1a0071b)
backend: make cache-deployer generate CSR using kubelet-serving signerName (#7273) (733a516)
backend: missing permissions for scheduled workflows (#7155) (3d63b9b)
backend: Update composite-controller.yaml to fix https://github.com/kubeflow/manifests/issues/2133 (#7311) (761873c)
backend: Updates nodeSelector, and Affinity Paths in cache-server. Fixes #6623 (#6656) (773e965)
components: move components to contrib/ folder (#6438) (7d6133b)
components: Update all GCPC components naming to be consistent (#6449) (163cfa9)
components/google-cloud: Custom Job resource proto output is incorrect (#6593) (6e32f42)
components/google-cloud: Change run_as_vertex_ai_custom_job to custom_training_job_op (#6616) (3c7d419)
components/google-cloud: Correct custom job default values (#6638) (ab58885)
components/google-cloud: Fix custom job wrapper error on empty input spec. (#6653) (ed80933)
components/google-cloud: Fix Dataflow component sample. (d7c5bae)
components/google-cloud: Fix Dataflow error log reporting. (927d2a9)
components/google-cloud: Fix Help string for dataflow python_module_path (9828007)
components/google-cloud: Fix the default value of preprocessing_bigquery_dataset (#6481) (3a88429)
components/google-cloud: Fix the keyword argument 'project_id' to 'project' after the yaml upd… (#6463) (e45782e)
components/google-cloud: Move the input param details out of the pipeline definition. (#6459) (c41e4a1)
components/google-cloud: Pipeline parameters with type Int cannot be passed to aiplatform components (#6607) (86828f2)
components/google-cloud: Remove executor_input (#6665) (d80f1dd)
components/google-cloud: Remove experimental modules from init (#6443) (a165cc6)
components/google-cloud: Remove googletests library from unit tests to make sure they can run on github presubmit. (7ea9454)
components/google-cloud: Remove special handing for "=" in remote_runner. (a66648d)
components/google-cloud: REVERT - remove custom python package training op (#6688) (c5e7861)
components/google-cloud: Set executor_input to empty for custom job (#6677) (4906ab2)
components/google-cloud: Stage requirements_file and setup_file locally for Dataflow component. (73a0551)
components/google-cloud: Switch to v2 types_util (#6453) (27b3511)
deployment: the viewer controller does not work because of missing permissions (#6892) (8f809c5)
frontend: Batch upgrade for fundamental dependency blocks. (Node, CRA, tailwind and more) Fix #7148 (#7144) (48e1e29)
frontend: Fix the styling for custom icons on sideNav (#6440) (aec5792)
frontend: fixes link to Vertex AI workbench in GettingStarted page. (#7227) (5a89f83)
frontend: Fulfill name/state/workspace for MLMD Executions and Artifacts list. (#6989) (181dd92)
manifests: Fix typo in default executor name (#7159) (4d660f8)
manifests: Remove redundant secret value in profile controller (#7087) (af9d058)
manifests: Upgrade kpt to 1.0.0-beta.6. Fix #5368 (#6595) (4abc4fd)
manifests: Upgrade rbac to v1 for multi-user mode (#7097) (632e71e)
manifests: Upgrade third_party CRDs to v1 for k8s 1.22. Partial #6014 (#7098) (f3a1b65)
manifests: Use number for cloudsql liveness port (#7252) (3953b59)
samples: Add kfp_package_path to importer sample test. (#6507) (236f67e)
samples: Use TFX library version for parameterized_tfx_oss. Fix #6974 (#6993) (3cfff3d)
sdk: Add missing retry policy. (#6808) (ec4ab2d)
sdk: change otherwise to else in yaml (#6952) (4484ccd)
sdk: fix bug in checking values in _param_values. Fixes #6799 (#6965) (a71fce0)
sdk: fix cloud scheduler's job name (#6844) (b2414f3)
sdk: fix input/outputspec and positional arguments (#6980) (85d7433)
sdk: Fix invalid doc example of python_component (#6841) (2f2ebf8)
sdk: fix load_test (#6978) (0842934)
sdk: fix resource spec test (#6979) (485a077)
sdk: Fix type_utils (#6719) (14b3d67)
sdk: fixes the specified 'mlpipeline-ui-metadata','mlpipeline-metrics' path is overrided by default value (#6796) (41275b4)
sdk: Get short name of complex input/output types to ensure we can map to appropriate de|serializer (#6504) (a0b18eb)
sdk: Make Artifact type be compatible with any sub-artifact types bidirectionally (#6859) (dea0823)
sdk: podSpecPatch bug. Fixes 6512 (#6514) (343350a)
sdk: refresh access token only when it expires. Fixes #6883 (#6941) (6d55e26)
sdk: removed duplicate function. Fixes #6586 (#6594) (8b55b16)
sdk: Require base and target images in built containers to be unique. (#6731) (b29c906)
sdk: sanitize op name. Fix #6433 (#6600) (4e89973)
sdk: Try alternatives for obtaining pip when ensurepip does not exist in container. (#6737) (77de39d)
sdk: Update upper bound on kubernetes constraint in Python SDK (#6532) (b613ca8)
sdk: update v2 yaml format (#6661) (5892bf9)
sdk: visualizations and metrics do not work with data_passing_methods (#6882) (b482ba8)
sdk.v2: Avoid pydantic union bug (#6957) (59dfffe)
sdk.v2: Block task dependency referencing tasks inside a sibling condition or loop group. (#7050) (6dfaeeb)
sdk.v2: Component is invalid if packages_to_install empty and install_kfp_package=False (#6527) (a52ac6d)
sdk.v2: fix Optional type hint causing executor to ignore user inputs for parameters. (#6541) (7626abf)
sdk.v2: Fix a couple of ParallelFor related bugs. Fixes #6383, fixes #6628 (#6643) (b466f59)
sdk.v2: fix boolean error for inputs (#7041) (1a0bef7)
sdk.v2: Fix bug on executor input with default value. (#6937) (2910d10)
sdk.v2: Fix display name support for groups (#6832) (0f2cab9)
sdk.v2: Fix executor getting None as value when float 0 is passed in. Fixes #6636 (#6682) (82ed66e)
sdk.v2: fix function-based components not preserving the namespace of GCPC artifact types. (#6702) (2b2e5fd)
sdk.v2: Fix importer ignoring reimport setting, and switch to Protobuf.Value for import uri. (#6827) (8c6843f)
sdk.v2: Fix InputPathPlaceholder changed to InputValuePlaceholder (#6794) (61fea6b)
sdk.v2: Fix missing experimental modules in setup.py (#6817) (7768f28)
sdk.v2: fix regression on nested loops (#6990) (c1b6715)
sdk.v2: Fix regression on optional inputs (#6905) (9ee4534)
sdk.v2: Fix the issue that using dsl. prefix in component I/O type annotation breaks component at runtime. (#6714) (073c819)
Remove some data from model_status when the size exceeds 3000 (#6528) (3a621fb)
Add workaround init container to fix test infra (#6618) (f870574)
disable cache for v1 mode in sample test (#6641) (0c2759d)
the cache server in combination with kubeflow profile quotas (#5743) (22f2693)
sdk.v2: fix importer not taking output from upstream (#6439) (06044dd)
sdk.v2: Fix passing in "" to a str parameter causes the parameter to receive it as None instead (#6533) (38e826b)
sdk.v2: fixes broken output parameter type checking in _handle_single_return_value (#6566) (866dfc7)
Rename 'python_module' to 'python_module_name' and fix type to str (#6473) (aebe8d7)
Update typing checks to include Python3.6 deprecated types. (#6520) (b6cb0b4)
sdk.v2: Support dict, list, bool typed input parameters from constant values and pipeline inputs. (#6523) (0fba85c)

Other Pull Requests

Fix the model eval and add more logs for debugability. (9760dd9)
update the status check when poll the query job. (7057bc6)
Creates a working list of FPC names with images (2e14672)
In BQ Predict/Eval components, enclose model name with `` for corner case that model name may container - in it. (d3a1f54)
Update version and README.md for v0.2.1 release (978a93b)
chore(component):update the doc (931cfcb)
1. Refactor the long if - action list in launcher to an action dictionary for error proof purpose. 2. Raise error if job type unsupported This is important to debug wrong job types as container logs show nothings in this case. 3. clear all lint warnings in existing code. (7713c03)
Add a new input - location for forecasting validation and preprocessing components. (275d642)
Fixed a bug in remote runner that checking string is None is not enough. The default input is empty string which may fail some of the assumptions. Therefore, adding is None and len(str) checks. (d916ed3)
Fixed a typo in predict model yaml file (f2df056)
feat(component):update label to dictionary (7b28539)
fix(deployment): the viewer controller does not work because of missing permissions (#7031) (ad5a767)
Make HyperparameterTuningJobOp serverless (8541b53)
bump kfp sdk version (5cfb4a5)
Fix missing experimental type module in setup.py. (#6816) (fca4c1c)
chore(components/google-cloud):Internal clean up. (003dfa4)
Update RELEASE.md (#6797) (ec9a8ed)
Update README links to point to newer versions of samples (20f2863)
Release KFP SDK and v2 launcher 1.8.6 (#6771) (0197e01)
Update API docs for CustomJob. (b861a89)
Update API docs for CustomJob. (#6756) (63c1520)
Minor updates on the documentation (#6747) (6506801)
1. Update API docs for AI platform components. (#6742) (3bf7059)
feat(component/google-cloud):Add Dataflow python component backend using Discovery API. (#6730) (ae77bbf)
Add Dataflow component using Discovery API - General structure of the remote runner code. (#6725) (e81727e)
add model export op to init file (#6701) (710db22)
add model export op to init file (#6723) (f008073)
Internal change (ba679ab)
Import module instead of functions; remove unneeded dependencies (#6671) (e7b6fd5)
bump kfp version (#6678) (dd92c5b)
Release KFP SDK and v2 launcher 1.8.4 (#6666) (0aa513e)
remove custom python package training op (#6474) (e9037a9)

Open source →

1.8.0rc010 Jan 2022pre-releaseBreaking

Upgrade Notes with notices and breaking changes

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api==1.8.0rc0 --upgrade

Refer to:

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

⚠ BREAKING CHANGES

Use Argo Emissary Executor instead of Docker by default. Partial #5718 (#7137)
sdk.v2: Block task dependency referencing tasks inside a sibling condition or loop group. (#7050)

Features

Use Argo Emissary Executor instead of Docker by default. Partial #5718 (#7137) (bea751c)
backend: Upgrade go client for MLMD to 1.5.0. Partial #7029 (#7101) (ba59a64)
backend: Upgrade MLMD to 1.5.0 (#6996) (7157764)
chore: add labels and error to the gcp_resources.proto (3da295e)
component: add BQ export model component (28c666f)
component: add generic custom training job component (a74eba1)
component: adding bq create model component (fef8c03)
component: adding bq evaluate model component (27d2ab5)
component: adding bq predict model component (90e759c)
component: adding component yaml and tests for bq query job component (accacf6)
component: rename bq query job remote runner to bq remote runner for more bq components. (57cc5c3)
components: Add a KServe component (#6881) (72185ff)
components: Add GetWorkerPoolSpecsOp and update sample notebook to include training new model from tuning results (30308ea)
components: Add IsMetricBeyondThresholdOp util function for HP tuning job; add tests for all util components (37b3397)
components: Add util components for getting best trial/hyperparameters; delete HP Tuning Job lightweight python version to avoid potential confusion (a62bdbe)
components/google-cloud: Accept UnmanagedContainerModel artifact in Batch Prediction component (e39eed1)
components/google-cloud: Commit experimental Vertex Evaluation component. (4cf9e5b)
components/google-cloud: Update ground_truth_column and update public container for Vertex Evaluation component. (b892888)
frontend: Upgrade MLMD client to 1.5.0. Partial #7029 (#7102) (ef8e092)
frontend: V2 create run from UI (#7001) (a1bef77)
google-cloud: Add 'generate_feature_attribution' and executor_output to Evaluation component inputs. (25508b6)
google-cloud: Update Evaluation Component to let the dataflow service account be an optional input (10df58c)
sdk: add metadata field for importer (#7112) (5a29d7f)
sdk: Improve CLI experience for archiving experiments, managing recurring runs and listing resources (#6934) (ddbfcde)
v2: control flow - conditional (#7088) (048218b)
adds secrets to ml-pipeline-ui-artifact deployments in user profiles (#5864) (c6c826d)
v2: iterator and sub-DAG basic support (#6985) (713558c)

Bug Fixes

components/google-cloud: Fix Help string for dataflow python_module_path (9828007)
frontend: Batch upgrade for fundamental dependency blocks. (Node, CRA, tailwind and more) Fix #7148 (#7144) (48e1e29)
frontend: Fulfill name/state/workspace for MLMD Executions and Artifacts list. (#6989) (181dd92)
manifests: Remove redundant secret value in profile controller (#7087) (af9d058)
manifests: Upgrade rbac to v1 for multi-user mode (#7097) (632e71e)
manifests: Upgrade third_party CRDs to v1 for k8s 1.22. Partial #6014 (#7098) (f3a1b65)
samples: Use TFX library version for parameterized_tfx_oss. Fix #6974 (#6993) (3cfff3d)
sdk: fix bug in checking values in _param_values. Fixes #6799 (#6965) (a71fce0)
sdk: fix input/outputspec and positional arguments (#6980) (85d7433)
sdk.v2: Block task dependency referencing tasks inside a sibling condition or loop group. (#7050) (6dfaeeb)
sdk.v2: fix boolean error for inputs (#7041) (1a0bef7)
sdk.v2: fix regression on nested loops (#6990) (c1b6715)

Other Pull Requests

Fix the model eval and add more logs for debugability. (9760dd9)
update the status check when poll the query job. (7057bc6)
Creates a working list of FPC names with images (2e14672)
In BQ Predict/Eval components, enclose model name with `` for corner case that model name may container - in it. (d3a1f54)
Update version and README.md for v0.2.1 release (978a93b)
chore(component):update the doc (931cfcb)
1. Refactor the long if - action list in launcher to an action dictionary for error proof purpose. 2. Raise error if job type unsupported This is important to debug wrong job types as container logs show nothings in this case. 3. clear all lint warnings in existing code. (7713c03)
Add a new input - location for forecasting validation and preprocessing components. (275d642)
Fixed a bug in remote runner that checking string is None is not enough. The default input is empty string which may fail some of the assumptions. Therefore, adding is None and len(str) checks. (d916ed3)
Fixed a typo in predict model yaml file (f2df056)
feat(component):update label to dictionary (7b28539)
fix(deployment): the viewer controller does not work because of missing permissions (#7031) (ad5a767)

Open source →

1.7.025 Aug 2021Breaking

Upgrade Notes with notices and breaking changes

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api==1.7.0 --upgrade

Refer to:

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

⚠ BREAKING CHANGES

use argo emissary executor by default. Fixes #5718 (#5926)

Features

api: Add env variable in ContainerSpec (#6251) (5b5568c)
api: Add PipelineTaskFinalStatus to pipeline_spec.proto (#6221) (bb22118)
backend: Add pipeline version description to backend. Part of #6256 (#6261) (943dc12)
backend/sdk: Rename pipeline-output-directory to pipeline-root. Fixes #6307 (#6329) (7559e27)
components: Add PyTorchJob launcher (#5170) (dfc9e95)
components/google-cloud: Add component for Custom Job (#6044) (bcca04e)
components/google-cloud: Add methods for creating forecasting preprocessing and validation components (#6382) (24cddb9)
components/google-cloud: Add NAB scoring to tfp_anomaly_detection (#6188) (c0df67a)
components/google-cloud: Add preprocess / postprocess components to tfp_anomaly_detection (#6216) (9a7dd4f)
components/google-cloud: add project_id to the inputs of preprocessing component (#6413) (c30a699)
components/google-cloud: Add tfp_anomaly_detection component (#6083) (13ace37)
components/google-cloud: Add tfp_anomaly_detection component to gcpc (#6128) (2cbe49f)
components/google-cloud: Add user agent to the CustomJob request header (#5935) (0615af5)
components/google-cloud: Add vertex forecasting preprocessing and validation components. (#6305) (4b80e83)
components/google-cloud: Add yaml files to python package (#6066) (0381deb)
components/google-cloud: Fix read the docs (#6345) (53eb3c7)
components/google-cloud: Include release notes in pypi (#6368) (fe97bcd)
components/google-cloud: Migrate experimental dataflow launch-python component to google_cloud_pipeline_components (#6037) (970a84d)
components/google-cloud: Update tfp_anomaly_detection to accept input csvs and show visualization (#6178) (055a3a5)
components/pytorch: Example to estimate best parameters using AX (#6041) (aef5037)
components/pytorch: Pytorch Distributed Training (#6021) (b1d0eb7)
deployment: gcp marketplace - provide emissary as an option. Part of #5718 (#6250) (3ec069d)
deployment: provide platform-agnostic-emissary and platform-agnostic-multi-user-emissary as options (#6248) (b5dc282)
frontend: Add KFPv2 and tutorial link in Get Started page. Partial #6306 (#6337) (0802f95)
frontend: Add version description to pipelineVersionList. Part of #6256 (#6405) (3866ccf)
frontend: Convert PipelineSpec to Staticflow for V2 pipeline. Fix #6270 (#6278) (579e46b)
frontend: integrate with TFX 1.2.0 metadata & visualization, no longer support previous versions. Part of #6138 (#6388) (d077e6e)
frontend: Introduce pipeline detail v2 page. Fix #6179 (#6180) (8a5e067)
frontend: Select node using execution ID via router param in run detail page. Partial #5977 (#6218) (a94e92e)
frontend: Show V2 PipelineDetail page when pipelineSpec is v2. Fix #6303 (#6318) (72c54c8)
frontend: Support v2 HTML/Markdown visualization. Fix #3970 (#6367) (3e83f32)
frontend: v2 Execution Node element. Partial #6280 (#6407) (ee718af)
google client: Enable to specify the service account to proxy function of scheduler in google client sdk (#6013) (d40985c)
sample: Use TFX 1.2.0 for Taxi tips prediction sample. Partial #6138 (#6381) (91dc80d)
sdk: add default schema_version to pipeline (#6366) (937cacd)
sdk: Add description to upload_pipeline_version in kfp. Part of #6256 (#6395) (8b51cc4)
added namespace as cache scope (#6342) (2639ba0)
upgrade argo to v3.1.6 (#6333) (ee8cbfa)
upgrade argo to v3.1.6-patch (#6376) (366b70e)
upgrade TFX to 1.2.0 (#6375) (59a0c9d)
backend: add V2 lightweight Python pipeline sample. Fixes #5689 (#5998) (27415fd)
backend: upgrade argo go module to V3. Part of #5718 (#5792) (00a0ffb)
backend: workflow validation. Fixes #3526. (#3965) (a44a225)
component: Add GCP launcher following the first party component interface (#5805) (dd365e2)
components: Added HuggingFace dataset components (#5707) (4cf420d)
components: Adding Mar component for PyTorch - KFP (#5780) (e20a1b8)
components: Adding Minio component for PyTorch - KFP (#5808) (f843121)
components: Adding samples for PyTorch - KFP (#5812) (703822c)
components: Adding Visualization component for PyTorch - KFP (#5810) (941879d)
components: Addressing Review comments on Trainer component for PyTorch - KFP (#5814) (72f7464)
components: KFP - Run component (#5338) (e996cdb)
components: Kubernetes - Create PersistentVolumeClaim (#5222) (1d38f6d)
components: PyTorch - Added the Create PyTorch Model Archive component (#5630) (b08b29f)
components/google-cloud: Add components for AutoMLForecasting (#5821) (e860fd6)
components/pytorch: Add requirement for mar generation (#5850) (2d7f3a2)
deployment: configurable v2 compatible mode default pipeline root. Part of #5680. Fixes #5704 (#5750) (adc1951)
frontend: Clickable link consistent css styling. Fix #5315 (#6023) (0ba947c)
frontend: MetricsTab for NodeDetail, ROC Curve from MLMD custom property. Issue #5668 (#5731) (601b104)
frontend: Show cache icon for v2 compatible. Partial #5977 (#6117) (17f1373)
frontend: Support new artifact name retrival and hide MLMD tab in v2 pipeline. Fix #5675 (#5956) (f524266)
frontend: Support v2 confusion matrix visualization. Issue #5668 (#5781) (ef33b9e)
sdk: add deprecation warning for AiPlatformClient (#6106) (1ad2015)
sdk: add GPU runtime resource request and fix spelling in runtime_resouce_request. Fixes #4877. Fixes #1252 (#5972) (e6becd7)
sdk: Add submission time caching option override for v2 compatible mode. (#6032) (158d15f)
sdk: Add support for executor input in dsl_util.py (#6057) (fdab2e6)
sdk: add_default_resource_spec container op transformer (#6203) (2181b2f)
sdk: Allow parametrization of ui_host in kfp.Client (#6124) (3ec8b82)
sdk: Enable the use of Executor from KFP package for v2 lightweight components (#6202) (c5591f2)
sdk: Surface kubernetes configuration in container builder (#6095) (eb41fdd)
sdk: update pipeline_spec.proto to add artifact schema version field (#6220) (6b48154)
sdk.v2: add enable_caching option in create_schedule_from_job_spec (#6119) (27051ab)
sdk.v2: Add HTML and Markdown artifact types (#6246) (f4657ba)
sdk.v2: Define BaseComponent, ComponentSpec, and PipelineTask. (#6210) (903b986)
change release tags to vX.Y.Z. Fixes #5954 (#5974) (b57bbee)
pipeline spec as a separate go module (#6000) (b653ea2)
upgrade to argo v3.1.5-patch (#6228) (36b815b)
use argo emissary executor by default. Fixes #5718 (#5926) (0c4129c)
use argo v3.1.1-patch. Part of #5930 (#6027) (cb75b44)
use argo v3.1.2-patch.2. Part of #5718 (#6082) (0434243)
frontend: Support Scalar metrics in V2 compatible mode. Partial #5668 (#5811) (22bc9e2)
frontend: Support v1 visualizations in v2 compatible mode. Partial #5666 (#5961) (df1ab4d)
launcher: nested directory artifacts support in v2 compatible (#5883) (045c717)
manifests: Expose sync.py frontend/visualization image as env var, add tests (#5732) (7607841)
sample: Add HTML sample for v1/v2 visualization (#5938) (036ef49)
sample: Add markdown visualization example for v1 and v2 compatible (#5936) (565d03e)
sample: ROC curve sample for v1 and v2 visualization (#5959) (db03151)
samples: Confusion Matrix sample for v1/v2 visualization (#5947) (5eef38b)
samples: update set parallelism limit example to not use container op. Part of #5687 (#5828) (57b120c)
samples: updated execution order, hello world, transformers and retry to not use Container.Op. Part of #5687 (#5852) (c397bd6)
samples: volume ops example to not use container op. Part of #5687 (#5826) (b3c2351)
samples/pytorch: Adding compilation test for PyTorch samples KFP (#5856) (a41a72f)
sdk: Add credentials to authenticate with ServiceAccountTokens. Part of #5138 (#5676) (52b0792)
sdk: Add interface for enable_caching at task level. (#6007) (c6cb8ac)
sdk: add runtime resource requests. Fixes #1956 (#5447) (5db8431)
sdk: add timeout option to kfp run submit. Fixes #5806 (#5892) (069d5a7)
sdk: Better error message when compiling for v2. Fixes #5688, Fixes #5727 (#5759) (2991602)
sdk: Dropped support for Python 3.5. Fixes #4584 (#4611) (4e5d724)
sdk: json serialization of dict/list items in run args, fixes #5896 (#5927) (cbb83a5)
sdk: Make kaniko_executor_image a parameter for ContainerBuilder (#5990) (4d75ec9)
sdk: set shell=False to run LocalClient both docker and local mode (#5751) (329ddca)
sdk: special handling for type annotation using typing.Optional (#5716) (58f74d3)
sdk: Support launching Dataflow Flex templates from Kubeflow pipelines (#5501) (24bc916)
sdk.v2: enable_caching in v2 client defaults to compile time settings. (#6052) (24c551d)
sdk.v2: Support Exit handler in v2 compiler. (#5784) (6b87155)
sdk/dsl/compiler: dsl-compile --mode flag to turn on V2_COMPATIBLE, defaults to KF_PIPELINES_COMPILER_MODE env var. Fixes #5840 (#5952) (8a256db)
v2,sdk: Add service_account as args to create_schedule_from_job_spec (#5994) (9a6b0e8)
update MLMD to 1.0.0 (#5786) (28431fc)
upgrade argo to v3.1.1. Part of #5718 (#5971) (4451c7d)
upgrade argo-workflows to v3.1.0. Part of #5718 (#5922) (3973407)
v2: enable v2 compatible mode in full Kubeflow with zero config. Fixes #5680 (#5697) (43994ab)
v2/dsl: support minio & s3 artifacts in v2 python component. Fixes #5838 (#5909) (a95bfd6)

Bug Fixes

backend: argo workflow not found in KFP DB should be GCed. Part of #6189 (#6190) (ef37236)
backend: Fixed workflow cannot be retried with node xxx in Omitted (#5738) (435f6b1)
backend: recognize "sidecar.istio.io/inject" annotation to use Istio service mesh optionally. Fixes #6200 (#6391) (702e48b)
cli: handle when pipeline.paramters is None (#5725) (4212110)
components: GCP - Fixed the BrokenPipe error. Fixes #5746 (#5760) (5bdb5e5)
components: Refactor Kubeflow E2E mnist example (#5433) (89c36d3)
components/gcp: add explicit command for gcp components. Fixes #6252 (#6352) (852e7f5)
components/google-cloud: Add temporary work around for key not found during compile (#6224) (0e1db0d)
components/google-cloud: Correct API reference doc link in README.md (#5890) (9c21701)
components/google-cloud: Custom job container must use the same container as AIPlatform (#6321) (6b6e090)
components/google-cloud: Fix custom job container args order to align with BE contract (#6322) (b432312)
components/google-cloud: Fix FileNotFoundError when loading component yaml from file. (#6396) (fa1cf41)
components/google-cloud: Fix parameter for TabularCreateDatasetOp breaks if it contains "=" (#6320) (b835800)
components/google-cloud: Skip serialization of value if value is instance of PipelineParam (#6290) (93951d1)
components/google-cloud: Undo fix for KFP compiler erroring with key not found (#6348) (7ffc4f6)
components/google-cloud: Update dependancies min version (#6357) (c02676f)
components/google-cloud: update version to include google-cloud-aiplatform fix (#5721) (d7b927f)
components/google-cloud: Use correct dataset type when passing dataset to Custom*Training. Fixes #5885 (#5964) (6c4a74f)
components/kubeflow: PyTorchJob launcher (#6172) (af921a5)
components/pytorch: Add sleep to cifar10 pipeline (#6042) (a4b915b)
components/pytorch: Clean up notebook and yaml files (#6070) (01ec07b)
components/pytorch: custom handler for cifar10, saves explain output as bytes image (#6050) (74d27e7)
components/pytorch: Custom package installation - torchserve (#6055) (7131771)
components/pytorch: Pytorch Cifar10 Captum Insights (#6105) (4d42624)
components/pytorch: pytorch kfp components and Sample - GPU updates (#5939) (d88394b)
components/pytorch: PyTorch Samples - Generating component.yaml using templates (#6231) (f9d47d0)
components/pytorch: readme update (#6081) (01acf5a)
components/pytorch: readme update (#6174) (8319eac)
components/pytorch: Update Readme docs (#6186) (105e100)
Components/pytorch: Add single docker file for cpu and gpu (#5863) (0f222f1)
deployment: GCP AI Platform Pipelines -- correct default values when some fields left empty. Fixes #5717 (#5777) (cf2807c)
deployment: reduce artifact name collision rate under heavy load (#5894) (e958156)
deployment: workaround fix, 34s timeout bug for argo crd on k8s1.20+ (#6075) (6e738ca)
Dockerfile: fixes the dockerfile by allowing apt-get update to ignore releaseinfo changes. (#6356) (fd52629)
frontend: Fix the enum iterator of tabs. Fix #6065 (#6077) (c89ed71)
frontend: Maximize width of HTML visualization. Partial #6296 (#6319) (1828768)
frontend: mock-backend didn't understand absolute path import in frontend/src/. Fix #6212 (#6213) (4804034)
frontend: Remove alphabetical sorting of the metrics column. Fixes #5215 (#5701) (ebf5310)
frontend: TFX artifact visualization update. Fix tensorflow/tfx#3933 (#5999) (290d201)
frontend: Truncate large Markdown file to prevent UI crashing. Fix #5513 (#5585) (2260793)
frontend: Update instruction link in upload pipeline page. (#5847) (039e8a6)
frontend: Update service account RBAC link (#6003) (ad419cd)
launcher: duplicate input artifacts logged (#5836) (77c1de8)
manifests: replace argo ClusterRoleBinding namespace (#6362) (7cfd462)
samples: Add kfp_package_path to a core sample test. (#6259) (402549d)
scheduledworkflow: Set location to make CRON timezone work, Fixes #2653 (#5800) (f61048b)
sdk: Add http status code check for token requests. Fixes #4312 (#5782) (0d84877)
sdk: Avoid inserting duplicate input parameters in v2 compatible mode. (#5740) (fa5ac25)
sdk: block dsl.importer usage in KFP OSS. Fixes: #6323 (#6330) (434e5c3)
sdk: Ensure pip exists for lightweight Python-based components. (#6359) (609e5b7)
sdk: Fix compiler bug for legacy outputs mlpipeline-ui-metadata and mlpipeline-metrics. Fixes #6311 (#6325) (f743dde)
sdk: Fix default value of component input not picked up. Fixes #5880. (#5903) (3f45f78)
sdk: Fix type hints for create_run_from_job_spec. Fixes #6092 (#6098) (9ebbe7b)
sdk: Fix URI placeholder in v2 compatible mode. (#6040) (b200e1b)
sdk: google client, Enable to provide app engine region that scheduled job is created in (#6145) (f116e2c)
sdk: google client: set ai platform endpoint by region (#6011) (78067d1)
sdk: pipeline name is required in v2 compatible mode. Fixes #6197 (#6332) (befd89c)
sdk: Relax the requirement that component inputs/outputs must appear on the command line. (#6268) (911562f)
fix a spelling error (#6236) (2f843f9)
sdk, v2: Fix error when replica_count isn't set (#5761) (5f34603)
sdk.v2: fix metrics outputs populating to unblock pipeline run (#5770) (ee77fd8)
sorting: use array index as key for TableRow (#6115) (7fa2433)
tfx: fix missing mlmd data when sdk label is overridden. Fixes #5303 (#6035) (a30e093)
ui: fixes cron format go.doc link. Fixes #6068 (#6069) (8dc1701)
Forked and applied fix to BigQuery to_gcs and to_table components (#6045) (4da78a8)
v2compat - handle parameter values with special characters stably. Fixes #5830 (#5973) (f3b369e)
sdk: resolve pipeline_root and run_uid placeholders in container env (#5772) (4eb722e)
sdk.v2: Fix the bug where string concat inside a loop may break compilation. (#5752) (f9f455e)
sdk/compiler: v2 compat - fix mlpipeline-ui-metadata artifact. Part of #5666 (#5832) (2bee21a)
test: fixed sample uri (#6177) (d73ffa7)

Other Pull Requests

feat(frontend): Prepare PipelineDetailPage to show different contents based on V1/V2 pipeline spec. Fix #6214 (#6258) (6c8309d)
SDK - Components - Added support for annotations to create_graph_component_from_pipeline_func (#5991) (9b679c5)
chore(samples/pytorch) - Build script fix - Updating Dockerfile name (#5928) (5b78b01)
chore(components/pytorch) Updating pytorch lightning version to 1.3.7 (#5905) (551756e)
chore(components/pytorch) Cifar 10 example - Confusion matrix Fix (#5918) (694cae4)
feat(frontend) Support Input/Output from MLMD for V2-compatible. Fix #5670 (#5859) (120da92)
chore(components/pytorch) [WIP] Updating docker image location - pytorch kfp samples (#5911) (a437b92)
fix(components/pytorch) Pytorch Lightning Arguments string pass through (#5870) (192b8e8)
fix(sdk compiler):Fix v2 compile for exit handler Fixes #5854 (#5899) (a1740b4)
feat(component/pytorch) Custom handler for cifar10 example ,saves multiple explain attributions methods (#5871) (1983037)
chore(components) - Add SageMaker component owners to AWS owners list (#4756) (60ed8e4)
Update README.md (ec9021f)
fix(components/pytorch) Pytorch - Tensorboard Profiler fix (#5860) (9cfa4df)
refactor(frontend) Set 1 Hour as a default value for recurring pipeline runs (#5824) (a8ffc50)
Update pipeline_spec.proto (#5813) (635f1ba)
feat(components): Adding Trainer component for PyTorch - KFP (#5767) (c5325db)
Update pipeline_spec.proto (#5787) (8501337)
Fix the cache-deployer to run as non root (#5742) (efe916c)

Open source →

1.7.0rc419 Aug 2021pre-release

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api==1.7.0rc4 --upgrade

See the Change Log

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

Features

api: Add env variable in ContainerSpec (#6251) (5b5568c)
backend: Add pipeline version description to backend. Part of #6256 (#6261) (943dc12)
backend/sdk: Rename pipeline-output-directory to pipeline-root. Fixes #6307 (#6329) (7559e27)
components/google-cloud: Add methods for creating forecasting preprocessing and validation components (#6382) (24cddb9)
components/google-cloud: Add user agent to the CustomJob request header (#5935) (0615af5)
components/google-cloud: Add vertex forecasting preprocessing and validation components. (#6305) (4b80e83)
components/google-cloud: Fix read the docs (#6345) (53eb3c7)
components/google-cloud: Include release notes in pypi (#6368) (fe97bcd)
frontend: Add KFPv2 and tutorial link in Get Started page. Partial #6306 (#6337) (0802f95)
frontend: Convert PipelineSpec to Staticflow for V2 pipeline. Fix #6270 (#6278) (579e46b)
frontend: integrate with TFX 1.2.0 metadata & visualization, no longer support previous versions. Part of #6138 (#6388) (d077e6e)
frontend: Show V2 PipelineDetail page when pipelineSpec is v2. Fix #6303 (#6318) (72c54c8)
sample: Use TFX 1.2.0 for Taxi tips prediction sample. Partial #6138 (#6381) (91dc80d)
added namespace as cache scope (#6342) (2639ba0)
upgrade argo to v3.1.6 (#6333) (ee8cbfa)
upgrade argo to v3.1.6-patch (#6376) (366b70e)
upgrade TFX to 1.2.0 (#6375) (59a0c9d)
frontend: Support v2 HTML/Markdown visualization. Fix #3970 (#6367) (3e83f32)
google client: Enable to specify the service account to proxy function of scheduler in google client sdk (#6013) (d40985c)
sdk: Surface kubernetes configuration in container builder (#6095) (eb41fdd)
sdk.v2: Add HTML and Markdown artifact types (#6246) (f4657ba)
sdk.v2: Define BaseComponent, ComponentSpec, and PipelineTask. (#6210) (903b986)

Bug Fixes

components/gcp: add explicit command for gcp components. Fixes #6252 (#6352) (852e7f5)
components/google-cloud: Add temporary work around for key not found during compile (#6224) (0e1db0d)
components/google-cloud: Custom job container must use the same container as AIPlatform (#6321) (6b6e090)
components/google-cloud: Fix custom job container args order to align with BE contract (#6322) (b432312)
components/google-cloud: Fix parameter for TabularCreateDatasetOp breaks if it contains "=" (#6320) (b835800)
components/google-cloud: Skip serialization of value if value is instance of PipelineParam (#6290) (93951d1)
components/google-cloud: Undo fix for KFP compiler erroring with key not found (#6348) (7ffc4f6)
components/google-cloud: Update dependancies min version (#6357) (c02676f)
components/pytorch: PyTorch Samples - Generating component.yaml using templates (#6231) (f9d47d0)
Dockerfile: fixes the dockerfile by allowing apt-get update to ignore releaseinfo changes. (#6356) (fd52629)
frontend: Maximize width of HTML visualization. Partial #6296 (#6319) (1828768)
manifests: replace argo ClusterRoleBinding namespace (#6362) (7cfd462)
samples: Add kfp_package_path to a core sample test. (#6259) (402549d)
sdk: block dsl.importer usage in KFP OSS. Fixes: #6323 (#6330) (434e5c3)
sdk: Ensure pip exists for lightweight Python-based components. (#6359) (609e5b7)
sdk: Fix compiler bug for legacy outputs mlpipeline-ui-metadata and mlpipeline-metrics. Fixes #6311 (#6325) (f743dde)
sdk: pipeline name is required in v2 compatible mode. Fixes #6197 (#6332) (befd89c)
sdk: Relax the requirement that component inputs/outputs must appear on the command line. (#6268) (911562f)

Other Pull Requests

feat(frontend): Prepare PipelineDetailPage to show different contents based on V1/V2 pipeline spec. Fix #6214 (#6258) (6c8309d)

Open source →

1.7.0a128 Jun 2021pre-release

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api==1.7.0a1 --upgrade

See the Change Log

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

Features

upgrade argo-workflows to v3.1.0. Part of #5718 (#5922) (3973407)
backend: upgrade argo go module to V3. Part of #5718 (#5792) (00a0ffb)
backend: workflow validation. Fixes #3526. (#3965) (a44a225)
component: Add GCP launcher following the first party component interface (#5805) (dd365e2)
components: Adding Mar component for PyTorch - KFP (#5780) (e20a1b8)
components: Adding Minio component for PyTorch - KFP (#5808) (f843121)
components: Adding samples for PyTorch - KFP (#5812) (703822c)
components: Adding Visualization component for PyTorch - KFP (#5810) (941879d)
components: Addressing Review comments on Trainer component for PyTorch - KFP (#5814) (72f7464)
components: PyTorch - Added the Create PyTorch Model Archive component (#5630) (b08b29f)
components/google-cloud: Add components for AutoMLForecasting (#5821) (e860fd6)
components/pytorch: Add requirement for mar generation (#5850) (2d7f3a2)
deployment: configurable v2 compatible mode default pipeline root. Part of #5680. Fixes #5704 (#5750) (adc1951)
frontend: MetricsTab for NodeDetail, ROC Curve from MLMD custom property. Issue #5668 (#5731) (601b104)
frontend: Support Scalar metrics in V2 compatible mode. Partial #5668 (#5811) (22bc9e2)
frontend: Support v2 confusion matrix visualization. Issue #5668 (#5781) (ef33b9e)
launcher: nested directory artifacts support in v2 compatible (#5883) (045c717)
manifests: Expose sync.py frontend/visualization image as env var, add tests (#5732) (7607841)
samples: update set parallelism limit example to not use container op. Part of #5687 (#5828) (57b120c)
samples: volume ops example to not use container op. Part of #5687 (#5826) (b3c2351)
samples/pytorch: Adding compilation test for PyTorch samples KFP (#5856) (a41a72f)
sdk: add runtime resource requests. Fixes #1956 (#5447) (5db8431)
sdk: add timeout option to kfp run submit. Fixes #5806 (#5892) (069d5a7)
sdk: Better error message when compiling for v2. Fixes #5688, Fixes #5727 (#5759) (2991602)
sdk: Dropped support for Python 3.5. Fixes #4584 (#4611) (4e5d724)
sdk: set shell=False to run LocalClient both docker and local mode (#5751) (329ddca)
sdk: special handling for type annotation using typing.Optional (#5716) (58f74d3)
sdk: Support launching Dataflow Flex templates from Kubeflow pipelines (#5501) (24bc916)
sdk.v2: Support Exit handler in v2 compiler. (#5784) (6b87155)
v2/dsl: support minio & s3 artifacts in v2 python component. Fixes #5838 (#5909) (a95bfd6)
update MLMD to 1.0.0 (#5786) (28431fc)
v2: enable v2 compatible mode in full Kubeflow with zero config. Fixes #5680 (#5697) (43994ab)

Bug Fixes

backend: Fixed workflow cannot be retried with node xxx in Omitted (#5738) (435f6b1)
cli: handle when pipeline.paramters is None (#5725) (4212110)
components/google-cloud: Correct API reference doc link in README.md (#5890) (9c21701)
components/google-cloud: update version to include google-cloud-aiplatform fix (#5721) (d7b927f)
Components/pytorch: Add single docker file for cpu and gpu (#5863) (0f222f1)
deployment: GCP AI Platform Pipelines -- correct default values when some fields left empty. Fixes #5717 (#5777) (cf2807c)
deployment: reduce artifact name collision rate under heavy load (#5894) (e958156)
frontend: Remove alphabetical sorting of the metrics column. Fixes #5215 (#5701) (ebf5310)
frontend: Truncate large Markdown file to prevent UI crashing. Fix #5513 (#5585) (2260793)
frontend: Update instruction link in upload pipeline page. (#5847) (039e8a6)
launcher: duplicate input artifacts logged (#5836) (77c1de8)
scheduledworkflow: Set location to make CRON timezone work, Fixes #2653 (#5800) (f61048b)
sdk: Add http status code check for token requests. Fixes #4312 (#5782) (0d84877)
sdk: Avoid inserting duplicate input parameters in v2 compatible mode. (#5740) (fa5ac25)
sdk: Fix default value of component input not picked up. Fixes #5880. (#5903) (3f45f78)
sdk: resolve pipeline_root and run_uid placeholders in container env (#5772) (4eb722e)
sdk, v2: Fix error when replica_count isn't set (#5761) (5f34603)
sdk.v2: fix metrics outputs populating to unblock pipeline run (#5770) (ee77fd8)
sdk.v2: Fix the bug where string concat inside a loop may break compilation. (#5752) (f9f455e)
sdk/compiler: v2 compat - fix mlpipeline-ui-metadata artifact. Part of #5666 (#5832) (2bee21a)

Other Pull Requests

chore(components/pytorch) Updating pytorch lightning version to 1.3.7 (#5905) (551756e)
chore(components/pytorch) Cifar 10 example - Confusion matrix Fix (#5918) (694cae4)
feat(frontend) Support Input/Output from MLMD for V2-compatible. Fix #5670 (#5859) (120da92)
chore(components/pytorch) [WIP] Updating docker image location - pytorch kfp samples (#5911) (a437b92)
fix(components/pytorch) Pytorch Lightning Arguments string pass through (#5870) (192b8e8)
fix(sdk compiler):Fix v2 compile for exit handler Fixes #5854 (#5899) (a1740b4)
feat(component/pytorch) Custom handler for cifar10 example ,saves multiple explain attributions methods (#5871) (1983037)
chore(components) - Add SageMaker component owners to AWS owners list (#4756) (60ed8e4)
Update README.md (ec9021f)
fix(components/pytorch) Pytorch - Tensorboard Profiler fix (#5860) (9cfa4df)
refactor(frontend) Set 1 Hour as a default value for recurring pipeline runs (#5824) (a8ffc50)
Update pipeline_spec.proto (#5813) (635f1ba)
feat(components): Adding Trainer component for PyTorch - KFP (#5767) (c5325db)
Update pipeline_spec.proto (#5787) (8501337)
Fix the cache-deployer to run as non root (#5742) (efe916c)

Open source →

1.6.024 May 2021

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api==1.6.0 --upgrade

See the Change Log

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

Features

api: Update pipeline_spec.proto - Add metadata to ArtifactSpec (#5143) (33a7d4d)
backend: Add qps & burst options in clientset. Fix #5095 (#5231) (15b0f50)
backend: Added multi-user pipelines (UI + API); Fixes #4197 (#4835) (5df2801)
backend: Support authentication with ServiceAccountTokens. Part of #5138 (#5286) (5db66e0)
backend: Support pipeline parameters macros in one-off runs (#5086) (b0cb5db)
backend: upgrade argo library to 2.12.6. Part of #4553 (#5041) (df48073)
components: Add Great Expectations validation component (#5235) (796a5f7)
components: Added a Python script sample (#5203) (e250160)
components: Added a Shell script sample (#5204) (1417e5e)
components: Added Kubernetes CRUD operations (#5239) (3b55207)
components: Added support for specifying Min and Max replicas for inference services (#5059) (0a04cba)
components: Converters - OnnxModel - to_TensorflowSavedModel (#5325) (42930d1)
components: Made the Chicago Taxi Trips dataset component compatible with volume-based data passing (#5216) (60a2612)
components: PyTorch - Convert to ONNX from PyTorch ScriptModule (#5207) (fc7afdd)
components: PyTorch - Create fully connected network in PyTorch network (#5205) (37da625)
components: PyTorch - Train fully-connected network pipeline (#5208) (b12d636)
components: PyTorch - Train PyTorch model from CSV (#5206) (dfa7563)
components/google-cloud: Add API documentation (#5603) (e693efa)
components/google-cloud: Add user_agent header for metrics reporting (#5600) (8872ea2)
components/google-cloud: Add version to package root (#5550) (e82d9a7)
components/google-cloud: Support input type union[str,list[str]] (#5637) (695e95e)
components/google-cloud: Use Kaniko for image build (#5536) (289d0f5)
deployment: [GCP marketplace] upgrade argo to v2.12.9. Part of #5232 (#5277) (78db53e)
deployment: keep defaulting to argo docker container runtime executor. Part of #5285 (#5312) (3bcaa52)
deployment: move k8s application manifests to third-party and hide log by default (#5336) (8079cca)
deployment: multi-user mode manifests for subject access review integration. Fixes #3513 (#5314) (4e2c455)
deployment: update mysql to 5.7 and support running as nonroot (#5278) (3cdb1d9)
frontend: Add a new page to monitor existing Jobs (recurring runs) (#5131) (4ff3ffb)
frontend: Hide sidebar in Kubeflow mode. Fixes #5199 (#5200) (0c93397)
frontend: Make pipeline and component clickable on Artifact list (#5311) (2910baa)
frontend: ResourceInfo supports MLMD struct value (#5618) (0fd1e6a)
frontend: Shorten output names (#5408) (5c27e7c)
frontend: Support showing Omitted node phase with new Argo version (#5339) (92ca406)
frontend: Upgrade argo template on UI. Fix #5358 (#5359) (a9f8eec)
samples: sample with kfp.dsl.RUN_ID_PLACEHOLDER. Part of #4700 (#5385) (e3fa90d)
sdk: Add local runner which will run ops in docker or locally. Fixes #1138 (#4983) (8af15a3)
sdk: Add sorted function to definitions to make constant result (#5248) (a0b57c3)
sdk: Adds Metrics & ClassificationMetrics ontology types (#5259) (2323568)
sdk: Enable setting OwnerReference on ResourceOps. Fixes #1779 (#4831) (a12e88d)
sdk: Introduce experimental v2-compatibility in KFP SDK (#5218) (83eded1)
sdk: Introduce new environment variables for kfp oauth2 clients (#5043) (075b83a)
sdk: Introduce the token credentials interface. Part of #4683 (#5287) (8936431)
sdk: Make cli.experiment use kfp.Client methods instead of using kfp_server_api directly (#5267) (7c1793c)
sdk: Refactor v2 component building (#5163) (d3a7fbf)
sdk: Support backoffs in retry strategy (#5060) (fa135fc)
sdk: Type checking to treat "explicit" generic artifact as any artifact type. (#5661) (559fe40)
sdk: Uses 'https' as the default host protocol if users didn't specify any. (#5298) (a1aea7b)
sdk.v1: Add pipeline_root as an argument to client's public run methods. (#5632) (f09131f)
sdk.v2: Add Google AIPlatformClient (#5677) (bf8681b)
sdk.v2: Parse pipeline param in argument values (#5183) (21a012d)
sdk.v2: Support bool, dict, list, JsonObject, and JsonArray type as parameter types (#5665) (2abc352)
adds v2 SDK compatible version of component (#5643) (315d0b0)
google_cloud_components sdk: Enable support for all components (#5442) (d8b2b24)
sdk: Add a component decorator for v2 components. (#5388) (539cbeb)
sdk: Enable InputArtifact/OutputArtifact in lightweight Python-based components (#5378) (bf8392d)
sdk: relax kubernetes upper constraint from <12 to <13 (#5349) (6603e8b)
sdk: Support return values in lightweight Python function components v2. (#5384) (2fc1016)
sdk.v2: Experimental custom job support. (#5604) (f20d20c)
sdk.v2: fix loop implementation and add subvar support (#5461) (85cb991)
sdk.v2: Raise error on artifacts vs. parameters mismatch in v2. (#5557) (d5a1853)
sdk.v2: remove pipeline_root from compiler interface (#5492) (5f97a45)
sdk.v2: Support optional component inputs in v2. (#5383) (eb3a092)
sdk.v2: Support setting custom job spec in v2. (#5454) (f32915e)
sdk.v2: surface metrics output artifacts to pipeline outputs (#5445) (a804211)
sdk/extensions: Add kfp.onprem.use_k8s_secret method that puts k8s secrets in env vars (#5365) (65f7669)
v2: added client side mlmd retry (#5629) (a62ea49)
customizable tensorboard image and env + sample pipeline with minio. Fixes #5471, fixes #4714, fixes #5449, fixes #699, fixes #4290 (#5515) (cadcac0)
sdk.v2: Support explicit importer (#5502) (5dafda1)
sdk.v2: support using placeholder class in custom job spec. (#5495) (3324459)
update argo image to v2.12.9 and automate update process. Fixes #5232 (#5266) (be64b32)
update MLMD to 0.30.0 (#5592) (3193908)
sdk.v2: Implement AI Platform (Unified) Custom job component in KFP v2 SDK (#5146) (ec7201d)
sdk.v2: support dsl.Condition in v2. (#5127) (a2e2e6c)
sdk.v2: Support dsl.ParallelFor in v2. (#5162) (280cae3)
sdk.v2: use consistent argument name in v2 compiler (#5430) (0e921c3)
v2: Support v2 components in KFP v2-compatible mode. (#5458) (5651bd1)

Bug Fixes

backend: api server panics on workflow with parameter without value. Fixes #5423 (#5424) (d95764b)
components: add bypass for BrokenPipeError in gcp ml engine client. Fixes #5085 (#5250) (3f91fe2)
components: Delete unused to_table component item (#4937) (d2fd499)
components: replace resource name class with vertex ai (#5702) (a1ecb3d)
components: Update types for GCS components so it's compatible with v2. (#5532) (961b17f)
components: Web - Download data - Fixed the image which does not support volumes (#5644) (54ac9a6)
deployment: fix default resource requests. Fixes #5148 (#5409) (5d0f3a3)
deployment: Manifests fixes for KF 1.3 (#5406) (ec2da24)
deployment: pipelines profile controller uses pipelines.kubeflow.org/enabled label (#5407) (5ba3054)
docs: updated readme wait condition (#5371) (a73a255)
frontend: Avoid crashing, and show error message for invalid trigger date/time format. (#5418) (19667e1)
frontend: Update the link text to the GCP CAIP Notebook (#5646) (9632509)
manifests: fix error when kpt pkg get manifests. Part of #5368 (#5390) (1b13be7)
manifests: Multi-User manifests fixes for 1.3 (#5416) (6033718)
metadata-writer: skip latest TFX pods (#5364) (6620dbd)
samples: Allow user to set the namespace and name for the sample as parameters. (#5212) (b397ec1)
sdk: Always create output paths for parameters in addition to artifacts in v2 Python components. (#5421) (1e32f01)
sdk: Enable FutureWarning with PipelineParam. Fix #5280 (#5299) (d8028c5)
sdk: faster get experiment by name respects casing. Fixes #5172 (#5370) (43486e7)
sdk: Fix a bug with creating dirname for parameter files in v2 lightweight components (#5574) (e4c6e12)
sdk: Fix bug where empty string was not preserved as pipeline param default value. (#5549) (2909f25)
sdk: Fix serialization of ExecutorInput. (#5403) (c46d6a2)
sdk: Minor API fixes to Metrics classes. (#5494) (55c14cc)
sdk: Support returning plain tuples in lightweight function components v2. (#5412) (a6a7255)
sdk: Update documentation for cron_expression parameter to kfp.Client().create_recurring_run (#5642) (8b97562)
sdk: Update syntax for lightweight components v2. (#5568) (b46f1b8)
sdk.v2: fix a small bug in io_types.is_artifact_annotation() (#5699) (b7084f2)
sdk.v2: fix bug for component input parameter/artifact referencing (#5419) (eb55842)
sdk.v2: Fix loop subvar support and remove redundant loop iterator component. (#5516) (74950d6)
v2: added default pipeline root (#5660) (ef547d0)
v2: Escape strings in RuntimeInfo. (#5601) (c5daa75)
viewer: viewer crd controller panics using tensorboard image without tag. Part of #5471 (#5522) (d106a65)
pass in class directly (#5488) (5d067f6)
backend: add brackets if metadata service host is ipv6. Fixes (#5161) (#5246) (1d8df97)
backend: add brackets if metadata service host is ipv6. Part of #5605 (#5160) (710ca26)
backend: Caching - Make the cache key depend on output names and paths (#4675) (5fbf224)
backend: fixes sql error when deploy KFP in ANSI_QUOTES sql mode. Fixes #5121 (#5156) (e82aec6)
backend: remove Bazel from building the API. Part of #3250 (#4906) (e669b22)
cache: do not cache TFX pods for newer TFX versions (#5188) (76cad52)
cache: update to use google/cloud-sdk:alpine instead of alpine for mutating cached steps. Fixes #4099 (#5184) (9c10d4f)
components: fixes issues with tfjob launcher component. (#4806) (dd31142)
components: Print logs for AWS SageMaker components (#4879) (079eea3)
deployment: Set default resource requests and limits for pipeline pods. Fixes #5214 (#5293) (67afca4)
deployment: Updated the manifests with resource requests. Fixes #5148 (#5158) (98f946a)
frontend: example-validator visualization for TFX 0.27.0. Part of #5137 (#5186) (8588c9a)
frontend: Show recurring run display name rather than ID. Fix #1786 (#5141) (468c65b)
manifests: Fix multi-user manifest (#5344) (a2e5568)
samples: Change Katib Trial image tag to v1beta1-45c5727 (#5330) (1249719)
samples: remove TFX iris example. Part of #5137 (#5189) (875137a)
samples: Update KFP version in sample to fix backslash recognization issue. (#5140) (d902cae)
samples: Update tfx image. Fixes #5137 (#5165) (1fee405)
sdk: fix bug in store_value_bytes when read old data. Fixes #5288 (#5290) (42630cb)
sdk: generate valid pipeline from non-ascii name. Fixes #5119 (#5120) (cfe5913)
sdk: include ontology_type_schemas/*.yaml in setuptools (#5241) (430bd6b)
sdk: Local runner - Fixed the local runner on Windows (#5313) (d0cbca7)
sdk: make workflow validation compatible with argo CLI v2.12+. Fixes #4775, Fixes #5296 (#4804) (8a32632)
sdk: only apply importer node when compiling to IR (#5264) (796ee37)
sdk: update default value for type_check in compiler docstrings (#5169) (37d6554)
sdk: v2 - Compiler - Fixed the v2 component detection (#5340) (e11241f)
test: Update reference link in test README.md (#5354) (3d441b0)
runasnonroot for kubeflow-pipelines-controller (#5294) (6ebff17)
tests: Alter hardcoded sdk version in aiplatform_e2e_test.py test data before comparison. (#5220) (14af539)
tests: Disable the SDK unittest that requires docker binary in local environment. (#5211) (45c5c18)
visualization: downgrade tfma to 0.26 (#5191) (9094fa3)
upgrade tfx samples & deps to 0.27.0. Part of #5137 (#5176) (a394f8c)

Other Pull Requests

Backend - Added modern time formatters (#5464) (584d5e5)
Fix pipelines with Kubeflow profile quota (#5695) (87d9dd5)
Update IR documentations (#5647) (0c6b995)
Update init.py (#5678) (0ae53a9)
remove task prefix (#5654) (d892dc0)
update client to allow equal signs (#5650) (929b56f)
Support extra params for cache server db connection (#5599) (13b27f6)
feat(components/google-cloud):Add support for URI prefix (#5625) (a89dc41)
Add configurable connection max lifetime for MySQL (#5583) (d25b001)
update xgboost sample pipeline (#5594) (00f3b3d)
Add OWNERS for v2 folder (#5571) (9160618)
fix(v2):added double value (#5591) (0a506f2)
[SDK] Add pod labels for telemetry purpose (#5582) (6717434)
Assigned copyright to the project authors (#5587) (cc83e10)
Add zijianjoy to frontend/OWNERS (#5570) (bb967e9)
chore(sdk.v2):add v2 compact metrics label (#5566) (0dba005)
feat(cache) : allow user to reset the node selector and affinity for cached steps. Fixes #4777 (#5456) (837429c)
Update Data passing in python components - Files.py (acc2c2a)
DOCS url 404 (#5455) (16e46ac)
[google-cloud-components] - change name to google-cloud-pipeline-components (#5514) (e6d3536)
[google-cloud-components] - update metadata for output (#5499) (cb4d644)
[google-cloud-components] add encryption key support (#5498) (eb00b7b)
[google-cloud-components sdk] casting on remote runner (#5497) (2c5e779)
[google-cloud-component-sdk] fix: Ignore none (#5490) (78b98c7)
Use artifacts for resource name (#5469) (3b8296f)
Fixes the Confusion Matrix storage format (#5463) (2c68229)
metadata-writer: Fix multi-user mode (#5417) (684d639)
Adds classification metrics type to io_types (#5387) (0e7e543)
Update pipeline_spec.proto (#5342) (1e8d499)
feat(components) Update KFServing component to use v0.5.1 (#5319) (9374c9a)
chore (sdk) typo err in cli/pipeline suppy -> supply (#5297) (66f410d)
fix(sdk/tests) fix parallelfor_item_argument_resolving compiler test, Fixes #5270 (#5271) (9708a2a)
Support baseline/candidate TFMA visualization (#5260) (63ec6c7)
feat(components) Update KFServing component to v0.5.0 (#5201) (5ffa2be)
Changes Artifact schema folder name (#5243) (81cdd82)
Artifact class update to support protobuf Struct (#5197) (3e55bd3)
fix(backend):Update with to allow eviction of pods with empty dir. Fixes #4530 (#5171) (1983041)
<feat>[components]: Add pipeline component to schedule spark jobs in Azure Synapse (#5070) (e976d6d)
Changes for kfserving 0.4.1 (#4479) (65bed9b)

Open source →

1.6.0rc013 May 2021pre-release

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install kfp-server-api package (python 3.6 above) by running:

python3 -m pip install kfp-server-api --pre --upgrade

See the Change Log

NOTE, kfp python SDK is NOT included and released separately.

Open source →

Additional notes

Features

components/google-cloud: Add API documentation (#5603) (e693efa)
components/google-cloud: Add user_agent header for metrics reporting (#5600) (8872ea2)
components/google-cloud: Add version to package root (#5550) (e82d9a7)
components/google-cloud: Support input type union[str,list[str]] (#5637) (695e95e)
components/google-cloud: Use Kaniko for image build (#5536) (289d0f5)
frontend: ResourceInfo supports MLMD struct value (#5618) (0fd1e6a)
sdk: Introduce new environment variables for kfp oauth2 clients (#5043) (075b83a)
sdk.v1: Add pipeline_root as an argument to client's public run methods. (#5632) (f09131f)
sdk.v2: Experimental custom job support. (#5604) (f20d20c)
customizable tensorboard image and env + sample pipeline with minio. Fixes #5471, fixes #4714, fixes #5449, fixes #699, fixes #4290 (#5515) (cadcac0)
google_cloud_components sdk: Enable support for all components (#5442) (d8b2b24)
sdk.v2: fix loop implementation and add subvar support (#5461) (85cb991)
sdk.v2: Raise error on artifacts vs. parameters mismatch in v2. (#5557) (d5a1853)
sdk.v2: remove pipeline_root from compiler interface (#5492) (5f97a45)
sdk.v2: Support setting custom job spec in v2. (#5454) (f32915e)
sdk.v2: support using placeholder class in custom job spec. (#5495) (3324459)
update MLMD to 0.30.0 (#5592) (3193908)
sdk.v2: Support explicit importer (#5502) (5dafda1)
sdk.v2: surface metrics output artifacts to pipeline outputs (#5445) (a804211)
v2: Support v2 components in KFP v2-compatible mode. (#5458) (5651bd1)

Bug Fixes

components: Update types for GCS components so it's compatible with v2. (#5532) (961b17f)
sdk: Fix a bug with creating dirname for parameter files in v2 lightweight components (#5574) (e4c6e12)
sdk: Fix bug where empty string was not preserved as pipeline param default value. (#5549) (2909f25)
sdk: Minor API fixes to Metrics classes. (#5494) (55c14cc)
sdk: Update syntax for lightweight components v2. (#5568) (b46f1b8)
sdk.v2: Fix loop subvar support and remove redundant loop iterator component. (#5516) (74950d6)
v2: Escape strings in RuntimeInfo. (#5601) (c5daa75)
viewer: viewer crd controller panics using tensorboard image without tag. Part of #5471 (#5522) (d106a65)
pass in class directly (#5488) (5d067f6)

Other Pull Requests

feat(components/google-cloud):Add support for URI prefix (#5625) (a89dc41)
Add configurable connection max lifetime for MySQL (#5583) (d25b001)
update xgboost sample pipeline (#5594) (00f3b3d)
Add OWNERS for v2 folder (#5571) (9160618)
fix(v2):added double value (#5591) (0a506f2)
[SDK] Add pod labels for telemetry purpose (#5582) (6717434)
Assigned copyright to the project authors (#5587) (cc83e10)
Add zijianjoy to frontend/OWNERS (#5570) (bb967e9)
chore(sdk.v2):add v2 compact metrics label (#5566) (0dba005)
feat(cache) : allow user to reset the node selector and affinity for cached steps. Fixes #4777 (#5456) (837429c)
Update Data passing in python components - Files.py (acc2c2a)
DOCS url 404 (#5455) (16e46ac)
[google-cloud-components] - change name to google-cloud-pipeline-components (#5514) (e6d3536)
[google-cloud-components] - update metadata for output (#5499) (cb4d644)
[google-cloud-components] add encryption key support (#5498) (eb00b7b)
[google-cloud-components sdk] casting on remote runner (#5497) (2c5e779)
[google-cloud-component-sdk] fix: Ignore none (#5490) (78b98c7)
Use artifacts for resource name (#5469) (3b8296f)
Fixes the Confusion Matrix storage format (#5463) (2c68229)

Open source →

1.5.0rc023 Mar 2021pre-release

api: Update pipeline_spec.proto - Add metadata to ArtifactSpec (\#5143)

Features

api: Update pipeline_spec.proto - Add metadata to ArtifactSpec (#5143) (33a7d4d)
backend: Add qps & burst options in clientset. Fix #5095 (#5231) (15b0f50)
backend: Added multi-user pipelines (UI + API); Fixes #4197 (#4835) (5df2801)
backend: Support authentication with ServiceAccountTokens. Part of #5138 (#5286) (5db66e0)
backend: Support pipeline parameters macros in one-off runs (#5086) (b0cb5db)
backend: upgrade argo library to 2.12.6. Part of #4553 (#5041) (df48073)
components: Add Great Expectations validation component (#5235) (796a5f7)
components: Added a Python script sample (#5203) (e250160)
components: Added a Shell script sample (#5204) (1417e5e)
components: Added Kubernetes CRUD operations (#5239) (3b55207)
components: Added support for specifying Min and Max replicas for inference services (#5059) (0a04cba)
components: Made the Chicago Taxi Trips dataset component compatible with volume-based data passing (#5216) (60a2612)
components: PyTorch - Convert to ONNX from PyTorch ScriptModule (#5207) (fc7afdd)
components: PyTorch - Create fully connected network in PyTorch network (#5205) (37da625)
components: PyTorch - Train fully-connected network pipeline (#5208) (b12d636)
components: PyTorch - Train PyTorch model from CSV (#5206) (dfa7563)
deployment: [GCP marketplace] upgrade argo to v2.12.9. Part of #5232 (#5277) (78db53e)
deployment: keep defaulting to argo docker container runtime executor. Part of #5285 (#5312) (3bcaa52)
deployment: move k8s application manifests to third-party and hide log by default (#5336) (8079cca)
deployment: multi-user mode manifests for subject access review integration. Fixes #3513 (#5314) (4e2c455)
deployment: update mysql to 5.7 and support running as nonroot (#5278) (3cdb1d9)
frontend: Add a new page to monitor existing Jobs (recurring runs) (#5131) (4ff3ffb)
frontend: Hide sidebar in Kubeflow mode. Fixes #5199 (#5200) (0c93397)
frontend: Make pipeline and component clickable on Artifact list (#5311) (2910baa)
sdk: Add local runner which will run ops in docker or locally. Fixes #1138 (#4983) (8af15a3)
sdk: Add sorted function to definitions to make constant result (#5248) (a0b57c3)
sdk: Adds Metrics & ClassificationMetrics ontology types (#5259) (2323568)
sdk: Enable setting OwnerReference on ResourceOps. Fixes #1779 (#4831) (a12e88d)
sdk: Introduce experimental v2-compatibility in KFP SDK (#5218) (83eded1)
sdk: Make cli.experiment use kfp.Client methods instead of using kfp_server_api directly (#5267) (7c1793c)
sdk: Refactor v2 component building (#5163) (d3a7fbf)
sdk: Support backoffs in retry strategy (#5060) (fa135fc)
sdk: Uses 'https' as the default host protocol if users didn't specify any. (#5298) (a1aea7b)
sdk.v2: Parse pipeline param in argument values (#5183) (21a012d)
update argo image to v2.12.9 and automate update process. Fixes #5232 (#5266) (be64b32)
sdk.v2: Implement AI Platform (Unified) Custom job component in KFP v2 SDK (#5146) (ec7201d)
sdk.v2: support dsl.Condition in v2. (#5127) (a2e2e6c)
sdk.v2: Support dsl.ParallelFor in v2. (#5162) (280cae3)

Bug Fixes

backend: add brackets if metadata service host is ipv6. Fixes (#5161) (#5246) (1d8df97)
backend: add brackets if metadata service host is ipv6. Part of #5605 (#5160) (710ca26)
backend: fixes sql error when deploy KFP in ANSI_QUOTES sql mode. Fixes #5121 (#5156) (e82aec6)
backend: remove Bazel from building the API. Part of #3250 (#4906) (e669b22)
cache: do not cache TFX pods for newer TFX versions (#5188) (76cad52)
cache: update to use google/cloud-sdk:alpine instead of alpine for mutating cached steps. Fixes #4099 (#5184) (9c10d4f)
components: fixes issues with tfjob launcher component. (#4806) (dd31142)
components: Print logs for AWS SageMaker components (#4879) (079eea3)
deployment: Set default resource requests and limits for pipeline pods. Fixes #5214 (#5293) (67afca4)
deployment: Updated the manifests with resource requests. Fixes #5148 (#5158) (98f946a)
frontend: example-validator visualization for TFX 0.27.0. Part of #5137 (#5186) (8588c9a)
frontend: Show recurring run display name rather than ID. Fix #1786 (#5141) (468c65b)
manifests: Fix multi-user manifest (#5344) (a2e5568)
samples: Change Katib Trial image tag to v1beta1-45c5727 (#5330) (1249719)
samples: remove TFX iris example. Part of #5137 (#5189) (875137a)
sdk: Local runner - Fixed the local runner on Windows (#5313) (d0cbca7)
sdk: make workflow validation compatible with argo CLI v2.12+. Fixes #4775, Fixes #5296 (#4804) (8a32632)
runasnonroot for kubeflow-pipelines-controller (#5294) (6ebff17)
samples: Update KFP version in sample to fix backslash recognization issue. (#5140) (d902cae)
samples: Update tfx image. Fixes #5137 (#5165) (1fee405)
sdk: generate valid pipeline from non-ascii name. Fixes #5119 (#5120) (cfe5913)
sdk: include ontology_type_schemas/*.yaml in setuptools (#5241) (430bd6b)
sdk: only apply importer node when compiling to IR (#5264) (796ee37)
sdk: update default value for type_check in compiler docstrings (#5169) (37d6554)
tests: Alter hardcoded sdk version in aiplatform_e2e_test.py test data before comparison. (#5220) (14af539)
tests: Disable the SDK unittest that requires docker binary in local environment. (#5211) (45c5c18)
visualization: downgrade tfma to 0.26 (#5191) (9094fa3)
upgrade tfx samples & deps to 0.27.0. Part of #5137 (#5176) (a394f8c)

Other Pull Requests

feat(components) Update KFServing component to use v0.5.1 (#5319) (9374c9a)
chore (sdk) typo err in cli/pipeline suppy -> supply (#5297) (66f410d)
fix(sdk/tests) fix parallelfor_item_argument_resolving compiler test, Fixes #5270 (#5271) (9708a2a)
Support baseline/candidate TFMA visualization (#5260) (63ec6c7)
feat(components) Update KFServing component to v0.5.0 (#5201) (5ffa2be)
Changes Artifact schema folder name (#5243) (81cdd82)
Artifact class update to support protobuf Struct (#5197) (3e55bd3)
fix(backend):Update with to allow eviction of pods with empty dir. Fixes #4530 (#5171) (1983041)
<feat>[components]: Add pipeline component to schedule spark jobs in Azure Synapse (#5070) (e976d6d)
Changes for kfserving 0.4.1 (#4479) (65bed9b)

1.4.1 (2021-02-25)

Bug Fixes

backend: fixes sql error when deploy KFP in ANSI_QUOTES sql mode. Fixes #5121 (#5156) (7178f9f)
backend: remove Bazel from building the API. Part of #3250 (#4906) (2a83336)
cache: do not cache TFX pods for newer TFX versions (#5188) (c2c5e83)
components: fixes issues with tfjob launcher component. (#4806) (c767329)
frontend: example-validator visualization for TFX 0.27.0. Part of #5137 (#5186) (89c3b61)
frontend: Show recurring run display name rather than ID. Fix #1786 (#5141) (f802b6d)
samples: remove TFX iris example. Part of #5137 (#5189) (09a7d03)
visualization: downgrade tfma to 0.26 (#5191) (3fbafc0)
upgrade tfx samples & deps to 0.27.0. Part of #5137 (#5176) (acf6f7b)
samples: Update KFP version in sample to fix backslash recognization issue. (#5140) (5e10ab7)
samples: Update tfx image. Fixes #5137 (#5165) (5b61bf5)

1.4.1-rc.1 (2021-02-25)

Bug Fixes

backend: fixes sql error when deploy KFP in ANSI_QUOTES sql mode. Fixes #5121 (#5156) (7178f9f)
backend: remove Bazel from building the API. Part of #3250 (#4906) (2a83336)
cache: do not cache TFX pods for newer TFX versions (#5188) (c2c5e83)
components: fixes issues with tfjob launcher component. (#4806) (c767329)
frontend: example-validator visualization for TFX 0.27.0. Part of #5137 (#5186) (89c3b61)
frontend: Show recurring run display name rather than ID. Fix #1786 (#5141) (f802b6d)
samples: remove TFX iris example. Part of #5137 (#5189) (09a7d03)
visualization: downgrade tfma to 0.26 (#5191) (3fbafc0)
upgrade tfx samples & deps to 0.27.0. Part of #5137 (#5176) (acf6f7b)
samples: Update KFP version in sample to fix backslash recognization issue. (#5140) (5e10ab7)
samples: Update tfx image. Fixes #5137 (#5165) (5b61bf5)

Open source →

1.2.018 Dec 2020

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.6 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

Open source →

Additional notes

Features

components: Add v1beta1 Katib launcher and samples (#4798) (89e4210)
sdk: Add artifact ontology and migrate compiler utils to onboard artifact types (#4901) (2386471)
sdk: Add settings of the dnsConfig field. Fixes #4836 (#4837) (5a4b70e)
sdk: allow calling GroupOp.after with multiple ops (#4788) (5169489)
sdk: Components - Restored stack traces in lightweight python components. Fixes #4273, #4849 (#4861) (7a66414)
SDK: adds Artifact base class. (#4895) (7591805)

Bug Fixes

frontend: fixed getting executions failure. Fixes #4903 (#4907) (df7a66f)
frontend: Use task_display_name annotation when displaying run metrics (#4875) (d4e8b87)
sdk: Do not wait for resource deletion (#4820) (8f70bf3)
sdk: make healthz exception visible in logs by default (#4904) (44fcda7)

1.1.2 (2020-12-14)

Features

backend: Make number of persistence worker goroutine configurable (#3904) (8a2d11c)
sdk: add ability to set retry policy (#4858) (2f3a686)
Set current namespace for in-cluster SDK in multi-user mode and add healthz endpoint to API backends (#4638) (0df9473)
API: exposing api for setting the default version of pipeline. Fixes #4049 (#4406) (5742991)
backend: allow configuring if default version should be updated when uploading new pipeline version. Fixes #4049 (#4476) (2317015)
backend: Metadata Writer - Record parameter argument values to MLMD (#4564) (ec65dfe)
backend: new server API to read run log. Fixes #4468 (#4493) (678ae0f)
backend: Refactor authz to perform SubjectAccessReview. Fixes #3513 (#4723) (39203d5)
backend: sort by run metrics - step 2 (#4235) (d4d3616)
backend: sort by run metrics - step 3. Part of #3591 (#4251) (9c6738f)
backend: sorting on run metrics - step 1 (#4203) (a96e8fe)
cache: Explicitly specifying which attributes affect the cache key (#4076) (f35462f)
compiled: set pod disruption budget for pipelines. Fixes #3877 (#4178) (c32ea23)
components: Added the component to convert from KerasModelHdf5 to TensorflowSavedModel (#4524) (4830e03)
components: [AWS SageMaker] Minimize inputs for mnist classification pipeline (#4192) (6860681)
components: Add yaml support to kfserving component (#4587) (2a1fe45)
components: Added a sample C# script component (#4815) (c5e6ca2)
components: Added a sample R script component (#4817) (3826edf)
components: Added Calculate classification metrics component (#4441) (9a00eb3)
components: Added converter components for Tensorflow Lite models (#4532) (1802db3)
components: Added converter components for TensorflowJS GraphModel (#4526) (3927a4f)
components: Added converter components for TensorflowJS LayersModel (#4527) (b1f46ac)
components: Added converters to OnnxModel from Tensorflow and Keras (#4550) (4ec84e6)
components: Added JSON query components (#4601) (b656fbf)
components: Added the Basics - Calculate data hash component (#4503) (68a367d)
components: Added the Google Cloud Optimizer components (#4621) (46eb6f2)
components: Added the Run notebook using papermill component. Fixes #497 (#4578) (329ed48)
components: AutoML Tables - Added the Create dataset from CSV component (#4558) (a316e1e)
components: AWS SageMaker - Add functionality to stop SageMaker jobs on run termination (#4167) (900eeae)
components: AWS SageMaker - Add optional parameter to allow training component to accept parameters related to Debugger (#4283) (3ebd075)
components: AWS SageMaker - Changes for updating an existing endpoint (#4424) (7c349f3)
components: AWS SageMaker - Support for assuming a role (#4212) (8014a44)
components: Azure ML component to deploy model (#4711) (ec721fe)
components: Google - AutoML - Added the Deploy model component (#4520) (5fe8ef4)
components: Google - AutoML - Added the Export model component (#4519) (d5ae723)
components: Google Cloud Optimizer - Added the "Suggest parameter sets from measurements" component (#4633) (f30aa09)
components: JSON - Added the "Build dict", "Build list" and "Combine lists" components (#4631) (616c48b)
components: Keras - Added the Train_classifier component (#4274) (5f7afe3)
components: ml_metrics - Calculate_regression_metrics - Added the metrics and number_of_items outputs (#4239) (1d8abb7)
components: Simplified the kubeflow - dnntrainer component (#4415) (b769416)
components: web - Download (#4068) (240543e)
deployment: Add env/platform-agnostic-pns standalone deployment option (#4278) (dac5a94)
deployment: GCP managed storage - detailed instructions to set up workload identity bindings before deployment (#4232) (45a91f6)
deployment: KFP standalone should keep user data when application deleted (#4332) (f738cb3)
deployment: Kustomize manifest for Azure deployments (#4567) (7a5c3c6)
deployment: marketplace - allow specifying gcs bucket directly (#4112) (d4a8329)
frontend: Show full node names. Fixes #3989 (#4663) (85bdbc1)
frontend: support tensorboard viewer and other visualize Results using volume mount path. Part of #4208 (#4236) (7f77e8b)
frontend: UX change to support downloading directory artifacts. Fixes #3667 (#4696) (e3992fa)
gcp: Revert "turn off archiveLogs for GCP marketplace #4758" (#4771) (aa162e6)
gcp: turn off archiveLogs for GCP marketplace (#4758) (1165ed2)
manifest: add support for Standalone KFP on AWS. Fixes #4337 (#4350) (5ec7ec8)
samples: Added "Output a directory" sample (#4437) (89cdaa4)
samples: Added the caching sample. Fixes #4636 (#4664) (9597c8b)
sdk: add aws region field in use_aws_secret in kfp sdk (#4363) (390e80e)
sdk: Added optional argument to specify description for pipeline upload (#4577) (dd1e3d8)
sdk: added pipeline name option to kfp run submit (#4535) (38946d8)
sdk: Adds missing short options to the cli (#4604) (ce6cfa5)
sdk: adds support for ephemeral-storage in container-op (#4504) (0653e7c)
sdk: Components - Prevent passing unserializable objects to components. Fixes #4040 (#4496) (0332584)
sdk: Python components - Parse component input/output descriptions from the function docstring (#4512) (5613db0)
add grafana as part of the customized kfp deployment (#4404) (f39d879)
add kpf pipeline list-versions command (#3928) (c9f621a)
add parallelism for dsl.ParallelFor. Fixes #4089 (#4149) (c6ac83f)
allow to use pipeline version (#3930) (40f06bf)
GCP marketplace - preserve install config in configmap and secret (#4471) (fbfc160)
sdk: CLI - Allow upload_version by the pipeline's name. Fixes #3901 (#4087) (aae1eba)
sdk: Client - Getting access token without gcloud CLI (#4811) (ab21ec9)
sdk: Components - Added Bool as a known type name (#4597) (d4fd6e1)
sdk: DSL - Added support for volatile components (#4104) (d707b93)
sdk: Expose upload pipeline version (#4146) (45c796e)
sdk: Fix print message for run submission with respect to output format (#4691) (a8bd468)
sdk: optional context in kubenetes client (#4401) (aed37c7)
sdk: Preserve parameter arguments and input names (#4563) (e8fb58a)
sdk: SDK - Components - Added annotations to InputSpec and OutputSpec (#4154) (9b15800)
sdk: SDK - Components - Added annotations to TaskSpec (#4153) (afd8161)
sdk: SDK - Deprecation warning when using ContainerOp (#4166) (2f94827)
sdk: SDK - Enable placeholders in task display names. Fixes #4163 (#4164) (db0af86)
sdk: support HTTP/S PROXY for SDK client (#4215) (ac66626)
sdk: Support to print the cli command output in json format (#4566) (0795597)
sdk/cli: add extra header for cli (#4400) (c8b4599)
sdk/client: add cookie configuration to kfp client (#2679) (d91ac0f)

Bug Fixes

backend: add MaxCallRecvMsgSize(math.MaxInt32) to proxy server (#4402) (cd9c9ff)
backend: Add a permission to create events to argo-role (#4449) (d40d66a)
backend: added setup structure to simplify adding new tests and removing duplicated code. Fixes #4630 (#4639) (92a932e)
backend: allow empty userid header prefix. Fixes #4091 (4098) (042ff09)
backend: Backend - Cache - Fixed reinstallation. Fixes #4299 (#4320) (fe77c19)
backend: Cache Deployer - Fixed grep call (#4568) (24217ff)
backend: Caching - Fixed deployer failure on Kubernetes v1.16+. Fixes #4627 (#4632) (dde7f9a)
backend: Caching - Only send cache-enabled pods to the caching webhook (#4429) (6b54eec)
backend: Caching - Reduced the cache webhook timeout (#4428) (ada18bc)
backend: fix the google-api-core to 1.16.0 for backend visualization server. (#4158) (cf29c61)
backend: fix typo in reference key type (#4376) (32c9c2a)
backend: fixes useless error message when visualization-server is not accessible. Fixes #4157 (#4201) (d3d4dcb)
backend: improve forward / backward compatibility on db status table (#4351) (384afac)
backend: improve forward / backward compatibility on experiment table (#4349) (5141201)
backend: job api -- deletion/disabling should succeed when swf not found. Fixes #4871 (#4884) (4df8925)
backend: logs error when failing to init mysql. Fixes #4334 (#4335) (bb21597)
backend: Metadata Writer - Fixed regression with artifact type retrieval. Fixes #3971 (#4231) (1990588)
backend: Metadata Writer - Fixed setting execution custom properties (#4670) (fc6b5d6)
backend: persistence agent - workflow not found error should be a permanent error (#4486) (29a6aaa)
backend: prevent seg fault if workflow manifest is deleted. Fixes #4389 (#4439) (ec733c9)
backend: reduce confusing ReadArtifact errors for metrics in api server. Fixes #3699 (#4338) (01a7998)
backend: skip reporting native Argo workflows which do not have Run ID label. Fixes #3584 (#4438) (1f2d417)
backend: updated the argo version too 2.7.7. Fixes #4392 (#4498) (d7793af)
cache: adds certificate approver permission to kubeflow-pipelines-cache-deployer-role. Fixes #4138 (#4246) (#4383) (265c9b7)
cache: Cache deployer - Using the same kubectl version as the server (#4525) (58584e1)
cache: Cache executions with no outputs. Fixes #3507 (#3808) (40353bf)
cache: cache-deployer should check both secret and config (#4186) (e4f4250)
cache: Fix cache deployer not regenerating secrets when secret not present (#4171) (75336f7)
component: Add new apply action to kfserving component (#4108) (f21e0fe)
component: Update default TensorFlow version for Watson ML example (4109) (de2e0f2)
components: Add explicit image tag and fix variable references. (#4298) (fed37f7)
components: add istio inject and ready status to kfserving component (#4362) (f3a02d1)
components: AWS SageMaker - Fix MinIO PID early exit (#4190) (c6754e3)
components: AWS SageMaker - Retry delete EKS Cluster after Integ test failure (#4662) (008985a)
components: Fix kfserving component url parsing (#4780) (7f32dde)
components: fix ml_engine/train interface (#4593) (d115d46)
components: Fixed the Download from GCS components for blob URIs (#4557) (0ffcbd7)
components: Fixed the GCP - ML Engine - Batch predict component. Fixes #4430 (#4432) (2beddd0)
components: GCP - AutoML - Fixed input type in Batch predict (#4499) (41ce68e)
components: Login to Dockerhub before running tests for Sa… (#4826) (c78781d)
components: Move output path as arguments for ibm components (#4647) (121bfdf)
components: Update watson ml sdk with the Aug 2020 release (#4331) (25ee59d)
samples: fixes name error and unused parameter in samples for loop_parallelism (#4187) (57bb7ac)
sdk: Fix opsgroups dependency resolution (#4370) (22b7b99)
all big enough ui proxy requests fail with error proxying with partial data (#4266) (c7300c3)
Bitbucket repo no longer available (#4395) (fe21cd2)
components: remove needless arguments from AWS EMR scripts (#4252) (262b288)
components: sending pyspark jobs and set generated step_id to /output.txt from the generated EMR step (#4725) (d6a2c23)
components: sending pyspark jobs to aws EMR with the correct py … (#4721) (b9e2259)
components: use default python version for launching dataflow jobs. Fixes #2601 (#4583) (a013edc)
deployment: fix database name prefix description (#4198) (c615e37)
deployment: gcp managed storage - add service account to minio and cloudsql proxy deployments for workload identity (#4188) (c5c9739)
doc: Use Managed NodeGroup for EKS Cluster (#4609) (1c2a4e8)
frontend: do not render Retry nodes in UI DAG as they are virtual nodes. Fixes #4425 #2705 (#4474) (ca29b5c)
frontend: Fix logic deciding if a node was cached. Fixes #4814 (#4819) (e3bbf16)
frontend: fix parsing large workflow graph. Fixes #4179 (#4180) (02b0899)
frontend: Fix the yaml cannot show in Safari browser issue. Fixes #4727 (#4818) (1108ad3)
Process: update backend development README. Fixes #4750 (#4774) (1b924e6)
proxy: fixes GCP inverse proxy url priority. Fixes #4284 (#4702) (9d6ab4c)
sample: Fix syntax error in openvino sample component (#4181) (70ac32c)
samples: Keras - Fixed the keras sample (#4523) (9287287)
samples: Switched the AutoML sample from BQ to GCS datasets (#4501) (3fd72e4)
samples: Updated and fixed the secrets sample (#4590) (c8ce4fa)
sdk: added clearer error for case when user uses watch and argo isn't installed (#4511) (37362a3)
sdk: Allow non-pythonic names for graph components' task's outputs. Fixes #4514. (#4515) (0b31879)
sdk: Client - Fixed wait_for_run_completion (#4808) (5c1b919)
sdk: Compiler - Fixed input artifact name sanitization when using raw string arguments. Fixes #4110 (#4120) (48889a9)
sdk: Compiler - Fixed pipeline parameters with empty default values (#4552) (f7874d3)
sdk: Compiler - Fixed the input argument mapping when using dsl.graph_component. Fixes #3915 (4082) (1da1cf4)
sdk: Compiler - Fixed the input argument mapping when using dsl.graph_component. Fixes #3915 (4082) (6960366)
sdk: Components - Fixed handling of typing.NamedTuple in Python 3.9 (#4614) (8699a05)
sdk: Components - Fixed ModelBase handling of open generic types in Python 3.9 (#4613) (be89972)
sdk: DSL - Enabled arbitrary ContainerOp names (#4554) (1aa8068)
sdk: Fix get_experiment when there are no experiments. Fixes #4176 (#4177) (749538d)
sdk: Fixed UI metadata and metrics (#4672) (80e1d70)
sdk: Fixes #4703: conflict between .pylintrc and .yapf (#4706) (0179584)
sdk: fixes dsl.ContainerOp deprecation warning not shown (#4658) (c52a81c)
sdk: fixes the default value for _is_refresh_token. Fixes #4189 (#4195) (d3ef0dc)
sdk: fixes too broad except when retrieving experiments. (#4765) (950fae4)
sdk: SDK - Avoiding deprecated ContainerOp methods (#4134) (d0b799e)
sdk: SDK - Components - Fixed the error message format string (#4130) (79f6a40)
sdk: TypeError in Client.get_pipeline_id() when pipeline does not exist (#4319) (d7e5894)
sdk: use a meaningful link text in ipython (#4570) (a840ce7)
sdk: VolumeOp has apiVersion as parameter (#4694) (5f992f5)
sdk/client: reserve the host protocal (http or https) so that http host can work. Fixes #4277 (#4285) (cc78bd1)
sdk/component: ml_engine deploy version is incorrect type. Fixes #4291 (#4292) (2394bb2)
increase TFX version from 0.20.2 to 0.22.0. Fixes #4084, fixes #4114 (#4133) (ce51c59)

Other Pull Requests

feat(components) Adds RoboMaker and SageMaker RLEstimator components (#4813) (4aa11c3)
[IR] Add deployment spec to IR spec. (#4881) (ee85ed5)
[Doc] update docs that still refer to KFP latest SDK reference (#4845) (518b8b8)
Update RELEASE.md (1d4ebdd)
Update RELEASE.md (6acd19b)
fix(sdk) - fixes missing import in KFP compiler (#4741) (9efc9e5)
kfp sdk version to 1.1.0 (#4740) (79e9f68)
SDK - Restored module decoupling (#4625) (ccc763f)
compiler for IR (#4529) (5020fd1)
Update IR ExecutorOutput message to use ArtifactList for artifacts field. (#4619) (72ce138)
sample azure components (#4509) (210eb8a)
update kfp installation instructions for a number of gcp components (#4594) (7e4d9f4)
Samples - Added the continue_training_from_prod sample (#4561) (2471f26)
Update ImporterSpec artifact_uri type (#4548) (629a980)
Package pipeline_spec proto into SDK (#4546) (097b186)
Pipeline IR (#4371) (d8e8260)
[AWS SageMaker] Fix small bugs (#4161) (05398cf)
Manifests - Added permissions for certificate approval (#4385) (d9ee639)
Fix (readme): fixed a typo. (#4410) (b49f5c9)
update owners file (#4316) (d135693)
Updated components/aws/sagemaker/README.md (#3983) (e3c33a6)
Update README.md (#4260) (ee66814)
Components - ml_metrics - Added the Aggregate_regression_metrics component (#4240) (d4d692e)
Components - XGBoost - Added the Train_regression_and_calculate_metrics component (#4243) (8d373e7)
Components - XGBoost - Added Train_and_cross-validate_regression component (#4245) (acb50e7)
Components - XGBoost - Added the Cross_validation_for_regression component (#4244) (34cb59d)
SDK - Components - Temporary removed Kubernetes options from ComponentSpec (#4230) (39975e3)
Add myself as approver/reviewer (#2254) (193d143)
SDK - Components - Fixed examples in docstrings (#4074) (aeb0401)
Components - tables - Remove_header (#4044) (b1c1431)
SDK - Compiler - Validating Argo validator (#3874) (bbc9ff5)
[AWS SageMaker] Integ test to check CloudWatch logs print feature (#4056) (799db47)
A brief release announcement/note for KFP 1.0 (#4059) (48f3075)
[AWS SageMaker] De-hardcode output paths in AWS components (#4119) (f0f8e5d)
SDK - Compiler - Removed the deprecated dsl-compile --package command (#4055) (229eff2)
[AWS SageMaker] GroundTruth Pre/Post Lambda function region additions (#3932) (131be23)
[AWS SageMaker] Processing job sample (#4009) (7619a01)
[AWS SageMaker] Add unit tests for cloudwatch logs (#4051) (e0e4b98)
Fix typo in error message (#3369) (50fd45f)
test(sdk) Restored the ParallelFor compiler test data (4103) (d24eb78)
Components - XGBoost - Added the recursive training pipeline (4067) (2268ddd)
Components - Added more GCP BigQuery components (3914) (c52a73e)
SDK - Compiler - Added support for volume-based data passing (3371) (54a596a)
SDK - Components - Python - Switched the default base image to python 3.7 (4054) (ceb860c)
SDK - Tests - Fixed two test failures in hermetic systems (4057) (51b1c74)
SDK - Compiler - Fixed error message (#4053) (757d43c)
Validate resourcekey to avoid apiserver being panic for invalid inputs (#3999) (91f08c4)
[Backend] Bug fix: applying filter in listing versions (#4052) (6fdf03a)
Components - Calculate_regression_metrics (#4050) (cd0e487)
Components - pandas - Transform DataFrame - Fixed transforms that replace dataframe (#4042) (cb4c1b2)
Components - XGBoost - Train and Predict from Apache Parquet (#4035) (c7ef9b4)
Components - Converters from ApacheParquet to CSV and TSV (#4031) (c4340f6)
Components - XGBoost - Parsing CSV using pandas (#4039) (ea94251)
better native-keras based sample (#3900) (7f39f18)
Metadata Writer - Added timeouts (#4037) (0417f13)
Components - CatBoost - Fixed small bug in sample (#4032) (091316b)
[SDK] Add list_pipeline_versions to kfp.Client (#3340) (a802dbb)
[AWS SageMaker] Print SageMaker job logs in kfp UI (#3954) (b3d8e04)
Docs - Show inherited members in the kfp.dsl docs (#4025) (d472d53)
Metadata-Writer: Updates metadata writer to use mlmd 0.22.0 (#4027) (24423ff)
Reduce ttl of persisted final workflow to 1 day (#4005) (8553497)
Update travis CI status badge to use .com version instead of outdated .org version (#3968) (7340fc2)
[AWS SageMaker] Component 0.4.1 Release (#4011) (c29ee5d)
Add missing import statement (#4010) (5717bc0)
SDK - Components - Added authentication support for component loading functions (#3976) (49fab02)
Components - Pandas - Transform_DataFrame (#3985) (a613f0c)
[AWS SageMaker] Processing job component (#3944) (bea6365)
enable pagination when expanding experiment in both the home page and the archive page (#4008) (53d0244)
Fix #3980 - mockapi tsconfig should reference root dir correctly (#4006) (4d2e317)
Simplified the style config (#4002) (2e9e750)
Components - CatBoost (#3844) (c809fc8)
Metadata Writer - Stopped using artifact properties (#4004) (4f5a7f0)
Fix/log unavailable warning (#3848) (040615c)
update refreshing mechanism (#3981) (c04b9f0)
[AWS Sagemaker] aws-samples kmeans-hpo pipeline test (#3905) (bc6aed9)
Cache - Deployer should check whether the secret is installed (#3992) ([8f8ac52](https://github.com/kubeflow/pipelines/commit/8f8ac52c347f6ba

Open source →

1.1.0a12 Nov 2020pre-release

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here.

Install python SDK (python 3.6 above) by running:

python3 -m pip install kfp kfp-server-api --pre --upgrade

See the Change Log

Open source →

Additional notes

Features

API: exposing api for setting the default version of pipeline. Fixes #4049 (#4406) (5742991)
backend: allow configuring if default version should be updated when uploading new pipeline version. Fixes #4049 (#4476) (2317015)
backend: Metadata Writer - Record parameter argument values to MLMD (#4564) (ec65dfe)
compiled: set pod disruption budget for pipelines. Fixes #3877 (#4178) (c32ea23)
components: Added the component to convert from KerasModelHdf5 to TensorflowSavedModel (#4524) (4830e03)
components: Add yaml support to kfserving component (#4587) (2a1fe45)
components: Added Calculate classification metrics component (#4441) (9a00eb3)
components: Added converter components for Tensorflow Lite models (#4532) (1802db3)
components: Added converter components for TensorflowJS GraphModel (#4526) (3927a4f)
components: Added converter components for TensorflowJS LayersModel (#4527) (b1f46ac)
components: Added converters to OnnxModel from Tensorflow and Keras (#4550) (4ec84e6)
components: Added JSON query components (#4601) (b656fbf)
components: Added the Basics - Calculate data hash component (#4503) (68a367d)
components: Added the Google Cloud Optimizer components (#4621) (46eb6f2)
components: Added the Run notebook using papermill component. Fixes #497 (#4578) (329ed48)
components: AutoML Tables - Added the Create dataset from CSV component (#4558) (a316e1e)
components: AWS SageMaker - Changes for updating an existing endpoint (#4424) (7c349f3)
components: Google - AutoML - Added the Deploy model component (#4520) (5fe8ef4)
components: Google - AutoML - Added the Export model component (#4519) (d5ae723)
components: Google Cloud Optimizer - Added the "Suggest parameter sets from measurements" component (#4633) (f30aa09)
components: JSON - Added the "Build dict", "Build list" and "Combine lists" components (#4631) (616c48b)
deployment: Kustomize manifest for Azure deployments (#4567) (7a5c3c6)
frontend: Show full node names. Fixes #3989 (#4663) (85bdbc1)
samples: Added "Output a directory" sample (#4437) (89cdaa4)
sdk: Added optional argument to specify description for pipeline upload (#4577) (dd1e3d8)
sdk: Adds missing short options to the cli (#4604) (ce6cfa5)
sdk: adds support for ephemeral-storage in container-op (#4504) (0653e7c)
sdk: Components - Added Bool as a known type name (#4597) (d4fd6e1)
sdk: Components - Prevent passing unserializable objects to components. Fixes #4040 (#4496) (0332584)
sdk: optional context in kubenetes client (#4401) (aed37c7)
sdk: Preserve parameter arguments and input names (#4563) (e8fb58a)
sdk: Python components - Parse component input/output descriptions from the function docstring (#4512) (5613db0)
sdk: Support to print the cli command output in json format (#4566) (0795597)
sdk/cli: add extra header for cli (#4400) (c8b4599)
GCP marketplace - preserve install config in configmap and secret (#4471) (fbfc160)
backend: Make number of persistence worker goroutine configurable (#3904) (8a2d11c)
backend: sort by run metrics - step 2 (#4235) (d4d3616)
backend: sort by run metrics - step 3. Part of #3591 (#4251) (9c6738f)
backend: sorting on run metrics - step 1 (#4203) (a96e8fe)
cache: Explicitly specifying which attributes affect the cache key (#4076) (f35462f)
components: [AWS SageMaker] Minimize inputs for mnist classification pipeline (#4192) (6860681)
components: AWS SageMaker - Add functionality to stop SageMaker jobs on run termination (#4167) (900eeae)
components: AWS SageMaker - Add optional parameter to allow training component to accept parameters related to Debugger (#4283) (3ebd075)
components: AWS SageMaker - Support for assuming a role (#4212) (8014a44)
components: Keras - Added the Train_classifier component (#4274) (5f7afe3)
components: ml_metrics - Calculate_regression_metrics - Added the metrics and number_of_items outputs (#4239) (1d8abb7)
components: Simplified the kubeflow - dnntrainer component (#4415) (b769416)
components: web - Download (#4068) (240543e)
add grafana as part of the customized kfp deployment (#4404) (f39d879)
deployment: Add env/platform-agnostic-pns standalone deployment option (#4278) (dac5a94)
deployment: GCP managed storage - detailed instructions to set up workload identity bindings before deployment (#4232) (45a91f6)
deployment: KFP standalone should keep user data when application deleted (#4332) (f738cb3)
deployment: marketplace - allow specifying gcs bucket directly (#4112) (d4a8329)
frontend: support tensorboard viewer and other visualize Results using volume mount path. Part of #4208 (#4236) (7f77e8b)
manifest: add support for Standalone KFP on AWS. Fixes #4337 (#4350) (5ec7ec8)
sdk: add aws region field in use_aws_secret in kfp sdk (#4363) (390e80e)
sdk: CLI - Allow upload_version by the pipeline's name. Fixes #3901 (#4087) (aae1eba)
sdk: DSL - Added support for volatile components (#4104) (d707b93)
sdk: Expose upload pipeline version (#4146) (45c796e)
sdk: SDK - Components - Added annotations to InputSpec and OutputSpec (#4154) (9b15800)
sdk: SDK - Components - Added annotations to TaskSpec (#4153) (afd8161)
sdk: SDK - Deprecation warning when using ContainerOp (#4166) (2f94827)
sdk: SDK - Enable placeholders in task display names. Fixes #4163 (#4164) (db0af86)
sdk: support HTTP/S PROXY for SDK client (#4215) (ac66626)
sdk/client: add cookie configuration to kfp client (#2679) (d91ac0f)
add kpf pipeline list-versions command (#3928) (c9f621a)
add parallelism for dsl.ParallelFor. Fixes #4089 (#4149) (c6ac83f)
allow to use pipeline version (#3930) (40f06bf)

Bug Fixes

backend: add MaxCallRecvMsgSize(math.MaxInt32) to proxy server (#4402) (cd9c9ff)
backend: Add a permission to create events to argo-role (#4449) (d40d66a)
backend: Cache Deployer - Fixed grep call (#4568) (24217ff)
backend: Caching - Fixed deployer failure on Kubernetes v1.16+. Fixes #4627 (#4632) (dde7f9a)
backend: Caching - Only send cache-enabled pods to the caching webhook (#4429) (6b54eec)
backend: Caching - Reduced the cache webhook timeout (#4428) (ada18bc)
backend: fix typo in reference key type (#4376) (32c9c2a)
backend: Metadata Writer - Fixed setting execution custom properties (#4670) (fc6b5d6)
backend: persistence agent - workflow not found error should be a permanent error (#4486) (29a6aaa)
backend: prevent seg fault if workflow manifest is deleted. Fixes #4389 (#4439) (ec733c9)
backend: skip reporting native Argo workflows which do not have Run ID label. Fixes #3584 (#4438) (1f2d417)
backend: updated the argo version too 2.7.7. Fixes #4392 (#4498) (d7793af)
cache: Cache deployer - Using the same kubectl version as the server (#4525) (58584e1)
cache: Cache executions with no outputs. Fixes #3507 (#3808) (40353bf)
components: add istio inject and ready status to kfserving component (#4362) (f3a02d1)
components: fix ml_engine/train interface (#4593) (d115d46)
components: Fixed the Download from GCS components for blob URIs (#4557) (0ffcbd7)
components: Fixed the GCP - ML Engine - Batch predict component. Fixes #4430 (#4432) (2beddd0)
components: GCP - AutoML - Fixed input type in Batch predict (#4499) (41ce68e)
components: Move output path as arguments for ibm components (#4647) (121bfdf)
components: use default python version for launching dataflow jobs. Fixes #2601 (#4583) (a013edc)
doc: Use Managed NodeGroup for EKS Cluster (#4609) (1c2a4e8)
frontend: do not render Retry nodes in UI DAG as they are virtual nodes. Fixes #4425 #2705 (#4474) (ca29b5c)
frontend: fix parsing large workflow graph. Fixes #4179 (#4180) (02b0899)
proxy: fixes GCP inverse proxy url priority. Fixes #4284 (#4702) (9d6ab4c)
samples: Keras - Fixed the keras sample (#4523) (9287287)
samples: Switched the AutoML sample from BQ to GCS datasets (#4501) (3fd72e4)
samples: Updated and fixed the secrets sample (#4590) (c8ce4fa)
sdk: added clearer error for case when user uses watch and argo isn't installed (#4511) (37362a3)
sdk: Allow non-pythonic names for graph components' task's outputs. Fixes #4514. (#4515) (0b31879)
sdk: Components - Fixed handling of typing.NamedTuple in Python 3.9 (#4614) (8699a05)
sdk: Components - Fixed ModelBase handling of open generic types in Python 3.9 (#4613) (be89972)
sdk: DSL - Enabled arbitrary ContainerOp names (#4554) (1aa8068)
sdk: Fix opsgroups dependency resolution (#4370) (22b7b99)
sdk: Fixed UI metadata and metrics (#4672) (80e1d70)
sdk: Fixes #4703: conflict between .pylintrc and .yapf (#4706) (0179584)
sdk: fixes dsl.ContainerOp deprecation warning not shown (#4658) (c52a81c)
sdk: use a meaningful link text in ipython (#4570) (a840ce7)
Bitbucket repo no longer available (#4395) (fe21cd2)
backend: allow empty userid header prefix. Fixes #4091 (4098) (042ff09)
backend: Backend - Cache - Fixed reinstallation. Fixes #4299 (#4320) (fe77c19)
backend: fix the google-api-core to 1.16.0 for backend visualization server. (#4158) (cf29c61)
backend: fixes useless error message when visualization-server is not accessible. Fixes #4157 (#4201) (d3d4dcb)
backend: improve forward / backward compatibility on db status table (#4351) (384afac)
backend: improve forward / backward compatibility on experiment table (#4349) (5141201)
backend: logs error when failing to init mysql. Fixes #4334 (#4335) (bb21597)
backend: Metadata Writer - Fixed regression with artifact type retrieval. Fixes #3971 (#4231) (1990588)
backend: reduce confusing ReadArtifact errors for metrics in api server. Fixes #3699 (#4338) (01a7998)
cache: adds certificate approver permission to kubeflow-pipelines-cache-deployer-role. Fixes #4138 (#4246) (#4383) (265c9b7)
cache: cache-deployer should check both secret and config (#4186) (e4f4250)
cache: Fix cache deployer not regenerating secrets when secret not present (#4171) (75336f7)
component: Add new apply action to kfserving component (#4108) (f21e0fe)
component: Update default TensorFlow version for Watson ML example (4109) (de2e0f2)
components: Add explicit image tag and fix variable references. (#4298) (fed37f7)
components: AWS SageMaker - Fix MinIO PID early exit (#4190) (c6754e3)
components: remove needless arguments from AWS EMR scripts (#4252) (262b288)
components: Update watson ml sdk with the Aug 2020 release (#4331) (25ee59d)
deployment: fix database name prefix description (#4198) (c615e37)
deployment: gcp managed storage - add service account to minio and cloudsql proxy deployments for workload identity (#4188) (c5c9739)
sample: Fix syntax error in openvino sample component (#4181) (70ac32c)
samples: fixes name error and unused parameter in samples for loop_parallelism (#4187) (57bb7ac)
sdk: Compiler - Fixed input artifact name sanitization when using raw string arguments. Fixes #4110 (#4120) (48889a9)
sdk: Compiler - Fixed the input argument mapping when using dsl.graph_component. Fixes #3915 (4082) (1da1cf4)
sdk: Compiler - Fixed the input argument mapping when using dsl.graph_component. Fixes #3915 (4082) (6960366)
sdk: Fix get_experiment when there are no experiments. Fixes #4176 (#4177) (749538d)
sdk: fixes the default value for _is_refresh_token. Fixes #4189 (#4195) (d3ef0dc)
sdk: SDK - Avoiding deprecated ContainerOp methods (#4134) (d0b799e)
sdk: SDK - Components - Fixed the error message format string (#4130) (79f6a40)
sdk: TypeError in Client.get_pipeline_id() when pipeline does not exist (#4319) (d7e5894)
sdk/client: reserve the host protocal (http or https) so that http host can work. Fixes #4277 (#4285) (cc78bd1)
sdk/component: ml_engine deploy version is incorrect type. Fixes #4291 (#4292) (2394bb2)
all big enough ui proxy requests fail with error proxying with partial data (#4266) (c7300c3)
increase TFX version from 0.20.2 to 0.22.0. Fixes #4084, fixes #4114 (#4133) (ce51c59)

Other Pull Requests

SDK - Restored module decoupling (#4625) (ccc763f)
compiler for IR (#4529) (5020fd1)
Update IR ExecutorOutput message to use ArtifactList for artifacts field. (#4619) (72ce138)
sample azure components (#4509) (210eb8a)
update kfp installation instructions for a number of gcp components (#4594) (7e4d9f4)
Samples - Added the continue_training_from_prod sample (#4561) (2471f26)
Update ImporterSpec artifact_uri type (#4548) (629a980)
Package pipeline_spec proto into SDK (#4546) (097b186)
Pipeline IR (#4371) (d8e8260)
[AWS SageMaker] Fix small bugs (#4161) (05398cf)
Manifests - Added permissions for certificate approval (#4385) (d9ee639)
Fix (readme): fixed a typo. (#4410) (b49f5c9)
update owners file (#4316) (d135693)
Updated components/aws/sagemaker/README.md (#3983) (e3c33a6)
Update README.md (#4260) (ee66814)
Components - ml_metrics - Added the Aggregate_regression_metrics component (#4240) (d4d692e)
Components - XGBoost - Added the Train_regression_and_calculate_metrics component (#4243) (8d373e7)
Components - XGBoost - Added Train_and_cross-validate_regression component (#4245) (acb50e7)
Components - XGBoost - Added the Cross_validation_for_regression component (#4244) (34cb59d)
SDK - Components - Temporary removed Kubernetes options from ComponentSpec (#4230) (39975e3)
Add myself as approver/reviewer (#2254) (193d143)
SDK - Components - Fixed examples in docstrings (#4074) (aeb0401)
Components - tables - Remove_header (#4044) (b1c1431)
SDK - Compiler - Validating Argo validator (#3874) (bbc9ff5)
[AWS SageMaker] Integ test to check CloudWatch logs print feature (#4056) (799db47)
A brief release announcement/note for KFP 1.0 (#4059) (48f3075)
[AWS SageMaker] De-hardcode output paths in AWS components (#4119) (f0f8e5d)
SDK - Compiler - Removed the deprecated dsl-compile --package command (#4055) (229eff2)
[AWS SageMaker] GroundTruth Pre/Post Lambda function region additions (#3932) (131be23)
[AWS SageMaker] Processing job sample (#4009) (7619a01)
[AWS SageMaker] Add unit tests for cloudwatch logs (#4051) (e0e4b98)
Fix typo in error message (#3369) (50fd45f)
test(sdk) Restored the ParallelFor compiler test data (4103) (d24eb78)
Components - XGBoost - Added the recursive training pipeline (4067) (2268ddd)
Components - Added more GCP BigQuery components (3914) (c52a73e)
SDK - Compiler - Added support for volume-based data passing (3371) (54a596a)
SDK - Components - Python - Switched the default base image to python 3.7 (4054) (ceb860c)
SDK - Tests - Fixed two test failures in hermetic systems (4057) (51b1c74)
SDK - Compiler - Fixed error message (#4053) (757d43c)
Validate resourcekey to avoid apiserver being panic for invalid inputs (#3999) (91f08c4)
[Backend] Bug fix: applying filter in listing versions (#4052) (6fdf03a)
Components - Calculate_regression_metrics (#4050) (cd0e487)
Components - pandas - Transform DataFrame - Fixed transforms that replace dataframe (#4042) (cb4c1b2)
Components - XGBoost - Train and Predict from Apache Parquet (#4035) (c7ef9b4)
Components - Converters from ApacheParquet to CSV and TSV (#4031) (c4340f6)
Components - XGBoost - Parsing CSV using pandas (#4039) (ea94251)
better native-keras based sample (#3900) (7f39f18)
Metadata Writer - Added timeouts (#4037) (0417f13)
Components - CatBoost - Fixed small bug in sample (#4032) (091316b)
[SDK] Add list_pipeline_versions to kfp.Client (#3340) (a802dbb)
[AWS SageMaker] Print SageMaker job logs in kfp UI (#3954) (b3d8e04)
Docs - Show inherited members in the kfp.dsl docs (#4025) (d472d53)
Metadata-Writer: Updates metadata writer to use mlmd 0.22.0 (#4027) (24423ff)
Reduce ttl of persisted final workflow to 1 day (#4005) (8553497)
Update travis CI status badge to use .com version instead of outdated .org version (#3968) (7340fc2)
[AWS SageMaker] Component 0.4.1 Release (#4011) (c29ee5d)
Add missing import statement (#4010) (5717bc0)
SDK - Components - Added authentication support for component loading functions (#3976) (49fab02)
Components - Pandas - Transform_DataFrame (#3985) (a613f0c)
[AWS SageMaker] Processing job component (#3944) (bea6365)
enable pagination when expanding experiment in both the home page and the archive page (#4008) (53d0244)
Fix #3980 - mockapi tsconfig should reference root dir correctly (#4006) (4d2e317)
Simplified the style config (#4002) (2e9e750)
Components - CatBoost (#3844) (c809fc8)
Metadata Writer - Stopped using artifact properties (#4004) (4f5a7f0)
Fix/log unavailable warning (#3848) (040615c)
update refreshing mechanism (#3981) (c04b9f0)
[AWS Sagemaker] aws-samples kmeans-hpo pipeline test (#3905) (bc6aed9)
Cache - Deployer should check whether the secret is installed (#3992) (8f8ac52)
[Manifest] Apply kustomize best practices to standalone manifest (#3978) (c0124cb)
[Doc] Cherry pick process for release branch (#3953) (7d6f2bd)
[Backend][Mutli-user] Allow shared read in the special multi-user mode. (#3858) (fcd2559)
[AWS SageMaker] Add integration test for sample pipeline train (#3876) (35019eb)
moved mnis examples (#3979) (370da26)
Update Dataflow component _client.py to use projects.locations.templates API endpoint (#3851) (79b9d40)
SDK - Compiler - Made compiler compatible with @wraps (#3956) (374b3b0)
[AWS SageMaker] Update GroundTruth integration test timeout (#3973) (d6920ca)
[Backend] Filter run on status (#3959) (2fa6e2b)
[Format] Format yaml files that will be automated (#3958) (3cf36cd)
Fix #3906 - mount_pvc transform should ignore non-ContainerOps (#3912) (4812d35)
[Python Client] Clean up generated python client template to facilitate version bump (#3937) (c6ac5e0)
fix length (#3934) (7fbb7ca)
SDK - Compiler - Using properly serialized pipeline parameter defaults (#3832) (40372e5)
Only pending or running workflows are considered not-final (#3940) (5f9e56a)
Add Bobgy to OWNERS (#3926) (57f415d)
Bump websocket-extensions from 0.1.3 to 0.1.4 in /frontend (#3929) (710c664)
[Release] Fix $(cat VERSION) not in bash script (#3924) (b0466cb)
[Release] Fix missing $TAG_NAME in .cloudbuild.yaml (#3921) (62959d5)
[Release] Various fixes to release script (#3918) (b12023e)
create cluster using 'cloud-platform' scope (#3892) (0324b2c)
SDK - Components - Added ComponentStore search (#3884) (1403b9b)
[Release] Regenerate api clients on release (#3910) (04e23d2)
Fix build failure because wcwidth LICENSE url changed (#3909) (8ba895d)
Docs - Moved the kfp_server_api documentation to a separate page (#3899) (180f0c5)
[API] Add license header to python api client files (#3897) (4f34be6)
[Release] Automate release script for all the changes (#3777) (c517fc5)
Regenerate clients after proto description fix (#3895) (7dfb34e)
Docs - Added docs for kfp_server_api.models (#3898) (c67bc7b)
[Python Client] Use openapi generator instead of swagger (#3872) (36eba13)
fix typo on sort order: 'des' --> 'desc' (#3893) (5d302b6)
[Samples] Change the data url to point to ml-pipeline instead of ml-pipeline-playground (#3890) (88ee54f)
Move the minio artifact download under try block (#3889) (a7be049)
Adds job_id parameter. (#3850) (58f1d13)
Fix the missing description in the CreatePipelineAPI (#3754) (b63dd3f)
SDK - Client - Use temporary directory context for pipeline package (#3878) (cd8a913)
Components - Added the pipeline component repository marker file (#3883) (4ce78dd)
SDK - Normalize line endings when calculating component hash (#3882) (da5f86e)
SDK - Components - Stabilize JSON serialization by sorting keys (#3879) (f773b9c)
[Backend] Fix user identity extraction logic in getUserIdentity() (#3842) (b23e1b4)
More documentation on backend API methods (#3758) (2864925)
Generate clickable artifact url for s3 URI (#3531) (3cae116)
SDK - Tests - Fixed the test_func_to_container_op_with_imported_func2 test case (#3837) (3010e85)
[AWS SageMaker] Add working FSx setup and test (#3831) (37a6363)
[SDK] Add first party component label (#3861) (1e2b9d4)
SDK - Python Components - Stop generating output saving code if no outputs (#3836) (da4acbb)
re-enable upgrade test since 0.5.2 is cut (#3696) (da1bc69)
fixing case when status is None (#3865) (58ff65f)
Modify docstrings to replace 'InitContainer' to 'UserContainer' (#3863) (699ce93)
Testing - Fixed SDK Travis tests (#3838) ([0e59b68](https://github.com/kubeflow/pipelines/commit/0e59b68775a7cb75c

Open source →

1.0.020 Jul 2020

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.5 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

Open source →

Additional notes

Features

deployment: marketplace - allow specifying gcs bucket directly (#4112) (92a8cfe)
sdk: DSL - Added support for volatile components (#4104) (07ebc27)
sdk: SDK - Components - Added annotations to InputSpec and OutputSpec (#4154) (98fccca)
sdk: SDK - Components - Added annotations to TaskSpec (#4153) (c439ec3)
sdk: SDK - Deprecation warning when using ContainerOp (#4166) (ebd62ec)
sdk/client: add cookie configuration to kfp client (#2679) (9206b5a)
deployment: GCP managed storage - detailed instructions to set up workload identity bindings before deployment (#4232) (11c1cd6)

Bug Fixes

backend: allow empty userid header prefix. Fixes #4091 (#4098) (9e00c79)
backend: fix the google-api-core to 1.16.0 for backend visualization server. (#4158) (8da057d)
backend: Metadata Writer - Fixed regression with artifact type retrieval. Fixes #3971 (#4231) (3ffd55e)
cache: cache-deployer should check both secret and config (#4186) (7b3654b)
cache: Fix cache deployer not regenerating secrets when secret not present (#4171) (d727c91)
deployment: fix database name prefix description (#4198) (2ea1a1f)
deployment: gcp managed storage - add service account to minio and cloudsql proxy deployments for workload identity (#4188) (bd7f3c9)
sdk: fixes the default value for _is_refresh_token. Fixes #4189 (#4195) (be47d6d)
increase TFX version from 0.20.2 to 0.22.0. Fixes #4084, fixes #4114 (#4133) (5d8a59c)

Other Pull Requests

SDK - Components - Temporary removed Kubernetes options from ComponentSpec (#4230) (9c16eff)
SDK - Compiler - Removed the deprecated dsl-compile --package command (#4055) (6bcb3ef)
SDK - Components - Python - Switched the default base image to python 3.7 (#4054) (090fffc)
Validate resourcekey to avoid apiserver being panic for invalid inputs (#3999) (6ced6ea)
[Backend] Bug fix: applying filter in listing versions (#4052) (3243f73)
better native-keras based sample (#3900) (842e8ce)
Metadata Writer - Added timeouts (#4037) (e808795)
Fix/log unavailable warning (#3848) (3c6f6db)
Metadata-Writer: Updates metadata writer to use mlmd 0.22.0 (#4027) (88d6da7)
Metadata Writer - Stopped using artifact properties (#4004) (4833d1a)
update refreshing mechanism (#3981) (efa5f1e)
enable pagination when expanding experiment in both the home page and the archive page (#4012) (8d34f6d)
Updated version to 1.0.0-rc.2 (0796340)
Cache - Deployer should check whether the secret is installed (#3992) (c60a046)
[Manifest] Apply kustomize best practices to standalone manifest (#3978) (7339da0)
Update Dataflow component _client.py to use projects.locations.templates API endpoint (#3851) (0e7c05c)
SDK - Compiler - Made compiler compatible with @wraps (#3956) (8d19aa8)
[Backend] Filter run on status (#3959) (3d41d5d)
Fix #3906 - mount_pvc transform should ignore non-ContainerOps (#3912) (e7beae4)
SDK - Compiler - Using properly serialized pipeline parameter defaults (#3832) (692b244)
Bump websocket-extensions from 0.1.3 to 0.1.4 in /frontend (#3929) (c69f097)
Only pending or running workflows are considered not-final (#3940) (#3957) (b021b7c)
Remove version from generated python client header comment (#3948) (19ff1b2)
Automated cherry pick of #3926: Add Bobgy to OWNERS (#3949) (f7623db)
fix length (#3934) (#3946) (524e162)
[Release] Fix $(cat VERSION) not in bash script (#3924) (#3925) (7d1de12)
[Release] Fix missing $TAG_NAME in .cloudbuild.yaml (#3921) (f9bcbbb)
[Release] Various fixes to release script (#3918) (105d43d)
Updated version to 1.0.0-rc.1 (2384d8f)
Update VERSION to 1.0.0-rc.1 (51c349b)
[Release] Regenerate api clients on release (#3910) (04e23d2)
Fix build failure because wcwidth LICENSE url changed (#3909) (8ba895d)
Docs - Moved the kfp_server_api documentation to a separate page (#3899) (180f0c5)
[API] Add license header to python api client files (#3897) (4f34be6)
[Release] Automate release script for all the changes (#3777) (c517fc5)
Regenerate clients after proto description fix (#3895) (7dfb34e)
Docs - Added docs for kfp_server_api.models (#3898) (c67bc7b)
[Python Client] Use openapi generator instead of swagger (#3872) (36eba13)
fix typo on sort order: 'des' --> 'desc' (#3893) (5d302b6)
[Samples] Change the data url to point to ml-pipeline instead of ml-pipeline-playground (#3890) (88ee54f)
Move the minio artifact download under try block (#3889) (a7be049)
Adds job_id parameter. (#3850) (58f1d13)
Fix the missing description in the CreatePipelineAPI (#3754) (b63dd3f)
SDK - Client - Use temporary directory context for pipeline package (#3878) (cd8a913)
Components - Added the pipeline component repository marker file (#3883) (4ce78dd)
SDK - Normalize line endings when calculating component hash (#3882) (da5f86e)
SDK - Components - Stabilize JSON serialization by sorting keys (#3879) (f773b9c)
[Backend] Fix user identity extraction logic in getUserIdentity() (#3842) (b23e1b4)
More documentation on backend API methods (#3758) (2864925)
Generate clickable artifact url for s3 URI (#3531) (3cae116)
SDK - Tests - Fixed the test_func_to_container_op_with_imported_func2 test case (#3837) (3010e85)
[AWS SageMaker] Add working FSx setup and test (#3831) (37a6363)
[SDK] Add first party component label (#3861) (1e2b9d4)
SDK - Python Components - Stop generating output saving code if no outputs (#3836) (da4acbb)
re-enable upgrade test since 0.5.2 is cut (#3696) (da1bc69)
fixing case when status is None (#3865) (58ff65f)
Modify docstrings to replace 'InitContainer' to 'UserContainer' (#3863) (699ce93)
Testing - Fixed SDK Travis tests (#3838) (0e59b68)
Components - Apache Parquet converters (#3834) (d737c44)
Kfserving component service account option (#3852) (8d738ea)
Cleanup - Removed unused code file (#3864) (f7acb71)
Testing - Using Argo validation in Travis tests (#3123) (7d94de1)
[AWS SageMaker] Add more unit tests (#3783) (b503050)
Testing - Handling spaces in paths in the component file loading test (#3856) (695573d)
revert the fix in proxy-agent (#3845) (cc46bef)
Components - XGBoost (#3843) (2d26a4c)
Datasets - Added Chicago Taxi Trips dataset (#3775) (5e3d9aa)
Add labels to plots (#3811) (76f7476)
A more accurate grpc error code for duplicate pipeline/pipeline version/experiment names (#3846) (ef73aab)
Added README for Amazon SageMaker Components for Kubeflow Pipelines (#3824) (3fe9b7e)
Components - Moved TFX components to deprecated directory (#3854) (5b50de0)
[AWS SageMaker] Integration Test for AWS SageMaker GroundTruth Component (#3830) (fb54953)
Testing - Pin numpy version to fix TFX installation instability in Travis tests (#3833) (9f1c596)
[UI] Also cloning recurring run schedule, fixes #3761 (#3840) (508f31a)
Client - Allow specifying pipeline description when uploading (#3828) (e52481a)
Adding HPO unit test (#3791) (bbe598d)
Docs - Added kfp.dsl placeholders to docs (#3813) (43ce639)
[UI] Fix confusion matrix wrong axes (#3817) (0199de0)
AWS SageMaker : Use IAM Roles for Service Account (#3719) (d18ad7a)
Add the 'kfp experiment' commands (#3705) (291f5b3)
[AWS SageMaker] Integration tests automation (#3768) (f2a860b)
Client - Added documentation for the generated members (#3787) (4a961ce)
Changing the default volume size to 30 (#3792) (6e2a55c)
Infer artifact store endpoint in metadata writer (#3530) (a764405)
[ScheduledWorkflow] Fix events permission missing (#3785) (53d35dd)
Allow PipelineParams in dict keys too. (#3565) (ec9445a)
[Backend] Make user identity header configurable (#3772) (9f8803e)
SDK - Tests - Use relative imports (#3784) (92394f6)
[Manifest] Use kustomize native image transformer to override image (#3776) (39805ac)
fix(testing) - Fix "1.14.10-gke.27" is unsupported (#3781) (205ca79)
SDK - Moved the tests closer to the code (#3774) (1dcea49)
SDK - Components - Removed the deprecated _python_op.get_default_base_image and set_default_base_image functions (#3773) (55d41df)
Add more approvers in AWS sagemaker components (#3740) (af4e8ef)
AWS Sagemaker Components - enhance integration test coverage (#3720) (bff8392)
SDK - Moved some data from the component_ref annotation to the component_spec annotation (#3751) (92a0d11)
manual merge as the change it self is correct (3d2c6ea)
Add probes to metadata grpc service (#3765) (61960ef)
[Servers] Add liveness and readiness probes (#3757) (f0c8eff)
Components - Tensorboard visualization (#3760) (0f40372)
[AWS SageMaker] Unit tests for Training component (#3722) (ddd1969)
SDK - Components - Improved stability of the input and output renaming (#3738) (d418f57)
[UI Server] Refactor for configurable auth header (#3753) (5435e87)
Metadata Writer - Preserve all Argo artifact information (#3725) (81dd6a2)
[UI] Show tooltip on long version names (#3750) (bb0c4d9)
[UI] Fix artifact preview with outdated content (#3749) (06cd029)
SDK - Made outputs with original names available in ContainerOp.outputs (#3734) (8ba366b)
SDK - Components - Calculate component hash digest (#3726) (fe30d54)
[UI] Wrap parameter/urls on overflow (#3747) (bd4be88)
Show version tag in UI (#3743) (e8356bc)
Fix bug in #3707 - href should show full artifact content instead of preview (#3745) (e57a619)
[Backend] Allow capital letters and underscore in metric names (#3741) (5740535)
Upgraded Argo to v2.7.5 (#3537) (e83bd7d)
[AWS SageMaker] Specify component input types (#3683) (bd8c1dd)
SDK - Prioritize lib2to3 when stripping type annotations (#3724) (b9aa106)
Travis - Use latest pip version (#3732) (84cd01a)
SDK - Annotate pods with component_ref (#3727) (2279bde)
Travis - Made flake8 test optional (#3739) (21e8ef1)
[frontend] Show artifact preview in UI (#2172) (#3707) (30345d1)
[UI] Make visualization tab easier to understand (#3717) (b1c9976)
When patching the {{}} placeholder in parameter, check for possible nil pointer (#3714) (d2c784a)
remove an accidentally committed debugging log (#3716) (fe01908)
Fixed small syntax error in a sample notebook (#3721) (d4cd70d)
[UI] Cleanup, remove types from urls in artifact/execution details page (#3715) (cb41f7a)
[UI] Hide empty resource op manifest tab in run details page (#3713) (c5ceaf5)
[UI] Improve TFX artifact visualization speed (#3712) (eab5d31)
0.5.1 changelog (#3706) (54c490c)
Integration tests for AWS SageMaker Components (#3654) (6beab22)
[UI] textbox to select KSA when creating runs/jobs (#3651) (e5bd2df)

Open source →

0.3.021 Mar 2020

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here

Install python SDK (python 3.5 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

Open source →

Additional notes

Full Changelog

Merged pull requests:

Reduce steps for release cloud build yaml #3331 (IronPan)
Bump version to 0.3.0 #3329 (neuromage)
Release be497983cda7a1d17f3883c67e39a969cf0868a9 #3327 (neuromage)
SDK - Tests - Moved imports to the start of the file #3325 (Ark-kun)
SDK - Moved the @python_component decorator test to dsl tests #3324 (Ark-kun)
SDK - Moved python op pipeline compilation test to bridge tests #3323 (Ark-kun)
[UI] Delete manifest.json #3320 (Bobgy)
SDK - Support kubernetes client v11 #3319 (Ark-kun)
SDK - Added support for maxCacheStaleness #3318 (Ark-kun)
Update Watson ML example to take output param path #3316 (Tomcli)
[Test] expose integration test flag #3307 (numerology)
Remove unused parameter #3306 (claytonlemons)
[Backend] Always disable istio sidecar injection #3305 (Bobgy)
[UI] Show step pod yaml and events in RunDetails page #3304 (Bobgy)
Enable archiving logs #3301 (Ark-kun)
[Test] Switch Travis dependency to TF2 #3299 (numerology)
[Test] Use a flag to separate integration test from sample test #3298 (numerology)
[UI] Auto refresh/redirect experiment list/detail page when user selects a different namespace #3297 (Bobgy)
[SDK] Improve host name handling #3287 (numerology)
Add Deletion button to archived runs #3285 (jingzhang36)
[Testing] Recycle upgrade- clusters and reduce time lapse #3282 (Bobgy)
[Frontend] Fix reported vulnerabilities #3281 (Bobgy)
Update IBM Watson ML sample to use V4 api #3274 (kevinyu98)
Add pipeline version to job/run integration test so that job/run is c… #3270 (jingzhang36)
SDK - Moved _dsl_bridge to dsl #3267 (Ark-kun)
[Backend]Cache - Cache logic with db interaction #3266 (rui5i)
Remove backend unit tests from Travis config #3264 (jingzhang36)
SDK - Components - Fixed handling collection return values #3263 (Ark-kun)
add presto pipeline component #3261 (mertkavi)
Update ai_platform.ipynb #3260 (gogasca)
SDK - Components - Make component structures hashable #3258 (Ark-kun)
SDK - Tests - Testing command-line resolving explicitly #3257 (Ark-kun)
Bump TFX to 0.21.2 #3255 (numerology)
Added 'workgroup' as optional argument to athena component #3254 (LeonardAukea)
[UI] Pass namespace to experiment api calls #3252 (Bobgy)
Fix ill-formed Cloud Build config #3245 (numerology)
[Backend][Multi-user] Add namespace to Experiment API implementation #3243 (chensun)
Clean up kataras dependency #3240 (jingzhang36)
[UI] Fix SideNav too tall when collapsed #3239 (Bobgy)
[Manifest] fix version for MLMD store #3237 (numerology)
Parametrize the image used for the Tensorboard viewer #3235 (rafaelbarreto87)
retry pipeline always fail (#3230) #3231 (xiaohanhuang)
[Frontend] Fix the TFMA/TFDV viz behavior when caching is used. #3228 (numerology)
[UI] Human readable workflow task display names #3225 (Bobgy)
[SDK] Use new released kfp-server-api package #3224 (Bobgy)
Bump TFX to 0.21.1 #3220 (numerology)
[Frontend] GKE metadata endpoint should fail when fetched response is not ok #3218 (Bobgy)
[Deployment] Add secure=false explicitly in manifests for better observability #3217 (Bobgy)
SDK/DSL: Enable the deletion of a resource via ResourceOp method #3213 (elikatsis)
add an optional name for dsl.Condition (kubeflow#3210) #3212 (xiaohanhuang)
Pin TFT to 0.21.0 #3209 (numerology)
[SDK] New fields for schedule pipeline #3201 (Bobgy)
Fix presubmit sample tests base image #3200 (rui5i)
[Backend][Multi-user] Add resource reference to Experiment API #3198 (chensun)
[UI] Better logging in UI server #3197 (Bobgy)
[UI] Get execution name from name field #3196 (Bobgy)
revert #3168 as it doesn't work for MKP, put myself to reviewer #3191 (rmgogogo)
Check Tensorboard Instance #3189 (jingzhang36)
[Tutorial] - Fixed lightweight component tutorial with bad metadata usage #3186 (luotigerlsx)
backend: Pin python dependencies #3161 (discordianfish)
Backend - Validate compiled samples #3120 (Ark-kun)
TFX Iris sample #3119 (numerology)
Fix config fieldname PipelineFolder to PipelinePath #3056 (eterna2)
SDK/CLI: Implement CLI delete pipeline command #2879 (sh-tatsuno)
[Testing] KFP standalone test infra for upgradability #1971 (Bobgy)

Open source →

0.1.331 Nov 2019

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here

Install python SDK (python 3.5 above) by running:

pip3 install kfp --upgrade

See the Change Log

Open source →

Additional notes

Full Changelog

Merged pull requests:

Update kustomization.yaml #2530 (IronPan)
Update setup.py #2528 (IronPan)
Update __init__.py #2527 (IronPan)
[Sample] Align preload TFX sample with TFX head #2526 (numerology)
SDK - Compiler - Fixed failures on Jinja placeholders #2522 (Ark-kun)
[Test] Fix TFX related Travis tests #2521 (numerology)
update location for logo #2520 (IronPan)
[MKP] Reduce the logo image size. #2519 (numerology)
Frontend - Added support for https artifact links #2517 (Ark-kun)
Pin tensorboard version to 1.13.2 #2513 (IronPan)
clean up viewer crd spec #2511 (IronPan)
remove unnecessary namespace in UI rolebinding spec #2510 (IronPan)
[UI] Fix metadata tabs loading state #2508 (Bobgy)
Regenerate api since https://github.com/kubeflow/pipelines/pull/2445 changed api proto #2506 (jingzhang36)
[Sample] Replace deprecated KubeflowRunner in TFX sample #2499 (numerology)
Samples - Renamed component build to container build #2496 (Ark-kun)
[Frontend] Enlarge choose pipeline dialog to show more description #2494 (Bobgy)
update inverse proxy custom permission setting and role setting #2493 (rmgogogo)
Fix pipeline description 255 characters length limit #2492 (Bobgy)
[MKP] Fix metadata DB configmap #2491 (numerology)
[Sample] Add the run ID place holder to TFX sample, fix metadb config in preload sample as well #2487 (numerology)
Fix inverse proxy matching regex #2486 (IronPan)
update base image to fix the GKE GC issue for marketplace deployment #2484 (IronPan)
better doc for MKP-KFP deployment #2481 (rmgogogo)
[Frontend] Fix log viewer cannot scroll horizontally + other minor issues #2480 (Bobgy)
[Sample] Update pre-load TFX::OSS sample #2476 (numerology)
SDK - Python components - Fixed bug when mixing file outputs with return value outputs #2473 (Ark-kun)
Update samples/core/ai_platform pipeline to follow data dependency #2472 (ucdmkt)
Add option to hide tensorboard artifact #2466 (hlu09)
Release notes for 0.1.32 #2465 (hongye-sun)
[Frontend] Update CONTRIBUTING.md with frontend code style info #2464 (Bobgy)
[Frontend] Check format in travis CI #2463 (Bobgy)
Format all source files under frontend/src using prettier #2462 (Bobgy)
[Frontend] UI shows specified task display name in PipelineDetail page #2459 (Bobgy)
clusterrole for pipeline-runner with seldondeployments #2458 (MingfeiPan)
Use string literals for reference resource relation and for reference resource type in frontend #2453 (jingzhang36)
SDK - Components - Added type to graph input references #2451 (Ark-kun)
Fix documentation for filter.proto #2447 (neuromage)
[Request for comments] Add config for yapf and pylintrc #2446 (numerology)
Runs and jobs can be created from pipeline version #2445 (jingzhang36)
Fix CustomTable.tsx layout problems #2444 (Bobgy)
Add --bind_all option for tensorboard #2441 (daikeshi)
[Test] Fix post-submit test #2439 (numerology)
SDK - Client - Makes the create_run output nicer #2438 (Ark-kun)
SDK - Setup - Added cloudpickle to requirements #2437 (Ark-kun)
Add owner files for marketplace deployment #2436 (IronPan)
Update backend OWNERS #2435 (IronPan)
Diagnose me dev env #2425 (SinaChavoshi)
[Doc] Update permission requirement in README #2422 (numerology)
Adding the core libraries for diagnose_me tool. #2417 (SinaChavoshi)
[Doc] Minor doc update #2394 (numerology)
update link so that user can easily jump to CAIP after deployment #2377 (rmgogogo)
[Frontend] Make links in pipeline description clickable #2376 (Bobgy)
[Doc] Fix a typo in MKP guide #2342 (numerology)
SDK/CLI: Implement kfp pipeline group #2340 (elikatsis)
SDK - Tests - Fixed most of the test warnings #2336 (Ark-kun)
SDK - Refactoring - Split the K8sHelper class #2333 (Ark-kun)
tech writer edits #2332 (jay-saldanha)
Tests - Use base image for frontend tests #190 (Ark-kun)

Open source →

0.1.3223 Oct 2019

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here

Install python SDK (python 3.5 above) by running:

pip3 install kfp --upgrade

See the Change Log

Open source →

Additional notes

Full Changelog

Merged pull requests:

Update sdk release version #2434 (hongye-sun)
Release e9b96de317989a9673ef88d88fb9dab9dac3005f #2433 (hongye-sun)
[Frontend] Configure tslint for better DX #2431 (Bobgy)
[Frontend] Upgrade typescript to 3.6 #2428 (Bobgy)
SDK - DSL - Make is_exit_handler unnecessary in ContainerOp #2411 (Ark-kun)
[Frontend] Prettier config to be consistent with existing code style #2409 (Bobgy)
tech writer edits #2403 (jay-saldanha)
[Test/Sample test] Fix model version in AI platform sample #2400 (numerology)
Update Watson ML default framework version #2398 (Tomcli)
Add Tomcli as kfs component reviewer #2396 (Tomcli)
License crawler for third party golang libraries #2393 (Bobgy)
quick custom spec fix #2390 (animeshsingh)
add test config comment #2389 (gaoning777)
Remove jingzhang36 and rmgogogo from frontend reviewer list #2388 (Bobgy)
enable the check for dataflow #2387 (gaoning777)
Update samples/core/tfx-oss to tfx==0.14.0 and kfp=0.1.31 #2385 (ucdmkt)
[Sample] Add back visualization in XGBoost sample #2384 (numerology)
move favicon path, root is override by inverse proxy #2382 (rmgogogo)
[Frontend] Script and documentation to start a frontend dev env that works with all API endpoints #2381 (Bobgy)
add animesh to the approvers of the kfserving #2380 (gaoning777)
SDK - Added version #2374 (Ark-kun)
tech writer edits #2373 (jay-saldanha)
[Samples] Add numerology as samples/OWNERS #2371 (numerology)
[Frontend] Fix cannot copy logs in LogViewer when scrolling #2370 (Bobgy)
KFServing move to v1alpha2 #2369 (animeshsingh)
Components - Updated the gcp dataproc create_cluster component image #2366 (Ark-kun)
[Doc] Fix some description of preload samples #2361 (numerology)
[Sample] Improve tfx oss sample #2360 (numerology)
add cloud-platform scope in the test to reclaim the ai platform sample models #2355 (gaoning777)
Fix potential issue of the ai platform sample when running it in the ai platform notebook #2349 (gaoning777)
Typo in Data passing in python components.ipynb #2347 (pingsutw)
[Test] Add unittest against TFX #2346 (numerology)
SDK - Python components - Fixed handling multiline decorators #2345 (Ark-kun)
[License] Fix third-party license #2344 (numerology)
Fix pipeline cannot run bug when using marketplace managed storage #2341 (Bobgy)
enlarge MKP cluster constraint #2339 (rmgogogo)
Add pipeline version api methods #2338 (jingzhang36)
tech writer edits #2331 (jay-saldanha)
Add sample test for multiple output #2328 (gaoning777)
add ai_platform test #2327 (gaoning777)
Tests - When testing SDK install it using pip #2325 (Ark-kun)
tech writer edits #2324 (jay-saldanha)
SDK - Compiler - Added the component spec annotations to the compiled workflow #2323 (Ark-kun)
[SDK/Compiler] Add _create_and_write_workflow method #2321 (numerology)
[Sample] Add new TFX::OSS sample #2319 (numerology)
SDK - Containers - Made python package installation more robust #2316 (Ark-kun)
Extend KFServing component with autoscaling and server mode #2315 (Tomcli)
SDK - Tests - Test creating component from the real AutoML pipeline #2314 (Ark-kun)
tech writer edits #2313 (jay-saldanha)
[MKP/doc] Update doc for changes of service account credential. #2309 (numerology)
[MKP] Remove service account credential from deployment page. #2308 (numerology)
SDK/DSL: ContainerOp.add_pvolume - Fix volume passed in add_volume #2306 (elikatsis)
[Frontend] Node detail view now can show workflow input/output artifacts #2305 (eterna2)
SDK - Compiler - Fixed deprecation warning when calling compile #2303 (Ark-kun)
tech writer edits #2301 (jay-saldanha)
[Component] Add VPC Interface Endpoint Support for SageMaker #2299 (RedbackThomson)
SDK - Compiler - Fix bugs in the data passing rewriter #2297 (deepio-oc)
Add CMLE deploy comment missing parameters #2296 (hongye-sun)
Samples - Simplified pipeline submission code in samples #2293 (Ark-kun)
SDK - Client - Added a way to set experiment name using environment variables #2292 (Ark-kun)
tech writer edits #2291 (jay-saldanha)
[MKP] Fix gcr paths in values.yaml #2289 (numerology)
fix for MKP #2288 (rmgogogo)
tech writer edits #2285 (jay-saldanha)
Disable cloudsql and update to v0.1.31 #2284 (rmgogogo)
tech writer edits #2282 (jay-saldanha)
Remove usage of deprecated ContainerOp methods in use_gcp_secret #2280 (andrewsmartin)
Samples - Switched the build_component sample to the new container API #2279 (Ark-kun)
Components - Added the "AutoML Export data to GCS" component #2278 (Ark-kun)
SDK - Components - Deprecated build_python_component #2277 (Ark-kun)
SDK - Containers - Deprecated build_docker_image #2276 (Ark-kun)
Refactor resource op sample for sample test coverage #2274 (numerology)
SDK - Components - Creating graph components from python pipeline function #2273 (Ark-kun)
SDK - Components - Verify the object type when serializing primitive arguments #2272 (Ark-kun)
SDK - Compiler - Make it possible to create more portable pipelines #2271 (Ark-kun)
SDK - Components - Reorganized TaskSpec execution options #2270 (Ark-kun)
SDK - Components - Fixed small bugs in graph component resolving #2269 (Ark-kun)
SDK - Notebooks - Deprecated the docker magic #2266 (Ark-kun)
SDK - Components - component_ref.name should only be set when component was loaded by name #2265 (Ark-kun)
SDK - Components - Added the ComponentSpec.save method #2264 (Ark-kun)
Components - Removed trailing whitespace from AutoML components code #2263 (Ark-kun)
SDK - Components - Restored attribute order when generating component.yaml files #2262 (Ark-kun)
SDK - Lightweight - Convert the names of file inputs and outputs #2260 (Ark-kun)
SDK - Compiler - Fixed small bug in data passing rewriter #2259 (Ark-kun)
Samples - Added the data passing tutorial #2258 (Ark-kun)
SDK - Components - Deprecate the get and set methods for default image in favor of plain variable #2257 (Ark-kun)
SDK - Containers - Getting namespace lazily #2256 (Ark-kun)
remove default namespace #2250 (SinaChavoshi)
SDK - Lightweight - Added package installation support to func_to_container_op #2245 (Ark-kun)
SDK: fix label check for ContainerOP entities #2243 (solovyevt)
Update doc for MKP release #2242 (rmgogogo)
Update changelog for release 0.1.31. (And also for 0.1.30 and 0.1.29, whose are not added before) #2232 (jingzhang36)
SDK - Compiler - Move Argo volume specifications to templates #2229 (Ark-kun)
Updated README Swagger CodeGen version #2228 (RedbackThomson)
SDK - Components - Fix - Stop serializing string values #2227 (Ark-kun)
third_party/metadata_envoy: Modify license file #2224 (dushyanthsc)
[SDK/Client] Improve the url format check for kfp.Client #2222 (numerology)
[Sample] update XGBoost sample #2220 (numerology)
[Component] Add Managed Spot Training Support for SageMaker #2219 (RedbackThomson)
SDK - Containers - Added support for container image cache #2216 (Ark-kun)
Add third party license + source code to argo and minio images to comply with their license #2201 (Bobgy)
SDK - Moved the _container_builder from kfp.compiler to kfp.containers #2192 (Ark-kun)
Added the backend Go module cache to .gitignote #2190 (Ark-kun)
Docs - Added the direct kfp module members to documentation #2183 (Ark-kun)
Components - Added AutoML Tables components and tests #2174 (Ark-kun)
GUI: should pop-up a correct info when delete more than one pipeline #2156 (QxiaoQ)
[SDK-compiler] Refactor Compiler to expose an API to write out yaml spec of pipeline. #2146 (numerology)
Add necessary data types to api and database to support pipeline version. #1873 (jingzhang36)
SDK - Hiding Argo's workflow.uid placeholder behind DSL #1683 (Ark-kun)
Allows uploading a pipeline in new run form #1643 (rileyjbauer)
SDK - Improve errors when ContainerOp.output is unavailable #1578 (Ark-kun)
Use Remote Build Execution for Bazel builds. #1031 (neuromage)

Open source →

0.1.2611 Aug 2019

To deploy Kubeflow Pipelines in an existing cluster, follow the instructions here.

To install the latest python SDK from PyPi, run:

pip3 install kfp --upgrade

See the Change Log

Open source →

Additional notes

Full Changelog

Merged pull requests:

update gcloud ml-engine to ai-platform #1863 (gaoning777)
Release 0517114dc2b365a4a6d95424af6157ead774eff3 #1859 (neuromage)
Reduce getPipeline calls in RunList #1852 (rileyjbauer)
Add back coveralls. #1849 (numerology)
Propagate pipeline name in pipeline spec #1842 (IronPan)
Create composite indexes [ResourceType, ReferenceUUID, ReferenceType] #1836 (IronPan)
Improve sql efficiency for getting the run #1835 (IronPan)
Adding a sample for serving component #1830 (SinaChavoshi)
Update for sample repo restructuring #1828 (zanedurante)
Fix run duration bug #1827 (rileyjbauer)
Reduce API usage by utilizing reference name in reference resource API #1824 (ajchili)
Update npm test to not use coverall #1819 (IronPan)
Add subprocess pip install example in lightweight component example notebook #1817 (Bobgy)
Build - Fix CloudBuild bug #1816 (Ark-kun)
Refactors toolbar buttons to use a map rather than an array #1812 (rileyjbauer)
Disable flaky tests temporarily #1809 (Bobgy)
Fix test loophole for loading samples during KFP startup #1807 (IronPan)
Container builder default gcr #1806 (gaoning777)
Fix the broken sample path in API #1805 (IronPan)
Garbage collect the completed workflow after persisted to database #1802 (IronPan)
Fix github security alert. #1798 (hongye-sun)
ContainerBuilder loading kube config #1795 (gaoning777)
Move TF installation to notebooks #1793 (numerology)
Move argo installation to dockerfile from bash script. #1792 (numerology)
fix sample reference link #1789 (gaoning777)
skip storing log to files #1788 (IronPan)
Remove yebrahim from approvers/reviewers #1787 (yebrahim)
update owner files in samples and test #1785 (gaoning777)
Fixed Dockerfile used for buildVisualizationServer in Cloud Build #1783 (ajchili)
Add retry button in Pipeline UI #1782 (IronPan)
add reference name to resource reference API proto #1781 (IronPan)
Update images, bug fixes, clean up code #1778 (carolynwang)
Container builder #1774 (gaoning777)
fix api server sort test #1769 (IronPan)
SDK - Containers - Returning image name with digest #1768 (Ark-kun)
Move imagepullsecrets sample to samples/core #1767 (numerology)
Not return error if run update doesn't change DB entry #1765 (IronPan)
remove copying the samples since we are not releasing the samples in the GCS #1764 (gaoning777)
Backend - Docker build should fail on sample compilation failures #1760 (Ark-kun)
Move samples to the correct location #1759 (gaoning777)
Change how Variables are Provided to Visualizations #1754 (ajchili)
Add preemtptible gpu sample #1749 (numerology)
Revert "Backend - Updated the version of the ml metadata package" #1747 (Ark-kun)
Revert "Backend - Starting the api-server container build from scratch" #1742 (Ark-kun)
Refactor aws samples to match new folder structure #1741 (carolynwang)
Components - Added the pymongo license URL #1740 (Ark-kun)
Add Visualization Server to Cloud Build yaml Files #1738 (ajchili)
Update Watson Machine Learning auth with IAM #1737 (Tomcli)
Delete KFP component before reinstalling again #1736 (IronPan)
Moving samples to match the new folder structure #1734 (SinaChavoshi)
fix cloudbuild failure #1733 (gaoning777)
Refactor sample tests configuration to reduce the efforts of adding samples. #1730 (numerology)
SDK - Lightweight - Fixed regression for components without outputs #1726 (Ark-kun)
Backend - Updated the version of the ml metadata package #1725 (Ark-kun)
Add API to rerun the pipeline #1720 (IronPan)
Remove outdated argo install instruction #1719 (Bobgy)
SDK - ContainerOp.set_display_name should return self to enable chaining #1718 (Ark-kun)
Rename InputPath -> Source for Visualization API definition #1717 (ajchili)
Add SageMaker create workteam and Ground Truth components, sample demo pipeline, other minor updates #1716 (carolynwang)
Support Single part PutFile #1713 (nirsagi)
Fixes cloning of recurring runs #1712 (rileyjbauer)
Restructure samples #1710 (gaoning777)
Simplify sample_test.yaml #1709 (numerology)
add jxzheng to the reviewers for samples #1705 (gaoning777)
Component build fix #1703 (gaoning777)
Allows creation of jobs without experiments #1702 (rileyjbauer)
Backend - Starting the api-server container build from scratch #1699 (Ark-kun)
Moving component_sdk to components/gcp/ #1698 (Ark-kun)
SDK - Lightweight - Added support for complex default values #1696 (Ark-kun)
Changelog 0.1.25 #1695 (jingzhang36)
Move kustomize manifests a dedicate directory #1690 (IronPan)
Clears the workflow's name in GetWorkflowSpec and uses it for the GenerateName #1689 (rileyjbauer)
API - Updated swagger-codegen-cli version #1686 (Ark-kun)
Update SageMaker components and sample pipeline #1682 (carolynwang)
Basic component build sample #1681 (SinaChavoshi)
Separate codegen from containerbuild 2 #1680 (gaoning777)
Separate codegen from containerbuild #1679 (gaoning777)
Add new PlotType to Allow for Visualization Creation #1677 (ajchili)
Container op mount secret sample #1676 (SinaChavoshi)
SDK/Lightweight - Updated default image to tensorflow:1.13.2-py3 #1671 (Ark-kun)
Adding a sample for explicitly defining the execution order #1668 (SinaChavoshi)
Adding multiple outputs into sdk with sample #1667 (zanedurante)
SDK - Removed the build_image parameter from build_python_component function #1657 (Ark-kun)
update kaniko executor version to speed up image build #1652 (gaoning777)
Add code for python visualization service #1651 (ajchili)
SDK/Client - Added the create_run_from_pipeline_package method #1523 (Ark-kun)
SDK - Using Airflow ops in Pipelines #1483 (Ark-kun)
SDK - Cleanup - Serialized PipelineParamTuple does not need value or type #1469 (Ark-kun)
Reorganize ResourceOp samples #1433 (elikatsis)
add default value type checking #1407 (gaoning777)
Seldon examples #1405 (ryandawsonuk)

Open source →