NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #5533 most downloaded on PyPI
Metaflow: More AI and ML, Less Engineering
Last release 1 months ago
02 Sep 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
7 years old
267 releases · first in 2019
This release fixes an issue with mapping values with spaces from the Argo events payload to flow parameters.
This release fixes an issue with mapping values with spaces from the Argo events payload to flow parameters.
@secrets by @oavdeev in https://github.com/Netflix/metaflow/pull/1474Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.7...2.9.8
This release includes new commands for managing workflows on Argo Workflows. When needed, commands can be authorized by supplying a production token w
This release includes new commands for managing workflows on Argo Workflows.
When needed, commands can be authorized by supplying a production token with --authorize.
argo-workflows deleteA deployed workflow can be deleted through the CLI with
python flow.py argo-workflows delete
argo-workflows terminateA run can be terminated mid-execution through the CLI with
python flow.py argo-workflows terminate RUN_ID
argo-workflows suspend/unsuspendA run can be suspended temporarily with
python flow.py argo-workflows suspend RUN_ID
Note that the suspended flow will show up as failed on Metaflow-UI after a period, due to this also suspending the heartbeat process. Unsuspending will resume the flow and its status will show as running again. This can be done with
python flow.py argo-workflows unsuspend RUN_ID
Previously the status for tasks running on Kubernetes was determined through the pod status, which can take a while to update after the last container finishes. This release changes the status checks to use container statuses directly instead.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.6...2.9.7
One column per quarter.
This release introduces the command step-functions delete for deleting state machines through the CLI.
This release introduces the command step-functions delete for deleting state machines through the CLI.
python flow.py step-functions delete
Comment out the @project decorator from the flow file, as we do not allow using --name with projects.
python project_flow.py step-functions --name project_a.user.saikonen.ProjectFlow delete
python project_flow.py --production step-functions delete
# or
python project_flow.py --branch custom step-functions delete
add --authorize PRODUCTION_TOKEN to the command if you do not have the correct production token locally
This release fixes an issue with the S3 server side encryption support, where some S3 compliant providers do not respond with the expected encryption method in the payload. This bug specifically affected regular operation when using MinIO.
--with environment in AirflowFixes a bug with the Airflow support for environment variables, where the env values set in the environment decorator could get overwritten.
--with environment in Airflow by @valayDave in https://github.com/Netflix/metaflow/pull/1459Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.5...2.9.6
There is now the possibility to choose which server side encryption method to use for S3 uploads by setting an environment variable METAFLOW_S3_SERVER
There is now the possibility to choose which server side encryption method to use for S3 uploads by setting an environment variable METAFLOW_S3_SERVER_SIDE_ENCRYPTION with an appropriate value, for example aws:kms or AES256
This release fixes an issue where using parameters on Argo Workflows caused the values to be unnecessarily quoted.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.4...2.9.5
Using an email address as the username when deploying with a @project decorator to Argo Workflows is now possible. This release fixes an issue with so
Using an email address as the username when deploying with a @project decorator to Argo Workflows is now possible. This release fixes an issue with some generated resources containing characters that are not permitted in names of Argo Workflow resources.
secrets decorator now supports assuming rolesThis release adds the capability to assume specific roles when accessing secrets with the @secrets decorator. The role for accessing a secret can be defined in the following ways
By setting the METAFLOW_DEFAULT_SECRET_ROLE environment variable, this role will be assumed when accessing any secret specified in the decorator.
This will assume the role secret-iam-role for accessing all of the secrets in the sources list.
@secrets(
sources=["first-secret-source", "second-secret-source"],
role="secret-iam-role"
)
Assuming a different role based on the secret in question can be done as well
@secrets(
sources=[
{"type": "aws-secrets-manager", "id": "first-secret-source", "role": "first-secret-role"},
{"type": "aws-secrets-manager", "id": "second-secret-source", "role": "second-secret-role"}
]
)
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.3...2.9.4
Duplicate Metaflow Extensions packages were not properly ignored in all cases. This release fixes this and will allow the loading of extensions even i
Duplicate Metaflow Extensions packages were not properly ignored in all cases. This release fixes this and will allow the loading of extensions even if they are present in duplicate form in your sys.path.
In some cases, packages from the outside environment (non Conda) could leak into the Conda environment when using the environment escape functionality. This release addresses this issue and ensures that no spurious packages are imported in the Conda environment.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.2...2.9.3
Introduce support for _image pull policy_ for @kubernetes
With this release, Metaflow users can specify image pull policy for their workloads through the @kubernetes decorator for Metaflow tasks.
@kubernetes(image='foo:tag', image_pull_policy='Always') # Allowed values are Always, IfNotPresent, Never
@step
def train(self):
...
...
If an image pull policy is not specified, and the tag for the container image is :latest or the tag for the container image is not specified, image pull policy is automatically set to Always.
If an image pull policy is not specified, and the tag for the container image is specified as a value that is not :latest, image pull policy is automatically set to IfNotPresent.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.1...2.9.2
Introduce Slack notifications support for workflow running on Argo Workflows
With this release, Metaflow users can get notified on Slack when their workflows succeed or fail on Argo Workflows. Using this feature is quite straightforward
https://hooks.slack.com/services/T0XXXXXXXXX/B0XXXXXXXXX/qZXXXXXX
python flow.py argo-workflows create --notify-on-error --notify-on-success --notify-slack-webhook-url <slack-webhook-url>
METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_SLACK_WEBHOOK_URL=<slack-webhook-url> in your environment instead of specifying --notify-slack-webhook-url on the CLI everytime.I deployed my workflow following the instructions above, but I haven’t received any notifications yet?
This issue may very well happen if you are running Kubernetes v1.24 or newer.
Since v1.24, Kubernetes stopped automatically creating a secret for every serviceAccount. Argo Workflows relies on the existence of these secrets to run lifecycle hooks responsible for the emission of these notifications.
Follow these steps for explicitly creating a secret for the service account that responsible for executing Argo Workflows steps:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: default-sa-token #change according to the name of the sa
annotations:
kubernetes.io/service-account.name: default #replace with your sa
type: kubernetes.io/service-account-token
EOF
$ kubectl edit sa default -n mynamespace
...
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2023-05-05T20:58:58Z"
name: default
namespace: jobs-default
resourceVersion: "6739507"
uid: 4a708eff-d6ba-4dd8-80ee-8fb3c4c1e1c7
secrets:
- name: default-sa-token # should match the secret above
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
metaflow configure kubernetes by @saikonen in https://github.com/Netflix/metaflow/pull/1405Full Changelog: https://github.com/Netflix/metaflow/compare/2.8.6...2.9.0
Introduce support for composing multiple interrelated workflows through external events
With this release, Metaflow users can architect sequences of workflows that conduct data across teams, all the way from ETL and data warehouse to final ML outputs. Detailed documentation and a blog post to follow very shortly! Keep watching this space.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
metaflow configure kubernetes by @saikonen in https://github.com/Netflix/metaflow/pull/1405Full Changelog: https://github.com/Netflix/metaflow/compare/2.8.6...2.9.0
Introduce support for persistent volume claims for executions on Kubernetes
With this release, Metaflow users can attach existing persistent volume claims to Metaflow tasks running on a Kubernetes cluster.
To use this functionality, simply list your persistent volume claim and mount point using the persistent_volume_claims arg in @kubernetes decorator - @kubernetes(persistent_volume_claims={"pvc-claim-name": "mount-point", "another-pvc-claim-name": "another-mount-point"}).
Here is an example:
from metaflow import FlowSpec, step, kubernetes, current
import os
class MountPVCFlow(FlowSpec):
@kubernetes(persistent_volume_claims={"test-pvc-feature-claim": "/mnt/testvol"})
@step
def start(self):
print('testing PVC')
mount = "/mnt/testvol"
file = f"zeros_run_{current.run_id}"
with open(os.path.join(mount, file), "w+") as f:
f.write("\0" * 50)
f.flush()
print(f"mount folder contents: {os.listdir(mount)}")
self.next(self.end)
@step
def end(self):
print("finished")
if __name__=="__main__":
MountPVCFlow()
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.8.5...2.8.6
Make pickled Metaflow client objects accessible across namespaces
Improvements
The previous release resulted in disabling a sequence of user operations that worked previously:
This release restores the previous behavior.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.8.4...2.8.5
Introduce support for tmpfs for executions on Kubernetes
Features
Improvements
It is typical for the user code in a Metaflow step to download assets from an object store, e.g. S3. Examples include serialized models and raw input data, such unstructured media or structured Parquet files. The amount of data loaded in a task is typically 10-100GB, allowing even terabytes to be handled in a foreach.
To reduce IO bottlenecks in such tasks, we provide an optimized client for S3, metaflow.S3 that makes it possible to download data using all available network bandwidth. Notably, in a modern instance the available network bandwidth can be higher than the local disk bandwidth. Consider: SATA 3.0 provides 6Gbit/s whereas a large instance can have 20Gbit/s network throughput. Even Gen3 NVMe provides just 16Git/s. To benefit from the full network bandwidth, local disk IO must be bypassed. The metaflow.S3 client accomplishes this by relying on the page cache: Nominally files are downloaded in a temporary directory on disk but practically all data stays in the page cache. This is assuming that the downloaded data can fit in memory, which can be ensured by having a high enough @resources(memory=) setting.
The above setup, which can provide excellent IO performance in general, has a small gotcha: The instance needs to have enough local disk space to back all the data, although no data actually hits the disk. Increasingly, instances may have more memory than local disk space available, so this superfluous requirement becomes a problem. This puts users in a strange situation: The instance has enough RAM to hold all the data in memory, and there are ways to download it quickly from S3, but the lack of local disk space (that is not even needed), makes it impossible to access the data.
Kubernetes supports mounting a tmpfs filesystem on the fly. Using this feature, the user can create a memory-backed file system which can be used as a temporary space for downloaded data. This removes the need to have to deal with any local disks. One can simply use a minimal root filesystem, which greatly simplifies the infrastructure setup.
With this release, we introduce a new config option - METAFLOW_TEMPDIR, which, if defined, is used as the default metaflow.S3(tmproot). If METAFLOW_TEMPDIR is not defined, tmproot=’.’ as before. In addition, a few new attributes are introduced for @kubernetes decorator -
| Attribute (default) | Default behavior | Override semantics |
|---|---|---|
| use_tmpfs=False | tmpfs disabled | use_tmpfs=True enables tmpfs |
| tmpfs_tempdir=True | sets METAFLOW_TEMPDIR=tmpfs_path | tmpfs_tempdir=False doesn't set METAFLOW_TEMPDIR |
| tmpfs_size=None | sets tmpfs size to 50% of @resources(memory) | tmpfs size in megabytes |
| tmpfs_path=None | use /metaflow_temp as tmpfs_path | custom mount point |
@kubernetes(memory=100000, use_tmpfs=True)
In this case, at most 50GB is available for tmpfs and it is used by S3 by default. Note that tmpfs only consumes the amount of memory corresponding to the data stored, so there is no downside in setting a large size by default.
@kubernetes(memory=100000, tmpfs_size=100000)
Let tmpfs use all available memory. Note that use_tmpfs=True doesn’t have to be specified redundantly.
@kubernetes(memory=100000, tmpfs_size=10000, tmpfs_path=’/data’, tmpfs_tempdir=False)
Full control over settings - metaflow.S3 doesn’t use the tmpfs volume in this case.
Besides metaflow.S3, the user may want to use the tmpfs volume for their own use cases. In particular, many modern ML libraries require a local cache. To support these use cases, tmpfs_path is exposed through the current object, as current.tempdir. This allows the user to leverage the volume straightforwardly:
AutoModelForSeq2SeqLM.from_pretrained(
model_path,
cache_dir=current.tempdir,
device_map='auto',
load_in_8bit=True,
)
With this release, you can access current.run and current.task within a running flow, allowing for use cases like
from metaflow import current
# add tags from inside a run
current.run.add_tag('foobar')
The previous release broke backward compatibility in cases where the metaflow client object is deserialized from an older version of Metaflow. This release preserves the functionality and provides explicit compatibility guarantees going forward.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
METAFLOW_S3_ENDPOINT_URL as a part of airflow by @valayDave in https://github.com/Netflix/metaflow/pull/1368run and task object. by @romain-intel in https://github.com/Netflix/metaflow/pull/1384Full Changelog: https://github.com/Netflix/metaflow/compare/2.8.3...2.8.4
Introduce support for tmpfs for executions on AWS Batch
Features
Improvements
It is typical for the user code in a Metaflow step to download assets from an object store, e.g. S3. Examples include serialized models and raw input data, such unstructured media or structured Parquet files. The amount of data loaded in a task is typically 10-100GB, allowing even terabytes to be handled in a foreach.
To reduce IO bottlenecks in such tasks, we provide an optimized client for S3, metaflow.S3 that makes it possible to download data using all available network bandwidth. Notably, in a modern instance the available network bandwidth can be higher than the local disk bandwidth. Consider: SATA 3.0 provides 6Gbit/s whereas a large instance can have 20Gbit/s network throughput. Even Gen3 NVMe provides just 16Git/s. To benefit from the full network bandwidth, local disk IO must be bypassed. The metaflow.S3 client accomplishes this by relying on the page cache: Nominally files are downloaded in a temporary directory on disk but practically all data stays in the page cache. This is assuming that the downloaded data can fit in memory, which can be ensured by having a high enough @resources(memory=) setting.
The above setup, which can provide excellent IO performance in general, has a small gotcha: The instance needs to have enough local disk space to back all the data, although no data actually hits the disk. Increasingly, instances may have more memory than local disk space available, so this superfluous requirement becomes a problem. The issue is further amplified by the fact that as of today, it is impossible to add ephemeral volumes on the fly on AWS Batch. This puts users in a strange situation: The instance has enough RAM to hold all the data in memory, and there are ways to download it quickly from S3, but the lack of local disk space (that is not even needed), makes it impossible to access the data.
AWS Batch supports mounting a tmpfs filesystem on the fly. Using this feature, the user can create a memory-backed file system which can be used as a temporary space for downloaded data. This removes the need to have to deal with any local disks. One can simply use a minimal root filesystem, which greatly simplifies the infrastructure setup.
With this release, we introduce a new config option - METAFLOW_TEMPDIR, which, if defined, is used as the default metaflow.S3(tmproot). If METAFLOW_TEMPDIR is not defined, tmproot=’.’ as before. In addition, a few new attributes are introduced for @batch decorator -
| Attribute (default) | Default behavior | Override semantics |
|---|---|---|
| use_tmpfs=False | tmpfs disabled | use_tmpfs=True enables tmpfs |
| tmpfs_tempdir=True | sets METAFLOW_TEMPDIR=tmpfs_path | tmpfs_tempdir=False doesn't set METAFLOW_TEMPDIR |
| tmpfs_size=None | sets tmpfs size to 50% of @resources(memory) | tmpfs size in megabytes |
| tmpfs_path=None | use /metaflow_temp as tmpfs_path | custom mount point |
@batch(memory=100000, use_tmpfs=True)
In this case, at most 50GB is available for tmpfs and it is used by S3 by default. Note that tmpfs only consumes the amount of memory corresponding to the data stored, so there is no downside in setting a large size by default.
@batch(memory=100000, tmpfs_size=100000)
Let tmpfs use all available memory. Note that use_tmpfs=True doesn’t have to be specified redundantly.
@batch(memory=100000, tmpfs_size=10000, tmpfs_path=’/data’, tmpfs_tempdir=False)
Full control over settings - metaflow.S3 doesn’t use the tmpfs volume in this case.
Besides metaflow.S3, the user may want to use the tmpfs volume for their own use cases. In particular, many modern ML libraries require a local cache. To support these use cases, tmpfs_path is exposed through the current object, as current.tempdir. This allows the user to leverage the volume straightforwardly:
AutoModelForSeq2SeqLM.from_pretrained(
model_path,
cache_dir=current.tempdir,
device_map='auto',
load_in_8bit=True,
)
With this release, Metaflow client objects will support autocomplete in ipython notebooks
from metaflow import Flow, Metaflow
Metaflow().flows
>>> [Flow('HelloFlow'), Flow('MovieStatsFlow')]
flow = Flow('HelloFlow') # No autocomplete here
flow._ipython_key_completions_()
>>>
['1680815181013681',
'1680815178214737',
'1680432265121345',
'1680430310127401']
run = flow["1680815178214737"]
run._ipython_key_completions_()
>>> ['end', 'hello', 'start']
step = run["hello"]
step._ipython_key_completions_()
>>> ['2']
task = step["2"]
task._ipython_key_completions_()
>>> ['name']
With this release, Metaflow flows should execute a tad bit faster since a few network calls to Metaflow's metadata service are now cached. Expect continued further improvements in flow execution times over the next few releases.
With this release, Metaflow card creation will handle non-JSON parseable types gracefully by replacing the column values with UnsupportedType : <TYPENAME>.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
pandas.DataFrame for default card by @valayDave in https://github.com/Netflix/metaflow/pull/1344Full Changelog: https://github.com/Netflix/metaflow/compare/2.8.2...2.8.3
Introduce support for Metaflow sandboxes for Metaflow tutorials
With this release, the Metaflow tutorials can now be executed within the Metaflow sandboxes, making it trivial to evaluate whether Metaflow is a good fit for your organization without committing to deploying the necessary cloud infrastructure upfront.
step-functions trigger or argo-workflows triggerWith this release, if the Metaflow config (in ~/.metaflow_config) includes a reference to the deployed Metaflow UI (assigned to METAFLOW_UI_URL), the user-facing logs in the terminal will indicate the direct link to the relevant run view in the Metaflow UI.
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
logs command in cases where the step/task hasn't finished by @romain-intel in https://github.com/Netflix/metaflow/pull/1315Full Changelog: https://github.com/Netflix/metaflow/compare/2.8.1...2.8.2
Add ec2 instance metadata in `task.metadata_dict` when a task executes on AWS Batch
task.metadata_dict when a task executes on AWS BatchWith this release, task.metadata_dict will include the fields - ec2-instance-id, ec2-instance-type, ec2-region, and ec2-availability-zone whenever the Metaflow task is executed on AWS Batch and the task container has access to ec2 metadata magic URL.
run or resumeWith this release, if the Metaflow config (in ~/.metaflow_config) includes a reference to the deployed Metaflow UI (assigned to METAFLOW_UI_URL), the user-facing logs in the terminal will indicate the direct link to the relevant run view in the Metaflow UI.
<img width="992" alt="Screen Shot 2023-03-15 at 12 46 01 PM" src="https://user-images.githubusercontent.com/763451/225425538-05cca220-a6d2-4c49-b01e-cc191d3e58a2.png">
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
Introduce capability to schedule Metaflow flows with Apache Airflow
With this release, we are introducing an integration with Apache Airflow similar to our integrations with AWS Step Functions and Argo Workflows where Metaflow users can easily deploy & schedule their DAGs by simply executing
python myflow.py airflow create mydag.py
which will create an Airflow DAG for them. With this feature, Metaflow users can now enjoy all the features of Metaflow on top of Apache Airflow - including a more user-friendly and productive development API for data scientists and data engineers, without needing to change anything in your existing pipelines or operational playbooks, as described in its announcement blog post. To learn how to deploy and operate the integration, see Using Airflow with Metaflow.
When running on Airflow, Metaflow code works exactly as it does locally: No changes are required in the code. With this integration, Metaflow users can inspect their flows deployed on Apache Airflow as before and debug and reproduce results from Apache Airflow on their local laptop or within a notebook. All tasks are run on Kubernetes respecting the @resources decorator as if the @kubernetes decorator was added to all steps, as explained in Executing Tasks Remotely.
The main benefits of using Metaflow with Airflow are:
In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.
New MF configs for Argo Workflows by @jackie-ob in https://github.com/Netflix/metaflow/pull/1267
metaflow_extensions, add an empty __init__.py file. by @romain-intel in https://github.com/Netflix/metaflow/pull/1276Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.22...2.7.23
Test metaflow.s3 on multiple Python versions across Linux and MacOS by @savingoyal in https://github.com/Netflix/metaflow/pull/1246
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.21...2.7.22
Fix extension support on Python 3.5 by @romain-intel in https://github.com/Netflix/metaflow/pull/1245
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.20...2.7.21
Bug/long card by @obgibson in https://github.com/Netflix/metaflow/pull/1233
If you are using the unsupported Metaflow Extensions mechanism, you may have to change them slightly. Please see https://github.com/Netflix/metaflow-extensions-template/blob/master/CHANGES.md for more details.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.19...2.7.20
Fix CVE-2007-4559 (tar.extractall) by @romain-intel in https://github.com/Netflix/metaflow/pull/1213
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.18...2.7.19
Adds check for tutorials dir and flattens if necessary by @ashrielbrian in https://github.com/Netflix/metaflow/pull/1211
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.17...2.7.18
Fix regression causing CL tool to not work. by @romain-intel in https://github.com/Netflix/metaflow/pull/1209
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.16...2.7.17
Deal with transient errors (like SlowDowns) more effectively for S3 by @romain-intel in https://github.com/Netflix/metaflow/pull/1186
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.15...2.7.16
Handle aborted Kubernetes workloads. by @shrinandj in https://github.com/Netflix/metaflow/pull/1195
._orig access for submodules for MF extensions by @romain-intel in https://github.com/Netflix/metaflow/pull/1174Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.14...2.7.15
fix pandas call bug by @mbalajew in https://github.com/Netflix/metaflow/pull/1173
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.13...2.7.14
Add cmd extension point to allow MF extensions to extend it by @romain-intel in https://github.com/Netflix/metaflow/pull/1143
cmd extension point to allow MF extensions to extend it by @romain-intel in https://github.com/Netflix/metaflow/pull/1143kubernetes_conn_id in Airflow integration by @valayDave in https://github.com/Netflix/metaflow/pull/1153Image.from_matplotlib by @valayDave in https://github.com/Netflix/metaflow/pull/1147Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.12...2.7.13
The Metaflow 2.7.12 release is a minor release
The Metaflow 2.7.12 release is a minor release
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.11...2.7.12
Fix DeprecationWarning on invalid escape sequence by @tommybrecher in https://github.com/Netflix/metaflow/pull/1133
The Metaflow 2.7.11 release is a minor release
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.10...2.7.11
Card bug fix when task-ids are non-unique by @valayDave in https://github.com/Netflix/metaflow/pull/1126
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.9...2.7.10
Fix issue with S3 URLs for packages by @savingoyal in https://github.com/Netflix/metaflow/pull/1130
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.8...2.7.9
Support airflow with metaflow on azure by @valayDave in https://github.com/Netflix/metaflow/pull/1127
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.7...2.7.8
The Metaflow 2.7.7 release is a minor release
The Metaflow 2.7.7 release is a minor release
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.6...2.7.7
Fix another issue with the escape hatch and paths by @romain-intel in https://github.com/Netflix/metaflow/pull/1105
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.5...2.7.6
Fix for env_escape bug when importing local packages by @hunsdiecker in https://github.com/Netflix/metaflow/pull/1100
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.4...2.7.5
Fix docstrings for the upcoming API reference (no functional changes!) by @tuulos in https://github.com/Netflix/metaflow/pull/1076
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.3...2.7.4
The Metaflow 2.7.3 release is a minor release
The Metaflow 2.7.3 release is a minor release
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.2...2.7.3
Metaflow 2.7.2 is a minor release
Metaflow 2.7.2 is a minor release
@conda in https://github.com/Netflix/metaflow/pull/1077Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.1...2.7.2
This is a patch release addressing a behavior of the environment escape mechanism.
This is a patch release addressing a behavior of the environment escape mechanism.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.0...2.7.1
This is a minor release which primarily adds the ability to do runtime tagging.
This is a minor release which primarily adds the ability to do runtime tagging.
tag) as well as methods in the client add_tags, replace_tags and remove_tags. If using the Metaflow Metadata service, a version greater than 2.3.0 is required to use this feature.Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.3...2.7.0
The Metaflow 2.6.3 release is a minor release
The Metaflow 2.6.3 release is a minor release
Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.2...2.6.3
The Metaflow 2.6.2 release is a minor release
The Metaflow 2.6.2 release is a minor release
@kubernetes (#1048 ). Metaflow allows you mount secrets in Kubernetes containers created by tasks. Now you can specify a set of secrets to be mounted by default via METAFLOW_KUBERNETES_SECRETS configuration option, in addition to existing @kubernetes(secrets="...") API. --run-id-file, the file is now written prior to execution when resuming a flow (#1051). That matches how run command behaves already.Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.1...2.6.2
The Metaflow 2.6.1 release is a minor release.
The Metaflow 2.6.1 release is a minor release.
card list command in https://github.com/Netflix/metaflow/pull/1044Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.0...2.6.1
The Metaflow 2.6.0 release is a minor release and introduces Metaflow's integration with Kubernetes and Argo Workflows
The Metaflow 2.6.0 release is a minor release and introduces Metaflow's integration with Kubernetes and Argo Workflows
tags in current object.This release enables brand new capabilities for Metaflow on top of Kubernetes. You can now run --with kubernetes all or parts of any Metaflow flow on top of any Kubernetes cluster from your workstation. To execute your flow asynchronously, you can deploy the flow to Argo Workflows (a Kubernetes-native workflow scheduler) with a single command - argo-workflows create.
To get started, take a look at the deployment guide for Kubernetes. Your feedback and feature requests are highly appreciated! - please reach out to us at slack.outerbounds.co
PR #992 addressed issue #50.
tags in current object.Metaflow tags are now available as part of the current singleton object.
@step
def my_step(self):
from metaflow import current
tags = current.tags
...
PR #1019 fixed issue #1007.
The Metaflow 2.5.4 release is a minor release.
The Metaflow 2.5.4 release is a minor release.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.3...2.5.4
The Metaflow 2.5.3 release is a minor release.
The Metaflow 2.5.3 release is a minor release.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.2...2.5.3
The Metaflow 2.5.2 release is a minor release.
The Metaflow 2.5.2 release is a minor release.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.1...2.5.2
The Metaflow 2.5.1 release is a minor release.
The Metaflow 2.5.1 release is a minor release.
@conda in https://github.com/Netflix/metaflow/pull/918 . Mamba promises faster package dependency resolution times, which should result in an appreciable speedup in flow environment initialization. It is not yet enabled by default; to use it you need to set METAFLOW_CONDA_DEPENDENCY_RESOLVER to mamba in Metaflow config.Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.0...2.5.1
The Metaflow 2.5.0 release is a minor release.
The Metaflow 2.5.0 release is a minor release.
:sparkles: Metaflow cards are now publicly available! For details, see a new section in the documentation, Visualizing Results, and a release blog post.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.4.9...2.5.0
The Metaflow 2.4.9 release is a patch release.
The Metaflow 2.4.9 release is a patch release.
@card ( https://github.com/Netflix/metaflow/pull/822 )Full Changelog: https://github.com/Netflix/metaflow/compare/2.4.8...2.4.9
The Metaflow 2.4.8 release is a patch release.
The Metaflow 2.4.8 release is a patch release.
aws_retry's S3_RETRY_COUNT now has to be >=1 ( https://github.com/Netflix/metaflow/pull/876 )Full Changelog: https://github.com/Netflix/metaflow/compare/2.4.7...2.4.8
The Metaflow 2.4.7 release is a patch release. We skipped 2.4.6 for technical reasons.
The Metaflow 2.4.7 release is a patch release. We skipped 2.4.6 for technical reasons.
@card decoratorFull Changelog: https://github.com/Netflix/metaflow/compare/2.4.5...2.4.7
The Metaflow 2.4.5 release is a patch release.
The Metaflow 2.4.5 release is a patch release.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.4.4...2.4.5
The Metaflow 2.4.4 release is a patch release.
The Metaflow 2.4.4 release is a patch release.
@batch & @kubernetes (#807)We're moving to a more consistent scheme for naming options related to docker images. You can read the details in #489, but this release introduces new config options DEFAULT_CONTAINER_IMAGE and DEFAULT_CONTAINER_REGISTRY that can be used to specify docker image in addition to plugin-specific options like KUBERNETES_CONTAINER_IMAGE
This adds a new configuration option to set the default namespace for the Kubernetes plugin
The Metaflow 2.4.3 release is a patch release
The Metaflow 2.4.3 release is a patch release
When accessing artifacts of a running task using Task(...).artifacts, a race condition existed and the call could return a difficult to understand error message. This release fixes this issue and making this call will either return the artifacts present or no artifacts at all if none are present yet.
@catch and @retry decorators (#776)A step as below:
@retry(times=2)
@catch(var='exception')
@step
def my_step(self):
raise ValueError()
would not retry 2 times as expected but instead the exception would be caught the first time around. This release fixes this issue and the step will now execute a total of 3 times and the exception will be caught on the third time.
On MacOS Big Sur, certain tutorials were broken due to using an older version of Pandas. This updates the tutorials to use 1.3.3 to solve this issue.
The Metaflow 2.4.2 release is a patch release
The Metaflow 2.4.2 release is a patch release
metaflow.client (#779)Metaflow v2.4.1 introduced a bug (due to a typo) in accessing legacy task logs through metaflow.client
Task("pathspec/to/task").stdout
This release fixes this issue.
A subtle bug was introduced in Metaflow 2.4.0 where the task datastore access fails when no task attempt was recorded. This release fixes this issue.
The Metaflow 2.4.1 release is a patch release
The Metaflow 2.4.1 release is a patch release
Prior to this release, non-pythonic dependencies in a conda environment were not automatically visible to a Metaflow task executing on AWS Batch (see #734) (they were available for tasks that were executed locally). For example
import os
from metaflow import FlowSpec, step, conda, conda_base, batch
class TestFlow(FlowSpec):
@step
def start(self):
self.next(self.use_node)
@batch
@conda(libraries={"nodejs": ">=16.0.0"})
@step
def use_node(self):
print(os.system("node --version"))
self.next(self.end)
@step
def end(self):
pass
if __name__ == "__main__":
TestFlow()
would print an error. This release fixes the issue with the incorrect PATH configuration.
This release exposes size properties for artifacts and logs (stderr and stdout) in metaflow.client. These properties are relied upon by the Metaflow UI (open-sourcing soon!).
In addition to the above mentioned properties, now users of Metaflow can access attempt specific Task metadata using the client
Task('42/start/452', attempt=1)
This release marks the alpha launch of @kubernetes decorator that allows farming off Metaflow tasks onto Kubernetes. The functionality works in exactly the same manner as @batch -
from metaflow import FlowSpec, step, resources
class BigSum(FlowSpec):
@resources(memory=60000, cpu=1)
@step
def start(self):
import numpy
import time
big_matrix = numpy.random.ranf((80000, 80000))
t = time.time()
self.sum = numpy.sum(big_matrix)
self.took = time.time() - t
self.next(self.end)
@step
def end(self):
print("The sum is %f." % self.sum)
print("Computing it took %dms." % (self.took * 1000))
if __name__ == '__main__':
BigSum()
python big_sum.py run --with kubernetes
will run all steps of this workflow on your existing EKS cluster (which can be configured with metaflow configure eks) and provides all the goodness of Metaflow!
To get started follow this guide! We would appreciate your early feedback at http://slack.outerbounds.co.
The Metaflow 2.4.0 release is a minor release and includes a *breaking change*
The Metaflow 2.4.0 release is a minor release and includes a breaking change
Prior to this release, the return type for created_at and finished_at properties in the Client API was a timestamp
string. This release changes this to a datetime object, as the old behavior is considered an unintentional mis-feature
(see below for details).
To keep the old behavior, append an explicit string conversion, .strftime('%Y-%m-%dT%H:%M:%SZ'), to
the created_at and finshed_at calls, e.g.
run.created_at.strftime('%Y-%m-%dT%H:%M:%SZ')
The first versions of Metaflow (internal to Netflix) returned a datetime object in all calls dealing with timestamps in
the Client API to make it easier to perform operations between timestamps. Unintentionally, the return type was changed
to string in the initial open-source release. This release introduces a number of internal changes, removing all
remaining discrepancies between the legacy version of Metaflow that was used inside Netflix and the open-source version.
The timestamp change is the only change affecting the user-facing API. While Metaflow continues to make a strong promise of backwards compatibility of user-facing features and APIs, the benefits of one-time unification outweigh the cost of this relatively minor breaking change.
Conda errors printed to stderr were not surfaced to the user; this release addresses this issue.
The code responsible for printing error messages from the metadata service had a problem that could cause it to be unable to print the correct error message and would instead raise another error that obfuscated the initial error. This release addresses this issue and errors from the metadata service are now properly printed.
This release allows you to set the number of times S3 access are retried (the default is 7). The relevant environment variable is: METAFLOW_S3_RETRY_COUNT.
The datastore implementation was reworked to make it easier to extend in the future. It also now uploads artifacts in parallel to S3 (as opposed to sequentially) which can lead to better performance. The changes also contribute to a notable improvement in the speed of resume which can now start resuming a flow twice as fast as before. Documentation can be found here.
The S3 datatools better handles small versus large files by using the download_file command for larger files and using get_object for smaller files to minimize the number of calls made to S3.
The Metaflow 2.3.6 release is a patch release.
The Metaflow 2.3.6 release is a patch release.
METAFLOW_DEFAULT_ENVIRONMENT is set to condaPrior to this release, setting default execution environment to conda through METAFLOW_DEFAULT_ENVIRONMENT would result in a recursion error.
METAFLOW_DEFAULT_ENVIRONMENT=conda python flow.py run
File "/Users/savin/Code/metaflow/metaflow/cli.py", line 868, in start
if e.TYPE == environment][0](ctx.obj.flow)
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 27, in __init__
if e.TYPE == DEFAULT_ENVIRONMENT][0](self.flow)
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 27, in __init__
if e.TYPE == DEFAULT_ENVIRONMENT][0](self.flow)
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 27, in __init__
if e.TYPE == DEFAULT_ENVIRONMENT][0](self.flow)
[Previous line repeated 488 more times]
File "/Users/savin/Code/metaflow/metaflow/plugins/conda/conda_environment.py", line 24, in __init__
from ...plugins import ENVIRONMENTS
RecursionError: maximum recursion depth exceeded
This release fixes this bug.
host_volumes attribute for @batch decoratorDots in volume names - @batch(host_volumes='/path/with/.dot') weren't being santized properly resulting in errors when a Metaflow task launched on AWS Batch. This release fixes this bug.
The Metaflow 2.3.5 release is a patch release.
The Metaflow 2.3.5 release is a patch release.
With this release, you can now mount and access instance host volumes within a Metaflow task running on AWS Batch. To access a host volume, you can add host-volumes argument to your @batch decorator -
@batch(host_volumes=['/home', '/var/log'])
list within a Metaflow Foreach taskThe following flow had a bug where the value for self.input was being imputed to None rather than the dictionary element. This release fixes this issue -
from metaflow import FlowSpec, Parameter, step, JSONType
class ForeachFlow(FlowSpec):
numbers_param = Parameter(
"numbers_param",
type=JSONType,
default='[1,2,3]'
)
@step
def start(self):
# This works, and passes each number to the run_number step:
#
# self.numbers = self.numbers_param
# self.next(self.run_number, foreach='numbers')
# But this doesn't:
self.next(self.run_number, foreach='numbers_param')
@step
def run_number(self):
print(f"number is {self.input}")
self.next(self.join)
@step
def join(self, inputs):
self.next(self.end)
@step
def end(self):
pass
if __name__ == '__main__':
ForeachFlow()
Your coding agent can read these notes before it upgrades. Set up the MCP server →