Added

• New datasets.

Changed

• CroissantBuilder's API to generate TFDS datasets from Croissant files.

Deprecated

Removed

Fixed

• Versions for existing datasets.

Security

Added

• Full support for Python 3.12.

Added

• Support to download and prepare datasets using the
  Parquet data format.

  builder = tfds.builder('fashion_mnist', file_format='parquet')
  builder.download_and_prepare()
  ds = builder.as_dataset(split='train')
  print(next(iter(ds)))

• tfds.data_source
  is pickable, thus working smoothly with
  PyGrain. Learn more by following the
  tutorial.

• TFDS plays nicely with
  Croissant. Learn more by
  following the
  recipe.

Changed

Deprecated

Removed

Fixed

Security

Added

• A new CroissantBuilder
  which initializes a DatasetBuilder based on a Croissant
  metadata file.
• New conversion options between different bounding boxes formats.
• Better support for HuggingfaceDatasetBuilder.
• A script
  to convert a dataset from one format to another.

Changed

Deprecated

• Python 3.9 support. TFDS now uses Python 3.10

Removed

Fixed

Security

Added

• Segment Anything
  (SA-1B) dataset.

Changed

• Hugging Face datasets accept None values for any features. TFDS has no
  tfds.features.Optional, so None values are converted to default values.
  Those default values used to be 0 and 0.0 for int and float. Now, it's
  -inf as defined by NumPy (e.g., np.iinfo(np.int32).min or
  np.finfo(np.float32).min). This avoids ambiguous values when 0 and 0.0
  exist in the values of the dataset. The roadmap is to implement
  tfds.features.Optional.

Deprecated

• Python 3.8 support. As per
  NEP 29, TFDS now
  uses Python>=3.9.

Removed

Fixed

Security

Added

• [Experimental] A list of freeform text tags can now be attached to a
  BuilderConfig. For example:

  BUILDER_CONFIGS = [
      tfds.core.BuilderConfig(name="foo", tags=["foo", "live"]),
      tfds.core.BuilderConfig(name="bar", tags=["bar", "old"]),
  ]

  The tags are recorded with the dataset metadata and can later be retrieved
  using the info object:

  builder.info.config_tags  # ["foo", "live"]

  This feature is experimental and there are no guidelines on tags format.

Changed

Deprecated

Removed

Fixed

• Fixed generated proto files (see issue 4858).

Security

Added

Changed

Deprecated

Removed

Fixed

• The installation on macOS now works (see issues
  4805 and
  4852). The ArrayRecord
  dependency is lazily loaded, so the
  TensorFlow-less path is
  not possible at the moment on macOS. A fix for this will follow soon.

Security

Added

• Native support for JAX and PyTorch. TensorFlow is no longer a dependency for reading datasets. See the documentation.
• Added minival split to LVIS dataset.
• Mixed-human and machine-generated robomimic datasets.
• WebVid dataset.
• ImagenetPI dataset.
• Wikipedia for 20230201.

Changed

• Support for tensorflow=2.12.

Deprecated

Removed

Fixed

Security

Added

Changed

Deprecated

• Python 3.7 support: this version and future version use Python 3.8.

Removed

Fixed

• Flag ignore_verifications from Hugging Face's datasets.load_dataset is deprecated, and used to cause errors in tfds.load(huggingface:foo).

Security

Deprecated

• Python 3.7 support: this is the last version of TFDS supporting Python 3.7. Future versions will use Python 3.8.

Fixed

• tfds new and tfds build better support the new recommended datasets organization, where individual datasets have their own package under datasets/, builder class is called Builder and is defined within module ${dsname}_dataset_builder.py.

Security

Changed

• Added file valid_tags.txt to not break builds.
• TFDS no longer relies on TensorFlow DTypes. We chose NumPy DTypes to keep the typing expressiveness, while dropping the heavy dependency on TensorFlow. We migrated all our internal datasets. Please, migrate accordingly:
  • tf.bool: np.bool_
  • tf.string: np.str_
  • tf.int64, tf.int32, etc: np.int64, np.int32, etc
  • tf.float64, tf.float32, etc: np.float64, np.float32, etc

Added

• [API] DatasetBuilder's description and citations can be specified in dedicated README.md and CITATIONS.bib files, within the dataset package (see https://www.tensorflow.org/datasets/add_dataset).
• Tags can be associated to Datasets, in the TAGS.txt file. For now, they are only used in the generated documentation.
• [API][Experimental] New ViewBuilder to define datasets as transformations of existing datasets. Also adds tfds.transform with functionality to apply transformations.
• Loggers are also called on tfds.as_numpy(...), base Logger class has a new corresponding method.
• tfds.core.DatasetBuilder can have a default limit for the number of simultaneous downloads. tfds.download.DownloadConfig can override it.
• tfds.features.Audio supports storing raw audio data for lazy decoding.
• The number of shards can be overridden when preparing a dataset: builder.download_and_prepare(download_config=tfds.download.DownloadConfig(num_shards=42)). Alternatively, you can configure the min and max shard size if you want TFDS to compute the number of shards for you, but want to have control over the shard sizes.

Changed

Deprecated

Removed

Fixed

Security

Added

• [API] Added TfDataBuilder that is handy for storing experimental ad hoc TFDS datasets in notebook-like environments such that they can be versioned, described, and easily shared with teammates.
• [API] Added options to create format-specific dataset builders. The new API now includes a number of NLP-specific builders, such as:
  • CoNNL
  • CoNNL-U
• [API] Added tfds.beam.inc_counter to reduce beam.metrics.Metrics.counter boilerplate
• [API] Added options to group together existing TFDS datasets into dataset collections and to perform simple operations over them.
• [Documentation] update, specifically:
  • New guide on format-specific dataset builders;
  • New guide on adding new dataset collections to TFDS;
  • Updated TFDS CLI documentation.
• [TFDS CLI] Supports custom config through Json (e.g. tfds build my_dataset --config='{"name": "my_custom_config", "description": "Abc"}')
• New datasets:
  • conll2003
  • universal_dependency 2.10
  • bucc
  • i_naturalist2021
  • mtnt Machine Translation of Noisy Text.
  • placesfull
  • tatoeba
  • user_libri_audio
  • user_libri_text
  • xtreme_pos
  • yahoo_ltrc
• Updated datasets:
  • C4 was updated to version 3.1.
  • common_voice was updated to a more recent snapshot.
  • wikipedia was updated with the 20220620 snapshot.
• New dataset collections, such as xtreme and LongT5

Changed

• The base Logger class expects more information to be passed to the as_dataset method. This should only be relevant to people who have implemented and registered custom Logger class(es).
• You can set DEFAULT_BUILDER_CONFIG_NAME in a DatasetBuilder to change the default config if it shouldn't be the first builder config defined in BUILDER_CONFIGS.

Deprecated

Removed

Fixed

• Various datasets
• In Linux, when loading a dataset from a directory that is not your home (~) directory, a new ~ directory is not created in the current directory (fixes #4117).

Security

Added

• Support for community datasets on GCS.
• [API] tfds.builder_from_directory and tfds.builder_from_directories, see https://www.tensorflow.org/datasets/external_tfrecord#directly_from_folder.
• [API] Dash ("-") support in split names.
• [API] file_format argument to download_and_prepare method, allowing user to specify an alternative file format to store prepared data (e.g. "riegeli").
• [API] file_format to DatasetInfo string representation.
• [API] Expose the return value of Beam pipelines. This allows for users to read the Beam metrics.
• [API] Expose Feature tf_example_spec to public.
• [API] doc kwarg on Features, to describe a feature.
• [Documentation] Features description is shown on TFDS Catalog.
• [Documentation] More metadata about HuggingFace datasets in TFDS catalog.
• [Performance] Parallel load of metadata files.
• [Testing] TFDS tests are now run using GitHub actions - misc improvements such as caching and sharding.
• [Testing] Improvements to MockFs.
• New datasets.

Changed

• [API] num_shards is now optional in the shard name.

Removed

• TFDS pathlib API, migrated to a self-contained etils.epath (see https://github.com/google/etils).

Fixed

• Various datasets.
• Dataset builders that are defined adhoc (e.g. in Colab).
• Better DatasetNotFoundError messages.
• Don't set deterministic on a global level but locally in interleave, so it only apply to interleave and not all transformations.
• Google drive downloader.

As always, thank you to all contributors!

Release notes:

• Fix import bug on Windows (#3709)
• Updated documentation

Release notes:

• Fix import bug on Windows (#3709)
• Add split=tfds.split_for_jax_process('train') (alias of tfds.even_splits('train', n=jax.process_count())[jax.process_index()])

This is the last version of TFDS supporting 3.6. Future version will use 3.7

• Better split API:

  • Splits can be selected using shards: split='train[3shard]'
  • Underscore supported in numbers for better readability: split='train[:500_000]'
  • Select the union of all splits with split='all'
  • tfds.even_splits is more precise and flexible:
    • Return splits exactly of the same size when passed tfds.even_splits('train', n=3, drop_remainder=True)
    • Works on subsplits tfds.even_splits('train[:75%]', n=3) or even nested
    • Can be composed with other splits: tfds.even_splits('train', n=3)[0] + 'test'

• FeatureConnectors:

  • Faster dataset generation (using tfrecords)
  • Features now have serialize_example / deserialize_example methods to encode/decode example to proto: example_bytes = features.serialize_example(example_data)
  • Audio now supports encoding='zlib' for better compression
  • Features specs exposed in proto for better compatibility with other languages

• Better testing:

  • Mock dataset now supports nested datasets
  • Customize the number of sub examples

• Documentation update:

  • Community datasets: https://www.tensorflow.org/datasets/community_catalog/overview
  • New guide on TFDS and determinism

• RLDS:

  • Nested datasets features are supported
  • New datasets: Robomimic, D4RL Ant Maze, RLU Real World RL, and RLU Atari with ordered episodes

• Misc:

  • Create beam pipeline using TFDS as input with tfds.beam.ReadFromTFDS
  • Support setting the file formats in tfds build --file_format=tfrecord
  • Typing annotations exposed in tfds.typing
  • tfds.ReadConfig has a new assert_cardinality=False to disable cardinality
  • Add a tfds.display_progress_bar(True) for functional control
  • Support for huge number of shards (>99999)
  • DatasetInfo exposes .release_notes

And of course, new datasets, bug fixes,...

Thank you to all our contributors for improving TFDS!

API:

• Add PartialDecoding support, to decode only a subset of the features (for performances)
• Catalog now expose links to KnowYourData visualisations
• tfds.as_numpy supports datasets with None
• Dataset generated with disable_shuffling=True are now read in generation order.
• Loading datasets from files now supports custom tfds.features.FeatureConnector
• tfds.testing.mock_data now supports
  • non-scalar tensors with dtype tf.string
  • builder_from_files and path-based community datasets
• File format automatically restored (for datasets generated with tfds.builder(..., file_format=)).
• Many new reinforcement learning datasets
• Various bug fixes and internal improvements like:
  • Dynamically set number of worker thread during extraction
  • Update progression bar during download even if downloads are cached

Dataset creation:

• Add tfds.features.LabeledImage for semantic segmentation (like image but with additional info.features['image_label'].name label metadata)
• Add float32 support for tfds.features.Image (e.g. for depth map)
• All FeatureConnector can now have a None dimension anywhere (previously restricted to the first position).
• tfds.features.Tensor() can have arbitrary number of dynamic dimension (Tensor(..., shape=(None, None, 3, None)))
• tfds.features.Tensor can now be serialised as bytes, instead of float/int values (to allow better compression): Tensor(..., encoding='zlib')
• Add script to add TFDS metadata files to existing TF-record (see doc).
• New guide on common implementation gotchas

Thank you all for your support and contribution!

API: • Add dataset.info.splits['train'].num_shards to expose the number of shards to the user • Add tfds.features.Dataset to have a field containing sub-datasets (e.g. used in RL datasets) • Add dtype and tf.uint16 supports for tfds.features.Video • Add DatasetInfo.license field to add redistributing information • Better tfds.benchmark(ds) (compatible with any iterator, not just tf.data, better colab representation)

Other • Faster tfds.as_numpy() (avoid extra tf.Tensor <> np.array copy) • Better tfds.as_dataframe visualisation (Sequence, ragged tensor, semantic masks with use_colormap) • (experimental) community datasets support. To allow dynamically import datasets defined outside the TFDS repository. • (experimental) Add a hugging-face compatibility wrapper to use Hugging-face datasets directly in TFDS. • (experimental) Riegelli format support • (experimental) Add DatasetInfo.disable_shuffling to force examples to be read in generation order. • Add .copy, .format methods to GPath objects • Many bug fixes

Testing: • Supports custom BuilderConfig in DatasetBuilderTest • DatasetBuilderTest now has a dummy_data class property which can be used in setUpClass • Add add_tfds_id and cardinality support to tfds.testing.mock_data

And of course, many new datasets and datasets updates.

We would like to thank all the TFDS contributors!

API:

• Add tfds build to the CLI. See documentation.
• DownloadManager now returns Pathlib-like objects
• Datasets returned by tfds.as_numpy are compatible with len(ds)
• New tfds.features.Dataset to represent nested datasets
• Add tfds.ReadConfig(add_tfds_id=True) to add a unique id to the example ex['tfds_id'] (e.g. b'train.tfrecord-00012-of-01024__123')
• Add num_parallel_calls option to tfds.ReadConfig to overwrite to default AUTOTUNE option
• tfds.ImageFolder now support tfds.decode.SkipDecoder
• Add multichannel audio support to tfds.features.Audio
• Better tfds.as_dataframe visualization (ffmpeg video if installed, bounding boxes,...)
• Add try_gcs to tfds.builder(..., try_gcs=True)
• Simpler BuilderConfig definition: class VERSION and RELEASE_NOTES are applied to all BuilderConfig. Config description is now optional.

Breaking compatibility changes:

• Removed configs for all text datasets. Only plain text version is kept. For example: multi_nli/plain_text -> multi_nli.
• To guarantee better deterministic, new validations are performed on the keys when creating a dataset (to avoid filenames as keys (non-deterministic) and restrict key to str, bytes and int). New errors likely indicates an issue in the dataset implementation.
• tfds.core.benchmark now returns a pd.DataFrame (instead of a dict)
• tfds.units is not visible anymore from the public API

Bug fixes:

• Support 0-len sequence with images of dynamic shape (Fix #2616)
• Progression bar correctly updated when copying files.
• Many bug fixes (GPath consistency with pathlib, s3 compatibility, TQDM visual artifacts, GCS crash on windows, re-download when checksums updated,...)
• Better debugging and error message (e.g. human readable size,...)
• Allow max_examples_per_splits=0 in tfds build --max_examples_per_splits=0 to test _split_generators only (without _generate_examples).

And of course, many new datasets and datasets updates.

Thank you the community for their many valuable contributions and to supporting us in this project!!!

• When generating a dataset, if download fails for any reason, it is now possible to manually download the data. See doc.

• Simplification of the dataset creation API.

  • We've made it is easier to create datasets outside TFDS repository (see our updated dataset creation guide).
  • _split_generators should now returns {'split_name': self._generate_examples(), ...} (but current datasets are backward compatible).
  • All dataset inherit from tfds.core.GeneratorBasedBuilder. Converting a dataset to beam now only require changing _generate_examples (see example and doc).
  • tfds.core.SplitGenerator, tfds.core.BeamBasedBuilder are deprecated and will be removed in future version.

• Better pathlib.Path, os.PathLike compatibility:

  • dl_manager.manual_dir now returns a pathlib-Like object. Example:

  text = (dl_manager.manual_dir / 'downloaded-text.txt').read_text()
  

  • Note: Other dl_manager.download, .extract,... will return pathlib-like objects in future versions
  • FeatureConnector,... and most functions should accept PathLike objects. Let us know if some functions you need are missing.
  • Add a tfds.core.as_path to create pathlib.Path-like objects compatible with GCS (e.g. tfds.core.as_path('gs://my-bucket/labels.csv').read_text()).

• Other bug fixes and improvement. E.g.

  • Add verify_ssl= option to tfds.download.DownloadConfig to disable SSH certificate during download.
  • BuilderConfig are now compatible with Beam datasets #2348
  • --record_checksums now assume the new dataset-as-folder model
  • tfds.features.Images can accept encoded bytes images directly (useful when used with img_name, img_bytes = dl_manager.iter_archive('images.zip')).
  • Doc API now show deprecated methods, abstract methods to overwrite are now documented.
  • You can generate imagenet2012 with only a single split (e.g. only the validation data). Other split will be skipped if not present.

• And of course new datasets

Thank you to all our contributors for improving TFDS!

• Fix tfds.load when generation code isn't present
• Fix improve GCS compatibility.

Thanks @carlthome for reporting and fixing the issue.

API changes, new features:

• Dataset-as-folder: Dataset can now be self-contained module in a folder with checksums, dummy data,... This simplify implementing datasets outside the TFDS repository.
• tfds.load can now load dataset without using the generation class. So tfds.load('my_dataset:1.0.0') can work even if MyDataset.VERSION == '2.0.0' (See #2493).
• Add a new TFDS CLI (see https://www.tensorflow.org/datasets/cli for detail)
• tfds.testing.mock_data does not require metadata files anymore!
• Add tfds.as_dataframe(ds, ds_info) with custom visualisation (example)
• Add tfds.even_splits to generate subsplits (e.g. tfds.even_splits('train', n=3) == ['train[0%:33%]', 'train[33%:67%]', ...]
• Add new DatasetBuilder.RELEASE_NOTES property
• tfds.features.Image now supports PNG with 4-channels
• tfds.ImageFolder now supports custom shape, dtype
• Downloaded URLs are available through MyDataset.url_infos
• Add skip_prefetch option to tfds.ReadConfig
• as_supervised=True support for tfds.show_examples, tfds.as_dataframe

Breaking compatible changes:

• tfds.as_numpy() now returns an iterable which can be iterated multiple times. To migrate next(ds) -> next(iter(ds))
• Rename tfds.features.text.Xyz -> tfds.deprecated.text.Xyz
• Remove DatasetBuilder.IN_DEVELOPMENT property
• Remove tfds.core.disallow_positional_args (should use Py3 *, instead)
• tfds.features can now be saved/loaded, you may have to overwrite FeatureConnector.from_json_content and FeatureConnector.to_json_content to support this feature.
• Stop testing against TF 1.15. Requires Python 3.6.8+.

Other bug fixes:

• Better archive extension detection for dl_manager.download_and_extract
• Fix tfds.__version__ in TFDS nightly to be PEP440 compliant
• Fix crash when GCS not available
• Script to detect dead-urls
• Improved open-source workflow, contributor guide, documentation
• Many other internal cleanups, bugs, dead code removal, py2->py3 cleanup, pytype annotations,...

And of course, new datasets, datasets updates.

A gigantic thanks to our community which has helped us debugging issues and with the implementation of many features, especially vijayphoenix@ for being a major contributor.

• Fix an issue with GCS on Windows.

Future breaking change:

• The tfds.features.text encoding API is deprecated. Please use tensorflow_text instead.

New features

API:

• Add a tfds.ImageFolder and tfds.TranslateFolder to easily create custom datasets with your custom data.
• Add a tfds.ReadConfig(input_context=) to shard dataset, for better multi-worker compatibility (#1426).
• The default data_dir can be controlled by the TFDS_DATA_DIR environment variable.
• Better usability when developing datasets outside TFDS
  • Downloads are always cached
  • Checksum are optional
• Added a tfds.show_statistics(ds_info) to display FACETS OVERVIEW. Note: This require the dataset to have been generated with the statistics.
• Open source various scripts to help deployment/documentation (Generate catalog documentation, export all metadata files,...)

Documentation:

• Catalog display images (example)
• Catalog shows which dataset have been recently added and are only available in tfds-nightly <span class="material-icons">nights_stay</span>

Breaking compatibility change:

• Fix deterministic example order on Windows when path was used as key (this only impact a few datasets). Now example order should be the same on all platforms.
• Remove tfds.load('image_label_folder') in favor of the more user-friendly tfds.ImageFolder

Other:

• Various performances improvements for both generation and reading (e.g. use __slot__, fix parallelisation bug in tf.data.TFRecordReader,...)
• Various fixes (typo, types annotations, better error messages, fixing dead links, better windows compatibility,...)

Thanks to all our contributors who help improving the state of dataset for the entire research community!

Beaking compatibility change:

• Rename tfds.core.NamedSplit, tfds.core.SplitBase -> tfds.Split. Now tfds.Split.TRAIN,... are instance of tfds.Split
• Remove deprecated num_shards argument from tfds.core.SplitGenerator. This argument was ignored as shards are automatically computed.

Future breaking compatibility changes:

• Rename interleave_parallel_reads -> interleave_cycle_length for tfds.ReadConfig.
• Invert ds, ds_info argument orders for tfds.show_examplesFuture breaking change:
• The tfds.features.text encoding API is deprecated. Please use tensorflow_text instead.

Other changes:

• Testing: Add support for custom decoders in tfds.testing.mock_data
• Documentation: shows which datasets are only present in tfds-nightly
• Documentation: display images for supported datasets
• API: Add tfds.builder_cls(name) to access a DatasetBuilder class by name
• API: Add info.split['train'].filenames for access to the tf-record files.
• API: Add tfds.core.add_data_dir to register an additional data dir
• Remove most ds.with_options which where applied by TFDS. Now use tf.data default.
• Other bug fixes and improvement (Better error messages, windows compatibility,...)

Thank you all for your contributions, and helping us make TFDS better for everyone!

Breaking changes:

• Legacy mode tfds.experiment.S3 has been removed
• New image_classification section. Some datasets have been move there from images.
• in_memory argument has been removed from as_dataset/tfds.load (small datasets are now auto-cached).
• DownloadConfig do not append the dataset name anymore (manual data should be in <manual_dir>/ instead of <manual_dir>/<dataset_name>/)
• Tests now check that all dl_manager.download urls has registered checksums. To opt-out, add SKIP_CHECKSUMS = True to your DatasetBuilderTestCase.
• tfds.load now always returns tf.compat.v2.Dataset. If you're using still using tf.compat.v1:
  • Use tf.compat.v1.data.make_one_shot_iterator(ds) rather than ds.make_one_shot_iterator()
  • Use isinstance(ds, tf.compat.v2.Dataset) instead of isinstance(ds, tf.data.Dataset)
• tfds.Split.ALL has been removed from the API.

Future breaking change:

• The tfds.features.text encoding API is deprecated. Please use tensorflow_text instead.
• num_shards argument of tfds.core.SplitGenerator is currently ignored and will be removed in the next version.

Features:

• DownloadManager is now pickable (can be used inside Beam pipelines)
• tfds.features.Audio:
  • Support float as returned value
  • Expose sample_rate through info.features['audio'].sample_rate
  • Support for encoding audio features from file objects
• Various bug fixes, better error messages, documentation improvements
• More datasets

Thank you to all our contributors for helping us make TFDS better for everyone!

New features:

• Datasets expose info.dataset_size and info.download_size. All datasets generated with 2.1.0 cannot be loaded with previous version (previous datasets can be read with 2.1.0 however).
• Auto-caching small datasets. in_memory argument is deprecated and will be removed in a future version.
• Datasets expose their cardinality num_examples = tf.data.experimental.cardinality(ds) (Requires tf-nightly or TF >= 2.2.0)
• Get the number of example in a sub-splits with: info.splits['train[70%:]'].num_examples

• This is the last version of TFDS that will support Python 2. Going forward, we'll only support and test against Python 3.
• The default versions of all datasets are now using the S3 slicing API. See the guide for details.
• The previous split API is still available, but is deprecated. If you wrote DatasetBuilders outside the TFDS repository, please make sure they do not use experiments={tfds.core.Experiment.S3: False}. This will be removed in the next version, as well as the num_shards kwargs from SplitGenerator.
• Several new datasets. Thanks to all the contributors!
• API changes and new features:
  • shuffle_files defaults to False so that dataset iteration is deterministic by default. You can customize the reading pipeline, including shuffling and interleaving, through the new read_config parameter in tfds.load.
  • urls kwargs renamed homepage in DatasetInfo
  • Support for nested tfds.features.Sequence and tf.RaggedTensor
  • Custom FeatureConnectors can override the decode_batch_example method for efficient decoding when wrapped inside a tfds.features.Sequence(my_connector)
  • Declaring a dataset in Colab won't register it, which allow to re-run the cell without having to change the name
  • Beam datasets can use a tfds.core.BeamMetadataDict to store additional metadata computed as part of the Beam pipeline.
  • Beam datasets' _split_generators accepts an additional pipeline kwargs to define a pipeline shared between all splits.
• Various other bug fixes and performance improvements. Thank you for all the reports and fixes!

Bug fixes and performance improvements.

Features

• Add shuffle_files argument to tfds.load function. The semantic is the same as in builder.as_dataset function, which for now means that by default, files will be shuffled for TRAIN split, and not for other splits. Default behaviour will change to always be False at next release.
• Most datasets now support the new S3 API (documentation)
• Support for uint16 PNG images

Misc

• Crash while shuffling on Windows
• Various documentation improvements

New datasets

• AFLW2000-3D
• Amazon_US_Reviews
• binarized_mnist
• BinaryAlphaDigits
• Caltech Birds 2010
• Coil100
• DeepWeeds
• Food101
• MIT Scene Parse 150
• RockYou leaked password
• Stanford Dogs
• Stanford Online Products
• Visual Domain Decathlon

Features

• Add in_memory option to cache small dataset in RAM.
• Better sharding, shuffling and sub-split
• It is now possible to add arbitrary metadata to tfds.core.DatasetInfo which will be stored/restored with the dataset. See tfds.core.Metadata.
• Better proxy support, possibility to add certificate
• Add decoders kwargs to override the default feature decoding (guide).

New datasets

More datasets added:

• downsampled_imagenet
• patch_camelyon
• coco 2017 (with and without panoptic annotations)
• uc_merced
• trivia_qa
• super_glue
• so2sat
• snli
• resisc45
• pet_finder
• mnist_corrupted
• kitti
• eurosat
• definite_pronoun_resolution
• curated_breast_imaging_ddsm
• clevr
• bigearthnet

• Add Apache Beam support
• Add direct GCS access for MNIST (with tfds.load('mnist', try_gcs=True))
• More datasets added
• Option to turn off tqdm bar (tfds.disable_progress_bar())
• Subsplit do not depends on the number of shard anymore (https://github.com/tensorflow/datasets/issues/292)
• Various bug fixes

Thanks to all external contributors for raising issues, their feedback and their pull request.

• Fixes bug #52 that was putting the process in Eager mode by default
• New dataset celeb_a_hq