This is a fairly major update to umoci.

This update to umoci includes support for v1.1.1 of the OCI image
specification. For the most part, this mostly involves supporting reading new
features added to the specification (such as embedded-data descriptors and
subject references used by OCI artifact images), but at the moment umoci does
not yet support creating images utilising these features.

In addition, umoci also now supports generating config.json blobs that are
compliant with v1.2.1 of the OCI runtime specification. Note that we do not
explicitly use any of the newer features, this is mostly a quality-of-life
update to move away from our ancient pinned version of the runtime-spec.

Breaking

• github.com/opencontainers/umoci/oci/config/generate.Generator has had the
  following breaking API changes made to it:
  • The existing ConfigExposedPorts and ConfigVolumes methods now return a
    sorted []string instead of a map.
  • The (Set)OS and (Set)Architecture methods have been renamed to have a
    Platform prefix (to match image-spec v1.1's organisational changes). They
    now read as (Set)PlatformOS and (Set)PlatformArchitecture respectively.

Added

• umoci stat now includes information about the manifest and configuration of
  the image, both in the regular and JSON-formatted outputs.

• umoci now has SOURCE_DATE_EPOCH support, to attempt to
  make it easier to create reproducible images. Our behaviour is modelled after
  tar --clamp-mtime, meaning that SOURCE_DATE_EPOCH will only be used to
  modify the timestamps of files newer than SOURCE_DATE_EPOCH.

  As umoci repack works based on diffs, this also means that only files that
  were modified (and will thus be usually be included in the new layer) will
  have their timestamps rewritten.

  --history.created and umoci config --created will also now default to
  SOURCE_DATE_EPOCH (if set).

  With this change, umoci should be fairly compliant with reproducible builds.
  Please let us know if you find any other problematic areas in umoci (we are
  investigating some other possible causes of instability such as Go map
  iteration).

• In order to avoid the need for a patched gomtree package
  that supports rootless mode, umoci now has a umoci raw mtree-validate
  subcommand that implements the key gomtree validate features we need for
  our integration tests.

  Note that this subcommand is not intended for wider use outside of our tests
  (and it is hidden from the help pages for a reason). Most users are probably
  better off just using gomtree.

• umoci --version now provides more information about the specification
  versions supported by the umoci binary as well as the Go version used.

• umoci config now supports specifying the architecture variant of the image
  with --platform.variant. In addition, --os and --architecture can now
  be set using --platform.os and --platform.arch respectively.

• umoci new will not automatically fill the architecture variant on ARM
  systems to match the host CPU.

Changed

• The output format of umoci stat has had some minor changes made to how
  special characters are escaped and when quoting is carried out.

Fixed

• Some minor aspects of how umoci stat would filter special characters in
  history entries have been resolved.
• umoci repack will now truncate the mtime of files added to the layer tar
  archives. Previously, we would defer to the Go stdlib's archive/tar which
  rounds to the nearest second (which is incompatible with gomtree and so in
  theory could lead to inconsistent results).
• Previously, when generating the runtime-spec config.json, umoci unpack
  would incorrectly prioritise the automatically generated annotations over
  explicitly configured labels. This precdence was the opposite of what the
  image-spec requires, and has now been resolved.

Thanks to the following contributors for making this release possible:

• Aleksa Sarai cyphar@cyphar.com
• Danish Prakash contact@danishpraka.sh

Signed-off-by: Aleksa Sarai cyphar@cyphar.com

This is a fairly minor update to umoci, containing a few bugfixes for
some potential issues, as well as finally removing the requirement for
oci-image-tool validation. We still do not support the latest image-spec
release, but decoupling for oci-image-tool is a very important first
step.

Fixed

• For images with an empty index.json, umoci will no longer incorrectly set
  the manifests entry to null (which was technically a violation of the
  specification, though such images cannot be pushed or interacted with outside
  of umoci).
• Based on some recent developments in the image-spec, umoci
  will now produce an error if it encounters descriptors with a negative size
  (this was a potential DoS vector previously) as well as a theoretical attack
  where an attacker would endlessly write to a blob (this would not be
  generally exploitable for images with descriptors).

Changed

• We now use go:embed to fill the version information of umoci --version,
  allowing for users to get a reasonable binary with go install. However, we
  still recommend using our official binaries, using distribution binaries, or
  building from source with make.

• Rather than using oci-image-tool validate for validating images in our
  tests, we now make use of some hand-written smoke tests as well as the
  jq-based validators maintained in docker-library/meta-scripts.

  This is intended to act as a stop-gap until umoci validate is implemented
  (and after that, we may choose to keep the jq-based validators as a
  double-check that our own validators are working correctly).

Thanks to the following contributors who made this release possible:

• Adam Korcz Adam@adalogics.com
• Akhil Mohan akhilerm@gmail.com
• Aleksa Sarai cyphar@cyphar.com

---

This release is dedicated to our cat Yuki who sadly passed away on
Friday. Most of the code I've written in the past four years was written
with him purring away on my chest, and he was the most loving cat I've
ever met. Rest in peace, little buddy. I hope you enjoyed your time with
us, and I'll always keep you in my heart. 🖤

Signed-off-by: Aleksa Sarai cyphar@cyphar.com

This is a long-awaited release of umoci containing some Go API breaking
changes, some new features, and many other minor changes and
improvements.

Note that the Go API is still considered to be unstable, so downstream
users should generally be aware that future updates may contain more
breaking changes until we release umoci v1.0.0. However, the umoci CLI
is considered to be stable (as it has been widely used for nearly a
decade now) and we will endeavour to not make breaking changes.

This version of umoci requires Go 1.23 to build.

Security

• A security flaw was found in the OCI image-spec, where it is possible to
  cause a blob with one media-type to be interpreted as a different media-type.
  As umoci is not a registry nor does it handle signatures, this vulnerability
  had no real impact on umoci but for safety we implemented the now-recommended
  media-type embedding and verification. CVE-2021-41190

Breaking

• The method of configuring the on-disk format and MapOptions in
  RepackOptions and UnpackOptions has been changed. The on-disk format is
  now represented with the OnDiskFormat interface, with DirRootfs and
  OverlayfsRootfs as possible options to use. MapOptions is now configured
  inside the OnDiskFormat setting, which will require callers to adjust their
  usage of the main umoci APIs. In particular, examples like

  unpackOptions := &layer.UnpackOptions{
      MapOptions: mapOptions,
      WhiteoutMode: layer.StandardOCIWhiteout, // or layer.OverlayFSWhiteout
  }
  err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)

  will have to now be written as

  unpackOptions := &layer.UnpackOptions{
      OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
          MapOptions: mapOptions,
      },
  }
  err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)

  and similarly

  repackOptions := &layer.RepackOptions{
      MapOptions: mapOptions,
      TranslateOverlayWhiteouts: false, // or true
  }
  layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)

  will have to now be written as

  repackOptions := &layer.RepackOptions{
      OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
          MapOptions: mapOptions,
      },
  }
  layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)

  Note that this means you can easily re-use the OnDiskFormat configuration
  between both UnpackOptions and RepackOptions, removing the previous need
  to translate between WhiteoutMode and TranslateOverlayWhiteouts.

  For users of the API that need to extract the MapOptions from
  UnpackOptions and RepackOptions, there is a new helper MapOptions which
  will help extract it without doing interface type switching. For
  OnDiskFormat there is also a Map method that gives you the inner
  MapOptions regardless of type.

• layer.NewTarExtractor now takes *UnpackOptions rather than
  UnpackOptions to match the signatures of the other layer.* APIs. Passing
  nil is equivalent to passing &UnpackOptions{}.

• In umoci 0.4.7, we added support for overlayfs unpacking using the
  still-unstable Go API. However, the implementation is still missing some key
  features and so we will now return errors from APIs that are still missing
  key features:

  • layer.UnpackManifest and layer.UnpackRootfs will now return an error
    if UnpackOptions.OnDiskFormat is set to anything other than DirRootfs
    (the default, equivalent to WhiteoutMode being set to
    OCIStandardWhiteout in umoci 0.4.7).

    This is because bundle-based unpacking currently tries to unpack all
    layers into the same rootfs and generate an mtree manifest -- this
    doesn't make sense for overlayfs-style unpacking and will produce garbage
    bundles as a result. As such, we expect that nobody actually made use of
    this feature (otherwise we would've seen bug reports complaining about it
    being completely broken in the past 4 years). opencontainers/umoci#574
    tracks re-enabling this feature (and exposing to umoci CLI users, if
    possible).

    Note that layer.UnpackLayer still supports OverlayfsRootfs
    (OverlayFSWhiteout in umoci 0.4.7).

  • Already-extracted bundles with OverlayfsRootfs (OverlayFSWhiteout in
    umoci 0.4.7) will now return an error when umoci operates on
    them -- we included the whiteout mode in our umoci.json but as the
    feature is broken, umoci will now refuse to operate on such bundles. Such
    bundles could only have been created using the now-error-inducing
    UnpackRootfs and UnpackManifest APIs mentioned above, and as mentioned
    above we expect there to have been no real users of this feature.

    Note that this only affects extracted bundles (a-la umoci unpack).
    Images created from such bundles are unaffected (even though their
    contents probably should be audited, since the implementation of this
    feature was quite broken in this usecase).

  Users should expect more breaking changes in the overlayfs-related Go APIs in
  a future umoci 0.6 release, as there is still a lot of work left to do.

Added

• umoci unpack now supports handling layers compressed with zstd. This is
  something that was added in image-spec v1.2 (which we do not yet support
  fully) but at least this will allow users to operate on zstd-compressed
  images, which are slowly becoming more common.
• umoci repack and umoci insert now support creating zstd-compressed
  layers. The default behaviour (called auto) is to try to match the last
  layer's compression algorithm, with a fallback to gzip if none of the layer
  algorithms were supported.
  • Users can specify their preferred compression algorithm using the new
    --compress flag. You can also disable compression entirely using
    --compress=none but --compress=auto will never automatically choose
    none compression.
• GenerateLayer and GenerateInsertLayer with OverlayfsRootfs
  (called TranslateOverlayWhiteouts in umoci 0.4.7) now support
  converting trusted.overlay.opaque=y and trusted.overlay.whiteout
  whiteouts into OCI whiteouts when generating OCI layers.
• OverlayfsRootfs now supports compatibility with the userxattr mount
  option for overlayfs (where user.overlay.* xattrs are used rather than
  the default trusted.overlay.*). This is a pretty key compatibility feature
  for users that use unprivileged overlayfs mounts and will hopefully remove
  the need for most downstream forks hacking in this functionality (such as
  stacker). For Go API users, to enable this just set UserXattr: true in
  OverlayfsRootfs. Note that (as with upstream overlayfs), only one xattr
  namespace is ever used (so if OverlayfsRootfs.UserXattr == true then
  trusted.overlay.* xattrs will be treated like any other non-overlayfs
  xattr).

Changes

• In this release, the primary development branch was renamed to main.
• The runtime-spec version of the config.json version we generate is no
  longer hard-coded to 1.0.0. We now use the version of the spec we have
  imported (with any -dev suffix stripped, as such a prefix causes havoc with
  verification tools -- ideally we would only ever use released versions of the
  spec but that's not always possible). #452
• Add the cgroup namespace to the default configuration generated by umoci unpack to make sure that our configuration plays nicely with runc when on
  cgroupv2 systems.
• umoci has been migrated away from github.com/pkg/errors to Go stdlib error
  wrapping.
• The gzip compression block size has been updated to be more friendly with
  Docker and other tools that might round-trip the layer blob data (causing the
  hash to change if the block size is different). #509

Fixed

• In 0.4.7, a performance regression was introduced as part of the
  VerifiedReadCloser hardening work (to read all trailing bytes) which would
  cause walk operations on images to hash every blob in the image (even blobs
  which we couldn't parse and thus couldn't recurse into). To resolve this, we
  no longer recurse into unparseable blobs. #373 #375 #394
• Handle EINTR on io.Copy operations. Newer Go versions have added more
  opportunistic pre-emption which can cause EINTR errors in io paths that
  didn't occur before. #437
• Quite a few changes were made to CI to try to avoid issues with fragility.
  #452
• umoci will now return an explicit error if you pass invalid uid or gid values
  to --uid-map and --gid-map rather than silently truncating the value.
• For Go users of umoci, GenerateLayer (but not GenerateInsertLayer) with
  OverlayfsRootfs (called TranslateOverlayWhiteouts in umoci
  0.4.7) had several severe bugs that made the feature unusable:
  • All OCI whiteouts added to the archive would incorrectly have the full host
    name of the path rather than the correctly rooted path, making the whiteout
    practically useless.
  • Any non-whiteout files would not be included in the layer, making the layer
    data incomplete and thus resulting in silent data loss.
    Given how severe these bugs were and the lack of bug reports of this issue in
    the past 4 years, it seems this feature has not really been used by anyone (I
    hope...).
• For Go users of umoci, UnpackLayer now correctly handles several aspects of
  OverlayfsRootfs (OverlayFSWhiteout in umoci 0.4.7) extraction
  that weren't handled correctly:
  • Unlike regular extractions, overlayfs-style extractions require us to
    create the parent directory of the whiteout (rather than ignoring or
    assuming the underlying path exists) because the whiteout is being created
    in a separate layer to the underlying file. We also need to make sure that
    opaque whiteout targets are directories.
  • trusted.overlay.opaque=y has very peculiar behaviour when a regular
    whiteout (i.e. mknod c 0 0) is placed inside an opaque directory -- the
    whiteout-ed file appears in readdir but the file itself doesn't exist. To
    avoid this confusion (and possible information leak), umoci will no longer
    extract plain whiteouts within an opaque whiteout directory in the same
    layer. (As per the OCI spec requirements, this is regardless of the order
    of the opaque whiteout and the regular whiteout in the layer archive.)
• UnpackLayer and Generate(Insert)Layer now correctly handle
  trusted.overlay.* xattr escaping when extracting and generating layers with
  the overlayfs on-disk format. This escaping feature has been supported by
  overlayfs since Linux 6.7, and
  allows for you to created images that contain an overlayfs layout inside the
  image (nested to arbitrary levels).
  • If an image contains trusted.overlay.* xattrs, UnpackLayer will
    rewrite the xattrs to instead be in the trusted.overlay.overlay.*
    namespace, so that when merged using overlayfs the user will see the
    expected xattrs.
  • If an on-disk overlayfs directory used with Generate(Insert)Layer
    contains escaped trusted.overlay.overlay.* xattrs, they will be rewritten
    so that the generated layer contains trusted.overlay.* xattrs. If we
    encounter an unescaped trusted.overlay.* xattr they will not be included
    in the image (though they may cause the file to be converted to a whiteout
    in the image) because they are considered to be an internal aspect of the
    host on-disk format (i.e. trusted.overlay.origin might be automatically
    set by whatever tool is using the overlayfs layers).
    Note that in the regular extraction mode, these xattrs will be treated like
    any other xattrs (this is in contrast to the previous behaviour where they
    would be silently ignored regardless of the on-disk format being used).
• When extracting a layer, umoci unpack would previously return an error if a
  tar entry was within a non-directory. In practice such cases are quite
  unlikely (as layer diffs would usually include an entry changing the type of
  the non-directory parent) but this could result in spurious errors with
  somewhat non-standard tar archive layers. Now, umoci will remove the
  offending non-directory parent component and re-create the parent path as a
  proper directory tree.
  • This also has the side-effect of fixing the behaviour when unpacking
    whiteouts with the OverlayfsRootfs on-disk format. If there is a plain
    whiteout of a regular directory, followed by parent components being made
    underneath that directory, then the directory should be converted to an
    opaque whiteout. This matches the behaviour of overlayfs (though again, it
    seems unlikely that a layer diff tool would generate such a layer).
    #546

Thanks to all of the following contributors for making this release
possible:

• AdamKorcz adam@adalogics.com
• Aleksa Sarai cyphar@cyphar.com
• Bibhas github@bibhasdn.com
• Cameron Nemo cnemo@tutanota.com
• Michael McCracken mikmccra@cisco.com
• Ramkumar Chinchani rchincha@cisco.com
• Serge Hallyn serge@hallyn.com
• Shengjing Zhu i@zhsj.me
• Tycho Andersen tycho@tycho.pizza
• guoguangwu guoguangwu@magic-shield.com

Signed-off-by: Aleksa Sarai cyphar@cyphar.com