magika

• Pin onnxruntime on Windows (#1099).
• Add docstrings for all relevant modules, classes, and methods.
• Improved READMEs and overall documentation.

• Mark python 3.13 as supported.
• New model standard_v3_3 model, with better support for TypeScript and non-ascii characters in textual files. See models' CHANGELOG for more information.
• identify_stream() now restores the stream's original position after reading from it, preventing side effects on subsequent stream operations. (#1020)
• Add asdict() utility method to MagikaResult.
• Set prediction.overwrite_reason to Overwrite.NONE if output.label is the same as dl.label. (#1023)
• Bugfix: limit the number of bytes we read in case of an input with just many whitespaces. (#1015)
• Bugfix: do not alter warnings' simplefilter as this has visible side effects for other modules. (#1017)
• Bugfix: magika's python client now properly warns for low-confidence predictions.
• Bugfix: update Magika's StrEnum string representation to be compatible with standard library.

Overview

Magika v0.6.1 is a significant update featuring a new model with 2x supported content types, a new command line client in Rust, performance improvements, API enhancements, and a few breaking changes. This changelog entry rolls up all changes from v0.5.1, the last stable release.

[!IMPORTANT] There are a few breaking changes! After reading about the new key features and improvements, we suggest to consult the migration guide below and the updated documentation.

Key Features and Improvements

• New deep learning model: We introduce a new model, standard_v3_2, which supports 2x content types (200+ in total, see full list here), has a similar ~99% average accuracy, and is ~20% faster, with an inference speed of about ~2ms on CPUs (YMMV depending on your testing setup). See models' CHANGELOG for more information.
• New command line client, written in Rust: We developed a new command line client, written in Rust, which is not affected by the one-time bootstrap overhead caused by the python's interpreter itself. This new client is packaged, pre-compiled, into the magika python package. This new client replaces the old client written in Python (but the old Python one is still available as a fallback for those platforms for which we don't have precompiled rust binaries).
• New stream-based identification: Added identify_stream(stream: typing.BinaryIO) API to infer content types from open binary streams. (#970)
• Improved path handling: identify_path and identify_paths now accept Union[str, os.PathLike] objects. You no longer need to explicitly use pathlib.Path. (#935)
• Improved python API: The new Python APIs offer a number of improvements, including: the inference APIs now return a MagikaResult, which is a absl::StatusOr-like object that wraps MagikaPrediction, with a clear separation between valid predictions and error situations; the output content types (label) are not just str anymore, but of type ContentTypeLabel, making integrations more robust (ContentTypeLabel extends StrEnum: thus, they are not just str, but you can treat them as such). The MagikaPrediction object now has additional is_text and extensions fields (in addition to the existing label, mime_type, group, and description).
• New debugging APIs: Added new APIs to ease debugging and introspection, such as get_output_content_types(), get_model_content_types(), get_module_version(), and get_model_name().

Breaking Changes and Migration Guide

This release introduces several breaking changes. Please review this guide carefully to update your code:

1. New identify_* API output format: The inference Python APIs now return a MagikaResult object, which is similar to absl::StatusOr; This provides a cleaner way to handle errors. dl.ct_label and output.ct_label are renamed to dl.label and output.label. labels are now of type ContentTypeLabel, which extends StrEnum (thus, they are not just str, but you can treat them as such). The score field is now at the top level, alongside dl and output. The magic field has been removed as it was often either incorrect or redundant; use description instead.

• Before (v0.5.x and earlier):

  import magika
  
  m = magika.Magika()
  result = m.identify_path("my_file.py")
  print(result.output.ct_label)  # Assumed success
  

• After (v0.6.1):

  import magika
  
  m = magika.Magika()
  result = m.identify_path("my_file.py")
  if result.ok():
      print(result.output.label)
  else:
      print(f"Error: {result.status}")
  

1. CLI Output Format Change (v0.6.0): The JSON output format of the CLI has changed. Those changes are analogous to the changes to the Python APIs. The score field is now at the top level, alongside dl and output, and is no longer nested within dl or output. The output also includes is_text and extensions fields. The magic metadata has been removed as it was often either incorrect or redundant; use description instead. Moreover, similarly to what happens under the hood with the StatusOr pattern, result.status indicates whether the prediction was successful, and the prediction results are available under the result.value key.

• Before (v0.5.x and earlier): (Illustrative example - adapt to your specific output)

  {
    "path": "code.py",
    "dl": {
      "ct_label": "python",
      "score": 0.9940916895866394,
      "group": "code",
      "mime_type": "text/x-python",
      "magic": "Python script",
      "description": "Python source"
    },
    "output": {
      "ct_label": "python",
      "score": 0.9940916895866394,
      "group": "code",
      "mime_type": "text/x-python",
      "magic": "Python script",
      "description": "Python source"
    }
  }
  

• After (v0.6.1):

  {
    "path": "code.py",
    "result": {
      "status": "ok",
      "value": {
        "dl": {
          "description": "Python source",
          "extensions": ["py", "pyi"],
          "group": "code",
          "is_text": true,
          "label": "python",
          "mime_type": "text/x-python"
        },
        "output": {
          "description": "Python source",
          "extensions": ["py", "pyi"],
          "group": "code",
          "is_text": true,
          "label": "python",
          "mime_type": "text/x-python"
        },
        "score": 0.9890000224113464
      }
    }
  }
  

1. dl.label == ContentTypeLabel.UNDEFINED when the model is not used: There are situations in which the deep learning model is not used, for example when the file is too small or empty. In these cases, dl.label is now set to ContentTypeLabel.UNDEFINED instead of having the full dl block being set to None.

• Before (v0.5.x and earlier):

  # ... (assuming successful result)
  if prediction.dl is not None:
      print(prediction.dl.ct_label)
  

• After (v0.6.1):

  # ... (assuming successful result)
  if prediction.dl.label != magika.ContentTypeLabel.UNDEFINED:
      print(prediction.dl.label)
  

1. Expanded List of Content Types: The model now supports over 200 content types.

• Migration: Review the updated list of supported content types and adjust any code that relies on specific content type labels returned by previous versions. Labels have not changed, but a file previously detected as javascript may not be detected as typescript. Consider using get_output_content_types() to dynamically retrieve the supported labels.

1. Pure Python Wheel and Rust Client Fallback: If you are installing Magika on a platform without pre-built wheels (e.g., Windows on ARM), you will automatically get the pure-python wheel. In this case, the package does not include the Rust binary client, but it does include the old python client as fallback; you can use such old python client with $ magika-python-client.

Full Changelog

For a detailed list of all changes, including those from the -rc releases, please refer to the individual changelog entries for each release candidate:

• 0.6.1-rc3
• 0.6.1-rc2
• 0.6.1-rc1
• 0.6.1-rc0
• 0.6.0-rc3
• 0.6.0-rc2
• 0.6.0-rc1

• Add support for python 3.12. Magika now supports python >=3.8 and <3.13.
• Fix bugs for features extraction to cover more corner cases.
• Remove MIME types from table of supported content types (Relevant for --list-output-content-types; see FAQs for context).
• Refactor features extraction around a Seekable abstraction; we now have only one reference implementation.
• Start groundwork for v2 of features extraction.
• Various clean ups and internal refactors.

• New public python APIs: identify_paths, identify_path, identify_bytes.
• The APIs now return a MagikaResult object.
• When the model's prediction has low confidence and we return a generic content type, print anyways (with a disclaimer) the model's best guess.
• Updated description for "unknown" to "Unknown binary data".
• Magika CLI now defaults to "high-confidence" mode. "default" mode is now called "medium-confidence".
• Magika CLI -p/--output-probability has been renamed to -s/--output-score for consistency.
• Default model is now called standard_v1.
• Major refactoring and clean up.

• First release.