Bug fixes

• Small fix for NTXentLoss with no negative pairs #272
• Fixed .detach() bug in NTXentLoss #282
• Fixed parameter override bug in MatchFinder.get_matching_pairs() #286 by @joaqo

New features and improvements

AccuracyCalculator now uses torch instead of numpy

• All the calculations (except for NMI and AMI) are done with torch. Calculations will be done on the same device and dtype as the input query tensor.
• You can still pass numpy arrays into AccuracyCalculator.get_accuracy, but the arrays will be immediately converted to torch tensors.

Faster custom label comparisons in AccuracyCalculator

• See #264 by @mlopezantequera

Numerical stability improvement for DistanceWeightedMiner

See #278 by @z1w

UniformHistogramMiner

This is like DistanceWeightedMiner, except that it works well with high dimension embeddings, and works with any distance metric (not just L2 normalized distance). Documentation

PerAnchorReducer

This converts unreduced pairs to unreduced elements. For example, NTXentLoss returns losses per positive pair. If you used PerAnchorReducer with NTXentLoss, then the losses per pair would first be converted to losses per batch element, before being passed to the inner reducer. See the documentation

BaseTester no longer converts embeddings from torch to numpy

This includes the get_all_embeddings function. If you want get_all_embeddings to return numpy arrays, you can set the return_as_numpy flag to True:

embeddings, labels = tester.get_all_embeddings(dataset, model, return_as_numpy=True)

The embeddings are converted to numpy only for the visualizer and visualizer_hook, if specified.

Reduced usage of .to(device) and .type(dtype)

Tensors are initialized on device and with the necessary dtype, and they are moved to device and cast to dtypes only when necessary. See this code snippet for details.

Simplified DivisorReducer

Replaced "divisor_summands" with "divisor".

New Features

Thanks to @mlopezantequera for adding the following features!

Testers: allow any combination of query and reference sets (#250)

To evaluate different combinations of query and reference sets, use the splits_to_eval argument for tester.test().

For example, let's say your dataset_dict has two keys: "dataset_a" and "train".

• The default splits_to_eval = None is equivalent to:

splits_to_eval = [('dataset_a', ['dataset_a']), ('train', ['train'])]

• dataset_a as the query, and train as the reference:

splits_to_eval = [('dataset_a', ['train'])]

• dataset_a as the query, and dataset_a + train as the reference:

splits_to_eval = [('dataset_a', ['dataset_a', 'train'])]

Then pass splits_to_eval to tester.test:

tester.test(dataset_dict, epoch, model, splits_to_eval = splits_to_eval)

Note that this new feature makes the old reference_set init argument obsolete, so reference_set has been removed.

AccuracyCalculator: allow arbitrary label comparion functions (#254)

AccuracyCalculator now has an optional init argument, label_comparison_fn, which is a function that compares two numpy arrays of labels and returns a boolean array. The default is numpy.equal. If a custom function is used, then you must exclude clustering based metrics ("NMI" and "AMI"). The following is an example of a custom function for two-dimensional labels. It returns True if the 0th column matches, and the 1st column does not match:

def example_label_comparison_fn(x, y):
    return (x[:, 0] == y[:, 0]) & (x[:, 1] != y[:, 1])

AccuracyCalculator(exclude=("NMI", "AMI"), 
                    label_comparison_fn=example_label_comparison_fn)

Other Changes

• BaseTrainer and BaseTester now take in an optional dtype argument. This is the type that the dataset output will be converted to, e.g. torch.float16. If set to the default value of None, then no type casting will be done.
• Removed self.dim_reduced_embeddings from BaseTester and the associated code in HookContainer, due to lack of use.
• tester.test() now returns all_accuracies, whereas before, it returned nothing and you'd have to access all_accuracies either through the end_of_testing_hook or by accessing tester.all_accuracies.
• tester.embeddings_and_labels is deleted at the end of tester.test() to free up memory.

New

BatchEasyHardMiner

This new miner is an implementation of Improved Embeddings with Easy Positive Triplet Mining. See the documentation. Thanks @marijnl!

New metric added to AccuracyCalculator

The new metric is mean_average_precision, which is the commonly used k-nn based mAP in information retrieval. Note that this differs from the already existing metric, mean_average_precision_at_r.

Bug fixes

• dtype casting in MultiSimilarityMiner changed to work with autocast. See #233 by @thinline72
• Added logic for dealing with zero rows in the weight matrix in DistanceWeightedMiner by ignoring them. For example, if the entire weight matrix is 0, then no triplets will be returned. Previously, the zero rows would cause a RuntimeError. See #230 by @tpanum

Various bug fixes and improvements

• A list or dictionary of miners can be passed into MultipleLosses. #212
• Fixed bug where MultipleLosses failed in list mode. #213
• Fixed bug where IntraPairVarianceLoss and MarginLoss were overriding sub_loss_names instead of _sub_loss_names. This likely caused embedding regularizers to have no effect for these two losses. #215
• ModuleWithRecordsAndReducer now creates copies of the input reducer when necessary. #216
• Moved cos.clone() inside torch.no_grad() in RegularFaceRegularizer. Should be more efficient? #219
• In utils.inference, moved faiss import inside of FaissIndexer since that is the only class that requires it. #222
• Added a copy_weights init argument to LogitGetter, to make copying optional #223

Small update

• Optimized get_random_triplet_indices, so if you were using DistanceWeightedMiner, or if you ever set the triplets_per_anchor argument to something other than "all" anywhere in your code, it should run a lot faster now. Thanks @AlexSchuy

New Features

DistributedLossWrapper and DistributedMinerWrapper

Added DistributedLossWrapper and DistributedMinerWrapper. Wrap a loss or miner with these when using PyTorch's DistributedDataParallel (i.e. multiprocessing). Most of the code is by @JohnGiorgi (https://github.com/JohnGiorgi/DeCLUTR).

from pytorch_metric_learning import losses, miners
from pytorch_metric_learning.utils import distributed as pml_dist
loss_func = pml_dist.DistributedLossWrapper(loss = losses.ContrastiveLoss())
miner = pml_dist.DistributedMinerWrapper(miner = miners.MultiSimilarityMiner())

For a working example, see the "Multiprocessing with DistributedDataParallel" notebook.

Added enqueue_idx to CrossBatchMemory

Now you can make CrossBatchMemory work with MoCo. This adds a great deal of flexibility to the MoCo framework, because you can use any tuple loss and tuple miner in CrossBatchMemory.

Previously this wasn't possible because all embeddings passed into CrossBatchMemory would go into the memory queue. In contrast, MoCo only queues the momentum encoder's embeddings.

The new enqueue_idx argument lets you do this, by specifying which embeddings should be added to memory. Here's a modified snippet from the MoCo on CIFAR10 notebook:

from pytorch_metric_learning.losses import CrossBatchMemory, NTXentLoss

loss_fn = CrossBatchMemory(loss = NTXentLoss(), embedding_size = 64, memory_size = 16384)

### snippet from the training loop ###
for images, _ in train_loader:
  ...
  previous_max_label = torch.max(loss_fn.label_memory)
  num_pos_pairs = encQ_out.size(0)
  labels = torch.arange(0, num_pos_pairs)
  labels = torch.cat((labels , labels)).to(device)

  ### add an offset so that the labels do not overlap with any labels in the memory queue ###
  labels += previous_max_label + 1

  ### we want to enqueue the output of encK, which is the 2nd half of the batch ###
  enqueue_idx = torch.arange(num_pos_pairs, num_pos_pairs*2)

  all_enc = torch.cat([encQ_out, encK_out], dim=0)

  ### now only encK_out will be added to the memory queue ###
  loss = loss_fn(all_enc, labels, enqueue_idx = enqueue_idx)
  ...

Check out the MoCo on CIFAR10 notebook to see the entire script.

TuplesToWeightsSampler

This is a simple offline miner. It does the following:

1. Take a random subset of your dataset, if you provide subset_size
2. Use a specified miner to mine tuples from the subset dataset.
3. Compute weights based on how often an element appears in the mined tuples.
4. Randomly sample, using the weights as probabilities.

from pytorch_metric_learning.samplers import TuplesToWeightsSampler
from pytorch_metric_learning.miners import MultiSimilarityMiner

miner = MultiSimilarityMiner(epsilon=-0.2)
sampler = TuplesToWeightsSampler(model, miner, dataset, subset_size = 5000)
# then pass the sampler into your Dataloader

LogitGetter

Added utils.inference.LogitGetter to make it easier to compute logits of classifier loss functions.

from pytorch_metric_learning.losses import ArcFaceLoss
from pytorch_metric_learning.utils.inference import LogitGetter

loss_fn = ArcFaceLoss(num_classes = 100, embedding_size = 512)
LG = LogitGetter(loss_fn)
logits = LG(embeddings)

Other

• Added optional batch_size argument to MPerClassSampler. If you pass in this argument, then each batch is guaranteed to have m samples per class. Otherwise, most batches will have m samples per class, but it's not guaranteed for every batch. Note there restrictions on the values of m and batch_size. For example, batch_size must be a multiple of m. For all the restrictions, see the documentation.

• Added trainable_attributes to BaseTrainer and to standardize the set_to_train and set_to_eval functions.

• Added save_models init argument to HookContainer. If set to False then models will not be saved.

• Added losses_sizes as a stat for BaseReducer

• Added a type check and conversion in common_functions.labels_to_indices to go from torch tensor to numpy

Bug Fixes and Improvements

• Fixed CircleLoss bug, by improving the logsumexp keep_mask implementation. See https://github.com/KevinMusgrave/pytorch-metric-learning/issues/173
• Fixed convert_to_weights bug, which caused a runtime error when an empty indices_tuple was passed in. See https://github.com/KevinMusgrave/pytorch-metric-learning/issues/174
• ProxyAnchorLoss now adds miner weights to the exponents which are fed to logsumexp. This is equivalent to scaling each loss component by e^(miner_weight). The previous behavior was to scale each loss component by just miner_weight.

Other updates

• Added an example notebook which shows how to use a customized loss + miner in a simple training loop.
• A new arxiv paper and an improved Readme give a better high level explanation of the library.
• Added better explanations for the testers and the default accuracy metrics

********** Summary **********

The main update is the new distances module, which adds an extra level of modularity to loss functions. It is a pretty big design change, which is why so many arguments have become obsolete. See the documentation for a description of the new module.

Other updates include support for half-precision, new regularizers and mixins, improved documentation, and default values for most initialization parameters.

********** Breaking Changes **********

Dependencies

This library now requires PyTorch >= 1.6.0. Previously there was no explicit version requirement.

Losses and Miners

All loss functions

normalize_embeddings has been removed

• If you never used this argument, nothing needs to be done.
• normalize_embeddings = True: just remove the argument.
• normalize_embeddings = False: remove the argument and instead pass it into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(normalize_embeddings=False))

ContrastiveLoss, GenericPairLoss, BatchHardMiner, HDCMiner, PairMarginMiner

use_similarity has been removed

• If you never used this argument, nothing needs to be done.
• use_similarity = True: remove the argument and:

### if you had set normalize_embeddings = False ###
from pytorch_metric_learning.distances import DotProductSimilarity
loss_func = ContrastiveLoss(distance=DotProductSimilarity(normalize_embeddings=False))

#### otherwise ###
from pytorch_metric_learning.distances import CosineSimilarity
loss_func = ContrastiveLoss(distance=CosineSimilarity())

squared_distances has been removed

• If you never used this argument, nothing needs to be done.
• squared_distances = True: remove the argument and instead pass power=2 into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = ContrastiveLoss(distance=LpDistance(power=2))

• squared_distances = False: just remove the argument.

ContrastiveLoss, TripletMarginLoss

power has been removed

• If you never used this argument, nothing needs to be done.
• power = 1: just remove the argument
• power = X, where X != 1: remove the argument and instead pass it into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(power=2))

TripletMarginLoss

distance_norm has been removed

• If you never used this argument, nothing needs to be done.
• distance_norm = 2: just remove the argument
• distance_norm = X, where X != 2: remove the argument and instead pass it as p into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(p=1))

NPairsLoss

l2_reg_weight has been removed

• If you never used this argument, nothing needs to be done.
• l2_reg_weight = 0: just remove the argument
• l2_reg_weight = X, where X > 0: remove the argument and instead pass in an LpRegularizer and weight:

from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = NPairsLoss(embedding_regularizer=LpRegularizer(), embedding_reg_weight=0.123)

SignalToNoiseRatioContrastiveLoss

regularizer_weight has been removed

• If you never used this argument, nothing needs to be done.
• regularizer_weight = 0: just remove the argument
• regularizer_weight = X, where X > 0: remove the argument and instead pass in a ZeroMeanRegularizer and weight:

from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = SignalToNoiseRatioContrastiveLoss(embedding_regularizer=ZeroMeanRegularizer(), embedding_reg_weight=0.123)

SoftTripleLoss

reg_weight has been removed

• If you never used this argument, do the following to obtain the same default behavior:

from pytorch_metric_learning.regularizers import SparseCentersRegularizer
weight_regularizer = SparseCentersRegularizer(num_classes, centers_per_class)
SoftTripleLoss(..., weight_regularizer=weight_regularizer, weight_reg_weight=0.2)

• reg_weight = X: remove the argument, and use the SparseCenterRegularizer as shown above.

WeightRegularizerMixin and all classification loss functions

• If you never specified regularizer or reg_weight, nothing needs to be done.
• regularizer = X: replace with weight_regularizer = X
• reg_weight = X: replace with weight_reg_weight = X

Classification losses

• For all losses and miners, default values have been set for as many arguments as possible. This has caused a change in ordering in positional arguments for several of the classification losses. The typical form is now:

loss_func = SomeClassificatinLoss(num_classes, embedding_loss, <keyword arguments>)

See the documentation for specifics

Reducers

ThresholdReducer

threshold has been replaced by low and high

• Replace threshold = X with low = X

Regularizers

All regularizers

normalize_weights has been removed

• If you never used this argument, nothing needs to be done.
• normalize_weights = True: just remove the argument.
• normalize_weights = False: remove the argument and instead pass normalize_embeddings = False into a distance object. For example:

from pytorch_metric_learning.distances import DotProductSimilarity
loss_func = RegularFaceRegularizer(distance=DotProductSimilarity(normalize_embeddings=False))

Inference

MatchFinder

mode has been removed

• Replace mode="sim" with either distance=CosineSimilarity() or distance=DotProductSimilarity()
• Replace mode="dist" with distance=LpDistance()
• Replace mode="squared_dist" with distance=LpDistance(power=2)

********** New Features **********

Distances

Distances bring an additional level of modularity to building loss functions. Here's an example of how they work.

Consider the TripletMarginLoss in its default form:

from pytorch_metric_learning.losses import TripletMarginLoss
loss_func = TripletMarginLoss(margin=0.2)

This loss function attempts to minimize [d_ap - d_an + margin]₊.

In other words, it tries to make the anchor-positive distances (d_ap) smaller than the anchor-negative distances (d_an).

Typically, d_ap and d_an represent Euclidean or L2 distances. But what if we want to use a squared L2 distance, or an unnormalized L1 distance, or completely different distance measure like signal-to-noise ratio? With the distances module, you can try out these ideas easily:

### TripletMarginLoss with squared L2 distance ###
from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(margin=0.2, distance=LpDistance(power=2))

### TripletMarginLoss with unnormalized L1 distance ###
loss_func = TripletMarginLoss(margin=0.2, distance=LpDistance(normalize_embeddings=False, p=1))

### TripletMarginLoss with signal-to-noise ratio###
from pytorch_metric_learning.distances import SNRDistance
loss_func = TripletMarginLoss(margin=0.2, distance=SNRDistance())

You can also use similarity measures rather than distances, and the loss function will make the necessary adjustments:

### TripletMarginLoss with cosine similarity##
from pytorch_metric_learning.distances import CosineSimilarity
loss_func = TripletMarginLoss(margin=0.2, distance=CosineSimilarity())

With a similarity measure, the TripletMarginLoss internally swaps the anchor-positive and anchor-negative terms: [s_an - s_ap + margin]₊. In other words, it will try to make the anchor-negative similarities smaller than the anchor-positive similarities.

All losses, miners, and regularizers accept a distance argument. So you can try out the MultiSimilarityMiner using SNRDistance, or the NTXentLoss using LpDistance(p=1) and so on. Note that some losses/miners/regularizers have restrictions on the type of distances they can accept. For example, some classification losses only allow CosineSimilarity or DotProductSimilarity as their distance measure between embeddings and weights. To view restrictions for specific loss functions, see the documentation

There are four distances implemented (LpDistance, SNRDistance, CosineSimilarity, DotProductSimilarity), but of course you can extend the BaseDistance class and write a custom distance measure if you want. See the documentation for more.

EmbeddingRegularizerMixin

All loss functions now extend EmbeddingRegularizerMixin, which means you can optionally pass in (to any loss function) an embedding regularizer and its weight. The embedding regularizer will compute some loss based on the embeddings alone, ignoring labels and tuples. For example:

from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = MultiSimilarityLoss(embedding_regularizer=LpRegularizer(), embedding_reg_weight=0.123)

WeightRegularizerMixin is now a subclass of WeightMixin

As in previous versions, classification losses extend WeightRegularizerMixin, which which means you can optionally pass in a weight matrix regularizer. Now that WeightRegularizerMixin extends WeightMixin, you can also specify the weight initialization function in object form:

from ..utils import common_functions as c_f
import torch

# use kaiming_uniform, with a=1 and mode='fan_out'
weight_init_func = c_f.TorchInitWrapper(torch.nn.kaiming_uniform_, a=1, mode='fan_out')
loss_func = SomeClassificationLoss(..., weight_init_func=weight_init_func)

New Regularizers

For increased modularity, the regularizers hard-coded in several loss functions were separated into their own classes. The new regularizers are:

• LpRegularizer
• SparseCentersRegularizer
• ZeroMeanRegularizer

Support for half-precision

In previous versions, various functions would break in half-precision (float16) mode. Now all distances, losses, miners, regularizers, and reducers work with half-precision, float32, and double (float64).

New collect_stats argument

All distances, losses, miners, regularizers, and reducers now have a collect_stats argument, which is True by default. This means that various statistics are collected in each forward pass, and these statistics can be useful to look at during experiments. However, if you don't care about collecting stats, you can set collect_stats=False, and the stat computations will be skipped.

Other updates

• You no longer have to explicitly call .to(device) on classification losses, because their weight matrices will be moved to the correct device during the forward pass if necessary. See issue https://github.com/KevinMusgrave/pytorch-metric-learning/issues/139

• Reasonable default values have been set for all losses and miners, to make these classes easier to try out. In addition, equations have been added to many of the class descriptions in the documentation. See issue https://github.com/KevinMusgrave/pytorch-metric-learning/issues/140

• Calls to torch.nonzero have been replaced by torch.where.

• The documentation for ArcFaceLoss and CosFaceLoss have been fixed to reflect the actual usage. (The documentation previously indicated that some arguments are positional, when they are actually keyword arguments.)

• The tensorboard_folder argument for utils.logging_presets.get_record_keeper is now optional. If you don't specify it, then there will be no tensorboard logs, which can be useful if speed is a concern.

• The loss dictionary in BaseTrainer is now cleared at the end of each epoch, to free up GPU memory. See issue https://github.com/KevinMusgrave/pytorch-metric-learning/issues/171

CrossBatchMemory

• Fixed bug where CrossBatchMemory would use self-comparisons as positive pairs. This was uniquely a CrossBatchMemory problem because of the nature of adding each current batch to the queue.
• Fixed bug where DistanceWeightedMiner would not work with CrossBatchMemory due to missing ref_label
• Changed 3rd keyword argument of forward() from input_indices_tuple to indices_tuple to be consistent with all other losses.

AccuracyCalculator

• Fixed bug in AccuracyCalculator where it would return NaN if the reference set contained none of query set labels. Now it will log a warning and return 0.

BaseTester

• Fixed bug where "compared_to_training_set" mode of BaseTester fails due to list(None) bug.

InferenceModel

• New get_nearest_neighbors function will return nearest neighbors of a query. By @btseytlin

Loss and miner utils

• Switched to fill_diagonal_ in the get_all_pairs_indices and get_all_triplets_indices code, instead of creating torch.eye.

Bug fix

Removed the circular import which caused an ImportError when the reducers module was imported before anything else. See #125

v0.9.87 comes with some major changes that may cause your existing code to break.

BREAKING CHANGES

Losses

• The avg_non_zero_only init argument has been removed from ContrastiveLoss, TripletMarginLoss, and SignalToNoiseRatioContrastiveLoss. Here's how to translate from old to new code:
  • avg_non_zero_only=True: Just remove this input parameter. Nothing else needs to be done as this is the default behavior.
  • avg_non_zero_only=False: Remove this input parameter and replace it with reducer=reducers.MeanReducer(). You'll need to add this to your imports: from pytorch_metric_learning import reducers
• learnable_param_names and num_class_per_param has been removed from BaseMetricLossFunction due to lack of use.
  • MarginLoss is the only built-in loss function that is affected by this. Here's how to translate from old to new code:
    • learnable_param_names=["beta"]: Remove this input parameter and instead pass in learn_beta=True.
    • num_class_per_param=N: Remove this input parameter and instead pass in num_classes=N.

AccuracyCalculator

• The average_per_class init argument is now avg_of_avgs. The new name better reflects the functionality.
• The old way to import was: from pytorch_metric_learning.utils import AccuracyCalculator. This will no longer work. The new way is: from pytorch_metric_learning.utils.accuracy_calculator import AccuracyCalculator. The reason for this change is to avoid an unnecessary import of the Faiss library, especially when this library is used in other packages.

New feature: Reducers

Reducers specify how to go from many loss values to a single loss value. For example, the ContrastiveLoss computes a loss for every positive and negative pair in a batch. A reducer will take all these per-pair losses, and reduce them to a single value. Here's where reducers fit in this library's flow of filters and computations:

Your Data --> Sampler --> Miner --> Loss --> Reducer --> Final loss value

Reducers are passed into loss functions like this:

from pytorch_metric_learning import losses, reducers
reducer = reducers.SomeReducer()
loss_func = losses.SomeLoss(reducer=reducer)
loss = loss_func(embeddings, labels) # in your training for-loop

Internally, the loss function creates a dictionary that contains the losses and other information. The reducer takes this dictionary, performs the reduction, and returns a single value on which .backward() can be called. Most reducers are written such that they can be passed into any loss function.

See the documentation for details.

Other updates

Utils

Inference

• InferenceModel has been added to the library. It is a model wrapper that makes it convenient to find matching pairs within a batch, or from a set of pairs. Take a look at this notebook to see example usage.

AccuracyCalculator

• The k value for k-nearest neighbors can optionally be specified as an init argument.
• k-nn based metrics now receive knn distances in their kwargs. See #118 by @marijnl

Other stuff

Unit tests were added for almost all losses, miners, regularizers, and reducers.

Bug fixes

Trainers

• Fixed a labels related bug in TwoStreamMetricLoss. See #112 by @marijnl

Loss and miner utils

• Fixed bug where convert_to_triplets could encounter a RuntimeError. See #95

Losses + miners

• Added assertions to make sure the number of input embeddings is equal to the number of input labels.
• MarginLoss
  • Fixed bug where loss explodes if self.nu > 0 and number of active pairs is 0. See https://github.com/KevinMusgrave/pytorch-metric-learning/issues/98#issue-618347291

Trainers

• Added freeze_these to the init arguments of BaseTrainer. This optional argument takes a list or tuple of strings as input. The strings must correspond to the names of models or loss functions, and these models/losses will have their parameters frozen during training. Their corresponding optimizers will also not be stepped.
• Fixed indices shifting bug in the TwoStreamMetricLoss trainer. By @marijnl

Testers

• BaseTester
  • Pass in epoch to visualizer_hook
  • Added eval option to get_all_embeddings. By default it is True, and will set the input trunk and embedder to eval() mode.

Utils

• HookContainer
  • Allow training to resume from best model, rather than just the latest model.
• The best models are now saved as <model_name>_best<epoch>.pth rather than <model_name>_best.pth. To easily get the new suffix for loading the best model you can do:

from pytorch_metric_learning.utils import common_functions as c_f
_, best_model_suffix = c_f.latest_version(your_model_folder, best=True)
best_trunk = "trunk_{}.pth".format(best_model_suffix)
best_embedder = "embedder_{}.pth".format(best_model_suffix)