pytorch-lightning

[1.3.7post0] - 2021-06-23

Fixed

• Fixed backward compatibility of moved functions rank_zero_warn and rank_zero_deprecation (#8085)

Contributors

@kaushikb11 @carmocca

[1.3.7] - 2021-06-22

Fixed

• Fixed a bug where skipping an optimizer while using amp causes amp to trigger an assertion error (#7975) This conversation was marked as resolved by carmocca
• Fixed deprecation messages not showing due to incorrect stacklevel (#8002, #8005)
• Fixed setting a DistributedSampler when using a distributed plugin in a custom accelerator (#7814)
• Improved PyTorchProfiler chrome traces names (#8009)
• Fixed moving the best score to device in EarlyStopping callback for TPU devices (#7959)

Contributors

@yifuwang @kaushikb11 @ajtritt @carmocca @tchaton

[1.3.6] - 2021-06-15

Fixed

• Fixed logs overwriting issue for remote filesystems (#7889)
• Fixed DataModule.prepare_data could only be called on the global rank 0 process (#7945)
• Fixed setting worker_init_fn to seed dataloaders correctly when using DDP (#7942)
• Fixed BaseFinetuning callback to properly handle parent modules w/ parameters (#7931)

Contributors

@awaelchli @Borda @kaushikb11 @Queuecumber @SeanNaren @senarvi @speediedan

[1.3.5] - 2021-06-08

Added

• Added warning to Training Step output (#7779)

Fixed

• Fixed LearningRateMonitor + BackboneFinetuning (#7835)
• Minor improvements to apply_to_collection and type signature of log_dict (#7851)
• Fixed docker versions (#7834)
• Fixed sharded training check for fp16 precision (#7825)
• Fixed support for torch Module type hints in LightningCLI (#7807)

Changed

• Move training_output validation to after train_step_end (#7868)

Contributors

@Borda, @justusschock, @kandluis, @mauvilsa, @shuyingsunshine21, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.3.4] - 2021-06-01

Fixed

• Fixed info message when max training time reached (#7780)
• Fixed missing __len__ method to IndexBatchSamplerWrapper (#7681)

Contributors

@awaelchli @kaushikb11

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.3.3] - 2021-05-26

Changed

• Changed calling of untoggle_optimizer(opt_idx) out of the closure function (#7563)

Fixed

• Fixed ProgressBar pickling after calling trainer.predict (#7608)
• Fixed broadcasting in multi-node, multi-gpu DDP using torch 1.7 (#7592)
• Fixed dataloaders are not reset when tuning the model (#7566)
• Fixed print errors in ProgressBar when trainer.fit is not called (#7674)
• Fixed global step update when the epoch is skipped (#7677)
• Fixed training loop total batch counter when accumulate grad batches was enabled (#7692)

Contributors

@carmocca @kaushikb11 @ryanking13 @Lucklyric @ajtritt @yifuwang

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.3.2] - 2021-05-18

Changed

• DataModules now avoid duplicate {setup,teardown,prepare_data} calls for the same stage (#7238)

Fixed

• Fixed parsing of multiple training dataloaders (#7433)
• Fixed recursive passing of wrong_type keyword argument in pytorch_lightning.utilities.apply_to_collection (#7433)
• Fixed setting correct DistribType for ddp_cpu (spawn) backend (#7492)
• Fixed incorrect number of calls to LR scheduler when check_val_every_n_epoch > 1 (#7032)

Contributors

@alanhdu @carmocca @justusschock @tkng

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.3.1] - 2021-05-11

Fixed

• Fixed DeepSpeed with IterableDatasets (#7362)
• Fixed Trainer.current_epoch not getting restored after tuning (#7434)
• Fixed local rank displayed in console log (#7395)

Contributors

@akihironitta @awaelchli @leezu

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Today we are excited to announce Lightning 1.3, containing highly anticipated new features including a new Lightning CLI, improved TPU support, integrations such as PyTorch profiler, new early stopping strategies, predict and validate trainer routines, and more.

https://medium.com/pytorch/pytorch-lightning-1-3-lightning-cli-pytorch-profiler-improved-early-stopping-6e0ffd8deb29

[1.3.0] - 2021-05-06

Added

• Added support for the EarlyStopping callback to run at the end of the training epoch (#6944)
• Added synchronization points before and after setup hooks are run (#7202)
• Added a teardown hook to ClusterEnvironment (#6942)
• Added utils for metrics to scalar conversions (#7180)
• Added utils for NaN/Inf detection for gradients and parameters (#6834)
• Added more explicit exception message when trying to execute trainer.test() or trainer.validate() with fast_dev_run=True (#6667)
• Added LightningCLI class to provide simple reproducibility with minimum boilerplate training CLI (#4492, #6862, #7156, #7299)
• Added gradient_clip_algorithm argument to Trainer for gradient clipping by value (#6123).
• Added a way to print to terminal without breaking up the progress bar (#5470)
• Added support to checkpoint after training steps in ModelCheckpoint callback (#6146)
• Added TrainerStatus.{INITIALIZING,RUNNING,FINISHED,INTERRUPTED} (#7173)
• Added Trainer.validate() method to perform one evaluation epoch over the validation set (#4948)
• Added LightningEnvironment for Lightning-specific DDP (#5915)
• Added teardown() hook to LightningDataModule (#4673)
• Added auto_insert_metric_name parameter to ModelCheckpoint (#6277)
• Added arg to self.log that enables users to give custom names when dealing with multiple dataloaders (#6274)
• Added teardown method to BaseProfiler to enable subclasses defining post-profiling steps outside of __del__ (#6370)
• Added setup method to BaseProfiler to enable subclasses defining pre-profiling steps for every process (#6633)
• Added no return warning to predict (#6139)
• Added Trainer.predict config validation (#6543)
• Added AbstractProfiler interface (#6621)
• Added support for including module names for forward in the autograd trace of PyTorchProfiler (#6349)
• Added support for the PyTorch 1.8.1 autograd profiler (#6618)
• Added outputs parameter to callback's on_validation_epoch_end & on_test_epoch_end hooks (#6120)
• Added configure_sharded_model hook (#6679)
• Added support for precision=64, enabling training with double precision (#6595)
• Added support for DDP communication hooks (#6736)
• Added artifact_location argument to MLFlowLogger which will be passed to the MlflowClient.create_experiment call (#6677)
• Added model parameter to precision plugins' clip_gradients signature (#6764, #7231)
• Added is_last_batch attribute to Trainer (#6825)
• Added LightningModule.lr_schedulers() for manual optimization (#6567)
• Added MpModelWrapper in TPU Spawn (#7045)
• Added max_time Trainer argument to limit training time (#6823)
• Added on_predict_{batch,epoch}_{start,end} hooks (#7141)
• Added new EarlyStopping parameters stopping_threshold and divergence_threshold (#6868)
• Added debug flag to TPU Training Plugins (PT_XLA_DEBUG) (#7219)
• Added new UnrepeatedDistributedSampler and IndexBatchSamplerWrapper for tracking distributed predictions (#7215)
• Added trainer.predict(return_predictions=None|False|True) (#7215)
• Added BasePredictionWriter callback to implement prediction saving (#7127)
• Added trainer.tune(scale_batch_size_kwargs, lr_find_kwargs) arguments to configure the tuning algorithms (#7258)
• Added tpu_distributed check for TPU Spawn barrier (#7241)
• Added device updates to TPU Spawn for Pod training (#7243)
• Added warning when missing Callback and using resume_from_checkpoint (#7254)
• DeepSpeed single file saving (#6900)
• Added Training type Plugins Registry (#6982, #7063, #7214, #7224)
• Add ignore param to save_hyperparameters (#6056)

Changed

• Changed LightningModule.truncated_bptt_steps to be property (#7323)
• Changed EarlyStopping callback from by default running EarlyStopping.on_validation_end if only training is run. Set check_on_train_epoch_end to run the callback at the end of the train epoch instead of at the end of the validation epoch (#7069)
• Renamed pytorch_lightning.callbacks.swa to pytorch_lightning.callbacks.stochastic_weight_avg (#6259)
• Refactor RunningStage and TrainerState usage (#4945, #7173)
  • Added RunningStage.SANITY_CHECKING
  • Added TrainerFn.{FITTING,VALIDATING,TESTING,PREDICTING,TUNING}
  • Changed trainer.evaluating to return True if validating or testing
• Changed setup() and teardown() stage argument to take any of {fit,validate,test,predict} (#6386)
• Changed profilers to save separate report files per state and rank (#6621)
• The trainer no longer tries to save a checkpoint on exception or run callback's on_train_end functions (#6864)
• Changed PyTorchProfiler to use torch.autograd.profiler.record_function to record functions (#6349)
• Disabled lr_scheduler.step() in manual optimization (#6825)
• Changed warnings and recommendations for dataloaders in ddp_spawn (#6762)
• pl.seed_everything will now also set the seed on the DistributedSampler (#7024)
• Changed default setting for communication of multi-node training using DDPShardedPlugin (#6937)
• trainer.tune() now returns the tuning result (#7258)
• LightningModule.from_datasets() now accepts IterableDataset instances as training datasets. (#7503)
• Changed resume_from_checkpoint warning to an error when the checkpoint file does not exist (#7075)
• Automatically set sync_batchnorm for training_type_plugin (#6536)
• Allowed training type plugin to delay optimizer creation (#6331)
• Removed ModelSummary validation from train loop on_trainer_init (#6610)
• Moved save_function to accelerator (#6689)
• Updated DeepSpeed ZeRO (#6546, #6752, #6142, #6321)
• Improved verbose logging for EarlyStopping callback (#6811)
• Run ddp_spawn dataloader checks on Windows (#6930)
• Updated mlflow with using resolve_tags (#6746)
• Moved save_hyperparameters to its own function (#7119)
• Replaced _DataModuleWrapper with __new__ (#7289)
• Reset current_fx properties on lightning module in teardown (#7247)
• Auto-set DataLoader.worker_init_fn with seed_everything (#6960)
• Remove model.trainer call inside of dataloading mixin (#7317)
• Split profilers module (#6261)
• Ensure accelerator is valid if running interactively (#5970)
• Disabled batch transfer in DP mode (#6098)

Deprecated

• Deprecated outputs in both LightningModule.on_train_epoch_end and Callback.on_train_epoch_end hooks (#7339)
• Deprecated Trainer.truncated_bptt_steps in favor of LightningModule.truncated_bptt_steps (#7323)
• Deprecated outputs in both LightningModule.on_train_epoch_end and Callback.on_train_epoch_end hooks (#7339)
• Deprecated LightningModule.grad_norm in favor of pytorch_lightning.utilities.grads.grad_norm (#7292)
• Deprecated the save_function property from the ModelCheckpoint callback (#7201)
• Deprecated LightningModule.write_predictions and LightningModule.write_predictions_dict (#7066)
• Deprecated TrainerLoggingMixin in favor of a separate utilities module for metric handling (#7180)
• Deprecated TrainerTrainingTricksMixin in favor of a separate utilities module for NaN/Inf detection for gradients and parameters (#6834)
• period has been deprecated in favor of every_n_val_epochs in the ModelCheckpoint callback (#6146)
• Deprecated trainer.running_sanity_check in favor of trainer.sanity_checking (#4945)
• Deprecated Profiler(output_filename) in favor of dirpath and filename (#6621)
• Deprecated PytorchProfiler(profiled_functions) in favor of record_functions (#6349)
• Deprecated @auto_move_data in favor of trainer.predict (#6993)
• Deprecated Callback.on_load_checkpoint(checkpoint) in favor of Callback.on_load_checkpoint(trainer, pl_module, checkpoint) (#7253)
• Deprecated metrics in favor of torchmetrics (#6505, #6530, #6540, #6547, #6515, #6572, #6573, #6584, #6636, #6637, #6649, #6659, #7131)
• Deprecated the LightningModule.datamodule getter and setter methods; access them through Trainer.datamodule instead (#7168)
• Deprecated the use of Trainer(gpus="i") (string) for selecting the i-th GPU; from v1.5 this will set the number of GPUs instead of the index (#6388)

Removed

• Removed the exp_save_path property from the LightningModule (#7266)
• Removed training loop explicitly calling EarlyStopping.on_validation_end if no validation is run (#7069)
• Removed automatic_optimization as a property from the training loop in favor of LightningModule.automatic_optimization (#7130)
• Removed evaluation loop legacy returns for *_epoch_end hooks (#6973)
• Removed support for passing a bool value to profiler argument of Trainer (#6164)
• Removed no return warning from val/test step (#6139)
• Removed passing a ModelCheckpoint instance to Trainer(checkpoint_callback) (#6166)
• Removed deprecated Trainer argument enable_pl_optimizer and automatic_optimization (#6163)
• Removed deprecated metrics (#6161)
  • from pytorch_lightning.metrics.functional.classification removed to_onehot, to_categorical, get_num_classes, roc, multiclass_roc, average_precision, precision_recall_curve, multiclass_precision_recall_curve
  • from pytorch_lightning.metrics.functional.reduction removed reduce, class_reduce
• Removed deprecated ModelCheckpoint arguments prefix, mode="auto" (#6162)
• Removed mode='auto' from EarlyStopping (#6167)
• Removed epoch and step arguments from ModelCheckpoint.format_checkpoint_name(), these are now included in the metrics argument (#7344)
• Removed legacy references for magic keys in the Result object (#6016)
• Removed deprecated LightningModule hparams setter (#6207)
• Removed legacy code to log or include metrics in the progress bar by returning them in a dict with the "log"/"progress_bar" magic keys. Use self.log instead (#6734)
• Removed trainer.fit() return value of 1. It has no return now (#7237)
• Removed logger_connector legacy code (#6733)
• Removed unused mixin attributes (#6487)

Fixed

• Fixed NaN errors in progress bars when training with iterable datasets with no length defined (#7306)
• Fixed attaching train and validation dataloaders when reload_dataloaders_every_epoch=True and num_sanity_val_steps=0 (#7207)
• Added a barrier in the accelerator teardown to synchronize processes before execution finishes (#6814)
• Fixed multi-node DDP sub-process launch by using local_rank instead of global_rank for main process assertion (#7061)
• Fixed incorrect removal of WORLD_SIZE environment variable in DDP training when launching with torch distributed/torchelastic (#6942)
• Made the Plugin.reduce method more consistent across all Plugins to reflect a mean-reduction by default (#6011)
• Move lightning module to correct device type when using LightningDistributedWrapper (#6070)
• Do not print top-k verbose log with ModelCheckpoint(monitor=None) (#6109)
• Fixed ModelCheckpoint(save_top_k=0, save_last=True) not saving the last checkpoint (#6136)
• Fixed .teardown(stage='fit') and .on_fit_{start,end}() getting called during trainer.test (#6386)
• Fixed LightningModule all_gather on cpu tensors (#6416)
• Fixed torch distributed not available in setup hook for DDP (#6506)
• Fixed trainer.tuner.{lr_find,scale_batch_size} not setting the Trainer state properly (#7258)
• Fixed bug where the learning rate schedulers did not follow the optimizer frequencies (#4868)
• Fixed pickle error checker to now check for pickle.PickleError to catch all pickle errors (#6917)
• Fixed a bug where the outputs object passed to LightningModule.training_epoch_end was different from the object passed to the on_train_end_epoch hook (#6969)
• Fixed a bug where the outputs passed to train_batch_end would be listed even when using a single optimizer and no truncated backprop through time steps (#6969)
• Fixed bug for trainer error handling which would cause hang for distributed training (#6864)
• Fixed self.device not returning the correct device in replicas of data-parallel (#6414)
• Fixed lr_find trying beyond num_training steps and suggesting a too high learning rate (#7076)
• Fixed logger creating incorrect version folder in DDP with repeated Trainer.fit calls (#7077)
• Fixed metric objects passed directly to self.log not being reset correctly (#7055)
• Fixed CombinedLoader in distributed settings for validation / testing (#7102)
• Fixed the save_dir in WandbLogger when the run was initiated externally (#7106)
• Fixed num_sanity_val_steps affecting reproducibility of training data shuffling (#7014)
• Fixed resetting device after fitting/evaluating/predicting (#7188)
• Fixed bug where trainer.tuner.scale_batch_size(max_trials=0) would not return the correct batch size result (#7262)
• Fixed metrics not being properly logged with precision=16 and manual_optimization (#7228)
• Fixed BaseFinetuning properly reloading optimizer_states when using resume_from_checkpoint (#6891)
• Fixed parameters_to_ignore not properly set to DDPWrapper (#7239)
• Fixed parsing of fast_dev_run=True with the built-in ArgumentParser (#7240)
• Fixed handling an IterableDataset that fails to produce a batch at the beginning of an epoch (#7294)
• Fixed LightningModule.save_hyperparameters() when attempting to save an empty container (#7268)
• Fixed apex not properly instantiated when running with ddp (#7274)
• Fixed optimizer state not moved to GPU (#7277)
• Fixed custom init args for WandbLogger (#6989)
• Fixed a bug where an error would be raised if the train dataloader sometimes produced None for a batch (#7342)
• Fixed examples (#6600, #6638, #7096, #7246, #6357, #6476, #6294, #6373, #6088, #7398)
• Resolved schedule step bug for PyTorch Profiler (#6674, #6681)
• Updated logic for checking TPUs availability (#6767)
• Resolve TPU miss rendezvous (#6781)
• Fixed auto-scaling mode when calling tune method on trainer (#7321)
• Fixed finetuning complex models correctly unfreezes (#6880)
• Ensure we set the eval/train flag correctly on accelerator model (#6877)
• Set better defaults for rank_zero_only.rank when training is launched with SLURM and torchelastic (#6802)
• Fixed matching the number of outputs of backward with forward for AllGatherGrad (#6625)
• Fixed the gradient_clip_algorithm has no effect (#6928)
• Fixed CUDA OOM detection and handling (#6934)
• Fixed unfreeze_and_add_param_group expects modules rather than module (#6822)
• Fixed DPP + SyncBN when move on device (#6838)
• Fixed missing arguments in lr_find call (#6784)
• Fixed set_default_tensor_type to torch.DoubleTensor with precision=64 (#7108)
• Fixed NeptuneLogger.log_text(step=None) (#7194)
• Fixed importing torchtext batch (#6365, #6323, #6211)

Contributors

@akihironitta, @alessiobonfiglio, @amisev, @amogkam, @ananthsub, @ArvinZhuang, @ashleve, @asnorkin, @awaelchli, @BloodAxe, @bmahlbrand, @Borda, @borisdayma, @camruta, @carmocca, @ceshine, @dbonner, @dhkim0225, @EdwardJB, @EliaCereda, @EricCousineau-TRI, @ethanwharris, @FlorianMF, @hemildesai, @ifsheldon, @kaushikb11, @mauvilsa, @maxfrei750, @mesejo, @ramonemiliani93, @rohitgr7, @s-rog, @sadiqj, @scart97, @SeanNaren, @shuyingsunshine21, @SkafteNicki, @SpontaneousDuck, @stllfe, @tchaton, @THasthika, @vballoli

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Fixing missing packaging package in dependencies, which was affecting the only installation to a very blank system.

[1.2.9] - 2021-04-20

Fixed

• Fixed the order to call for world ranks & the root_device property in TPUSpawnPlugin (#7074)
• Fixed multi-gpu join for Horovod (#6954)
• Fixed parsing for pre-release package versions (#6999)

Contributors

@irasit @Borda @kaushikb11

[1.2.8] - 2021-04-14

Added

• Added TPUSpawn + IterableDataset error message (#6875)

Fixed

• Fixed process rank not being available right away after Trainer instantiation (#6941)
• Fixed sync_dist for tpus (#6950)
• Fixed AttributeError for require_backward_grad_sync` when running manual optimization with sharded plugin (#6915)
• Fixed --gpus default for parser returned by Trainer.add_argparse_args (#6898)
• Fixed TPU Spawn all gather (#6896)
• Fixed EarlyStopping logic when min_epochs or min_steps requirement is not met (#6705)
• Fixed csv extension check (#6436)
• Fixed checkpoint issue when using Horovod distributed backend (#6958)
• Fixed tensorboard exception raising (#6901)
• Fixed setting the eval/train flag correctly on accelerator model (#6983)
• Fixed DDP_SPAWN compatibility with bug_report_model.py (#6892)
• Fixed bug where BaseFinetuning.flatten_modules() was duplicating leaf node parameters (#6879)
• Set better defaults for rank_zero_only.rank when training is launched with SLURM and torchelastic:
  • Support SLURM and torchelastic global rank environment variables (#5715)
  • Remove hardcoding of local rank in accelerator connector (#6878)

Contributors

@ananthsub @awaelchli @ethanwharris @justusschock @kandluis @kaushikb11 @liob @SeanNaren @skmatz

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.7] - 2021-04-06

Fixed

• Fixed resolve a bug with omegaconf and xm.save (#6741)
• Fixed an issue with IterableDataset when len is not defined (#6828)
• Sanitize None params during pruning (#6836)
• Enforce an epoch scheduler interval when using SWA (#6588)
• Fixed TPU Colab hang issue, post training (#6816])
• Fixed a bug where TensorBoardLogger would give a warning and not log correctly to a symbolic link save_dir (#6730)

Contributors

@awaelchli, @ethanwharris, @karthikprasad, @kaushikb11, @mibaumgartner, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.6] - 2021-03-30

Changed

• Changed the behavior of on_epoch_start to run at the beginning of validation & test epoch (#6498)

Removed

• Removed legacy code to include step dictionary returns in callback_metrics. Use self.log_dict instead. (#6682)

Fixed

• Fixed DummyLogger.log_hyperparams raising a TypeError when running with fast_dev_run=True (#6398)
• Fixed error on TPUs when there was no ModelCheckpoint (#6654)
• Fixed trainer.test freeze on TPUs (#6654)
• Fixed a bug where gradients were disabled after calling Trainer.predict (#6657)
• Fixed bug where no TPUs were detected in a TPU pod env (#6719)

Contributors

@awaelchli, @carmocca, @ethanwharris, @kaushikb11, @rohitgr7, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.5] - 2021-03-23

Changed

• Added Autocast in validation, test and predict modes for Native AMP (#6565)
• Update Gradient Clipping for the TPU Accelerator (#6576)
• Refactored setup for typing friendly (#6590)

Fixed

• Fixed a bug where all_gather would not work correctly with tpu_cores=8 (#6587)
• Fixed comparing required versions (#6434)
• Fixed duplicate logs appearing in console when using the python logging module (#6275)

Contributors

@awaelchli, @Borda, @ethanwharris, @justusschock, @kaushikb11

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.4] - 2021-03-16

Changed

• Changed the default of find_unused_parameters back to True in DDP and DDP Spawn (#6438)

Fixed

• Expose DeepSpeed loss parameters to allow users to fix loss instability (#6115)
• Fixed DP reduction with collection (#6324)
• Fixed an issue where the tuner would not tune the learning rate if also tuning the batch size (#4688)
• Fixed broadcast to use PyTorch broadcast_object_list and add reduce_decision (#6410)
• Fixed logger creating directory structure too early in DDP (#6380)
• Fixed DeepSpeed additional memory use on rank 0 when default device not set early enough (#6460)
• Fixed DummyLogger.log_hyperparams raising a TypeError when running with fast_dev_run=True (#6398)
• Fixed an issue with Tuner.scale_batch_size not finding the batch size attribute in the datamodule (#5968)
• Fixed an exception in the layer summary when the model contains torch.jit scripted submodules (#6511)
• Fixed when Train loop config was run during Trainer.predict (#6541)

Contributors

@awaelchli, @kaushikb11, @Palzer, @SeanNaren, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.3] - 2021-03-09

Fixed

• Fixed ModelPruning(make_pruning_permanent=True) pruning buffers getting removed when saved during training (#6073)
• Fixed when _stable_1d_sort to work when n >= N (#6177)
• Fixed AttributeError when logger=None on TPU (#6221)
• Fixed PyTorch Profiler with emit_nvtx (#6260)
• Fixed trainer.test from best_path hangs after calling trainer.fit (#6272)
• Fixed SingleTPU calling all_gather (#6296)
• Ensure we check deepspeed/sharded in multinode DDP (#6297)
• Check LightningOptimizer doesn't delete optimizer hooks (#6305)
• Resolve memory leak for evaluation (#6326)
• Ensure that clip gradients is only called if the value is greater than 0 (#6330)
• Fixed Trainer not resetting lightning_optimizers when calling Trainer.fit() multiple times (#6372)

Contributors

@awaelchli, @carmocca, @Chizuchizu, @frankier, @SeanNaren, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.2] - 2021-03-02

Added

• Added checkpoint parameter to callback's on_save_checkpoint hook (#6072)

Changed

• Changed the order of backward, step, zero_grad to zero_grad, backward, step (#6147)
• Changed default for DeepSpeed CPU Offload to False, due to prohibitively slow speeds at smaller scale (#6262)

Fixed

• Fixed epoch level schedulers not being called when val_check_interval < 1.0 (#6075)
• Fixed multiple early stopping callbacks (#6197)
• Fixed incorrect usage of detach(), cpu(), to() (#6216)
• Fixed LBFGS optimizer support which didn't converge in automatic optimization (#6147)
• Prevent WandbLogger from dropping values (#5931)
• Fixed error thrown when using valid distributed mode in multi node (#6297)

Contributors

@akihironitta, @borisdayma, @carmocca, @dvolgyes, @SeanNaren, @SkafteNicki

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.1] - 2021-02-23

Fixed

• Fixed incorrect yield logic for the amp autocast context manager (#6080)
• Fixed priority of plugin/accelerator when setting distributed mode (#6089)
• Fixed error message for AMP + CPU incompatibility (#6107)

Contributors

@awaelchli, @SeanNaren, @carmocca

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.2.0] - 2021-02-18

Added

• Added DataType, AverageMethod and MDMCAverageMethod enum in metrics (#5657)
• Added support for summarized model total params size in megabytes (#5590)
• Added support for multiple train loaders (#1959)
• Added Accuracy metric now generalizes to Top-k accuracy for (multi-dimensional) multi-class inputs using the top_k parameter (#4838)
• Added Accuracy metric now enables the computation of subset accuracy for multi-label or multi-dimensional multi-class inputs with the subset_accuracy parameter (#4838)
• Added HammingDistance metric to compute the hamming distance (loss) (#4838)
• Added max_fpr parameter to auroc metric for computing partial auroc metric (#3790)
• Added StatScores metric to compute the number of true positives, false positives, true negatives and false negatives (#4839)
• Added R2Score metric (#5241)
• Added LambdaCallback (#5347)
• Added BackboneLambdaFinetuningCallback (#5377)
• Accelerator all_gather supports collection (#5221)
• Added image_gradients functional metric to compute the image gradients of a given input image. (#5056)
• Added MetricCollection (#4318)
• Added .clone() method to metrics (#4318)
• Added IoU class interface (#4704)
• Support to tie weights after moving model to TPU via on_post_move_to_device hook
• Added missing val/test hooks in LightningModule (#5467)
• The Recall and Precision metrics (and their functional counterparts recall and precision) can now be generalized to Recall@K and Precision@K with the use of top_k parameter (#4842)
• Added ModelPruning Callback (#5618, #5825, #6045)
• Added PyTorchProfiler (#5560)
• Added compositional metrics (#5464)
• Added Trainer method predict(...) for high performence predictions (#5579)
• Added on_before_batch_transfer and on_after_batch_transfer data hooks (#3671)
• Added AUC/AUROC class interface (#5479)
• Added PredictLoop object (#5752)
• Added QuantizationAwareTraining callback (#5706, #6040)
• Added LightningModule.configure_callbacks to enable the definition of model-specific callbacks (#5621)
• Added dim to PSNR metric for mean-squared-error reduction (#5957)
• Added promxial policy optimization template to pl_examples (#5394)
• Added log_graph to CometLogger (#5295)
• Added possibility for nested loaders (#5404)
• Added sync_step to Wandb logger (#5351)
• Added StochasticWeightAveraging callback (#5640)
• Added LightningDataModule.from_datasets(...) (#5133)
• Added PL_TORCH_DISTRIBUTED_BACKEND env variable to select backend (#5981)
• Added Trainer flag to activate Stochastic Weight Averaging (SWA) Trainer(stochastic_weight_avg=True) (#6038)
• Added DeepSpeed integration (#5954, #6042)

Changed

• Changed stat_scores metric now calculates stat scores over all classes and gains new parameters, in line with the new StatScores metric (#4839)
• Changed computer_vision_fine_tunning example to use BackboneLambdaFinetuningCallback (#5377)
• Changed automatic casting for LoggerConnector metrics (#5218)
• Changed iou [func] to allow float input (#4704)
• Metric compute() method will no longer automatically call reset() (#5409)
• Set PyTorch 1.4 as min requirements, also for testing and examples torchvision>=0.5 and torchtext>=0.5 (#5418)
• Changed callbacks argument in Trainer to allow Callback input (#5446)
• Changed the default of find_unused_parameters to False in DDP (#5185)
• Changed ModelCheckpoint version suffixes to start at 1 (#5008)
• Progress bar metrics tensors are now converted to float (#5692)
• Changed the default value for the progress_bar_refresh_rate Trainer argument in Google COLAB notebooks to 20 (#5516)
• Extended support for purely iteration-based training (#5726)
• Made LightningModule.global_rank, LightningModule.local_rank and LightningModule.logger read-only properties (#5730)
• Forced ModelCheckpoint callbacks to run after all others to guarantee all states are saved to the checkpoint (#5731)
• Refactored Accelerators and Plugins (#5743)
  • Added base classes for plugins (#5715)
  • Added parallel plugins for DP, DDP, DDPSpawn, DDP2 and Horovod (#5714)
  • Precision Plugins (#5718)
  • Added new Accelerators for CPU, GPU and TPU (#5719)
  • Added Plugins for TPU training (#5719)
  • Added RPC and Sharded plugins (#5732)
  • Added missing LightningModule-wrapper logic to new plugins and accelerator (#5734)
  • Moved device-specific teardown logic from training loop to accelerator (#5973)
  • Moved accelerator_connector.py to the connectors subfolder (#6033)
  • Trainer only references accelerator (#6039)
  • Made parallel devices optional across all plugins (#6051)
  • Cleaning (#5948, #5949, #5950)
• Enabled self.log in callbacks (#5094)
• Renamed xxx_AVAILABLE as protected (#5082)
• Unified module names in Utils (#5199)
• Separated utils: imports & enums (#5256, #5874)
• Refactor: clean trainer device & distributed getters (#5300)
• Simplified training phase as LightningEnum (#5419)
• Updated metrics to use LightningEnum (#5689)
• Changed the seq of on_train_batch_end, on_batch_end & on_train_epoch_end, on_epoch_end hooks (#5688)
• Refactored setup_training and remove test_mode (#5388)
• Disabled training with zero num_training_batches when insufficient limit_train_batches (#5703)
• Refactored EpochResultStore (#5522)
• Update lr_finder to check for attribute if not running fast_dev_run (#5990)
• LightningOptimizer manual optimizer is more flexible and expose toggle_model (#5771)
• MlflowLogger limit parameter value length to 250 char (#5893)
• Re-introduced fix for Hydra directory sync with multiple process (#5993)

Deprecated

• Function stat_scores_multiple_classes is deprecated in favor of stat_scores (#4839)
• Moved accelerators and plugins to its legacy pkg (#5645)
• Deprecated LightningDistributedDataParallel in favor of new wrapper module LightningDistributedModule (#5185)
• Deprecated LightningDataParallel in favor of new wrapper module LightningParallelModule (#5670)
• Renamed utils modules (#5199)
  • argparse_utils >> argparse
  • model_utils >> model_helpers
  • warning_utils >> warnings
  • xla_device_utils >> xla_device
• Deprecated using 'val_loss' to set the ModelCheckpoint monitor (#6012)
• Deprecated .get_model() with explicit .lightning_module property (#6035)
• Deprecated Trainer attribute accelerator_backend in favor of accelerator (#6034)

Removed

• Removed deprecated checkpoint argument filepath (#5321)
• Removed deprecated Fbeta, f1_score and fbeta_score metrics (#5322)
• Removed deprecated TrainResult (#5323)
• Removed deprecated EvalResult (#5633)
• Removed LoggerStages (#5673)

Fixed

• Fixed distributed setting and ddp_cpu only with num_processes>1 (#5297)
• Fixed the saved filename in ModelCheckpoint when it already exists (#4861)
• Fixed DDPHPCAccelerator hangs in DDP construction by calling init_device (#5157)
• Fixed num_workers for Windows example (#5375)
• Fixed loading yaml (#5619)
• Fixed support custom DataLoader with DDP if they can be re-instantiated (#5745)
• Fixed repeated .fit() calls ignore max_steps iteration bound (#5936)
• Fixed throwing MisconfigurationError on unknown mode (#5255)
• Resolve bug with Finetuning (#5744)
• Fixed ModelCheckpoint race condition in file existence check (#5155)
• Fixed some compatibility with PyTorch 1.8 (#5864)
• Fixed forward cache (#5895)
• Fixed recursive detach of tensors to CPU (#6007)
• Fixed passing wrong strings for scheduler interval doesn't throw an error (#5923)
• Fixed wrong requires_grad state after return None with multiple optimizers (#5738)
• Fixed add on_epoch_end hook at the end of validation, test epoch (#5986)
• Fixed missing process_dataloader call for TPUSpawn when in distributed mode (#6015)
• Fixed progress bar flickering by appending 0 to floats/strings (#6009)
• Fixed synchronization issues with TPU training (#6027)
• Fixed hparams.yaml saved twice when using TensorBoardLogger (#5953)
• Fixed basic examples (#5912, #5985)
• Fixed fairscale compatible with PT 1.8 (#5996)
• Ensured process_dataloader is called when tpu_cores > 1 to use Parallel DataLoader (#6015)
• Attempted SLURM auto resume call when non-shell call fails (#6002)
• Fixed wrapping optimizers upon assignment (#6006)
• Fixed allowing hashing of metrics with lists in their state (#5939)

Contributors

@alanhdu, @ananthsub, @awaelchli, @Borda, @borisdayma, @carmocca, @ddrevicky, @deng-cy, @ducthienbui97, @justusschock, @kartik4949, @kaushikb11, @manipopopo, @marload, @neighthan, @peblair, @prampey, @pranjaldatta, @rohitgr7, @SeanNaren, @sid-sundrani, @SkafteNicki, @tadejsv, @tchaton, @teddykoker, @titu1994, @yuntai

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.1.8] - 2021-02-08

Fixed

• Separate epoch validation from step validation (#5208)
• Fixed toggle_optimizers not handling all optimizer parameters (#5775)

Contributors

@ananthsub, @rohitgr7

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.1.7] - 2021-02-03

Fixed

• Fixed TensorBoardLogger not closing SummaryWriter on finalize (#5696)
• Fixed filtering of pytorch "unsqueeze" warning when using DP (#5622)
• Fixed num_classes argument in F1 metric (#5663)
• Fixed log_dir property (#5537)
• Fixed a race condition in ModelCheckpoint when checking if a checkpoint file exists (#5144)
• Remove unnecessary intermediate layers in Dockerfiles (#5697)
• Fixed auto learning rate ordering (#5638)

Contributors

@awaelchli @guillochon @noamzilo @rohitgr7 @SkafteNicki @sumanthratna

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.1.6] - 2021-01-26

Changed

• Increased TPU check timeout from 20s to 100s (#5598)
• Ignored step param in Neptune logger's log_metric method (#5510)
• Pass batch outputs to on_train_batch_end instead of epoch_end outputs (#4369)

Fixed

• Fixed toggle_optimizer to reset requires_grad state (#5574)
• Fixed FileNotFoundError for best checkpoint when using DDP with Hydra (#5629)
• Fixed an error when logging a progress bar metric with a reserved name (#5620)
• Fixed Metric's state_dict not included when child modules (#5614)
• Fixed Neptune logger creating multiple experiments when GPUs > 1 (#3256)
• Fixed duplicate logs appearing in console when using the python logging module (#5509)
• Fixed tensor printing in trainer.test() (#5138)
• Fixed not using dataloader when hparams present (#4559)

Contributors

@awaelchli @bryant1410 @lezwon @manipopopo @PiotrJander @psinger @rnett @SeanNaren @swethmandava @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.1.5] - 2021-01-19

Fixed

• Fixed a visual bug in the progress bar display initialization (#4579)
• Fixed logging on_train_batch_end in a callback with multiple optimizers (#5521)
• Fixed reinit_scheduler_properties with correct optimizer (#5519)
• Fixed val_check_interval with fast_dev_run (#5540)

Contributors

@awaelchli, @carmocca, @rohitgr7

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.1.4] - 2021-01-12

Added

• Add automatic optimization property setter to lightning module (#5169)

Changed

• Changed deprecated enable_pl_optimizer=True (#5244)

Fixed

• Fixed transfer_batch_to_device for DDP with len(devices_ids) == 1 (#5195)
• Logging only on not should_accumulate() during training (#5417)
• Resolve interpolation bug with Hydra (#5406)
• Check environ before selecting a seed to prevent warning message (#4743)

Contributors

@ananthsub, @SeanNaren, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

[1.1.3] - 2021-01-05

Added

• Added a check for optimizer attached to lr_scheduler (#5338)
• Added support for passing non-existing filepaths to resume_from_checkpoint (#4402)

Changed

• Skip restore from resume_from_checkpoint while testing (#5161)
• Allowed log_momentum for adaptive optimizers in LearningRateMonitor (#5333)
• Disabled checkpointing, earlystopping and logging with fast_dev_run (#5277)
• Distributed group defaults to WORLD if None (#5125)

Fixed

• Fixed trainer.test returning non-test metrics (#5214)
• Fixed metric state reset (#5273)
• Fixed --num-nodes on DDPSequentialPlugin (#5327)
• Fixed invalid value for weights_summary (#5296)
• Fixed Trainer.test not using the latest best_model_path (#5161)
• Fixed existence check for hparams not using underlying filesystem (#5250)
• Fixed LightningOptimizer AMP bug (#5191)
• Fixed casted key to string in _flatten_dict (#5354)

Contributors

@8greg8, @haven-jeon, @kandluis, @marload, @rohitgr7, @tadejsv, @tarepan, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Overview

Detail changes

Added

• Support number for logging with sync_dist=True (#5080)
• Added offset logging step when resuming for Wandb logger (#5050)

Removed

• enable_pl_optimizer=False by default to temporarily fix AMP issues (#5163)

Fixed

• Metric reduction with Logging (#5150)
• Remove nan loss in manual optimization (#5121)
• Un-balanced logging properly supported (#5119)
• Fix hanging in DDP HPC accelerators (#5157)
• Fix saved filename in ModelCheckpoint if it already exists (#4861)
• Fix reset TensorRunningAccum (#5106)
• Updated DALIClassificationLoader to not use deprecated arguments (#4925)
• Corrected call to torch.no_grad (#5124)

Contributors

@8greg8, @ananthsub, @borisdayma, @gan3sh500, @rohitgr7, @SeanNaren, @tchaton, @VinhLoiIT

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Overview

Detail changes

Added

• Add a notebook example to reach a quick baseline of ~94% accuracy on CIFAR10 using Resnet in Lightning (#4818)

Changed

• Simplify accelerator steps (#5015)
• Refactor load in checkpoint connector (#4593)

Removed

• Drop duplicate metrics (#5014)
• Remove beta arg from F1 class and functional (#5076)

Fixed

• Fixed trainer by default None in DDPAccelerator (#4915)
• Fixed LightningOptimizer to expose optimizer attributes (#5095)
• Do not warn when the name key is used in the lr_scheduler dict (#5057)
• Check if optimizer supports closure (#4981)
• Extend LightningOptimizer to exposure underlying Optimizer attributes + update doc (#5095)
• Add deprecated metric utility functions back to functional (#5067, #5068)
• Allow any input in to_onnx and to_torchscript (#4378)
• Do not warn when the name key is used in the lr_scheduler dict (#5057)

Contributors

@Borda, @carmocca, @hemildesai, @rohitgr7, @s-rog, @tarepan, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Overview

Lightning 1.1 is out! You can now train models with twice the parameters and zero code changes with the new sharded model training! We also have a new plugin for sequential model parallelism, more logging options, and a lot of improvements! Release highlights: https://bit.ly/3gyLZpP

Learn more about sharded training: https://bit.ly/2W3hgI0

Detail changes

Added

• Added "monitor" key to saved ModelCheckpoints (#4383)
• Added ConfusionMatrix class interface (#4348)
• Added multiclass AUROC metric (#4236)
• Added global step indexing to the checkpoint name for a better sub-epoch checkpointing experience (#3807)
• Added optimizer hooks in callbacks (#4379)
• Added option to log momentum (#4384)
• Added current_score to ModelCheckpoint.on_save_checkpoint (#4721)
• Added logging using self.log in train and evaluation for epoch end hooks (#4913)
• Added ability for DDP plugin to modify optimizer state saving (#4675)
• Added casting to python types for NumPy scalars when logging hparams (#4647)
• Added prefix argument in loggers (#4557)
• Added printing of total num of params, trainable and non-trainable params in ModelSummary (#4521)
• Added PrecisionRecallCurve, ROC, AveragePrecision class metric (#4549)
• Added custom Apex and NativeAMP as Precision plugins (#4355)
• Added DALI MNIST example (#3721)
• Added sharded plugin for DDP for multi-GPU training memory optimizations (#4773)
• Added experiment_id to the NeptuneLogger (#3462)
• Added Pytorch Geometric integration example with Lightning (#4568)
• Added all_gather method to LightningModule which allows gradient-based tensor synchronizations for use-cases such as negative sampling. (#5012)
• Enabled self.log in most functions (#4969)
• Added changeable extension variable for ModelCheckpoint (#4977)

Changed

• Removed multiclass_roc and multiclass_precision_recall_curve, use roc and precision_recall_curve instead (#4549)
• Tuner algorithms will be skipped if fast_dev_run=True (#3903)
• WandbLogger does not force wandb reinit arg to True anymore and creates a run only when needed (#4648)
• Changed automatic_optimization to be a model attribute (#4602)
• Changed Simple Profiler report to order by percentage time spent + num calls (#4880)
• Simplify optimization Logic (#4984)
• Classification metrics overhaul (#4837)
• Updated fast_dev_run to accept integer representing num_batches (#4629)
• Refactored optimizer (#4658)

Deprecated

• Deprecated prefix argument in ModelCheckpoint (#4765)
• Deprecated the old way of assigning hyper-parameters through self.hparams = ... (#4813)
• Deprecated mode='auto' from ModelCheckpoint and EarlyStopping (#4695)

Removed

• Removed reorder parameter of the auc metric (#5004)

Fixed

• Added feature to move tensors to CPU before saving (#4309)
• Fixed LoggerConnector to have logged metrics on root device in DP (#4138)
• Auto convert tensors to contiguous format when gather_all (#4907)
• Fixed PYTHONPATH for DDP test model (#4528)
• Fixed allowing logger to support indexing (#4595)
• Fixed DDP and manual_optimization (#4976)

Contributors

@ananyahjha93, @awaelchli, @blatr, @Borda, @borisdayma, @carmocca, @ddrevicky, @george-gca, @gianscarpe, @irustandi, @janhenriklambrechts, @jeremyjordan, @justusschock, @lezwon, @rohitgr7, @s-rog, @SeanNaren, @SkafteNicki, @tadejsv, @tchaton, @williamFalcon, @zippeurfou

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Detail changes

Added

• Added casting to python types for numpy scalars when logging hparams (#4647)
• Added warning when progress bar refresh rate is less than 20 on Google Colab to prevent crashing (#4654)
• Added F1 class metric (#4656)

Changed

• Consistently use step=trainer.global_step in LearningRateMonitor independently of logging_interval (#4376)
• Metric states are no longer as default added to state_dict (#4685)
• Renamed class metric Fbeta >> FBeta (#4656)
• Model summary: add 1 decimal place (#4745)
• Do not override PYTHONWARNINGS (#4700)

Fixed

• Fixed checkpoint hparams dict casting when omegaconf is available (#4770)
• Fixed incomplete progress bars when total batches not divisible by refresh rate (#4577)
• Updated SSIM metric (#4566)(#4656)
• Fixed batch_arg_name - add batch_arg_name to all calls to _adjust_batch_sizebug (#4812)
• Fixed torchtext data to GPU (#4785)
• Fixed a crash bug in MLFlow logger (#4716)

Contributors

@awaelchli, @jonashaag, @jungwhank, @M-Salti, @moi90, @pgagarinov, @s-rog, @Samyak2, @SkafteNicki, @teddykoker, @ydcjeff

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Detail changes

Added

• Added lambda closure to manual_optimizer_step (#4618)

Changed

• Change Metrics persistent default mode to False (#4685)

Fixed

• Prevent crash if sync_dist=True on CPU (#4626)
• Fixed average pbar Metrics (#4534)
• Fixed setup callback hook to correctly pass the LightningModule through (#4608)
• Allowing decorate model init with saving hparams inside (#4662)
• Fixed split_idx set by LoggerConnector in on_trainer_init to Trainer (#4697)

Contributors

@ananthsub, @Borda, @SeanNaren, @SkafteNicki, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Detail changes

Added

• Added metrics aggregation in Horovod and fixed early stopping (#3775)
• Added manual_optimizer_step which work with AMP Native and accumulated_grad_batches (#4485)
• Added persistent(mode) method to metrics, to enable and disable metric states being added to state_dict (#4482)
• Added congratulations at the end of our notebooks (#4555)

Changed

• Changed fsspec to tuner (#4458)
• Unify sLURM/TorchElastic under backend plugin (#4578, #4580, #4581, #4582, #4583)

Fixed

• Fixed feature-lack in hpc_load (#4526)
• Fixed metrics states being overridden in DDP mode (#4482)
• Fixed lightning_getattr, lightning_hasattr not finding the correct attributes in datamodule (#4347)
• Fixed automatic optimization AMP by manual_optimization_step (#4485)
• Replace MisconfigurationException with warning in ModelCheckpoint Callback (#4560)
• Fixed logged keys in mlflow logger (#4412)
• Fixed is_picklable by catching AttributeError (#4508)

Contributors

@dscarmo, @jtamir, @kazhang, @maxjeblick, @rohitgr7, @SkafteNicki, @tarepan, @tchaton, @tgaddair, @williamFalcon

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Detail changes

Added

• Added PyTorch 1.7 Stable support (#3821)
• Added timeout for tpu_device_exists to ensure process does not hang indefinitely (#4340)

Changed

• W&B log in sync with Trainer step (#4405)
• Hook on_after_backward is called only when optimizer_step is being called (#4439)
• Moved track_and_norm_grad into training loop and called only when optimizer_step is being called (#4439)
• Changed type checker with explicit cast of ref_model object (#4457)

Deprecated

• Deprecated passing ModelCheckpoint instance to checkpoint_callback Trainer argument (#4336)

Fixed

• Disable saving checkpoints if not trained (#4372)
• Fixed error using auto_select_gpus=True with gpus=-1 (#4209)
• Disabled training when limit_train_batches=0 (#4371)
• Fixed that metrics do not store computational graph for all seen data (#4313)
• Fixed AMP unscale for on_after_backward (#4439)
• Fixed TorchScript export when module includes Metrics (#4428)
• Fixed CSV logger warning (#4419)
• Fixed skip DDP parameter sync (#4301)

Contributors

@ananthsub, @awaelchli, @borisdayma, @carmocca, @justusschock, @lezwon, @rohitgr7, @SeanNaren, @SkafteNicki, @ssaru, @tchaton, @ydcjeff

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Detail changes

Added

• Added dirpath and filename parameter in ModelCheckpoint (#4213)
• Added plugins docs and DDPPlugin to customize ddp across all accelerators (#4258)
• Added strict option to the scheduler dictionary (#3586)
• Added fsspec support for profilers (#4162)
• Added autogenerated helptext to Trainer.add_argparse_args (#4344)
• Added support for string values in Trainer's profiler parameter (#3656)

Changed

• Improved error messages for invalid configure_optimizers returns (#3587)
• Allow changing the logged step value in validation_step (#4130)
• Allow setting replace_sampler_ddp=True with a distributed sampler already added (#4273)
• Fixed santized parameters for WandbLogger.log_hyperparams (#4320)

Deprecated

• Deprecated filepath in ModelCheckpoint (#4213)
• Deprecated reorder parameter of the auc metric (#4237)
• Deprecated bool values in Trainer's profiler parameter (#3656)

Fixed

• Fixed setting device ids in DDP (#4297)
• Fixed synchronization of best model path in ddp_accelerator (#4323)
• Fixed WandbLogger not uploading checkpoint artifacts at the end of training (#4341)

Contributors

@ananthsub, @awaelchli, @carmocca, @ddrevicky, @louis-she, @mauvilsa, @rohitgr7, @SeanNaren, @tchaton

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Detail changes

Added

• Added persistent flag to Metric.add_state (#4195)

Changed

• Used checkpoint_connector.hpc_save in SLURM (#4217)
• Moved base req. to root (#4219)

Fixed

• Fixed hparams assign in init (#4189)
• Fixed overwrite check for model hooks (#4010)

Contributors

@Borda, @EspenHa, @teddykoker

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Fixes the last major bugs for validation logging. Also removes duplicate charts for metric / metric_loss. Doing this minor release because correct validation metrics logging is critical.

Details changes

Added

• Added trace functionality to the function to_torchscript (#4142)

Changed

• Called on_load_checkpoint before loading state_dict (#4057)

Removed

• Removed duplicate metric vs step log for train loop (#4173)

Fixed

• Fixed the self.log problem in validation_step() (#4169)
• Fixed hparams saving - save the state when save_hyperparameters() is called [in __init__] (#4163)
• Fixed runtime failure while exporting hparams to yaml (#4158)

Contributors

@Borda, @NumesSanguis, @rohitgr7, @williamFalcon

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Obligatory post 1.0 minor release. Main fix is to make Lightning module fully compatible with Jit (had some edge-cases we had not covered).

Overview

...

Detail changes

• Added Explained Variance Metric + metric fix (#4013)
• Added Metric <-> Lightning Module integration tests (#4008)
• Added parsing OS env vars in Trainer (#4022)
• Added classification metrics (#4043)
• Updated explained variance metric (#4024)
• Enabled plugins (#4041)
• Enabled custom clusters (#4048)
• Enabled passing in custom accelerators (#4050)
• Added LightningModule.toggle_optimizer (#4058)
• Added LightningModule.manual_backward (#4063)

Changed

• Integrated metrics API with self.log (#3961)
• Decoupled Appex (#4052, #4054, #4055, #4056, #4058, #4060, #4061, #4062, #4063, #4064, #4065)
• Renamed all backends to Accelerator (#4066)
• Enabled manual returns (#4089)

Removed

• Removed output argument from *_batch_end hooks (#3965, #3966)
• Removed output argument from *_epoch_end hooks (#3967)
• Removed support for EvalResult and TrainResult (#3968)
• Removed deprecated trainer flags: overfit_pct, log_save_interval, row_log_interval (#3969)
• Removed deprecated early_stop_callback (#3982)
• Removed deprecated model hooks (#3980)
• Removed deprecated callbacks (#3979)
• Removed trainer argument in LightningModule.backward [#4056)

Fixed

• Fixed current_epoch property update to reflect true epoch number inside LightningDataModule, when reload_dataloaders_every_epoch=True. (#3974)
• Fixed to print scaler value in progress bar (#4053)
• Fixed mismatch between docstring and code regarding when on_load_checkpoint hook is called (#3996)

Contributors

@ananyahjha93, @Borda, @edenlightning, @hbredin, @rohitgr7, @SkafteNicki, @teddykoker, @williamFalcon

If we forgot someone due to not matching commit email with GitHub account, let us know :]

This release is a buffer in case 1.0 breaks any compatibility for people who upgrade. 0.10.0 has all the bug fixes and features of 1.0 but is 100% backward compatible. The 1.0 release following in the next 24 hours.

Overview

The major changes are:

• Results objects are deprecated (we hated them too haha)
• This means dataflow and logging have been decoupled

To log:

def any_step(...):
   self.log('something', i_computed)

Separately, return whatever you want from methods:

def training_step(...):
  return loss

or

def training_step(...):
   return {'loss': loss, 'whatever': [1, 'want']}

Detail changes

Added

• Added new Metrics API. (#3868, [#3921)
• Enable PyTorch 1.7 compatibility (#3541)
• Added LightningModule.to_torchscript to support exporting as ScriptModule (#3258)
• Added warning when dropping unpicklable hparams (#2874)
• Added EMB similarity (#3349)
• Added ModelCheckpoint.to_yaml method (#3048)
• Allow ModelCheckpoint monitor to be None, meaning it will always save ([3630)
• Disabled optimizers setup during testing (#3059)
• Added support for datamodules to save and load checkpoints when training (#3563
• Added support for datamodule in learning rate finder (#3425)
• Added gradient clip test for native AMP (#3754)
• Added dist lib to enable syncing anything across devices (#3762)
• Added broadcast to TPUBackend (#3814)
• Added XLADeviceUtils class to check XLA device type (#3274)

Changed

• Refactored accelerator backends:
  • moved TPU xxx_step to backend (#3118)
  • refactored DDP backend forward (#3119)
  • refactored GPU backend __step (#3120)
  • refactored Horovod backend (#3121, #3122)
  • remove obscure forward call in eval + CPU backend ___step (#3123)
  • reduced all simplified forward (#3126)
  • added hook base method (#3127)
  • refactor eval loop to use hooks - use test_mode for if so we can split later (#3129)
  • moved ___step_end hooks (#3130)
  • training forward refactor (#3134)
  • training AMP scaling refactor (#3135)
  • eval step scaling factor (#3136)
  • add eval loop object to streamline eval loop (#3138)
  • refactored dataloader process hook (#3139)
  • refactored inner eval loop (#3141)
  • final inner eval loop hooks (#3154)
  • clean up hooks in run_evaluation (#3156)
  • clean up data reset (#3161)
  • expand eval loop out (#3165)
  • moved hooks around in eval loop (#3195)
  • remove _evaluate fx (#3197)
  • Trainer.fit hook clean up (#3198)
  • DDPs train hooks (#3203)
  • refactor DDP backend (#3204, #3207, #3208, #3209, #3210)
  • reduced accelerator selection (#3211)
  • group prepare data hook (#3212)
  • added data connector (#3285)
  • modular is_overridden (#3290)
  • adding Trainer.tune() (#3293)
  • move run_pretrain_routine -> setup_training (#3294)
  • move train outside of setup training (#3297)
  • move prepare_data to data connector (#3307)
  • moved accelerator router (#3309)
  • train loop refactor - moving train loop to own object (#3310, #3312, #3313, #3314)
  • duplicate data interface definition up into DataHooks class (#3344)
  • inner train loop (#3359, #3361, #3362, #3363, #3365, #3366, #3367, #3368, #3369, #3370, #3371, #3372, #3373, #3374, #3375, #3376, #3385, #3388, #3397)
  • all logging related calls in a connector (#3395)
  • device parser (#3400, #3405)
  • added model connector (#3407)
  • moved eval loop logging to loggers (#3408)
  • moved eval loop (#3412[#3408)
  • trainer/separate argparse (#3421, #3428, #3432)
  • move lr_finder (#3434)
  • organize args (##3435, #3442, #3447, #3448, #3449, #3456)
  • move specific accelerator code (#3457)
  • group connectors (#3472)
  • accelerator connector methods x/n (#3469, #3470, #3474)
  • merge backends (#3476, #3477, #3478, #3480, #3482)
  • apex plugin (#3502)
  • precision plugins (#3504)
  • Result - make monitor default to checkpoint_on to simplify (#3571)
  • reference to the Trainer on the LightningDataModule (#3684)
  • add .log to lightning module (#3686, #3699, #3701, #3704, #3715)
  • enable tracking original metric when step and epoch are both true (#3685)
  • deprecated results obj, added support for simpler comms (#3681)
  • move backends back to individual files (#3712)
  • fixes logging for eval steps (#3763)
  • decoupled DDP, DDP spawn (#3733, #3766, #3767, #3774, #3802, #3806)
  • remove weight loading hack for ddp_cpu (#3808)
  • separate torchelastic from DDP (#3810)
  • separate SLURM from DDP (#3809)
  • decoupled DDP2 (#3816)
  • bug fix with logging val epoch end + monitor (#3812)
  • decoupled DDP, DDP spawn (#3733, #3817, #3819, #3927)
  • callback system and init DDP (#3836)
  • adding compute environments (#3837, [#3842)
  • epoch can now log independently (#3843)
  • test selecting the correct backend. temp backends while slurm and TorchElastic are decoupled (#3848)
  • fixed init_slurm_connection causing hostname errors (#3856)
  • moves init apex from LM to apex connector (#3923)
  • moves sync bn to each backend (#3925)
  • moves configure ddp to each backend (#3924)
• Deprecation warning (#3844)
• Changed LearningRateLogger to LearningRateMonitor (#3251)
• Used fsspec instead of gfile for all IO (#3320)
  • Swaped torch.load for fsspec load in DDP spawn backend (#3787)
  • Swaped torch.load for fsspec load in cloud_io loading (#3692)
  • Added support for to_disk() to use remote filepaths with fsspec (#3930)
  • Updated model_checkpoint's to_yaml to use fsspec open (#3801)
  • Fixed fsspec is inconsistant when doing fs.ls (#3805)
• Refactor GPUStatsMonitor to improve training speed (#3257)
• Changed IoU score behavior for classes absent in target and pred (#3098)
• Changed IoU remove_bg bool to ignore_index optional int (#3098)
• Changed defaults of save_top_k and save_last to None in ModelCheckpoint (#3680)
• row_log_interval and log_save_interval are now based on training loop's global_step instead of epoch-internal batch index (#3667)
• Silenced some warnings. verified ddp refactors (#3483)
• Cleaning up stale logger tests (#3490)
• Allow ModelCheckpoint monitor to be None (#3633)
• Enable None model checkpoint default (#3669)
• Skipped best_model_path if checkpoint_callback is None (#2962)
• Used raise .. from .. to explicitly chain exceptions (#3750)
• Mocking loggers (#3596, #3617, #3851, #3859, #3884, #3853, #3910, #3889, #3926)
• Write predictions in LightningModule instead of EvalResult [#3882

Deprecated

• Deprecated TrainResult and EvalResult, use self.log and self.write from the LightningModule to log metrics and write predictions. training_step can now only return a scalar (for the loss) or a dictionary with anything you want. (#3681)
• Deprecate early_stop_callback Trainer argument (#3845)
• Rename Trainer arguments row_log_interval >> log_every_n_steps and log_save_interval >> flush_logs_every_n_steps (#3748)

Removed

• Removed experimental Metric API (#3868, #3943, #3949, #3946), listed changes before final removal:
  • Added EmbeddingSimilarity metric (#3349, [#3358)
  • Added hooks to metric module interface (#2528)
  • Added error when AUROC metric is used for multiclass problems (#3350)
  • Fixed ModelCheckpoint with save_top_k=-1 option not tracking the best models when a monitor metric is available (#3735)
  • Fixed counter-intuitive error being thrown in Accuracy metric for zero target tensor (#3764)
  • Fixed aggregation of metrics (#3517)
  • Fixed Metric aggregation (#3321)
  • Fixed RMSLE metric (#3188)
  • Renamed reduction to class_reduction in classification metrics (#3322)
  • Changed class_reduction similar to sklearn for classification metrics (#3322)
  • Renaming of precision recall metric (#3308)

Fixed

• Fixed on_train_batch_start hook to end epoch early (#3700)
• Fixed num_sanity_val_steps is clipped to limit_val_batches (#2917)
• Fixed ONNX model save on GPU (#3145)
• Fixed GpuUsageLogger to work on different platforms (#3008)
• Fixed auto-scale batch size not dumping auto_lr_find parameter (#3151)
• Fixed batch_outputs with optimizer frequencies (#3229)
• Fixed setting batch size in LightningModule.datamodule when using auto_scale_batch_size (#3266)
• Fixed Horovod distributed backend compatibility with native AMP (#3404)
• Fixed batch size auto scaling exceeding the size of the dataset (#3271)
• Fixed getting experiment_id from MLFlow only once instead of each training loop (#3394)
• Fixed overfit_batches which now correctly disables shuffling for the training loader. (#3501)
• Fixed gradient norm tracking for row_log_interval > 1 (#3489)
• Fixed ModelCheckpoint name formatting ([3164)
• Fixed auto-scale batch size (#3151)
• Fixed example implementation of AutoEncoder (#3190)
• Fixed invalid paths when remote logging with TensorBoard (#3236)
• Fixed change t() to transpose() as XLA devices do not support .t() on 1-dim tensor (#3252)
• Fixed (weights only) checkpoints loading without PL (#3287)
• Fixed gather_all_tensors cross GPUs in DDP (#3319)
• Fixed CometML save dir (#3419)
• Fixed forward key metrics (#3467)
• Fixed normalize mode at confusion matrix (replace NaNs with zeros) (#3465)
• Fixed global step increment in training loop when training_epoch_end hook is used (#3673)
• Fixed dataloader shuffling not getting turned off with overfit_batches > 0 and distributed_backend = "ddp" (#3534)
• Fixed determinism in DDPSpawnBackend when using seed_everything in main process (#3335)
• Fixed ModelCheckpoint period to actually save every period epochs (#3630)
• Fixed val_progress_bar total with num_sanity_val_steps (#3751)
• Fixed Tuner dump: add current_epoch to dumped_params (#3261)
• Fixed current_epoch and global_step properties mismatch between Trainer and LightningModule (#3785)
• Fixed learning rate scheduler for optimizers with internal state (#3897)
• Fixed tbptt_reduce_fx when non-floating tensors are logged (#3796)
• Fixed model checkpoint frequency (#3852)
• Fixed logging non-tensor scalar with result breaks subsequent epoch aggregation (#3855)
• Fixed TrainerEvaluationLoopMixin activates model.train() at the end (#3858)
• Fixed overfit_batches when using with multiple val/test_dataloaders (#3857)
• Fixed enables training_step to return None (#3862)
• Fixed init nan for checkpointing (#3863)
• Fixed for load_from_checkpoint (#2776)
• Fixes incorrect batch_sizes when Dataloader returns a dict with multiple tensors (#3668)
• Fixed unexpected signature for validation_step (#3947)

Contributors

@abrahambotros, @akihironitta, @ananthsub, @ananyahjha93, @awaelchli, @Borda, @c00k1ez, @carmocca, @f4hy, @GimmickNG, @jbschiratti, @justusschock, @LeeJZh, @lezwon, @Lucas-Steinmann, @maxjeblick, @monney, @mpariente, @nateraw, @nrupatunga, @patrickorlando, @PhilJd, @rohitgr7, @s-rog, @ShomyLiu, @SkafteNicki, @Sordie, @teddykoker, @tgaddair, @Vozf, @williamFalcon, @XDynames, @ydcjeff

If we forgot someone due to not matching the commit email with GitHub account, let us know :]

Overview

The newest PyTorch Lightning release includes final API clean-up with better data decoupling and shorter logging syntax.

Were happy to release PyTorch Lightning 0.9 today, which contains many great new features, more bugfixes than any release we ever had, but most importantly it introduced our mostly final API changes! Lightning is being adopted by top researchers and AI labs around the world, and we are working hard to make sure we provide a smooth experience and support for all the latest best practices.

Detail changes

Added

• Added SyncBN for DDP (#2801, #2838)
• Added basic CSVLogger (#2721)
• Added SSIM metrics (#2671)
• Added BLEU metrics (#2535)
• Added support to export a model to ONNX format (#2596)
• Added support for Trainer(num_sanity_val_steps=-1) to check all validation data before training (#2246)
• Added struct. output:
  • tests for val loop flow (#2605)
  • EvalResult support for train and val. loop (#2615, #2651)
  • weighted average in results obj (#2930)
  • fix result obj DP auto reduce (#3013)
• Added class LightningDataModule (#2668)
• Added support for PyTorch 1.6 (#2745)
• Added call DataModule hooks implicitly in trainer (#2755)
• Added support for Mean in DDP Sync (#2568)
• Added remaining sklearn metrics: AveragePrecision, BalancedAccuracy, CohenKappaScore, DCG, Hamming, Hinge, Jaccard, MeanAbsoluteError, MeanSquaredError, MeanSquaredLogError, MedianAbsoluteError, R2Score, MeanPoissonDeviance, MeanGammaDeviance, MeanTweedieDeviance, ExplainedVariance (#2562)
• Added support for limit_{mode}_batches (int) to work with infinite dataloader (IterableDataset) (#2840)
• Added support returning python scalars in DP (#1935)
• Added support to Tensorboard logger for OmegaConf hparams (#2846)
• Added tracking of basic states in Trainer (#2541)
• Tracks all outputs including TBPTT and multiple optimizers (#2890)
• Added GPU Usage Logger (#2932)
• Added strict=False for load_from_checkpoint (#2819)
• Added saving test predictions on multiple GPUs (#2926)
• Auto log the computational graph for loggers that support this (#3003)
• Added warning when changing monitor and using results obj (#3014)
• Added a hook transfer_batch_to_device to the LightningDataModule (#3038)

Changed

• Truncated long version numbers in progress bar (#2594)
• Enabling val/test loop disabling (#2692)
• Refactored into accelerator module:
  • GPU training (#2704)
  • TPU training (#2708)
  • DDP(2) backend (#2796)
  • Retrieve last logged val from result by key (#3049)
• Using .comet.config file for CometLogger (#1913)
• Updated hooks arguments - breaking for setup and teardown (#2850)
• Using gfile to support remote directories (#2164)
• Moved optimizer creation after device placement for DDP backends (#2904](https://github.com/PyTorchLightning/pytorch-lighting/pull/2904))
• Support **DictConfig for hparam serialization (#2519)
• Removed callback metrics from test results obj (#2994)
• Re-enabled naming metrics in ckpt name (#3060)
• Changed progress bar epoch counting to start from 0 (#3061)

Deprecated

• Deprecated Trainer attribute ckpt_path, which will now be set by weights_save_path (#2681)

Removed

• Removed deprecated: (#2760)
  • core decorator data_loader
  • Module hook on_sanity_check_start and loading load_from_metrics
  • package pytorch_lightning.logging
  • Trainer arguments: show_progress_bar, num_tpu_cores, use_amp, print_nan_grads
  • LR Finder argument num_accumulation_steps

Fixed

• Fixed accumulate_grad_batches for last batch (#2853)
• Fixed setup call while testing (#2624)
• Fixed local rank zero casting (#2640)
• Fixed single scalar return from training (#2587)
• Fixed Horovod backend to scale LR schedlers with the optimizer (#2626)
• Fixed dtype and device properties not getting updated in submodules (#2657)
• Fixed fast_dev_run to run for all dataloaders (#2581)
• Fixed save_dir in loggers getting ignored by default value of weights_save_path when user did not specify weights_save_path (#2681)
• Fixed weights_save_path getting ignored when logger=False is passed to Trainer (#2681)
• Fixed TPU multi-core and Float16 (#2632)
• Fixed test metrics not being logged with LoggerCollection (#2723)
• Fixed data transfer to device when using torchtext.data.Field and include_lengths is True (#2689)
• Fixed shuffle argument for the distributed sampler (#2789)
• Fixed logging interval (#2694)
• Fixed loss value in the progress bar is wrong when accumulate_grad_batches > 1 (#2738)
• Fixed correct CWD for DDP sub-processes when using Hydra (#2719)
• Fixed selecting GPUs using CUDA_VISIBLE_DEVICES (#2739, #2796)
• Fixed false num_classes warning in metrics (#2781)
• Fixed shell injection vulnerability in subprocess call (#2786)
• Fixed LR finder and hparams compatibility (#2821)
• Fixed ModelCheckpoint not saving the latest information when save_last=True (#2881)
• Fixed ImageNet example: learning rate scheduler, number of workers and batch size when using DDP (#2889)
• Fixed apex gradient clipping (#2829)
• Fixed save apex scaler states (#2828)
• Fixed a model loading issue with inheritance and variable positional arguments (#2911)
• Fixed passing non_blocking=True when transferring a batch object that does not support it (#2910)
• Fixed checkpointing to remote file paths (#2925)
• Fixed adding val_step argument to metrics (#2986)
• Fixed an issue that caused Trainer.test() to stall in DDP mode (#2997)
• Fixed gathering of results with tensors of varying shape (#3020)
• Fixed batch size auto-scaling feature to set the new value on the correct model attribute (#3043)
• Fixed automatic batch scaling not working with half-precision (#3045)
• Fixed setting device to root GPU (#3042)

Contributors

@ananthsub, @ananyahjha93, @awaelchli, @bkhakshoor, @Borda, @ethanwharris, @f4hy, @groadabike, @ibeltagy, @justusschock, @lezwon, @nateraw, @neighthan, @nsarang, @PhilJd, @pwwang, @rohitgr7, @romesco, @ruotianluo, @shijianjian, @SkafteNicki, @tgaddair, @thschaaf, @williamFalcon, @xmotli02, @ydcjeff, @yukw777, @zerogerc

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Overview

The point of this release is more bug fixes ahead of v 1.0.0. We now have CI tests on TPU thanks to @zcain117 from Google! :slightly_smiling_face: This means we fixed many TPU bugs we hadn’t caught before because we had no tests. In addition, we fixed:

• all the file path errors with loggers (txs @awaelchli)
• pickling errors with loggers (txs @awaelchli)
• fixed all the .test() calls

Detail changes

Added

• Added a PSNR metric: peak signal-to-noise ratio (#2483)
• Added functional regression metrics (#2492)

Removed

• Removed auto val reduce (#2462)

Fixed

• Flattening Wandb Hyperparameters (#2459)
• Fixed using the same DDP python interpreter and actually running (#2482)
• Fixed model summary input type conversion for models that have input dtype different from model parameters (#2510)
• Made TensorBoardLogger and CometLogger pickleable (#2518)
• Fixed a problem with MLflowLogger creating multiple run folders (#2502)
• Fixed global_step increment (#2455)
• Fixed TPU hanging example (#2488)
• Fixed argparse default value bug (#2526)
• Fixed Dice and IoU to avoid NaN by adding small eps (#2545)
• Fixed accumulate gradients schedule at epoch 0 (continued) (#2513)
• Fixed Trainer .fit() returning last not best weights in "ddp_spawn" (#2565)
• Fixed passing (do not pass) TPU weights back on test (#2566)
• Fixed DDP tests and .test() (#2512, #2570)

Contributors

@anthonytec2, @awaelchli, @bernardomig, @Borda, @EspenHa, @HHousen, @InCogNiTo124, @rohitgr7, @williamFalcon

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Detail changes

Added

• Added reduce ddp results on eval (#2434)
• Added a warning when an IterableDataset has __len__ defined (#2437)

Changed

• Enabled no returns from eval (#2446)

Fixed

• Fixes train outputs (#2428)
• Fixes Conda dependencies (#2412)
• Fixed Apex scaling with decoupled backward (#2433)
• Fixed crashing or wrong displaying progressbar because of missing ipywidgets (#2417)
• Fixed TPU saving dir (fc26078e395f8a001f4c6dd7b3fe7ca202f914a3, 04e68f022fc03dd5f1555ee86dea997d42a448ad)
• Fixed logging on rank 0 only (#2425)

Contributors

@awaelchli, @Borda, @olineumann, @williamFalcon

Fixed

• Fixed AMP wrong call (593837e1da24ff6c942b24ed803fc1496a304609)
• Fixed batch typo (92d1e75b2638a493d9d21ed5fe00a22093888285)

Overview

Fixing critical bugs in newly added hooks and hparams assignment. The recommended data following:

1. use prepare_data to download and process the dataset.
2. use setup to do splits, and build your model internals

Detail changes

• Fixed the load_from_checkpoint path detected as URL bug (#2244)
• Fixed hooks - added barrier (#2245, #2257, #2260)
• Fixed hparams - remove frame inspection on self.hparams (#2253)
• Fixed setup and on fit calls (#2252)
• Fixed GPU template (#2255)

Overview

Highlights of this release are adding support for TorchElastic enables distributed PyTorch training jobs to be executed in a fault-tolerant and elastic manner; auto-scaling of batch size; new transfer learning example; an option to provide seed to random generators to ensure reproducibility.

Detail changes

Added

• Added callback for logging learning rates (#1498)
• Added transfer learning example (for a binary classification task in computer vision) (#1564)
• Added type hints in Trainer.fit() and Trainer.test() to reflect that also a list of dataloaders can be passed in (#1723).
• Added auto scaling of batch size (#1638)
• The progress bar metrics now also get updated in training_epoch_end (#1724)
• Enable NeptuneLogger to work with distributed_backend=ddp (#1753)
• Added option to provide seed to random generators to ensure reproducibility (#1572)
• Added override for hparams in load_from_ckpt (#1797)
• Added support multi-node distributed execution under torchelastic (#1811, #1818)
• Added using store_true for bool args (#1822, #1842)
• Added dummy logger for internally disabling logging for some features (#1836)

Changed

• Enable non-blocking for device transfers to GPU (#1843)
• Replace mata_tags.csv with hparams.yaml (#1271)
• Reduction when batch_size < num_gpus (#1609)
• Updated LightningTemplateModel to look more like Colab example (#1577)
• Don't convert namedtuple to tuple when transferring the batch to target device (#1589)
• Allow passing hparams as a keyword argument to LightningModule when loading from checkpoint (#1639)
• Args should come after the last positional argument (#1807)
• Made DDP the default if no backend specified with multiple GPUs (#1789)

Deprecated

• Deprecated tags_csv in favor of hparams_file (#1271)

Fixed

• Fixed broken link in PR template (#1675)
• Fixed ModelCheckpoint not None checking file path (#1654)
• Trainer now calls on_load_checkpoint() when resuming from a checkpoint (#1666)
• Fixed sampler logic for DDP with the iterable dataset (#1734)
• Fixed _reset_eval_dataloader() for IterableDataset (#1560)
• Fixed Horovod distributed backend to set the root_gpu property (#1669)
• Fixed wandb logger global_step affects other loggers (#1492)
• Fixed disabling progress bar on non-zero ranks using Horovod backend (#1709)
• Fixed bugs that prevent LP finder to be used together with early stopping and validation dataloaders (#1676)
• Fixed a bug in Trainer that prepended the checkpoint path with version_ when it shouldn't (#1748)
• Fixed LR key name in case of param groups in LearningRateLogger (#1719)
• Fixed saving native AMP scaler state (introduced in #1561)
• Fixed accumulation parameter and suggestion method for learning rate finder (#1801)
• Fixed num processes wasn't being set properly and auto sampler was DDP failing (#1819)
• Fixed bugs in semantic segmentation example (#1824)
• Fixed saving native AMP scaler state (#1561, #1777)
• Fixed native AMP + DDP (#1788)
• Fixed hparam logging with metrics (#1647)

Contributors

@ashwinb, @awaelchli, @Borda, @cmpute, @festeh, @jbschiratti, @justusschock, @kepler, @kumuji, @nanddalal, @nathanbreitsch, @olineumann, @pitercl, @rohitgr7, @S-aiueo32, @SkafteNicki, @tgaddair, @tullie, @tw991, @williamFalcon, @ybrovman, @yukw777

If we forgot someone due to not matching commit email with GitHub account, let us know :]

We made a few changes to Callbacks to test ops on detached GPU tensors to avoid CPU transfer. However, it made callbacks unpicklable which will crash DDP.

This release fixes that core issue

Changed

• Allow logging of metrics together with hparams (#1630)
• Allow metrics logged together with hparams (#1630)

Removed

• Removed Warning from trainer loop (#1634)

Fixed

• Fixed ModelCheckpoint not being fixable (#1632)
• Fixed CPU DDP breaking change and DDP change (#1635)
• Tested pickling (#1636)

Contributors

@justusschock, @quinor, @williamFalcon

We had a few (subtle) bugs that affected DDP and a few key things in 0.7.2 so we released 0.7.3 to fix them because they are critical for DDP. sorry about that! still, no API changes, but please do skip straight to 0.7.3 upgrade for those fixes

Detail changes

Added

• Added rank_zero_warn for warning only in rank 0 (#1428)

Fixed

• Fixed default DistributedSampler for DDP training (#1425)
• Fixed workers warning not on windows (#1430)
• Fixed returning tuple from run_training_batch (#1431)
• Fixed gradient clipping (#1438)
• Fixed pretty print (#1441)

Contributors

@alsrgv, @Borda, @williamFalcon

Monir bug fix with print issues and data_loader (#1080)

This release focused on a ton of bug fixes, small optimizations to training but most importantly, clean new docs!

Major changes

We have released New documentation, please bear with us as we fix broken links and patch in missing pieces. This project moved to new org PyTorchLightning, so no longer the root sits on WilliamFalcon/PyTorchLightning. We have added own custom Tensorboard logger as default logger. We have upgrade Continues Integration to speed up the automatic testing. We have fixed GAN training - supporting multiple optimizers.

Complete changelog

Added

• Added support for resuming from a specific checkpoint via resume_from_checkpoint argument (#516)
• Added support for ReduceLROnPlateau scheduler (#320)
• Added support for Apex mode O2 in conjunction with Data Parallel (#493)
• Added option (save_top_k) to save the top k models in the ModelCheckpoint class (#128)
• Added on_train_start and on_train_end hooks to ModelHooks (#598)
• Added TensorBoardLogger (#607)
• Added support for weight summary of model with multiple inputs (#543)
• Added map_location argument to load_from_metrics and load_from_checkpoint (#625)
• Added option to disable validation by setting val_percent_check=0 (#649)
• Added NeptuneLogger class (#648)
• Added WandbLogger class (#627)

Changed

• Changed the default progress bar to print to stdout instead of stderr (#531)
• Renamed step_idx to step, epoch_idx to epoch, max_num_epochs to max_epochs and min_num_epochs to min_epochs (#589)
• Renamed several Trainer atributes: (#567)
  • total_batch_nb to total_batches,
  • nb_val_batches to num_val_batches,
  • nb_training_batches to num_training_batches,
  • max_nb_epochs to max_epochs,
  • min_nb_epochs to min_epochs,
  • nb_test_batches to num_test_batches,
  • and nb_val_batches to num_val_batches (#567)
• Changed gradient logging to use parameter names instead of indexes (#660)
• Changed the default logger to TensorBoardLogger (#609)
• Changed the directory for tensorboard logging to be the same as model checkpointing (#706)

Deprecated

• Deprecated max_nb_epochs and min_nb_epochs (#567)
• Deprecated the on_sanity_check_start hook in ModelHooks (#598)

Removed

• Removed the save_best_only argument from ModelCheckpoint, use save_top_k=1 instead (#128)

Fixed

• Fixed a bug which ocurred when using Adagrad with cuda (#554)
• Fixed a bug where training would be on the GPU despite setting gpus=0 or gpus=[] (#561)
• Fixed an error with print_nan_gradients when some parameters do not require gradient (#579)
• Fixed a bug where the progress bar would show an incorrect number of total steps during the validation sanity check when using multiple validation data loaders (#597)
• Fixed support for PyTorch 1.1.0 (#552)
• Fixed an issue with early stopping when using a val_check_interval < 1.0 in Trainer (#492)
• Fixed bugs relating to the CometLogger object that would cause it to not work properly (#481)
• Fixed a bug that would occur when returning -1 from on_batch_start following an early exit or when the batch was None (#509)
• Fixed a potential race condition with several processes trying to create checkpoint directories (#530)
• Fixed a bug where batch 'segments' would remain on the GPU when using truncated_bptt > 1 (#532)
• Fixed a bug when using IterableDataset (#547](https://github.com/PyTorchLightning/pytorch-lightning/pull/547))
• Fixed a bug where .item was called on non-tensor objects (#602)
• Fixed a bug where Trainer.train would crash on an uninitialized variable if the trainer was run after resuming from a checkpoint that was already at max_epochs (#608)
• Fixed a bug where early stopping would begin two epochs early (#617)
• Fixed a bug where num_training_batches and num_test_batches would sometimes be rounded down to zero (#649)
• Fixed a bug where an additional batch would be processed when manually setting num_training_batches (#653)
• Fixed a bug when batches did not have a .copy method (#701)
• Fixed a bug when using log_gpu_memory=True in Python 3.6 (#715)
• Fixed a bug where checkpoint writing could exit before completion, giving incomplete checkpoints (#689)
• Fixed a bug where on_train_end was not called when early stopping (#723)

Contributors

@akhti, @alumae, @awaelchli, @Borda, @borisdayma, @ctlaltdefeat, @dreamgonfly, @elliotwaite, @fdiehl, @goodok, @haossr, @HarshSharma12, @Ir1d, @jakubczakon, @jeffling, @kuynzereb, @MartinPernus, @matthew-z, @MikeScarp, @mpariente, @neggert, @rwesterman, @ryanwongsa, @schwobr, @tullie, @vikmary, @VSJMilewski, @williamFalcon, @YehCF

If we forgot someone due to not matching commit email with GitHub account, let us know :]