Checkpointing¶

It is sometimes the case that an optimization proceeds for a longer time than is available on computational infrastructure. For example, a shared high performance computing center with job limits of 72 hours. To address this, GloMPO incorporates checkpointing functionality. This constructs a snapshot of a managed optimization at some point in time and persists it to disk. This file can be loaded by a new GloMPOManager instance at a later time to resume the optimization.

Checkpointing options are provided to the GloMPOManager through a CheckpointingControl instance.

Checkpointing tries to create an entire image of the GloMPO state, it is the user’s responsibility to ensure that the used optimizers are restartable.

Tip

Within tests/test_optimizers.py there is the TestSubclassesGlompoCompatible class which can be used to ensure an optimizer is compatible with all of GloMPO’s functionality.

The optimization task can sometimes not be reduced to a pickled state depending on it complexity and interfaces to other codes. GloMPO will first attempt to pickle the object, failing that GloMPO will attempt to call checkpoint_save(). If this also fails, the checkpoint is created without the optimization task. GloMPO can be restarted from an incomplete checkpoint if the missing components are provided.

Similarly to manual stopping of optimizers (see User Interventions), manual checkpoints can also be requested by creating a file named CHKPT in the working directory. Note, that this file will be deleted by the manager when the checkpoint is created.

Checkpointing & Log Files¶

Caution

Please pay close attention to how GloMPO handles log files and loading checkpoints.

The HDF5 log file is not included inside the checkpoints since they can become extremely large if they are being used to gather lots of data. GloMPO always aims to continue an optimization by appending data to a matching log file rather than creating a new one. To do this, the following conditions must be met:

A log file called glompo_log.h5 must be present in the working directory.
The log file must contain a key matching the one in the checkpoint.

If a file named glompo_log.h5 is not present then a warning is issued and GloMPO will begin logging into a new file of this name.

If the file exists but does not contain a matching key an error will be raised.

It is the user’s responsibility to ensure that log files are located and named correctly in the working directory when loading checkpoints.

Caution

GloMPO will overwrite existing data in if a matching log is found in the working directory, but it contains more iteration information that the checkpoint.

For example, a checkpoint was created at the 1000th function evaluation of an optimization, but the manager continued until convergence after 1398 function evaluations. If the checkpoint is loaded, it will expect a the log file to only have 1000 function evaluations.

The only way to load this checkpoint (and ensure duplicate iterations are not included in the log) is to remove any values in the log which were generated after the checkpoint. To avoid data being overwritten, the user can manually copy/rename the log file they wish to retain before loading a checkpoint.

Checkpointing & Visualisation¶

If you are visualizing the optimization (see GloMPO Scope), it is unfortunately not possible to continue saving a movie through a checkpoint. One can continue the visualisation through the checkpoint but the recording will go into a new file. More specifically, it will go into a file with the same name as was configured before the checkpoint, this means that the pre-checkpoint movie file is likely to be overwritten. Thus, to save the first movie file, the user should rename it before continuing an optimization from a checkpoint, or change GloMPOManager.visualisation_args when loading the checkpoint. It is possible to stitch together several movies into a single file at a later stage using ffmpeg.

Checkpointing Control Settings¶

class glompo.core.checkpointing.CheckpointingControl(checkpoint_time_frequency: float = inf, checkpoint_iter_frequency: float = inf, checkpoint_at_init: bool = False, checkpoint_at_conv: bool = False, raise_checkpoint_fail: bool = False, force_task_save: bool = False, keep_past: int = -1, naming_format: str = 'glompo_checkpoint_%(date)_%(time)', checkpointing_dir: Union[pathlib.Path, str] = 'checkpoints')[source]¶

Class to setup and control the checkpointing behaviour of the GloMPOManager. This class has limited functionality and is mainly a container for various settings. The initialisation arguments match the class attributes of the same name.

checkpoint_at_conv¶

If True a checkpoint is built when the manager reaches convergence and before it exits.

Type:	bool

checkpoint_at_init¶

If True a checkpoint is built at the very start of the optimization. This can make starting duplicate jobs easier.

Type:	bool

checkpoint_iter_frequency¶

Frequency (in number of function evaluations) with which GloMPO will save its state to disk during an optimization. Function call based checkpointing not performed if this parameter is not provided.

Type:	float

checkpoint_time_frequency¶

Frequency (in seconds) with which GloMPO will save its state to disk during an optimization. Time based checkpointing not performed if this parameter is not provided.

Type:	float

checkpointing_dir¶

Directory in which checkpoints are saved. Defaults to 'checkpoints'

Important

This path is always converted to an absolute path, if a relative path is provided it will be relative to the current working directory when this object is created. There is no relation to GloMPOManager.working_dir.

Type:	Union[pathlib.Path, str]

count¶

Counter for checkpoint naming patterns which rely on incrementing filenames.

Type:	int

force_task_save¶

Some tasks may pickle successfully but fail to load properly, if this is an issue then setting this parameter to True will cause the manager to bypass the pickle task step and immediately attempt the checkpoint_save() method.

Type:	bool

keep_past¶

The number of newest checkpoints retained when a new checkpoint is made. Any older ones are deleted. Default is -1 which performs no deletion. keep_past = 0 retains no previous results, only the newly constructed checkpoint will exist.

Note

GloMPO will only count the directories in checkpointing_dir and matching the supplied naming_format.
Existing checkpoints will only be deleted if the new checkpoint is successfully constructed.

Type:	int

naming_format¶

Convention used to name the checkpoints. Special keys that can be used:

Naming Format Key Checkpoint Name Result

'%(date)' Current calendar date in YYYYMMDD format

'%(year)' Year formatted to YYYY

'%(yr)' Year formatted to YY

'%(month)' Numerical month formatted to MM

'%(day)' Calendar day of the month formatted to DD

'%(time)' Current calendar time formatted to HHMMSS (24-hour style)

'%(hour)' Hour formatted to HH (24-hour style)

'%(min)' Minutes formatted to MM

'%(sec)' Seconds formatted to SS

'%(count)' Index count of the number of checkpoints constructed. Count starts from the largest existing match in checkpointing_dir or zero otherwise. Formatted to 3 digits.

Type:	str

raise_checkpoint_fail¶

If True a failed checkpoint will cause the manager to end the optimization in error. Note, that GloMPO will always write out some data when it terminates. This can be a way of preserving data if the checkpoint fails. If False an error in constructing a checkpoint will simply raise a warning and pass.

Type:	bool

get_name() → str[source]¶: Returns a new name for a checkpoint matching the naming format.

matches_naming_format(name: str) → bool[source]¶: Returns True if the provided name matches the pattern in the naming_format.