Skip to content

Technical Overview & Definitions

Technical Diagram

To aid in understanding how model-ensembler works, Figure 1 provides a visual overview of how the configuration and templates interact to generate our ensemble.

In green, Config and Templates are highlighted, these are the files that are changed by the user.

In gray, the different processes that are generated from the user files are shown. For example, the pre-processing tasks are controlled by the pre-processing section of the config, and pre-run scripts within a batch are generated by a template called pre-run.j2.

The arrows indicate which process is generated from the respective config/template elements.

Definitions diagram, dark mode Definitions diagram, light mode

Figure 1: Visual overview of the definitions used throughout model-ensembler documentation. Boxes in green are edited by the user. Two batches are shown here for illustrative purposes.

As the user, you will be changing the Config and Templates to construct your ensemble. How to do this is expanded on in the Building Configuration and Building Templates sections.

Definitions

Before reading these, an overview of definitions used throughout the documentation:

  • a run: A run refers to the execution of a model, with a given set of parameters. A run will be submitted to an HPC backed (such as SLURM).
  • a batch: We refer to the collection of model runs as a batch. model-ensembler can be used to configure a single batch or a list of batches. The batch also includes common pre-and post batch tasks, which are common for all runs but not for all batches, as well as pre-and post run tasks which are unique for each run.
  • pre-processing: Common task(s) that are executed before all batch execution.
  • post-processing: Common task(s) that are executed after all batches are completed.

Also note, not pictured:

  • a job: Once a run has been submitted to an HPC backend (such as SLURM) as part of a batch, we define it as a job. It is distinguished from a run, as model-ensembler enables configuration that can control the number of jobs that are executed concurrently on an HPC cluster. This is independent from the run configuration, and follows the SLURM naming convention.