Configuration

Configuration#

Please note that configuration options of law contrib packages might also appear in sections of the general law configuration.

Features of the law configuration parser#

The law home directory#

The law home directory is the default location of a small number of files and directories that are created when working with law. Please refer to the particular configuration options below for more information. The law home directory defaults to $LAW_HOME or $HOME/.law.

Configuration file resolution order#

Law configuration files are looked up in a certain order: $LAW_CONFIG_FILE, ./law.cfg, config in the law home directory, or etc/law/config. The config parser only reads the first existing file. Please also note the optional synchronization with environment variables described below.

Inheritance and extensions#

The law configuration can inherit from or can be extended by additional configuration files. See the options inherit and extend in the [core] section of the law configuration below. The only difference between the two is that options defined in inherit do not overwrite those in the actual config, whereas options of extend do (options of extended config files are added after those in the actual config).

Option referencing#

To avoid that some configurations are copied to other options (which is required in some cases), the law configuration parser supports references. The syntax for references to options in the same section is &::other_option, and &::other_section::other_option for options in other sections. None is returned in case a reference cannot be resolved. The same applies to circular references.

Environment variable expansion#

Configuration values that contain environment variables are automatically expanded by most getters of the law configuration parser, such as get_expanded(), get_expanded_int(), etc. If desired, you can avoid this expansion by backslash-escaping characters such as $ or ~ to \$ or \~.

The “None” value#

Getters of the law configuration parser such as get_default(), get_expanded()``, get_expanded_int(), etc., are configured to return a default value (which itself defaults to None (the NoneType)) in case an option is not existing. Sometimes it can be useful to actively declare an option as missing. This can be achieved by setting an option to "None" (string).

Synchronization with environment variables#

In some scenarios it might be useful to control the value of some options via environment variables, for instance, when testing features or prototyping outside of an existing configuration file. For these cases, a synchronization mechanism is implemented that reads variables named LAW__the_section__the_option and overwrites the respective option with its value. This is enabled by default, but can be configured by the sync_env option in the [core] section.

Synchronization with luigi configuration#

law is built on top of luigi and does not replace it. Therefore, another luigi-specfic configuration file is actually required. However, law provides a mechanism to define the luigi configuration right within the law configuration file. All sections starting with luigi_, e.g. [luigi_worker] are forwarded to luigi. Environment variable expansion is performed as well. This behavior is configurable by the sync_luigi_config option in the [core] section.

law configuration#

[core]#

law_home

  • Description: The law home directory is the default location of a small number of files and directories that are created when working with law.

  • Type: str

  • Default: "$LAW_HOME", "$HOME/.law"

index_file

  • Description: The location of the file that contains a list of all tasks, their module, and their parameters for the purpose of fast autocompletion on the command line.

  • Type: str

  • Default: "$LAW_INDEX_FILE", "$LAW_HOME/index"

software_dir

  • Description: The default location where softwar is copied to by the law software command. This can be especially useful for (e.g.) forwarding law and its dependencies into a container. Type law software --help for more info.

  • Type: str

  • Default: "$LAW_SOFTWARE_DIR", "$LAW_HOME/software"

inherit

  • Description: A comma-separated list of files that this config is inheriting, i.e., their options do not overwrite the ones defined in this file. Relative paths are resolved with respect to this file. Brace expansion, such as "myfile{1,2}.cfg" to "myfile1.cfg", "myfile2.cfg" is applied.

  • Type: str, list[str], None

  • Default: None

extend

  • Description: A comma-separated list of files that this config is extended by, i.e., their options do overwrite the ones defined in this file. Relative paths are resolved with respect to this file. Brace expansion, such as "myfile{1,2}.cfg" to "myfile1.cfg", "myfile2.cfg" is applied.

  • Type: str, list[str], None

  • Default: None

sync_env

  • Description: A boolean flag that decides whether the config should be synchronized with environment variables that have the format LAW__the_section__the_option. When True, the synchronization takes place after all defaults and config files were read.

  • Type: bool

  • Default: True

sync_luigi_config

  • Description: A boolean flag that decides whether sections starting with "luigi_" should be synchronized with the luigi configuration.

  • Type: bool

  • Default: True

[logging]#

law

  • Description: The log level of the law root logger. Accepted values are the default python log levels, i.e., "NOTSET", "DEBUG", "INFO", "WARNING", "ERROR", or "CRITICAL".

  • Type: str

  • Default: "$LAW_LOG_LEVEL", "WARNING"

gfal2

  • Description: Operations on remote targets are performed using the gfal2 package. This value defines the level of the gfal2 logger.

  • Type: str

  • Default: "WARNING"

Other loggers can be configured conveniently by adding my_logger: my_level to this section. For instance, law.workflow.remote: DEBUG will set the log level of the logger defined in law/workflow/remote.py to "DEBUG". Loggers that are not part of the law namespace are supported as well.

[modules]#

This section only accepts options without values. They are interpreted as a list of modules that are loaded and checked for law tasks classes when law index is executed. This command creates the index file that contains a list of task names and their parameters for fast autocompletion on the command line.

[task]#

colored_repr

  • Description: A boolean flag that decides whether the output of task.repr() and repr(task) is colorized by default.

  • Type: bool

  • Default: False

colored_str

  • Description: A boolean flag that decides whether the output of str(task) is colorized by default.

  • Type: bool

  • Default: True

interactive_format

  • Description: The style of interactive task outputs, e.g. when adding --print-status N to a command. Possible values: "plain", "compact", "fancy", "fancy_compact".

  • Type: str

  • Default: "fancy"

interactive_line_breaks

  • Description: A boolean flag that decides whether smart line breaks should be used when printing interactive task outputs to terminal. Automatically disabled when the terminal width cannot be determined.

  • Type: bool

  • Default: True

interactive_line_width

  • Description: The width of interactive task output when interactive_line_breaks is enabled. When zero or negative, the terminal size is obtained dynamically if possible.

  • Type: int

  • Default: 0

interactive_status_skip_seen

  • Description: A boolean flag that decides whether during interactive task status traversal, e.g. when adding --print-status N to a command, tasks whose status has already been checked are not checked again in case they appear more than once in the dependency tree.

  • Type: bool

  • Default: False

[target]#

colored_repr

  • Description: A boolean flag that decides whether the output of target.repr() and repr(target) is colorized by default.

  • Type: bool

  • Default: False

colored_str

  • Description: A boolean flag that decides whether the output of str(target) is colorized by default.

  • Type: bool

  • Default: True

expand_path_repr

  • Description: A boolean flag that decides whether environment variables in paths of file targets are expanded in repr(target) and str(target).

  • Type: bool

  • Default: False

filesize_repr

  • Description: A boolean flag that decides whether the file size is part of the representation for targets inheriting from law.FileSystemTarget.

  • Type: bool

  • Default: False

default_local_fs
tmp_dir

  • Description: The directory in which temporary local targets are stored.

  • Type: str

  • Default: "$LAW_TARGET_TMP_DIR", "tempfile.gettempdir()"

tmp_dir_perm

  • Description: When the tmp_dir does not exist, it is created with this permission. When None or when has_permissions is False, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: 0o0770

collection_remove_threads Desciption: Number of threads to use for removing targets in “TargetCollection“‘s in parallel. No multi-threading is used when the number is 0. Type: int Default: 0

interactive_removal_local_sync

  • Description: Whether the interactive removal via “–remove-output” of “MirroredTarget“‘s should enforce synchronization with the local representation.

  • Type: bool

  • Default: True

Options of contrib packages#

default_wlcg_fs

  • Description: The name of the section to lookup for getting the configuration of a law.wlcg.WLCGFileSystem.

  • Type: str

  • Default: "wlcg_fs"

default_dropbox_fs

  • Description: The name of the section to lookup for getting the configuration of a law.dropbox.DropboxFileSystem.

  • Type: str

  • Default: "dropbox_fs"

[local_fs]#

base

  • Description: The base path for accessing files on this file system. Its default refers to the root of the file system which should serve most uses cases where local file targets assigned to it define absolute paths on their own. However, setting a different value will prepend it to the paths used in all its operations.

  • Type: str

  • Default: "/"

Options defined by law.target.file.FileSystem#

has_permissions

  • Description: A boolean flag that signifies whether the file system should handle permissions, i.e., whether chmod operations should be evaluated. When False, the current umask is used.

  • Type: bool

  • Default: True

default_file_perm

  • Description: The default permission of newly created files. When None, or when has_permissions is False, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: None

default_dir_perm

  • Description: The default permission of newly created directories. When None or when has_permissions is False, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: None

create_file_dir

  • Description: A boolean flag that signifies whether directories should be created when a file is created via open, copy, move or dump operations.

  • Type: bool

  • Default: True

Options defined by law.target.local.LocalFileSystem#

, local_root_depth

  • Description: Number of fragments of the absolute local path to check when local targets are used in law.MirroredTarget’s. The check is done pre-emptively on the local file system to ensure that a mount point with a path of the configured depth exists.

  • Type: integer

  • Default: 1

[wlcg_fs]#

Note: Remote file operations are handled by GFAL2. For more info, see cern-fts/gfal2.

Options defined by law.target.file.FileSystem#

has_permissions

  • Description: A boolean flag that signifies whether the file system should handle permissions, i.e., whether chmod operations should be evaluated. When False, the current umask is used.

  • Type: bool

  • Default: True

default_file_perm

  • Description: The default permission of newly created files. When None or when has_permissions` is ``False, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: None

default_dir_perm

  • Description: The default permission of newly created directories. When None or when has_permissions is False, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: None

create_file_dir

  • Description: A boolean flag that signifies whether directories should be created when a file is created via open, copy, move or dump operations. When False, the gfal_transfer_create_parent options should be set to True to ensure that intermediate directories are created.

  • Type: bool

  • Default: False

Options defined by law.target.remote.RemoteFileInterface#

base

  • Description: The base path for accessing files on a WLCG storage element. It should with a protocol, such as "root://" or "gsiftp://", that can be handled by gfal. This value is required for the law.wlcg.WLCGFileSystem to work. Brace expansion into a list of paths is applied.

  • Type: str, list[str], None

  • Default: None

base_stat, base_exists, base_chmod, base_unlink, base_rmdir, base_mkdir, base_listdir, base_filecopy

  • Description: Different base paths for certain file or directory operations. When empty, the value of base is used for that operation. As an example, if you use a base path starting with "srm://...", you might want to choose a different protocol for listdir operations as SRM is too inefficient for directory listings (runtime linear to number of items). Brace expansion into a list of paths is applied.

  • Type: str, list[str], None

  • Default: None

retries

  • Description: The default number of retries to perform when a file operation fails. The value can also be set per operation.

  • Type: int

  • Default: 1

retry_delay

  • Description: The default amount of time to wait before retrying a failed file operation. The default unit is seconds. The value can also be set per operation.

  • Type: int, str

  • Default: "5s"

random_base Desciption: A boolean flag that, if a base path above is given as a list of paths, decides whether, by default, a random path should be chosen per file operation. When True and a file operation fails, the next attempt is made with a different yet still random base path. The exact behavior can also be set per operation. Type: bool Default: True

Options defined by law.target.remote.RemoteFileSystem#

validate_copy

  • Description: A boolean flag that decides whether, by default, a validation should be performed after each file copy (or move) operation. The exact behavior can also be set per operation.

  • Type: bool

  • Default: False

use_cache

  • Description: A boolean flag that decides whether, by default, certain operations (copy, move, open, etc.) should use the local cache (when existing). The exact behavior can also be set per operation.

  • Type: bool

  • Default: False

Options defined by law.target.remote.RemoteCache#

cache_root

  • Description: The directory in which cached files from remote file operation should be stored. When None or empty, file caching is disabled. When "__TMP__", a temporary directory is created.

  • Type: str, None

  • Default: None

cache_cleanup

  • Description: A boolean flag that decides whether the cache should be removed when the program terminates.

  • Type: bool

  • Default: False

cache_max_size

  • Description: The maximum size of the cache directory. When a file should be cached, but the maximum size is reached, files are deleted from the cache in order of their smallest modification time until enough disk space is freed. The default unit is MB. When zero or negative, the maximum cache size is equal to 90% of the available disk space.

  • Type: int, str

  • Default: "0MB"

cache_mtime_patience

  • Description: The maximum amount of seconds that the modification time of remote of cached files can differ before considering a file invalid. A negative value means causes all files to be considered valid, independent of modification times. This feature might only be useful for file systems that are accessible via multiple protocols that might respond to stat requests with different timestamps for files that are known to be identical (e.g. large, identical files that are provided simultaneously by multiple central storage sites).

  • Type: float

  • Default: 1.0

cache_file_perm

  • Description: The permission of files in the cache. When None or empty, the current umask is used. The value can be expressed by an octal number.

  • Type: int

  • Default: 0o0660

cache_dir_perm

  • Description: The permission of the cache root directory in case it is not existing yet and has to be created. When None or empty, the current umask is used. The value can be expressed by an octal number.

  • Type: int

  • Default: 0o0770

cache_wait_delay

  • Description: The amount of time to wait for a file in the cache to be unlocked. The default unit is seconds.

  • Type: int, str

  • Default: "5s"

cache_max_waits

  • Description: The maximum number of times to wait for a duration of cache_wait_delay for a file in the cache to be unlocked before an error is thrown that the file is unavailable.

  • Type: int

  • Default: 120

cache_global_lock

  • Description: A boolean flag that decides whether each cache operation is locked by a global locking mechanism that prevents simultaneous cache access.

  • Type: bool

  • Default: False

Options defined by law.gfal.GFAlFileInterface#

gfal_atomic_contexts

  • Description: A boolean flag that decides whether each operation of the gfal2 interface should be placed in its own gfal2 context. For the law.wlcg.WLCGFileSystem it is highly recommended to set this option to True to enfore that FTP connections are closed as soon as possible and remote resources are freed properly.

  • Type: bool

  • Default: True

gfal_transfer_timeout

  • Description: The number of seconds after which file operations should be considered timed out.

  • Type: int

  • Default: 3600

gfal_transfer_checksum_check

  • Description: A boolean flag that decides whether checksums should be checked after operations that create new files.

  • Type: bool

  • Default: False

gfal_transfer_nbstreams

  • Description: The number of parallel streams for copying files.

  • Type: int

  • Default: 1

gfal_transfer_overwrite

  • Description: A boolean flag that decides whether existing files should by overwritten by operations that would recreate them.

  • Type: bool

  • Default: True

gfal_transfer_create_parent

  • Description: A boolean flag that decides whether intermediate, missing directories should be created automatically in certan file operations.

  • Type: bool

  • Default: True

gfal_transfer_strict_copy

  • Description: A boolean flag that decides whether file copying should be done without source and destination checks.

  • Type: bool

  • Default: False

[dropbox_fs]#

Note: Remote file operations are handled by GFAL2. For more info, see cern-fts/gfal2. To access files on your Dropbox, you to have to register an application to obtain credentials. Visit https://www.dropbox.com/developers for more information.

Options defined by law.dropbox.DropboxFileSystem#

app_key
app_secret
access_token

Options defined by law.target.file.FileSystem#

has_permissions

  • Description: A boolean flag that signifies whether the file system should handle permissions, i.e., whether chmod operations should be evaluated. When False, the current umask is used.

  • Type: bool

  • Default: True

default_file_perm

  • Description: The default permission of newly created files. When None or empty, or when has_permissions is False, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: None

default_dir_perm

  • Description: The default permission of newly created directories. When None or empty, or when has_permissions is False, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: None

create_file_dir

  • Description: A boolean flag that signifies whether directories should be created when a file is created via open, copy, move or dump operations. When False, the gfal_transfer_create_parent options should be set to True to ensure that intermediate directories are created.

  • Type: bool

  • Default: False

Options defined by law.target.remote.RemoteFileInterface#

base

  • Description: The base path for accessing files on your Dropbox. Normally, it should with "dropbox://dropbox.com/...". This value is required for the law.dropbox.DropboxFileSystem to work.

  • Type: str, list[str], None

  • Default: None

base_stat, base_exists, base_chmod, base_unlink, base_rmdir, base_mkdir, base_mkdir_rec, base_listdir, base_filecopy

  • Description: Different base paths for certain file or directory operations. When empty, the value of base is used for that operation. For the law.dropbox.DropboxFileSystem, setting different base paths is usually not required as Dropbox does not provide different entry points per operation type.

  • Type: str, list[str], None

  • Default: None

retries

  • Description: The default number of retries to perform when a file operation fails. The value can also be set per operation.

  • Type: int

  • Default: 1

retry_delay

  • Description: The default amount of time to wait before retrying a failed file operation. The default unit is seconds. The value can also be set per operation.

  • Type: int, str

  • Default: "5s"

random_base Desciption: A boolean flag that, if a base path above is given as a list of paths, decides whether, by default, a random path should be chosen per file operation. When True and a file operation fails, the next attempt is made with a different yet still random base path. The exact behavior can also be set per operation. Type: bool Default: True

Options defined by law.target.remote.RemoteFileSystem#

validate_copy

  • Description: A boolean flag that decides whether, by default, a validation should be performed after each file copy (or move) operation. The exact behavior can also be set per operation.

  • Type: bool

  • Default: False

use_cache

  • Description: A boolean flag that decides whether, by default, certain operations (copy, move, open, etc.) should use the local cache (when existing). The exact behavior can also be set per operation.

  • Type: bool

  • Default: False

Options defined by law.target.remote.RemoteCache#

cache_root

  • Description: The directory in which cached files from remote file operation should be stored. When None or empty, file caching is disabled. When "__TMP__", a temporary directory is created.

  • Type: str, None

  • Default: None

cache_cleanup

  • Description: A boolean flag that decides whether the cache should be removed when the program terminates.

  • Type: bool

  • Default: False

cache_max_size

  • Description: The maximum size of the cache directory. When a file should be cached, but the maximum size is reached, files are deleted from the cache in order of their smallest modification time until enough disk space is freed. The default unit is MB. When zero or negative, the maximum cache size is equal to 90% of the available disk space.

  • Type: int, str

  • Default: "0MB"

cache_file_perm

  • Description: The permission of files in the cache. When None or empty, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: 0o0660

cache_dir_perm

  • Description: The permission of the cache root directory in case it is not existing yet and has to be created. When None or empty, the current umask is used. The value can be expressed by an octal number.

  • Type: int, None

  • Default: 0o0770

cache_wait_delay

  • Description: The amount of time to wait for a file in the cache to be unlocked. The default unit is seconds.

  • Type: int, str

  • Default: "5s"

cache_max_waits

  • Description: The maximum number of times to wait for a duration of cache_wait_delay for a file in the cache to be unlocked before an error is thrown that the file is unavailable.

  • Type: int

  • Default: 120

cache_global_lock

  • Description: A boolean flag that decides whether each cache operation is locked by a global locking mechanism that prevents simultaneous cache access.

  • Type: bool

  • Default: False

Options defined by law.gfal.GFAlFileInterface#

gfal_atomic_contexts

  • Description: A boolean flag that decides whether each operation of the gfal2 interface should be placed in its own gfal2 context.

  • Type: bool

  • Default: True

gfal_transfer_timeout

  • Description: The number of seconds after which file operations should be considered timed out.

  • Type: int

  • Default: 3600

gfal_transfer_checksum_check

  • Description: A boolean flag that decides whether checksums should be checked after operations that create new files.

  • Type: bool

  • Default: False

gfal_transfer_nbstreams

  • Description: The number of parallel streams for copying files.

  • Type: int

  • Default: 1

gfal_transfer_overwrite

  • Description: A boolean flag that decides whether existing files should by overwritten by operations that would recreate them.

  • Type: bool

  • Default: True

gfal_transfer_create_parent

  • Description: A boolean flag that decides whether intermediate, missing directories should be created automatically in certan file operations.

  • Type: bool

  • Default: True

gfal_transfer_strict_copy

  • Description: A boolean flag that decides whether file copying should be done without source and destination checks.

  • Type: bool

  • Default: False

[job]#

job_file_dir

  • Description: The location of the directory in which a job file factory saves temporary files that are used for job submission.

  • Type: str

  • Default: "$LAW_JOB_FILE_DIR", tempfile.gettempdir()

job_file_dir_mkdtemp

  • Description: A boolean flag or string that decides if a temporary directory inside job_file_dir should be created in which the actual job files are stored. It is discouraged to set this option to False in order to have a clear separation between files created by different job file factories. When a string is used it serves as a prefix for name of the temporary directory. More than three X’s will be replaced with random characters and template variables “{{task_id}}” and “{{task_family}}” will be expanded with task values.

  • Type: bool, str

  • Default: True

job_file_dir_cleanup

  • Description: A boolean flag that decides whether a job file factory should remove its file diretory once it is destructed. Set this option to False when the job submission system requires all jobs files to be present as long as jobs are running, or in case you want to debug the files later on.

  • Type: bool

  • Default: False

job_query_timeout

  • Description: If set, this option controls the maximum duration a job status query can take until it is deemed hanging and terminated consequently. The default unit is seconds.

  • Type: integer, string

  • Default: None

Options of contrib packages#

arc_job_file_dir, arc_job_file_dir_mkdtemp, arc_job_file_dir_cleanup, arc_job_query_timeout

  • Description: These three options are identical to the ones above without the "arc" prefix, but only apply to the law.arc.ARCJobFileFactory. When None or not existing, the values above are used.

arc_chunk_size_submit

  • Description: Number of jobs that can be submitted in parallel inside a single call to law.arc.ARCJobManager.submit, i.e., in a single arcsub command.

  • Type: int

  • Default: 25

arc_chunk_size_cancel

  • Description: Number of jobs that can be cancelled in parallel inside a single call to law.arc.ARCJobManager.cancel, i.e., in a single arckill command.

  • Type: int

  • Default: 25

arc_chunk_size_cleanup

  • Description: Number of jobs that can be cancelled in parallel inside a single call to law.arc.ARCJobManager.cleanup, i.e., in a single arcclean command.

  • Type: int

  • Default: 25

arc_chunk_size_query

  • Description: Number of jobs whose status can be queried in parallel inside a single call to law.arc.ARCJobManager.query, i.e., in a single arcstat command.

  • Type: int

  • Default: 25

arc_cmd_arcsub

  • Description: Custom “arcsub” command.

  • Type: string

  • Default: arcsub

arc_cmd_arckill

  • Description: Custom “arckill” command.

  • Type: string

  • Default: arckill

arc_cmd_arcclean

  • Description: Custom “arcclean” command.

  • Type: string

  • Default: arcclean

arc_cmd_arcstat

  • Description: Custom “arcstat” command.

  • Type: string

  • Default: arcstat

glite_job_file_dir, glite_job_file_dir_mkdtemp, glite_job_file_dir_cleanup, glite_job_query_timeout

  • Description: These three options are identical to the ones above without the "glite" prefix, but only apply to the law.glite.GLiteJobFileFactory. When None or not existing, the values above are used.

glite_chunk_size_cancel

  • Description: Number of jobs that can be cancelled in parallel inside a single call to law.glite.GLiteJobManager.cancel, i.e., in a single glite-ce-job-cancel command.

  • Type: int

  • Default: 25

glite_chunk_size_cleanup

  • Description: Number of jobs that can be cancelled in parallel inside a single call to law.glite.GLiteJobManager.cleanup, i.e., in a single glite-ce-job-purge command.

  • Type: int

  • Default: 25

glite_chunk_size_query

  • Description: Number of jobs whose status can be queried in parallel inside a single call to law.glite.GLiteJobManager.query, i.e., in a single glite-ce-job-status command.

  • Type: int

  • Default: 25

glite_cmd_submit

  • Description: Custom “glite-ce-job-submit” command.

  • Type: string

  • Default: glite-ce-job-submit

glite_cmd_cancel

  • Description: Custom “glite-ce-job-cancel” command.

  • Type: string

  • Default: glite-ce-job-cancel

glite_cmd_purge

  • Description: Custom “glite-ce-job-purge” command.

  • Type: string

  • Default: glite-ce-job-purge

glite_cmd_status

  • Description: Custom “glite-ce-job-status” command.

  • Type: string

  • Default: glite-ce-job-status

htcondor_job_file_dir, htcondor_job_file_dir_mkdtemp, htcondor_job_file_dir_cleanup, htcondor_job_query_timeout

  • Description: These three options are identical to the ones above without the "htcondor" prefix, but only apply to the law.htcondor.HTCondorJobFileFactory. When None or not existing, the values above are used. The only exception is htcondor_job_file_dir_cleanup whose default value is False.

htcondor_job_grouping_submit

  • Description: Whether to use job grouping (cluster submission in HTCondor nomenclature) or not. If not, the standard batched submission is used and settings such as “htcondor_chunk_size_submit” and “htcondor_merge_job_files” are considered.

  • Type: boolean

  • Default: True

htcondor_chunk_size_submit

  • Description: Number of jobs that can be submitted in parallel inside a single call to law.htcondor.HTCondorJobManager.submit, i.e., in a single “condor_submit” command. Ignored when job grouping is enabled in “htcondor_job_grouping_submit”.

  • Type: int

  • Default: 25

htcondor_chunk_size_cancel
htcondor_chunk_size_query

  • Description: Number of jobs whose status can be queried in parallel inside a single call to law.htcondor.HTCondorJobManager.query, i.e., in a single condor_q or condor_history command.

  • Type: int

  • Default: 25

htcondor_merge_job_files

  • Description: A boolean flag that decides whether multiple job description files should be merged into a single file before submission. Ignored when job grouping is enabled in “htcondor_job_grouping_submit”. When “False”, the “htcondor_chunk_size_submit” option is not considered either.

  • Type: bool

  • Default: True

htcondor_cmd_q

  • Description: Custom “condor_q” command.

  • Type: string

  • Default: condor_q

htcondor_cmd_history

  • Description: Custom “condor_history” command.

  • Type: string

  • Default: condor_history

htcondor_cmd_submit

  • Description: Custom “condor_submit” command.

  • Type: string

  • Default: condor_submit

htcondor_cmd_rm

  • Description: Custom “condor_rm” command.

  • Type: string

  • Default: condor_rm

lsf_job_file_dir, lsf_job_file_dir_mkdtemp, lsf_job_file_dir_cleanup, lsf_job_query_timeout

  • Description: These three options are identical to the ones above without the "lsf" prefix, but only apply to the law.lsf.LSFJobFileFactory. When None or not existing, the values above are used. The only exception is lsf_job_file_dir_cleanup whose default value is False.

lsf_chunk_size_cancel

  • Description: Number of jobs that can be cancelled in parallel inside a single call to law.lsf.LSFJobManager.cancel, i.e., in a single bkill command.

  • Type: int

  • Default: 25

lsf_chunk_size_query

  • Description: Number of jobs whose status can be queried in parallel inside a single call to law.lsf.LSFJobManager.query, i.e., in a single bjobs command.

  • Type: int

  • Default: 25

lsf_cmd_bsub

  • Description: Custom “bsub” command.

  • Type: string

  • Default: bsub

lsf_cmd_bkill

  • Description: Custom “bkill” command.

  • Type: string

  • Default: bkill

lsf_cmd_bjobs

  • Description: Custom “bjobs” command.

  • Type: string

  • Default: bjobs

slurm_job_file_dir, slurm_job_file_dir_mkdtemp, slurm_job_file_dir_cleanup, slurm_job_query_timeout

  • Description: These three options are identical to the ones above without the "slurm" prefix, but only apply to the law.slurm.SlurmJobFileFactory. When None or not existing, the values above are used. The only exception is slurm_job_file_dir_cleanup whose default value is False.

slurm_chunk_size_cancel

  • Description: Number of jobs that can be cancelled in parallel inside a single call to law.slurm.SlurmJobManager.cancel, i.e., in a single scancel command.

  • Type: int

  • Default: 25

slurm_chunk_size_query

  • Description: Number of jobs whose status can be queried in parallel inside a single call to law.slurm.SlurmJobManager.query, i.e., in a single squeue or sacct command.

  • Type: int

  • Default: 25

slurm_cmd_sbatch

  • Description: Custom “sbatch” command.

  • Type: string

  • Default: sbatch

slurm_cmd_scancel

  • Description: Custom “scancel” command.

  • Type: string

  • Default: scancel

slurm_cmd_squeue

  • Description: Custom “squeue” command.

  • Type: string

  • Default: squeue

slurm_cmd_sacct

  • Description: Custom “sacct” command.

  • Type: string

  • Default: sacct

crab_job_file_dir, crab_job_file_dir_mkdtemp, crab_job_file_dir_cleanup, crab_job_query_timeout

  • Description: These three options are identical to the ones above without the "crab" prefix, but only apply to the law.cms.CrabJobFileFactory. When None or not existing, the values above are used.

crab_work_area

  • Description: The directory in which the law.cms.CrabJobManager will create crab project directories upon submission.

  • Type: str

  • Default: job.crab_job_file_dir

crab_sandbox_name

  • Description: The name of the cmssw sandbox to use which provides the crab executable. In its simplest form, this is just the version string of the CMSSW release to use. However, additional settings can be appended with the "::" delimiter in the format "cmssw_version::setting=value::setting=value" for more configurablity. Supported settings are "setup", an additional setup script that is executed inside the src directory during installation, "dir", a custom install directory, "arch", a custom scram architecture, and "cores", the number of CPU cores to use for installation.

  • Type: str

  • Default: "CMSSW_14_2_1::arch=el9_amd64_gcc12"

crab_password_file

  • Description: A file containing the X509 certificate password for automatic proxy delegations by the law.crab.CrabJobManager.

  • Type: str, None

  • Default: None

crab_cmd_crab

  • Description: Custom “crab” command.

  • Type: string

  • Default: crab

[notifications]#

mail_recipient
mail_sender
mail_smtp_host
mail_smtp_port

Options of contrib packages#

slack_token

  • Description: The token used for notifications sent by law.slack.notify_slack.

  • Type: str, None

  • Default: None

slack_channel

  • Description: The channel used for notifications sent by law.slack.notify_slack.

  • Type: str, None

  • Default: None

slack_mention_user

  • Description: When set, notifications sent by law.slack.notify_slack mention this user explicitly.

  • Type: str, None

  • Default: None

telegram_token

  • Description: The token used for notifications sent by law.telegram.notify_telegram.

  • Type: str, None

  • Default: None

telegram_chat

  • Description: The chat used for notifications sent by law.telegram.notify_telegram.

  • Type: str, None

  • Default: None

telegram_mention_user

  • Description: When set, notifications sent by law.telegram.notify_telegram mention this user explicitly.

  • Type: str, None

  • Default: None

mattermost_hook_url Desciption: Address of the mattermost webhook to use. For more info, see https://developers.mattermost.com/integrate/webhooks/incoming. Type: str, None Default: None

mattermost_header Desciption: A custom header to add to mattermost notifications. Not used when empty. Type: str, None Default: None

mattermost_channel Desciption: A custom channel name to send the notification to. When empty, the default configured for the webhook is used. Type: str, None Default: None

mattermost_user Desciption: A custom user name to send the notification with. When empty, the default configured for the webhook is used. Type: str, None Default: None

mattermost_mention_user Desciption: If set, the user with that name is mentioned. Type: str, None Default: None

mattermost_icon_url Desciption: The address of a custom icon to use. When empty, the default configured for the webhook is used. Type: str, None Default: “https://media.githubusercontent.com/media/riga/law/refs/heads/master/assets/logo_profile.png”

mattermost_icon_emoji Desciption: Name of a custom emoji to use instead of the icon url. Type: str, None Default: None

mattermost_success_emoji Desciption: Emoji to append to the status text in case of success. Type: str, None Default: “:tada:”

mattermost_failure_emoji Desciption: Emoji to append to the status text in case of a failure. Type: str, None Default: “:rotating_light:”

[bash_sandbox]#

Note: This section defines the default options for all bash sandboxes. To configure options per bash initialization file, create a section [bash_sandbox_<init_file>] with the desired options.

stagein_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and to which input files are staged-in outside of the sandbox and provided to the sandboxed task. When None, no stage-in is performed and the task is assumed to be able to access inputs directly from within the sandbox.

  • Type: str, None

  • Default: "stagein"

stageout_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and provided to sandboxed tasks to store outputs and from which those files are staged-out outside the sandbox. When None, no stage-out is performed and the task is assumed to be able to store outputs directly from within the sandbox.

  • Type: str, None

  • Default: "stageout"

law_executable

  • Description: The law executable to use within sandboxes, e.g. law or python -m law.

  • Type: str

  • Default: "law"

login

  • Description: A boolean flag that decides whether the bash sandbox should be invoked as a login shell.

  • Type: bool

  • Default: False

[bash_sandbox_env]#

Note: Here you can define environment variables via key-value pairs that are accessible in a bash sandbox. When an option has no value, i.e., when only a key is given, the value of the variable of the host environment is used. Also note that, unless escaped with a blackslach, "$" and "~" are expanded with variables of the outer task environment. The environment variables defined in this section are applied to all bash sandboxes. To configure variables per bash initialization file, create a section [bash_sandbox_env_<init_file>] with the desired values.

[venv_sandbox]#

Note: This section defines the default options for all venv sandboxes. To configure options per bash initialization file, create a section [venv_sandbox_<venv_dir>] with the desired options.

stagein_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and to which input files are staged-in outside of the sandbox and provided to the sandboxed task. When None, no stage-in is performed and the task is assumed to be able to access inputs directly from within the sandbox.

  • Type: str, None

  • Default: "stagein"

stageout_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and provided to sandboxed tasks to store outputs and from which those files are staged-out outside the sandbox. When None, no stage-out is performed and the task is assumed to be able to store outputs directly from within the sandbox.

  • Type: str, None

  • Default: "stageout"

law_executable

  • Description: The law executable to use within sandboxes, e.g. law or python -m law.

  • Type: str

  • Default: "law"

[venv_sandbox_env]#

Note: Here you can define environment variables via key-value pairs that are accessible in a venv sandbox. When an option has no value, i.e., when only a key is given, the value of the variable of the host environment is used. Also note that, unless escaped with a blackslach, "$" and "~" are expanded with variables of the outer task environment. The environment variables defined in this section are applied to all venv sandboxes. To configure variables per venv directory, create a section [venv_sandbox_env_<venv_dir>] with the desired values.

[docker_sandbox]#

Note: This section defines the default options for all docker sandboxes. To configure options per image, create a section [docker_sandbox_<image_name>] with the desired options.

stagein_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and to which input files are staged-in outside of the sandbox and provided to the sandboxed task. When None, no stage-in is performed and the task is assumed to be able to access inputs directly from within the sandbox.

  • Type: str, None

  • Default: "stagein"

stageout_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and provided to sandboxed tasks to store outputs and from which those files are staged-out outside the sandbox. When None, no stage-out is performed and the task is assumed to be able to store outputs directly from within the sandbox.

  • Type: str, None

  • Default: "stageout"

law_executable

  • Description: The law executable to use within sandboxes, e.g. law or python -m law.

  • Type: str

  • Default: "law"

uid

  • Description: The user id of the account inside the docker container. When None, the container is started with its default value.

  • Type: int, None

  • Default: None

gid

  • Description: The grouo id of the account inside the docker container. When None, the container is started with its default value.

  • Type: int, None

  • Default: None

forward_dir

  • Description: The container directory in which forwarded files and directories are located.

  • Type: str

  • Default: "/law_forward"

python_dir

  • Description: The container directory in which forwarded python modules are located, relative to forward_dir.

  • Type: str

  • Default: "py"

bin_dir

  • Description: The container directory in which forwarded executables are located, relative to forward_dir.

  • Type: str

  • Default: "bin"

[docker_sandbox_env]#

Note: Here you can define environment variables via key-value pairs that are accessible in a docker sandbox. When an option has no value, i.e., when only a key is given, the value of the variable of the host environment is used. Also note that, unless escaped with a blackslach, "$" and "~" are expanded with variables of the outer task environment. The environment variables defined in this section are applied to all docker sandboxes. To configure variables per image, create a section [docker_sandbox_env_<image_name>] with the desired values.

[docker_sandbox_volumes]#

Note: Here you can define volumes that are mounted in docker containers via key-value pairs that represent host and container directory, respectively. The volumes defined in this section are applied to all docker sandboxes. To configure volumes per image, create a section [docker_sandbox_volumes_<image_name>] with the desired values.

[singularity_sandbox]#

Note: This section defines the default options for all singularity sandboxes. To configure options per image, create a section [singularity_sandbox_<image_name>] with the desired options.

stagein_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and to which input files are staged-in outside of the sandbox and provided to the sandboxed task. When None, no stage-in is performed and the task is assumed to be able to access inputs directly from within the sandbox.

  • Type: str, None

  • Default: "stagein"

stageout_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and provided to sandboxed tasks to store outputs and from which those files are staged-out outside the sandbox. When None, no stage-out is performed and the task is assumed to be able to store outputs directly from within the sandbox.

  • Type: str, None

  • Default: "stageout"

law_executable

  • Description: The law executable to use within sandboxes, e.g. law or python -m law.

  • Type: str

  • Default: "law"

uid

  • Description: The user id of the account inside the docker container. When None, the container is started with its default value.

  • Type: int, None

  • Default: None

gid

  • Description: The group id of the account inside the docker container. When None, the container is started with its default value.

  • Type: int, None

  • Default: None

forward_dir

  • Description: The container directory in which forwarded files and directories are located.

  • Type: str

  • Default: "/law_forward"

python_dir

  • Description: The container directory in which forwarded python modules are located, relative to forward_dir.

  • Type: str

  • Default: "py"

bin_dir

  • Description: The container directory in which forwarded executables are located, relative to forward_dir.

  • Type: str

  • Default: "bin"

allow_binds

  • Description: A boolean flag that decides whether singularity binds are allowed.

  • Type: bool

  • Default: True

forward_law

  • Description: A boolean flag that decides whether the law python package and executable should be forwarded into the conatiner. When False, the container is assumed to have law accessible.

  • Type: bool

  • Default: True

[singularity_sandbox_env]#

Note: Here you can define environment variables via key-value pairs that are accessible in a singularity sandbox. When an option has no value, i.e., when only a key is given, the value of the variable of the host environment is used. Also note that, unless escaped with a blackslach, "$" and "~" are expanded with variables of the outer task environment. The environment variables defined in this section are applied to all singularity sandboxes. To configure variables per image, create a section [singularity_sandbox_env_<image_name>] with the desired values.

[singularity_sandbox_volumes]#

Note: Here you can define volumes that are bound to singularity containers via key-value pairs that represent host and container directory, respectively. The volumes defined in this section are applied to all singularity sandboxes. To configure volumes per image, create a section [singularity_sandbox_volumes_<image_name>] with the desired values.

[cmssw_sandbox]#

Note: This section defines the default options for all cmssw sandboxes. To configure options per cmssw version, create a section [cmssw_sandbox_<cmssw_version>] with the desired options.

stagein_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and to which input files are staged-in outside of the sandbox and provided to the sandboxed task. When None, no stage-in is performed and the task is assumed to be able to access inputs directly from within the sandbox.

  • Type: str, None

  • Default: "stagein"

stageout_dir_name

  • Description: Name of a directory which is placed automatically inside a temporary directory and provided to sandboxed tasks to store outputs and from which those files are staged-out outside the sandbox. When None, no stage-out is performed and the task is assumed to be able to store outputs directly from within the sandbox.

  • Type: str, None

  • Default: "stageout"

law_executable

  • Description: The law executable to use within sandboxes, e.g. law or python -m law.

  • Type: str

  • Default: "law"

login

  • Description: A boolean flag that decides whether the bash beneath the sandbox should be invoked as a login shell.

  • Type: bool

  • Default: False

[cmssw_sandbox_env]#

Note: Here you can define environment variables via key-value pairs that are accessible in a cmssw sandbox. When an option has no value, i.e., when only a key is given, the value of the variable of the host environment is used. Also note that, unless escaped with a blackslach, "$" and "~" are expanded with variables of the outer task environment. The environment variables defined in this section are applied to all cmssw sandboxes. To configure variables per cmssw version, create a section [cmssw_sandbox_env_<cmssw_version>] with the desired values.

Contents