htcondor

htcondor#

HTCondor contrib functionality.

Class HTCondorWorkflow#

class HTCondorWorkflow(*args, **kwargs)#

Bases: BaseRemoteWorkflow

workflow_proxy_cls#

alias of HTCondorWorkflowProxy

htcondor_workflow_run_context()#

Hook to provide a context manager in which the workflow run implementation is placed. This can be helpful in situations where resurces should be acquired before and released after running a workflow.

abstract htcondor_output_directory()#

Hook to define the location of submission output files, such as the json files containing job data, and optional log files (in case htcondor_log_directory() is not defined). This method should return a FileSystemDirectoryTarget.

htcondor_log_directory()#

Hook to define the location of log files if any are written. When set, it has precedence over htcondor_output_directory() for log files. This method should return a FileSystemDirectoryTarget or a value that evaluates to False in case no custom log directory is desired.

htcondor_job_resources(job_num, branches)#

Hook to define resources for a specific job with number job_num, processing branches. This method should return a dictionary.

htcondor_dump_intermediate_job_data()#

Whether to dump intermediate job data to the job submission file while jobs are being submitted.

htcondor_post_submit_delay()#

Configurable delay in seconds to wait after submitting jobs and before starting the status polling.

htcondor_poll_callback(poll_data)#

Configurable callback that is called after each job status query and before potential resubmission. It receives the variable polling attributes poll_data (PollData) that can be changed within this method.

If False is returned, the polling loop is gracefully terminated. Returning any other value does not have any effect.

htcondor_post_poll_callback(success, duration)#

Configurable callback that is called after the polling loop has ended. It receives a boolean success that indicates whether the job polling was successful, and the duration of the job polling in seconds.

Class HTCondorJobManager#

class HTCondorJobManager(pool=None, scheduler=None, user=None, threads=1)#

Bases: BaseJobManager

cleanup(*args, **kwargs)#

Abstract atomic or group job cleanup. Can throw exceptions. Should return a dictionary mapping job ids to per-job return values.

cleanup_batch(*args, **kwargs)#

Cleans up a batch of jobs given by job_ids via a thread pool of size threads which defaults to its instance attribute. When chunk_size, which defaults to chunk_size_cleanup, is not negative, job_ids are split into chunks of that size which are passed to cleanup().

When callback is set, it is invoked after each successful job (or job chunk) cleaning with the index of the corresponding job id (starting at 0) and either None or an exception if any occurred. All other kwargs are passed to cleanup().

Exceptions that occured during job cleaning are stored in a list and returned. An empty list means that no exceptions occured.

submit(job_file, job_files=None, pool=None, scheduler=None, spool=False, retries=0, retry_delay=3, silent=False, _processes=None)#

Abstract atomic or group job submission. Can throw exceptions. Should return a list of job ids.

cancel(job_id, pool=None, scheduler=None, silent=False, _processes=None)#

Abstract atomic or group job cancellation. Can throw exceptions. Should return a dictionary mapping job ids to per-job return values.

query(job_id, pool=None, scheduler=None, user=None, silent=False, _processes=None)#

Abstract atomic or group job status query. Can throw exceptions. Should return a dictionary mapping job ids to per-job return values.

Class HTCondorJobFileFactory#

class HTCondorJobFileFactory(file_name='htcondor_job.jdl', command=None, executable=None, arguments=None, input_files=None, output_files=None, log='log.txt', stdout='stdout.txt', stderr='stderr.txt', postfix_output_files=True, postfix=None, universe='vanilla', notification='Never', custom_content=None, absolute_paths=False, **kwargs)#

Bases: BaseJobFileFactory

create(grouped_submission=False, **kwargs)#

Abstract job file creation method that must be implemented by inheriting classes.