law.task.base

Custom luigi base task definitions.

Class BaseRegister

class BaseRegister(classname, bases, classdict)

Bases: Register

Class BaseTask

class BaseTask(*args, **kwargs)

Bases: Task

classmethod deregister(task_cls=None)

Removes a task class task_cls from the luigi task register. When None, this class is used. Task family strings and patterns are accepted as well. True is returned when at least one class was successfully removed, and False otherwise.

classmethod modify_param_args(params, args, kwargs)

Hook to modify command line arguments before they are event created within get_param_values().

classmethod modify_param_values(params)

Hook to modify command line arguments before instances of this class are created.

classmethod get_param_values(params, args, kwargs)

Get the values of the parameters from the args and kwargs.

Parameters:

• params – list of (param_name, Parameter).

• args – positional arguments

• kwargs – keyword arguments.

Returns:

list of (name, value) tuples, one for each parameter.

complete()

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

input()

Returns the outputs of the Tasks returned by requires()

See Task.input

Returns:

a list of Target objects which are specified as outputs of all required Tasks.

abstract run()

The task run method, to be overridden in a subclass.

See Task.run

property live_task_id

The task id depends on the task family and parameters, and is generated by luigi once in the constructor. As the latter may change, this property returns to the id with the current set of parameters.

Class Register

class Register(classname, bases, classdict)

Bases: BaseRegister

Class Task

class Task(*args, **kwargs)

Bases: BaseTask

Class WrapperTask

class WrapperTask(*args, **kwargs)

Bases: Task

Use for tasks that only wrap other tasks and that by definition are done if all their requirements exist.

complete()

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

output()

The output that this Task produces.

The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single Target or a list of Target instances.

Implementation note

If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.

See Task.output

run()

The task run method, to be overridden in a subclass.

See Task.run

Class ExternalTask

class ExternalTask(*args, **kwargs)

Bases: Task

run = None