law.parameter

Custom luigi parameters.

Data:

NO_STR	String value denoting an empty parameter.
NO_INT	Integer value denoting an empty parameter.
NO_FLOAT	Float value denoting an empty parameter.

Functions:

is_no_param(value)	Checks whether a parameter value denotes an empty parameter, i.e., if the value is either NO_STR, NO_INT, or NO_FLOAT.
get_param(value[, default])	Returns the passed value when it does not refer to an empty parameter value, checked with is_no_param().

Classes:

Parameter(*args[, parse_empty])	Custom base class of law-based parameters that adds additional features.
TaskInstanceParameter(*args, **kwargs)	Parameter that can be used to pass the instance of a task.
OptionalBoolParameter(*args, **kwargs)	Same as luigi's BoolParameter (and unlike luigi's OptionalBoolParameter) but parses and serializes "None" strings transparently to None values and vice-versa.
DurationParameter([unit])	Parameter that interprets a string (or float) value as a duration, represented by a float number with a configurable unit.
BytesParameter([unit])	Parameter that interprets a string (or float) value as a number of bytes, represented by a float number with a configurable unit.
CSVParameter(*args, **kwargs)	__init__(*args, cls=luigi.Parameter, inst=None, unique=False, sort=False, min_len=None,
MultiCSVParameter(*args, **kwargs)	__init__(*args, cls=luigi.Parameter, inst=None, unique=False, sort=False, min_len=None,
RangeParameter(*args[, require_start, ...])	Parameter that parses a range in the format start:stop and returns a tuple with two integer elements.
MultiRangeParameter(*args[, require_start, ...])	Parameter that parses several integer ranges (each in the format start-end), separated by comma, and produces a nested tuple.
NotifyParameter(*args, **kwargs)	Base class for notification parameters.
NotifyMultiParameter([parameters])	Parameter that takes multiple other NotifyParameter's to join their notification functionality in a single parameter.
NotifyMailParameter(*args, **kwargs)	Notification parameter defining a basic email transport.

NO_STR = 'NO_STR'

String value denoting an empty parameter.

NO_INT = -1

Integer value denoting an empty parameter.

NO_FLOAT = -1.0

Float value denoting an empty parameter.

is_no_param(value)

Checks whether a parameter value denotes an empty parameter, i.e., if the value is either NO_STR, NO_INT, or NO_FLOAT.

get_param(value, default=None)

Returns the passed value when it does not refer to an empty parameter value, checked with is_no_param(). Otherwise, default is returned, which defaults to None.

class Parameter(*args, parse_empty=False, **kwargs)

Bases: Parameter

Custom base class of law-based parameters that adds additional features.

As per luigi’s default behavior, passing empty strings on the command line for a parameter leads to its default value and parsing is not triggered. When parse_empty is True, the parser is still called and should implement a custom behavior.

All other args and kwargs are passed to luigi.Parameter.

parse_empty

type: bool

Whether the parameter parsing should be triggered for empty command line arguments (usually empty strings, but not for *None*s).

class TaskInstanceParameter(*args, **kwargs)

Bases: Parameter

Parameter that can be used to pass the instance of a task. This class does not implement parameter value parsing.

class OptionalBoolParameter(*args, **kwargs)

Bases: BoolParameter, Parameter

Same as luigi’s BoolParameter (and unlike luigi’s OptionalBoolParameter) but parses and serializes "None" strings transparently to None values and vice-versa.

class DurationParameter(unit='s', *args, **kwargs)

Bases: Parameter

Parameter that interprets a string (or float) value as a duration, represented by a float number with a configurable unit. unit is forwarded as both the unit and input_unit argument of law.util.parse_duration() which is used for the conversion. For optimal precision, value serialization uses law.util.human_duration() with colon_format. Example:

p = DurationParameter(unit="s")
p.parse("5")                      # -> 5.0 (using the unit implicitly)
p.parse("5s")                     # -> 5.0
p.parse("5m")                     # -> 300.0
p.parse("05:10")                  # -> 310.0
p.parse("5 minutes, 15 seconds")  # -> 310.0
p.serialize(310)                  # -> "05:15"

p = DurationParameter(unit="m")
p.parse("5")                      # -> 5.0 (using the unit implicitly)
p.parse("5s")                     # -> 0.083
p.parse("5m")                     # -> 5.0
p.parse("05:10")                  # -> 5.167
p.parse("5 minutes, 15 seconds")  # -> 5.25
p.serialize(310)                  # -> "05:15:00"

For more info, see law.util.parse_duration() and law.util.human_duration().

class BytesParameter(unit='MB', *args, **kwargs)

Bases: Parameter

Parameter that interprets a string (or float) value as a number of bytes, represented by a float number with a configurable unit. unit is forwarded as both the unit and input_unit argument of law.util.parse_bytes() which is used for the conversion. Example:

p = BytesParameter(unit="MB")
p.parse("5")                      # -> 5.0 (using the unit implicitly)
p.parse("5 MB")                   # -> 5.0
p.parse(5)                        # -> 5.0
p.parse("1 GB")                   # -> 1024.0
p.serialize(310)                  # -> "310MB"

p = BytesParameter(unit="GB")
p.parse("5")                      # -> 5.0 (using the unit implicitly)
p.parse("1024 MB")                # -> 1.0
p.serialize("2048 MB")            # -> "2GB"

For more info, see law.util.parse_bytes() and law.util.human_bytes().

class CSVParameter(*args, **kwargs)

Bases: Parameter

__init__(*args, cls=luigi.Parameter, inst=None, unique=False, sort=False, min_len=None,

max_len=None, choices=None, brace_expand=False, escape_sep=True, force_tuple=True, **kwargs)

Parameter that parses a comma-separated value (CSV) and produces a tuple. cls (inst) can refer to an other parameter class (instance) that will be used to parse and serialize the particular items.

When unique is True, both parsing and serialization methods make sure that values are unique. sort can be a boolean or a function for sorting parameter values.

When min_len (max_len) is set to an integer, an error is raised in case the number of elements to serialize or parse (evaluated after potentially ensuring uniqueness) deceeds (exceeds) that value. Just like in luigi’s ChoiceParamater, choices can be a sequence of accepted values.

When brace_expand is True, brace expansion is applied, potentially extending the list of values. However, note that in this case commas that are not meant to act as a delimiter cannot be quoted in csv-style with double quotes, but they should rather be backslash-escaped instead. Unless escape_sep is False, escaped separators (comma) are not split when parsing strings and, likewise, separators contained in values to serialze are escaped.

By default, single values are parsed such that they result in a tuple containing a single item. However, when force_tuple is False, single values that do not end with a comma are not wrapped by a tuple. Likewise, during serialization they are converted to a string as is, whereas tuple containing only a single item will end with a trailing comma.

Example:

p = CSVParameter(cls=luigi.IntParameter)
p.parse("4,5,6,6")
# => (4, 5, 6, 6)
p.serialize((7, 8, 9))
# => "7,8,9"

# "," that should not be used as delimiter
p = CSVParameter()
p.parse("a,b,\"c,d\"")
# -> ("a", "b", "c,d")
# same as
p.parse("a,b,c\,d")
# -> ("a", "b", "c,d")

# uniqueness check
p = CSVParameter(cls=luigi.IntParameter, unique=True)
p.parse("4,5,6,6")
# => (4, 5, 6)

# length check
p = CSVParameter(cls=luigi.IntParameter, max_len=2)
p.parse("4,5,6")
# => ValueError

# choices
p = CSVParameter(cls=luigi.IntParameter, choices=(1, 2))
p.parse("2,3")
# => ValueError

# brace expansion
p = CSVParameter(cls=luigi.IntParameter, brace_expand=True)
# (note that with brace_expand enabled, the quoting if "," only works with back slashes)
p.parse("1{2,3,4}9")
# => (129, 139, 149)

# do not force tuples to wrap single values
p = CSVParameter(cls=luigi.IntParameter, force_tuple=False)
p.parse("1")
# => 1
# note: the result would be (1,) with force_tuple left at True (default)
p.parse("1,")
# => (1,)
p.serialize(1)
# => "1"
p.serialize((1,))
# => "1,"
p.serialize((1, 2))
# => "1,2"

Note

Due to the way instance caching is implemented in luigi, parameters should always have hashable, immutable values. Therefore, this parameter produces a tuple and, in particular, not a list. To avoid undesired side effects, the default value given to the constructor is also converted to a tuple.

_inst

type: cls

Instance of the luigi parameter class cls or inst directory, that is used internally for parameter parsing and serialization.

class MultiCSVParameter(*args, **kwargs)

Bases: CSVParameter

__init__(*args, cls=luigi.Parameter, inst=None, unique=False, sort=False, min_len=None,

max_len=None, choices=None, brace_expand=False, escape_sep=True, force_tuple=True, **kwargs)

Parameter that parses several comma-separated values (CSV), separated by colons, and produces a nested tuple. cls (inst) can refer to an other parameter class (instance) that will be used to parse and serialize the particular items.

Except for the additional support for multiple CSV sequences, the parsing and serialization implementation is based on CSVParameter, which also handles the features controlled by unique, sort, max_len, min_len, choices, brace_expand, escape_sep and force_tuple per sequence of values.

However, note that in this case colon characters that are not meant to act as a delimiter cannot be quoted in csv-style with double quotes, but they should rather be backslash-escaped instead. Unless escape_sep is False, escaped separators (colon) are not split when parsing strings and, likewise, separators contained in values to serialze are escaped.

Example:

p = MultiCSVParameter(cls=luigi.IntParameter)
p.parse("4,5:6,6")
# => ((4, 5), (6, 6))
p.serialize((7, 8, (9,)))
# => "7,8:9"

# ":" that should not be used as delimiter
p = MultiCSVParameter()
p.parse("a,b:\"c:d\"")
# -> (("a", "b"), ("c:d",))
# same as
p.parse("a,b:c\:d")
# -> (("a", "b"), ("c:d",))

# uniqueness check
p = MultiCSVParameter(cls=luigi.IntParameter, unique=True)
p.parse("4,5:6,6")
# => ((4, 5), (6,))

# length check
p = MultiCSVParameter(cls=luigi.IntParameter, max_len=2)
p.parse("4,5:6,7,8")
# => ValueError

# choices
p = MultiCSVParameter(cls=luigi.IntParameter, choices=(1, 2))
p.parse("1,2:2,3")
# => ValueError

# brace expansion
# (note that with brace_expand enabled, the quoting if ":" only works with back slashes)
p = MultiCSVParameter(cls=luigi.IntParameter, brace_expand=True)
p.parse("4,5:6,7,8{8,9}")
# => ((4, 5), (6, 7, 88, 89))

Note

Due to the way instance caching is implemented in luigi, parameters should always have hashable values. Therefore, this parameter produces a (nested) tuple and, in particular, not a list. To avoid undesired side effects, the default value given to the constructor is also converted to a tuple.

_inst

type: cls

Instance of the luigi parameter class cls or inst directly, that is used internally for parameter parsing and serialization.

class RangeParameter(*args, require_start=True, require_end=True, single_value=False, **kwargs)

Bases: Parameter

Parameter that parses a range in the format start:stop and returns a tuple with two integer elements.

When require_start or require_stop are False, the formats :stop and start:, respectively, are accepted as well. In these cases, the tuple will contain an attribute OPEN do denote that either side is unconstrained.

When single_value is True, single integer values are accepted and lead to a tuple with one element.

p = RangeParameter()
p.parse("4:8")
# => (4, 8)
p.serialize((5, 9))
# => "5:9"
p.parse("4:")
# => ValueError
p.parse("4")
# => ValueError

p = RangeParameter(require_start=False, require_stop=False)
p.parse("4:5")
# => (4, 5)
p.parse("4:")
# => (4, OPEN)
p.parse(":5")
# => (OPEN, 5)
p.parse(":")
# => (OPEN, OPEN)
p.serialize((OPEN, 8))
# => ":8"

p = RangeParameter(single_value=True)
p.parse("4")
# => (4,)
p.serialize((5,))
# => "4"

classattribute RANGE_SEP

type: string

Character used as a separator between range edges when parsing strings and serializing values. Defaults to ":".

classattribute OPEN

type: None

Value denoting open edges in parsed ranges.

Methods:

expand(range, **kwargs)

Expands range (as returned by parse()) to a sorted list of unique integers.

classmethod expand(range, **kwargs)

Expands range (as returned by parse()) to a sorted list of unique integers. Additional kwargs are forwarded to law.util.range_expand().

RangeParameter.expand((4, 8))
# -> [4, 5, 6, 7]

class MultiRangeParameter(*args, require_start=True, require_end=True, single_value=False, **kwargs)

Bases: RangeParameter

Parameter that parses several integer ranges (each in the format start-end), separated by comma, and produces a nested tuple.

Except for the additional support for multiple ranges, the parsing and serialization implementation is based on RangeParameter, which also handles the control of open edges with require_start and require_end, and the acceptance of single integer values with single_value.

Example:

p = MultiRangeParameter()
p.parse("4:8,12:14")
# => ((4, 8), (12, 14))
p.serialize(((5, 9), (13, 15)))
# => ""5:9,13:15""

classattribute MULTI_RANGE_SEP

type: string

Character used as a separator between ranges when parsing strings and serializing values. Defaults to ",".

Methods:

expand(ranges, **kwargs)

Expands ranges (as returned by parse()) to a sorted list of unique integers.

classmethod expand(ranges, **kwargs)

Expands ranges (as returned by parse()) to a sorted list of unique integers. Additional kwargs are forwarded to law.util.range_expand().

MultiRangeParameter.expand(((4, 8), (12, 14)))
# -> [4, 5, 6, 7, 8, 12, 13, 14]

class NotifyParameter(*args, **kwargs)

Bases: BoolParameter, Parameter

Base class for notification parameters. A notification parameter must provide a notification transport in get_transport(), e.g.

def get_transport(self):
    return {
        "func": notification_func,
        "raw": True,  # or False
    }

When a task has a specific notification parameter set to True and its run method is decorated with the law.notify() function, notification_func is called with at least three arguments: success, title and message. success is a boolean which is True when the decorated function did not raise an exception. title is always a string. When raw is False (the default), message is also a string. Otherwise, it is an ordered dictionary containing key value pairs describing the message content. All options passed to law.decorator.notify() are forwarded to notification_func as optional arguments.

Methods:

get_transport()

Method to configure the transport that is toggled by this parameter.

get_transport()

Method to configure the transport that is toggled by this parameter. Should return a dictionary with "func" and "raw" (optional) fields.

class NotifyMultiParameter(parameters=[], *args, **kwargs)

Bases: NotifyParameter

Parameter that takes multiple other NotifyParameter’s to join their notification functionality in a single parameter. Example:

class MyTask(law.Task):

    notify = law.NotifyMultiParameter(parameters=[
        law.NotifyMailParameter(significant=False),
        ...  # further NotifyParameters
    ])

class NotifyMailParameter(*args, **kwargs)

Bases: NotifyParameter

Notification parameter defining a basic email transport. Uses law.notification.notify_mail() internally.