law.target.remote

Base classes and tools for defining and working with remote targets.

Class RemoteFileSystem

class RemoteFileSystem(file_interface, validate_copy=False, use_cache=False, cache_config=None, local_fs=None, **kwargs)

Bases: FileSystem

classmethod split_remote_kwargs(kwargs, include=None, skip=None)

Takes keyword arguments kwargs, splits them into two separate dictionaries depending on their content, and returns them in a tuple. The first one will contain arguments related to potential remote file operations (e.g. "cache" or "retries"), while the second one will contain all remaining arguments. This function is used internally to decide which arguments to pass to target formatters. include (skip) can be a list of argument keys that are considered as well (ignored).

exists(path, stat=False, **kwargs)

Return True if file or directory at path exist, False otherwise

Parameters:

path (str) – a path within the FileSystem to check for existence.

isdir(path, rstat=None, **kwargs)

Return True if the location at path is a directory. If not, return False.

Parameters:

path (str) – a path within the FileSystem to check as a directory.

Note: This method is optional, not all FileSystem subclasses implements it.

remove(path, **kwargs)

Remove file or directory at location path

Parameters:

• path (str) – a path within the FileSystem to remove.

• recursive (bool) – if the path is a directory, recursively remove the directory and all of its descendants. Defaults to True.

mkdir(path, perm=None, recursive=True, **kwargs)

Create directory at location path

Creates the directory at path and implicitly create parent directories if they do not already exist.

Parameters:

• path (str) – a path within the FileSystem to create as a directory.

• parents (bool) – Create parent directories when necessary. When parents=False and the parent directory doesn’t exist, raise luigi.target.MissingParentDirectory

• raise_if_exists (bool) – raise luigi.target.FileAlreadyExists if the folder already exists.

listdir(path, pattern=None, type=None, **kwargs)

Return a list of files rooted in path.

This returns an iterable of the files rooted at path. This is intended to be a recursive listing.

Parameters:

path (str) – a path within the FileSystem to list.

Note: This method is optional, not all FileSystem subclasses implements it.

copy(src, dst, perm=None, dir_perm=None, **kwargs)

Copy a file or a directory with contents. Currently, LocalFileSystem and MockFileSystem support only single file copying but S3Client copies either a file or a directory as required.

move(src, dst, perm=None, dir_perm=None, **kwargs)

Move a file, as one would expect.

Class RemoteTarget

class RemoteTarget(path, fs, **kwargs)

Bases: FileSystemTarget

fs = None

directory_class

alias of RemoteDirectoryTarget

file_class

alias of RemoteFileTarget

Class RemoteFileTarget

class RemoteFileTarget(path, fs, **kwargs)

Bases: FileSystemFileTarget, RemoteTarget

Class RemoteDirectoryTarget

class RemoteDirectoryTarget(path, fs, **kwargs)

Bases: FileSystemDirectoryTarget, RemoteTarget

Class RemoteFileInterface

class RemoteFileInterface(base=None, bases=None, retries=0, retry_delay=0, random_base=True, **kwargs)

Bases: object

abstract exists(path, base=None, stat=False, **kwargs)

Returns True when the path exists and False otherwise. When stat is True, returns the stat object or None.

abstract stat(path, base=None, **kwargs)

Returns a stat object or raises an exception when path does not exist.

abstract isdir(path, stat=None, base=None, **kwargs)

Returns True when path refers to an existing directory, optionally using a precomputed stat object instead, and False otherwise.

abstract isfile(path, stat=None, base=None, **kwargs)

Returns True when path refers to a existing file, optionally using a precomputed stat object instead, and False otherwise.

abstract chmod(path, perm, base=None, silent=False, **kwargs)

Changes the permission of a path to perm. Raises an exception when path does not exist or returns False when silent is True, and returns True on success.

abstract unlink(path, base=None, silent=True, **kwargs)

Removes a file at path. Raises an exception when path does not exist or returns False when silent is True, and returns True on success.

abstract rmdir(path, base=None, silent=True, **kwargs)

Removes a directory at path. Raises an exception when path does not exist or returns False when silent is True, and returns True on success.

abstract remove(path, base=None, silent=True, **kwargs)

Removes any file or directory at path. Directories are removed recursively. Raises an exception when path does not exist or returns False when silent is True, and returns True on success.

abstract mkdir(path, perm, base=None, silent=True, **kwargs)

Creates a directory at path with permissions perm. Raises an exception when path already exists or returns False when silent is True, and returns True on success.

abstract mkdir_rec(path, perm, base=None, **kwargs)

Recursively creates a directory and intermediate missing directories at path with permissions perm. Raises an exception when path already exists or returns False when silent is True, and returns True on success.

abstract listdir(path, base=None, **kwargs)

Returns a list of elements in and relative to path.

abstract filecopy(src, dst, base=None, **kwargs)

Copies a file from src to dst. Returns the full, schemed src and dst URIs used for copying in a 2-tuple.

Class RemoteCache

class RemoteCache(*args, **kwargs)

Bases: object