law.target.file#
Custom luigi file system and target objects.
Class FileSystem
#
- class FileSystem(name=None, has_permissions=True, default_file_perm=None, default_dir_perm=None, create_file_dir=True, **kwargs)[source]#
Bases:
FileSystem
- abstract exists(path, stat=False, **kwargs)[source]#
Return
True
if file or directory atpath
exist,False
otherwise- Parameters:
path (str) – a path within the FileSystem to check for existence.
- abstract isdir(path, **kwargs)[source]#
Return
True
if the location atpath
is a directory. If not, returnFalse
.- Parameters:
path (str) – a path within the FileSystem to check as a directory.
Note: This method is optional, not all FileSystem subclasses implements it.
- abstract remove(path, recursive=True, silent=True, **kwargs)[source]#
Remove file or directory at location
path
- abstract mkdir(path, perm=None, recursive=True, silent=True, **kwargs)[source]#
Create directory at location
path
Creates the directory at
path
and implicitly create parent directories if they do not already exist.- Parameters:
path (str) – a path within the FileSystem to create as a directory.
parents (bool) – Create parent directories when necessary. When parents=False and the parent directory doesn’t exist, raise luigi.target.MissingParentDirectory
raise_if_exists (bool) – raise luigi.target.FileAlreadyExists if the folder already exists.
- abstract listdir(path, pattern=None, type=None, **kwargs)[source]#
Return a list of files rooted in path.
This returns an iterable of the files rooted at
path
. This is intended to be a recursive listing.- Parameters:
path (str) – a path within the FileSystem to list.
Note: This method is optional, not all FileSystem subclasses implements it.
Class FileSystemTarget
#
- class FileSystemTarget(path, fs=None, **kwargs)[source]#
Bases:
Target
,FileSystemTarget
- file_class#
alias of
FileSystemFileTarget
- directory_class#
alias of
FileSystemDirectoryTarget
- exists(**kwargs)[source]#
Returns
True
if the path for this FileSystemTarget exists;False
otherwise.This method is implemented by using
fs
.
- remove(silent=True, **kwargs)[source]#
Remove the resource at the path specified by this FileSystemTarget.
This method is implemented by using
fs
.
- abstract property fs#
The
FileSystem
associated with this FileSystemTarget.
Class FileSystemFileTarget
#
- class FileSystemFileTarget(path, fs=None, **kwargs)[source]#
Bases:
FileSystemTarget
- open(mode, **kwargs)[source]#
Open the FileSystem target.
This method returns a file-like object which can either be read from or written to depending on the specified mode.
- Parameters:
mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.
Class FileSystemDirectoryTarget
#
- class FileSystemDirectoryTarget(path, fs=None, **kwargs)[source]#
Bases:
FileSystemTarget
- open = None#
Functions#
- localize_file_targets(struct, *args, **kwargs)[source]#
Takes an arbitrary struct of targets, opens the contexts returned by their
FileSystemFileTarget.localize()
implementations and yields their localized representations in the same structure as passed in struct. When the context is closed, the contexts of all localized targets are closed.