File Transfer Strategies
A set of classes with a shared interface to perform copy, download, and upload operations given two (source & destination) File objects that support various protocols.
Classes:
Blob([client_id, client_secret, tenant_id])
Implements FileTransferStrategy class to transfer files to/from Azure Blob Storage.
GCloud([credentials, project_id])
Implements FileTransferStrategy class to transfer files to/from Google Cloud Storage.
HTTP()
Implements Base FileTransferStrategy class to use HTTP to download files from public URLs.
Rsync([user, host, private_key_path])
Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems.
S3([credentials, profile, region_name]
Implements Base FileTransferStrategy class to upload/download files from S3 Bucket.
class covalent.fs_strategies.Blob
Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements FileTransferStrategy class to transfer files to/from Azure Blob Storage.
Parameters
client_id (Optional[str]) - ID of a service principal authorized to perform the transfer
client_secret (Optional[str]) - Corresponding secret key for the service principal credentials
tenant_id (Optional[str]) - The Azure Active Directory tenant ID which owns the cloud resources.
credentials
A tuple containing (client_id, client_secret, tenant_id)
Methods:
download(from_file[, to_file])
Download files or the contents of folders from Azure Blob Storage.
upload(from_file[, to_file])
Upload files or the contents of folders to Azure Blob Storage.
download
Download files or the contents of folders from Azure Blob Storage.
upload
Upload files or the contents of folders to Azure Blob Storage.
class covalent.fs_strategies.GCloud
Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements FileTransferStrategy class to transfer files to/from Google Cloud Storage.
Parameters
credentials(Optional[str]) - IPath to OAuth 2.0 credentials JSON file for a service account
project_id (Optional[str]) - ID of a project in GCP
credentials
String containing OAuth 2.0 credentials.
project_id
ID of a project in GCP
download
Return Type
callable
upload
Return Type
callable
class covalent.fs_strategies.HTTP
Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements Base FileTransferStrategy class to use HTTP to download files from public URLs.
class covalent.fs_strategies.Rsync
Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems. Rsync via ssh is used if one of the provided files is marked as remote.
user
(optional) Determine user to specify for remote host if using rsync with ssh
host
(optional) Determine what host to connect to if using rsync with ssh
private_key_path
(optional) Filepath for ssh private key to use if using rsync with ssh
Methods:
cp(from_file[, to_file])
RTYPE None
get_rsync_cmd(from_file, to_file[, …])
RTYPE str
get_rsync_ssh_cmd(local_file, remote_file[, …])
RTYPE str
RTYPE None
upload(from_file, to_file)
RTYPE None
cp
Return Type
None
get_rsync_cmd
Return Type
str
get_rsync_ssh_cmd
Return Type
str
return_subprocess_callable
Return Type
None
upload
Return Type
None
class covalent.fs_strategies.S3
Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements Base FileTransferStrategy class to upload/download files from S3 Bucket.
Methods:
download(from_file[, to_file])
Download files or the contents of folders from S3 bucket.
upload(from_file, to_file)
Upload files or folders to S3 bucket.