Covalent
Main Covalent public functionality.
Classes
DepsBash([commands])
Shell commands to run before an electron
DepsCall([func, args, kwargs, …])
Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment
DepsPip([packages, reqs_path])
PyPI packages to be installed before executing an electron
Lepton([language, library_name, …])
A generalization of an Electron to languages other than Python.
TransportableObject(obj)
A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized.
Alias of covalent.RESULT_STATUS
Funcions
cancel(dispatch_id[, task_ids, dispatcher_addr])
Cancel a running dispatch.
dispatch(orig_lattice[, dispatcher_addr, …])
Wrapping the dispatching functionality to allow input passing and server address specification.
dispatch_sync(lattice[, dispatcher_addr])
Wrapping the synchronous dispatching functionality to allow input passing and server address specification.
electron([_func, backend, executor, files, …])
Electron decorator to be called upon a function.
get_config([entries])
Return a configuration setting.
get_result(dispatch_id[, wait, …])
Lattice decorator to be called upon a function.
lattice([_func, backend, executor, …])
Wrapping the dispatching functionality to allow input passing and server address specification.
redispatch(dispatch_id[, dispatcher_addr, …])
Reload the configuration from the TOML file.
Reload the configuration from the TOML file.
set_config(new_config[, new_value])
Update the configuration.
stop_triggers(dispatch_ids[, …])
Stop observing on all triggers of all given dispatch ids registered on the Triggers server.
sync([dispatch_id])
Synchronization call.
wait(child, parents)
Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched.
class covalent.DepsBash
Bases: covalent._workflow.deps.Deps
Shell commands to run before an electron
Deps class to encapsulate Bash dependencies for an electron.
The specified commands will be executed as subprocesses in the same environment as the electron.
Commands
A list of bash commands to execute before the electron runs.
Methods
from_dict(object_dict)
Rehydrate a dictionary representation
to_dict()
Return a JSON-serializable dictionary representation of self
Rehydrate a dictionary representation
Parameters
object_dict – a dictionary representation returned by to_dict
Return Type
DepsBash
Returns
self
Instance attributes will be overwritten.
class covalent.DepsCall
Bases: covalent._workflow.deps.Deps.
Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment.
Deps class to encapsulate python functions to be called in the same execution environment as the electron.
func
A callable
args
args list
kwargs
kwargs dict
retval_keyword
An optional string referencing the return value of func. If retval_keyword is specified, the return value of func will be passed during workflow execution as an argument to the electron corresponding to the parameter of the same name.
NOTES
Electron parameters to be injected during execution must have default parameter values. It is the user’s responsibility to ensure that retval_keyword is actually a parameter of the electron. Unexpected behavior may occur otherwise.
Methods
from_dict(object_dict)
Rehydrate a dictionary representation
to_dict()
Return a JSON-serializable dictionary representation of self
Rehydrate a dictionary representation
Parameters
object_dict – a dictionary representation returned by to_dict
Return Type
Returns
self
Instance attributes will be overwritten.
class covalent.DepsPip
Bases: covalent._workflow.deps.Deps
PyPI packages to be installed before executing an electron
A specification of Pip packages to be installed
packages
A list of PyPI packages to install
reqs_path
Path to requirements.txt (overrides packages).
These packages are installed in an electron’s execution environment just before the electron is run.
Methods
from_dict(object_dict)
Rehydrate a dictionary representation
to_dict()
Return a JSON-serializable dictionary representation of self
Rehydrate a dictionary representation
Parameters
object_dict – a dictionary representation returned by to_dict
Return Type
Returns
self
Instance attributes will be overwritten.
class covalent.Lepton
Bases: covalent._workflow.electron.Electron
A generalization of an Electron to languages other than Python.
Leptons inherit from Electrons, overloading the function attribute with a wrapper function. Users specify the foreign function’s signature as well as its location by providing a library and entrypoint. When one of the executors invokes the task’s function, the foreign function is called by way of the wrapper function defined here. If compilation scripts are required, these must be separately copied to the backend.
language
Language in which the task specification is written.
library_name
Name of the library or module which specifies the function.
function_name
Name of the foreign function.
argtypes
List of tuples specifying data types and input/output properties.
executor
Alternative executor object to be used for lepton execution. If not passed, the dask executor is used by default
files
An optional list of FileTransfer objects which copy files to/from remote or local filesystems
Methods
Return a lepton wrapper function.
INPUT = 0
INPUT_OUTPUT = 2
OUTPUT = 1
class covalent.TransportableObject
Bases: object
A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized. The object will also contain additional info like the python version used to serialize it.
Methods
deserialize(serialized, *[, header_only, …])
Deserialize the transportable object from the archived transportable object.
deserialize_dict(collection)
Recursively deserializes a dict of TransportableObjects.
deserialize_from_json(json_string)
Reconstruct a transportable object from JSON
deserialize_list(collection)
Recursively deserializes a list of TransportableObjects.
from_dict(object_dict)
Rehydrate a dictionary representation
Get the deserialized transportable object.
Get the deserialized transportable object.
make_transportable(obj)
RTYPE TransportableObject
Serialize the transportable object.
Serialize the transportable object to JSON.
to_dict()
Return a JSON-serializable dictionary representation of self
property attrs
Deserialize the transportable object.
Parameters
data (bytes) – Cloudpickled function.
Return Type
object
Returns
The deserialized transportable object.
Recursively deserializes a dict of TransportableObjects. More precisely, collection is a dict, each of whose entries is assumed to be either a TransportableObject, a list, or dict.
Return Type
dict
Reconstruct a transportable object from JSON.
Parameters
json_string (str) – A JSON string representation of a TransportableObject
Return Type
str
Returns
A TransportableObject instance
Recursively deserializes a list of TransportableObjects. More precisely, collection is a list, each of whose entries is assumed to be either a TransportableObject, a list, or dict
Return Type
list
Rehydrate a dictionary representation
Parameters
object_dict – a dictionary representation returned by to_dict
Return Type
Returns
A TransportableObject represented by object_dict
Get the deserialized transportable object.
Parameters
None
Return Type
The deserialized object/callable function.
Returns
function
Get the serialized transportable object.
Parameters
None
Return Type
The serialized transportable object.
Returns
object
property json
Return Type
object_string
python_version
Serialize the transportable object.
Parameters
None
Returns
The serialized object along with the python version.
Return Type
pickled_object
Serialize the transportable object to JSON.
Parameters
None
Returns
A JSON string representation of the transportable object.
Return Type
str
covalent.cancel
Cancel a running dispatch.
Parameters
dispatch_id (str) – The dispatch id of the dispatch to be canceled.
task_ids (Optional[List[int]]) – Optional, list of task ids to cancel within the workflow
dispatcher_addr (Optional[str]) – Dispatcher server address, if None then defaults to the address set in Covalent’s config.
Return Type
str
Returns
Cancelation response
covalent.dispatch
Wrapping the dispatching functionality to allow input passing and server address specification.
Afterwards, send the lattice to the dispatcher server and return the assigned dispatch id.
Parameters
orig_lattice (Lattice) – The lattice/workflow to send to the dispatcher server.
dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then defaults to the address set in Covalent’s config.
disable_run (bool) – Whether to disable running the worklow and rather just save it on Covalent’s server for later execution
Return Type
Callable
Returns
Wrapper function which takes the inputs of the workflow as arguments.
covalent.dispatch_sync
Wrapping the synchronous dispatching functionality to allow input passing and server address specification.
Afterwards, sends the lattice to the dispatcher server and return the result of the executed workflow.
Parameters
orig_lattice – The lattice/workflow to send to the dispatcher server.
dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then defaults to the address set in Covalent’s config.
Return Type
Callable
Returns
Wrapper function which takes the inputs of the workflow as arguments.
covalent.electron
Electron decorator to be called upon a function. Returns the wrapper function with the same functionality as _func.
Parameters
_func (Optional[Callable]) – function to be decorated
Keyword Arguments
backend
DEPRECATED: Same as executor
executor
Alternative executor object to be used by the electron execution. If not passed, the dask executor is used by default.
deps_bash
An optional DepsBash object specifying a list of shell commands to run before _func
deps_pip
An optional DepsPip object specifying a list of PyPI packages to install before running _func
call_before
An optional list of DepsCall objects specifying python functions to invoke before the electron
call_after
An optional list of DepsCall objects specifying python functions to invoke after the electron
files
An optional list of FileTransfer objects which copy files to/from remote or local filesystems.
Returns
Electron object inside which the decorated function exists.
Return Type
Electron
covalent.get_config
Return a configuration setting.
Invocation with no arguments returns the full configuration description; with a list of arguments returns a dictionary of configuration settings; with a string key name returns the corresponding value for a single setting.
Parameters
entries (Union[str, List, None]) – A string or list of strings specifying key names.
Returns
A dictionary or string describing the corresponding configuration settings.
Return Type
config
covalent.get_result
Get the results of a dispatch from the Covalent server.
Parameters
dispatch_id (str) – The dispatch id of the result.
wait (bool) – Controls how long the method waits for the server to return a result. If False, the method will not wait and will return the current status of the workflow. If True, the method will wait for the result to finish and keep retrying for sys.maxsize.
dispatcher_addr (Optional[str]) – Dispatcher server address, if None then defaults to the address set in Covalent’s config.
status_only (bool) – If true, only returns result status, not the full result object, default is False.
Return Type
Result
Returns
The Result object from the Covalent server
covalent.lattice(_func=None, *, backend=None, executor=None, workflow_executor=None, deps_bash=None, deps_pip=None, call_before=[], call_after=[], triggers=None)
Lattice decorator to be called upon a function. Returns a new Lattice <covalent._workflow.lattice.Lattice> object.
Parameters
_func (Optional[Callable]) – function to be decorated
Keyword Arguments
- backend – DEPRECATED: Same as executor.
- executor – Alternative executor object to be used by the electron execution. If not passed, the dask executor is used by default.
- workflow_executor – Executor for postprocessing the workflow. Defaults to the built-in dask executor or the local executor depending on whether Covalent is started with the –no-cluster option.
- deps_bash – An optional DepsBash object specifying a list of shell commands to run before _func
- deps_pip – An optional DepsPip object specifying a list of PyPI packages to install before running _func
- call_before – An optional list of DepsCall objects specifying python functions to invoke before the electron
- call_after – An optional list of DepsCall objects specifying python functions to invoke after the electron
- triggers – Any triggers that need to be attached to this lattice, default is None
Returns
Lattice object inside which the decorated function exists.
Return Type
covalent.lattice
Lattice decorator to be called upon a function. Returns a new Lattice <covalent._workflow.lattice.Lattice> object.
Parameters
_func (Optional[Callable]) – function to be decorated
Keyword Arguments
backend
DEPRECATED: Same as executor.
executor
Alternative executor object to be used by the electron execution. If not passed, the dask executor is used by default.
workflow_executor
Executor for postprocessing the workflow. Defaults to the built-in dask executor or the local executor depending on whether Covalent is started with the –no-cluster option.
deps_bash
An optional DepsBash object specifying a list of shell commands to run before _func
deps_pip
An optional DepsPip object specifying a list of PyPI packages to install before running _func
call_before
An optional list of DepsCall objects specifying python functions to invoke before the electron
call_after
An optional list of DepsCall objects specifying python functions to invoke after the electron
triggers
Any triggers that need to be attached to this lattice, default is None
Returns
Lattice object inside which the decorated function exists.
Return Type
covalent.redispatch
Wrapping the dispatching functionality to allow input passing and server address specification.
Parameters
dispatch_id (str) – The dispatch id of the workflow to re-dispatch.
dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then then defaults to the address set in Covalent’s config.
replace_electrons (Optional[Dict[str, Callable]]) – A dictionary of electron names and the new electron to replace them with.
reuse_previous_results (bool) – Boolean value whether to reuse the results from the previous dispatch.
Return Type
Callable
Returns
Wrapper function which takes the inputs of the workflow as arguments.
covalent.reload_config
Reload the configuration from the TOML file.
Parameters
None-
Return Type
None
Returns
None
covalent.set_config
Update the configuration.
Users may pass a dictionary of new settings, or a string key with a value to set a single configuration setting.
Parameters
new_config (Union[Dict, str]) – The new configuration dictionary, or a string key name.
new_value (Optional[Any]) – A new configuration value, if the first argument is a string.
Return Type
None
Returns
None
covalent.status
Alias of covalent.RESULT_STATUS
Attributes:
CANCELED
COMPLETED
FAILED
NEW_OBJECT
PENDING_POSTPROCESSING
POSTPROCESSING
POSTPROCESSING_FAILED
RUNNING
covalent.stop_triggers
Stop observing on all triggers of all given dispatch ids registered on the Triggers server. :type dispatch_ids: Union[str, List[str]] :param dispatch_ids: Dispatch ID(s) for whose triggers are to be stopped :type triggers_server_addr: Optional[str] :param triggers_server_addr: Address of the Triggers server; configured dispatcher’s address is used as default
Return Type
None
Returns
None
covalent.sync
Synchronization call. Returns when one or more dispatches have completed.
Parameters
dispatch_id (Union[str, List[str], None]) – One or more dispatch IDs to wait for before returning.
Return Type
None
Returns
None
covalent.wait
Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched.
Parameters
child – the dependent electron
parents – Electron(s) which must complete before waiting_electron starts
Returns
waiting_electron
Useful when execution of an electron relies on a side-effect from another one.