Workflows

Workflows in Covalent are also known as “lattices”. Each lattice composes one or more tasks (a.k.a. “electrons”) and executes them in a coordinated manner. Lattices are Python functions just like electrons, so complex processing sequences can be defined inside them with ease.

Creating Lattices

Ensure that you’ve installed Covalent Open-Source. (It is included with covalent_cloud)

pip install -U covalent

Here’s an example of a minimal workflow.

import random
import covalent as ct

@ct.electron
def prepare_data():
    # Code to prepare data
    return random.randint(0, 100)

@ct.electron
def process_data(data):
    # Code to process data
    return data + 1

@ct.lattice
def data_processing_workflow():
    results = []

    for _ in range(10):  # this loop will create 10 concurrent tasks
        data = prepare_data()
        results.append(process_data(data))

    return results

This workflow composes the tasks prepare_data() and preprocess_data(), using the outputs of one as inputs for the other. Covalent detects these interdependencies automatically to execute tasks concurrently (on their respective backends), wherever possible. Users can easily access the result of a workflow, as well as all task results, in the Covalent platform.

Dispatching Lattices

Lattices need to be dispatched in order to execute in Covalent Cloud. To do this, wrap the lattice function with cc.dispatch() and call the resulting function to dispatch the lattice.

import covalent_cloud as cc

# Dispatch to Covalent Cloud.
result_id = cc.dispatch(data_processing_pipeline)()

Tip

Pass a cc.volume() to attach workflow-wide persistent storage to any dispatch.

The workflow identifier can then be used to query the result.

result = cc.get_result(result_id)  # check for result
result = cc.get_result(result_id, wait=True)  # wait for result

The workflow identifier can also be used to “re-dispatch” a workflow with new inputs. This can be convenient because it requires neither the source code nor the environment of the original dispatch. See Reusing Dispatched Workflows for more information.

Workflow basics

• Create electrons: Begin by defining the individual tasks that comprise your workflow, using the @ct.electron decorator (see Defining Tasks). Each electron should encapsulate a distinct, well-defined step of your process.
• Build the workflow: Use the @ct.lattice decorator to assemble electrons into a workflow. The logic of the lattice function determines the order of execution and data dependencies.

Best Practices for Lattices

See also, Best Practices for Electrons.

• Orchestration: Lattices define the structure of your computation. Break down complex workflows into modular, reusable tasks while establishing clear dependencies and data flow.
• Hardware abstraction: Assign compute resources (CPUs, GPUs, memory) on a per-task or workflow level, ensuring efficient utilization of Covalent infrastructure. See GPU Access for details. Execute code seamlessly across different hardware without modifications.
• Dynamic workflows, i.e. workflows that execute tasks conditionally (based results from other tasks) are more challenging than workflows with a fixed number of task executions. See Dynamic Workflows for instructions and more information.

Pitfalls

• Complexity: Avoid creating overly complex lattices. A small amount of code can go a long way here.
• Workflow Patterns: Inside a lattice, all function calls must be electrons. Covalent dry-runs the lattice build process for dependency analysis, so result-dependent control flow is done via dynamic workflows in a sublattice pattern. See the following GitHub discussions for common pitfalls and examples: Discussion 1, Discussion 2, Discussion 3.