Covalent Tasks

A task in Covalent (a.k.a. an "electron") is a shippable unit of computational work. Tasks are defined by decorating a normal Python function with @ct.electron. As an electron, a function can be assigned arbitrary compute resources, along with an arbitrary software environment, and be executed in Covalent Cloud.

Electrons represent distinct components of a workflow and perform specific tasks, like data processing or other complex calculations.

Creating Electrons

Ensure that you’ve installed Covalent Open-Source. The open-source core of Covalent provides the framework used to create and compose tasks. (It is included with covalent_cloud)

pip install -U covalent

Here’s an example of a minimal task.

import covalent as ct

def helper_function():
    return "Covalent"

@ct.electron
def hello_task():

    # Code here is shipped by Covalent.

    word = helper_function()
    return f"Hello, {word}!"

Covalent only ships an electron when a workflow (a.k.a. a “lattice”) containing that electron is dispatched. Workflows and workflow tasks behave like normal Python functions in every other context — this can be quite useful for debugging your code.

To go beyond the basics, see any of our tutorials for myriad non-trivial examples.

Assigning Compute Resources

Compute resources are assigned to electrons in a modular way, using executors. This means resources can be delegated in proportion to what each task requires. Click here for more information about compute resources in Covalent Cloud.

GPU resources, for example, are essential for many applications.

import covalent_cloud as cc

resource = cc.CloudExecutor(
    env="optimized-environment",
    gpu_type="h100",
    num_cpus=12,
    num_gpus=2,
    memory="24GB"
)

@ct.electron(executor=resource)  # assign 2 H100 GPUs, 24 CPUs etc.
def model_fine_tuning(model_id, dataset_id):
    # load base model and dataset
    # run fine-tuning on H100 GPUs
    # save fine-tuned model
    return model_path

Tip

See here for a complete example of a fine-tuning electron.

Best Practices for Electrons

• Resource optimization: Assign the minimum compute resources required by each task to balance performance and cost. For example, use two or more executors that specify the same env but different num_gpus or num_cpus, thereby decoupling software and hardware requirements.
• Error handling: Errors inside electrons are reported in the Covalent UI. Implement robust error handling within tasks to manage exceptions and ensure reliability.

Pitfalls

• Over-partitioning: Avoid dividing processes into too many tasks. This can complicate workflow management and incur unnecessary costs. Prefer looping inside electrons to looping over electrons, unless concurrent iterations are a must.
• Improper resourcing: Assigning insufficient resources can lead to performance bottlenecks and failed executions. Conversely, over-assigning unnecessary resources can be costly and inefficient.

Data Management

• Caching: Electron inputs and outputs are automatically cached by the platform to streamline data handling across tasks.
• Size limits: The maximum size for inputs and outputs is 5 GB. Using storage volumes is recommended for exchanging larger data between tasks.

When to Use Electrons

• Tasks that require significant compute power or specific environmental configurations.
• Operations that are distinct and can be isolated from other processes.

When NOT to Use Electrons

• Processes that require frequent interaction with other tasks and/or continuous data exchange.
• Simple, quick operations that aren’t worth the overhead of running spinning up a new virtual machine. If an operation is too small to be an electron, just run it inside the electron that needs its result. The helper function above illustrates the basic idea.

Further Resources

Explore complex task management and integration in our Workflow Orchestration guide.