Redispatch Simplifying Scalability

Covalent's Redispatch feature represents a major leap in scalable and efficient workflow management. With the ability to re-run previously dispatched workflows using only the dispatch ID—without needing to redefine the initial workflow—users can now effectively utilize Covalent as a robust computational backend for a myriad of applications.

The Power of Redispatch

Redispatch offers a refined approach to managing and redefining computational workflows. Its significance lies in the following areas:

1. Scalability for Platforms

Platforms often encounter a constant influx of computational requests with unique inputs. Traditional methods demand a new workflow for every distinct input, leading to inefficiency.

Redispatch tackles this challenge by leveraging an executed workflow's dispatch ID. Platforms can then re-initiate these complex workflows using only this ID and fresh inputs. This offers:

• Efficiency:No need to repeatedly design new workflows for each input, leading to smoother scalability.

• Environment Independence: The initiating system doesn't have to mirror the original workflow's computational setup. Be it an intricate ML model, particular software, or intensive computations; the triggering mechanism doesn't bear their requirements. This aspect is particularly beneficial for lightweight, serverless functions like AWS Lambda, allowing them to activate high-compute tasks without the associated computational or software overhead.

2. Optimized Research Workflows

Research often involves iterative changes to models, algorithms, or parameters. Conventionally, each alteration would demand a full workflow re-run, using up time and resources.

With Redispatch, users can modify specific tasks (known as electrons in Covalent) within a workflow, leading to computation only for the altered sections. This means faster iterations and resource savings. For instance, switching between machine learning models in a workflow doesn't require reprocessing the data, resulting in expedited outcomes.

Redispatch in Action

Here's how to leverage Redispatch in Covalent:

Redefining Tasks in a Workflow

Once a workflow is dispatched using Covalent, tasks within it can be redefined using the replace_electrons parameter. For instance, swapping out an accuracy computation with a recall computation is as simple as:

@ct.electron
def compute_recall(model, test_data):
    ...

redispatch_func = ct.redispatch(
    dispatch_id,
    replace_electrons={"compute_accuracy": compute_recall}
)
redispatch_id=redispatch_func()

Input Parameter Modifications

With Redispatch, modifying input parameters is straightforward:

redispatch_func = ct.redispatch(dispatch_id)
redispatch_id=redispatch_func({C=1.5, gamma=0.8})

Leveraging Previous Results

For workflows where certain results don't change, Redispatch can reuse previously computed results, enhancing efficiency:

redispatch_func = ct.redispatch(
    dispatch_id,
    replace_electrons={"compute_accuracy": compute_recall},
    reuse_previous_results=True
)
redispatch_id=redispatch_func()

However, it's crucial to ensure that reusing results is appropriate, especially in workflows with inherent stochasticity.

Redispatch How-to Guide

For further information on redispatching, checkout out the how-to guide:

• Re-executing a workflow