Creating Custom Environments

Creating environments in Covalent Cloud helps you isolate and manage software dependencies across individual tasks, workflows, and projects. Environments can be created in the Covalent Cloud UI or with the Covalent Cloud SDK, using the cc.create_env() function.

Every environment is named so it can be identified, used, and modified after creation.

Environment creation with the SDK looks like the following. This environment features both conda and pip dependencies. Note that the cc.create_env() function accepts requirements.txt (or environment.yml) files as an alternative to explicit dependencies.

cc.create_env(
    name="my_custom_env",
    pip="./requirements.txt",  # Can be a string of path to reqs.txt or list of packages
    conda={
        "channels": ["conda-forge", "defaults"],
        "dependencies": ["numpy=1.21.*", "scipy=1.5.*"],
    },
)

Caution

Always be mindful of the trade-off between complexity and startup time. Initialization is quick for environments with around 10-20 packages, averaging a mere 5-20 seconds of overhead. However, installing more/larger packages can slow things down significantly. We're dedicated to improving this, and soon, even the most elaborate environments will launch in milliseconds. Stay tuned!

Python versions

The user's local Python version is chosen by default when creating an environment. It is possible to overload this setting by passing e.g. "python=3.8" as a conda dependency inside cc.create_env(). Note that this is only recommended if you intend to dispatch from elsewhere, in a local environment with that same Python version.

Environment builds

Environments can take a few minutes build, so initiating creation is asynchronous by default. Alternatively, setting wait=True will not allow cc.create_env() to exit until the environment build completes.

Tip

You can now inspect build logs in the Environments page during any new environment build.

After an environment is built, it's ready to go.

How to use environments

Specify the name of an environment inside a cloud executor to use it with your tasks.

custom_ex = cc.CloudExecutor(env="my_custom_env")

@ct.electron(executor=custom_ex)
def compute_with_numpy():
    return np.array([1, 2, 3]) * 3

Installing packages from private repositories

We recommend using secrets to store your credentials and access them during environment builds. For example, use the following pattern to download a package from a private GitHub repository.

import covalent_cloud as cc

# Create a secret with your private repo access token
cc.store_secret(name="MY_SECRET_NAME", value="MY_SECRET_VALUE")

Specify stored secrets as follows to use them with cc.create_env().

# Create an environment that installs a package from a private repo
cc.create_env(
   name="my_custom_env",
   conda=["python=3.8"],
   pip=[
       "git+https://username:{covalent.secrets.MY_SECRET_NAME}@github.com/user/project.git@main"
   ]
)

Best Practices and Considerations

It's important to consider the impact of environment size on both creation time and task execution. Here are some best practices regarding environments:

1. Creation Time: Expect 2-20 minutes for environment setup, depending on size and package count. Plan accordingly to avoid delays.
2. Overhead: Keep environments as lean as possible, as larger environments will increase startup times.
3. Optimizations: Separate dependencies into specific environments for specific tasks to optimize for efficiency.

Upcoming Improvements

Stay tuned for upcoming announcements about support for pre-built custom images! This feature will significantly reduce the setup times for complex environments by enabling users to leverage existing images.