Skip to main content

Creating Custom Environments

For developers looking to tailor their compute environment on Covalent Cloud, the create_env function is your starting point. This functionality allows you to specify the exact software landscape your electron will operate in, whether it's a suite of pip packages or a carefully curated Conda environment or installing from a private repository.

Let's take a peek at how create_env works, allowing you to send a creation request to the Covalent Cloud with a tailored environment dependency list. This versatile function accepts a mix of pip and conda packages, along with environment variables to fine-tune your execution environment.

cc.create_env(
name="my_custom_env",
pip="requirements.txt", # Can be a string of path to reqs.txt or list of packages
conda={
"channels": ["conda-forge", "defaults"],
"dependencies": ["python=3.8","numpy=1.21.*", "scipy=1.5.*"],
"variables": [{'name': 'MY_VARIABLE', 'value': '123', 'sensitive': False}]
}
)

The create_env function can interpret a requirements.txt or an environment.yml, providing flexibility in how you define your dependencies. Plus, for a more visual setup, Covalent's UI is available to help guide you through creating environments. Once the create_env function is called, please look at the status of the environment build in the UI at environments page, which lists all your environments.

Once your environment is built (green check in the UI) and set, you can utilize it in an electron:

import covalent as ct
import numpy as np

cc.create_env(
name="my_custom_env",
pip=["numpy"], # Can be a string path to reqs.txt or list of packages
conda={
"channels": ["conda-forge", "defaults"],
"dependencies": ["scipy=1.5"],
"variables": [{'name': 'MY_VARIABLE', 'value': '123', 'sensitive': False}]
}
)

#wait for the env creation to complete, follow progress in UI

@ct.electron(executor=cc.CloudExecutor(env="my_custom_env"))
def compute_with_numpy():
return np.array([1, 2, 3]) * 3

This code illustrates how to deploy a function that uses numpy, executed in the cloud with the environment you've just created. Note that it is critical to use the same major python version installed in the environment as that is being used to dispatch the workflow, this can be defined by conda=["python=3.8"]. In future we may relax this constrain based on user feedback.

Warning

When configuring your environment, consider the trade-off between complexity and startup time. The container initialization is swift for environments with 10-20 packages, averaging a mere 5-20 seconds of overhead. We're dedicated to improving this, and soon, even the most extensive containers will launch in milliseconds—stay tuned for these exciting enhancements!

Installing from private repositories

Incorporating code from private repositories into your Covalent Cloud environment is streamlined with the secret feature. Here’s how to securely manage access to private repositories and install them, for example from Github.

First, store your access credentials as a secret:

import covalent_cloud as cc

# Create a secret with your private repo access token
cc.store_secret(name="MY_SECRET_NAME", value="MY_SECRET_VALUE")

You can manage your secrets by listing them or deleting them as needed. When you’re ready to create an environment that pulls from your private repository, specify your secret in the pip field:

# Create an environment that installs a package from a private repo
cc.create_env(
name="my_custom_env",
conda=["python=3.8"],
pip=[
"git+https://username:{covalent.secrets.MY_SECRET_NAME}@github.com/user/project.git@main"
]
)

This integration allows your electron to leverage private code securely and efficiently. Keep an eye on the environment's build status through Covalent’s UI, and once it's ready, your custom environment is all set for high-performance cloud computations.