CosmicAC Logo

Use team storage volumes

Create, list, rename, resize, and delete team storage volumes, and attach a volume to a GPU Container Job.

A team storage volume is persistent storage that your team attaches to a GPU Container Job. When you delete a job, the volume keeps its data, so you can reuse datasets and checkpoints in later jobs.

Prerequisites

Before you start, make sure that you have the following.

  • The role for your task. Team leads and platform administrators can create, rename, resize, and delete volumes. Members can attach a volume to a job that they create. Viewers can view the team's volumes. For more information, see Teams and roles.
  • A deployment that serves storage volumes. Volumes work only on a cluster that runs on Gcore. For more information, see Team storage volumes.
  • For the CLI steps, CosmicAC CLI version 1.3.0 or later, signed in to your deployment. To check which version you installed, run npm ls -g @tetherto/cosmicac-cli. For more information, see Install the CLI.

Understand team storage volumes

  • Shared use: more than one job can use a volume at the same time.
  • One node: a job that uses a volume must run on one node.
  • Fixed location: a volume stays in the location where you create it. A job that uses the volume runs in that location.
  • Provisioning: a new volume takes a few minutes to provision.
  • Size: you set the size in GiB when you create the volume, with a minimum of 100 GiB. You can increase the size later, but you can't decrease it.
  • Volume limit: each team can hold a limited number of volumes. A failed volume keeps its slot until you delete it. To see how many slots the team uses, check the Volume slots card on the Storage volumes page, or run cosmicac teams storage-volumes list.
  • Mount path: inside the job, the volume appears at /mnt/team-volume.

Create a volume

A volume name must be unique in your team. The name must have 3 to 63 lowercase letters, numbers, and hyphens. It must start and end with a letter or number.

In Settings > Storage volumes, click New volume. In the New storage volume dialog, select a Location, enter a Name, and enter a Size. Then, click Create volume. The volume's status is Provisioning until the volume is ready, and then the status changes to Available.

You can also create a volume while you create a job. For the steps, see Create a team storage volume (optional).

List volumes

To check each volume's status and the jobs that use it, list your team's volumes.

In the sidebar, click Settings, and then click Storage volumes.

Attach a volume to a job

You attach a volume when you create a GPU Container Job. You can't attach a volume to a job that already exists. You can attach a volume that has the status Available, In use, or Resizing. For a volume with another status, such as Provisioning, the Team storage volume list shows the status in place of the volume's size.

If you select a volume after you select a GPU in a different location, the new job form clears the GPU. Select a GPU again.

To attach a volume, follow the steps in one of the following guides.

Use the volume in the job

Connect to the job, and then work with the files in /mnt/team-volume. To check the volume's size and free space, run the following command.

df -h /mnt/team-volume

A new volume is empty. The first job that uses the volume creates an ext4 file system on it. You don't need to format the volume yourself. Later jobs keep the file system, so they can read the files that earlier jobs wrote. When you delete a job, the volume and the files on it remain.

Rename or resize a volume

You can rename or resize a volume in the web interface. Jobs that use the volume keep running.

To rename a volume, do the following.

  1. In Settings > Storage volumes, click Rename in the volume's row.
  2. In the Name field, enter the new name. The name follows the same rules as the name of a new volume.
  3. Click Rename volume.

The new name appears in the volume list and in the new job form.

To increase the size of a volume, do the following.

  1. In Settings > Storage volumes, click Resize in the volume's row.
  2. In the New size field, enter a size in GiB that's larger than the current size.
  3. Click Resize volume.

The volume's status is Resizing until the resize completes. During the resize, jobs keep using the volume at its old size. After the resize, jobs see the extra space. You can't undo a resize.

You can rename a volume whose status is Available, In use, or Failed. You can resize a volume whose status is Available or In use. To replace a failed volume, delete it, and then create a new one.

Delete a volume

Deleting a volume permanently deletes all the data on it. Before you delete a volume, copy everything you want to keep.

You can delete a volume only when its status is Available or Failed. If the status is In use, first stop or delete each job that uses the volume. To find the jobs, check the Attached to column in the volume list. The volume's status changes to Available a few minutes after you stop or delete the last job that uses it.

  1. In Settings > Storage volumes, click Delete in the volume's row.
  2. To confirm, enter the volume's name.
  3. Click Delete volume.

The status changes to Deleting. When the deletion completes, the volume disappears from the list.

Help and troubleshooting

The Location list or the CLI shows that no location can serve storage volumes

No rack in your deployment is set up for storage volumes. Ask your platform administrator to set up a rack. For more information, see Team storage volumes.

The New volume button doesn't appear or is unavailable

To create a volume, you must be a platform administrator or a team lead of the active team. For other roles, the New volume button is unavailable on the Storage volumes page and doesn't appear in the new job form. If you're a member or a viewer, ask a team lead to create the volume.

In the new job form, the button appears only for a GPU Container Job. The button is also unavailable when your team reaches its volume limit. To free a slot, delete a volume.

CosmicAC also hides the button if your deployment doesn't serve storage volumes. In this case, the Storage volumes settings page shows the message Storage volumes are unavailable. Ask your platform administrator to set up a rack. For more information, see Team storage volumes.

The New volume button is unavailable, or the CLI reports that the team already holds its maximum number of volumes

Your team has used all of its volume slots. Delete a volume that you no longer need, including any volume with the status Failed. To raise the limit, ask your platform administrator. For more information, see Team storage volumes.

Next steps

On this page