Local NVMe, RAID 0 and moving data in and out

#Identify your disks

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL
findmnt /                  # the device that holds the operating system
sudo apt install -y nvme-cli
sudo nvme list             # model, serial number and capacity of each NVMe drive

NVMe drives appear as nvme0n1, nvme1n1 and so on, their partitions as nvme0n1p1, nvme0n1p2. A drive with no partitions, no filesystem (FSTYPE) and no mount point is unused. The drive, partition or RAID device shown by findmnt / holds the system: leave it, and any RAID members behind it, alone.

#Partition and format a data disk

DISK=/dev/nvme1n1                                  # the unused drive you identified
sudo parted -s $DISK mklabel gpt mkpart data 0% 100%
sudo udevadm settle
sudo mkfs.ext4 -L data ${DISK}p1
# or XFS: sudo apt install -y xfsprogs && sudo mkfs.xfs -L data ${DISK}p1

ext4 is the Ubuntu default and a safe choice. XFS handles very large files and parallel writes well and is common for datasets and checkpoints. Either is fine; pick one and move on.

#Mount by UUID in /etc/fstab

sudo mkdir -p /data
UUID=$(sudo blkid -s UUID -o value ${DISK}p1)
echo "UUID=$UUID  /data  ext4  defaults,noatime,nofail  0  2" | sudo tee -a /etc/fstab
sudo findmnt --verify
sudo systemctl daemon-reload
sudo mount -a
findmnt /data
sudo chown $USER:$USER /data

• For XFS, write xfs and 0 0 instead of ext4 and 0 2.
• Device names such as nvme1n1 can change between boots; UUIDs do not.
• nofail lets the server finish booting even if the drive is missing, which matters on a machine you reach only over SSH.
• noatime skips a metadata write on every file read.
• Run sudo findmnt --verify after every edit of /etc/fstab.

#Optional: RAID 0 across NVMe drives

If your server has several unused NVMe drives, RAID 0 stripes them into one larger and faster volume.

sudo apt install -y mdadm xfsprogs
sudo wipefs -a /dev/nvme1n1 /dev/nvme2n1            # erases both drives
sudo mdadm --create /dev/md0 --level=0 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1
sudo mkfs.xfs -L scratch /dev/md0
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u
sudo mkdir -p /scratch
echo "UUID=$(sudo blkid -s UUID -o value /dev/md0)  /scratch  xfs  defaults,noatime,nofail  0  0" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload && sudo mount -a
cat /proc/mdstat

Adjust --raid-devices and the device list to the number of drives. A single NVMe drive is already fast; striping helps most with large sequential reads and writes, such as streaming a big dataset.

#Moving data in and out

# from your computer or another server
rsync -avhP ./dataset/ gpu1:/data/dataset/

# server to server: run on the new server, with an SSH key the old one accepts
rsync -avhP [email protected]:/data/ /data/

# models from Hugging Face, straight onto NVMe
curl -LsSf https://hf.co/cli/install.sh | bash      # installs the hf command
export HF_HOME=/data/hf
hf download Qwen/Qwen3-4B

# many small files: pack them, move one archive
sudo apt install -y zstd
tar -I 'zstd -T0' -cf dataset.tar.zst dataset/
tar -I zstd -xf dataset.tar.zst

Transfers over SSH are encrypted end to end, and rsync -P resumes where it stopped. For millions of small files, one archive moves much faster than the files one by one. The hf CLI also handles hf auth login for gated models; more on SSH transfers in SSH access.

#S3-compatible storage with rclone

For backups and datasets you reuse, S3-compatible object storage works well with rclone. Distribution packages of rclone are often out of date, so use the official install script:

sudo -v ; curl https://rclone.org/install.sh | sudo bash
rclone config        # n = new remote, name it s3remote, storage type s3, then provider, keys and endpoint
rclone lsd s3remote:                                   # list buckets

rclone copy /data/checkpoints s3remote:my-bucket/checkpoints --progress --transfers 16
rclone check /data/checkpoints s3remote:my-bucket/checkpoints   # compare both sides
rclone copy s3remote:my-bucket/datasets/web /data/datasets/web --progress

rclone copy never deletes anything. rclone sync makes the destination identical to the source and deletes what is not in the source, so run it with --dry-run first. For sensitive data, rclone’s crypt remote encrypts files before they leave the server.

#Back up before the term ends

If a term is not renewed, the server is stopped at the end of the paid month and its disks are wiped. Plan as if nothing on the server survives that date.

1. List what must survive: checkpoints, fine-tuned weights, datasets you produced, configuration, notebooks, logs, scripts and Docker volumes (docker volume ls).
2. Copy it to object storage or another machine a few days before the end date, not in the last hour.
3. Verify the copy with rclone check, or with checksums as below.
4. If you still need the server, renew before the term ends (see Billing and payments).

cd /data
find checkpoints -type f -print0 | xargs -0 sha256sum > checkpoints.sha256
rclone copy checkpoints.sha256 s3remote:my-bucket/
# after restoring elsewhere:
sha256sum -c checkpoints.sha256