HomeServer/STORAGE.md

Storage and Backup Guide

Storage Overview

This home server uses a bcache configuration for performance with redundant backup.

Physical Storage

Device	Type	Capacity	Mount Point	Purpose
/dev/nvme0n1p2	NVMe SSD	70GB	/	Root filesystem
/dev/nvme0n1p3	NVMe SSD	163GB	-	Bcache cache device
/dev/sdb	HDD (IronWolf)	4TB	-	Bcache backing device
/dev/bcache0	Bcache	3.6TB	/mnt/bcache	Main data storage (HDD + NVMe cache)
/dev/sda1	HDD (IronWolf)	3.6TB	/mnt/backup-mirror	Weekly backup mirror

Data Layout

/mnt/bcache/               # Main storage (933GB used)
├── nextcloud/             # Nextcloud files (~2TB)
├── jellyfin/              # Media library
├── git/                   # Gitea repositories
├── data/                  # General data
│   └── media/             # Media files
├── bitwarden/             # Password vault data
├── home-assistant/        # Home automation config
└── [other services]/

/mnt/backup-mirror/        # Mirror backup (synced weekly)
└── [same structure as above]

Bcache Status

Check Cache Status

# Should show "clean" or "dirty" (NOT "no cache")
cat /sys/block/bcache0/bcache/state

# View cache performance
cat /sys/block/bcache0/bcache/stats_total/cache_hits
cat /sys/block/bcache0/bcache/stats_total/cache_misses

Re-attach Detached Cache

If bcache state shows "no cache":

echo "74a7d177-65f4-4902-9fe5-e596602c28d4" | sudo tee /sys/block/bcache0/bcache/attach
cat /sys/block/bcache0/bcache/state  # Should now show "clean"

Backup System

Automated Backups

• Schedule: Every Sunday at 2:00 AM
• Source: /mnt/bcache/
• Destination: /mnt/backup-mirror/
• Method: Rsync incremental sync

Manual Backup

# Run backup manually
sudo /usr/local/bin/backup-mirror-sync.sh

# Monitor backup progress
sudo tail -f /var/log/backup-mirror-sync.log

# Check backup status
systemctl status backup-mirror-sync.service
systemctl list-timers backup-mirror-sync.timer

Restore from Backup

Full Restore

# Ensure backup drive is mounted
mountpoint /mnt/backup-mirror

# Restore everything
sudo rsync -avh --delete /mnt/backup-mirror/ /mnt/bcache/

Selective Restore

# Restore specific service
sudo rsync -avh /mnt/backup-mirror/nextcloud/ /mnt/bcache/nextcloud/

# Restore with dry-run first
sudo rsync -avhn /mnt/backup-mirror/gitea/ /mnt/bcache/gitea/

Drive Health Monitoring

Check Drive Health

Note: /dev/sda (backup mirror) spins down after 10 minutes to save energy. Accessing it will wake the drive (takes ~10 seconds).

# Quick health check
sudo smartctl -H /dev/sda   # Backup drive (may spin up if in standby)
sudo smartctl -H /dev/sdb   # Main storage drive
sudo smartctl -H /dev/nvme0n1  # Cache/system drive

# Detailed SMART info
sudo smartctl -a /dev/sda

# Check if backup drive is spun down
sudo hdparm -C /dev/sda

View Drive Temperature

# HDDs
sudo smartctl -a /dev/sda | grep Temperature
sudo smartctl -a /dev/sdb | grep Temperature

# NVMe
sudo smartctl -a /dev/nvme0n1 | grep Temperature

SMART Test History

# View recent self-tests
sudo smartctl -l selftest /dev/sda

# Monitor smartd service
systemctl status smartmontools
sudo journalctl -u smartmontools -n 50

Disk Space Management

Check Usage

# Overall usage
df -h

# Specific mounts
df -h /
df -h /mnt/bcache
df -h /mnt/backup-mirror

# Find large directories
du -sh /mnt/bcache/*
du -sh /mnt/backup-mirror/*

Cleanup Commands

# Clean journal logs (keeps 30 days)
sudo journalctl --vacuum-time=30d

# Prune unused container images
sudo crictl rmi --prune

# Remove old packages
sudo apt autoremove -y
sudo apt autoclean -y

Emergency Procedures

If bcache drive fails (/dev/sdb)

1. Boot into recovery mode
2. Mount backup drive: sudo mount /dev/sda1 /mnt/backup-mirror
3. Replace failed drive
4. Rebuild bcache configuration (see bcache docs)
5. Restore data: sudo rsync -avh /mnt/backup-mirror/ /mnt/bcache/

If backup drive fails (/dev/sda)

1. Replace drive
2. Partition: sudo parted /dev/sda mklabel gpt && sudo parted /dev/sda mkpart primary ext4 0% 100%
3. Format: sudo mkfs.ext4 -L backup-mirror /dev/sda1
4. Update fstab if UUID changed
5. Run manual backup: sudo /usr/local/bin/backup-mirror-sync.sh

If cache drive fails (/dev/nvme0n1p3)

1. Data is safe on backing device (/dev/sdb)
2. Replace NVMe drive
3. Recreate cache device (see bcache docs)
4. Re-attach to bcache set

Maintenance Logs

All automated maintenance logs are located in /var/log/:

• /var/log/backup-mirror-sync.log - Backup operations
• /var/log/k3s-maintenance.log - System maintenance
• /var/log/k3s-cert-rotation.log - Certificate rotation
• journalctl -u smartmontools - SMART monitoring

Power Management

Backup Drive Auto-Spindown

The backup mirror drive (/dev/sda) is configured to automatically spin down after 10 minutes of inactivity to save energy.

Configuration:

• Spindown timeout: 10 minutes (120 * 5 seconds)
• Managed by: udev rule /etc/udev/rules.d/99-backup-drive-power.rules
• Auto-spinup: Drive wakes automatically when accessed

Power Savings:

• Active power: ~5-8W
• Standby power: <1W
• Expected savings: ~35-50 kWh/year (~6 hours active per week for backups)

Check Power State:

# Check if drive is spun down
sudo hdparm -C /dev/sda

# Manually spin down (for testing)
sudo hdparm -y /dev/sda

# Manually spin up (wake drive)
ls /mnt/backup-mirror

SMART Monitoring: The backup drive is excluded from continuous SMART monitoring to avoid waking it up. Instead:

• SMART health is checked automatically during weekly backup runs
• Manual checks: sudo smartctl -H /dev/sda (will wake drive if needed)
• Status logged in /var/log/backup-mirror-sync.log

Performance Tips

1. Bcache is most effective for frequently accessed data - First access is slow (reads from HDD), subsequent accesses are fast (cached in NVMe)

2. Monitor cache hit ratio to understand effectiveness:

   hits=$(cat /sys/block/bcache0/bcache/stats_total/cache_hits)
   misses=$(cat /sys/block/bcache0/bcache/stats_total/cache_misses)
   echo "scale=2; $hits / ($hits + $misses) * 100" | bc
   

3. Backup timing - Sunday 2AM chosen to minimize impact on media streaming and other services

4. SMART tests timing - Daily tests only on active drives (/dev/sdb, /dev/nvme0n1). Backup drive checked during weekly backups.