Move application volumes to an additional disk with volumectl #8259

Since NethServer 8.7, an application volume can be placed on an additional disk, but only when the application is installed, restored or cloned. When an installed application grows and fills the root disk, the only way to move its data to a new disk is to clone the application and remove the old one: it takes a long time, needs twice the space, and changes the application name. The purpose of this feature is to move the data of one volume of an installed application to another disk, in place, with a short downtime.

Proposed solution

A new volumectl move-volume command moves one volume of an installed application in two steps:

volumectl move-volume dovecot-data --for mail1 --target /mnt/disk00 --sync
volumectl move-volume dovecot-data --for mail1 --target /mnt/disk00

1. Sync (--sync) copies the volume data to the target while the application keeps running. It can be repeated as many times as wanted: each run copies only what changed since the previous one, and it shows progress, because the first copy can take hours. The copy keeps ownership, permissions, access control lists, extended attributes (including SELinux labels), hard links and sparse files.
2. Finalize (without --sync) stops the application, copies the last changes, recreates the volume at the new location and starts the application again. It is the only step with downtime: the time to stop the application, plus the last copy, plus the time to start it again.

The command also:

• accepts as target a base path printed by volumectl list-base-paths, or the root disk mountpoint (for example /) to move a volume back to the root disk
• checks that the target has enough free space for the volume data
• refuses to finalize while the application agent is running a task; with --force it stops the agent anyway, and the interrupted task never returns a result to its caller
• leaves the original data untouched until finalize completes; if finalize fails or is interrupted with Ctrl-C, the original volume is restored and the application is started again
• after a successful finalize, removes the old data in background, so the removal does not add to the downtime

To stop the application, finalize stops the whole user session of the application: its services, its timers and its agent. While the session is stopped, Podman runs as the application user through runuser, and the volume is removed and created again. Stopped containers that still refer to the volume are removed too: all NethServer applications create their containers again when their services start.

The command supports rootless applications only. The existing rootfull applications do not store large data, so they do not need it. /etc/nethserver/volumes.conf is not changed, because it is read only when a volume is created.

Alternative solutions

These paths were explored and discarded:

• Clone the application to the same node, choosing the additional disk, then remove the original: it works today, but it changes the application name and needs twice the space.
• Update volumes.conf at finalize, so the assignment survives later operations: not needed. The file is read only when a volume is created, and application instance names are never reused.
• Stop only the application services, like clone does with systemctl --user isolate transfer-state.target: tests showed the command returns before the services are stopped, timers keep running and can start services again, and the agent can start services while the move runs. Stopping the whole user session is simpler and safer.
• Stop only the containers that use the volume: a shorter downtime, but the application data could become inconsistent, for example between a database and its files.
• Run Podman with a login shell (runuser -l), as the runagent command does: when it runs without an interactive login (for example from a future UI action), it starts the user session again. The command uses runuser -u instead, with the usual runtime directory, because Podman records that path in its database.
• Copy with cp -a in the first sync, to use XFS reflinks: reflinks work only inside one filesystem, and a move always goes from one disk to another. In a test, cp -a was not faster than rsync, and it shows no progress.
• Remove the old data during the last copy (rsync --remove-source-files): it adds the removal time to the downtime. The old data is moved aside instead (an instant rename) and removed in background.
• A UI action: it can come later, once the command is reliable.

Additional context

The command was tested on Rocky Linux 9 (with SELinux) and Debian 13. Debian has no rsync command, so the copy runs in the core rsync container image.

The admin manual must describe the procedure, including when to run the sync steps and the expected downtime of finalize. The developer manual is updated by the implementation pull request. The draft "Disk usage and size of applications" would help choose which volume to move.

See also

• Implementation: feat(volumectl): move app volumes to another disk ns8-core#1399
• Assign specific app volumes to dedicated storage: Assign specific app volumes to dedicated storage #7665
• Developer manual, volumes: https://nethserver.github.io/ns8-core/modules/volumes/
• Discussion https://mattermost.nethesis.it/nethesis/pl/z1xcei4sqtd6mr3agkkhayxysr