| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
software:files [2024/04/04 09:34]
cyril [Creation] |
software:files [2024/04/22 00:04] (current)
cyril [Tools] |
| === Partitions === | === Partitions === |
| |
| * If you want to use LVM and don't need to boot on the disk, nothing to do in this section, skip to the next one to create the LVM volumes directly on the raw device. | * If you want to use LVM and don't need to boot on the disk, not much to do in this section, just remove all the existing partitions with ''gparted'', then go directly to the next one to create the LVM volumes directly on the raw device. |
| * Otherwise, create a ''gpt'' partition table with ''gparted'' | * Otherwise, create a ''gpt'' partition table with ''gparted'' |
| * If you need to boot on the disk, create the required ''boot'' and ''grub'' partitions with ''gparted'' | * If you need to boot on the disk, create the required ''boot'' and ''grub'' partitions with ''gparted'' |
| === LVM === | === LVM === |
| |
| * Note that depending on your use case, i.e. whether you want multiple partitions having the same password or not, you may want | |
| * create physical volume: ''pvcreate <device-name>''. | * create physical volume: ''pvcreate <device-name>''. |
| * Check with ''pvdisplay'' or ''pvs''. | * Check with ''pvdisplay'' or ''pvs''. |
| * ''<absolute-size>'': ''200G'', ''3T'', ... | * ''<absolute-size>'': ''200G'', ''3T'', ... |
| * ''<relative-size>'': ''+100%FREE'' | * ''<relative-size>'': ''+100%FREE'' |
| | * If using an SSD drive, [[#ssd_trim|TRIM commands]] from the layers below (eg filesystem) will be transparently forwarded without any special configuration. However if you wish that LVM issues its own TRIM commands when some space is not allocated by LVM, you can set the ''issue_discards'' option to 1 in ''/etc/lvm/lvm.conf''. |
| | |
| |
| === LUKS === | === LUKS === |
| * By default it will configure the key derivation take 2 seconds | * By default it will configure the key derivation take 2 seconds |
| * Open (decrypt) the volume: ''cryptsetup luksOpen <volume-name> <evolume-name!>'' | * Open (decrypt) the volume: ''cryptsetup luksOpen <volume-name> <evolume-name!>'' |
| | * If using an SSD drive, you probably should enable [[#ssd_trim|TRIM-forwarding]]: ''cryptsetup --allow-discards --persistent refresh <evolume-name>'' (check [[https://wiki.archlinux.org/title/Solid_state_drive#dm-crypt|security implications]] though) |
| |
| === Filesystem === | === Filesystem === |
| * Choose the filesystem: | * Choose the filesystem: |
| * ''ext4'' for a default robust journaled filesystem | * ''ext4'' for a default robust journaled filesystem |
| * ''btrfs'': modern filesystem based on copy-on-write (instead of journal), offering more features (integrity checks, subvolumes, snapshots / deduplication, compression, encryption, RAID, ...), but also some drawbacks (less stable, requires more resources, [[https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/|wasted allocated data]], ...). It is a good choice for a work volume (because integrity checks and snapshots are really useful), but less obvious for backup volumes (because integrity checks are already handled by the backup software, and native deduplication is less performant with moved files and modified files, especially if not using ''btrfs-send''). | * ''btrfs'': modern filesystem based on copy-on-write (instead of journal), offering more features (integrity checks, subvolumes, snapshots / deduplication, compression, encryption, RAID, ...), but also some drawbacks (slightly less stable, requires more resources, [[https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/|wasted allocated data]], ...). It is a good choice for a work volume (because integrity checks and snapshots are really useful), but less obvious for backup volumes (because native deduplication is less performant with moved files and modified files, especially if not using ''btrfs-send'', so it is more efficient to use a deduplicating backup software, which will also handle integrity checks). |
| * ''zfs'': TODO | * ''zfs'': similar to ''btrfs'', using both copy-on-write and a journal (for improved performance with synchonous writes), more mature and sligthly more stable, but not included in kernel due to licensing (though easy to use). |
| * Create the filesystem: ''mkfs.<fs> <evolume-name> | * Create the filesystem: ''mkfs.<fs> <evolume-name>'' |
| * Tune the filesystem: | * Tune the filesystem: |
| * With ''ext4'', if not the system partition, you can remove the 5% reserved for ''root'': ''tune2fs -r 0 /dev/mapper/<evolume-name> | * With ''ext4'', if not the system partition, you can remove the 5% reserved for ''root'': ''tune2fs -r 0 /dev/mapper/<evolume-name>'' |
| * Mount the filesystem: ''mount /dev/mapper/<evolume-name> /mnt/<evolume-name>'' | * Mount the filesystem: ''mount /dev/mapper/<evolume-name> /mnt/<evolume-name>'' |
| | * Some filesystem tuning must be done after mount: |
| | * With ''btrfs'', you can enable compression: ''btrfs property set <fs-root> compression <algo>'' with ''<algo>'' equal to ''lzo'' (fastest) or ''zstd'' (compromise). Note that this syntax does not support configuring levels, nor forcing compression to disable heuristics. For that you have to use instead a mount option in the previous step: ''compress=zstd:1'' (default is '':3'') or ''compress-force=lzo''. |
| | |
| |
| === Sub-volumes === | === Sub-volumes === |
| * ''cryptsetup luksHeaderBackup /dev/DEVICE --header-backup-file /path/to/backupfile'' | * ''cryptsetup luksHeaderBackup /dev/DEVICE --header-backup-file /path/to/backupfile'' |
| * ''vgcfgbackup -f /path/to/backup/file vg01'' | * ''vgcfgbackup -f /path/to/backup/file vg01'' |
| | |
| | === Disk usage === |
| | |
| | * ''btrfs filesystem usage <subvolume>'' (option ''-g'' to display GB only). |
| | |
| | === Defragmentation === |
| | |
| | * With ''btrfs'': |
| | * ''btrfs filesystem defrag'' for **defragmenting files**. |
| | * can also be used to change compression of existing files (but breaks deduplication) with option ''-czstd'' (inherits level specified at mount). |
| | * ''btrfs filesystem balance -dusage=<percentage>'' for **defragmenting free space** (only data chunks less full than ''<percentage>'' will be compacted). |
| | * ''filefrag -v <file>'' to analyze the fragmentation of a file, and list all extents. |
| | |
| | === Compression === |
| | |
| | * With ''btrfs'': |
| | * ''compsize <subvolume-path>'' in order to get statistics about quantity of compressed files, and compression ratio. |
| | * ''compsize <file-path>'' in order to get compression details about a specific file. |
| | |
| | === SSD TRIM === |
| | |
| | * TRIM (or discard) operation means informing the SSD drive about the unused memory, so that it can perform efficiently wear leveling. |
| | * Checking TRIM support: run ''lsblk --discard'', and check for non-zero values in columns DISC-GRAN (DISCard GRANularity) and DISC-MAX (DISCard MAX bytes). |
| | * **Warning**: make sure that your device supports TRIM before using it, or data loss can occur. |
| | * Each layer must forward the TRIM commands to the layer above, until it reaches the drive. If you haven't done it persistently for LUKS as suggested in the [[#luks|create]] section, you can open it with this option: ''cryptsetup <...> --allow-discards'' |
| | * Then two options are available to enable it: |
| | * Continuous TRIM, i.e. configuring the filesystem to notify instantly each block that is freed. |
| | * It is not advised because doing it to often can reduce the lifetime of poor quality SSDs. |
| | * Periodic TRIM, i.e. explicitly notifying the free blocks periodically. |
| | * Using the ''fstrim'' utils from the util-linux package. |
| | * Manually: run ''fstrim --verbose <mount-point>'' for a single volume, or ''fstrim --verbose -A'' for all mounted filesystems listed in ''/etc/fstab'' and the root filesystem inferred from the kernel command line. |
| | * Weekly: enable the timer ''systemctl start fstrim.timer'' |
| | |
| | |
| | Source : https://wiki.archlinux.org/title/Solid_state_drive |
| |
| === Resizing === | === Resizing === |
| * do a full mirror backup of the data partition regularly (eg with rsync or a deduplicate software such as Attic) on an external hard drive or a network drive. Try to keep at least one copy somewhere else from your home (network drive, or one at home and one at work). | * do a full mirror backup of the data partition regularly (eg with rsync or a deduplicate software such as Attic) on an external hard drive or a network drive. Try to keep at least one copy somewhere else from your home (network drive, or one at home and one at work). |
| * take precautions to put the odds on your side in case of problem: make copies of your disks MBR (output of command p of fdisk), of your encrypted partitions headers, etc. | * take precautions to put the odds on your side in case of problem: make copies of your disks MBR (output of command p of fdisk), of your encrypted partitions headers, etc. |
| | |
| | ==== Tools ==== |
| | === rsync === |
| | |
| | === Borg Backup === |
| | |
| | * Create a Borg repository in the current folder: <code>borg init -e <encryption> [--append-only] .</code> |
| | * ''<encryption>'' can be: |
| | * ''none'' to disable it, for instance on an already encrypted volume |
| | * ''repokey'' to enable it (or ''repokey-blake2'' to use Blake2 instead of Sha256, which is often faster) |
| | * ''authenticated'' to disable encryption but still enable authentication (or ''authenticated-blake2'' to use Blake2) |
| | * ''--append-only'' means that no data can be removed with borg, archives can only be added. It can be used to protect an online repository against malware. |
| | * Create archives: <code>borg create <repo>::<!archive> <path> --stats --progress |
| | --compression auto,zstd,12 --chunker-params 15,23,19,4095 --noctime -x --exclude-caches</code> |
| | * ''--compression'': it can make sense to adjust the compression level depending on your computer speed and your storage speed, so that compression does not slow down the backup, but still save as much space as possible under this constraint. However it is not always easy to find an universal value (data that compress very well are mostly limited by the input storage speed, while data that compress less well are mostly limited by the output storage speed). You have roughly the choice between LZ4 (very quick), LZMA (very high compression ratio), and ZSTD (wide-range) in between. |
| | * ''--chunker-params'': this is also an important but a bit complicated tuning. Originally default value was creating small chunks causing huge cache and memory usage, so they switched to much larger chunks, but which can be too large for some applications (for instance when modifying only metadata of an image file, we want to deduplicate the data), so I came with this compromise ''15,23,19,4095''. |
| | * ''borg info <repo>'' |
| | * ''borg list <repo>'' |
| | * ''borg info <repo>::<archive>'' |
| | |
| | === Restic === |
| | |
| | * Create a Restic repository in the current folder: <code>restic init --repo .</code> |
| | * Note that encryption **and** password are mandatory, [[https://github.com/restic/restic/issues/4326|because]]. However you can store the password in a file in the repository, or use the a password file with ''--password-file''. |
| | * Create snapshots: <code>restic --repo <repo> --verbose --compression auto --ignore-ctime backup <path></code> |
| | * The chunker cannot be configured, contrary to Borg. It is equivalent to [[https://restic.readthedocs.io/en/stable/100_references.html#backups-and-deduplication|19,23]],[[https://restic.net/blog/2015-09-12/restic-foundation1-cdc/|21,512]], similarly to Borg's default [[https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#buzhash-chunker|19,23,21,4095]], but unlike my chosen values. |
| | * ''--compression'': unlike Borg, there is only choicies ''auto'', ''max'' and ''off'' |
| | |
| | |
| | === BTRFS snapshots === |
| | |
| | The BTRFS filesystem allows to perform some sorts of backups: |
| | * On the work disk, regularly creating snapshots allows to keep an history, for recovery in case of bad manual operation |
| | * It is also useful in order to "freeze" the content, so that the backup with another tool saves a consistent copy |
| | * On a backup disk, snapshots can also be used to keep an history. |
| | * If you update the backup with ''rsync'' for instance, then moved files and modified files will not be deduplicated (because they are sent again by rsync and won't be recognized). |
| | * However if you moved or modified files on a btrfs filesystem, you can send the increment between two snapshots: ''btrfs send -p <parent-src-snapshot> <src-snapshot> | pv | btrf receive <target-snapshot>'' (''<parent-source-snapshot>'' must have been sent already). |
| | * You can also deduplicate afterwards using offline tools for out-of-band deduplication (cf [[https://wiki.tnonline.net/w/Btrfs/Deduplication|list]]) |
| | |
| | |
| | |
| | ===== Container files ===== |
| | |
| | Sources: |
| | * [[https://serverfault.com/questions/696554/creating-a-grow-on-demand-encrypted-volume-with-luks|Excellent demonstration]] for creating a growing container file with LUKS and EXT4 using sparse files. |
| |
| ===== Digital Will ===== | ===== Digital Will ===== |
