| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
software:files [2024/04/16 12:46]
cyril [Digital Will] |
software:files [2024/07/05 08:09] (current)
cyril [Tools] |
| * If you want to use LVM and don't need to boot on the disk, not much to do in this section, just remove all the existing partitions with ''gparted'', then go directly to the next one to create the LVM volumes directly on the raw device. | * If you want to use LVM and don't need to boot on the disk, not much to do in this section, just remove all the existing partitions with ''gparted'', then go directly to the next one to create the LVM volumes directly on the raw device. |
| * Otherwise, create a ''gpt'' partition table with ''gparted'' | * Otherwise, create a ''gpt'' partition table with ''gparted'' |
| * If you need to boot on the disk, create the required ''boot'' and ''grub'' partitions with ''gparted'' | * If you need to boot on the disk, with ''gparted'': |
| | * create the EFI partition: set size around 200-400MB, format it to fat32, set ''boot'' and ''esp'' flags |
| | * create the boot partition: set size around 300-500MB, format it to ext2 |
| | * mount everything: root partition to /mnt, boot partition to /mnt/boot, efi partition to /mnt/boot/efi, ''mount --bind'' /mnt/{dev,proc,sys}, ''mount -t efivarfs efivarfs /mnt/sys/firmware/efi/efivars'' |
| | * chroot to the new root: ''chroot /mnt'' |
| | * install grub: ''grub-install --root-directory=/ --boot-directory=/boot --efi-directory=/boot/efi --bootloader-id=<os-name>'' |
| | * you can check the EFI install with ''efibootmgr -v'' (and remove an entry with ''efibootmgr -b <0005> -B'' |
| | * grub-mkconfig -o /boot/grub/grub.cfg |
| * If you want to use LVM, create a single large partition with the remaining space with ''gparted'' to create the LVM volumes on this partition. | * If you want to use LVM, create a single large partition with the remaining space with ''gparted'' to create the LVM volumes on this partition. |
| * Otherwise create the required system and data partitions with ''gparted'' | * Otherwise create the required system and data partitions with ''gparted'' |
| === LVM === | === LVM === |
| |
| * create physical volume: ''pvcreate <device-name>''. | * create physical volume: ''pvcreate <device-name>'' (device can be the whole device if not a boot device, or a partition). |
| * Check with ''pvdisplay'' or ''pvs''. | * Check with ''pvdisplay'' or ''pvs''. |
| * if it complains with the error ''Cannot use <device-name>: device is partitioned'', you need to remove existing traces of partition table or filesystem with the command ''wipefs --all <device-name>'' | * if it complains with the error ''Cannot use <device-name>: device is partitioned'', you need to remove existing traces of partition table or filesystem with the command ''wipefs --all <device-name>'' |
| * ''<absolute-size>'': ''200G'', ''3T'', ... | * ''<absolute-size>'': ''200G'', ''3T'', ... |
| * ''<relative-size>'': ''+100%FREE'' | * ''<relative-size>'': ''+100%FREE'' |
| | * If using an SSD drive, [[#ssd_trim|TRIM commands]] from the layers below (eg filesystem) will be transparently forwarded without any special configuration. However if you wish that LVM issues its own TRIM commands when some space is not allocated by LVM, you can set the ''issue_discards'' option to 1 in ''/etc/lvm/lvm.conf''. |
| | |
| |
| === LUKS === | === LUKS === |
| |
| * Encrypt the volume/partition/device: ''cryptsetup luksFormat -c aes-xts-plain64 -h sha256 -s 512 <volume-name>'' | * Encrypt the volume/partition/device: ''cryptsetup luksFormat -c aes-xts-plain64 -h sha256 -s 512 <volume-name>'' |
| * ''<volume-name>>'' is the device or partition name if not using LVM, or ''/dev/mapper/<vgroup-name>-<lvolume-name>'' if using it. | * ''<volume-name>'' is the device or partition name if not using LVM, or ''/dev/mapper/<vgroup-name>-<lvolume-name>'' if using it. |
| * Choose a strong passphrase as it can be brute-forced (at least 80 bits of entropy) | * Choose a strong passphrase as it can be brute-forced (at least 80 bits of entropy) |
| * By default it will configure the key derivation take 2 seconds | * By default it will configure the key derivation take 2 seconds |
| * Open (decrypt) the volume: ''cryptsetup luksOpen <volume-name> <evolume-name!>'' | * Open (decrypt) the volume: ''cryptsetup luksOpen <volume-name> <evolume-name!>'' |
| | * If using an SSD drive, you probably should enable [[#ssd_trim|TRIM-forwarding]]: ''cryptsetup --allow-discards --persistent refresh <evolume-name>'' (check [[https://wiki.archlinux.org/title/Solid_state_drive#dm-crypt|security implications]] though). Check with ''cryptsetup luksDump <volume-name>''. If you enabled it by mistake (for instance on a non-SSD), you can disable it with ''cryptsetup --persistent refresh <evolume-name>'' (it resets flags). |
| |
| === Filesystem === | === Filesystem === |
| |
| * With ''btrfs'': | * With ''btrfs'': |
| * ''compsize <subvolume-path>'' in order to get statistics about quantity of compressed files, and compression ratio | * ''compsize <subvolume-path>'' in order to get statistics about quantity of compressed files, and compression ratio. |
| | * ''compsize <file-path>'' in order to get compression details about a specific file. |
| | |
| | === SSD TRIM === |
| | |
| | * TRIM (or discard) operation means informing the SSD drive about the unused memory, so that it can perform efficiently wear leveling. |
| | * Checking TRIM support: run ''lsblk --discard'', and check for non-zero values in columns DISC-GRAN (DISCard GRANularity) and DISC-MAX (DISCard MAX bytes). |
| | * **Warning**: make sure that your device supports TRIM before using it, or data loss can occur. |
| | * Each layer must forward the TRIM commands to the layer above, until it reaches the drive. If you haven't done it persistently for LUKS as suggested in the [[#luks|create]] section, you can open it with this option: ''cryptsetup <...> --allow-discards'' |
| | * Then two options are available to enable it: |
| | * Continuous TRIM, i.e. configuring the filesystem to notify instantly each block that is freed. |
| | * It is not advised because doing it to often can reduce the lifetime of poor quality SSDs. |
| | * Periodic TRIM, i.e. explicitly notifying the free blocks periodically. |
| | * Using the ''fstrim'' utils from the util-linux package. |
| | * Manually: run ''fstrim --verbose <mount-point>'' for a single volume, or ''fstrim --verbose -A'' for all mounted filesystems listed in ''/etc/fstab'' and the root filesystem inferred from the kernel command line. |
| | * Weekly: enable the timer ''systemctl start fstrim.timer'' |
| | |
| | |
| | Source : https://wiki.archlinux.org/title/Solid_state_drive |
| |
| === Resizing === | === Resizing === |
| | |
| | ==== Extending ==== |
| | |
| | * Resize the LVM logical volume: ''lvresize -L <absolute-size> <lvolume-name>'' |
| | * ''<absolute-size>'' can also be an increment, e.g. ''+50G'' |
| | * Open the volume with Luks: ''cryptsetup luksOpen <lvolume-name> <evolume-name>'' |
| | * Resize the filesystem: |
| | * ext4: ''e2fsck -f <evolume-name> ; resize2fs <evolume-name>'' |
| | * btrfs: mount the filesystem then ''btrfs filesystem resize max /mnt/<evolume-name>'' |
| | |
| | |
| | ==== Reducing ==== |
| |
| TODO | TODO |
| === Borg Backup === | === Borg Backup === |
| |
| * Create a Borg repository in the current folder: <code>borg init -e <encryption> [--append-only]</code> | * Create a Borg repository in the current folder: <code>borg init -e <encryption> [--append-only] .</code> |
| * ''<encryption>'' can be: | * ''<encryption>'' can be: |
| * ''none'' to disable it, for instance on an already encrypted volume | * ''none'' to disable it, for instance on an already encrypted volume |
| * ''--append-only'' means that no data can be removed with borg, archives can only be added. It can be used to protect an online repository against malware. | * ''--append-only'' means that no data can be removed with borg, archives can only be added. It can be used to protect an online repository against malware. |
| * Create archives: <code>borg create <repo>::<!archive> <path> --stats --progress | * Create archives: <code>borg create <repo>::<!archive> <path> --stats --progress |
| --compression auto,zstd,12 --chunker-params 15,23,19,4095 --noatime --noctime -x --exclude-caches</code> | --compression auto,zstd,12 --chunker-params 15,23,19,4095 --noctime -x --exclude-caches</code> |
| * ''--compression'': it can make sense to adjust the compression level depending on your computer speed and your storage speed, so that compression does not slow down the backup, but still save as much space as possible under this constraint. However it is not always easy to find an universal value (data that compress very well are mostly limited by the input storage speed, while data that compress less well are mostly limited by the output storage speed). You have roughly the choice between LZ4 (very quick), LZMA (very high compression ratio), and ZSTD (wide-range) in between. | * ''--compression'': it can make sense to adjust the compression level depending on your computer speed and your storage speed, so that compression does not slow down the backup, but still save as much space as possible under this constraint. However it is not always easy to find an universal value (data that compress very well are mostly limited by the input storage speed, while data that compress less well are mostly limited by the output storage speed). You have roughly the choice between LZ4 (very quick), LZMA (very high compression ratio), and ZSTD (wide-range) in between. |
| * ''--chunker-params'': this is also an important but a bit complicated tuning. Originally default value was creating small chunks causing huge cache and memory usage, so they switched to much larger chunks, but which can be too large for some applications (for instance when modifying only metadata of an image file, we want to deduplicate the data), so I came with this compromise ''15,23,19,4095''. | * ''--chunker-params'': this is also an important but a bit complicated tuning. Originally default value was creating small chunks causing huge cache and memory usage, so they switched to much larger chunks, but which can be too large for some applications (for instance when modifying only metadata of an image file, we want to deduplicate the data), so I came with this compromise ''15,23,19,4095''. |
| * ''borg list <repo>'' | * ''borg list <repo>'' |
| * ''borg info <repo>::<archive>'' | * ''borg info <repo>::<archive>'' |
| | * ''borg diff <repo>::<archive1> <archive2>'' |
| | * ''borg mount <repo>::<archive> <mountpoint>'' |
| | |
| |
| === Restic === | === Restic === |
| | |
| | * Create a Restic repository in the current folder: <code>restic init --repo .</code> |
| | * Note that encryption **and** password are mandatory, [[https://github.com/restic/restic/issues/4326|because]]. However you can store the password in a file in the repository, or use the a password file with ''--password-file''. |
| | * Create snapshots: <code>restic --repo <repo> --verbose --compression auto --ignore-ctime backup <path></code> |
| | * The chunker cannot be configured, contrary to Borg. It is equivalent to [[https://restic.readthedocs.io/en/stable/100_references.html#backups-and-deduplication|19,23]],[[https://restic.net/blog/2015-09-12/restic-foundation1-cdc/|21,512]], similarly to Borg's default [[https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#buzhash-chunker|19,23,21,4095]], but unlike my chosen values. |
| | * ''--compression'': unlike Borg, there is only choicies ''auto'', ''max'' and ''off'' |
| | * ''restic --repo <repo> snapshots'' to list snapshots |
| | |
| |
| === BTRFS snapshots === | === BTRFS snapshots === |
