# Backup module specification

Author: Ladislav Slezak < lslezak@suse.cz>

Changes:

- 2003-02-17: Profiles
- 2003-02-06: Star support (ACL backup), documentation update
- 2002-02-01: Added default excluded directories and file systems
- 2002-01-03: Multi volume problems with tar utility
- 2001-12-18: Changed archive file internal structure
- 2001-11-29: Changed solutions of some problems
- 2001-10-31: Multi volume archive
- 2001-10-25: Partition table storing updated
- 2001-10-15: New options in Backup settings dialog
- 2001-10-05: Screenshots update
- 2001-09-26: Detailed batch mode description
- 2001-09-06: Initial version

## Backup and system recovery solutions

### Backup

Utilities like dump, kbackup and others are indended for full system backup. These utilities do not use rpm database to get changed files, but they support incremental backup or client/server architecture or other features.

#### YaST1

YaST1 backup module compares file flags stored in rpm database with actual file properties and backups only files which differ, the result is small backup archive.

How is backup done in YaST1:

1. get list of installed packages
2. get list of files and modification flags for each installed package
3. select from list only modified files
4. remove duplicate entries from file list
5. find files which do not belong to any package (on demand): iterate through directory content (start at root directory), for each:

- file - check whether it is in package files list, if no add it to the backup list
- directory - check if it is in exclude directory list or it is on excluded filesystem, if no then get recursive file list from it

6. display list of files to backup, user choose files from list
7. display archiving options
8. run tar to create backup archive

YaST1 can not restore files from backup archive. Restoration of system is done by new system installation and manually updating files from backup archive. There is no way how to check whether installed packages are same as in backup time or if they are installed in same place (not relocated to another directory).

### System recovery utilities

Make CDROM recovery makes a bootable recovery ISO images, with full backup of system. System can be booted from CD and complete recovered from backup. Others utilities only make bootable floppy/CD with useful utilities, for example rip or zdisk. Gpart utility can guess lost partition table from disk content, this is useful if no partition backup is available.

## Documentation

YaST1 source, tar, fdisk, e2image and rpm manual pages, Partition rescue mini HOWTO, Maximum RPM - Verification, Bugzilla entry

## Features

List of features is available here.

## Implementation

### Backup

Backup archive contains files which differ from files in RPM packages and files which do not belong to any RPM package.

Changed files which belong to one package are stored to one subarchive with name

<package_name>-<package_version>-<date_of_backup>-<num>.<ext>

Currently num is 0 - this value is for future enhacement (backup archive created by package update have similar structure, so it is possible to include already created backup files to archive and this number can be used to differ file names). File extension ext is one of '.tar.gz', '.tar.bz2', '.tar' or '.star.gz', '.star.bz2', 'star' - depends on selected archiver (tar or star).

Some additional information will be stored in subdirectory info in the archive. This directory should contain files:

- comment - user comment, description of backup archive
- date - date of backup
- hostname - name of machine where archive was created
- installed_packages list of installed packages - output from `rpm -qa` command
- files list of all files present in the backup archive (useful for multiple volume archive)
- packages_info - list of files (with absolute path) and packages to which they belong

  Format of packages_info file:

      Package: package_name_with_version
      Installed: installation_prefix
      file1
      file2
      Package: package_name_with_version
      Installed: installation_prefix
      file1
      file2
      file3
      ...
      Nopackage:
      file1
      file2
      ...

#### Files not owned by any package

Files not owned by any package are stored in NOPACKAGE subcharive. It is posible to exclude directory or file system type from search. This allows to exclude from backup /tmp directory or directories with mounted NFS file system.

Default excluded directories are: /media, /windows, /tmp, /var/lock, /var/run, /var/tmp and /var/cache. Default excluded file system is iso9660, ntfs and file systems with "nodev" flag in /proc/filesystems file (e.g. nfs, proc, tmpfs...).

#### Access Control Lists (ACLs)

Backup module can store ACLs using star archiver - in this case subarchives (with *.star extension) have exustar format and they are created with -acl flag to store ACLs.

#### System backup

System backup will be stored in system subdirectory in the archive. This subdirectory can contain files:

- partition_table_<device>.txt - partition table info obtained by `sfdisk -d <device>` (human readable, can be used as input file for sfdisk - see sfdisk man page)
- partition_table_<device> - partition table stored by `dd if=/dev/<device> of=<file> bs=512 count=1` (raw data)
- e2image_<device>.tar.bz2 - compressed ext2 metadata obtained by `e2image <device>`

All package archives, files in info and system subdirectories are archived to one big archive by tar utility.

It is possible to store only list of changed files to file - script backup_archive.pl outputs file names from packages_info file. This list of files can be used to create another archive type.

#### Autoinstallation profile

Backup module also creates autoinstallation XML profile. Profile file name is same as backup archive except extension which is ".xml".

Autoinstallation profile contains installation settings which can be used in automatic installation withou user interaction.

#### Profiles

Yast2 Backup module contains profile manager. After backup archive is created it is possible to store entered settings to profile with specified name.

When backup module find any stored profile after start it opens profile dialog. Action which can be selected:

- don't use profile - start backup with default settings, standard workflow
- use profile without changes - read settings from profile, start backup immediately
- allow changes before use - read settings from profile, start standard workflow to allow changes

#### Problems and solutions

- How to detect changed files? Most reliable detecting of changed files is MD5 sum stored in RPM database. But computing MD5 sum of all files takes long time. There is option to use MD5 sum or only use size and modification time flags - this is faster, but less reliable.
- What if some package was updated after backup? For restoring must be present package of the same version!
  Solution: Store package version with list of changed files
- What if relocation was used in installation? (Package is reinstalled in another place.)
  Solution: Store installation prefix
- What if some file is owned by more packages? File can be owned by multiple packages, but if in one packge it has different modify time (content is equal) then this file will be unnecessary backed up. (This is possible if MD5 sum is not used.)
  Solution: Get list of files which are owned by more than one package. If a file has only different modify time in the RPM database, but MD5 sum and size tests passes and it's in the duplicate list then do not backup it. This check takes some time, but there should be a minimum of files which are owned by more than one package.
- Symbolic link problem: If a file in RPM package is installed in directory which is symbolic link to another directory then it seems like file in linked directory is not in any package (because it was not reported by rpm -qal command), but rpm -qf tells right owner.
  Solution: Stat all files owned by packages, remember device and inode numbers. While searching files not owned by any package check if it is in list of all files. If not check if its device and inode is in the list.
- Updated/other packages: If a package was updated or it isn't available on the installation sources it is not enought to store only changed files because RPM file cannot be available at restoration time (e.g. package update was modified, lost installation sources...).
  Solution: Get list of available packagest on the installation sources and backup unavailable packages completly.

#### Multi volume archive

Size of created archive can be larger than size of backup media. In this case archive file can be manually splitted to more files, but for beginners is more acceptable automatic splitting.

Tar archiver creates multiple volume archives if -M or -L is used. Each volume is standalone archive - files can be extracted without need of other volumes (except files which were splitted to more volumes).

Each volume is named according to pattern <volume_number>_<archive_name>. If user enters archive name /tmp/archive.tar then volume names are: /tmp/01_archive.tar, /tmp/02_archive.tar, ...

There is small problem with tar and multi volume archives: Volume size is multiply of block size, which is default 20 * 512 bytes = 10kiB. So if requested volume size is 32kiB (with parametr -L 32) created volume will have size 40kiB. There is option to set block size, but with parameter -b 1 (block size 1*512B) or -b 2 tar exits on SIGSEGV. With -b 4 it seems that it works, so default block size is set to 4*512 bytes. This was tested with tar version 1.13.18.

Solution is to create smaller volumes - if requested volume size is 35kiB create volumes with size 34kiB (with 2kiB blocks).

#### Architecture

Backup module has two parts: backup scripts and YaST2 frontend.

Backup scripts are written in Perl. First script (backup_search.pl) collects files to backup, second script (backup_archive.pl)creates archive file. Two scripts are required so that user can exclude some files from backup after search. These scripts are independent on YaST2 and they can be used in batch mode backup. In batch mode all parametres are entered on command line and backup is done without any interaction with user. This feature can be used in shell scripts to do backup automatically.

YaST2 frontend collects information from user, then it runs backup scripts and displays progress of backup.

### Restore

YaST2 retore module will use some information about system stored in backup archive and will allow simple restoring of files. Restoring partition table or restoring ext2 critical areas are dangerous operations, therefore these operations should be done only manually by expert user. Restoring partition table is described in Partition rescue mini HOWTO - chapter 8.1 Ext2 data are stored in image obtained by e2image utility, see e2image man page.

More information about restoring can be found in Yast2 restore module documentation.

## Workflow

### Overview

This is backup workflow overview. (source)

### Dialog details

(Outdated Qt designer workflow dialogs are available here.)

Archive options - Archive file name, achive type and optional archive description are entered in this dialog.

Archive file options - Multi volume archive options

Backup options - Number of selected parts to backup affects number of dialogs in workflow.

Exclude directory - Directories can be excluded from search.

Exclude file system - Filesystems can be excluded from search.

System backup - Partition table and ext2 partitions can be selected here.

Searching modified files - Progress bar is displayed while searching modified files.

Searching files - Progress of searching files is displayed here.

File selection - All found files are displayed here.

Creating archive - Progess is displayed while creating archive.

Backup summary - Results of backup procedure are displayed here.

Summary details - Detailed results are displayed here.