June 9, 2017: New automatic backup system: user crontab and zenity popups

For automatic backup of my user, I configured a system some years ago (using rdiff-backup), and I made a number of conception errors:

I used anacron, which is run by cron by default on Ubuntu. So I changed /etc/anacrontab by adding a backup task named "bckup". Problem: anacron does not examine the return code of the task, it considers it has been done in any case. The consequence is that in case of error I was compelled to start a program in the background, changing the date in the file /var/spool/anacron/bckup after anacron made his modification. Thus I used a sleep() command, which is in general a bad idea, as there are always some cases where the duration being set is not suitable. I was also compelled to catch the SIGTERM signal in the backup script to run the background program mentioned above if the computer was stopped whereas the backup was not complete or started (e.g. popup raised but not accepted); in this case the sleep duration is hard to set.
I changed the system crontabs, though only my user is concerned. This means in particular that DISPLAY=:0 had to be added at the top of /etc/cron.d/anacron such that the anacron task is able to display popups. I had also to add in my script lines as:
```
xauth merge ~jscordi/.Xauthority
cp -f /home/jscordi/.Xauthority ~
```
These lines were necessary for some Linux distribution (maybe Mandrake, or Ubuntu, I can’t remember), I cannot even say if these lines are still necessary at this time.

So I have decided to employ a user crontab instead, reading and writing a user file containing the date of the last successful backup. A user crontab file "crontab_file" is set with

$ crontab crontab_file

It is listed with

$ crontab -l

It is edited with

$ crontab -e

And removed with

$ crontab -r

Using the crontab command allows to avoid reloading cron manually. Moreover it is an abstraction layer, which is better to use than editing a cron file manually; as written in the manual page:

There is one file for each user’s crontab under the /var/spool/cron/crontabs directory. Users are not allowed to edit the files under that directory directly to ensure that only users allowed by the system to run periodic tasks can add them, and only syntactically correct crontabs will be written there. This is enforced by having the directory writable only by the crontab group and configuring crontab command with the setgid bit set for that specific group.

Indeed, on my machine:

$ sudo ls -ld /var/spool/cron/crontabs
drwx-wx--T 2 root crontab 4096 Aug 13 22:21 /var/spool/cron/crontabs
$ sudo ls -l /var/spool/cron/crontabs
total 4
[…]
-rw------- 1 jscordi crontab 1246 Aug 13 22:21 jscordi
$ ll /usr/bin/crontab
-rwxr-sr-x 1 root crontab 36K 2016-04-05 23:59 /usr/bin/crontab

How does it work? /usr/bin/crontab is executable by anybody; but only people of the crontab group (and root) can list (’x’) and write (’w’) in directory /var/spool/cron/crontabs. As /usr/bin/crontab has the setgid bit ’s’ activated (see here), and has group crontab, any user executing it will get the group rights of the crontab group, thus allowing to write in /var/spool/cron/crontabs; the created files will have crontab as group and the user name as owner: this is the case here for the file "jscordi": I am not in the crontab group, but I have written this file in directory /var/spool/cron/crontabs by using /usr/bin/crontab.

The sticky bit (see here) on /var/spool/cron/crontabs means that only the owner of a file in this directory can delete it (another user having write access will be able to modify it, possibly to an empty file, but not to remove it). Here this means that /usr/bin/crontab is not able to remove file jscordi; this is perhaps to ensure that every user of the system has a crontab file, possibly empty.

For a list of caveats preventing crontab to work, see here. I made one of the listed errors here:

The cron job specification format differs between users’ crontab files (/var/spool/cron/username or /var/spool/cron/crontabs/username) and the system crontabs (/etc/crontab and the the files in /etc/cron.d).

Indeed the user name ("jscordi" below) is not needed for a user crontab. I would rapidly have found the problem, but while using this syntax, one of the lines of my user cron file worked correctly (one-liner allowing to put on foreground and at random position any window containing "Automatic backup" in its title):

*/1 * * * *  jscordi unset w;for i in ‘wmctrl -d|sed -n "s/.*\* DG: \([0-9]*\)x\([0-9]*\).*/\1 \2/p"‘;do w[${#w[*]}]=$(( $RANDOM \% i ));done;wmctrl -R "Automatic backup" -e 0,${w[0]},${w[1]},-1,-1;wmctrl -R "Automatic backup"

So I thought that the syntax was correct. I have fallen in a common trap in bash: the use of ’;’ to delimit commands. If one command fails with an error, the other ones are still executed. "jscordi unset w" is interpreted as a command; it fails and the next command i.e. "for i in ..." is executed and indeed works, as the failure of the first part does not prevent the second part to work. Other lines of my crontab did not contain ";" and all failed. To avoid this trap, "&&" has to be used instead:

*/1 * * * *  jscordi unset w && for i in ‘wmctrl -d|sed -n "s/.*\* DG: \([0-9]*\)x\([0-9]*\).*/\1 \2/p"‘;do w[${#w[*]}]=$(( $RANDOM \% i ));done;wmctrl -R "Automatic backup" -e 0,${w[0]},${w[1]},-1,-1;wmctrl -R "Automatic backup"

This trap is related to the security problem in the Python subprocess module (cf here). By the way, note that ’;’ is not the equivalent of ’||’ (cf "man bash").

Curiously kdialog crashes when calling it from my user crontab file. Fortunately zenity works correctly. For instance:

$ zenity --question --text "toto?" --title "Automatic backup" --width 200

Here is my crontab file after all my experiments. I have included some interesting comments:

$ crontab -l
# $PATH is empty when entering this script, so we are compelled to indicate all
# necessary directories in the hard-coded way. Note that /bin and /usr/bin are
# necessary for some of the programs used in bckup_exthd.sh below.
PATH=/usr/local/bin/:/bin:/usr/bin
SHELL=/bin/sh
# June 2017: DISPLAY=:0 is necessary for wmctrl to work below. Otherwise
# "cannot open display" error message is obtained.
DISPLAY=:0
# Random positioning of zenity windows for backup, and raise them on top of the window
# stack. Note how it is important to escape the "%" in the following line (see
# http://www.hcidata.info/crontab.htm)
*/1 * * * *  unset w && for i in ‘wmctrl -d|sed -n "s/.*\* DG: \([0-9]*\)x\([0-9]*\).*/\1 \2/p"‘;do w[${#w[*]}]=$(( $RANDOM \% i ));done;wmctrl -R "Periodic backup" -e 0,${w[0]},${w[1]},-1,-1;wmctrl -R "Periodic backup"
*/1 * * * *  bckup_exthd.sh > /dev/null

And here is the backup script bckup_exthd.sh, provided "as is":

#!/bin/sh
####### CONSTANTS
# List of directories to backup.
DirToSave=(
"/home/"
"/usr/local"
)
DirNotToSave=(
""
)
MAX_DAYS_WITHOUT_BACKUP=3
EVENING_HOUR_WHEN_TO_START_BACKUP=22
PERIODIC_BACKUP_POPUP_TITLE_STRING="Periodic backup"
####### COMMAND LINE ARGUMENTS
# fflag to force backup (no hour test).
fflag=off
while getopts fc opt; do
    case "$opt" in
        f)  fflag="on";;
        \?)       # unknown flag
        echo >&2 \
        "usage: $0 [-f]"
        exit 1;;
    esac
done
shift ‘expr $OPTIND - 1‘
####### CODE
# Creation of the directory containing timestamp file if needed.
if [ ! -d /home/jscordi/.bckup_exthd_sh/ ];then
    mkdir /home/jscordi/.bckup_exthd_sh
fi
BackupReminder()
{
    # $1: directory to backup.
    zenity --title "$PERIODIC_BACKUP_POPUP_TITLE_STRING" --warning --text ’I will repeat my request for backup soon.
    Please be ready at this time’ 2>/dev/null
    exit
}
BackupError()
{
    # $1: error code, $2: error message
    zenity --title "$PERIODIC_BACKUP_POPUP_TITLE_STRING" --error --text "$1\n$2" 2>/dev/null &
    mail -s "bckup_extd.sh — error.  $1" jscordi << EOF
    $2
    A new backup attempt will occur soon.
EOF
    exit
}
BackupSuccess()
{
    # $1: directory to save
    # $2: backup directory on external hard drive.
    zenity --title "$PERIODIC_BACKUP_POPUP_TITLE_STRING" --info --text \
    "Complete backup of $1 on external hard \
drive (directory $2$1)" 2>/dev/null &
    mail -s "bckup_extd.sh — success (backup of $1 on external hard \
drive $2$1" jscordi << EOF
The backup is complete.
EOF
}
DirectoryBckupcopy()
{
    # $1: removable path (e.g. /media/)
    # $2: directory to backup.
    # $3: directories not to backup
    if [ ! -e $1/$2 ];then
        # The backup directory on external disk does not exist
        zenity --title "$PERIODIC_BACKUP_POPUP_TITLE_STRING" --question --text "First copy of $2?" 2>/dev/null
        if [ "$?" = "0" ];then
            mail -s "bckup_extd.sh - First copy of $2 on external hard drive" jscordi << EOF
            First copy of $2 on external hard drive.
            $(display_size $1)
EOF
        else
            BackupReminder $2
        fi
        echo "First copy of $2 to $1/$2"
        if [ ! -d $1/$2 ];then
            mkdir -p $1/$2
        fi
    else
        # The backup directory on external disk exists
        echo "Remove older than 1 month incremental backups for $1/$2"
        sleep 1
        set -x
        nice -n 19 rdiff-backup --force --remove-older-than 1M $1/$2
        error_code=$?
        set +x
        if [ "$error_code" != "0" ];then
            BackupError $error "An error occured during \
removal of oldest backup increments of $2 on external hard drive."
        fi
    fi
    exclude_dirs=
    for i in $3;do
        if [ ! -z "‘echo $i|grep $2‘" ];then
            exclude_dirs="$exclude_dirs $i"
        fi
    done
    echo "Incremental copy of $2 to $1/$2"
    # Two cases as the --exclude command does not accept an empty argument.
    if [ ! -z "$exclude_dirs" ];then
        nice -n 19 rdiff-backup -b --exclude $exclude_dirs --force $2 $1/$2
        error_code=$?
    else
        nice -n 19 rdiff-backup -b --force $2 $1/$2
        error_code=$?
    fi
    if [ "$error_code" != "0" ];then
        BackupError $error "An error occured during \
incremental backup of $2 on external hard drive."
    fi
    BackupSuccess $2 $4
}
last_backup_date=$(</home/jscordia/.bckup_exthd_sh/last_backup)
last_backup_date_in_seconds=$(date --date $last_backup_date +%s)
max_difference_in_seconds=$(($MAX_DAYS_WITHOUT_BACKUP*24*3600))
max_date_without_backup_in_seconds=$(($last_backup_date_in_seconds + $max_difference_in_seconds))
current_date=$(date +%G%m%d)
current_date_in_seconds=$(date --date $current_date +%s)
current_hour=$(date +%H)
if [ "$(pgrep -c bckup_exthd.sh)" -ge "2" ];then
    # An instance of the backup program is already running.
    exit
elif [ $current_date_in_seconds -lt $max_date_without_backup_in_seconds ] && [ "$fflag" = "off" ];then
    # Backup is already up to date. Use the ’-f’ option to force the backup.
    exit
elif [ $current_hour -lt $EVENING_HOUR_WHEN_TO_START_BACKUP ] && [ "$fflag" = "off" ];then
    # It is too early in the day to make backup. Wait $EVENING_HOUR_WHEN_TO_START_BACKUP hour, or use the ’-f’ option.
    exit
else
    echo "Starting backup program."
    removable_path=
    while [ -z "$removable_path" ];do
        list_of_size=‘df -h |grep "/media"|grep -v "/media/win"|awk ’{print $((NF-4))}’‘
        for i in $list_of_size;do
            corresponding_dir=‘df -h|sed -n "s/.*${i}.*\(\/media\/.*\)/\1/p"‘
            # We check that this directory is not in
            # ExcludedBackupDestination
            for j in ${ExcludedBackupDestination[*]};do
                if [ "$corresponding_dir" = "$j" ];then
                    continue 2
                fi
            done
            zenity --title "$PERIODIC_BACKUP_POPUP_TITLE_STRING" --question --text "Use $corresponding_dir for backup (size ${i})?" 2>/dev/null
            if [ "$?" = "0" ];then
                removable_path=$corresponding_dir
                answer=‘zenity --title "$PERIODIC_BACKUP_POPUP_TITLE_STRING" --list --text "What type of backup?" --column "Choice" \
                 --column "Type of backup" 1 "complete backup" 2 "quick backup (not Download directory)" --print-column=1 2>/dev/null‘
                # If answer is 2, we add ~/Downloads to the excluded directories.
                if [ "$answer" = "2" ];then
                    DirNotToSave+=(’/home/jscordi/Downloads’)
                fi
                break 2
            else
                continue
            fi
        done
        if [ -z "$removable_path" ];then
            # Display a request
            answer=‘zenity --title "$PERIODIC_BACKUP_POPUP_TITLE_STRING" --list --text "Please plug in an external hard drive for backup of your data, and mount it"\
                --column "Choice" \
                --column "Description" 1 "It is just mounted" 2 "Want to do it later" --print-column=1 2>/dev/null‘
            if [ "$answer" != "1" ];then
                BackupReminder
            fi
        fi
    done
    # ii instead of i because a variable named "i" already used in
    # DirectoryBckupcopy.
    for ii in ‘seq 1 ${#DirToSave[*]}‘;do
        ii=$((ii-1))
        DirectoryBckupcopy $removable_path ${DirToSave[$ii]} ${DirNotToSave[*]}
        # It seems that when a copy has succeeded, the beginning of a
        # new copy just after raises an error. So we sleep 10s, except
        # for the last directory to backup.
        if [ "$((ii+1))" != "${#DirToSave[*]}" ];then
            sleep 10
        fi
    done
    # Correct backup of all directories, we can update the timestamp.
    echo $current_date > /home/jscordia/.bckup_exthd_sh/last_backup
fi

Perusal, Synthesis, Bliss

June 9, 2017: New automatic backup system: user crontab and zenity popups