I have been introducing myself to Linux shell scripting.
I maintain a fair number of WordPress installations, and part of that maintenance is the task of backing them up. The actual backups are easy as they are done by simple scripts on the servers which are called by CRON tabs. Each server has three tabs – one to backup the database (on a nightly basis), one to backup critical files, such as uploaded files and configuration files (run once a week) and one to backup the entire site (run once a month).
While backup files on the server are all very well, they are not much use if the server collapses, so I need to transfer those backups, and this is where the need for shell scripting came in.
To retrieve the files, I initially had to fire up my FTP programme and then log into each server in turn, download the files and then delete redundant files from the server. That proved to be an extremely tedious job and of course it had to be done every day.
My first and obvious idea was to create a simple shell script to log onto a server and FTP a file down onto the local machine. There were two initial problems – I would have to have a script for each individual server, and there was the problem of specifying the file I wanted to download. For various reasons I wanted to be able to specify precisely which file I wanted to download (and the actual names varied from server to server), so this involved wildcards.
My initial solution was to create an array of variables – one set for each server containing the server address, the username, the password and where the file was to be stored locally, and some variables that were specific to the day, as each backup file was time stamped.
MYDATE=$(date +”%d.%m.%y”)
REMOTEFILE=*$MYDATE.*SERVER[0]=”domain1.com”
USER[0]=”user1″
PASSW[0]=”password1″
LOCAL[0]=”/home/sites/BlogBackups/Site1″SERVER[1]=”domain2.com”
USER[1]=”user2″
PASSW[1]=”password2″
LOCAL[1]=”/home/sites/BlogBackups/Site2″
I then created a simple loop which cycled through the array, connecting to each server in turn, downloading the files and then deleting the files on the server.
count=0
while [[ $count -lt ${#SERVER[@]} ]]
do
ftp -v -i -n ${SERVER[$count]} <<END_OF_SESSION
user ${USER[$count]} ${PASSW[$count]}
cd webspace/www/backups
lcd ${LOCAL[$count]}
mget $REMOTEFILE
mdelete $REMOTEFILE
bye
END_OF_SESSION
(( count++ ))
done
I as quite pleased with this effort as it ran perfectly and did precisely what I intended.
Until the end of the month, that is.
On the first of the month, I ran my script and it locked up.
A little investigation found the cause – it was downloading files greater than 200Mb, but on completing the download the script just stopped, for no apparent reason. There were no error messages, no timesouts, nothing. I searched the Web but couldn’t find any mention of a possible cure for this. In the end I posted a query on a couple of forums, and while they didn’t provide an answer they did suggest something else – that I try something other than FTP.
Unfortunately, for various reasons none of the suggestions worked, so I started my own search and found LFTP.
LFTP is essentially FTP with a load more bells and whistles. It took a while to sort it out as my only reference was the Web.
The basic command line to retrieve a file from the server is
lftp -u username,password ServerURL -e “get /webspace/www/backups/remotefile”
I modified this using mget rather than get as that allowed for wildcards. I used a loop to cycle through my array and that’s where the fun began.
First of all it started throwing up warnings left right and centre about bad certificates.
WARNING: Certificate verification: Not trusted
WARNING: Certificate verification: Expired
WARNING: Certificate verification: certificate common name doesn’t match
This was easily cured by adding a line at the bottom of /etc/lftp.conf – set ssl:verify-certificate no
My next problem was that it would access the first server all right but would refuse to connect to any other, giving an error that there was “No Route to Host”. It was logging in all right on Port 21, but was then trying random ports to do the actual transfer and kept cycling through them.
I cured the long wait between cycles by adding three more lines to /etc/lftp.conf –
set net:timeout 10
set net:reconnect-interval-base 5
set net:reconnect-interval-multiplier 1
While that sped things up a bit, it didn’t solve the problem! I tried forcing Active mode over Passive but that had no effect.
The solution in the end was to add yet another line to /etc/lftp.conf –
set ftp:ssl-allow false
I ran the script, deliberately including some large files and it worked.
My final script –
count=0
while [[ $count -lt ${#SERVER[@]} ]]
do
lftp -u ${USER[$count]},${PASSW[$count]} ${SERVER[$count]} -e “mget -E /webspace/httpdocs/backups/$REMOTEFILE -O ${LOCAL[$count]}; ; exit”
(( count++ ))
done
I added in a couple of extra features – the –E flag deletes the remote file but only on successful download, and the –O flag sets the target location for the file. I ran it against the full server load (having previously uploaded some of the large files) and it ran perfectly.
LFTP is an immensely powerful tool as it provides such commands as MIRROR which can be used to mirror a local folder with a remote one. I can see myself using it a lot more!