Author Archives: Richard

Rebuilding a WordPress site from nothing

A Pipe and a Keyboard Posted on by

Some weeks ago I took on a project.

The project was to rebuild a major blog with next to nothing to start with. The blog had been running for years and had an enormous following. The author had been diagnosed with cancer some years ago but had continued her writings. Last December she decided to discontinue the site, as her health was seriously deteriorated at that stage. She erased the site. Not long after, she expressed a sincere regret that the site was gone, and it was one of her dying wishes that somehow the site be restored. Last August, she died without seeing that wish fulfilled.

I assumed there were some backups lying around and offered to restore the site to its former glory. Unfortunately my assumption was wrong. I contacted the company which had hosted the site. Sorry! We don’t keep backups that long. So the call went out to see if anyone had miraculously got a backup of their own.

The first to respond was the blogger’s husband who was as anxious to restore the site as his wife had been. He announced that there were backups on the author’s PC. I duly received 1,419 files (one per post) to find that they were all in MHT format. MHT contains all the information for an individual page including HTML, JS, CSS and images. However, MHT is not a universally recognised format so putting them on-line would only work in Internet Explorer (or in some other browsers with a specific add-on installed). I had to find a way of extracting them, each into its component parts.

After some searching, I came across MHT2HTM which happily extracts the required contents and works on Linux. I installed it and ran it. It worked perfectly.

So I now had 1,419 folders, each containing everything required to reconstruct the posts. However, files in folders – 59,337 files weighing 834Mb – are not very efficient as an archive. They are bulky in the extreme and aren’t searchable and can’t be listed by anything other than folder name. The answer was to extract the information from each folder, to move the image files to a central point and to populate a database with the correct information.

I set about writing a programme (in PHP). The programme was simple enough in concept – it had to cycle through each folder, reading the HTML file and extracting al the required information to place in the database. This meant stripping all the HTML off the file leaving just two parts – the post content and the comments. These two were added so that the comments became part of the post. That was moved to the database along with other information which had been extracted such as the author (there had been quite a few contributors to the site), the title, the name (slightly different to the title as far as the database is concerned) and the date of posting.

In the meantime I had taken over the domain name and had set up a server with a WordPress site. I uploaded the database I had created and the first generation of the archive was live.

Also in the meantime I had received a few (!) more files from others who had had the foresight to backup some of the original.

One batch of 1,462 files was also in MHT format. I ran that through MHT2HTM and I now had 1,462 folders. Unfortunately the two archives had been saved using slightly different parameters so I couldn’t compare the two directly. It would have to be done automatically.

I took my home-grown PHP setup and modified the code. It now had to make sure that the record didn’t exist already from the first run. This wasn’t quite so simple as because of the different parameters, there were some variations in post titles. I had to generate a unique key for each post (which meant running the first batch again) and compare keys. The key was simple enough – just strip all non-alpha-numeric characters from the title and replace spaces with dashes.

Next I had received some more copies of the site. A couple had used WGET to download the site and a couple had used HTTrack.. These of course gave me a load of files all with different layouts and small but important differences in the HTML coding.

Once again I rewrote my PHP programme and updated the database. I was now on a third and fourth generation of the site, each time adding a few more missing posts.

I had been dipping through the WayBack machine which had also sporadically taken copies of the site. Then I discovered wayback-machine-downloader. I downloaded and installed it and did a test run. It seemed to do the job so I did a live run. About eighteen hours later it finished.

Naturally, the downloader had dumped all its results into a series of folders with yet another layout. This time I tried a different approach – I used “locate” to find the index files and piped the result into a file. I discovered that if I created an array from that file using “explode” and “/” that the required files were to be found in all elements with a count of 12. This made the job easier. Once again, I used a modified version of the PHP programme to generate database entries.

I think I have gone as far as I can. There are still some small tweaks to be made to the site, including a few duplicate posts that somehow crept in. I still have to find some lost images as well, and maybe sort out additions to the various categories.

The result of all this effort can be found here:

The Anna Raccoon Archive.

 

PhpMyAdmin Settings

A Pipe and a Keyboard Posted on by

I have a LAMP server running on my local machine. The version I use is Bitnami.

I use this server quite extensively for testing code, site layout and various tests, and therefore use PhpMyAdmin a lot. There are two things that bug me constantly though – the first is the limit on file size to import data and the other is the timeout. While the need for these on a live production platform is obvious, they are just an irritation on a local device.

To change the timeout, open the file apps/phpmyadmin/htdocs/config.inc.php The location may depend on your variant of LAMP but a quick search should find it.

Change the setting $cfg[‘LoginCookieValidityDisableWarning’] to ‘true’.

Next find the line containing $cfg[‘LoginCookieValidity’] and change the value to the required timeout.

To change the maximum upload filesize, open the file apps/phpmyadmin/conf/httpd-app.conf Again, your location of this file may be different.

Change php_value upload_max_filesize and php_value post_max_size to the required value

Keeping up to date

A Pipe and a Keyboard Posted on by

This site has been somewhat quiet for a while.

Usually I post here on Linux related matters, but things have been very quiet on that front.  Instead I have been turning my attention to WordPress and hosting.

There is a major upheaval on the horizon with the arrival of PHP7.  Most sites currently run on PHP5 but a lot of coding has been depreciated and in 7 has been eliminated altogether.  So at some stage in the future, hosting companies are going to upgrade to PHP7 and sites across the board will suddenly lose a lot of functionality, or worse.

So I have been spending a considerable amount of time trawling through my sites (there's a lot of them) finding plugins that aren't PHP7 compatible.  A lot of them aren't.  Some of the latter are essential, in which case I contacted the authors and they updated their code (they were extremely grateful for the warning!).  Some plugins that were not so critical I just replaced with suitable alternatives and some I just left to chance.  The plugin I used in this part of the exercise is PHP Compatibility Checker.

Ultimately I made the switch on the majority of my sites, leaving just a few that as yet haven't got PHP7 as an option on the servers.  I hope to migrate those sites to newer servers in the coming weeks.

Another factor I have been working on is to upgrade sites to secure connections, as this another way the Internet is moving.  Google apparently intend to (or actually do) penalise non-secure sites.

The main problem with moving to SSL is the need for certificates, which are expensive and are beyond the scope of the hobbyist or even the small business.  The simple answer I found is to move all sites to CloudFlare Content Delivery Network.  As part of their free package they provide easily installable certificates.

Lastly (but by no means least) I used Weaver II as a theme builder on many of my sites.  While this was a very powerful theme creator/customiser it is now depreciated and has been replaced by Weaver Xtreme.  At first I found that the latter had a very steep learning curve and was reluctant to switch.  However I persevered and eventually found that Weaver Xtreme is immensely powerful and is far superior to the old Weaver.  It is also fully PHP7 compatible, uses more efficient coding and is therefore a lot faster.

In the course of all this work, I kept a stern eye on page speed, tweaking plugins (and replacing many) with an ultimate goal of getting every site on my books to load in under 3 seconds.  That was not easy.  One client in particular had a great fondness for sliding image galleries on his front page – six or seven of them.  On of those galleries alone contained 36 images, none of which was reduced in dimensions or compressed.  The overall size of the front page was well in excess of 50Mb.  Plugins and tweaks can do a lot, but this was the only case where I had to intervene and strongly suggest some alternative layouts!

So, as you can see, I haven't been idle! 

Bad interpreter error

A Pipe and a Keyboard Posted on by

I use shell scripts for a few jobs.

Recently I rebuilt a laptop and installed a few scripts from a working machine.  The other day I went to run one of the scripts from a command line and got the following:

/bin/bash^M: bad interpreter: No such file or directory

Now the new machine had an identical setup to the old and the script I was trying to run had the right permissions(and of course ran perfectly on the old machine), so the problem had to lie with the formatting of the script file ( and the ^M was a bit of a hint too?).  Somehow the file transfer had corrupted the script file and trying to edit it didn't fix it.

A simple fix. 

I used the following:

sed -i -e 's/\r$//' myfile.sh

The file then ran perfectly.

 

Hidden traffic

A Pipe and a Keyboard Posted on by

This site has been somewhat dormant for quite a while?

It's not that I had forgotten it – I have just been busy elsewhere.  Also my Linux has been running very smoothly so I have had no problems to resolve.

The intention of this site was to act as a notebook for myself, to which I could refer if I ran into similar problems again, and in fact I frequently do refer to it.  However, others find my little hints and tricks helpful so at least the site is doing its job properly!

While traffic to here is low but constant, I would consider it a bit of a backwater on the Internet.  So why are hackers so interested in it?  In the last 48 hours, there have been 2,770 attempts to hack the Admin password, and they're still coming.  As each of those 2,770 attempts have instantly been locked out of the site, I shudder to think how many further attempts would have been made if they had been allowed unlimited access to the login page?

It just demonstrates how important security is.  Even a very small site like this is prone to attack, and without security, it could be now hosting a dodgy pharmaceuticals site, dishing out porn or just acting as a slave to attack other sites.

Popularity isn't all it's made out to be!