belial.co.uk

I have run an Ubuntu desktop environment for a fileserver I run at home for several years. About 2 years ago I had a backup-scare, so plumped for some 1Tb drives configured as RAID mirror to ensure the same thing didn’t happen again.

Since then I have a good routine of backing-up crucial data, and felt the RAID was overkill, and the benefits were negligible. Since my 8.04 Ubuntu install had fallen out of LTS and was creaking (i.e. java VMs kept falling over and other assorted oddness) I decided to rebuild the machine from the ground up with Ubuntu 12.04 which lots of peopls had been raving about online.

Trying to get rid of RAID was my first obsticle. Turing off the RAID controller… not a problem. Quick tampering with the BIOS sorted that out. But since RAID writes some special data at the beginning of the drive not normally accessible by a format, the drives were still showing up as a RAID array. A quick DBAN session sorted that out. I say quick… it took 6 hrs… Anyway, I burnt my Ubuntu CD off, and slapped it in the machine, went through the setup/install only to be told that the 60Gb drive I was using for my root file system was dirty/bad/not-ok. This wasn’t too much of a surprise since I had salvaged it from an old laptop years before. Ok, so lets try this 60Gb instead? Nope, ok… what about this 250Gb I have lying around for a rainy day. No… ok hang that, I’ll install the root file system to one of the Tb drives, and then mount my home directory to the other.

All goes swimmingly, until the final boot. I get the new Unity desktop, but it’s juddery and slow. I have a GTX260 in my server (!) so wtf. a quick “top” identifies that compiz is causing 100% cpu usage. Huh… well this is a server, lets kill off compiz. Right, that’s dead but now (doh!) the desktop manager won’t show because it needs compiz. Some googling later and various attempts to correct it lead me to the decision fork that rebuilding the machine and disabling compiz properly (like I should have done first time around) is going to be quicker than trawling through askubuntu. Ok installation screen… huh? My 1Tb drive is bad and can’t be installed to. Argghh. Ok install on to the other one… this one is also bad? Ok, back to DBAN. 5+ hours later. Back to the install screen… they still show up as bad. This is crazy.

Short trip to Centos, and I now have a working fileserver again. Ok, the fact that selinux was on by default caught me out when I was trying to sort out the samba shares, but otherwise it’s been the dream I should have had with Ubuntu.

I’ve really enjoyed Ubuntu, but 12.04 seems to be a massive regression in usability. I’ll probably put it on my eeePC later in the year, but for now it can sit pretty on other peoples machines, cos I’m not going to touch it with a bargepole.

So, belial.co.uk and a number of other sites I run, are serviced by a single VPS running with 1&1. I have a nice vhost setup that lets them all run on the same apache server, without them colliding in any way. It’s economic, and for the amount of actual traffic I get, perfect for my needs.

On the 11th May 2012 all my websites became unavailable. I’m still not entirely sure why it fell out of the sky, but what I do know is that some crucial libraries went walkies after a regular site backup. It’s possible I could have done some forensics to find out quite what happened, but in all honesty, there is nothing there of much value or that couldn’t be recreated from backups. If it was a hack, they didn’t do a very good job.

What made the failure a little more interesting was that SSH, MySQL and even ICMP services were down and also refusing to start back up? Certainly the SSHd service being down was a major pain in the backside, because I had no way of then dialling in to administer the server. My only recourse (as confirmed by 1&1 support) was to re-image the server. i.e. re-install the operating system and build it from the ground up again. Nice opportunity to fix the server up to the latest versions of everything, but came at the same time as my personal intranet fileserver started creaking… so not great timing really.

Anyway, as you can tell, it’s all working again now… Maybe I start using this for some other work. It’s been over 5 years since gothornot.com had an overhaul, so probably time it received some care and attention, it’s looking rather haggered IMHO.

My next blog post will be on my woes fixing my fileserver which was utter, unadulterated joy! :/

About 4 years ago I bought my dad a Buffalo Linkstation NAS. It has about 450Gb of capacity, and basically exposes itself as a Windows File Server. You can set up a bunch of clever shares, users and groups etc… but for simplicity I’ve left it with just guest accounts.

He has used it for moving important data onto, from his laptop. This has largely been documents and images, but it’s now becoming an unwieldy task for him to manage. He needs a scheduled backup!

I toyed with using the default Windows Backup thang, but for some reason it won’t play ball. I guess its the unusual Samba implementation that is throwing it off… anyway, I need another route. At home I use a neato Synctoy setup, but I wanted to do this in the least invasive way, without using any downloaded applications. A straight “copy” is not going to cut it. He now has about 148Gb of data that he wants to backup, and copying is going to bottleneck his PC while it’s occuring. I needed something a bit more inteligent… robocopy!

Robocopy is available in Windows 7, and does a pretty decent job of backing up, even over Samba. First to design the switches. Thankfully I had previously set up his PC so that all his documents, pictures, music etc. lived in directories rooted on the D drive. So

robocopy D:\ \\buffalo\Sabre_Backup /S /Z /XA:H /W:5 /R:1 /COPY:DT /dcopy:t /XO /XD “D:\System Volume Information” D:\$

• Ok, so lets walk though this. “D:\” is the source and “\\buffalo\Sabre_Backup” is the target… this much is pretty simple. Now the switches.
• “/S” traverses directories recursively. so everything from “D:\” downwards will be matched.
• “/Z” is supposed to allow robocopy to restart if it is stopped mid-way through a copy process. Very useful for scheduled tasks.
• “/XA:H” excludes any hidden files or directories. The only things that should be backed up are normal files… not hidden system ones. It’s a bit of a precaution really.
• “/W:5 /R:1″ reduces the retry timeout to 5 seconds rather than 30, and makes it only retry once before skipping. This is required incase my dad is watching a picture or such like and the file-lock won’t release to allow the copy.
• “/COPY:DT” makes robocopy only copy the data and the timestamp in the file and nothing more. Security permissions passed on may not be recognised in the instance of a recovery situation, so I want all these files to be simple.
• “/dcopy:t” does the same as “/copy:d” but for directories also. Both these retentions of the file date allow robocopy to get inteligent about which files it has to copy and which it can omit.
• “/XO” excludes files which it already has the latest version on the destination… really crucial for this task!
• “/XD” makes the following space-separated patterns excluded from the match. In this case I wanted to ignore the System Volume Information directory and the Recycle Bin respectfully.

Now you have your command line setup. Definately run it up a couple of times to make sure you’re getting the target backup you expect. Dropping in new temp files in the source tree to make sure it’s picking them up is also a good idea.

You want to make this run on a schedule… you’ll need the task scheduler. Go to your start bar and type the following in to the box provided:

taskschd.msc /s

In the top right of the opened application you should see the option to “Create Basic Task…”. Click this. Name and describe your task as you see fit. Trigger defines how often your task will run. Action; pick “Start a program”. Program is:

robocopy

and your arguments are:

D:\ \\buffalo\Sabre_Backup /S /Z /XA:H /W:5 /R:1 /COPY:DT /dcopy:t /XO /XD “D:\System Volume Information” D:\$

Don’t leave your arguments in the program box, it makes a bad choice about how your arguments are listed and will result in you messing about trying to find out what it is that is making your beautiful command-line instruction fail.

And thats it… you should try running it from the scheduler to make sure everything is working as it should. A command box should appear with taskeng.exe in it’s title… it will rattle though your files and you’re done! Another satisfied customer!

Struts2 is a MVC framework I’m trying to get to grips with right now. One part that I found that took and inordinate amount of time to sort out was the IDE setup, and getting a simple HelloWorld working.

There are numerous good websites that explain how to use Struts2 once you’ve got it running. I’m a Java novice, and so what a lot of programmers were taking for granted, was a steep learning curve for me. Here are my own notes I used to get a second system up and running:

<version>${project.version}</version>

<packaging>war</packaging>

<name>Struts 2 Blank Webapp</name>

<properties>

<struts2.version>${project.version}</struts2.version>

</properties>

<version>1.0-SNAPSHOT</version>

<packaging>war</packaging>

<name>Struts 2 Blank Webapp</name>

<properties>

<struts2.version>2.3.1.2</struts2.version>

</properties>

I really can’t take any credit for the following passage. When I was setting up my SSD these are the instructions I followed, and other than a small amount of registry wrestling to allow certain security patches to find the right drive they have served me well. The page that they are sliced from is sadly no longer with us (www.highlyillogical.org) but with the magic of the www.waybackmachine.org I’m able to recover the article and re-post it here for you all to enjoy:

Danger! Doing this might mess up your computer PERMANENTLY.

I’ve been messing around with my computer, trying to get all my programs installed on a very large software RAID0 drive on Windows 7. (Might also work on Vista.)

I hate the way that Windows by default puts everything on your C: drive, with no option to change it. My Program Files and User directory often account for most of the stuff on my computer, so I like to have them on the biggest (and fastest) drive possible.

To this end, I built a new install of Windows 7 on my machine, which contains 6 500G SATA drives. I installed Windows into a 50G partition on the first drive, and created a software RAID0 (stripe set) volume across all the remaining space. – Doing this is *significantly* faster than using the fakeraid RAID0 controller on my motherboard. (More on this in another post.)

So, I have a 50G C: drive, and a >2TB E: drive. Belive it or not, it *is* possible to get all the data onto the E: drive without problems. It’s best to do this with a new install, or at least make sure you have very good backups though.

Unfortunately, you can’t just move Program Files, Users, etc., while logged onto Windows. Even in safe mode, it’s using your registry data from your profile, and a number of files from Program Files. Some files in Program Files are also owned by “TrustedInstaller” and can’t even be moved by Administrator. There is a way around this though.

The solution involves using NTFS Junctions – basically a magic directory that works as a pointer from one place to another. So, you can create a junction on your C: drive called “Program Files” that actually points at “E:\Program Files”. Any request to a file in “C:\Program Files” actually gets redirected to a folder on E:.

In order to create the junctions, we need a program from Microsoft called… wait for it… “Junction”. You can download it here. Open the zip file, and copy junction.exe to c:\Windows.

Of course as already mentioned, the files can’t be moved whilst Windows is running, even in safe mode. So, in order to move them to another drive, the computer must be started in rescue mode from the original install CD. BUT there’s no access to the Junction command from there. So… The first real step is to create “dummy” junctions, which will be renamed later… As follows:

Open a command prompt as administrator and run the following commands:

• mkdir “e:\Program Files”
• mkdir “e:\Program Files (x86)”
• mkdir “e:\Users”
• mkdir “e:\ProgramData”

Then create the junctions:

• junction “c:\JProgram Files” “e:\Program Files”
• junction “c:\JProgram Files (x86)” “e:\Program Files (x86)”
• junction c:\JUsers e:\Users
• junction c:\JProgramData e:\ProgramData

Of course, you only need “Program Files (x86)” if you’re on 64-bit Windows.

You should now notice that there are 4 junctions on your C drive that point to the empty folders on E:.

Next up, reboot the computer with the Windows DVD in the drive. Ensure your BIOS is configured to boot from the CD drive first, and hit a key when it says “Press any key to boot from DVD” before Windows starts.

When the install system is booted, choose to “repair your computer”. It should find your copy of Windows, so do a manual repair. When prompted, choose to repair using the command prompt.

You should now be able to move the files to the new drive:

Remove the dummy folders as they have the wrong permissions:

• rmdir “e:\Program Files”
• rmdir “e:\Program Files (x86)”
• rmdir “e:\Users”
• rmdir “e:\ProgramData”

Copy the files from C: to E:. After each xcopy command, press D for directory, when prompted

• xcopy /Q /E /B /K /X /H “c:\Program Files” “e:\Program Files”
• xcopy /Q /E /B /K /X /H “c:\Program Files (x86)” “e:\Program Files (x86)”
• xcopy /Q /E /B /K /X /H “c:\Users” “e:\Users”
• xcopy /Q /E /B /K /X /H “c:\ProgramData” “e:\ProgramData”

If you got no errors above, remove the copies from C:

• rmdir /Q /S “c:\Program Files”
• rmdir /Q /S “c:\Program Files (x86)”
• rmdir /Q /S “c:\Users”
• rmdir /Q /S “c:\ProgramData”

Finally, rename the dummy junctions to have the names of the folders on C: you just removed

• ren “c:\JProgram Files” “c:\Program Files”
• ren “c:\JProgram Files (x86)” “c:\Program Files (x86)”
• ren “c:\JUsers” “c:\Users”
• ren “c:\JProgramData” “c:\ProgramData”

After this you should be able to reboot, and if everything has worked Windows should boot normally, but all your files now exist on the other drive.

You don’t even need to change anything in the registry, as anything that points at the default location on C: gets automatically redirected!

There are several other instructions for doing similar effects, but this is the one I followed and I can vouch for.