Because my current work style is very fluid, where I hop between my office and home PCs, a laptop, a netbook and an iPhone, it made sense to try and keep my data where I could access it regardless of device. I basically have 3 categories of data: email, my web browser environment (bookmarks, stored passwords, history) and unstructured data (Word, PowerPoint, Excel, JPEGs etc. files). I use a few cloud based services to supplement these, but more on them another time.

Documents: I’m fairly neat when it comes to organising data on my laptop. Everything is in the My Documents folder. I used to use Beyond Compare to manually sync my desktop and laptop but for the last while I’ve been using Microsoft Live Mesh Beta to keep everything in sync automatically. A change or addition on any system ripples around to all the rest in real time. This means I can start work on a document in the office and seamlessly pick up where I left off when I get home without having to do a thing. On the new PC I just installed the Mesh client and left it to do it’s thing over night. It pulled my data from the online copy and placed it just where I’d expect it.

Email: POP is for noobs; IMAP is king. Seeing as Google are generous enough to give away 7GB+ of email storage I’d be a fool not to use it. A few years ago (pretty much as soon as it was available) I created an @garypigott.net Google Apps account and shoved all my email up there. Right now there are 118,527 emails, occupying 2298MB of space, and vitally, it’s completely searchable in exactly the same way you’d search the web using Google. I access exactly the same data using Gmail, Outlook, Outlook Express, Windows Live Mail, Thunderbird, and the iPhone mail client.

Web Browser Environment: My browser of choice is Firefox,  which is handy a) because it’s multi-platform and b) because of Weave. Weave is an add-on that moves your bookmarks, history and passwords into the cloud. You can install the add-on on each of your computers, regardless of OS and everything is kept in sync. Weave is open source, so surely some enterprising and generous developer out there will churn out a Weave add-on for IE, Opera and Safari…

Version 1 had a web UI for day to day management (starting & stopping VMs, tweaking networks etc.) and a separate Windows application for console access. VMware tried to make an all powerful web UI for version 2, and integrated the console functionality.

There must have been a backlash during the beta, because they’ve included VMware Infrastructure Client at the last minute. It leads to an odd division of capabilities. On the VMware Infrastructure Client I can create a new VM or import an appliance, but I can’t open an existing VM. On the Web UI I can create a new VM or open an existing one, but I can’t import an appliance.

What’s happened is that in version 1 you had a web UI for management and a console application for maintenance. Now both functions are split between interfaces, which is a pain in the rear.

Clouds are cool ‘n all, but what ever happened to K.I.S.S. (Keep It Simple Stupid!). Simple architectures go wrong in simple ways which are easy to understand and quick to fix. The team in MediaMax obviously didn’t fully understand what they needed to do to safely move all their customer’s data over, and everyone is out of a job as a result.

There are two elements to online security: data encryption and identity verification. If your data encryption is good enough it doesn’t matter who gets hold of your data. Being sure you’re sending the data to the right server adds to the security. Online backup typically deploys two layers of encryption: the data is encrypted using a key, and then the data is transmitted through an SSL tunnel that encrypts everything again. The SSL tunnel uses a digital certificate to verify the server is who it says it is. These certificates are issued by a trusted source, (Thawte, Verisign, Geotrust etc.) and provide the sender with a key to encrypt the traffic (the public key). Data encrypted with a public key can only be decrypted with a matching private key, and vice-versa. Only the owner of the certificate has the private key.

Software on the user’s end should throw up a big fat warning if the certificate isn’t issued by one of these trusted sources. Unlike Carbonite and Mozy, Ahsay doesn’t throw up a warning and just sends the data anyway. A determined individual can create their own certificate and keys and trick your PC into mis-routing your backup to their server (DNS cache poisoning is easier than you think). This gets them through the first layer; the SSL tunnel. They then have access to the username and password (which are transmitted unencrypted, because the SSL tunnel should be secure enough). This in itself isn’t a huge deal except for one thing…… To make things simple Ahsay’s default behaviour is to use the password as the data encryption key! They can then log in to your account and download all your data and decrypt it with your key. Carbonite & Mozy also use the same password/key shortcut but at least Mozy throws up a warning and Carbonite refuses to go any further if the key isn’t genuine.

Outsourcing to a managed service provider is the only thing that makes sense if you’re small and have these functionality requirements. I could sign up with someone like Intermedia on a syndicated Exchange server for $125/month. ActiveSync (mobile device synchronisation) is another $2.95/inbox/month, and there’s loads more optional, but essential extras, like additional storage. Basically at the end of it you’re looking at €150/inbox/year, which is a big improvement, especially when it means that I’ve no hardware to manage, and no housekeeping to do.

Google have had their Google Apps for Domains suite out for a year or so now. You get Gmail with your own domain name instead of @gmail.com, a calendar app, a basic browser based word processor, presentation & spreadsheet app that allows multiple users update the same file simultaneously, and a Wiki. We started using it informally when they launched the “Team Edition”, that gave you everything but Gmail, so you didn’t have to move away from your email provider. The Premier Edition gives you 25GB inboxes for $50/inbox/year, and the standard edition, with conventional Gmail adverts and a 6GB inbox is free!

We went with Google and threw the switch last Tuesday and it’s bedding in nicely. Over the past week though a couple of “wouldn’t it be nice if…” things have come up:

• Full contact information, and not just name & email address would be good.
• Proper contact sharing would be nice. Contacts are shared, but only in as much as the auto-complete function can pull addresses from other employee’s address books when you’re composing an email on gmail.com.
• The ability to sync contacts with Outlook and mobile devices would be a huge step forward.

Backups should be automatic. If a human has to do something to ensure your backups happen, then once and a while that human will do it wrong, so your backups are not 100% reliable.

Backups should be redundant. There’s no point doing a backup to a local disk, because it’s possible that the event that causes you to lose your data will cause you to lose your backup. he uses the example of Francis Ford Coppola, who had his laptop stolen earlier this year. Much of the data was irreplaceable, including a script for the movie he was about to start work on. He had a backup alright, but it was a USB hard disk. the thief stole the disk at the same time. A similar scenario I came across before was the company who instructed their sales reps to backup their laptops to CD-Rs. Unfortunately the reps had a habit of storing the backup discs in the laptop case….

Backups should be restorable. It’s an obvious one, but often overlooked. People diligently follow their backup procedure, but fail to ensure it actually works. With tapes this means frequent test restores, but tapes by their nature have a finite lifespan. The verification puts more wear on the tape and can damage the data while it’s being verified. Basically, even after you verify the tape you’re never certain that the act of verification didn’t damage the backup, so you need to verify it again, and again, and again…….

To be honest, it makes me feel a bit warmer in side that the solution my company offers ticks all the boxes….. Our online backup platform is a pure software solution, so there’s nothing to interact with. It just works in the background, and is monitored by the service provider, so it ticks the automatic box. Data is immediately transferred off-site over the Internet, so a theft or natural disaster isn’t going to take out your server and ours at the same time, so that’s redundancy taken care of. Hard disks can take a lot more read and write cycles than tape, RAID and high end storage technologies prevent data loss, and built in continuous integrity checking mean that backup data is difficult to corrupt and is spotted and corrected immediately if it happens, so data on our service is always restorable.

The Pirate Backup System – Network World

Asigra supports Fedora Core, Red Hat ES and SUSE ES. I don’t like Fedora for a number of reasons, and this is for home use so I’m not paying for RH or SUSE, so I’ll use CentOS, the free clone of RHES. I’ve allocated a 4GB disk file for the VM. The OS itself should fit under 1GB, but the Postgresql database can hit 2GB, so I need the space. I’ve allocated 256MB RAM to the VM, but backups will run a lot faster with more.

At the OS “Installation Type” screen, you select “custom” to keep the disk footprint as small as possible. Disable the firewall and SELinux. On the Package Group Selection screen, tick the box for minimal install and complete the installation.

The application requires Postgres, so firstly you install it with:
yum install postgresql-server postgresql-jdbc

You need to run it once to build the database and create the config files
/etc/init.d/postgresql start
does the trick.
To make postgresql launch at boot, I use “chkconfig –add postgresql“, “chkconfig –level 345 postgresql on” and to confirm “chkconfig –list postgresql“. Postgres needs some minor tweeks for Asigra. you must add the line “tcpip_socket = true” to postgresql.conf and “host all all 127.0.0.0 255.255.255.255 trust” to pg_hba.conf. You then need to restart postgres to load the new config by running “/etc/init.d/postgresql restart“.

The client has some other dependencies that get resolved by installing “yum install compat-libstdc*” The client installer must be run with the “-console” switch unless you’ve installed X. Agree to the EULA and accept the default install path of “/usr/local/DS-Client”. We’re only going to install the DS-Client on this box, with the Java DS-User UI to be installed elsewhere. We’ll select a normal installation. The self contained install is an evaluation mode where everything is stored locally. We enter “localhost” for the DB host, and “postgres” as the username. We can put in anything for the password. We also want the service to start at boot time. It is normal for the client to give a warning about a missing database during an initial install. At this stage you get prompted for username, client ID and server location. These would be provided to you by your service provider.

The next thing is the encryption level. Asigra supports everything from DES (very weak, avoid) right up to military grade AES256. AES is slow, and the higher the key length, the slower the progress. I usually go for AES128. It’s significantly faster than 256, and it will take millions of years to break, rather than trillions, so it’s still good enough. Asigra uses two encryption keys; a key specific to the PC and an account key shared amongst all backup clients on the WAN. This enables common files to be encrypted and stored once, with this common key, rather than multiple copies being stored with individual keys (inefficient), or all sites sharing a single key (insecure).

iSCSI is significantly cheaper because, in theory, you can use the same LAN to connect to your storage as you connect to the outside world. Basically standard SCSI data packets are wrapped in a TCP/IP wrapper and transmitted between hardware. Everything has an IP address, and it can even be routed over the internet. In practice however the CPU hit involved in achieving fast data transfer when every piece of data has to be processed by the CPU twice in each direction (once to wrap the data in a SCSI packet and once to wrap the SCSI packet in a TCP packet) means that the server isn’t much use for anything else. Thankfully you can get network cards that will do the process in hardware, but they’re even dearer than the fibre cards, so iSCSI isn’t a cheep option either.

AoE dispenses with the TCP layer altogether. It’s a very lightweight protocol that just uses the Ethernet network switch for routing, so you can use your server’s existing LAN hardware. Coraid have taken the decision to create a very low entry cost. It uses generic SATA disks, so all you need to buy from them is the AoE enclosure, which starts at < ???????4K for a chassis that'll take 15 disks. If it doesn't work out, you can re-use the disks elsewhere and just bin the enclosure. 750GB disks are ???????400 each, so I could deploy a 10TB array for < ???????1/GB.

I sincerely hope these guys succeed. Right now Coraid are the only manufacturer of this kit, and even though they're selling a lot of hardware to universities and research organisations (NASA are a customer), the iSCSI vendors are probably not happy with an upstart eating at their table, and may decide to kill them off, leaving customers in the lurch...

link

What do you do once you need more than 24TB in a single volume. Do you put everything behind a gateway, and consolidate it there, (and deal with a performance bottleneck and single point of failure), or use them as components of an copper Ethernet (rather than fibre channel) SAN and assemble the volumes at each server (which is a management overhead, and a big CPU load on each application server)?

The other tricky thing is what to do if a drive pops? Sun are touting hot swap capabilities, so you don't have to power off the server to swap out the disk, but they're not front accessible, so you need to pull the server out of the rack and pop off the top to change out a disk. You need to be very careful with your cable management at the back of the rack to reliably pull a server forward without disconnecting power, network, KVM etc. So far we've not taken the risk and just gone with a mix of front accessible EMC units and an IBM ESS 800, which is a standalone unit, designed for high-density storage.

Still, Sun do a 60 day trial, so I might just give my friendly neighborhood Sun reseller a call ….