I don’t have a backup system for home (which is where this site, and others are located), and I have generally relied on duplicating enough of my important stuff between friends and other computers. That, and I have a RAID5 setup for my large storage, and then home directories and website stuff is on a RAID1 ZFS volume. This doesn’t prevent accidental “oh-no”s, but it does protect me from some hardware failures.

Last year when I upgraded to the new server, I lost a lot of data because I forgot to backup all of my MySQL databases. I like to think I can learn from my mistakes, so a full year later I finally did something about it and signed up for Amazon’s S3 service.

The pricing is pretty nice, and I don’t have all that much data to backup. I figure, I’ll use up a few GB in total, and keep the monthly price around $1 - $2. That seems worth the price for off-site backup’s.

Now, I have 3 main websites that I need to backup, and one test site that I like to play around with:

• http://www.m87-blackhole.org/ <- This is the first domain that I owned, and its the site where my family checks out new photos
• http://www.willowoakboarding.com/ <- My parent’s site for their boarding ranch. I’m glad they have no concept of a SLA, or that things need to be backed up
• http://www.mywushublog.com/ <- This site of course, where I claim my own identity on the internet
• http://www.evil-genius-network.com/ <- a test domain, but now I run a little OpenID service, For one…

After a quick “FreeBSD s3 backup” Google search, I found Gary Dalton’s blog post: http://dvector.com/oracle/2008/10/18/backing-up-to-amazon-s3/. After reading this post, I formulated my plan of attack:

• Sign up for S3, create a “bucket” for each site
• Use something to interface with S3 ( duplicity )
• Automate MySQL and PostgreSQL backups
• Create a service account to run both s3 and db backup scrīpts as
• Set up a cron job for backups

So, after I signed up for S3, I had to create the buckets. I couldn’t find a way to do this though my Amazon account settings, so I created a little ruby scrīpt.

$ sudo gem install aws-s3
$ vim make-bucket.rb


#!/usr/local/bin/ruby
 
require 'aws/s3'
 
AWS::S3::Base.establish_connection!(
:access_key_id     => 'my-s3-key-id',
:secret_access_key => 'my-s3-secret-access-key'
)
AWS::S3::Bucket.create('mywushublog')
AWS::S3::Bucket.create('willowoak')
AWS::S3::Bucket.create('m87-blackhole')
AWS::S3::Bucket.create('evil-genius-network')

$ ./make-bucket.rb
Next, I had to install duplicity and py-boto
[root@server ~] cd /usr/ports/sysutils/duplicity
[root@server duplicity] make install
…
[root@server duplicity] cd ../../devel/py-boto
[root@server py-boto] make install clean
…
[root@server py-boto]
Next step, create a user (with access to shared data, and website data) to run the backups with the adduser command…
[root@server py-boto] adduser -g shared-data -G www -s /bin/tcsh -w random s3backupuser
…
[roott@server py-boto] su - s3backupuser
In tcsh, you can `set autolist’ to have the shell automatically show
all the possible matches when doing filename/directory expansion.
%
I’ll have to set my Access ID and Access Key in the s3backupuser’s environment, as well as a GnuPG passphrase so the backups are encrypted (and compressed). I mean, I trust Amazon, but not THAT much
% vim .cshrc
setenv AWS_ACCESS_KEY_ID my-s3-key-id
setenv AWS_SECRET_ACCESS_KEY my-s3-secrect-access-key
setenv PASSPRASE AVeryRandonPassphraseForGnuPG
Next, I copied the very useful automysqlbackup.sh scrīpt into a separate scrīpt for each website. I could have just dumped every database that was running, but I wanted to segregate each site’s databases into a different directory. So, I’m complicating my cron job by running multiple backup scrīpts, but I really want to make the end result easily readable and identifiable by me. So for each site, I create a directoy under /u01/backups:
%ll /u01/backups/
total 8
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:46 evil-genius-network
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:47 m87-blackhole
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:46 mywushublog
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:47 willowoak
Next was the s3-backups.sh scrīpt, which is very crude and simple. If I’m really motivated, I’ll make it nicer but I’m lazy and if I don’t need anymore functionality then I’ll just leave it.

~/bin/s3-backups.sh:

#!/bin/sh
duplicity /www/www.willowoakboarding.com s3+http://s3.amazon.com/willowoak/www
duplicity /www/www.mywushublog.com s3+http://s3.amazon.com/mywushublog/www
duplicity /www/www.m87-blackhole.org s3+http://s3.amazon.com/m87-blackhole/www
duplicity /www/www.evil-genius-network.com s3+http://s3.amazon.com/evil-genius-network/www
 
duplicity /u01/backups/willowoak s3+http://s3.amazon.com/willowoak/db
duplicity /u01/backups/mywushublog s3+http://s3.amazon.com/mywushublog/db
duplicity /u01/backups/m87-blackhole s3+http://s3.amazon.com/m87-blackhole/db
duplicity /u01/backups/evil-genius-network s3+http://s3.amazon.com/evil-genius-network/db

And last but not least, a cronjob to tie it all together:
% crontab -e
@weekly ~/bin/s3-backups.sh
@weekly ~/bin/mywushublog-mysql-backup.sh
@weekly ~/bin/willowoak-mysql-backup.sh
@weekly ~/bin/m87-blackhole-mysql-backup.sh
@weekly ~/bin/evil-genius-network-mysql-backup.sh
I can check the status of a backup by running duplicity with the ‘collection-status‘ flag:
%duplicity collection-status s3+http://s3.amazon.com/mywushublog/db
Last full backup date: Sat Apr 25 15:08:02 2009
Collection Status
-----------------
Connecting with backend: BotoBackend
Archive dir: None
Found 0 backup chains without signatures.
Found a complete backup chain with matching signature chain:
-------------------------
Chain start time: Sat Apr 25 15:08:02 2009
Chain end time: Sat Apr 25 15:08:02 2009
Number of contained backup sets: 1
Total number of contained volumes: 1
Type of backup set:                            Time:      Num volumes:
Full         Sat Apr 25 15:08:02 2009                 1
-------------------------
No orphaned or incomplete backup sets found.
I can also list the files:
%duplicity list-current-files s3+http://s3.amazon.com/mywushublog/db
Last full backup date: Sat Apr 25 15:08:02 2009
Sat Apr 25 15:05:11 2009 .
Sat Apr 25 15:05:10 2009 daily
Sat Apr 25 15:05:10 2009 daily/mywushublog
Sat Apr 25 15:05:10 2009 monthly
Sat Apr 25 15:05:10 2009 weekly
Sat Apr 25 15:05:11 2009 weekly/mywushublog
Sat Apr 25 15:05:11 2009 weekly/mywushublog/mywushublog_week.17.2009-04-25_15h05m.sql.gz
Pretty sweet automated backup process. It is a lot cheaper than tapes or additional disk storage. With S3, I also don’t have to worry about buying additional hardware, the maintenance of a library or tape drive (which is what I had a few years ago, what a headache).

Lately at work, I’ve been involved with a very large file system that is being export from Solaris 10/ZFS to windows and OS X users via Samba. Even with a very large Sun server (T5220) a lot of users are complaining about the slow performance of the system. I’m not going to go into details, but what it has prompted me to do is to look into what I use at home (FreeBSD + Samba) for my network storage needs, and see if I can improve the performance of it.

Well, I was checking out the very useful Whats cooking for FreeBSD 8, unrelated to my Samba needs, when I noticed this post on Ivan Voras’ blog. Ivan has some really cool information there, and with this new knowledge I began my update from running Samba 3.0.28a,2 to 3.3.3,0 (the comma represents the Ports version).

Hvala Ivan!

I’ve discussed my new FreeBSD environment before here, with that, here are some quick details:

• OS: FreeBSD 7.1
• Intel Core 2 Duo E6750 (2.66Mhz 4MB cache)
• Intel S975XBX2 workstation motherboard
• AMCC 3Ware 9650SE 4 port SATA RAID controller (4x PCI-e)
  • BatterY backup for the 3Ware so I can enable cached writes
• 2GB ECC Crucial Memory Kit
• ASUS EN6200 LE 16x PCI-e nVidia GFX card
• 4 Western Digital 1TB Drives

The 4 1TB drives create a nice 2.6TB (RAID5) array which I used Samba to share out to my 4 other systems in the house (which is a mix of Windows XP and Vista, sorry, no OS X). I do a lot with this array, any work that Michele and I do like photo editing, word documents, media files, etc.. all gets saved to this volume. Needless to say, this volume is accessed A LOT, and if the house ever caught on fire, I’d save the server before the family (Hey, I could still look at their pictures and videos…).

The Old Configuration

samba 3.0.28a:

[root@server samba3]> make showconfig
===> The following configuration options are available for samba-3.0.34,1:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=on "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     ACL_SUPPORT=on "With ACL support
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=off "With Disk quota support"
     UTMP=on "With UTMP accounting support"
     PAM_SMBPASS=off "With PAM authentication vs passdb backends"
     CLUSTER=off "With experimental cluster support"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     EXP_MODULES=off "With experimental modules"
     POPT=on "With system-wide POPT library"
     PCH=on "With precompiled headers optimization"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

The New Configuration

samba 3.3 config:

[root@server samba33]> make showconfig
===> The following configuration options are available for samba-3.3.3:
     LDAP=on "With LDAP support"
     ADS=off "With Active Directory support"
     CUPS=on "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=off "With File Alteration Monitor"
     SYSLOG=off "With Syslog support"
     QUOTAS=off "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=off "With experimental modules"
     SHARED_LIBS=off "With shared libraries"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

If you have never dealt with FreeBSD’s Ports system, setting these compile time options is a breeze with ‘make config’. These options are presented in a nurses interface (which is great for ssh or other terminal based sessions) like this:

FreeBSD's make config screen

The noticable options that I chose are AIO=yes and ADS=off. I’ve always compiled ADS support thinking that I would also get around to configuring Samba as an Active Directory-like server. But you know, there are really only two active users here, myself and Michele, and I don’t see the benefit right now. It could also slow down samba with the extra system calls so again, I’m leaving it out. AIO, Asyncronous IO, is new, and it is reported to increase size (oh wait, wrong advertisement) I mean, network IO.

smb.conf options

The performance related options in smb.conf are here:

        socket ōptions=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY IPTOS_LOWDELAY
        min receivefile size=16384
        aio read size = 16384
        aio write size = 16384
        aio write behind = true

Again, I got these from Ivan’s post, plus what I’ve used for the past 8 years.

Benchmarking with IOzone

I ran a simple

iozone -Ra -b samba3.0.28.xls

for both versions of samba. I’d create some nice 3d charts but I really don’t know excel well enough. So, I’ll just link these here and you can see for yourself what the substantial differences are. Some quick glances while the tests ran, I saw around 50MB/sec for Samba 3.0.28 here and there, topping out at ~60MB/sec. This was only with 256MB and above files. Smaller files always stayed around 5-10MB/sec.

Samba 3.3.3 - with AIO enabled started up fine, but iozone crashed after wrting a few small 64K bytes. This was a little disappointing, however, I did continue the benchmark with the new send and recieve sizes.

UPDATE:
After doing a little more reading, I found out I was supposed to load the aio kernel module. After running:

$ kldload aio

I restarted Samba with the AIO options enabled, re-ran iozone, and it all worked.

ufs2-3ware-raid5-freebsd71

samba3028

samba333

Tonight was Mark’s last class before he leaves for Idaho for a month. Last week, he joked that he wanted everyone to still be recovering when he gets back in June… He tried his best to make that happen, and I think Patti bailed us out in the end

We got some time to warm up for a few minutes, I think he wanted us ready by 7:20, so we jogged around for a bit and did some light stretching. Mark went around to assist us, which always helps out, but its a tad painful. Ahh good ol’ pain, its such a gift. Then I think Mark rubbed his hands together, muttered “Let us begin…” and cackled…

Tonight was all about basics, so our first drill was simple: keep your arms out straight, palms flat, and do a series or 20 front stretch kicks. The trick was, is we would do a round of kicks, using a bit of our momentum from the previous kick, to really get it up there - head to toe if you can. The alternating set was to NOT use the momentum from the previous kick. So you would kick, and stop immediately on the way down, wait, and do it again. Doing it this was is a lot more difficult, and its great because it really strengthens your leg muscles more, and you end up working your core more as well. Between those, and I think we did around 100 kicks, we still had to keep our arms up. I’m happy to say I never wussed out and took a break, but as of Saturday, days later, I can still feel it in my forearms and shoulders.

The next drill was normal front stretch kicks, and we were broken up in the three typical groups. Simple, but the trick this time was Mark wanted all of us to synchronize our kicks. Since we have a few people that always like to march to their own drum, this can take a while to get right So its always hilarious when people are getting frustrated after the 5th or 6th try and still hearing Mark say “Almost, but not quite, again.” Felicia is yelling at people, Eli is ready to quit, everyone is throwing in their two cents on who should follow who and who needs to slow down Its funny I tell ya.

This lasted an hour, and I think Mark wanted to work on more basic stance work but Patti suggested we all work on weapons. This was good, because my quads felt like they were pulled off, they were so exhausted that I think they detached themselves and walked off, giving me the finger on the way out.

Mark had us alternate between a group of long weapons (which I was in, staff), and then a group of short weapons (straight sword/broadsword). We did basic stuff like figure 8’s and reverse flowers for a full minute, switching out groups. After doing that for a while, Mark had Johnny and myself work on some nangun basics. Now, most of what I’ve learned has been from one seminar 4 or so years ago when the Beijing Wushu Team came out. I’ve practiced this one form from video clips over and over again, and when Mark worked with me on some of the basic moves, it was like learning it all over again. It really breathes new life into things when you get to correct the simple moves. I’ll have to make sure I get more from him in June.

After class, we went to Diamo’s to eat, and I stayed until midnight we had a lot of fun, and some good conversations.

Sunday we celebrated Easter, as well as Owens 1st birthday (its really April 22nd, but we took advantage of having most of the family in one place), at my parents place in Knightsen.

Here is a link to the gallery on my other site, m87-blackhole.org:
http://www.m87-blackhole.org/photos/2009-04-12-easter/

I also took some pictures of my parents house in the making. It is starting to shape up nicely, my step dad is a pretty talented carpenter.