Archive for October, 2006

Does it really have to be so hard?

Wednesday, October 18th, 2006

Fair warning, rant incoming. I’m working on a server development project, which will get deployed to CentOS 3 or possibly 4. Since I don’t currently have a spare machine that’s fast enough and able to blown away, I revived my linux dual boot on my main development workstation (2x opteron 242, tyan thunder k8w). I initially went with Fedora Core 5, since I figured I could easily enough rebuild using mock for centos4 and centos3. Except I’m getting my butt kicked by the ‘udev hangs’ bug. No combination of kernel versions, acpi=off, noapic, udevtimout, selinux=disabled, apm=off or bios changes seems to make the machine boot consistently, let alone assign my ethernet cards consistently.

Since I have work to get done, I went ahead and installed CentOS 4.4, which luckily did install and with the addition of the Nvidia binary drivers is running my dual screens beautifully and stably. Now all I need in order to work happily is music! Unfortunately, most of said music is in mp3 format, and redhat doesn’t supply an mp3 decoding library, due to licensing issues. I understand the nature of the situation and don’t fault redhat for this, but nonetheless its really annoying.

There is no livna repository for rhel/centos, and trying to build the repos for fc3 or fc5 tends to introduce all sorts of dependency problems. And besides, I’m not getting paid to rebuild mp3 players, I have real work to do. I just want to listen to mp3s! Luckily for me theres a guy out there by the name of Dag Wieers who maintains a very nice set of RPMS for fedora AND rhel distributions, and his xmms package compiles cleanly with dependencies I’ve already got, saving me from working in misery for another few days.

However, all of this points to a major problem in the Linux world. As much as we love to hate it, Microsoft absolutely understands binary application compatibility. For a given version of windows, application developers know exactly what library versions are guaranteed to be available, and how to include the rest of their dependencies in a safe way. Standard practice is to include the libraries that are needed inline with the application. Windows provides reasonably standard interfaces allowing the application register itself as a type handler and to add desktop shortcuts, quicklaunch icons or start menu entries. Every windows application developer writes one or perhaps two versions of application foo and makes up a nice little installer, and after a few iterations has an installer that works on windows, period. Sure it makes the average download kinda big, but honestly in the days of free after rebate spindles of CD-ROM’s, broadband for $15/mo, and 250gb hard drives for under $100, who cares? My time is worth a WHOLE lot more than a few extra megs used up on my drive.

Compare that to linux. Say I found out about this great new mp3 player. I go to their website, and see a source tarball. I try to build the source tarball and if I’m lucky the’ll have a competent ./configure script which will find out what dependencies I’m missing and tell me about them. At that point I’m looking at a significant period of time trying to find prebuilt versions of those build dependencies, or rebuilding my entire system from scratch. If I’m lucky, they’ll have an pre-built rpm package. However, 9/10 times that rpm is going to have a whole slew of unmet dependencies and I’ll have to spend several ours ferreting them out and downloading them. If I’m really unlucky, the application will depend on a whole slew of CPAN libraries, some of which will come with my OS, and some of which will require a perl version not included with my distro. I can understand needing to integrate directly with the OS when we’re talking about truly system level stuff, but 99% of the time thats not what I’m talking about. An mp3 player does NOT need to know what kernel version you have, nor does it generally need to know much of anything about your system. It doesn’t need to know what version of python, perl, or php is included with your distribution, nor what extensions those libraries include.

Linux distributions absolutely must get away from the ‘distribution contains every application’ mentality. Right now, if an application doesn’t come integrated with an additional application, or have a version packaged specifically for that distribution, life sucks for the user. Life also sucks for the application developer, who would like to see his software able to be used by everybody, but can’t, because he uses debian and doesn’t know how to package for redhat.

The answer to this, of course, is the Linux Standards Base (LSB). It describes a set of ABI’s that the OS should guarantee, and defines how an application developer can package up the rest of their dependencies so their application can run on any system providing LSB support. Every major linux distro out there supports LSB applications, except I don’t see a single open source application providing LSB binary downloads!

Why? I’m not positive to be honest, but I’m willing to take a guess. It’s too dang hard to build for. I’ve not done a lot of research on it yet, but I just haven’t seen an easy to use buildroot for LSB on redhat. Nor have I seen an easy to follow howto for building LSB applications. In addition, it’s probably just sorta inconvienient for developers, who pick a disitribution they like and just run with it, following the dependencies.

In our zealotry for open source, we continue to paint ourselves into a binary incompatibility corner that is simply unsustainable. The OS (distribution) absolutely needs to be able to evolve, or not evolve, as it needs. Applications need to be able to evolve (or not evolve) independantly. If I want to listen to be able to mp3’s, I should be able to choose from a variety of mp3 players (the code is out there!) and use them easily. That means we need real binary application compatibility, just like consumer OS’s. If I want to run a bleeding edge browser on my otherwise stable machine, thats my prerogative, and it shouldn’t require me to invest a day resolving all the requisite dependencies and building the thing from scratch. Regardless of what Gentoo folk will try to convince you, rebuilding everything from scratch and customized is not a sustainable way to run an operating system. It takes too long, and is too complicated to ever allow mainsteam users to do it, or even people who just have better things to do!

Burning DVD ISO images on Linux

Friday, October 13th, 2006

So I had a great idea… Why don’t I buy a DVD burner and then start using DVD’s to install software, make backups, etc, etc. Well… it was a nice idea, but then I had another great idea… Why don’t I run windows XP 64 since I have an AMD Opteron system! Turns out none of the wonderful bundled DVD burning software will even install on XP64, let alone run, so there it sat.

I’d always used the win32 build cdrecord to burn iso images under windows anyway, since the [ISO Recorder Powertoy](http://isorecorder.alexfeinman.com/isorecorder.htm) doesn’t work if windows doesn’t have a driver for your CD burner, and I’d been stuck in that situation a few times. I figured cdrecord would work the same for burning DVD’s. Well, aside from the possible licensing issues, at least with my drive, it doesn’t. For reference, my drive is this:

hdd: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive

I saw a bunch of documents out on the web saying that cdrecord was able to burn DVD’s, just like a cd image using `cdrecord -dao `. I tried that. Several times. Every DVD was a coaster. Last time I had linux installed, I tried it under linux too, but since it was just a passing fancy I didn’t dig into it.

Well yesterday I committed to running linux on my desktop for a while to get a project finished since I don’t have any other linux development suitable machines available currently. I needed a DVD burned, and when I tried to have Dave burn one for me on his windows machine, it didn’t work either! Something was up! I tried it using the `cdrecord`, and made another coaster. But this time I noticed something:

Device type    : Removable CD-ROM
Version        : 0
Response Format: 2
Capabilities   :
Vendor_info    : '_NEC    '
Identifikation : 'DVD_RW ND-3500AG'
Revision       : '2.18'
Device seems to be: Generic mmc2 DVD-R/DVD-RW.
cdrecord: Found DVD media: using cdr_mdvd.
Using Session At Once (SAO) for DVD mode.
Using Session At Once (SAO) for DVD mode.
Using Session At Once (SAO) for DVD mode.
Using generic SCSI-3/mmc DVD-R(W) driver (mmc_mdvd).
Driver flags   : SWABAUDIO BURNFREE
Supported modes: PACKET SAO
scsi_set_streaming
Speed set to 8467 KB/s
Starting to write CD/DVD at speed   6.0 in dummy SAO mode for single session.

Aha! Theres two types of DVDR media, and I have +R. This is a problem. Well, the cdrecord manpage claims autodetection support for plusr devices and media, but when I tried to force a +R driver…

erik@bambi ~]$ cdrecord  -dao driver=mmc_dvdplus dev=/dev/hdd /tmp/sol-nv-b49-x86-dvd.iso
Illegal driver type 'mmc_dvdplus'.

So therein lies the problem. Well, the good news is that I found the solution. A little bit of googling taught me about a great new program, growisofs (http://fy.chalmers.se/~appro/linux/DVD+RW/). And it’s even included in FC5, in the dvd+rw-tools package! Well, don’t ask me why I didn’t know this before, but growisofs works like a charm.

[erik@bambi ~]$ growisofs --Z /dev/hdd=/tmp/sol-nv-b49-x86-dvd.iso
Executing 'builtin_dd if=/tmp/sol-nv-b49-x86-dvd.iso of=/dev/hdd obs=32k seek=0'

To boot, it even displays progress information AND my disc was readable finally!

Fedora/CentOS with a Microsoft Laser Mouse 6000

Friday, October 13th, 2006

So I’m working on some linux software and am running linux on my desktop. One of my major pet peeves every time I go back to a linux desktop is that out of the box my button 4 and 5 don’t work correctly under firefox. Here’s the fix.

Under Fedora Core 5, use this:


cat > /etc/X11/xinit/xinitrc.d/mouse.sh <<-EOF
#!/bin/sh
# /etc/X11/xinit/xinitrc.d/mouse.sh
# Required for the configuration of a 5-button mouse
xmodmap -e "pointer = 1 2 3 8 9 4 5 6 7 10 11"
EOF
chmod a+x /etc/X11/xinit/xinitrc.d/mouse.sh

Under CentOS 4 / RHEL4, use this:


cat > /etc/X11/xinit/xinitrc.d/mouse.sh <<-EOF "
#!/bin/sh
# /etc/X11/xinit/xinitrc.d/mouse.sh
# Required for the configuration of a 5-button mouse
xmodmap -e "pointer = 1 2 3 6 7 4 5 "
EOF
chmod a+x /etc/X11/xinit/xinitrc.d/mouse.sh

And then use the following for the mouse inputdevice:


Section "InputDevice"
        Identifier  "Mouse0"
        Driver      "mouse"
        Option      "Protocol" "ExplorerPS/2"
        Option      "Device" "/dev/input/mice"
        Option      "ZAxisMapping" "6 7"
        Option      "Buttons"   "7"
        Option      "Emulate3Buttons" "no"
EndSection


Solaris BrandZ Zones

Thursday, October 12th, 2006

Sun has really been pushing innovation with Solaris recently, and since it’s now freely available and open sourced, what better time to give it a test drive? With the Solaris Express Nevada 49 release (get it from http://www.opensolaris.org/os/downloads/on/ and click the CD Version or DVD Version links under step 3b) Sun has officially included their BrandZ extension to Solaris containers. BrandZ allows containers to be ‘Branded’, the upshot of it is that you can run an entire system under a lxrun-like technology.

Currently they fully support running CentOS 3 in a zone, which conveniently enough is what all our production services still run on. See http://www.opensolaris.org/os/community/brandz/install/ for the official howto guide. Here’s how to I set up a CentOS 3 development system for testing on my SNV49 machine:


#zonecfg -z centos3-dev-2

centos3-dev-2: No such zone configured
Use 'create' to begin configuring a new zone.

zonecfg:centos3-dev-2> create -t SUNWlx
zonecfg:centos3-dev-2> set zonepath=/tank/centos3-dev-2
zonecfg:centos3-dev-2> add net
zonecfg:centos3-dev-2:net> set address=192.168.2.31/24
zonecfg:centos3-dev-2:net> set physical=e1000g0
zonecfg:centos3-dev-2:net> end
zonecfg:centos3-dev-2> commit
zonecfg:centos3-dev-2> exit



#zoneadm -z centos3-dev-2 install -d /tank/public/centos_fs_image.tar.bz2


cannot create ZFS dataset tank/centos3-dev-2: dataset already exists
Installing zone 'centos3-dev-2' at root directory '/tank/centos3-dev-2'
from archive '/tank/public/centos_fs_image.tar.bz2'

This process may take several minutes.

Security Appliance Roundup Part 2

Thursday, October 12th, 2006

Smoothwall came through with a demo license for me in just a matter of minutes, and I spend a couple hours playing with it. It has a fairly complete web interface, but unfortunately even with all its fancy features I saw absolutely nothing that would allow me to operate it as a layer 2 firewall (bridging my static ip addresses into a dmz) nor does it have support for routing said static ip address without NAT. Given we have clients and servers on static IP addresses and a class C address block to boot, it seems a waste to have to static nat them all and deal with that complexity when a dual dmz solution with layer 2 support would take care of it. So it’s back to the drawing board.

Given none of the software firewall packages seem to support 4 or more interfaces well, nor do many support layer 2 firewalling, I’m looking at hardware solutions. Going with a hardware appliance type solution seems to open up the options significantly with respect to high availability, as well, which actually seems like a really intelligent thing to do since I really can’t afford for the system to be down as long as it would take to get a replacement.

Currently, my top pick seems to be the
[netscreen 25](http://www.juniper.net/products/integrated/ns_2550.html). It offers the features I need, and at a retail price around $2500 it seems like a solid deal.

Other units I’m looking at are the [WatchGuard Firebox x750e](http://www.watchguard.com/products/core-e.asp) and the
[sonicwall 2040 or 3060](http://www.sonicwall.com/products/vpnapp.html)

Disk Benchmarks, Round 1

Tuesday, October 10th, 2006

In the process of trying to figure out my VM performance problems, I’ve been doing a lot of filesystem benchmarking. Unfortunately, there isn’t a lot of consistency between platforms or machine classes as to benchmarking methodology, so I’ve had some trouble generating comparable numbers. However, I’ve gotten the solaris filebench suite running on linux, and bonnie++ running on solaris, so I can now generate comparable numbers across both platforms.

One of my primary interests is the throughput I can get out of the 3ware 7506 raid controller in my unix nas box, both in order to optimize it and in order to compare to other solutions and determine if they will actually be an upgrade or not. In the process, I’ve been benchmarking an older Dell Precision Workstation 420. It has 4 wd1200jb drives plugged into its onboard IDE boards (yes, they are sharing ide channels), and is currently running opensolaris nv47. I played around with a few different ZFS configurations, but eventually settled on raidz, leaving me with 360G usable disk space. Here’s some of the numbers I got:

Filebench Benchmarks

Webserver IO Summary: 848559 ops 16865.4 ops/s, (5440/544 r/w)  91.8mb/s,    249us cpu/op,   0.3ms latency
Varmail IO Summary: 96110 ops 1913.1 ops/s, (294/295 r/w)   9.6mb/s,    618us cpu/op,  26.1ms latency
Fileserver IO Summary: 8000 ops 3964.1 ops/s, (497/493 r/w)  72.8mb/s,   1130us cpu/op,   0.9ms latency

Here are some benchmarks from my production file server, at a low load time but nonetheless serving vmdk images for 8 virtual machines. It is a 2x P4 Xeon 2.4ghz ,3GB RAM, 3ware 7506 controller with raid5 x 5 + 1 hotswap WD1200JB PATA disks.

Varmail IO Summary: 87754 ops 1454.5 ops/s, (224/224 r/w)   7.3mb/s,    142us cpu/op,  35.5ms latency
Webserver  IO Summary: 20312 ops 20208.2 ops/s, (6499/660 r/w) 108.4mb/s,    215us cpu/op,   0.7ms latency
Fileserver IO Summary: 7997 ops 1587.9 ops/s, (202/196 r/w)  29.8mb/s,   1122us cpu/op,  37.8ms latency

Security Appliance Roundup Part 1

Tuesday, October 10th, 2006

So I want to simplify and strengthen our network security at the office. Currently we’re using a mishmash of cisco IOS ipsec, pptp, cbac/NAT and linux iptables host based firewalls. I’d like to centralize everything, and add a more reliable vpdn solution to the mix. In addition, I’d like better logging and traffic shaping control.

### [Cisco 2600 with IOS 12 advanced firewall](http://www.cisco.com) ###

I’ve been running this for the last couple of years, on 2600 class hardware. In general, its a very complete solution, but configuring it can be extremely trying. I can’t even remember the number of times I’ve been stumped by some issue that ’should be working’, only to find out a month or two later that the problem was software revision I was using. Cisco is very difficult to work with wrt getting firmware updates for their software, and I typically don’t have good luck findout out what other version I’d need, anyway. The cisco ipsec vpn implementation seems to be solid, as are their routing abilities. The CBAC packet inspection system is where the cisco starts to weaken as a firewall platform, however. Even just inspecting standard TCP traffic can easily put the 2600 under enough load that I can’t really afford to run it between my internal network and DMZ.

Cisco’s single file configuration, plethora of hardware interfaces, utterly complete routing support and general hardware reliability are second to none however. If it were easier to get a bunch of ethernet interfaces tied into a 2600, and it was in general easier to get ahold of firmware updates and modules for the hardware I’d be hardpressed to want to change.

We’ve also been using Cisco hardware for our vpn endpoints. At this point they consist of a soho91 and a pair of 857w’s. In general, these are very capable endpoint routers, and obviously their ipsec vpn connectivity to the 2600 is second to none. However, they lack support for any sort of split horizon dns, so using dns at client sides is trying at best. In addition, I’ve not gotten so far as to even attempt to control access to the vpn at the remote ends, so I’m stuck trusting that anybody managing to connect to one of the remote endpoint networks (wired, wireless, or otherwise) is trustworthy. I believe I ought to be able to apply CBAC or packet filter rules to traffic arriving from the VPN but the complexity of figuring out how to apply them in the face of cisco’s ip stack traversal order has prevented it.

In addition, the ios nat implementation doesn’t seem to be able to do any sort of hairpinning, which is pretty frustrating since our mail server is snatted through the router. This means I can’t send smtp traffic from any of the vpn endpoints currently.

### [M0n0Wall](http://m0n0.ch) ###
This is another very nicely integrated freebsd based web managed firewall implementation. It is designed to run on embedded hardware such as Soekris or WRAP single board computers. It supports IPSEC tunnels, split horizon dns using dnsmasq, but again is targetted fairly directly at the SOHO market, with no support for vpdn or static routing. I used it for several months running on my soekris 3ethernet box and had no problems with the functionality it provided.

### [LEAF](http://leaf.sourceforge.net) ###
This is an interesting amalgam of linux software packages for embedded hardware. Unfortunately, the price to be paid for the flexibility of a full linux system is complexity. You’ll have to edit the files in /etc folder on a ramdisk, and figure out which package they are a part of in order to get them ‘backed up’ to compact flash so they will be available post reboot. The firewall portion of LEAF is typically handled with shorewall.

### [Shoreline Firewall](http://www.shorewall.net/) ###

This is basically an iptables configuration script. It uses a series of files (typically placed in /etc/shorewall) defining things like interfaces, rules, and policies, and turns it all into a complete set of iptables rules. It seems to be able to configure just about any aspect of iptables you might want to, but the downside is that it generates copious amounts of rules, which makes reading the iptables -L output the old fashioned way pretty tough. The other downside is obviously that you have to edit the files, and do it correctly, or you risk blowing up all kinds of stuff.

### [Proxmox Firewall](http://www.proxmox.com) ###
This appears to be another shorewall based firewall solution. I did manage to download and install it, but in its install process it appears to have assigned itself an IP address using dhcp and then started using it statically, and I have no idea what it is or how to get it back. Oh well.

### [Gibraltar](http://www.gibraltar.at/) ###
Based on debian . Looks to be utterly feature complete (including multiple dmvpn options, heartbeat failover support, lots of proxies, etc), and it still knows how to back itself up / restore easily. Runs off read only media, and according to their site they have some VERY nice hardware solutions for a reasonable price. What I don’t like is that the UI is anything but user friendly. It’s basically just a series of web forms to cover up the standard debian configuration files. When I tried to save the configuration to my vmware hard drive, it wouldn’t work, either, so at this point I’m gonna have to backburner this option.

### [Smoothwall (GPL)](http://www.smoothwall.org) ###
This is very nicely integrated firewall package. It has explicit dsl modem support, and a fairly strong community and selection of add-on packages. However it is clearly aimed at the SOHO market. It doesn’t support multiple DMZ’s, nor any static routing. The web interface appears to be very nice, but you really have to be sure your needs will fit directly within their model.

### [IPCop](http://www.ipcop.org) ###
This is the ‘more open source’ fork of smoothwall. It adds support for the snort IDS, complete with automatic rule updates, a ‘blue’ dmz interface (for wireless networks, I think), and in general is another very nice package. But again it only supports ipsec vpn (although it will handle dynamic vpn terminals using dyndns). It also doesn’t support any static routing that I can find.

### [Smoothwall Advanced Firewall](http://www.smoothwall.net) ###
This is the commercial version of smoothwall. They’ve been in the business for a long time, and feature wise this looks like an extremely compelling option. They provided me with a free evaluation license, so I downloaded it and spent a few hours checking it out while running in a VPC. In general it has a quite nice web configuration interface, but again it really isn’t designed for a more complicated network. In particular, it doesn’t appear to have any mechanism to support static routing or layer 2 filtering, making it difficult for me to use my class C block of IP addresses effectively.

First Post!

Sunday, October 1st, 2006

So, as it happens, I’ve been doing a lot of ‘research’ lately. Such that it is, a lot of my ‘research’ ends up being on the web, and a lot of information comes from blogs, so I decided I’d best get with the program and start posting what I’ve been up to on the web as well. Not that I expect it to be interesting to anybody else, but at least I’ll start to accumulate the stuff I find somewhere more useful than a browser cache!

Right now, the primary areas of my ‘research’ are storage and virtualization. The last time I went around this merry-go-round it was VOIP, so perhaps I’ll get around to trying to consolidate some of what I’ve picked up on that front as well. We’ve recently been nailed with some hardware failures at Interlink and I’m really hoping that the latest round of storage and virtualization hype will help me to insulate us from service outages and spending the amount of time I have attempting to resurrect systems with a soldering iron and box of new fans. Interlink is a pretty small company, but we’re more and more dependant on the Internet. This puts us in an interesting position, in that we don’t really have the bankroll to play with ‘Enterprise’ hardware, but our dependance on the internet (and computing services in general) is high enough that we really can’t afford to take the usual small business ‘buy a dell and stick it in the corner and hope it doesn’t die’ route.

As it happened, VMWare had just released the final version of the their newly free ‘VMWare Server’ product when one of my machines died in the rack, and I’d been evaluating it for a while, so I got a crash course in ‘putting VMWare into production’. Unfortunately, in my case, just about every issue I’ve had with VMWare seems to be IO related, so the interest I’d had in moving beyond our Dell Windows Storage Server NAS to a real storage system was piqued.