1. Right-click an open panel area
2. Select “properties”
3. Right-click on a notebook tab 3 times

The Linux Plumbers Conference was created to bring together the key developers involved in Linux plumbing - the “Linux plumbers” - and give them an opportunity to discuss problems face-to-face, both within subsystems and across subsystems. Participants include invited attendees, speakers selected through an open, competitive review process, and students. Registration is open to the general public as well.
…
The goal of the Plumbers Conference is to solve problems. The conference is arranged as a series of microconferences, each on a topic that is narrow enough to identify specific problem areas and brainstorm workable solutions. Each microconference is led by an expert in the field and organized to encourage discussion and problem solving.

You are in a mall when zombies attack. You have:

1. One weapon
2. One song blasting on the speakers
3. One famous person to fight along side you.

I can’t resist.

1. BFG9000: ranged weapon that vaporizes multiple zombies at once. I should be out of the mall before I’m out of ammo.
2. “Good Vibrations” by the Beach Boys: up beat and a little silly.
3. Jet Li: he could totally handle the zombies within slicing/kicking/clubbing range.

Looking back through my build logs, I can see that my desktop spent 55 hours, 14 minutes, and 3 seconds on 406 builds related to edgy-security updates I was involved in publishing. These times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the porting machines, etc. Comparing to my prior post on this topic, here are the standings for other releases:

dapper: 44:48:24
feisty: 58:49:04
gutsy: 37:06:08
hardy: 86:25:58

Hmm… I think my hardy numbers include devel builds times… I’ll have to sort that out. :)

Thank you Edgy! I will remember you for your wonderful default -fstack-protector.

To watch a movie:

• Plug in projector
• Open System/Preferences/Screen Resolution
• Pick nice big resolution for the attached projector
• Run “xvattr -a XV_CRTC -v 1″ (”-v 0″ will push Xv back to the LCD)
• Eat popcorn

As a result, if AppArmor isn’t the MAC system you want, you can now install a functional SELinux system on Ubuntu with just a simple “sudo apt-get install selinux”.

  fd_set rfds;
  int rc;

  FD_ZERO(&rfds);
  FD_SET(fd, &rfds);

  tv.tv_sec = 1;
  tv.tv_usec = 0;

  rc = select(fd + 1, &rfds, NULL, NULL, &tv);
  if (rc > 0) {
    char buf[80];
    ssize_t got = read(fd, buf, sizeof(buf));
    if (got < 0) {
      perror("read");
      return 1;
    }
    else if (got == 0) {
      printf("EOF\n");
      return 1;
    }
    else {
      printf("read bytes: %d\n", got);
    }
  }

When dealing with sockets, the above loop is sane — EOF means the other end hung up and you’ll never get more data from the file descriptor. In the case of a FIFO, however, “0 length read” means there are no more FIFO writers — but more could attach later and continue feeding in data! The problem with this is that select misbehaves and marks the file descriptor as “EOF” forever. Only the initial select() call blocks until there is something to read — once it hits EOF, select will always immediately return, defeating the purpose of select().

One solution is to re-open the FIFO, but you might miss writers between your 0-length read() and the next open().

The seemingly correct solution is rather simple: the FIFO reader should open the FIFO as a writer also. In this case, select() never thinks all the writers have vanished, so there is never an EOF condition, and the reader never misses any writers. So, instead of O_RDONLY, use:

fd = open(FIFO_PATHNAME, O_RDWR | O_NONBLOCK);

$ cat <<EOM | as - -o /tmp/fake.ko
> .section .modinfo
> .string "license=GPL"
> EOM

$ modinfo /tmp/fake.ko
filename:       /tmp/fake.ko
license:        GPL

1. Create /etc/initramfs-tools/conf.d/cryptroot for your partition, based on the logic and defaults in /usr/share/initramfs-tools/scripts/local-top/cryptroot.
2. Convert the existing encrypted swap to the new configuration.
3. Update initrd, reboot, enjoy.

Assuming your swap partition (in encrypted form) is stored at /dev/laptopvg/swaprawlv, and you want your accessible swap partition as /dev/mapper/swap, here are the above steps in detail:

Doing step 1 is simple, we’re assuming the defaults from the cryptroot script above:

echo source=/dev/laptopvg/swaprawlv target=swap > /etc/initramfs-tools/conf.d/cryptroot

Step 2 hurt my head. Make sure you’ve unmounted your swap before attempting this, or you can destroy the partition contents. The parameters come from the cryptroot script again:

swapoff /dev/mapper/swap
vol_id /dev/mapper/swap
cryptsetup -c aes-essiv:sha256 -h sha256 -s 256 create swap2 /dev/laptopvg/swaprawlv
dd if=/dev/mapper/swap of=/dev/mapper/swap2 bs=4k
cryptsetup remove swap
vol_id /dev/mapper/swap2

Step 3 is simple again:

update-initramfs -u
shutdown -r now

Ta-da!

In disk management, a lot of time was spent discussing LVM, which I’m very fond of myself. (Even LVM snapshots were covered!) I have a hard time imagining running any computer without LVM, so it was great to see it get a solid chunk of attention. The only thing I felt was missing from disk management was a discussion of RAID (md). For server environments, I think this is a critical topic. Providing redundancy against drive failure is, I think, even more important than demonstrating how to easily manage partition layouts with LVM.

In filesystem management, basic ACLs were covered as well as quota management. I think quota management is an often neglected part of administration, so I was glad to see this covered. In network management, basic iptables were outlined with good examples. (Hardy’s “ufw” will help make this section even simpler in future revisions of the book.) IPv6 was touched on, though I would have liked to see slightly more details.

Under service management I enjoyed the introduction to PKI, which is critical to understanding the basics of SSH and other services using SSL. The examples for DNS, DHCP, NFS, and Samba were all very well done. I think they make handy references for how to get a network or file-sharing server up and running in short order.

As another Hardy feature to call out, the addition of “virt-manager” will make the Virtualization section on KVM much nicer to deal with.

I took some notes for ideas and corrections that may be a benefit to other readers of this book:

• I personally like suffixing VG and LV names with “vg” and “lv” just to be able to quickly distinguish them when looking at device names.
• Administrators watching long-running “tail -f” output would benefit from using “tail -F” for when log files are rotated.
• In the section on “Finding Files” I was expecting to see mention of “locate”.
• When viewing compressed files: “zless” instead of “zcat FILE.gz | less”.
• When discussing Job Control, I would have liked to see a mention of “screen” for managing long-running processes (kernel compiles, “top”, etc). Not enough people know about “screen”. :)
• While the book was written to Feisty, it would be nice to have a short section in future versions on how to generate and use AppArmor profiles for the various running network services.
• Instead of the manual symlink management for Apache modules and sites, administrators can use the “a2{dis,en}{site,mod}” tools.
• Typos I saw: tailing “sudo” in mysql db creation example, “_netdec” should be “_netdev” in NFS fstab example.

As I mentioned at the start — I think this is a great book for someone either new to Ubuntu server management or looking for simple service configuration references in a single place. Thanks again to Apress for sending me a copy; I tried not to be too biased. :)

bash -c "MOZ_NO_REMOTE=1 firefox -ProfileManager"

And in a recent bug-hunting session, I had a firefox profile that just kind of didn’t load javascript correctly any more (”change_feedback_state is not defined” on facebook). I have no idea what was causing the issue (something not extensions — it didn’t go away in “-safe-mode“), and so I just reconstructed the profile one bit at a time, eventually leaving all of prefs.js out. I used the migration checklist I found at mozillazine.

For binaries that have been compiled with -pie (Position Independent Executable), the kernel is finally able to take advantage of it. As an example, openssh is already using this compile option, and the results are easy to see. Here are the processes from two SSH connections:

$ pstree -lp | grep sshd
        |-sshd(7243)-+-sshd(9136)—sshd(9140)—bash(9142)-+-grep(15380)
        |            +-sshd(9181)—sshd(9185)—bash(9186)

If we examine the memory layout of both sshd processes (9136 and 9181), we can see no user-space memory locations are shared:

$ sudo cat /proc/9136/maps
7ff69df86000-7ff69e0c6000 rw-s 00000000 00:09 34320                      /dev/zero (deleted)
7ff69e0c6000-7ff69e0c9000 r-xp 00000000 fe:15 480495                     /lib/security/pam_limits.so
...
7ff6a1fc8000-7ff6a1fd0000 rw-p 7ff6a1fc8000 00:00 0
7ff6a1ff7000-7ff6a1ffa000 rw-p 7ff6a1ff7000 00:00 0
7ff6a1ffa000-7ff6a1ffc000 rw-p 0001d000 fe:15 1040531                    /lib/ld-2.7.so
7ff6a1ffc000-7ff6a205b000 r-xp 00000000 fe:15 98598                      /usr/sbin/sshd
7ff6a225a000-7ff6a225d000 rw-p 0005e000 fe:15 98598                      /usr/sbin/sshd
7ff6a225d000-7ff6a2289000 rw-p 7ff6a225d000 00:00 0                      [heap]
7fffaa045000-7fffaa05a000 rw-p 7ffffffea000 00:00 0                      [stack]
7fffaa1fe000-7fffaa200000 r-xp 7fffaa1fe000 00:00 0                      [vdso]
...

$ sudo cat /proc/9181/maps
7f05a07b8000-7f05a08f8000 rw-s 00000000 00:09 35989                      /dev/zero (deleted)
7f05a08f8000-7f05a08fb000 r-xp 00000000 fe:15 480495                     /lib/security/pam_limits.so
...
7f05a47fa000-7f05a4802000 rw-p 7f05a47fa000 00:00 0
7f05a4829000-7f05a482c000 rw-p 7f05a4829000 00:00 0
7f05a482c000-7f05a482e000 rw-p 0001d000 fe:15 1040531                    /lib/ld-2.7.so
7f05a482e000-7f05a488d000 r-xp 00000000 fe:15 98598                      /usr/sbin/sshd
7f05a4a8c000-7f05a4a8f000 rw-p 0005e000 fe:15 98598                      /usr/sbin/sshd
7f05a4a8f000-7f05a4abb000 rw-p 7f05a4a8f000 00:00 0                      [heap]
7fffac877000-7fffac88c000 rw-p 7ffffffea000 00:00 0                      [stack]
7fffac9fe000-7fffaca00000 r-xp 7fffac9fe000 00:00 0                      [vdso]
...

The larger the memory space, the more effective ASLR is, so 64bit is the way to go. And, as always, using 64bit kernels automatically gives you the NX bit protections too. Running a 64bit Hardy system is going to rock. :)

Grab the three updated tarballs from the “vmware-any-any” tar.gz here. Currently update115 works for me just fine.

/usr/sbin/sendmail -v -o MinQueueAge=0 -qI${ID_GOES_HERE}

<Printer lj4200>
...
JobSheets shared-banner none
...
</Printer>

Then, I could drop a modified banner into /usr/share/cups/banners with the filename “shared-banner”. The banner is just a regular PostScript file, so I could muck around with it. While looking at the “standard” banner, I saw some PS variables being used that had been defined by CUPS:

...
  (Job ID: ) RIGHT
  2 copy                % Copy X & Y
  moveto
  ({printer-name}-{job-id}) show
…

I couldn’t find documentation on the available variables, but managed to track down some of the list at cupsGetJobs2 in cups/utils.c:

job-id
job-priority
job-k-octets
job-state
time-at-completed
time-at-creation
time-at-processing
job-printer-uri
document-format
job-name
job-originating-user-name

None of these had the sending host listed, so I continued searching. Additional ones are defined in scheduler/ipp.c, including:

printer-name
job-id
job-billing
job-hold-until
job-sheets
job-media-sheets-completed
job-originating-host-name

“job-originating-host-name” did the trick for my banner:

...
  (Host: ) RIGHT
  moveto
  ({job-originating-host-name}) show
...

I’ve seen some other partial lists, but I haven’t found an official complete list. It’d be handy to see this documented better, since some variables aren’t valid until after the job is processed (job-sheets), so it’s only valid in the trailing banner, not the leading banner.

*-bank:0
description: DIMM DDR Synchronous 333 MHz (3.0 ns)
…
size: 1GB
width: 64 bits
clock: 333MHz (3.0ns)

On reboot, I also noted that my BIOS said I only had 3G. I started to get worried, but managed to find a setting on my Northbridge for Memory to enable “Hardware memory hole”. After that, both the BIOS and Linux were happy and seeing the full 4G. I assume the BIOS just bumps the memory in the 3G region to above 4G, which makes for a silly kernel message:

[ 24.617275] Memory: 3977852k/5259264k available (2281k kernel code, 150272k reserved, 1182k data, 300k init)

But I don’t care. :) It works now, and my “free” output makes me happy again:

             total       used       free     shared    buffers     cached
Mem:       3986156    3958396      27760          0      68268    2949472
-/+ buffers/cache:     940656    3045500
Swap:      3903672      38676    3864996

sudo apt-get –reinstall install $(grep ^2007-09-24 /var/log/dpkg.log | cut -d\ -f4)

I’m fairly used to seeing things like this from my gdb sessions:

Program received signal SIGSEGV, Segmentation fault.
0×41414141 in ?? ()
(gdb)

It means I’ve managed to gain control of the instruction pointer, and I’m now to the stage of needing to locate and deliver a shellcode.

Over the weekend I had the pleasure of causing my kernel to do something similar, via an unprivileged userspace process, using the vulnerability discovered by Wojciech Purczynski:

[119647.578349] general protection fault: 0000 [3] SMP
[119647.578357] CPU 0
…
[119647.578759] Code: Bad RIP value.
[119647.578774] RIP [<4141414141414141>]

I hadn’t had an opportunity to play with kernel shellcode before, so I ended up learning a lot from Brad Spengler. Before the day was up, I was left staring at a root shell.

This was a nasty bug. Luckily, it’s “only” a local exploit, and only for x86_64 kernels. But that’s still a very large number of installations. Please make sure your x86_64 machines are patched against CVE-2007-4573 (for Ubuntu, this is USN-518-1).

Here is the function preamble:

0x08081940 <wpa_driver_wext_get_scan_results+0>:        push   %ebp
0x08081941 <wpa_driver_wext_get_scan_results+1>:        mov    %esp,%ebp
0x08081943 <wpa_driver_wext_get_scan_results+3>:        push   %edi
0x08081944 <wpa_driver_wext_get_scan_results+4>:        push   %esi
0x08081945 <wpa_driver_wext_get_scan_results+5>:        push   %ebx

Save registers, prepare %ebp.

0x08081946 <wpa_driver_wext_get_scan_results+6>:        mov    $0x1000,%ebx
0x08081951 <wpa_driver_wext_get_scan_results+17>:       mov    0x8(%ebp),%eax
0x08081954 <wpa_driver_wext_get_scan_results+20>:       mov    0xc(%ebp),%edx
0x08081957 <wpa_driver_wext_get_scan_results+23>:       lea    0xffffffb0(%ebp),%esi

Make room for local variables, copy some function arguments and local variables into registers.

0x0808195a <wpa_driver_wext_get_scan_results+26>:       mov    %gs:0x14,%ecx
0x08081961 <wpa_driver_wext_get_scan_results+33>:       mov    %ecx,0xffffffec(%ebp)
0×08081964 <wpa_driver_wext_get_scan_results+36>:       xor    %ecx,%ecx

Here’s the stack canary getting set, and the register cleared. It’s saved at %ebp minus 0×14 (0xffffffec signed is -0×14):

(gdb) printf "0x%xn", 0-0xffffffec
0x14

Now for the function play-out:

0x08081a37 <wpa_driver_wext_get_scan_results+247>:      mov    0xffffffec(%ebp),%edx
0×08081a3a <wpa_driver_wext_get_scan_results+250>:      xor    %gs:0×14,%edx
0×08081a41 <wpa_driver_wext_get_scan_results+257>:      jne    0×8081eae <wpa_driver_wext_get_scan_results+1390>

There is the canary check.

0x08081a47 <wpa_driver_wext_get_scan_results+263>:      add    $0xec,%esp
0x08081a4d <wpa_driver_wext_get_scan_results+269>:      pop    %ebx
0x08081a4e <wpa_driver_wext_get_scan_results+270>:      pop    %esi
0x08081a4f <wpa_driver_wext_get_scan_results+271>:      pop    %edi
0x08081a50 <wpa_driver_wext_get_scan_results+272>:      pop    %ebp
0x08081a51 <wpa_driver_wext_get_scan_results+273>:      ret
...
0x08081eae <wpa_driver_wext_get_scan_results+1390>:     call   0x804bdc8 <__stack_chk_fail@plt>

Release local stack, pop saved registers and return. Nearer the end is the call to __stack_chk_fail when the canary doesn’t match.

So, to watch the canary, we need to set up a memory watch after it as been set, and tear it down before we leave the function. Respectively, we can use addresses 0×08081964 and 0×08081a3a (in bold above):

(gdb) br *0x08081964
Breakpoint 1 at 0x8081964
(gdb) br *0x08081a3a
Breakpoint 2 at 0x8081a3a

At the first breakpoint, we set a memory watch using a gdb-local variable, based on %ebp (we can’t use %ebp directly since it will change in lower function calls):

(gdb) commands 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>silent
>set variable $cow = (unsigned long*)($ebp - 0x14)
>watch *$cow
>cont
>end

Since I couldn’t find an easy way to track the memory watch number that was created during the first breakpoint, I just built a gdb counter, and deleted the memory watch when leaving, since I could predict gdb’s numbering (first watch will be “3″, following our breakpoints 1 and 2):

(gdb) set variable $count = 3
(gdb) commands 2
Type commands for when breakpoint 2 is hit, one per line.
End with a line saying just "end".
>silent
>delete $count
>set variable $count = $count + 1
>cont
>end

Now we can run, and wait for the canary to get overwritten:

(gdb) cont
Continuing.
Hardware watchpoint 3: *$cow
Hardware watchpoint 4: *$cow
...
Hardware watchpoint 12: *$cow
Hardware watchpoint 13: *$cow
Hardware watchpoint 13: *$cow

Old value = 4278845440
New value = 4278845546
0x0804eae6 in ?? ()

We see the canary value is 0xFF0A0000 getting it’s little-endian first byte overwritten to FF0A006A. We catch it before it has wrecked the stack, and we can see very clearly where we are:

(gdb) bt
#0  hexstr2bin (hex=0x080a239d "6151663870517a74", buf=0x080a2395 "aQf8pQzt00000000j", len=8)
    at ../src/utils/common.c:88
#1  0x08082297 in wpa_driver_wext_get_scan_results (priv=0xb7dd816c,
    results=0x080a239d, max_size=0x79)
    at ../src/drivers/driver_wext.c:1383
...
(gdb) x/1i $eip
0x804eae6 <hexstr2bin +54>:      addl   $0x1,0xfffffff0(%ebp)

On a closer look at the source, we realize wext_get_scan_custom got inlined into the function (it was static and only called from one place, so the compiler optimized it). Further tracking in the source shows that the “16″ value passed in should actually be “8″ (the limit of the destination, not the source, buffer size).

Looking back through my build logs, I can see that my desktop spent 18 hours, 49 minutes, and 4 seconds on 108 builds related to the roughly 64 breezy-security updates I was involved in publishing. So far, Dapper is at 132 builds totaling 19:59:40, and Edgy is at 142 builds totaling 23:32:28. These times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the PPC machine, etc. Even if it’s a bit incomplete, I think it’s fun to be able to point to some hard numbers about CPU time spent on Breezy updates. :)

Thank you Breezy! You have housed my MythTV installation very nicely, but now it’s time for some long over-due upgrades…

sudo apt-get install apparmor-modules-source dpatch
sudo m-a -v -t prepare
sudo m-a -v -t build apparmor-modules
sudo m-a -v -t install apparmor-modules
sudo apt-get install apparmor apparmor-utils apparmor-profiles libterm-readline-gnu-perl


With the default profiles, you can see one quick example of a confined process. Try doing this:

ping localhost >/dev/null &
sudo ps aZ | grep ping


In the first column, you should see what profile is being used to confine the process:

/bin/ping 14351 pts/14 S 0:00 ping localhost
unconstrained 15381 pts/14 S+ 0:00 grep ping


The list of active profiles can be seen as root in /sys/kernel/security/apparmor/profiles, which are loaded from /etc/apparmor.d/.

To confine a process, use aa-autodep and aa-logprof. For example, I wanted to confine my PDF document browser to only use /tmp (since I tend to only use it when browsing PDFs online):

• First, I create an empty profile in “complain” mode: sudo aa-autodep evince
• Next, I run evince like I normally would, including as many actions as I can think of (printing, preferences, help, etc). Watching the output of dmesg you can follow the trail of all the actions evince is taking. When I’m finished, I quit evince.
• Next, I run aa-logprof, which runs through all the kernel audit output and offers suggestions on what to allow from evince. Where appropriate, I select “abstrations” for things like Gnome, DNS, fonts, tmp dir usage, etc. When a whole directory tree should be allowed, I double-glob the path (/usr/share/evince/**). Once all the items from the log have been processed, the profile is saved.
• Finally, I enable the profile with aa-enforce evince. Any disallowed actions will show up in the kernel logs.

Check out the resulting profile for evince.

Now if I end up reading a malicious PDF that takes advantage of some currently-unknown vulnerability in evince, it will be confined to the above AppArmor profile, unable to exec new processes, and only able to write to the Gnome preferences for evince. (It’s also unable to read files out of /home, so that the above profile may be way too strict for common usage. And to even get caught by AppArmor, the imaginary exploit would have to avoid the randomized stack, randomized heap, stack protector, and, since I’m running 64bit, the NX processor bit.)

Be aware, this is still a new bit of packaging for Ubuntu, so you may run into sneaky gotchas. If that happens, please open a bug.

I personally use “4 spaces” for code indentation, and if I’m working on code that uses 8, I just hit “tab” twice, and if I’m working on code that uses 2, I can just backspace over the 2-too-many spaces. When the code has actual tabs, things break. When the code has a mix of tabs and spaces, it becomes a serious head-ache.

I wrote some vim insanity to detect which indentation type was being used “the most” in a given source file. If anyone has a simpler way to solve this (without switching to a different editor), I’m all ears. What follows are some bits from my .vimrc.

First, my space-indentation defaults:

set noai ts=4 sw=8 expandtab

Next, Makefiles and debian/rules files always use tabs, so I have a base set of overrides:

" Makefile sanity
autocmd BufEnter ?akefile* set noet ts=8 sw=8
autocmd BufEnter */debian/rules set noet ts=8 sw=8

Finally, define a function that compares the number of lines that start with a tab to those that start with a space. If the tabs outnumber the spaces, disable my defaults, and don’t expand tabs:

function Kees_settabs()
    if len(filter(getbufline(winbufnr(0), 1, "$"), 'v:val =~ "^\t"')) > len(filter(getbufline(winbufnr(0), 1, "$"), 'v:val =~ "^ "'))
        set noet ts=8 sw=8
    endif
endfunction
autocmd BufReadPost * call Kees_settabs()

• Inkscape rocks!
• Ubuntu is the best Linux desktop
• Duran Duran was neither Duran nor Duran

Or just see what’s been claimed about linux in general. Yay for silly social networking sites! :)

Since MozDev’s “linkify.user.js” was a bit heavy-handed, I wrote up a quick hack based on similar code to only look at mitre.org’s LI tags: “cve-links.user.js“.

One big hurdle is audio, since FMOD doesn’t have a Free license. I hope it can get replaced; I’d be curious to hear what Second Life needs from FMOD that some of the other Free stacks can’t do.

So, if you’re in a mood to play with getting the Second Life client running, hopefully my stab at packaging can help (I’ve solved a number of gotchas in the assumptions their build system made), and so far it built:

$ ls -lh secondlife-x86_64-bin
-rwxr-xr-x 1 kees kees 34M 2007-01-10 05:33 secondlife-x86_64-bin*
$ ldd ./secondlife-x86_64-bin | wc -l
77

Unfortunately, it immediately crashes when I load it. :)