AWMN @ oZoNet

The product: a wireless networking device

The hardware: an embedded PC-like board, with a serial port at ttyS0, a
CompactFlash card at hda, and PCMCIA for the wireless cards.

The software: a Linux 2.4.x kernel, and a good collection of network
utilities (iptables, brctl, iptraf, tcpdump, rp-pppoe, dhcpd, openssh, and
more)

The attitude: Don't even think about seeing the source code, or modifying the
software.

The detective story:

Removing the CF card and sticking it in a more cooperative host reveals 3
partitions. hda2 and hda3 are unidentifiable. hda1 is an ext2 filesystem. It
contains what looks like a stripped-down /boot directory. There is a bzImage,
an initrd, a lilo map file, and some small mysterious files. The initrd is
not a recognizable filesystem.

The boot sector and second-stage loader are from LILO 22.5.7 and appear
unmodified, which eliminates the possibility that the initrd was being XORed
with some magic sequence as it's being loaded. Checked for the possibility
that the files named "bzImage" and ".initrd" don't correspond to the sectors
that LILO is actually loading, but that was a dead end. There's no trickery
in the bootloader.

That leaves the bzImage itself as the only possible place where deobfuscation
is being done. Copied the bzImage to a regular hard drive and booted it with
my regular root filesystem and no initrd, in order to gain clues about what
sort of encryption code or weird filesystems it might contain. Operation in
this mode is a little awkward because there is no VGA console support (no VGA
or keyboard port on the embedded board), so everything has to be done off a
serial port.

The bzImage supports the kerneli.org cryptoapi with the "serpent" cipher, and
has built-in support for ext2 and cramfs. None of that is helpful yet. How
can an initrd be encrypted? The answer must be in the loading of the initrd
itself. So we must compare the bzImage with a stock kernel - the decryption
of initrd will be will be an identifiable difference.

The big gzip-format hunk of bzImage is vmlinux, with its ELF skeleton
removed. Adding a new ELF skeleton is a matter of writing a custom ld
script, and now there's a nice vmlinux that can be analyzed with objdump -D
(Never did figure out how to make one that objdump -d would like). The
section boundaries and symbol table are missing, but this wouldn't be fun if
it was easy.

The files involved in initrd processing are drivers/block/rd.c and
init/do_mounts.c. Now it's time to ask yourself: "if I were an initrd
deobfuscator, where would I live?" I think I'd like to live a high level,
like in prepare_namespace. A little side-by-side assembly vs. C comparison,
and it looks clean, unchanged. So does initrd_load, and rd_load_image. The
4th guess provides a hit: identify_ramdisk_image() has been modified to frob
the ramdisk data before using it. (It actually tries everything twice: once
without frobbing and once with frobbing.)

The frobbing consists of a setup function, called once, which apparently
generates a key from a passphrase (the passphrase is simply a hardcoded
8-byte sequence in the kernel image), and the decrypt function, which takes a
buffer and length and frobs it in place.

Those functions are short, so it didn't take long to translate them into a
standalone C program that frobs from stdin to stdout, and applying that to
the initrd... bingo! gzipped ext2.

Now the decoded initrd can be mounted. On it you will find a bare minimum set
of unix commands, all of them symlinked to busybox, and a /linuxrc script
which (among other chores) mounts the mystery partitions. hda2 and hda3 are
both subject to a cryptoloop mount. The losetup command (which is actually
provided by busybox) has been modified to use a fixed passphrase that is
a mixture of a hardcoded string and the md5sums of some files located on the
unencrypted partition.

Using that passphrase directly doesn't work for some reason. This wonderful
document:
  http://clemens.endorphin.org/Cryptoloop_Migration_Guide
reveals that there are multiple incompatible versions of cryptoloop, and it
gets worse when the serpent cipher is involved. We need to know exactly what
that modified losetup is doing.

The modified losetup doesn't produce the correct results under a 2.6 kernel.
It will only work with 2.4+kerneli, and the only one of those that happens to
be laying around is the modified one on the CF card. As mentioned before,
that bzImage will boot and run a normal userland; the main thing it's missing
is a VGA console. So back to the serial port we go, running the modified
kernel, mounting its initrd this time (not as a root filesystem though, still
using the normal root filesystem from the host's hard drive), and now knowing
the magic incantation to get the modified losetup to use its internal key.
hda2 and hda3 are finally decrypted! One is an ext2 filesystem, the other one
is cramfs.

At this point, the whole filesystems could just be copied from /dev/loop*
onto the hard drive for further inspection. That's not as satisfying, though,
as mounting them directly under a 2.6 kernel would be. From the Cryptoloop
Migration Guide, there is a tool called "lo-tracker" which wraps a losetup
invocation and prints the actual key given to the kernel. Unfortunately, it
does this by preloading ioctl(), and this losetup, being part of busybox,
uses no dynamic linking. LD_PRELOAD, and therefore lo-tracer, is worthless.

strace in theory is the right tool for the job, but it doesn't know how to
decode loop-device ioctls. Time to modify strace? Doesn't sound fun (which is
probably why there are lots of things strace doesn't decode). There is
another way. The busybox ioctl() can't be overrided through LD_PRELOAD, but
it can be found pretty easily (there aren't too many int $0x80's in there;
find the one that's preceded by mov $NR_ioctl,%eax).

Set a gdb breakpoint on the statically-linked ioctl() within busybox.
condition it to only stop if the request is one of the LOOP_* values. Run
the losetup, and when it stops, print out 32 bytes pointed to by the arg.
This is the key, we hope (the serial terminal experience is getting old).
Reboot, with normal 2.6 kernel (plus the tnepres patch from the Cryptoloop
Migration Guide). Install a normal losetup, plus the patch from the
Cryptoloop Migration Guide (all glory to the CMG) so it can take a raw key
from stdin rather than a passphrase.

The filesystems are decrypted. The addition of a proper shell, accessible
through ssh when the CF card is placed back in the device, is now easily
accomplished.

Hoping to inspire some discussion, or at least more heroic tales of struggle
against pointless obstacles,

-- 
pacman