Contact the author.
Running OpenGFS under User Mode Linux (UML)
Copyright 2003 The OpenGFS Project.
Introduction
------------
User mode Linux is a set of patches for the Linux kernel that allows running
a nested Linux kernel as a userland process. As it is, UML is mainly
interesting for developers to debug kernel components. It may also make sense
to use UML as a replacement for stomith hardware: when a box falls out of the
configured cluster, use "kill -9" as a stomith method.
This document assumes you are already familiar with setting up OpenGFS on a
normal cluster. Please refer to the HOWTO-generic for further details.
Compiling for UML is supported only for opengfs-0.3.0 and later.
UML Setup
---------
1. Download the target kernel source tarball, for example linux-2.4.20.tar.gz.
2. Download a matching UML patch from http://user-mode-linux.sourceforge.net/
Example: uml-patch-2.4.20-1.bz2
3. OpenGFS now (as of July 30, 2003, and any releases after that date)
requires patches for the DM device mapper to be in your kernel.
You have (at least) 2 choices for obtaining them:
A. If you are going to use EVMS, see (OpenGFS doc) HOWTO-evms for
information on how to install EVMS and DM (EVMS uses DM), including
information on the convenient tarballs available on the OpenGFS
download site (*not* part of the OpenGFS CVS source tree).
B. If you will not use EVMS, and want to keep your patching as compact
as possible, obtain patches directly from DM project website,
http://sources.redhat.com/dm/
Select the latest patches that apply to your kernel.
4. Unpack the Linux sources and the patches and apply the patches.
HINT: To avoid patch conflicts, you may want to start with a pristine
kernel.
$ tar xvzf linux-2.4.20.tar.gz
$ bunzip2 uml-patch-2.4.20-1.bz2
$ cd linux-2.4.20
$ patch -p1 < ../
$ patch -p1 < ../uml-patch-2.4.20-1
5. Apply the OpenGFS kernel patch(es).
$ patch -p1 < /kernel_patches/2.4.20/linux-2.4.20-ogfs.patch
6. Configure and compile the kernel. In the basic setup you do not need to
change any configuration options.
$ make menuconfig ARCH=um
$ make dep ARCH=um
$ make clean ARCH=um
$ make linux ARCH=um
IMPORTANT NOTE: If you forget to add "ARCH=um" at the end of any of these
commands, either the UML kernel or OpenGFS will not build properly.
Specifically, you can not build OpenGFS with a kernel source tree that was
configured without ARCH=um, even if you repeat the process later. You have
to remove the whole kernel tree and begin from scratch if that happens.
Believe me.
7. Fetch a UML root file system of your choice. You need to compile OGFS on the
UML machine, so you will need a few libraries and development tools
installed. You can download a small Debian-3.0 form the OpenGFS sourceforge
page that should help ease the initial setup (skip to step H):
http://sourceforge.net/projects/opengfs
If it is not good enough for you, you have to make your own. For Debian,
try this:
A. Install the debootstrap packacge on your regular machine.
B. Create a file for the root_fs image
# 1GB sparse root image
$ dd if=/dev/zero of=root_fs bs=1M seek=1023 count=1
C. Create a file system.
$ mke2fs root_fs
(answer 'y'es in the confirmation dialog)
D. Mount the image via the loop driver.
$ mount root_fs /mnt -t ext2 -o loop
E. Use debootstrap to create a minimal Debian installation:
$ debootstrap woody /mnt
F. Install more packages:
$ chroot /mnt
$ mount -t proc proc /proc
# install packages as usual.
# ...
$ umount /proc
$ exit
Do not forget to unmount the procf. If you forget it, you can not
unmount the file system image anymore.
G. Unmount the image and the procfs
$ umount /mnt
If you start UML before unmounting, you have to reboot your machine to
remove the stale mount.
H. Start the UML machine
$ nice ./linux mem=200M
OGFS Setup
----------
Below this point, all commands are typed in the UML machine.
I. Log in as root and mount the host file system in the UML machine:
$ mount -t hostfs hostfs /mnt
Note: You need to compile hostfs support into the UML kernel to use this
feature.
J. Copy the opengfs tarball to the UML machine and unpack it
$ cd /usr/local
$ cp /mnt//opengfs-x.y.z.tar.gz .
$ tar xvzf opengfs-x.y.z.tar.gz
K. Configure opengfs
$ cd opengfs-x.y.z
$ ./configure --enable-uml \
--with-linux_srcdir=/mnt//linux-2.4.20 \
--with-linux_moduledir=/lib/modules/2.4.20-1um
L. Build and install opengfs
$ make
$ make install
The instructions below assume you want to run an ogfs file system locally on a
single UML machine. Things become much more complicated for a cluster setup.
M. Load the kernel modules
$ insmod lock_harness
$ insmod divdi3
$ insmod ogfs
$ insmod nolock
N. Create a file for the ogfs file system and a mountpoint and format the
file
$ mkdir /ogfs
$ dd if=/dev/zero of=/ogfs_file bs=1M seek=255 count=1
$ mkfs.ogfs -i -j 1 -J 32 -p nolock /ogfs_file
O. Mount the ogfs file system
$ mount -t ogfs /ogfs /ogfsmnt -o loop
Debugging
---------
Now we want to debug the kernel modules. The following lines are typed on the
normal machine.
IMPORTANT NOTE: OGFS versions prior to 0.3.0 are compiled with the
-fomit-frame-pointer gcc option by default which makes it difficult to use a
debugger with the modules.
P. Attaching a debugger
When the UML starts, it prints a message "tracing thread pid = ".
Find out the and start the UML debugger with
$ kill -USR1
An xterm running gdb should show up.
If you are running only one UML machine, you can use this script instead:
--------------------------------- snip ----------------------------------
#!/bin/sh
TTPID=`find /proc -name cmdline 2> /dev/null | xargs grep -il "\[(tracing thread)\]" 2> /dev/null | sed -e 's:/proc/\([[:digit:]]*\)/.*:\1:g' 2> /dev/null`
if test x"$TTPID" = x; then
echo no tracing thread found
exit 1
else
echo "sending SIGUSR1 to thread $TTPID"
kill -USR1 $TTPID
fi
--------------------------------- snip ----------------------------------
Q. Loading the symbol table
On the debugger prompt type
(gdb) p module_list->name
(gdb) p module_list->next->name
(gdb) p module_list->next->next->name
...
Until you find the "ogfs" module. On my UML machine it is always
module_list->next->name. Get the address of the corresponding structure
and add a certain constant offset:
(gdb) p /x (unsigned int)module_list->next + module_list->size_of_struct
This prints an address to the gdb window. With this address, type
(gdb) dir /src/fs/arch_linux
(gdb) dir /src/include
(gdb) add-symbol-file /src/fs/arch_linux/ogfs.o
is the path to the ogfs sources on the normal machine. If you
start UML and ogfs in the same way each time, the address will probably
have the same value each time. You can then put the above steps in a
.gdbinit file (in the directory from which you started the UML):
--------------------------------- snip ----------------------------------
dir /src/fs/arch_linux
dir /src/include
add-symbol-file /src/fs/arch_linux/ogfs.o
--------------------------------- snip ----------------------------------
R. Set a breakpoint and start debugging
For example:
(gdb) b ogfs_mkdir
(gdb) cont
That's it.