Contact the author.
Generic (storage-device-agnostic) HOWTO install OpenGFS This document describes how to build/install/stop/start OpenGFS so a cluster of computers can share a single storage device. It provides a simple example configuration for one shared drive with 2 partitions, shared between 2 computers. NOTE: This document describes operation using the OpenGFS "pool" volume manager, which is no longer (as of Spring, 2003) being maintained by the project, and will eventually be removed from the project. This document applies to releases 0.2.1 and earlier, which require pool. If you would like to use OpenGFS without using pool (recommended), you must use current CVS code, or releases *later* than 0.2.1 (as they become available), and see the "HOWTO-nopool" document instead of this one. Depending on your setup, you may want to see other documentation as well: IEEE-1394 (firewire-attached) devices: See the OpenGFS HOWTO-1394 document for instructions on patching and re-building the kernel, including some 1394-specific patches. Then return to this document for the remaining steps of setting up OpenGFS. iSCSI (Internet SCSI): See the OpenGFS HOWTO-iSCSI document for instructions on setting up iSCSI, then return to this document for setting up OpenGFS. UML (User-Mode Linux): See the OpenGFS HOWTO-uml document. You may want to read this document in addition, but all instructions are in the HOWTO-uml document. Note that the UML setup does not use pool, so UML instructions are are significantly different from this document. This document assumes that you already have set up a shared storage device. Typically, this means installing interface hardware (e.g. a fibre channel host bus adapter) and drivers (e.g. qla2300 driver for QLogic 23x0 HBA) on each computer that will be sharing the storage device(s), and connecting each computer to the storage device (e.g. with fibre cable). You do not need to partition or initialize the storage device/drives (yet). Installation success is indicated by the drive(s) appearing in /proc/partitions on each computer in the shared storage cluster. Check for success: cat /proc/partitions shows shared drive(s) This document also assumes that your cluster member computers are interconnected by a LAN, and that you know the IP address of each computer. Most of the steps below must be executed on *all* computers in the cluster, while some (the steps that write to the shared storage device) need to be executed just once, on only one computer. You may find it easiest to follow the instructions by building the code separately on each computer in the cluster (especially if different computers run different kernel versions for some reason). However, if you want to do one build (e.g. on your development platform), and install on multiple target computers, there are two options: The opengfs.spec file in the CVS download version allows you to create an RPM package for easy installation on multiple computers. Use --prefix and --with-linux_moduledir options when running ./configure (see below), to install the build results into isolated directories, which you can copy to the other machines. Note that instructions below use names current for post-version-0.2 OpenGFS. If you use these instructions for 0.2 or earlier, you can usually substitute "gfs" wherever "ogfs" appears, and be okay. For example, the make utility changed names from "mkfs_gfs" to "mkfs.ogfs" (also note "_" vs. ".") after 0.2. BUILDING AND INSTALLING OpenGFS ------------------------------- 1. Get the OpenGFS source, either from CVS or via tarball (some releases are also available as source RPMs). Copy to each computer in the cluster. A. Retrieve (check out) the OpenGFS source from CVS: export CVSROOT=:pserver:firstname.lastname@example.org:/cvsroot/opengfs cvs login (just hit "enter" key for the password) cvs -z3 co opengfs (-z3 invokes compression, if desired) HINT: Any subsequent updates of the code can use "cvs up opengfs". *OR* B. Download a release tarball from http://opengfs.sourceforge.net. Click on "Downloads", then on "Sourceforge Download Page", and download the release you want, then: tar -xvzf opengfs-n.n.n.tar.gz (substitute version for n.n.n) or use bunzip2 if the tarball has a .bz2 suffix. 2. Patch/rebuild kernel with the OpenGFS patches (on each computer). These patches only modify the existing kernel code. They do not contain the OpenGFS kernel modules: cd /usr/src/linux (or wherever your kernel source tree lives) cat /path/to/opengfs/kernel_patches/2.4.x/*.patch | patch -p1 HINT: For kernels <= 2.4.18, instead of using *, you can selectively apply just the patches that you need, but applying them all doesn't hurt anything. For kernels >= 2.4.19, the various patches are gathered into one file, so all patches will apply automatically. Rebuild kernel Reboot (Return here from HOWTO-1394 . . . ) 3. Build and install OpenGFS kernel modules/tools/man-pages (on each computer). Do the bootstrap step only if you downloaded from CVS. It is not required for the tarball. Make sure you provide a correct path to your patched kernel, using --with-linux_srcdir, when running ./configure. Some of the patches modify include files needed for building ogfs. cd /path/to/opengfs ./bootstrap (ONLY FOR CVS DOWNLOADS) ./configure [options] HINT: If the bootstrap script spits out cryptic warnings, double check that you have the right versions of autoconf and automake installed. Some versions of RedHat and Debian Linux (and maybe others), came with a flawed wrapper script that tried to allow installing old and new versions of the auto... tools in parallel. Using them usually ends in disaster. Deinstall all autoconf and automake versions from your system, download the latest autoconf tarball and the automake-1.6.x tarball and install the tools from them, then try again. HINT: ./configure --help shows options. Some interesting ones: --with-linux_srcdir=/some/path, location of patched(!) linux source --with-opendlm=/some/path, location of OpenDLM source, see HOWTO-opendlm --prefix=/some/path, installs user binaries and man pages under here --with-linux_moduledir=/some/path, installs kernel modules under here --enable-extras, builds OpenGFS test tools --enable-*-debug, enables debug features in various OpenGFS components --enable-*-stats, enables statistics in various OpenGFS components --enable-uml, compile for User-Mode Linux, see HOWTO-uml --enable-opendlm, compile with OpenDLM lock module, see HOWTO-opendlm make su (you need root privilege for all following steps) make install Check for success*: /sbin contains "ptool" and many other ogfs tools Check for success*: /lib/modules/2.4.x/ogfs contains ogfs.o + others Check for success*: /usr/man/man8 contains ogfs.8, memexpd.8 + others * Default locations without --prefix or --with-linux_moduledir options. 4. Insert OpenGFS modules (on each computer): modprobe memexp modprobe ogfs Hint: If modprobe says it can't find memexp, try running depmod to update /lib/modules/(version)/modules.dep Check for success: cat /proc/filesystems shows "ogfs", among others Check for success: cat /proc/modules shows "memexp", "ogfs" + other modules used by memexp and ogfs, among others 5. Partition the shared drive (using only one computer), into 2 partitions. A small partition (~4MB) will be used for Cluster Information (ci). A large partition (the rest of the drive) will be used for data. Use the dev name of your shared drive in place of "sdX" below. See the man page for sfdisk for more information: sfdisk -R /dev/sdX (make sure no drive partitions are in use) sfdisk /dev/sdX (this partitions the disk; follow the prompts) Hint: sfdisk works in units of "cylinders". When partitioning my drive, sfdisk showed that each cylinder was 1048576 bytes. So, for the first (small) partition, I entered 0,4 to start at cylinder 0, with a size of 4 cylinders (~4MB). For the next partition, I entered nothing (except the Enter key), which defaulted to use the rest of the drive. For the next two (sfdisk asks for 4 partitions), I also entered nothing (except the Enter key). These last two "partitions", of course, were empty. After you enter all 4 partitions, sfdisk asks you if you really want to write to the disk, so you can experiment a bit before committing. Check for success: cat /proc/partitions shows the new partitions HINT: These partitions must show up on all computers in the cluster (although they do not need to be named consistently). After you create new partitions using one machine, you may need to find a way for the other machines to re-scan for partitions, or simply reboot the other machines (don't forget to re-modprobe memexp and ogfs). 6. Write OpenGFS partition labels to shared drive (using only one computer). These labels describe the shared drive pool(s), and will be read from the drive(s) by any computer joining the cluster. See opengfs/docs/ogfs-pool for more information. See opengfs/docs/pooltemplate to understand the configuration files: A. Edit a configuration file named pool0.cf to read like the following, using the dev name of the *large* partition (created in step 5) in place of "sdxn". This is for the data partition/pool. You can find a copy of this file, with comments, as opengfs/docs/pool-1d2p.cf. This file, and all other opengfs configuration files, may reside anywhere on your computer. poolname pool0 subpools 1 subpool 0 0 1 ogfs_data pooldevice 0 0 /dev/sdxn 0 Note: The /dev/sdxn dev name tells *this* computer how to access the physical drive partition to write the label. This dev name does not need to match dev names of the partition as seen by other cluster member computers; they do not use it to access the drive pool. B. Edit a file named pool0cidev.cf to read like the following, using the dev name of the *small* partition (created in step 5) in place of "sdxn". This is for the cluster information (ci) partition/pool. You can find a copy of this file, with comments, as opengfs/docs/poolcidev-1d2p.cf. poolname pool0_cidev subpools 1 subpool 0 0 1 ogfs_data pooldevice 0 0 /dev/sdxn 0 C. Run OpenGFS tool "ptool" to write partition labels to the disks. See the man page for ptool for more information. ptool pool0.cf ptool pool0cidev.cf Check for success: ptool messages indicate success 7. Read Drive Pool labels from disk, set up Drive Pools (on each computer): Run the OpenGFS tool "passemble" to cause each computer to read the partition labels written in step 6, to obtain Drive Pool information. See the man page for passemble for more information. passemble Check for success: passemble messages indicate success 8. Make the OpenGFS Filesystem (using only one computer). Use the OpenGFS tool "mkfs.ogfs" to create the file system on disk. The command below uses two special options (-j 2 and -i) that create two journal subpools (one for each computer in the cluster) within the one pool you created earlier. See opengfs/docs/ogfs-pool. See the man page for mkfs.ogfs for more information on options: mkfs.ogfs -p memexp -t /dev/pool/pool0_cidev -j 2 -i /dev/pool/pool0 HINT: This can take some time to write to disk, and shows no output until it is done. Be patient. 9. Configure the OpenGFS Cluster, write it to disk (using only one computer). Now that the drive pools have been set up, and the file system has been created on the storage medium, we need to describe the computers (nodes) that will be sharing the drive pools. This includes the STOMITH ("Shoot The Other Machine In The Head") method(s) used for resetting a computer when it fails, and the heartbeat period used for detecting failure. For more information, see the man page for ogfsconf. Also see opengfs/docs/ogfscftemplate to understand the configuration file. HINT: When using Manual STOMITH, after resetting a dead computer (but *not* now, though, while you initially set up the cluster), you must run the following OpenGFS tool to continue the recovery process. See the man page for do_manual for more information: do_manual -s $IP_ADDR_OF_DEAD_NODE (don't do this now, though) A. Edit a configuration file named ogfscf.cf to read like the following. You'll need to substitute appropriate IP addresses for the lock server computer and the cluster member computers (nodes). However, you'll want to use the same port numbers (15697 and 3001). See ogfsconf man page. datadev: /dev/pool/pool0 cidev: /dev/pool/pool0_cidev lockdev: 192.168.0.37:15697 cbport: 3001 timeout: 30 STOMITH: manual name: manual node: 192.168.0.37 0 SM: manual 1 node: 192.168.0.203 1 SM: manual 2 B. Write the config info to the Cluster Information (ci) partition. ogfsconf -c ogfscf.cf 10. Start the Lock Server (on only one computer, the lock server computer). Launch the OpenGFS memexpd lock server daemon (see memexpd man page): memexpd & 11. Mount the Filesystem (on each computer). The following commands mount the ogfs file system at the /ogfs mount point. For hostdata, you'll need to substitute the IP address of the computer on which you are currently mounting the filesystem (i.e. *this* computer). mkdir /ogfs mount -t ogfs /dev/pool/pool0 /ogfs -o hostdata=192.168.0.x That's it, you are done! You should now be able to use this like any other filesystem. SHUTTING DOWN CLEANLY -------------------- As an alternative to manually executing the steps below, look in opengfs/scripts for pool.* and ogfs.* startup scripts for Debian and Red Hat distributions. If you want to shut down your computers, you will need to unmount OpenGFS. 1. Unmount the Filesystem (on each computer). umount /ogfs (this assumes you mounted onto /ogfs) 2. If you find that you need to unload the pool kernel module for some reason (usually not necessary for shutting down the computer), you will need to run the following to remove all knowledge of pools from the module, and reduce usage count to 0: passemble -r all Then unload the modules: modprobe -r ogfs modprobe -r memexp 3. If you want to uninstall the tools/modules/man-pages, go to your build directory, and run: make uninstall If you deleted the tree (oops!), it can be rebuilt from scratch, but make sure you use the same paths. STARTING OpenGFS (e.g. after boot-up) ------------------------------------ As an alternative to manually executing the steps below, look in opengfs/scripts for pool.* and ogfs.* startup scripts for Debian and Red Hat distributions. Once OpenGFS has been installed on your computers and storage media, only a few steps are needed to get it going after a boot-up. The following steps assume that your storage media hardware and drivers are installed and visible by all computers in the cluster. Check for success: cat /proc/partitions shows shared drive(s) You will need root privilege for all steps below: 1. Load OpenGFS kernel modules (on each computer). modprobe memexp modprobe ogfs Check for success: cat /proc/filesystems shows "ogfs", among others Check for success: cat /proc/modules shows "memexp", "ogfs" + other modules used by memexp and ogfs, among others 2. Read Drive Pool labels from disk, set up Drive Pools (on each computer): passemble Check for success: passemble messages indicate success 3. Start the Lock Server (on only one computer, the lock server computer). This must be started before mounting the filesystem on *any* of the computers in the cluster. memexpd & (on lock server computer only!) 4. Mount the Filesystem (on each computer). Remember, "hostdata" is the IP address of the computer on which you are currently mounting the filesystem (i.e. *this* computer). mount -t ogfs /dev/pool/pool0 /ogfs -o hostdata=192.168.0.x That's it, you are done! You should now be able to use this like any other filesystem. Copyright 2002-2003 The OpenGFS Project