HOWTO_build_and_install_OpenGFS_(with_pool,_old) (Jan 16 2004)


Contact the author.

Generic (storage-device-agnostic) HOWTO install OpenGFS

This document describes how to build/install/stop/start OpenGFS so a cluster
of computers can share a single storage device.  It provides a simple example
configuration for one shared drive with 2 partitions, shared between
2 computers.

NOTE:  This document describes operation using the OpenGFS "pool" volume
manager, which is no longer (as of Spring, 2003) being maintained by the
project, and will eventually be removed from the project.  This document
applies to releases 0.2.1 and earlier, which require pool.  If you would like
to use OpenGFS without using pool (recommended), you must use current CVS code,
or releases *later* than 0.2.1 (as they become available), and see the
"HOWTO-nopool" document instead of this one.

Depending on your setup, you may want to see other documentation as well:

  IEEE-1394 (firewire-attached) devices:  See the OpenGFS HOWTO-1394 document
  for instructions on patching and re-building the kernel, including some
  1394-specific patches.  Then return to this document for the remaining steps
  of setting up OpenGFS.

  iSCSI (Internet SCSI):  See the OpenGFS HOWTO-iSCSI document for instructions
  on setting up iSCSI, then return to this document for setting up OpenGFS.

  UML (User-Mode Linux):  See the OpenGFS HOWTO-uml document.  You may want to
  read this document in addition, but all instructions are in the HOWTO-uml
  document.  Note that the UML setup does not use pool, so UML instructions are
  are significantly different from this document.

This document assumes that you already have set up a shared storage device.
Typically, this means installing interface hardware (e.g. a fibre channel
host bus adapter) and drivers (e.g. qla2300 driver for QLogic 23x0 HBA) on
each computer that will be sharing the storage device(s), and connecting
each computer to the storage device (e.g. with fibre cable).  You do not
need to partition or initialize the storage device/drives (yet).  Installation
success is indicated by the drive(s) appearing in /proc/partitions on each
computer in the shared storage cluster.

Check for success:  cat /proc/partitions shows shared drive(s)

This document also assumes that your cluster member computers are
interconnected by a LAN, and that you know the IP address of each computer.

Most of the steps below must be executed on *all* computers in the cluster,
while some (the steps that write to the shared storage device) need to be
executed just once, on only one computer.

You may find it easiest to follow the instructions by building the code
separately on each computer in the cluster (especially if different computers
run different kernel versions for some reason).  However, if you want to do
one build (e.g. on your development platform), and install on multiple target
computers, there are two options:

  The opengfs.spec file in the CVS download version allows you to create an
  RPM package for easy installation on multiple computers.

  Use --prefix and --with-linux_moduledir options when running ./configure
  (see below), to install the build results into isolated directories, which
  you can copy to the other machines.

Note that instructions below use names current for post-version-0.2 OpenGFS.
If you use these instructions for 0.2 or earlier, you can usually substitute
"gfs" wherever "ogfs" appears, and be okay.  For example, the make utility
changed names from "mkfs_gfs" to "mkfs.ogfs" (also note "_" vs. ".") after 0.2.


BUILDING AND INSTALLING OpenGFS
-------------------------------

1. Get the OpenGFS source, either from CVS or via tarball (some releases are
	also available as source RPMs).  Copy to each computer in the cluster.

	A.  Retrieve (check out) the OpenGFS source from CVS:

	export CVSROOT=:pserver:anonymous@cvs.opengfs.sourceforge.net:/cvsroot/opengfs
	cvs login		(just hit "enter" key for the password)
	cvs -z3 co opengfs	(-z3 invokes compression, if desired)

	HINT:  Any subsequent updates of the code can use "cvs up opengfs".

		*OR*

	B.  Download a release tarball from http://opengfs.sourceforge.net.
	Click on "Downloads", then on "Sourceforge Download Page", and
	download the release you want, then:

	tar -xvzf opengfs-n.n.n.tar.gz	(substitute version for n.n.n)

	or use bunzip2 if the tarball has a .bz2 suffix.


2. Patch/rebuild kernel with the OpenGFS patches (on each computer).
	These patches only modify the existing kernel code.  They do not
	contain the OpenGFS kernel modules:

	cd /usr/src/linux	(or wherever your kernel source tree lives)
	cat /path/to/opengfs/kernel_patches/2.4.x/*.patch | patch -p1

	HINT:  For kernels <= 2.4.18, instead of using *, you can
		selectively apply just the patches that you need,
		but applying them all doesn't hurt anything.
		For kernels >= 2.4.19, the various patches are gathered
		into one file, so all patches will apply automatically.

	Rebuild kernel
	Reboot


(Return here from HOWTO-1394 . . . )
3. Build and install OpenGFS kernel modules/tools/man-pages (on each computer).
	Do the bootstrap step only if you downloaded from CVS.  It is not
	required for the tarball.

	Make sure you provide a correct path to your patched kernel, using
	--with-linux_srcdir, when running ./configure.  Some of the patches
	modify include files needed for building ogfs.

	cd /path/to/opengfs
	./bootstrap (ONLY FOR CVS DOWNLOADS)
	./configure [options]

	HINT:  If the bootstrap script spits out cryptic warnings, double check
	that you have the right versions of autoconf and automake installed.
	Some versions of RedHat and Debian Linux (and maybe others), came with
	a flawed wrapper script that tried to allow installing old and new
	versions of the auto... tools in parallel.  Using them usually ends in
	disaster.  Deinstall all autoconf and automake versions from your
	system, download the latest autoconf tarball and the automake-1.6.x
	tarball and install the tools from them, then try again.

	HINT:  ./configure --help shows options.  Some interesting ones:
	--with-linux_srcdir=/some/path, location of patched(!) linux source
	--with-opendlm=/some/path, location of OpenDLM source, see HOWTO-opendlm
	--prefix=/some/path, installs user binaries and man pages under here
	--with-linux_moduledir=/some/path, installs kernel modules under here
	--enable-extras, builds OpenGFS test tools
	--enable-*-debug, enables debug features in various OpenGFS components
	--enable-*-stats, enables statistics in various OpenGFS components
	--enable-uml, compile for User-Mode Linux, see HOWTO-uml
	--enable-opendlm, compile with OpenDLM lock module, see HOWTO-opendlm

	make
	su		(you need root privilege for all following steps)
	make install

	Check for success*:  /sbin contains "ptool" and many other ogfs tools
	Check for success*:  /lib/modules/2.4.x/ogfs contains ogfs.o + others
	Check for success*:  /usr/man/man8 contains ogfs.8, memexpd.8 + others

	* Default locations without --prefix or --with-linux_moduledir options.


4. Insert OpenGFS modules (on each computer):

	modprobe memexp
	modprobe ogfs

	Hint:  If modprobe says it can't find memexp, try running depmod to
	update /lib/modules/(version)/modules.dep

	Check for success:  cat /proc/filesystems shows "ogfs", among others
	Check for success:  cat /proc/modules shows "memexp", "ogfs"
			+ other modules used by memexp and ogfs, among others


5. Partition the shared drive (using only one computer), into 2 partitions.
	A small partition (~4MB) will be used for Cluster Information (ci).
	A large partition (the rest of the drive) will be used for data.
	Use the dev name of your shared drive in place of "sdX" below.
	See the man page for sfdisk for more information:

	sfdisk -R /dev/sdX  (make sure no drive partitions are in use)
	sfdisk /dev/sdX	    (this partitions the disk; follow the prompts)

	Hint:  sfdisk works in units of "cylinders".  When partitioning
	my drive, sfdisk showed that each cylinder was 1048576 bytes.
	So, for the first (small) partition, I entered 0,4 to start at
	cylinder 0, with a size of 4 cylinders (~4MB).  For the next
	partition, I entered nothing (except the Enter key), which
	defaulted to use the rest of the drive.  For the next two (sfdisk
	asks for 4 partitions), I also entered nothing (except the
	Enter key).  These last two "partitions", of course, were empty.
	After you enter all 4 partitions, sfdisk asks you if you really
	want to write to the disk, so you can experiment a bit
	before committing.

	Check for success:  cat /proc/partitions shows the new partitions

	HINT:  These partitions must show up on all computers in the cluster
	(although they do not need to be named consistently).  After you
	create new partitions using one machine, you may need to find a way
	for the other machines to re-scan for partitions, or simply reboot
	the other machines (don't forget to re-modprobe memexp and ogfs).


6. Write OpenGFS partition labels to shared drive (using only one computer).
	These labels describe the shared drive pool(s), and will be read
	from the drive(s) by any computer joining the cluster.
	See opengfs/docs/ogfs-pool for more information.
	See opengfs/docs/pooltemplate to understand the configuration files:

	A.  Edit a configuration file named pool0.cf to read like the
		following, using the dev name of the *large* partition
		(created in step 5) in place of "sdxn".  This is for
		the data partition/pool.  You can find a copy of this
		file, with comments, as opengfs/docs/pool-1d2p.cf.
		This file, and all other opengfs configuration files,
		may reside anywhere on your computer.

poolname	pool0
subpools	1
subpool		0	0	1	ogfs_data
pooldevice	0	0	/dev/sdxn	0

		Note:  The /dev/sdxn dev name tells *this* computer how
		to access the physical drive partition to write the label.
		This dev name does not need to match dev names of the partition
		as seen by other cluster member computers; they do not use it
		to access the drive pool.

	B.  Edit a file named pool0cidev.cf to read like the following, using
		the dev name of the *small* partition (created in step 5)
		in place of "sdxn".  This is for the cluster information
		(ci) partition/pool.  You can find a copy of this file,
		with comments, as opengfs/docs/poolcidev-1d2p.cf.

poolname	pool0_cidev
subpools	1
subpool		0	0	1	ogfs_data
pooldevice	0	0	/dev/sdxn	0

	C.  Run OpenGFS tool "ptool" to write partition labels to the disks.
		See the man page for ptool for more information.

	ptool pool0.cf
	ptool pool0cidev.cf

	Check for success:  ptool messages indicate success


7.  Read Drive Pool labels from disk, set up Drive Pools (on each computer):
	Run the OpenGFS tool "passemble" to cause each computer to read the
	partition labels written in step 6, to obtain Drive Pool information.
	See the man page for passemble for more information.

	passemble

	Check for success:  passemble messages indicate success


8. Make the OpenGFS Filesystem (using only one computer).
	Use the OpenGFS tool "mkfs.ogfs" to create the file system on disk.
	The command below uses two special options (-j 2 and -i) that create
	two journal subpools (one for each computer in the cluster) within the
	one pool you created earlier.  See opengfs/docs/ogfs-pool.
	See the man page for mkfs.ogfs for more information on options:

	mkfs.ogfs -p memexp -t /dev/pool/pool0_cidev -j 2 -i /dev/pool/pool0

	HINT:  This can take some time to write to disk, and shows no output
	until it is done.  Be patient.

9. Configure the OpenGFS Cluster, write it to disk (using only one computer).
	Now that the drive pools have been set up, and the file system
	has been created on the storage medium, we need to describe
	the computers (nodes) that will be sharing the drive pools.
	This includes the STOMITH ("Shoot The Other Machine In The Head")
	method(s) used for resetting a computer when it fails, and the
	heartbeat period used for detecting failure.  For more information,
	see the man page for ogfsconf.  Also see opengfs/docs/ogfscftemplate
	to understand the configuration file.

	HINT:  When using Manual STOMITH, after resetting a dead computer
	(but *not* now, though, while you initially set up the cluster),
	you must run the following OpenGFS tool to continue the recovery
	process.  See the man page for do_manual for more information:

	do_manual -s $IP_ADDR_OF_DEAD_NODE  (don't do this now, though)

	A.  Edit a configuration file named ogfscf.cf to read like the
		following.  You'll need to substitute appropriate
		IP addresses for the lock server computer and the
		cluster member computers (nodes).  However, you'll want
		to use the same port numbers (15697 and 3001).
		See ogfsconf man page.

datadev: /dev/pool/pool0
cidev: /dev/pool/pool0_cidev
lockdev: 192.168.0.37:15697
cbport: 3001

timeout: 30
STOMITH: manual
name: manual

node: 192.168.0.37 0 SM: manual 1
node: 192.168.0.203 1 SM: manual 2


	B.  Write the config info to the Cluster Information (ci) partition.

	ogfsconf -c ogfscf.cf


10. Start the Lock Server (on only one computer, the lock server computer).
	Launch the OpenGFS memexpd lock server daemon (see memexpd man page):

	memexpd &


11. Mount the Filesystem (on each computer).
	The following commands mount the ogfs file system at the /ogfs
	mount point.  For hostdata, you'll need to substitute the IP address
	of the computer on which you are currently mounting the filesystem
	(i.e. *this* computer).

	mkdir /ogfs
	mount -t ogfs /dev/pool/pool0 /ogfs -o hostdata=192.168.0.x


That's it, you are done!
You should now be able to use this like any other filesystem.



SHUTTING DOWN CLEANLY
--------------------
As an alternative to manually executing the steps below, look in
opengfs/scripts for pool.* and ogfs.* startup scripts for Debian and Red Hat
distributions.

If you want to shut down your computers, you will need to unmount OpenGFS.

1.  Unmount the Filesystem (on each computer).

	umount /ogfs   (this assumes you mounted onto /ogfs)


2.  If you find that you need to unload the pool kernel module for some
	reason (usually not necessary for shutting down the computer), you
	will need to run the following to remove all knowledge
	of pools from the module, and reduce usage count to 0:

	passemble -r all

	Then unload the modules:

	modprobe -r ogfs
	modprobe -r memexp


3.  If you want to uninstall the tools/modules/man-pages, go to your
	build directory, and run:

	make uninstall

	If you deleted the tree (oops!), it can be rebuilt from scratch, but
	make sure you use the same paths.


STARTING OpenGFS (e.g. after boot-up)
------------------------------------
As an alternative to manually executing the steps below, look in
opengfs/scripts for pool.* and ogfs.* startup scripts for Debian and Red Hat
distributions.

Once OpenGFS has been installed on your computers and storage media, only a
few steps are needed to get it going after a boot-up.  The following steps
assume that your storage media hardware and drivers are installed and visible
by all computers in the cluster.

Check for success:  cat /proc/partitions shows shared drive(s)

You will need root privilege for all steps below:

1.  Load OpenGFS kernel modules (on each computer).

	modprobe memexp
	modprobe ogfs

	Check for success:  cat /proc/filesystems shows "ogfs", among others
	Check for success:  cat /proc/modules shows "memexp", "ogfs"
			+ other modules used by memexp and ogfs, among others


2.  Read Drive Pool labels from disk, set up Drive Pools (on each computer):

	passemble

	Check for success:  passemble messages indicate success


3.  Start the Lock Server (on only one computer, the lock server computer).
	This must be started before mounting the filesystem on *any* of the
	computers in the cluster.

	memexpd &  (on lock server computer only!)


4.  Mount the Filesystem (on each computer).
	Remember, "hostdata" is the IP address of the computer on which you
	are currently mounting the filesystem (i.e. *this* computer).

	mount -t ogfs /dev/pool/pool0 /ogfs -o hostdata=192.168.0.x


That's it, you are done!
You should now be able to use this like any other filesystem.

Copyright 2002-2003 The OpenGFS Project
SourceForge Logo