Installing Amber 11 on CentOS 4.3

Amber 11 on CentOS 4.3

This article will explain how to get Amber 11 ( http://ambermd.org ) to compile and run on my cluster with MPICH2. The systems I work with are old, well in terms of computers anyway. It consists of a cluster of PowerEdge 1950's running CentOS 4.3. Specifically:

• Dual Intel Xeon E5320 1.86GHz
• 4GB or 8GB 533MHz DDR2
• 146GB 15kRPM Hardrive
• CentOS 4.3
• gcc-4.1.2

Before we get started, some acknowledments are in order, many of the procedures posted herein are pulled from Mark Williamson [1], the MPICH2 readme [2] and from the Amber 11 and AmberTools 1.4 manuals [3].

[1] http://www.wmd-lab.org/mjw/howtos/compiling_gcc_4_4_4_under_RHEL4.txt

[2] http://www.mcs.anl.gov/research/projects/mpich2/index.php

[3] http://ambermd.org

The main reason for this, is that Amber 11 does not compile properly with gcc-4.1.2 which is available on CentOS 4.3. Therefore we must compile gcc-4.4.4, then MPICH2-1.2.1p1 and Amber 11 using it.

Setup

All the programs we setup here will be placed in /opt, and things are built in $BUILD as much as possible

$ BUILD=/scratch

Compiling gcc-4.4.4

gcc-4.4.4 requires gmp ( GNU MP, a library for abritrary precision arithmitic ). It is available on CentOS and must be installed prior to compilation.

 yum install gmp gmp-devel

This installs gmp-4.1.4.

Get gcc-4.4.4:

$ cd $BUILD

$ wget ftp://mirrors.kernel.org/gnu/gcc/gcc-4.4.4/gcc-4.4.4.tar.gz

$ sha1sum gcc-4.4.4.tar.bz2

45d1e7242a08cbb7a343ed2bf640a5afd0ebe098 gcc-4.4.4.tar.bz2

Untar it.

$ tar -jxf gcc-4.4.4.tar.bz2

Gcc requires MPFR 2.3.2+ for it compile. MPFR is not available on CentOS 4.3, so we must provide it within the gcc source directory. The highest version of MPFR that is recommended with our version of gmp is MPFR-2.3.2

Get MPFR 2.3.2

$ wget http://www.mpfr.org/mpfr-2.3.2/mpfr-2.3.2.tar.bz2

$ sha1sum mpfr-2.3.2.tar.bz2

37dbd478e1c136f37cf4c68eb1522e86e2307288 mpfr-2.3.2.tar.bz2

The mpfr source must be placed within the gcc source directory in a directory

named 'mpfr'.

$ tar -jxf mpfr-2.3.2.tar.bz2

$ sha1sum mpfr-2.3.2.tar.bz2

37dbd478e1c136f37cf4c68eb1522e86e2307288 mpfr-2.3.2.tar.bz2

$ mv mpfr-2.3.2 gcc-4.4.4/mpfr

Create a build directory for building gcc outside the main source directory.

$ mkdir build

$ cd build

Run the gcc configure script from the build directory. We only need c,c++ and

fortran, so we will enable only them.

$ ../gcc-4.4.4/configure --enable-languages=c,c++,fortran --prefix=/opt/gcc-4.4.4

This is running on an 8 core node, so make with '-j8' to enable compililation on

all cores.

$ make -j8

This takes about 16 minutes.

Install the compiler.

$ sudo make install

Compilation MPICH2

Setup the environment to use the newly installed gcc-4.4.4.

$ export PATH=/opt/gcc-4.4.4/bin:$PATH

$ export LD_LIBRARY_PATH=/opt/gcc-4.4.4/lib:/opt/gcc-4.4.4/lib64:$LD_LIBRARY_PATH

$ export MANPATH=/opt/gcc-4.4.4/man:$MANPATH

Set the fortran environment variables so that MPICH2 enables fortran modules.

$ export F90=gfortran

$ export F77=gfortran

Go back to the build directory and get mpich2 source code.

$ cd $BUILD

$ wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.2.1p1/mpich2-1.2.1p1.tar.gz

$ sha1sum mpich2-1.2.1p1.tar.gz

26bd5620b45bc654aa3945755d11450be63d650c mpich2-1.2.1p1.tar.gz

$ tar -zxf mpich2-1.2.1p1.tar.gz

Configure and make.

$ cd mpich2-1.2.1p1

$ ./configure --prefix=/opt/mpich2-1.2.1p1 2>&1 myconf.log

$ make

This takes about 4 minutes.

And finally Install

$ sudo make install

Setup the environment to use our newly compilied MPICH2 install.

$ export PATH=/opt/mpich2-1.2.1p1/bin:$PATH

$ export LD_LIBRARY_PATH=/opt/mpich2-1.2.1p1/lib:$LD_LIBRARY_PATH

$ export MANPATH=/opt/mpich2-1.2.1p1/share/man:$MANPATH

Do some quick MPICH2 envrioment tests. See the mpich2-1.2.1p1 README[4] for more details.

$ which mpd

/opt/mpich2-1.2.1p1/bin/mpd

$ which mpiexec

/opt/mpich2-1.2.1p1/bin/mpiexec

$ mpd &

$ mpdtrace

compute-x-y

$ mpdallexit

[4] https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.2.1/README.vin

Amber 11

Unpack Amber

$ cd $BUILD

$ sha1sum Amber11.tar.bz2 AmberTools-1.4.tar.bz

fdf3e840da58a8ae4f13cdf7d9ef037c4c69fa67 Amber11.tar.bz2

ef0f9a44c8767ed35f942fd41afc636b67da0389 AmberTools-1.4.tar.bz2

$ tar -jxf Amber11.tar.bz2

$ tar -jxf AmberTools-1.4.tar.bz2

Get the latest bugfixes and patch the source.

$ wget http://ambermd.org/bugfixes/11.0/bugfix.all

$ mv bugfix.all amber11.p1-6.patch

$ cd amber11

$ patch -N -p0 < ../amber11.p1-6.patch patching file src/sander/qm_mm.f patching file src/sander/nonbond_list.f patching file src/pmemd/src/mdin_ctrl_dat.fpp patching file src/pmemd/src/cuda/gpu.cpp patching file src/pmemd/src/cuda/gpu.cpp patching file src/sander/makedepend patching file src/sander/Makefile patching file src/sander/mdread.f

Amber has a poor compile/make setup. It has a bad habit of hardcoding paths

into libraries and executables. This forces the compilation of Amber in the

destination directory. In production environments such as mine, this is not

ideal, but it is something we must deal with. In order to do a cheap workaround

I copy the amber11 tools to my target directory and do some permission magic so

I can compile as a user. Then change permissions back so I can rollout to my

cluster.

Copy the amber11 directory to the target directory. I name it amber11p6 to

indicate that it has upto bugfix 6 applied. And change the permissions to my

user account.

$ sudo cp -r $BUILD/amber11 /opt/amber11p6

$ sudo chown -R 1000:1000 /opt/amber11p6

Now start the Amber 11 configure/make process

$ cd /opt/amber11p6

$ export AMBERHOME=`pwd`

$ cd AmberTools/src

$ ./configure -noX11 gnu

Make AmberTools. Unfortuanely the Makefiles don't play nice with parallel make

options available from the make tool. So we must do it serially. This takes

about 9 minutes.

$ make install

Here comes the fun part. We'll start the testing of AmberTools while compiling

amber11.

Start AmberTools testing. This takes about 1/2 hour.

$ cd ../test

$ make test

While this is testing, open a new window and setup the enviroment. I can do this

because I'm using an 8 core node. Each test is single threaded.

$ cd /opt/amber11p6

$ export PATH=/opt/mpich2-1.2.1p1/bin:/opt/gcc-4.4.4/bin/:$PATH

$ export LD_LIBRARY_PATH=/opt/mpich2-1.2.1p1/lib:/opt/gcc-4.4.4/lib:/opt/gcc-4.4.4/lib64:$LD_LIBRARY_PATH

$ export AMBERHOME=`pwd`

In window 2, compile amber11. This takes a 3 minutes.

$ cd src

$ make install

Now, in Window 2 run the amber11 tests. This takes about 75 minutes.

$ cd ../test

$ make test

Close the second window, we don't need it anymore.

When both serial tests are done, we can move on to

compiling the parallel versions of Amber11. This takes about

5 minutes.

$ cd $AMBERHOME/AmberTools/src

$ ./configure -noX11 -mpi gnu

$ cd ../../src

$ make clean

$ make parallel

Setup a simple Parallel environment to test the MPI environment. We'll try doing

it with 2 and 4 processors. Testing beyond this causes PIMD tests to hang. This

is a known bug and is being worked on as of this writing. ( July 2010 )

2 Cores, about 24 minutes.

$ mpd &

$ export DO_PARALLEL='mpiexec -n 2'

$ cd ../test

$ make test.parellel

4 Cores, about 16 minutes.

$ export DO_PARALLEL='mpiexec -n 4'

$ make test.parellel

All done for now

$ mpdallexit

Taring everything up

This is pretty easy, just use tar.

$ cd /

$ tar -cP --owner=root --group=root -f ~/gcc-4.4.4-x86_64.tar /opt/gcc-4.4.4

$ tar -cP --owner=root --group=root -f ~/mpich2-1.2.1p1-x86_64.tar /opt/mpich2-1.2.1p1

Amber and AmberTools do not need all the compiliation directories to work.

So we only tar the files we need.

$ tar -cP --owner=root --group=root -f ~/amber11p6-x86_64.tar \

 /opt/amber11p6/AmberTools/{bin,include,exe,lib} \

/opt/amber11p6/{bin,include,exe,lib,dat}

Now compress them

$ cd ~

$ gzip -9 {gcc-4.4.4,mpich2-1.2.1p1,amber11p6}-x86_64.tar

And we are done.

UPS Battery Death

Yesterday my APC Back-UPS XS 800 started beeping like mad and turned on the 'Replace Battery' light. "Great!" I thought, "How much is this going to cost me?"

Like the good little consumer I am, I searched the APC site to see what it would cost for a new battery. $90!? Wow. When I purchased the unit 3 years ago, if I remember correctly, it was about $130 on sale. It's not worth replacing the battery at 3/4 the cost of a new unit which is probably more efficient anyway. APC has a little 5 point 'document' that tells you why to use only APC batteries is their units. The only point that has any real weight is the "Voids APC's Equipment Protection Policy". This unit is 3 years old, and the protection is probably out of date anyway. Moreover, I can't find my original receipt for the unit. Still, I'm not going to pay $90 for a battery. ( Pair of batteries really, but more on this later. )

I remember driving down the street some time ago and seeing a sign in a storefront that reads "We do UPS batteries". Considering the store was called Battery emporium or somesuch, it may be worth a try. "What will it cost to replace this?" Quickly the clerk responded, "about $45". They have done this before. "Let's do it"

It turns out the battery for this UPS is just two standard 12v 7.2Ah sealed Lead-Acid batteries with a bracket in between them to keep them apart and to allow space for the connectors. Nothing fancy or proprietary about it except, maybe, the cable connecting the batteries to the UPS.

The clerk cut off the APC label, disconnected the wires, checked a couple numbers and went to the back. He returned with two new batteries, connected them to the bracket, and used good-ol duct tape to hold the batteries together. Some quick connectivity tests, and $48 later I was out of there. That was easy.

Now that the battery is good, I need to reset the battery date on the UPS. In unix, the tool to do this is called 'apcupsd'. I looked around and found it in the opensolaris contib package repository. I tried it, but it kept coredumping on me. No biggie, I just compiled from scratch. I may look into the core dumps later, but for the moment I can't be bothered.

The only thing left to do is to reset the battery date in the UPS. Not difficult.

# apctest
-> Set battery date

and set the date. Done for now.

Now I need to look into the coredumps or make my own package. I really don't like 'make install' putting things on my system.

Onward.

A Sysadmin Workstation

I have been thinking about getting a new workstation for some time now. I want a rig that I can pound on, use for Virutalization, and just plain play with little fear of breaking it. I'm a sysadmin and could care less about graphics performance. I use fluxbox and mutt and xterm for the most part. Surfing is done by Firefox. My requirements were, lots of Memory, lots of 'guts', and not lots of money. I looked around and decided to build my own.

After some minor research, I went down to my local computer store and purchased all the parts. Memory Express has a free service called 'quickmount'. If you purchase all the parts, they will mount your CPU, and memory to the motherboard and update the BIOS if required. It's a nice little service, and this time is saved me much trouble. It turns out the motherboard I was originally given was defective. This little service cost me an hour ( with mobo replacement and the minor paperwork involved) and saved me much headache. Good on them.

Okay enough preamble. Here is my hardware:

ASUS M4A78-E Motherboard
AMD Phenom II X4 955 Quad-Core Black Edition 3.2GHz
Corsair 4GB XMS2-8500CS TWIN2X Dual Channel DDR2 Kit (x2, so 8GB total )
ANTEC New Solutions Series NSK6580 Super Mid Tower w/ EarthWatts 430W
Seagate 500GB Barracuda 7200.12 SATAII ( x2, for mirroring. see more below. )
and a monitor, keyboard and mouse I had laying around.

Assembly
I enjoyed assembling this rig. The case is for the most part will layed out. The only exception is the location of the main power connector to the motherboard. The removable 3.5" HD rack is a little too close to the motherboard, so I needed to put my harddrives at the bottom of the rack instead of at the top as I would have liked. No functional deficiency though.

Setup
For the record, I hate onboard video, it just eats system memory. Luckilly this motherboard has some fancy video 'sideport' memory. I'm not sure what the real purpose of it is ( I read it is used for buffering some Crossfire stuff, but I can't remember where). In the BIOS there is an option to turn off using system memory for video, and just use the sideport memory for it. I first tried to just set the Internal Graphics Mode option to Sideport, but that failed as I forgot to also select the Primary Video Controller option to Internal. Now that I have, everything works fine for me.

I have 1066MHz memory for this computer. But to my surprise the Motherboard defaults to 800MHz when all slots are populated. Hmm... The processor documentation says it fully supports it, but the motherboard defaults to 800MHz for 'stability'. I am currently running memtest86+ in both configurations to see what happens.

Operating System
Open Solaris.
Yup, I'm a *nix guy. Any unix-like operation system works for me. I want to use Open Solaris (OS) for tools I get out of the box, like ZFS, zones and dtrace. I have two 500GB harddrives so I can mirror then using ZFS. I will be doing some testing for performance. There are some different SATA drive access options in the BIOS (namely IDE or AHCI). We'll see what happens.

More to come as it becomes available.