Christopher Jackson's

hacks, thoughts, and utter ramblings.

Creating Your Own Cluster of VM's to Run MPI Applications. Using Ubuntu Server 10.04 LTS.

Notes: I’m using VMWare Fusion 3.1, Ubuntu Server 10.04 LTS (64 bit).
1) Create a new virtual machine and install Ubuntu Server 10.04 LTS (64 bit). I used bridged networking for this machine. This machine will be the base for all other nodes on the cluster.

2) Once the virtual machine is created, log in and run the following commands to install the necessary packages:
1
sudo apt-get install openssh-server openssh-client libcr-dev build-essentials libmpich2-dev mpich2 nfs-common port map
3) Now that we have the packages we need lets set up Hydra (mpd replacement). First we need to set up a hosts file.
1
2
3
cd ~
touch hosts
vi hosts
You will need to put an entry for each node on the cluster on a new line in this file. My setup looks like this:
1
2
3
4
5
6
10.0.1.10 ubu0
10.0.1.11 ubu1
10.0.1.12 ubu2
10.0.1.13 ubu3
10.0.1.14 ubu4
10.0.1.15 ubu5
Next we will have to update the /etc/hosts file. My file looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
127.0.0.1 localhost
10.0.1.10 ubu0
10.0.1.11 ubu1
10.0.1.12 ubu2
10.0.1.13 ubu3
10.0.1.14 ubu4
10.0.1.15 ubu5
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
4) Now that we have hydra set up. We will now set up the shared directory that we will export with NFS at a later step. Since we are using this VM as our base install we will only set up the client side of the NFS share right now. To do this we will add an entry into /etc/fstab. Run the following command:
1
2
3
4
5
sudo mkdir -p /srv/research
sudo mkdir -p /shared/research
sudo chown `whoami`:`whoami` /srv/research
sudo chown `whoami`:`whoami` /shared/research
echo -e "10.0.1.10:/srv/research\t/shared/research\tnfs\tsoft,intr,rsize=8192,wsize=8192 0 0" | sudo tee -a /etc/fstab
5) Now we will need to set up a ssh key pair that the VM’s will use to communicate with one another without using a password. Run the following commands:
1
2
3
4
5
ssh-keygen -t dsa    # Hit enter a few times until you get back to the prompt
cd ~/.ssh
cat id_dsa.pub >> authorized_keys
rm id_dsa.pub
chmod 400 authorized_keys
6) Now we need to make configuration changes so that we can connect as a client to an NFS share. Run the following commands:
1
2
echo "portmap : ALL" | sudo tee -a /etc/hosts.deny
echo "portmap : 10.0.1.10" | sudo tee -a /etc/hosts.allow
7) We have now finished setting up our base install for the VM. We now want to stop the VM and make as many copies as we need. Once you’ve copied the VM’s start all the copies. On each of them you will need to update /etc/hostname with the appropriate name. Once you have done this shutdown each of them. You will issue the following commands on each host to perform the aforementioned tasks:
1
2
sudo vi /etc/hostname
sudo shutdown -h now
8 ) Now we need to set up a NFS export on the root node of the cluster (for me that is ubu0). To do this run the following commands:
1
2
3
4
5
touch /srv/research/test
sudo apt-get install nfs-kernel-server
echo "/srv/research 10.0.1.0/255.255.255.0(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports
sudo exportfs -ra
sudo mount -a
If all went well you should see a file called “test” in /shared/research.
9) Now bring up the client nodes (the copies) and do the following to make sure everything is working.
First from the root node ssh into all the client nodes to establish an ECDSA key fingerprint. Once you have done this run the following command:
1
mpiexec -f ~/hosts -n 3 hostname
Output should look something like this:
1
2
3
ubu0
ubu2
ubu1
10) That is it you now have a cluster that you can run MPI applications on.