Grid5000 Tutorial Expo and Kameleon

This demonstration aims to present Kameleon and Expo, two tools that respectively ease the process of environment construction and experiment deployment and execution. This demonstration targets two common activities in an infrastructure as Grid’5000. The creation and customization of an environment so that it meets the user requirements and its corresponding deployment and experiment management. The demonstration will focus in three primary aspects:

The objective of this first part is to create a deployable Grid5000 image with PAPI and TAU packages. Everything in the user machine. Without a client/server architecture, without features to maintain a living system or other heavy features We are going to follow the following steps:

In the second part we are going to get familiarize with Expo and we are going to write a simple experiment to study the behavior of matrix multiplication algorithm in different machines. These are going to be the steps:

Let’s get started:

Installing kameleon

Kameleon must be run as root and as it constructs a system image into a chroot directory, it may start/stop some services. For the moment this may have unpredictable effects on your running system. So, for the purpose of this tutorial and to experiment Kameleon with no risk, we are going to use a machine in Grid5000. To do so let’s deploy an environment in one site:

 oarsub -I -l/nodes=1 -t deploy 

After deploy the environment:

kadeploy3 -e squeeze-x64-big -f $OAR_NODEFILE

Install git:

aptitude install git

And set some environmental variables and git configuration stuff to have access to some web sites from the outside:

export http_proxy="http://proxy:3128" ; export https_proxy="http://proxy:3128"
git config --global http.proxy http://proxy:3128

Checkout kameleon repository

$ git clone https://gforge.inria.fr/git/kameleon/kameleon.git

Kameleon depends on a non-standard ruby module sessison. Installation tarball can be found in the redist directory. Go to kameleon/redist/ and you will find a tarball session-2.4.0.tgz, untar it and install it like this:

$ sudo ruby install.rb

Install some debian packages needed:

$ aptitude install debootstrap qemu-utils kvm 

Make sure that ruby, rsync, parted, kpartx, losetup, dmsetup, grub-install, awk, sed are installed on your computer, you may also need VBoxManage to generate qemu or VirtualBox images. But for this tutorial we are going to generate only Grid5000 images.

Creating a Debian base System

Simple Example of a recipe file:

global:
  distrib: debian-lenny
  workdir_base: /var/tmp/kameleon/
  distrib_repository: http://ftp.us.debian.org/debian/
  arch: i386
  kernel_arch: "686"
steps:
  - check_deps
  - bootstrap
  - debian/system_config
  - software_install
  - kernel_install
  - strip
  - build_appliance:
    - create_raw_image
    - copy_system_tree
    - install_grub
    - save_as_raw
    - save_as_qcow2
    - clean

Every recipe is mainly divided in two sections:

Steps: Steps are classified as Macrosteps and Microsteps. Macrosteps are YAML files that define Microsteps. Microsteps are simple kameleon commands that expand to pieces of bash script.

We start by creating the following recipe to build a basic debian system.

#### Basic Debian Kameleon recipe ###
global:
  workdir_base: /tmp/kameleon

# Debian specific
  distrib: debian
  debian_version_name: squeeze
  distrib_repository: http://ftp.fr.debian.org/debian/
  output_environment_file_system_type: ext4

  include_dir: scripts
# Architecture
  arch: amd64
  kernel_arch: "amd64"
   
# Network configuration
  network_hostname: kameleon
 
steps:
# Checking availability of tools used during the process
  - debian_check_deps
  - check_deps:
    - rsync
    - building_kvm_images
# Creation of the debian base system using debootstrap 
  - bootstrap
# Some system coniguration
  - system_config:
    - fstab
  - root_passwd
  - mount_proc
# Grid5000 network management
  - g5k/g5k-update-host-name
  - kernel_install
  - strip
  - umount_proc
# Building the appliance
  - build_appliance:
    - clean_udev
    - create_raw_image
    - create_nbd_device
    - mkfs
    - mount_image
    - copy_system_tree
    - umount_image
    - save_as_raw 
# Creating Grid5000 deployable image
  - save_as_g5k
  - clean

To run kameleon, run as root (because we need to create a chroot)

./kameleon recipes/debian/squeeze_amd64_g5k.yaml

If we take a look to our working directory defined by the variable workdir_base, you will find a directory like this:

/tmp/kameleon/2012-11-22-08-43-56/

We will see the following structure:

/tmp/kameleon/2012-11-22-08-43-56$ ls
debian.g5k.dsc  debian.g5k.tgz  debian.raw  image.raw  kameleon_env

Therefore:

So, we can use the debian.g5k.tgz and debian.g5k.dsc and transfer them to any site in Grid5000 and deploy a base debian system. But instead of that let’s add more software to this image.

Adding PAPI and TAU packages to the initial image

If the software we want to add is available in the debian repository we just create in the section global: a variable called extra_packages, which looks something like this:

extra_packages: "vim less bzip2 openssh-server rsync gnupg locales debian-keyring console-tools mingetty gcc g++ make patch build-essential"

And once this has been done, we need to add a step:

- software_install:
  - extra_packages

The step software install execute the debian application manager to install the packages defined in the variable extra_packages

If the software is not available in the debian repositories or you have to compile it form source because you need a special configuration, you have to create a step which is a YAML file with Macrosteps and Microsteps. We are going to write one for installing PAPI in the base image. This is how it looks like:

## Macro steps for installing PAPI 
### http://icl.cs.utk.edu/projects/papi/downloads/papi-4.4.0.tar.gz
papi:
  - get_unpack:
    - exec_chroot: bash -c "cd /root/ ; wget $$papi_repository/papi-$$papi_version.tar.gz"
    - exec_chroot: bash -c "cd /root/ ; tar -xzf papi-$$papi_version.tar.gz -C ."
    - exec_chroot: bash -c "cd /root/ ; rm papi-$$papi_version.tar.gz"
    
  - papi_install:
    - exec_chroot: bash -c "cd /root/papi-$$papi_version/src/ ; ./configure"
    - exec_chroot: bash -c "cd /root/papi-$$papi_version/src/; make; make install"

Here we keep it simple, we just execute commands but it can do more things such as writing into files.

A step for installing TAU.

## Macro steps for installing TAU profiling tool
tau:
  - get_unpack:
    - exec_chroot: bash -c "cd /root/ ; wget $$pdt_repository/pdt.tgz"
    - exec_chroot: bash -c "cd /root/ ; wget $$tau_repository/tau.tgz"
    - exec_chroot: bash -c "cd /root/ ; tar -xzf pdt.tgz -C ."
    - exec_chroot: bash -c "cd /root/ ; tar -xzf tau.tgz -C ."
    - exec_chroot: bash -c "cd /root/ ; rm pdt.tgz"
    - exec_chroot: bash -c "cd /root/ ; rm tau.tgz"
    
  - pdt_install:
    - exec_chroot: bash -c "cd /root/pdtoolkit-$$pdt_version ; ./configure "
    - exec_chroot: bash -c "cd /root/pdtoolkit-$$pdt_version; make clean install"
    
  - tau_install_with_mpi:
    - exec_chroot: bash -c "cd /root/tau-$$tau_version ; ./configure -pdt=/root/pdtoolkit-$$pdt_version -mpiinc=/usr/include/ -mpilib=/usr/lib/"
    - exec_chroot: bash -c "cd /root/tau-$$tau_version; make clean install"
    
  - tau_install_with_papi:
    - exec_chroot: bash -c "cd /root/tau-$$tau_version ; ./configure -pdt=/root/pdtoolkit-$$pdt_version -papi=/root/papi-4.4.0/ "
    - exec_chroot: bash -c "cd /root/tau-$$tau_version; make clean install"

And Finally we create a recipe with both steps.

This is how it looks like the global section for this final recipe.

 global:
   workdir_base: /tmp/kameleon
     # Debian specific
     distrib: debian
     debian_version_name: squeeze
     distrib_repository: http://ftp.fr.debian.org/debian/
     output_environment_file_system_type: ext4
     #
     include_dir: scripts
     # Architecture
     arch: amd64
     kernel_arch: "amd64"
     #
     extra_packages: "vim less bzip2 openssh-server rsync gnupg locales debian-keyring console-tools mingetty gcc g++ make patch build-essential"
     # Network configuration
     network_hostname: kameleon
     # 
     checkpoint_file: /some_directory/debian_base.tgz
     #
     #PDT variable definition
     pdt_repository: http://tau.uoregon.edu/
     pdt_version: 3.18.1 
     # TAU variable definition
     tau_repository: http://tau.uoregon.edu/
     tau_version: 2.22-p1
     #PAPI variable definition
   papi_repository: http://icl.cs.utk.edu/projects/papi/downloads/
   papi_version: 4.4.0 

For this particular example, since everything is going to be executed inside Grid5000, we have to change the web accesses to `http://public.luxembourg.grid5000.fr/~cruizsanabria/

And the steps section will look like this:

steps:
   - checkpoint_resume
   - software_install:
     - extra_packages
   - PAPI/papi
   - TAU/tau:
   - get_unpack
   - pdt_install
   - tau_install_with_papi  
   - umount_proc
   
   - build_appliance:
     - clean_udev
     - create_raw_image
     - create_nbd_device
     - mkfs
     - mount_image
     - copy_system_tree
     - umount_image
     - save_as_raw 
   - save_as_g5k
   - clean

Run Kameleon with this new recipe. An environment with PAPI and TAU ready to use will be created. It’s the time for Expo.

Installing Expo

We have to install it first, please go to Expo Getting Started, upon installation, you have to copy the final image and the description file produce by Kameleon, to one site of Grid5000 and do the setup in order to make the image available form a url inside Grid5000, to see how to do this Kadeploy tutorial

Playing with Expo commands

Execute the Expo Console:

$ ./expo
ruby 1.8.7 (2010-08-16 patchlevel 302) [x86_64-linux]
Welcome to Expo Interactive Mode
All the libraries have been loaded
Opening Experiment
Preparing resource container $all
Connecting to the Grid5000 API
Expo Console > 

And then create a reservation object:

Expo Console > reserv=ExpoEngine::new(@connection)

Assign the environment url for your image to the reservation:

Expo Console > reserv.environment="http://public.grenoble.grid5000.fr/~cruizsanabria/environments/debian_papi_tau_g5k.dsc"

As well as others parameters:

Expo Console > reserv.site=["grenoble","lille","toulouse","reims"]
=> ["grenoble"]
Expo Console > reserv.walltime=7200
=> 7200

We are going to reserve a machine in several sites for this experiment, so run the reservation:

2012-11-26 21:31:54 +0100 [DEBUG] In /home/cruizsanabria/Repositories/git/expo/bin
2012-11-26 21:31:54 +0100 [INFO] [ Expo Engine Grid5000 API ] Asking for Resources
2012-11-26 21:31:54 +0100 [INFO] [ Expo Engine Grid5000 API ] Number of nodes to reserve in site: bordeaux => nodes=2
2012-11-26 21:31:54 +0100 [INFO] [bordeaux] Launching job [no-submit=false]...
2012-11-26 21:31:54 +0100 [INFO] [ Expo Engine Grid5000 API ] Number of nodes to reserve in site: lille => nodes=2
2012-11-26 21:31:54 +0100 [INFO] [lille] Launching job [no-submit=false]...
2012-11-26 21:31:54 +0100 [INFO] [ Expo Engine Grid5000 API ] Number of nodes to reserve in site: sophia => nodes=2
2012-11-26 21:31:54 +0100 [INFO] [sophia] Launching job [no-submit=false]...
2012-11-26 21:31:54 +0100 [INFO] [ Expo Engine Grid5000 API ] Number of nodes to reserve in site: toulouse => nodes=2
2012-11-26 21:31:54 +0100 [INFO] [toulouse] Launching job [no-submit=false]...
2012-11-26 21:31:59 +0100 [INFO] [sophia] Got the following job: 512842
2012-11-26 21:31:59 +0100 [INFO] [sophia] Waiting for state=running for job #512842 (expected start time="unknown")...
2012-11-26 21:31:59 +0100 [INFO] [toulouse] Got the following job: 380230
2012-11-26 21:31:59 +0100 [INFO] [toulouse] Waiting for state=running for job #380230 (expected start time="Mon Nov 26 21:31:59 +0100 2012")...
2012-11-26 21:32:03 +0100 [INFO] [bordeaux] Got the following job: 949246
2012-11-26 21:32:03 +0100 [INFO] [bordeaux] Waiting for state=running for job #949246 (expected start time="Mon Nov 26 21:32:04 +0100 2012")...
2012-11-26 21:32:09 +0100 [INFO] [sophia] Job is running:
2012-11-26 21:32:10 +0100 [INFO] [toulouse] Job is running:
2012-11-26 21:32:13 +0100 [INFO] [lille] Got the following job: 1189952
2012-11-26 21:32:13 +0100 [INFO] [lille] Waiting for state=running for job #1189952 (expected start time="Mon Nov 26 21:32:12 +0100 2012")...
2012-11-26 21:32:14 +0100 [INFO] [bordeaux] Job is running:
2012-11-26 21:32:23 +0100 [INFO] [lille] Job is running:
2012-11-26 21:32:23 +0100 [INFO] [ Expo Engine Grid5000 API ] Total Time Spent waiting for resources 29.465673 secs
2012-11-26 21:32:23 +0100 [INFO] [ Expo Engine Grid5000 API ] Deploying Environments
2012-11-26 21:32:23 +0100 [INFO] [ Expo Engine Grid5000 API ] Deploying 2 machines in site: sophia
2012-11-26 21:32:23 +0100 [INFO] [ Expo Engine Grid5000 API ] Deploying 2 machines in site: toulouse
2012-11-26 21:32:23 +0100 [INFO] [sophia] Launching deployment [no-deploy=false]...
2012-11-26 21:32:23 +0100 [INFO] [ Expo Engine Grid5000 API ] Deploying 2 machines in site: bordeaux
2012-11-26 21:32:23 +0100 [INFO] [toulouse] Launching deployment [no-deploy=false]...
2012-11-26 21:32:23 +0100 [INFO] [ Expo Engine Grid5000 API ] Deploying 2 machines in site: lille
2012-11-26 21:32:23 +0100 [INFO] [bordeaux] Launching deployment [no-deploy=false]...
2012-11-26 21:32:23 +0100 [INFO] [lille] Launching deployment [no-deploy=false]...

After the deployment, we can start using expo to interact with the machines, if we take a look at the variable $all, we’ll see something like this:

Expo Console > $all
=> #<Expo::ResourceSet:0x7fd86f908d60
@properties={},
@resource_files={},
@resources=
[#<Expo::ResourceSet:0x7fd86d01a048
  @properties=
  {:site=>"grenoble",
  :alias=>"edel",
  :gateway=>"frontend.grenoble.grid5000.fr",
  :name=>"ExpoEngine",
  :id=>1393107},
  @resource_files={},
  @resources=
    [#<Expo::Resource:0x7fd86d019df0
    @properties=
      {:site=>"grenoble",
      :gateway=>"frontend.grenoble.grid5000.fr",
      :name=>"edel-25.grenoble.grid5000.fr"},
    @type=:node>],
    @type=:resource_set>],
@type=:resource_set>

This is a ResourceSet object, which abstracts away the complexity when dealing with several nodes. It maps the Grid5000 topology and allows Taktuk to efficiently execute commands over these nodes taking advantage of their topology.

A simple command to test:

Expo Console > task $all.first.name, "hostname"
edel-25.grenoble.grid5000.fr
2012-11-22 14:03:31 +0100 [INFO] [ Sequential Task:3 ]  [ Executed ]   hostname  
2012-11-22 14:03:31 +0100 [INFO] [ Sequential Task:3 ]  [ On Node ]  edel-25.grenoble.grid5000.fr  
2012-11-22 14:03:31 +0100 [INFO] [ Sequential Task:3 ]  [ Elapsed Time ] 0.00247 secs 
=> [3,
[{"stdout"=>"edel-25.grenoble.grid5000.fr\n",
  "end_time"=>Thu Nov 22 14:03:30 +0100 2012,
  "host_name"=>"edel-25.grenoble.grid5000.fr",
  "stderr"=>"",
  "rank"=>"1",
  "status"=>"0",
  "command_line"=>"cd . ; hostname ",
  "start_time"=>Thu Nov 22 14:03:30 +0100 2012}]]

Just to get used to the syntax, we can get and treat the output of commands

Expo Console > id,res=task $all.first, "ls"
papi-4.4.0
pdtoolkit-3.18.1
postinst.log
tau-2.22-p1
2012-11-22 14:05:22 +0100 [INFO] [ Sequential Task:5 ]  [ Executed ]   ls  
2012-11-22 14:05:22 +0100 [INFO] [ Sequential Task:5 ]  [ On Node ]  edel-25.grenoble.grid5000.fr  
2012-11-22 14:05:22 +0100 [INFO] [ Sequential Task:5 ]  [ Elapsed Time ] 0.00313 secs 
=> [5,
[{"stdout"=>"papi-4.4.0\npdtoolkit-3.18.1\npostinst.log\ntau-2.22-p1\n",
"end_time"=>Thu Nov 22 14:05:22 +0100 2012,
"host_name"=>"edel-25.grenoble.grid5000.fr",
"stderr"=>"",
"rank"=>"1",
"status"=>"0",
"command_line"=>"cd . ; ls ",
"start_time"=>Thu Nov 22 14:05:22 +0100 2012}]

The output was stored in the res variable, and then its contents can be checked doing:

Expo Console > res[0]["stdout"]
=> "papi-4.4.0"
=> "pdtoolkit-3.18.1"
=> "postinst.log"
=> "tau-2.22-p1"

Expo hooks in pry make the visualization of outputs easier recognizing that the string is the result of stdout and helping the readability.

Now let’s execute something more serious. Let’s compile one of the examples of TAU (This could have been done in the kameleon step but it was done here as an illustrative example)

Expo Console > id,res=task $all.first, "make -C tau-2.22-p1/examples/papi/" 
2012-11-22 14:14:13 +0100 [INFO] [ Sequential Task:6 ]  [ Executed ]   make  
2012-11-22 14:14:13 +0100 [INFO] [ Sequential Task:6 ]  [ On Node ]  edel-25.grenoble.grid5000.fr  
2012-11-22 14:14:13 +0100 [INFO] [ Sequential Task:6 ]  [ Elapsed Time ] 0.29028 secs 
=> [6,
[{"stdout"=>
"make: Entering directory `/root/tau-2.22-p1/examples/papi'\ng++   -I/root/tau-2.22-p1/include -DPROFILING_ON    simple.o -o simple  -L/root/tau-2.22-p1/x86_64/lib -lpapi",
"end_time"=>Thu Nov 22 14:14:11 +0100 2012,
"host_name"=>"edel-25.grenoble.grid5000.fr",
"stderr"=>"",
"rank"=>"1",
"status"=>"0",
"command_line"=>"cd . ; make -C tau-2.22-p1/examples/papi/",
"start_time"=>Thu Nov 22 14:14:11 +0100 2012}]]

Expo offers other abstraction called Task, which is implemented as a Ruby Object. This Class makes clearer the command execution through the experiment description.

To create a Task object do the following:

task_1=Task::new("/root/tau-2.22-p1/examples/papi/simple",$all.first,"Test 1")

Here we created a task that is going to execute the example we have just compiled. This task is going to be executed on the first machine of the resource set $all. This example shows the effect of using a strip mining optimization with the matrix multiply algorithm. Although both the regular multiply algorithm and the strip mining algorithm have the same number of floating point operations, they have dramatically different cache behaviors. Using PAPI we can see the exact number of floating point operations and secondary data cache misses as we will see. TAU is the profiler which offers a nice interface to the PAPI counters.

To specify the counter we want to use in PAPI, we have to specify it by defining an environmental variable. We should do something like this in a shell.

export TAU_METRICS=TIME:PAPI_L2_DCM

For measuring the Level 2 data cache misses. By default the profiling output is generated in a directory located in the same directory where the application is located. This can be changed by setting the environmental variable PROFILEDIR.

export PROFILEDIR=/root/

In order to do that with Expo we can specify the environment variables :

Expo Console > task_1.env_var="TAU_METRICS=TIME:PAPI_L2_DCM PROFILEDIR=/root/"

Then we can proceed with the execution:

Expo Console > task_1.execute
012-11-23 14:06:37 +0100 [INFO] [ (Test 1): Parallel Task:21 ]  [ Executed ]   ./simple  
2012-11-23 14:06:37 +0100 [INFO] [ (Test 1): Parallel Task:21 ]  [ On Node ]  chinqchint-43.lille.grid5000.fr  
2012-11-23 14:06:37 +0100 [INFO] [ (Test 1): Parallel Task:21 ]  [ Elapsed Time ] 0.24925 secs 
=> [21,
[{"stdout"=>"",
"end_time"=>Fri Nov 23 14:06:35 +0100 2012,
"host_name"=>"chinqchint-43.lille.grid5000.fr",
"stderr"=>"",
"rank"=>"2",
"status"=>"0",
"command_line"=>
"cd /root/tau-2.22-p1/examples/papi ; export TAU_METRICS=TIME:PAPI_L2_DCM PROFILEDIR=/root/ ; ./simple ",
"start_time"=>Fri Nov 23 14:06:35 +0100 2012}]]

This will produce two directories called MULTI__TIME and MULTI__PAPI_L2_CDM respectively. These two directories contain values of the performance counters. To see the content of these directories we need a tool called pprof. So, we create a task to execute this command and given that we dont have it in the PATH, we should specify it to Expo.

Expo Console > analysis=Task::new("pprof -f MULTI__TIME/profile.0.0.0",$all,"Analysis")
Expo Console > analysis.env_var="PATH=/root/tau-2.22-p1/x86_64/bin/"

And run it:

Expo Console > id,res=analysis.execute

Reading Profile files in MULTI__PAPI_L2_DCM/profile.0.0.0.*

MULTI__PAPI_L2_DCM/profile.0.0.0
---------------------------------------------------------------------------------------
Time   Exclusive   Inclusive       #Call      #Subrs Count/Call Name
  counts total counts                            
---------------------------------------------------------------------------------------
100.0           6          57           1           1         57 main() int (int, char **)
89.5          21          51           1           2         51 multiply void (void)
36.8          21          21           1           0         21 multiply-regular void (void)
15.8           9           9           1           0          9 multiply-with-strip-mining-optimization void (void)

2012-11-23 14:06:27 +0100 [INFO] [ (Analysis): Parallel Task:20 ]  [ Executed ]   pprof  
2012-11-23 14:06:27 +0100 [INFO] [ (Analysis): Parallel Task:20 ]  [ On Node ]  chinqchint-43.lille.grid5000.fr  
2012-11-23 14:06:27 +0100 [INFO] [ (Analysis): Parallel Task:20 ]  [ Elapsed Time ] 0.00471 secs 
=> [20,
[{"stdout"=>
"Reading Profile files in MULTI__PAPI_L2_DCM/profile.0.0.0.*--\n%Time   Exclusive   Inclusive       #Call      #Subrs Count/Call Name
  \n           counts total counts                            
  ---------------------------------------------------------------------------------------
  100.0           6          57           1           1         57 main() int (int, char **)
  89.5          21          51           1           2         51 multiply void (void)
  36.8          21          21           1           0         21 multiply-regular void (void)
  15.8           9           9           1           0          9 multiply-with-strip-mining-optimization void (void)",
"end_time"=>Fri Nov 23 14:06:26 +0100 2012,
"host_name"=>"chinqchint-43.lille.grid5000.fr",
"stderr"=>"",
"rank"=>"2",
"status"=>"0",
"command_line"=>
"cd . ; export PATH=/root/tau-2.22-p1/x86_64/bin/ ; pprof -f MULTI__PAPI_L2_DCM/profile.0.0.0",
"start_time"=>Fri Nov 23 14:06:26 +0100 2012}]]

To know how much time it took, we execute:

Expo Console > res.duration
=> 0.00471

Now let’s do it in several machines, for compiling the code we can use ptask, which execute parallel tasks

Expo Console > id,res=ptask $all, "make -C tau-2.22-p1/examples/papi/"

The syntax is almost the same as task but instead of passing one machine, we pass the complete set of machines. After executing we have each of the results in the variable res, this is the log produced:

2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Executed ]   make  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ On Node ]  genepi-12.grenoble.grid5000.fr  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Elapsed Time ] 1.48813 secs 
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Executed ]   make  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ On Node ]  pastel-53.toulouse.grid5000.fr  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Elapsed Time ] 1.99214 secs 
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Executed ]   make  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ On Node ]  pastel-105.toulouse.grid5000.fr  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Elapsed Time ] 1.96169 secs 
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Executed ]   make  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ On Node ]  pastel-107.toulouse.grid5000.fr  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Elapsed Time ] 1.97811 secs 
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Executed ]   make  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ On Node ]  pastel-109.toulouse.grid5000.fr  
2012-11-24 13:14:24 +0100 [INFO] [ Parallel Task:1 ]  [ Elapsed Time ] 2.07666 secs 

We can ask for the duration of a given compilation in a certain machine:

Expo Console > res[1].duration
=> 1.99214

And also the mean duration

Expo Console > res.mean_duration
=> 1.899346

For creating the task we dont change at all.

Expo Console > task_1=Task::new("/root/tau-2.22-p1/examples/papi/simple",$all,"Test 1")
Expo Console > task_1.env_var="TAU_METRICS=TIME:PAPI_L2_DCM PROFILEDIR=/root/"
Expo Console > task_1.execute

We create also the analysis task.

Expo Console > analysis=Task::new("pprof -f MULTI__TIME/profile.0.0.0",$all,"Analysis")
Expo Console > analysis.env_var="PATH=/root/tau-2.22-p1/x86_64/bin/"
Expo Console > id,res=analysis.execute

We can get the results of the execution and check each output with the variable res like this:

Expo Console > res[1]['stdout']

=> "Reading Profile files in MULTI__TIME/profile.0.0.0.*"
=> ""
=> "MULTI__TIME/profile.0.0.0"
=> "---------------------------------------------------------------------------------------"
=> "%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name"
=> "              msec   total msec                          usec/call "
=> "---------------------------------------------------------------------------------------"
=> "100.0        0.067           45           1           1      45963 main() int (int, char **)"
=> " 99.9            1           45           1           2      45896 multiply void (void)"
=> " 49.2           22           22           1           0      22626 multiply-with-strip-mining-optimization void (void)"
=> " 48.4           22           22           1           0      22248 multiply-regular void (void)"
Expo Console > res[3]['stdout']
=> "Reading Profile files in MULTI__TIME/profile.0.0.0.*"
=> ""
=> "MULTI__TIME/profile.0.0.0"
=> "---------------------------------------------------------------------------------------"
=> "%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name"
=> "              msec   total msec                          usec/call "
=> "---------------------------------------------------------------------------------------"
=> "100.0        0.069           45           1           1      45844 main() int (int, char **)"
=> " 99.8        0.987           45           1           2      45775 multiply void (void)"
=> " 49.2           22           22           1           0      22553 multiply-with-strip-mining-optimization void (void)"
=> " 48.5           22           22           1           0      22235 multiply-regular void (void)"
Expo Console > res[4]['stdout']
=> "Reading Profile files in MULTI__TIME/profile.0.0.0.*"
=> ""
=> "MULTI__TIME/profile.0.0.0"
=> "---------------------------------------------------------------------------------------"
=> "%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name"
=> "              msec   total msec                          usec/call "
=> "---------------------------------------------------------------------------------------"
=> "100.0        0.071           45           1           1      45884 main() int (int, char **)"
=> " 99.8            1           45           1           2      45813 multiply void (void)"
=> " 49.1           22           22           1           0      22544 multiply-with-strip-mining-optimization void (void)"
=> " 48.5           22           22           1           0      22246 multiply-regular void (void)"

When reserving resources from different sites. We got different clusters and we can access to these nodes easily, if for example the set of resources includes nodes form genepi cluster we can execute the parallel task like this:

Expo Console > id, res = ptask $all["genepi"], "make -C tau-2.22-p1/examples/papi/"

Writing an experiment description file

Let’s first download the source code for the example on all the machines.

Expo Console > id,res = ptask $all, "wget http://public.luxembourg.grid5000.fr/~cruizsanabria/g5k_school.tar" 

Expo Console > res[0]['stderr']
=> "--2012-11-27 07:49:12--  http://public.luxembourg.grid5000.fr/~cruizsanabria/g5k_school.tar"
=> "Resolving public.luxembourg.grid5000.fr... 172.16.191.60"
=> "Connecting to public.luxembourg.grid5000.fr|172.16.191.60|:80... connected."
=> "HTTP request sent, awaiting response... 200 OK"
=> "Length: 20480 (20K) [application/x-tar]"
=> "Saving to: `g5k_school.tar'"
=> ""
=> "     0K .......... ..........                                 100%  365K=0.05s"
=> ""
=> "2012-11-27 07:49:12 (365 KB/s) - `g5k_school.tar' saved [20480/20480]"

Untar it :

Expo Console > id,res = ptask $all, "tar -xvf g5k_school.tar"

And compile it:

Expo Console > id,res = ptask $all, "make -C ~/Grid5000_school/"
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Executed ]   make  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ On Node ]  stremi-7.reims.grid5000.fr  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Elapsed Time ] 1.5425 secs 
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Executed ]   make  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ On Node ]  paradent-2.rennes.grid5000.fr  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Elapsed Time ] 1.73446 secs 
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Executed ]   make  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ On Node ]  chinqchint-9.lille.grid5000.fr  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Elapsed Time ] 1.9158 secs 
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Executed ]   make  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ On Node ]  pastel-97.toulouse.grid5000.fr  
2012-11-27 08:50:48 +0100 [INFO] [ Parallel Task:3 ]  [ Elapsed Time ] 2.02048 secs 

Run it with a parameter:

Expo Console > id,res = ptask $all, "~/Grid5000_school/matrix_mul 128"

The following example could take approximately 8 minutes so try with shorter examples

Expo Console > id,res = ptask $all, "~/Grid5000_school/matrix_mul 2024"
size:2024
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ On Node ]  stremi-7.reims.grid5000.fr  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Elapsed Time ] 455.76539 secs 
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ On Node ]  paradent-2.rennes.grid5000.fr  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Elapsed Time ] 327.41454 secs 
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ On Node ]  chinqchint-9.lille.grid5000.fr  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Elapsed Time ] 294.32948 secs 
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ On Node ]  pastel-97.toulouse.grid5000.fr  
2012-11-27 09:01:22 +0100 [INFO] [ Parallel Task:7 ]  [ Elapsed Time ] 418.34645 secs 
=> [7,
[{"stdout"=>"size:2024\n",
"end_time"=>Tue Nov 27 09:01:21 +0100 2012,
"host_name"=>"stremi-7.reims.grid5000.fr",
"stderr"=>"",
"rank"=>"6",
"status"=>"0",
"command_line"=>"cd ~/Grid5000_school ; ./matrix_mul 2024",
"start_time"=>Tue Nov 27 08:53:46 +0100 2012},
{"stdout"=>"size:2024\n",
"end_time"=>Tue Nov 27 08:59:12 +0100 2012,
"host_name"=>"paradent-2.rennes.grid5000.fr",
"stderr"=>"",
"rank"=>"8",
"status"=>"0",
"command_line"=>"cd ~/Grid5000_school ; ./matrix_mul 2024",
"start_time"=>Tue Nov 27 08:53:45 +0100 2012},
{"stdout"=>"size:2024\n",
"end_time"=>Tue Nov 27 08:58:40 +0100 2012,
"host_name"=>"chinqchint-9.lille.grid5000.fr",
"stderr"=>"",
"rank"=>"2",
"status"=>"0",
"command_line"=>"cd ~/Grid5000_school ; ./matrix_mul 2024",
"start_time"=>Tue Nov 27 08:53:45 +0100 2012},
{"stdout"=>"size:2024\n",
"end_time"=>Tue Nov 27 09:00:44 +0100 2012,
"host_name"=>"pastel-97.toulouse.grid5000.fr",
"stderr"=>"",
"rank"=>"4",
"status"=>"0",
"command_line"=>"cd ~/Grid5000_school ; ./matrix_mul 2024",
"start_time"=>Tue Nov 27 08:53:46 +0100 2012}]]

Expo Console > res.each{ |t| puts t.duration }
455.76539
327.41454
294.32948
418.34645
Expo Console > res.each{ |t| puts " #{t["host_name"]}\t#{t.duration}" }
stremi-7.reims.grid5000.fr:     455.76539
paradent-2.rennes.grid5000.fr:  327.41454
chinqchint-9.lille.grid5000.fr: 294.32948
pastel-97.toulouse.grid5000.fr: 418.34645

Experiment: measure the behavior of executing two matrix multiplication algorithms with different parameters over different machines. We are going to measure time and L2 Cache misses.

Let’s write an experiment description file for the experiment mentioned above. First to measure the run-times:

You can use shell commands directly from this console, so let’s execute a text editor to create our Experiment description file.

Expo Console > .emacs experiment_test.rb

This will open emacs, now we can write the following

File.open("run_times.txt","w+") do |f|

  f.puts "Hostname\tSize\tRun_time"
  [128,256,512].each{ |size|
  
  id,res = ptask $all, "~/Grid5000_school/matrix_mul #{size}"
  ## putting the results in the file
  res.each{ |result|
          f.puts "#{ result["host_name"]}\t #{size}\t#{result.duration }"
      }
  }
end

And let’s load it with:

Expo Console > load_exp("experiment_test.rb")

This will run the experiment :), and we will see something like this:

Expo Console > load_exp("experiment_test.rb")
.
.
.
size:1024
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ On Node ]  stremi-7.reims.grid5000.fr  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Elapsed Time ] 54.40883 secs 
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ On Node ]  paradent-2.rennes.grid5000.fr  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Elapsed Time ] 39.94993 secs 
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ On Node ]  chinqchint-9.lille.grid5000.fr  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Elapsed Time ] 35.74112 secs 
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Executed ]   ./matrix_mul  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ On Node ]  pastel-97.toulouse.grid5000.fr  
2012-11-27 09:25:24 +0100 [INFO] [ Parallel Task:17 ]  [ Elapsed Time ] 52.35911 secs 
2012-11-27 09:25:24 +0100 [INFO] [ Expo Experiment ] The experiment executed in 70.135224 secs 

Lets put everything together and write an experiment to get the cache misses. So we need to execute the program and execute the profiler to get the output.

Expo Console > .emacs experiment_cache.rb

And write this.

File.open("cache_behavior.txt","w+") do | f|

f.puts "Cache Behaviour for two different algorithms in different architectures"
f.puts "Hostname\tSIZE\t%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name"

task_m=Task::new(" ",$all,"Testing matrix")
task_m.env_var="TAU_METRICS=TIME:PAPI_L2_DCM"

analysis=Task::new("pprof -f ~/Grid5000_school/MULTI__TIME/profile.0.0.0",$all,"Analysis")
analysis.env_var="PATH=/root/tau-2.22-p1/x86_64/bin/"

[128,256,512].each{ |size|

   task_m.command="~/Grid5000_school/matrix_mul #{size}"
   task_m.execute
   id,res = analysis.execute
   res.each{ |result|
     output= result['stdout'].split("\n")
     f.puts "#{result["host_name"]}\t#{size}  "+output.pop
     f.puts "#{result["host_name"]}\t#{size}  "+output.pop
   }    }    end

And let’s load it with:

Expo Console > load_exp("experiment_cache.rb")

This is the output

.
.
.
2012-11-27 11:25:10 +0100 [INFO] [ (Testing matrix): Parallel Task:21 ]  [ Executed ]   ./matrix_mul  
2012-11-27 11:25:10 +0100 [INFO] [ (Testing matrix): Parallel Task:21 ]  [ On Node ]  pastel-92.toulouse.grid5000.fr  
2012-11-27 11:25:10 +0100 [INFO] [ (Testing matrix): Parallel Task:21 ]  [ Elapsed Time ] 6.50205 secs 
Reading Profile files in /root/Grid5000_school/MULTI__TIME/profile.0.0.0.*

/root/Grid5000_school/MULTI__TIME/profile.0.0.0
---------------------------------------------------------------------------------------
Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
               msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.084        5,661           1           1    5661167 main() int (int, char **)
100.0           10        5,661           1           2    5661083 multiply void (void)
60.5        3,427        3,427           1           0    3427061 multiply-regular void (void)
39.3        2,223        2,223           1           0    2223095 multiply-with-strip-mining-optimization void (void)
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Executed ]   pprof  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ On Node ]  stremi-29.reims.grid5000.fr  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Elapsed Time ] 0.0065 secs 
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Executed ]   pprof  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ On Node ]  parapluie-23.rennes.grid5000.fr  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Elapsed Time ] 0.00648 secs 
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Executed ]   pprof  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ On Node ]  chinqchint-9.lille.grid5000.fr  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Elapsed Time ] 0.00467 secs 
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Executed ]   pprof  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ On Node ]  pastel-92.toulouse.grid5000.fr  
2012-11-27 11:25:13 +0100 [INFO] [ (Analysis): Parallel Task:22 ]  [ Elapsed Time ] 0.00568 secs 
2012-11-27 11:25:13 +0100 [INFO] [ Expo Experiment ] The experiment took 23.185795 secs 
=> nil
Expo Console > 

And if you look at the file written:

Expo Console > .cat cache_behavior.txt
Cache Behaviour for two different algorithms in different architectures
Hostname  SIZE %Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
stremi-29.reims.grid5000.fr 128 47.8           34           34           1           0      34268 multiply-with-strip-mining-optimization void (void)
stremi-29.reims.grid5000.fr 128 51.1           36           36           1           0      36653 multiply-regular void (void)
parapluie-23.rennes.grid5000.fr 128 47.8           34           34           1           0      34292 multiply-with-strip-mining-optimization void (void)
parapluie-23.rennes.grid5000.fr 128 51.1           36           36           1           0      36661 multiply-regular void (void)
chinqchint-9.lille.grid5000.fr  128 48.4           18           18           1           0      18003 multiply-with-strip-mining-optimization void (void)
chinqchint-9.lille.grid5000.fr  128 49.8           18           18           1           0      18550 multiply-regular void (void)
pastel-92.toulouse.grid5000.fr  128 48.7           39           39           1           0      39290 multiply-with-strip-mining-optimization void (void)
pastel-92.toulouse.grid5000.fr  128 50.4           40           40           1           0      40667 multiply-regular void (void)
stremi-29.reims.grid5000.fr 256 49.5          273          273           1           0     273841 multiply-with-strip-mining-optimization void (void)
stremi-29.reims.grid5000.fr 256 50.0          276          276           1           0     276797 multiply-regular void (void)
parapluie-23.rennes.grid5000.fr 256 49.6          273          273           1           0     273420 multiply-with-strip-mining-optimization void (void)
parapluie-23.rennes.grid5000.fr 256 49.9          275          275           1           0     275475 multiply-regular void (void)
chinqchint-9.lille.grid5000.fr  256 48.3          143          143           1           0     143897 multiply-with-strip-mining-optimization void (void)
chinqchint-9.lille.grid5000.fr  256 51.1          152          152           1           0     152233 multiply-regular void (void)
pastel-92.toulouse.grid5000.fr  256 49.6          316          316           1           0     316811 multiply-with-strip-mining-optimization void (void)
pastel-92.toulouse.grid5000.fr  256 50.0          319          319           1           0     319057 multiply-regular void (void)
stremi-29.reims.grid5000.fr 512 39.3        2,223        2,223           1           0    2223095 multiply-with-strip-mining-optimization void (void)
stremi-29.reims.grid5000.fr 512 60.5        3,427        3,427           1           0    3427061 multiply-regular void (void)
parapluie-23.rennes.grid5000.fr 512 39.2        2,218        2,218           1           0    2218457 multiply-optimization void (void)
parapluie-23.rennes.grid5000.fr 512 60.6        3,424        3,424           1           0    3424351 multiply-regular void (void)
chinqchint-9.lille.grid5000.fr  512 40.1        1,149        1,149           1           0    1149238 multiply-optimization void (void)
chinqchint-9.lille.grid5000.fr  512 59.7        1,712        1,712           1           0    1712791 multiply-regular void (void)
pastel-92.toulouse.grid5000.fr  512 40.1        2,602        2,602           1           0    2602363 multiply-optimization void (void)
pastel-92.toulouse.grid5000.fr  512 59.7        3,875        3,875           1           0    3875938 multiply-regular 

When you log off of the Expo Console, two logs are generated:

One log tracks all the main activities, the reservation process, all the execution of commands, times, etc. The other store all the data structures used such as: jobs, deployments, resources, commands results, etc.

This is the end of the tutorial, these were simple examples of how to use Kameleon and Expo to conduct your experiment and make it more reproducible. Thanks for reading.