MXP: Toy Example

This is a very simple and artificial example, and it is described here in very detail. It should help you to grasp the basic MXP concepts.

Before trying this example, you have to install MXP. Do not forget to add path to MXP binaries to PATH!

It is assumed that everything that will be created during running this example is located in  ~/MXP_Examples  directory. If you prefer another location, replace paths given below accordingly.

In the code snippets given below, what you have type in the terminal is typeset in blue, and what computer responds is in black. When computer response is insignificant, it is omitted.


Create directory for running examples and download Toy-1.tar.gz to it and unpack:

cd ~
mkdir MXP_Examples
cd MXP_Examples
wget --content-disposition
tar -xzf Toy-1.tar.gz

What you get
Inspect your directory:

[sourcecode language=”plain” highlight=”1″] tree Toy-1
└── mxp
└── MkRes

2 directories, 7 files

This is what downloaded Toy-1 file contains.

Now, we have to initialize directory Toy-1. Initialization means creation of mxp.conf file in mxp subdirectory, that specifies (a) dataset name, and (b) parent pipeline.

In our case, dataset name is insignificant (name of our directory is OK, and it is default), and parent pipeline is MXP base, which also is a default. Thus, command for initialization is very simple:

mxp-init Toy-1

MXP: initializing directory ~/MXP_Examples/Toy-1
-- using existing directory ~/MXP_Examples/Toy-1/mxp
-- creating new file       ~/MXP_Examples/Toy-1/mxp/mxp.conf
+ Contents of created file ~/MXP_Examples/Toy-1/mxp/mxp.conf

You are ready to play with Toy-1 example. To simplify subsequent steps, change current directory to Toy-1:

cd Toy-1

All commands below assume that the current directory is Toy-1.

Examine Makefile

cat mxp/
# This is Toy-1 Makefile.
# It is just a very simple example of MXP usage.


MXP_MAKEFILE[t02_something]="(idata_DIR=t01_idata) do_something"

MXP_MAKEFILE[t03_result1]="(idata_DIR=t01_idata) r1 : make_results"
MXP_MAKEFILE[t03_result2]="(idata_DIR=t01_idata) r2 : make_results"

MXP_MAKEFILE[t04_combined_results]="  ( r1 = t03_result1,  r2 = t03_result2 )    combine_results"

It defines 5 targets: t01_idata, t02_something, t03_result1, t03_result2, t04_combined_results.

It also says that 4 methods (idata, do_something, make_results, combine_results) and 2 parameter sets (r1, r2) are needed. You can find corresponding scripts (,,,,, in the mxp subdirectory.

mxp subdirectory structure

When you look inside mxp subdirectory, you will find:

[sourcecode language=”plain” highlight=”1″] tree mxp
├── MkRes
│   ├──
│   ├──
│   └──
└── mxp.conf

1 directory, 8 files

You can see that some files are on the top level, and some files are inside subdirectory MkRes. MXP requires only files mxp.conf and to be on the top level. All other files may be either on the top level, or arbitrary deep in the directory structure. This allows for convenient organization of your files. In our example, we place methods and parameter sets related to making results into MkRes subdirectory. When we have only 8 files, placing them into subdirectories does not help much — but it is really helpful for dozens or hundreds of files.

As you may recall, is a Bash script. It defines elements of associative array MXP_MAKEFILE. The name (index) of an element is the name of the target, and the value is a description of how this target should be obtained.

As is a Bash script, it must obey Bash syntax. In particular, no spaces are allowed before the opening quotation mark. The line:

MXP_MAKEFILE[t01_idata]   = "idata"

is syntactically incorrect from Bash point of view.

In contrast, the description of how the target should be obtained can use spaces freely to increase readability — the last line of is an example.

Obtain the first target

Let us try to obtain target t01_idata. According to Makefile, it has no required targets (the definition does not contain expression in parentheses), and in order to obtain it, script should be executed. You can find this script in mxp subdirectory. If you look inside, you can see that it just writes two files to the target directory and reports its actions to the terminal. So, enter command:

mxp t01_idata

Output of   mxp t01_idata
[sourcecode language=”plain” highlight=””] **************************************************************************************************
**** mxp t01_idata (Dataset: Toy-1) 2018-02-09 14:36:41
**** Main work directory: /home/mk42/MXP_Examples/Toy-1

**** Trying to obtain lock… Lock successfully obtained, proceeding.

**** Preprocessing Makefile:
—- target names are OK.

**** Checking required targets:
t01_idata [OK-] Need to rebuild — does not exist.

==== Building target t01_idata <= idata 2018-02-09 14:36:41

— Toy-1 — idata started.
Created: /home/mk42/MXP_Examples/Toy-1/t01_idata/InitialFile-1.txt
Created: /home/mk42/MXP_Examples/Toy-1/t01_idata/InitialFile-2.txt
— Toy-1 — idata completed.

==== t01_idata : Successfully built target. 2018-02-09 14:36:41
Wall=0:00:00 Usr=0:00:00 Sys=0:00:00 CPU=30% Mem=357K

**** mxp t01_idata : Successfully built target. 2018-02-09 14:36:41
Wall=0:00:00 Usr=0:00:00 Sys=0:00:00 CPU=47% Mem=458K