A distributed Matlab job sample

This sample shows how to submit a distributed matlab job containing multiple tasks to eddie

Step 1. create a scheduler using the generic model: 

>> sched=findResource('scheduler','type','generic');

Step 2. to set the location of your job data, replace the value in the command below by your shared directory: 

>> set(sched, 'DataLocation', '/exports/work/is_iti_ug/ywan/matlab-test')

Note: the data location could be a subdirectory on your shared disk, it may looks various on different clients. For example:

Step 3. tell the scheduler that you are using a shared filesystem: 

>> set(sched, 'HasSharedFileSystem', true)

Step 4. set the root directory of the Matlab Distributed Computing Engine on eddie: 

>> set(sched, 'ClusterMatlabRoot', '/exports/home/local/Cluster-Apps/matlab-mdce/3.1-r2007a')

Note: the Cluster Matlab Root directory is same no matter what client you are using.

Step 5. estimate how long your job will run and register the submission function. The time is used to set the runtime limit in GridEngine (<='48:00:00'), and is written as HH:MM:SS: 

>> time_limit='00:05:00';
>> set(sched, 'SubmitFcn', {@sgeSubmitFcn,time_limit})

Step 6. create a job which contains several tasks. 

>> j = createJob(sched);
>> createTask(j, @sum, 1, {[1 1]});
>> createTask(j, @sum, 1, {[2 2]});
>> createTask(j, @sum, 1, {[3 3]});

Note: You will need to set file dependencies if you want to use custom functions (instead of @sum) and custom data:

i.e. To execute a function named process_luxmlx100_b2.m which depends on functions invdist.m, neighbours.m, coord2dist.m and data sets problux_mlx100.mat and mlx100b2.mat (All these functions and data sets are already in Matlab Path) 

>> j = createJob(sched);
>> set(j, 'FileDependencies', {'invdist.m', 'neighbours.m', 'coord2dist.m', 'problux_mlx100.mat', 'mlx100b2.mat'})
>> createTask(j, @process_luxmlx100_b2, 2, {});

Step 7. submit your job, the output shows how Matlab submits your job to GridEngine: 

>> submit(j)

Submitting task 1
qsub -v MDCE_DECODE_FUNCTION=sgeDecodeFunc,\
MDCE_STORAGE_LOCATION=PC{}:UNIX{/exports/work/is_iti_ug/ywan/matlab-test/interactive}:,\
MDCE_STORAGE_CONSTRUCTOR=makeFileStorageObject,MDCE_JOB_LOCATION=Job3,MDCE_DEBUG=true,\
MDCE_MATLAB_EXE=/exports/home/local/Cluster-Apps/matlab-mdce/3.1-r2007a/bin/worker,MDCE_MATLAB_ARGS=,\
MDCE_TASK_LOCATION=Job3/Task1 \
-cwd -N Job3.1 -j yes -o "/exports/work/is_iti_ug/ywan/matlab-test/interactive/Job3_Task1.out" \
-l s_rt="00:05:00" "/exports/home/local/Cluster-Apps/matlab/3.1-r2007a/work/sgeWrapper.sh"
Job output will be written to: /exports/work/is_iti_ug/ywan/matlab-test/interactive/Job3_Task1.out
QSUB output: Your job 4190219 ("Job3.1") has been submitted

Submitting task 2
qsub -v MDCE_DECODE_FUNCTION=sgeDecodeFunc,\
MDCE_STORAGE_LOCATION=PC{}:UNIX{/exports/work/is_iti_ug/ywan/matlab-test/interactive}:,\
MDCE_STORAGE_CONSTRUCTOR=makeFileStorageObject,MDCE_JOB_LOCATION=Job3,MDCE_DEBUG=true,\
MDCE_MATLAB_EXE=/exports/home/local/Cluster-Apps/matlab-mdce/3.1-r2007a/bin/worker,MDCE_MATLAB_ARGS=,\
MDCE_TASK_LOCATION=Job3/Task2 \
-cwd -N Job3.2 -j yes -o "/exports/work/is_iti_ug/ywan/matlab-test/interactive/Job3_Task2.out" \
-l s_rt="00:05:00" "/exports/home/local/Cluster-Apps/matlab/3.1-r2007a/work/sgeWrapper.sh"
Job output will be written to: /exports/work/is_iti_ug/ywan/matlab-test/interactive/Job3_Task2.out
QSUB output: Your job 4190220 ("Job3.2") has been submitted

Submitting task 3
qsub -v MDCE_DECODE_FUNCTION=sgeDecodeFunc,\
MDCE_STORAGE_LOCATION=PC{}:UNIX{/exports/work/is_iti_ug/ywan/matlab-test/interactive}:,\
MDCE_STORAGE_CONSTRUCTOR=makeFileStorageObject,MDCE_JOB_LOCATION=Job3,MDCE_DEBUG=true,\
MDCE_MATLAB_EXE=/exports/home/local/Cluster-Apps/matlab-mdce/3.1-r2007a/bin/worker,MDCE_MATLAB_ARGS=,\
MDCE_TASK_LOCATION=Job3/Task3 \
-cwd -N Job3.3 -j yes -o "/exports/work/is_iti_ug/ywan/matlab-test/interactive/Job3_Task3.out" \
-l s_rt="00:05:00" "/exports/home/local/Cluster-Apps/matlab/3.1-r2007a/work/sgeWrapper.sh"
Job output will be written to: /exports/work/is_iti_ug/ywan/matlab-test/interactive/Job3_Task3.out
QSUB output: Your job 4190221 ("Job3.3") has been submitted

Note: the output may be slightly different if you submit job from your Linux/Windows desktop

Step 8. (optional) set synchronisation with your submitted job. This means that the command prompt won't appear until all job tasks have completed. Unless you have a console session on Eddie, this will be the only way you know when your job has finished. 

>> waitForState(j)

Step 9. retrieve job results 

>> results=getAllOutputArguments(j)

results = 

    [2]
    [4]
    [6]

Step 10. Tidy up the job submission files created by Matlab: 

>> destroy(j)
>>

A distributed Matlab job sample (last edited 2007-11-29 12:05:55 by Yuan WAN)