A parallel Matlab job sample

This sample shows how to submit a parallel matlab job containing single task to eddie

Step 1. create a scheduler using the generic model: 

>> sched=findResource('scheduler','type','generic');

Step 2. to set the location of your job data, replace the value in the command below by your shared directory: 

>> set(sched, 'DataLocation', '/exports/work/is_iti_ug/ywan/matlab-test')

Note: the data location could be a subdirectory on your shared disk, it may looks various on different clients. For example:

Step 3. tell the scheduler that you are using a shared filesystem: 

>> set(sched, 'HasSharedFileSystem', true)

Step 4. set the root directory of the Matlab Distributed Computing Engine on eddie: 

>> set(sched, 'ClusterMatlabRoot', '/exports/home/local/Cluster-Apps/matlab-mdce/3.1-r2007a')

Note: the Cluster Matlab Root directory is same no matter what client you are using.

Step 5. estimate how long your job will run and register the parallel submission function. The time is used to set the runtime limit in GridEngine (<='48:00:00'), and is written as HH:MM:SS: 

>> time_limit='00:05:00';
>> set(sched, 'ParallelSubmitFcn', {@sgeParallelSubmitFcn, time_limit});

Step 6. create a job which contains several tasks. 

>> pjob = createParallelJob(sched);
>> set(pjob, 'FileDependencies', {'colsum.m'});
>> set(pjob, 'MaximumNumberOfWorkers', 4)
>> set(pjob, 'MinimumNumberOfWorkers', 4)
>> t = createTask(pjob, @colsum, 1, {});

Note: You will need to set file dependencies if you want to use custom functions (i.e. colsum.m) and custom data:

Step 7. submit your job, the output shows how Matlab submits your job to GridEngine: 

>> submit(pjob)
qsub -v MDCE_DECODE_FUNCTION=sgeParallelDecode,\
MDCE_STORAGE_LOCATION=PC{}:UNIX{/exports/work/is_iti_ug/ywan/matlab-test/interactive}:,\
MDCE_STORAGE_CONSTRUCTOR=makeFileStorageObject,MDCE_JOB_LOCATION=Job5,MDCE_DEBUG=true,\
MDCE_CMR=/exports/home/local/Cluster-Apps/matlab-mdce/3.1-r2007a,\
MDCE_MATLAB_EXE=/exports/home/local/Cluster-Apps/matlab-mdce/3.1-r2007a/bin/worker,\
MDCE_MATLAB_ARGS=" -parallel",MDCE_TOTAL_TASKS=4 \
-cwd -N Job5 -j yes -o "/exports/work/is_iti_ug/ywan/matlab-test/interactive/Job5.mpiexec.out" \
-l s_rt="00:05:00" -pe matlab 4 "/usr/local/Cluster-Apps/matlab/3.1-r2007a/work/sgeParallelWrapper.sh"
Job output will be written to: /exports/work/is_iti_ug/ywan/matlab-test/interactive/Job5.mpiexec.out
QSUB output: Your job 4190223 ("Job5") has been submitted

Note: the output may be slightly different if you submit job from your Linux/Windows desktop

Step 8. (optional) set synchronisation with your submitted job. This means that the command prompt won't appear until all job tasks have completed. Unless you have a console session on Eddie, this will be the only way you know when your job has finished. 

>> waitForState(pjob)

Step 9. retrieve job results 

>> results=getAllOutputArguments(pjob)

results = 

    [136]
    [136]
    [136]
    [136]

Step 10. Tidy up the job submission files created by Matlab: 

>> destroy(pjob)
>>

A parallel Matlab job sample (last edited 2007-11-29 12:06:31 by Yuan WAN)