Grid Setup (with Globus Toolkit)
This document will show you how to set up your account to use the EaStCHEM grid system using the Globus Toolkit. Once you have completed this How To you should be able to submit jobs to either of the EaStCHEM hardware facilities from either site.
Setting up your account for the grid requires a number of steps. The first step is to obtain a UK e-Science Certificate (this usually takes a week or so), then you must install the certificate on the EaStCHEM clusters before finally sending a request to your RCO to have you registered as a grid user.
Obtaining a UK e-Science Certificate
Firstly, go to https://ca.grid-support.ac.uk and select Request a Certificate -> User Certificate. Fill in all the fields and choose Edinburgh NeSC as your registration authority. You will now receive an e-mail confirming that you requested a certificate and asking you to go along to the Registration Authority operator with photographic ID. For both Edinburgh and St. Andrews the Registration Authority is NeSC in Edinburgh. In Edinburgh you have a wide choice of people to visit - at KB you can see Andrew Turner in the School of Chemistry or people in the EPCC, in town you can go along to NeSC. St. Andrews are in the process of setting up their Registration Authority, until it is up and running you will have to come to Edinburgh.
Once your identity has been verified you will receive a further e-mail with a link to download the certificate into your browser. Follow the link.
Installing the certificate on the clusters
Next, we have to convert the certificate in your web browser into a format that Globus Toolkit can use. We must extract the certificate from the browser to do this and the procedure depends on the web browser that you are using:
For Firefox
In Windows open: Tools > Options. In Linux open: Edit > Preferences. In Mac open: Firefox > Preferences
- Select the Certificates button to open the Certificate Manager
- Under Your Certificates tab click on the certificate you want to export and select Backup
- Enter a filename for the exported certificate and select Save
- Choose a password or passphrase to protect the certificate you are exporting. You will need to enter this twice. Select OK.
For Internet Explorer
Select: Tools > Internet Options
- Select the Content tab and click Certificates...
- Under the Personal tab click on the certificate you want to export
- Select Export... to open the Certificate Export Wizard and select Next
- select Yes, export the private key and select Next
- Ensure Personal Information Exchange - PKCS #12 (.PFX) is selected and select Next
- Enter a passphrase to protect the certificate once it is exported and select Next
- Enter a filename for the exported certificate and select Next and then Finish
Now we have the exported certificate we should transfer it to one of the clusters to convert it into a format for use with Globus Toolkit (make sure you select binary transfer.)
We will use OpenSSL to convert our certificate into two parts: the public certificate and the private key. To create the public certificate (usercert.pem) use (assuming that your exported certificate is in cert.p12, if you exported your certificate from Internet Explorer your file will probably have the pfx extension rather than p12.)
> openssl pkcs12 -in cert.p12 -clcerts -nokeys -out usercert.pem
The private key needs to have a passphrase associated with it. This passphrase should be 16 characters long and contain both upper and lowercase letters (this is a requirement of UK e-Science Certificates.) To create the private key (userkey.pem) use the command
> openssl pkcs12 -in cert.p12 -nocerts -out userkey.pem
and enter your passphrase at the prompts. The two files we have created (usercert.pem and userkey.pem) are a public/private key pair that you can use to identify yourself and should be kept backed up and secure along with the p12 file we created from the web browser (for example on your USB memory stick.) You will need them if they ever get deleted from the clusters. It is important to realize that these files have similar significance to your passport or drivers license and could be used by someone to fraudulently pose as you! Do not leave them lying about in publicly accessible places.
The final step in installing the certificates is to create a directory called .globus in your home directory, move the files there and change the permissions on the files to make them secure. This is done with the following commands:
> mkdir ~/.globus > mv usercert.pem ~/.globus > mv userkey.pem ~/.globus > chmod 644 ~/.globus/usercert.pem > chmod 600 ~/.globus/userkey.pem > chmod 700 ~/.globus
Now, e-mail your RCO to request access to the EaStCHEM grid.
Testing the Grid
Once your RCO has informed you that you have access to the EaStCHEM grid we can test to check that everything is working properly.
Firstly we need to initialize a proxy from our UK e-Science Certificate that will allow us to access the grid. This is done using the following command (enter your passphrase for the certificate at the prompts.)
> grid-proxy-init Your identity: /C=UK/O=eScience/OU=Edinburgh/L=NeSC/CN=andrew turner Enter GRID pass phrase for this identity: Creating proxy .............................. Done Your proxy is valid until: Wed Oct 17 23:12:46 2007
Now we will test that we can see the queues on the other host. The command for checking the queue status using the Globus Toolkit is gtstat.
> gtstat burke.st-and.ac.uk
job-ID prior name user state submit/start at queue slots
------------------------------------------------------------------------------------------------------
753 0.50500 mCarbDFT dp12 r 09/06/2007 14:17:44 fat.q@comp00.st-and.ac.uk 1
748 0.50500 pCarbDFT dp12 r 09/06/2007 10:34:32 parallel-medium.q@comp31.st-an 1
712 0.51929 MetaDimer dp12 r 09/04/2007 17:24:57 parallel-medium.q@comp29.st-an 2
691 0.51929 gpt-b3lyp- tanja r 09/02/2007 12:08:34 parallel-long.q@comp28.st-and. 2
726 0.54786 upa_rigid. cmorrison r 09/04/2007 23:00:48 parallel-medium.q@comp25.st-an 4
742 0.60500 go.castep sema r 09/06/2007 06:57:30 parallel-long.q@comp24.st-and. 8
729 0.57643 go.castep sema r 09/05/2007 08:04:25 parallel-long.q@comp20.st-and. 6
754 0.51929 go.castep kej2 r 09/06/2007 14:22:58 parallel-long.q@comp06.st-and. 2
679 0.53357 runPb2 au10 r 08/31/2007 17:15:13 parallel-medium.q@comp03.st-an 3
752 0.00000 go.castep sema hqw 09/06/2007 14:16:40 12
(Of course, if you are on burke you would substitute alchemy.epcc.ed.ac.uk for burke.st-and.ac.uk.) If everything is OK you should see a listing of the queue status on the remote host. If this does not work then you will need to contact your RCO to investigate the cause of the problem.
Next we will test if you can submit a simple job that involves file transfer. Our job script will look like (test.bash and input.txt):
#!/bin/bash # #$ -cwd -V #$ -N Globus_Test #$ -q serial-medium.q #% -host burke.st-and.ac.uk #% -input input.txt /bin/cat input.txt > output.txt
(Again, make sure you use the correct host on the -host line.) Run this job using gtsub. You should see something like the following
> gtsub globus_test.bash ******************************************************************************** Creating Globus job. Submission script will be in submit_1192616173.bash. ******************************************************************************** Processing script file... Remote host: burke.st-and.ac.uk Input files: input.txt ...finished. Remote script in rm_test.bash Creating RSL job descriptions... Written 1192616173_mkdir.rsl <= Create remote directory Written 1192616173_unzip.rsl <= Unzip input files Written 1192616173_qsub.rsl <= Submit job to remote batch system Written 1192616173_zip.rsl <= Zip output Written 1192616173_rm.rsl <= Remove remote directory ...finished Using Web Services and XML RSL Write Globus scripts... Written submit_1192616173.bash <= Submit remote job via Globus Written retrieve_1192616173.bash <= Retrieve remote job output via Globus Written tidy_1192616173.bash <= Tidy up after remote job via Globus ...finished ******************************************************************************** Globus job created. Submit job with submit_1192616173.bash Check status with gtstat burke.st-and.ac.uk Once job has finished run retrieve_1192616173.bash to retrieve output files. Use tidy_1192616173.bash to clean up after remote job. ********************************************************************************
As the output from the command tells you, you should use the submit file to submit the grid job.
> ./submit_1192616173.bash adding: input.txt (stored 0%) adding: rm_test.bash (deflated 16%) 1192616173_mkdir.rsl <= Make remote directory 1192616173_unzip.rsl <= Transfer input files 1192616173_qsub.rsl <= Submit remote job
Now use 'gtstat' to monitor the progress of your job and once it has completed use the retrieve script to get the output back on to your local computer. (The retrieve script can sometimes 'hang' and produce no output, if this happens just use Ctrl-c to cancel the command and reissue it.)
> ./retrieve_1192616173.bash 1192616173_zip.rsl <= Get output files Archive: 1192616173_out.zip extracting: 1192616173_in.zip extracting: Globus_test.o1200 extracting: Globus_test.e1200 extracting: output.txt
Now we have got all the output back we can tidy up after the job using the tidy script.
> ./tidy_1192616173.bash 1192616173_rm.rsl <= Remove remote directory Removing 1192616173_in.zip Removing 1192616173_out.zip Removing 1192616173_mkdir.rsl Removing 1192616173_qsub.rsl Removing 1192616173_rm.rsl Removing 1192616173_unzip.rsl Removing 1192616173_zip.rsl Removing retrieve_1192616173.bash Removing submit_1192616173.bash Removing tidy_1192616173.bash
Check that the job ran correctly by listing the contents of the output.txt file. It should look like
> cat output.txt I am the text of a simple Grid test
If this is successful it means that everything is working fine and you can now submit jobs to remote (or local) machines over the grid. If it does not work then you need to consult your RCO.
For more information about running jobs over the EaStCHEM grid see How To Submit Jobs to the Grid, How To Write Grid Submission Scripts and The EaStCHEM Grid User Guide.