Globus Toolkit 4.0.8 (GRID) with Weka4ws

Table of Contents (Globus Toolkit) -- INCOMPLETE

1. Introduction
2. Setting up the first machine

2.1. Pre-requisites
2.2. Building the Toolkit
2.3. Setting up security on your first machine
2.4. Set up GridFTP
2.5. Starting the webservices container
2.6. Configuring RFT
2.7. Setting up WS GRAM

3. Setting up your second machine

3.1. Setting up your second machine: Prereqs
3.2. Setting up your second machine: Installation
3.3. Setting up your second machine: Security
3.4. Setting up your second machine: GridFTP
3.5. Setting up your second machine: Webservices
3.6. Setting up your second machine: WS GRAM

4. VO-level services

4.1. Setting up an Index Service hierarchy
4.2. Configuring WebMDS
4.3. Creating a MyProxy server

5. Setting up your cluster

5.1. Cluster: prereqs
5.2. Cluster: install
5.3. Security on the cluster
5.4. GridFTP on the cluster
5.5. Cluster: Container
5.6. Cluster: RFT
5.7. Cluster: GRAM
5.8. Cluster: MDS

6. Establishing Cross-CA Trust
7. Now what?

1. Introduction

This is a quickstart that shows a full installation of the Toolkit on two Centos 5 machines. It shows the installation of prereqs, installation of the toolkit, creation of certificates, and configuration of services. It is designed to supplement the main admin guide.

The installer used throughout this document is the GT4.0.8 installer. There are no changes required to use this document with later 4.0.8 installers. You should use the most current version available.
2. Setting up the first machine
2.1. Pre-requisites

I will be installing all of the toolkit from source, so I'm going to double-check my system for pre-requisites. The full list of prereqs is available at Software Prerequisites in the Admin Guide.

First I'll check for zlib development libraries for GSI-OpenSSH:

choate % yum install zlib zlib-devel gcc gcc-c++ tar sed make perl sudo postgresql postgresql-server postgresql-odbc xinetd

Next, I'll install java from Sun, you can download from here.

root@choate:~# cd /opt
root@choate:~# ./jdk-6u13-linux-i586.bin
root@choate:~# mv jdk1.6.0_13 java

Next, we download from here, install ant:

root@choate:~# cd /opt
root@choate:~# tar xzf apache-ant-1.7.1-bin.tar.gz
root@choate:~# mv apache-ant-1.7.1 ant
Aslo need to set JAVA_HOME and ANT_HOME
Add below entry in /etc/profile
export JAVA_HOME=/opt/java
export ANT_HOME=/opt/ant
export GLOBUS_LOCATION=/usr/local/globus-4.0.1
export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$GLOBUS_LOCATION/bin:$GLOBUS_LOCATION/sbin:$PATH

2.2. Building the Toolkit

That completes the list of build prereqs, so now I will download the installer from here and build it. The long version of these instructions is at Installing in the Admin Guide.

root@cognito:~# useradd globus
root@choate:/etc/init.d# mkdir /usr/local/globus-4.0.1/
root@choate:/etc/init.d# chown globus:globus /usr/local/globus-4.0.1/

Now, as the newly created globus user:

root@cognito:~# su - globus
globus@choate:~$ tar xzf gt4.0.8-all-source-installer.tar.gz
globus@choate:~$ cd gt4.0.8-all-source-installer
globus@choate:~/gt4.0.8-all-source-installer$ ./configure --prefix=/usr/local/globus-4.0.1/ \
--with-iodbc=/usr/lib
checking build system type... i686-pc-linux-gnu
checking for javac... /usr/java/j2sdk1.4.2_10//bin/javac
checking for ant... /usr/local/apache-ant-1.6.5/bin/ant
configure: creating ./config.status
config.status: creating Makefile

Much better!

[Note]


Note



The machine I am installing on doesn't have access to a scheduler. If it did, I would have specified one of the wsgram scheduler options, like --enable-wsgram-condor, --enable-wsgram-lsf, or --enable-wsgram-pbs.



Now it's time to build the toolkit:

globus@choate:~/gt4.0.1-all-source-installer$ make | tee installer.log
cd gpt-3.2autotools2004 && OBJECT_MODE=32 ./build_gpt
build_gpt ====> installing GPT into /usr/local/globus-4.0.1/
...
Time for a coffee break here, the build will take over an hour, possibly
longer depending on how fast your machine is
...
echo "Your build completed successfully. Please run make install."
Your build completed successfully. Please run make install.
globus@choate:~/gt4.0.1-all-source-installer$ make install
/usr/local/globus-4.0.1//sbin/gpt-postinstall
...
..Done
globus@choate:~/gt4.0.1-all-source-installer$

2.3. Setting up security on your first machine

Now that the toolkit is installed, we're going to want hostcerts for the machine, and a usercert for me. To do that, we're going to use the SimpleCA that is distributed with the toolkit. Here's how we set it up, based on the instructions at SimpleCA Admin:

globus@choate:~$ source $GLOBUS_LOCATION/etc/globus-user-env.sh
globus@choate:~$ $GLOBUS_LOCATION/setup/globus/setup-simple-ca
WARNING: GPT_LOCATION not set, assuming:
GPT_LOCATION=/usr/local/globus-4.0.1
C e r t i f i c a t e A u t h o r i t y S e t u p
This script will setup a Certificate Authority for signing Globus
users certificates. It will also generate a simple CA package
that can be distributed to the users of the CA.
The CA information about the certificates it distributes will
be kept in:
/home/globus/.globus/simpleCA/
/usr/local/globus-4.0.1/setup/globus/setup-simple-ca: line 250:
test: res: integer expression expected
The unique subject name for this CA is:
cn=Globus Simple CA, ou=simpleCA-choate.mcs.anl.gov, ou=GlobusTest, o=Grid
Do you want to keep this as the CA subject (y/n) [y]:
y
Enter the email of the CA (this is the email where certificate
requests will be sent to be signed by the CA): bacon@choate
The CA certificate has an expiration date. Keep in mind that
once the CA certificate has expired, all the certificates
signed by that CA become invalid. A CA should regenerate
the CA certificate and start re-issuing ca-setup packages
before the actual CA certificate expires. This can be done
by re-running this setup script. Enter the number of DAYS
the CA certificate should last before it expires.
[default: 5 years (1825 days)]:RETURN
Enter PEM pass phrase:******
Verifying - Enter PEM pass phrase:******
/bin/sed: can't read /tmp//globus_tmp_ca_setup//pkgdata/pkg_data_src.gpt.tmpl:
No such file or directory
creating CA config package...
A self-signed certificate has been generated
for the Certificate Authority with the subject:
/O=Grid/OU=GlobusTest/OU=simpleCA-choate.mcs.anl.gov/CN=Globus Simple CA
If this is invalid, rerun this script
/usr/local/globus-4.0.1/setup/globus/setup-simple-ca
and enter the appropriate fields.
-------------------------------------------------------------------
The private key of the CA is stored in /home/globus/.globus/simpleCA//private/cakey.pem
The public CA certificate is stored in /home/globus/.globus/simpleCA//cacert.pem
The distribution package built for this CA is stored in
/home/globus/.globus/simpleCA//globus_simple_ca_ebb88ce5_setup-0.18.tar.gz
This file must be distributed to any host wishing to request
certificates from this CA.
CA setup complete.
The following commands will now be run to setup the security
configuration files for this CA:
$GLOBUS_LOCATION/sbin/gpt-build \
/home/globus/.globus/simpleCA//globus_simple_ca_ebb88ce5_setup-0.18.tar.gz
$GLOBUS_LOCATION/sbin/gpt-postinstall
-------------------------------------------------------------------
setup-ssl-utils: Configuring ssl-utils package
Running setup-ssl-utils-sh-scripts...
***************************************************************************
Note: To complete setup of the GSI software you need to run the
following script as root to configure your security configuration
directory:
/usr/local/globus-4.0.1/setup/globus_simple_ca_ebb88ce5_setup/setup-gsi
For further information on using the setup-gsi script, use the -help
option. The -default option sets this security configuration to be
the default, and -nonroot can be used on systems where root access is
not available.
***************************************************************************
setup-ssl-utils: Complete
globus@choate:~$

That's quite a lot of output. Here's what has happened:

globus@choate:~$ ls ~/.globus/
simpleCA
globus@choate:~$ ls ~/.globus/simpleCA/
cacert.pem globus_simple_ca_ebb88ce5_setup-0.18.tar.gz newcerts
certs grid-ca-ssl.conf private
crl index.txt serial

That's the directory where my simpleCA has been created. Now I need to make my machine trust that new CA, which I do by running the following command as root:

globus@choate:~$ exit
root@choate:~# export GLOBUS_LOCATION=/usr/local/globus-4.0.1
root@choate:~# $GLOBUS_LOCATION/setup/globus_simple_ca_ebb88ce5_setup/setup-gsi -default
setup-gsi: Configuring GSI security
Making /etc/grid-security...
mkdir /etc/grid-security
Making trusted certs directory: /etc/grid-security/certificates/
mkdir /etc/grid-security/certificates/
Installing /etc/grid-security/certificates//grid-security.conf.ebb88ce5...
Running grid-security-config...
Installing Globus CA certificate into trusted CA certificate directory...
Installing Globus CA signing policy into trusted CA certificate directory...
setup-gsi: Complete
root@choate:~# ls /etc/grid-security/
certificates globus-host-ssl.conf globus-user-ssl.conf grid-security.conf
root@choate:~# ls /etc/grid-security/certificates/
ebb88ce5.0 globus-user-ssl.conf.ebb88ce5
ebb88ce5.signing_policy grid-security.conf.ebb88ce5
globus-host-ssl.conf.ebb88ce5

Those are the configuration files that establish trust for the simpleCA for my Globus Toolkit installation. Notice that the hash value ebb88ce5 matches the hash value of my SimpleCA. These files are all explained in Security Admin. If you don't have root, you can use the -nonroot option to place the files under your GLOBUS_LOCATION instead.

Now that we've created a CA and trust it, we'll get a hostcert for the machine:

root@choate:~# source $GLOBUS_LOCATION/etc/globus-user-env.sh
root@choate:~# grid-cert-request -host `hostname`
Generating a 1024 bit RSA private key
..++++++
...................................................++++++
writing new private key to '/etc/grid-security/hostkey.pem'
...
Your certificate will be mailed to you within two working days.
If you receive no response, contact Globus Simple CA at bacon@choate

We need to sign the certificate using our simpleCA, as globus:

root@choate:~$ su - globus
globus@choate:~$ grid-ca-sign -in /etc/grid-security/hostcert_request.pem -out hostsigned.pem
To sign the request
please enter the password for the CA key:******
The new signed certificate is at: /home/globus/.globus/simpleCA//newcerts/01.pem

Our last step is to copy that signed certificate into /etc:

globus@choate:~$ exit
root@choate:~# cp ~globus/hostsigned.pem /etc/grid-security/hostcert.pem

The hostcert and hostkey are owned by root, and will be used by the GridFTP server. Because the webservices container runs non-root, we need a certificate owned by globus. In the end, we need one host certificate/key owned by root, and one host certificate/key owned by globus. We do that by copying the files:

root@choate:/etc/grid-security# cp hostcert.pem containercert.pem
root@choate:/etc/grid-security# cp hostkey.pem containerkey.pem
root@choate:/etc/grid-security# chown globus:globus container*.pem
root@choate:/etc/grid-security# ls -l *.pem
-r-------- 1 globus globus 887 2005-11-15 07:48 containerkey.pem
-rw-r--r-- 1 globus globus 2710 2005-11-15 07:48 containercert.pem
-rw-r--r-- 1 root root 2710 2005-11-15 07:47 hostcert.pem
-rw-r--r-- 1 root root 1404 2005-11-15 07:40 hostcert_request.pem
-r-------- 1 root root 887 2005-11-15 07:40 hostkey.pem

Now we'll get a usercert for bacon. In this example I'm running tcsh, just to show that the version of globus-user-env depends on your shell:

bacon@choate :~$ setenv GLOBUS_LOCATION /usr/local/globus-4.0.1/
bacon@choate :~$ source $GLOBUS_LOCATION/etc/globus-user-env.sh
bacon@choate :~$ grid-cert-request
A certificate request and private key is being created.
You will be asked to enter a PEM pass phrase.
This pass phrase is akin to your account password,
and is used to protect your key file.
If you forget your pass phrase, you will need to
obtain a new certificate.
Generating a 1024 bit RSA private key
.........................................................++++++
.........................++++++
unable to write 'random state'
writing new private key to '/home/bacon/.globus/userkey.pem'
Enter PEM pass phrase: ****
Verifying - Enter PEM pass phrase: ****
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
-----
Level 0 Organization [Grid]:
Level 0 Organizational Unit [GlobusTest]:
Level 1 Organizational Unit [simpleCA-choate.mcs.anl.gov]:
Level 2 Organizational Unit [mcs.anl.gov]:
Name (e.g., John M. Smith) []:
A private key and a certificate request has been generated with the subject:
/O=Grid/OU=GlobusTest/OU=simpleCA-choate.mcs.anl.gov/OU=mcs.anl.gov/CN=Charles Bacon
If the CN=Charles Bacon is not appropriate, rerun this
script with the -force -cn "Common Name" options.
Your private key is stored in /home/bacon/.globus/userkey.pem
Your request is stored in /home/bacon/.globus/usercert_request.pem
Please e-mail the request to the Globus Simple CA bacon@choate
You may use a command similar to the following:
cat /home/bacon/.globus/usercert_request.pem | mail bacon@choate
Only use the above if this machine can send AND receive e-mail. if not, please
mail using some other method.
Your certificate will be mailed to you within two working days.
If you receive no response, contact Globus Simple CA at bacon@choate

Now I need to get that certificate request to the globus user so it can be signed, then send the signed cert back to bacon:

bacon@choate :~$ cat /home/bacon/.globus/usercert_request.pem | mail globus@choate

Please note that the point of the "mail" command is to move the request from the second machine to the first machine. I need it on the first machine because that's where SimpleCA is installed. You can use scp or some other tool to move the request over to choate. Once it is on choate, sign it as user globus:

bacon@choate :~$ exit
root@choate :~$ su - globus
globus@choate:~$ grid-ca-sign -in request.pem -out signed.pem
To sign the request
please enter the password for the CA key: ******
The new signed certificate is at: /home/globus/.globus/simpleCA//newcerts/02.pem
globus@choate:~$ cat signed.pem | mail bacon@choate

Now user bacon checks his mail and copies the cert to the proper location:

globus@choate :~$ exit
bacon@choate :~$ su - bacon
bacon@choate :~$ cp signed.pem ~/.globus/usercert.pem
bacon@choate :~$ ls -l ~/.globus/
total 12
-rw-r--r-- 1 bacon globdev 895 2005-11-15 07:57 usercert.pem
-rw-r--r-- 1 bacon globdev 1426 2005-11-15 07:51 usercert_request.pem
-r-------- 1 bacon globdev 963 2005-11-15 07:51 userkey.pem

Our last act will be to create a grid-mapfile as root for authorization:

root@choate:/etc/grid-security# vim /etc/grid-security/grid-mapfile
root@choate:/etc/grid-security# cat /etc/grid-security/grid-mapfile
"/O=Grid/OU=GlobusTest/OU=simpleCA-choate.mcs.anl.gov/OU=mcs.anl.gov/CN=Charles Bacon" bacon

[Note]


Note



The globus user doesn't need a user certificate! It's a dummy account that we're using to own the GLOBUS_LOCATION. When it starts the container, it will use the containercert. Only real people need user certs.

2.4. Set up GridFTP

Now that we have our secure credentials in place, we can start a service. This setup comes from the GridFTP Admin Guide.

root@choate:/etc/grid-security# vim /etc/xinetd.d/gridftp
root@choate:/etc/grid-security# cat /etc/xinetd.d/gridftp
service gsiftp
{
instances = 100
socket_type = stream
wait = no
user = root
env += GLOBUS_LOCATION=/usr/local/globus-4.0.1
env += LD_LIBRARY_PATH=/usr/local/globus-4.0.1/lib
server = /usr/local/globus-4.0.1/sbin/globus-gridftp-server
server_args = -i
log_on_success += DURATION
nice = 10
disable = no
}
root@choate:/etc/grid-security# vim /etc/services
root@choate:/etc/grid-security# tail /etc/services
vboxd 20012/udp
binkp 24554/tcp # binkp fidonet protocol
asp 27374/tcp # Address Search Protocol
asp 27374/udp
dircproxy 57000/tcp # Detachable IRC Proxy
tfido 60177/tcp # fidonet EMSI over telnet
fido 60179/tcp # fidonet EMSI over TCP
# Local services
gsiftp 2811/tcp
root@choate:/etc/grid-security# /etc/init.d/xinetd reload
Reloading internet superserver configuration: xinetd.
root@choate:/etc/grid-security# netstat -an | grep 2811
tcp 0 0 0.0.0.0:2811 0.0.0.0:* LISTEN

Now the gridftp server is waiting for a request, so we'll run a client and transfer a file:

bacon@choate :~$ grid-proxy-init -verify -debug
User Cert File: /home/bacon/.globus/usercert.pem
User Key File: /home/bacon/.globus/userkey.pem
Trusted CA Cert Dir: /etc/grid-security/certificates
Output File: /tmp/x509up_u1817
Your identity: /O=Grid/OU=GlobusTest/OU=simpleCA-choate.mcs.anl.gov/OU=mcs.anl.gov/CN=Charles Bacon
Enter GRID pass phrase for this identity: ****
Creating proxy .....++++++++++++
..++++++++++++
Done
Proxy Verify OK
Your proxy is valid until: Tue Nov 15 20:15:46 2005
bacon@choate :~$ globus-url-copy gsiftp://choate.mcs.anl.gov/etc/group file:///tmp/bacon.test.copy
bacon@choate :~$ diff /tmp/bacon.test.copy /etc/group
bacon@choate :~$

The globus-url-copy takes two arguments, the source and the destination. In this case we're using the gsiftp:// to indicate a GridFTP transfer from the host choate.mcs.anl.gov of the file /etc/group as the source. The destination is just going to be a file on the local system under /tmp/bacon.test.copy.

Okay, so the GridFTP server works. If you had trouble, start with the GridFTP Troubleshooting guide. If the trouble is with your certificates, check the security troubleshooting. Now we can move on to starting the webservices container.
2.5. Starting the webservices container

Now we'll setup an /etc/init.d entry for the webservices container. You can find more details about the container at Container Admin Guide.

globus@choate:~$ vim $GLOBUS_LOCATION/start-stop
globus@choate:~$ cat $GLOBUS_LOCATION/start-stop
#! /bin/sh
set -e
export GLOBUS_LOCATION=/usr/local/globus-4.0.1
export JAVA_HOME=/opt/java
export ANT_HOME=/opt/ant
export GLOBUS_OPTIONS="-Xms256M -Xmx512M"
. $GLOBUS_LOCATION/etc/globus-user-env.sh
cd $GLOBUS_LOCATION
case "$1" in
start)
$GLOBUS_LOCATION/sbin/globus-start-container-detached -p 8443
;;
stop)
$GLOBUS_LOCATION/sbin/globus-stop-container-detached
;;
*)
echo "Usage: globus {start|stop}" >&2
exit 1
;;
esac
exit 0
globus@choate:~$ chmod +x $GLOBUS_LOCATION/start-stop
Now, as root, we'll create an /etc/init.d script to call the globus user's start-stop script:
root@choate:~# vim /etc/init.d/globus-4.0.1
root@choate:~# cat /etc/init.d/globus-4.0.1
#!/bin/sh -e
case "$1" in
start)
su - globus /usr/local/globus-4.0.1/start-stop start
;;
stop)
su - globus /usr/local/globus-4.0.1/start-stop stop
;;
restart)
$0 stop
sleep 1
$0 start
;;
*)
printf "Usage: $0 {start|stop|restart}\n" >&2
exit 1
;;
esac
exit 0
root@choate:~# chmod +x /etc/init.d/globus-4.0.1
root@choate:~# /etc/init.d/globus-4.0.1 start
Starting Globus container. PID: 29985
root@choate:~# cat /usr/local/globus-4.0.1/var/container.log
2005-11-15 08:48:00,886 ERROR service.ReliableFileTransferImpl [main,:68]
Unable to setup database driver with pooling.A connection error has occurred:
FATAL: no pg_hba.conf entry for host "140.221.8.31", user "globus",
database "rftDatabase", SSL off
2005-11-15 08:48:02,183 WARN service.ReliableFileTransferHome [main,initialize:97]
All RFT requests will fail and all GRAM jobs that require file staging will fail.
A connection error has occurred: FATAL: no pg_hba.conf entry for host
"140.221.8.31", user "globus", database "rftDatabase", SSL off
Starting SOAP server at: https://140.221.8.31:8443/wsrf/services/
With the following services:
[1]: https://140.221.8.31:8443/wsrf/services/TriggerFactoryService
[2]: https://140.221.8.31:8443/wsrf/services/DelegationTestService
[3]: https://140.221.8.31:8443/wsrf/services/SecureCounterService
[4]: https://140.221.8.31:8443/wsrf/services/IndexServiceEntry
[5]: https://140.221.8.31:8443/wsrf/services/DelegationService
[6]: https://140.221.8.31:8443/wsrf/services/InMemoryServiceGroupFactory
[7]: https://140.221.8.31:8443/wsrf/services/mds/test/execsource/IndexService
[8]: https://140.221.8.31:8443/wsrf/services/mds/test/subsource/IndexService
[9]: https://140.221.8.31:8443/wsrf/services/SubscriptionManagerService
[10]: https://140.221.8.31:8443/wsrf/services/TestServiceWrongWSDL
[11]: https://140.221.8.31:8443/wsrf/services/SampleAuthzService
[12]: https://140.221.8.31:8443/wsrf/services/WidgetNotificationService
[13]: https://140.221.8.31:8443/wsrf/services/AdminService
[14]: https://140.221.8.31:8443/wsrf/services/DefaultIndexServiceEntry
[15]: https://140.221.8.31:8443/wsrf/services/CounterService
[16]: https://140.221.8.31:8443/wsrf/services/TestService
[17]: https://140.221.8.31:8443/wsrf/services/InMemoryServiceGroup
[18]: https://140.221.8.31:8443/wsrf/services/SecurityTestService
[19]: https://140.221.8.31:8443/wsrf/services/ContainerRegistryEntryService
[20]: https://140.221.8.31:8443/wsrf/services/NotificationConsumerFactoryService
[21]: https://140.221.8.31:8443/wsrf/services/TestServiceRequest
[22]: https://140.221.8.31:8443/wsrf/services/IndexFactoryService
[23]: https://140.221.8.31:8443/wsrf/services/ReliableFileTransferService
[24]: https://140.221.8.31:8443/wsrf/services/mds/test/subsource/IndexServiceEntry
[25]: https://140.221.8.31:8443/wsrf/services/Version
[26]: https://140.221.8.31:8443/wsrf/services/NotificationConsumerService
[27]: https://140.221.8.31:8443/wsrf/services/IndexService
[28]: https://140.221.8.31:8443/wsrf/services/NotificationTestService
[29]: https://140.221.8.31:8443/wsrf/services/ReliableFileTransferFactoryService
[30]: https://140.221.8.31:8443/wsrf/services/DefaultTriggerServiceEntry
[31]: https://140.221.8.31:8443/wsrf/services/TriggerServiceEntry
[32]: https://140.221.8.31:8443/wsrf/services/PersistenceTestSubscriptionManager
[33]: https://140.221.8.31:8443/wsrf/services/mds/test/execsource/IndexServiceEntry
[34]: https://140.221.8.31:8443/wsrf/services/DefaultTriggerService
[35]: https://140.221.8.31:8443/wsrf/services/TriggerService
[36]: https://140.221.8.31:8443/wsrf/services/gsi/AuthenticationService
[37]: https://140.221.8.31:8443/wsrf/services/TestRPCService
[38]: https://140.221.8.31:8443/wsrf/services/ManagedMultiJobService
[39]: https://140.221.8.31:8443/wsrf/services/RendezvousFactoryService
[40]: https://140.221.8.31:8443/wsrf/services/WidgetService
[41]: https://140.221.8.31:8443/wsrf/services/ManagementService
[42]: https://140.221.8.31:8443/wsrf/services/ManagedExecutableJobService
[43]: https://140.221.8.31:8443/wsrf/services/InMemoryServiceGroupEntry
[44]: https://140.221.8.31:8443/wsrf/services/AuthzCalloutTestService
[45]: https://140.221.8.31:8443/wsrf/services/DelegationFactoryService
[46]: https://140.221.8.31:8443/wsrf/services/DefaultIndexService
[47]: https://140.221.8.31:8443/wsrf/services/ShutdownService
[48]: https://140.221.8.31:8443/wsrf/services/ContainerRegistryService
[49]: https://140.221.8.31:8443/wsrf/services/TestAuthzService
[50]: https://140.221.8.31:8443/wsrf/services/CASService
[51]: https://140.221.8.31:8443/wsrf/services/ManagedJobFactoryService
2005-11-15 08:48:29,063 INFO impl.DefaultIndexService [ServiceThread-10,processConfigFile:107]
Reading default registration configuration from file:
/usr/local/globus-4.0.1/etc/globus_wsrf_mds_index/hierarchy.xml
2005-11-15 08:48:31,705 ERROR impl.QueryAggregatorSource [Thread-12,pollGetMultiple:149]
Exception Getting Multiple Resource Properties from
https://140.221.8.31:8443/wsrf/services/ReliableFileTransferFactoryService:
java.rmi.RemoteException: Failed to serialize resource property
org.globus.transfer.reliable.service.factory.TotalNumberOfBytesTransferred@e8eeca;
nested exception is:
org.apache.commons.dbcp.DbcpException: A connection error has occurred:
FATAL: no pg_hba.conf entry for host "140.221.8.31", user "globus",
database "rftDatabase", SSL off

1


The RFT warnings are expected right now because we haven't setup our database yet. Otherwise, things look good.

2


140.221.8.31 is my IP address. Some people following the quickstart may see "127.0.0.1" here. You need to fix that! Edit $GLOBUS_LOCATION/etc/globus_wsrf_core/server-config.wsdd and client-server-config.wsdd, add a line reading 140.221.8.32" /> under the section. For instance:

You can also use this to select the interface to publish for a multi-homed host. See Global Configuration for more container config options.

At this point, we can use one of the sample clients/services to interact with the container:

choate % setenv JAVA_HOME /usr/java/j2sdk1.4.2_10/
choate % setenv ANT_HOME /usr/local/apache-ant-1.6.5/
choate % setenv PATH $ANT_HOME/bin:$JAVA_HOME/bin:$PATH
choate % counter-client -s https://choate.mcs.anl.gov:8443/wsrf/services/CounterService
Got notification with value: 3
Counter has value: 3
Got notification with value: 13



Whenever you see me testing against "choate.mcs.anl.gov" in this document, use your own fully qualified hostname. Connections to choate will timeout because the host is behind a firewall.

That is the expected output, so it looks like the container is up and running. Next we'll configure a database for RFT to get rid of that pesky warning, and so we can reliably transfer files using GridFTP!
2.6. Configuring RFT

Following the instructions at RFT Admin, we'll first configure the system to allow TCP/IP connections to postgres, as well as adding a trust entry for our current host. More recent versions of postgres have deprecated the "-i" option. For version 8 and newer, instead set "listen_addresses = '*'" in the postgres configuration file.

root@choate:~# vim /var/lib/psql/data/pg_hba.conf
root@choate:~# grep rftDatabase /etc/postgresql/pg_hba.conf
host rftDatabase "globus" "140.221.8.31" 255.255.255.255 md5
root@choate:~# /etc/init.d/postgresql restart
Stopping PostgreSQL database server: postmaster.
Starting PostgreSQL database server: postmaster.
root@choate:~# su postgres -c "createuser -P globus"
Enter password for new user: *****
Enter it again: *****
Shall the new user be allowed to create databases? (y/n) y
Shall the new user be allowed to create more new users? (y/n) n
CREATE USER

[Note]


Note



This is one of the most system-dependent steps of this quickstart. Your pg_hba.conf and postmaster.conf files may be located in a different directory. Please consult your vendor's notes for details.

Now the globus user can create the rftDatabase:

globus@choate:~$ createdb rftDatabase
CREATE DATABASE
globus@choate:~$ psql -d rftDatabase -f $GLOBUS_LOCATION/share/globus_wsrf_rft/rft_schema.sql
psql:/usr/local/globus-4.0.1/share/globus_wsrf_rft/rft_schema.sql:6: NOTICE:
CREATE TABLE / PRIMARY KEY will create implicit index "requestid_pkey" for table "requestid"
CREATE TABLE
psql:/usr/local/globus-4.0.1/share/globus_wsrf_rft/rft_schema.sql:11: NOTICE:
CREATE TABLE / PRIMARY KEY will create implicit index "transferid_pkey" for table "transferid"
CREATE TABLE
psql:/usr/local/globus-4.0.1/share/globus_wsrf_rft/rft_schema.sql:30: NOTICE:
CREATE TABLE / PRIMARY KEY will create implicit index "request_pkey" for table "request"
CREATE TABLE
psql:/usr/local/globus-4.0.1/share/globus_wsrf_rft/rft_schema.sql:65: NOTICE:
CREATE TABLE / PRIMARY KEY will create implicit index "transfer_pkey" for table "transfer"
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE INDEX
globus@choate:~$ vim $GLOBUS_LOCATION/etc/globus_wsrf_rft/jndi-config.xml
globus@choate:~$ grep -C 3 password $GLOBUS_LOCATION/etc/globus_wsrf_rft/jndi-config.xml



password


*****

I have created the database, loaded the RFT schema, and changed the password in the jndi-config.xml file. If your database isn't owned by the same user as the container, you will also need to change the username parameter in the jndi-config.xml. In this example, we installed as globus and made the database as globus, so I only changed the password.

The database is setup, so we restart the container to load the new RFT configuration:

root@choate:~# /etc/init.d/globus-4.0.1 restart
Stopping Globus container. PID: 29985
Starting Globus container. PID: 8620
root@choate:~# head /usr/local/globus-4.0.1/var/container.log
Starting SOAP server at: https://140.221.8.31:8443/wsrf/services/
With the following services:
[1]: https://140.221.8.31:8443/wsrf/services/TriggerFactoryService
[2]: https://140.221.8.31:8443/wsrf/services/DelegationTestService
[3]: https://140.221.8.31:8443/wsrf/services/SecureCounterService
[4]: https://140.221.8.31:8443/wsrf/services/IndexServiceEntry
[5]: https://140.221.8.31:8443/wsrf/services/DelegationService
[6]: https://140.221.8.31:8443/wsrf/services/InMemoryServiceGroupFactory
[7]: https://140.221.8.31:8443/wsrf/services/mds/test/execsource/IndexService
...

Great, we got rid of the warning. Now let's try an RFT transfer to make sure the service is really working:

bacon@choate :~$ cp /usr/local/globus-4.0.1/share/globus_wsrf_rft_test/transfer.xfr /tmp/rft.xfr
bacon@choate :~$ vim /tmp/rft.xfr
bacon@choate :~$ cat /tmp/rft.xfr
true
16000
16000
false
1
true
1
null
null
false
10
gsiftp://choate.mcs.anl.gov:2811/etc/group
gsiftp://choate.mcs.anl.gov:2811/tmp/rftTest_Done.tmp
bacon@choate :~$ rft -h choate.mcs.anl.gov -f /tmp/rft.xfr
Number of transfers in this request: 1
Subscribed for overall status
Termination time to set: 60 minutes
Overall status of transfer:
Finished/Active/Failed/Retrying/Pending
0/1/0/0/0
Overall status of transfer:
Finished/Active/Failed/Retrying/Pending
1/0/0/0/0
All Transfers are completed
bacon@choate :~$ diff /etc/group /tmp/rftTest_Done.tmp
bacon@choate :~$

RFT did its job, starting up a reliable transfer and notifying us of the status and results.
2.7. Setting up WS GRAM

Now that we have GridFTP and RFT working, we can setup GRAM for resource management. First we have to setup sudo so the globus user can start jobs as a different user. For reference, you can see the GRAM Admin Guide.

root@choate:~# visudo
root@choate:~# cat /etc/sudoers
globus ALL=(bacon) NOPASSWD: /usr/local/globus-4.0.1/libexec/globus-gridmap-and-execute
-g /etc/grid-security/grid-mapfile /usr/local/globus-4.0.1/libexec/globus-job-manager-script.pl *
globus ALL=(bacon) NOPASSWD: /usr/local/globus-4.0.1/libexec/globus-gridmap-and-execute
-g /etc/grid-security/grid-mapfile /usr/local/globus-4.0.1/libexec/globus-gram-local-proxy-tool *

Make sure they're all on one line. I split them up in the HTML to keep the page width down. Also, if your sudoers has a line reading "requiretty", remove it. The container won't have a tty while it is running, but it needs to use sudo. With that addition, we can now run jobs:

bacon@choate :~$ globusrun-ws -submit -c /bin/true
Submitting job...Done.
Job ID: uuid:3304e3f2-55f2-11da-8b8f-00d0b7b7c0bc
Termination time: 11/16/2005 16:09 GMT
Current job state: Active
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
bacon@choate :~$ echo $?
0
choate % globusrun-ws -submit -c /bin/false
Submitting job...Done.
Job ID: uuid:456b7c9a-55f2-11da-9b0d-00d0b7b7c0bc
Termination time: 11/16/2005 16:09 GMT
Current job state: Active
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
bacon@choate :~$ echo $?
1

Success. Now we've got a working GRAM installation.
3. Setting up your second machine
3.1. Setting up your second machine: Prereqs

Alas, it's not much of a grid with just one machine. So let's start up on another machine and add it to this little test grid. For a change of pace, I'm going to use the binary installer on this machine. First, though, let's get some prereqs out of the way:

root@cognito:~# cd /opt
root@cognito:~# ./jdk-6u13-linux-i586.bin
root@cognito:~# mv jdk1.6.0_13 java

Next, we download from here, install ant:

root@cognito:~# cd /opt
root@cognito:~# tar xzf apache-ant-1.7.1-bin.tar.gz
root@cognito:~# mv apache-ant-1.7.1 ant

Also need to set JAVA_HOME and ANT_HOME
Add below entry in /etc/profile

export JAVA_HOME=/opt/java
export ANT_HOME=/opt/ant
export GLOBUS_LOCATION=/usr/local/globus-4.0.1
export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$GLOBUS_LOCATION/bin:$GLOBUS_LOCATION/sbin:$PATH
root@cognito:~# adduser globus
root@cognito:~# mkdir /usr/local/globus-4.0.1
root@cognito:~# chown globus:globus /usr/local/globus-4.0.1

Then, as user globus:
globus@cognito:~$ tar xzf gt4.0.8-all-source-installer.tar.gz

[Note]


Note



You might notice that I didn't install Postgres on this machine. That's because my grid can actually share the services of the RFT located on my first machine. Even if I weren't planning on that, I could add this new machine to the pg_hba.conf on the first machine and re-use the existing DB server.

3.2. Setting up your second machine: Installation

Now we can install from binaries:

globus@cognito:~/gt4.0.8-all-source-installer$ ./configure \
--prefix=/usr/local/globus-4.0.1
checking for javac... /usr/java/j2sdk1.4.2_10//bin/javac
checking for ant... /usr/local/apache-ant-1.6.5//bin/ant
configure: creating ./config.status
config.status: creating Makefile
globus@cognito:~/gt4.0.8-all-source-installer$ make
cd gpt-3.2autotools2004 && OBJECT_MODE=32 ./build_gpt
...
Binaries are much faster! This is done in less than 10 minutes.
...
tar -C /usr/local/globus-4.0.1 -xzf binary-trees/globus_wsrf_rft_test-*/*.tar.gz
tar -C /usr/local/globus-4.0.1 -xzf binary-trees/globus_rendezvous-*/*.tar.gz
echo "Your build completed successfully. Please run make install."
Your build completed successfully. Please run make install.
globus@cognito:~/gt4.0.8-all-source-installer$ make install
ln -s /usr/local/globus-4.0.1/etc/gpt/packages /usr/local/globus-4.0.1/etc/globus_packages
...
config.status: creating fork.pm
..Done

3.3. Setting up your second machine: Security

Now let's get security setup on the second machine. We're going to just add trust for the original simpleCA to this new machine, there's no need to create a new one. This is the multiple machines section of the SimpleCA guide.

Please make sure that your two machines agree on the time! These certificates have dates that tell you when they are valid. If your two machines don't agree about the time, you might get errors saying a certificate is not yet valid. If you use NTP, this won't be a problem.

globus@cognito:~$ scp choate:.globus/simpleCA/globus_simple_ca_ebb88ce5_setup-0.18.tar.gz .

globus@cognito:~$ $GLOBUS_LOCATION/sbin/gpt-build globus_simple_ca_ebb88ce5_setup-0.18.tar.gz
gpt-build ====> CHECKING BUILD DEPENDENCIES FOR globus_simple_ca_ebb88ce5_setup
gpt-build ====> Changing to /sandbox/globus/BUILD/globus_simple_ca_ebb88ce5_setup-0.18/
gpt-build ====> BUILDING globus_simple_ca_ebb88ce5_setup
gpt-build ====> Changing to /sandbox/globus/BUILD
gpt-build ====> REMOVING empty package globus_simple_ca_ebb88ce5_setup-noflavor-data
gpt-build ====> REMOVING empty package globus_simple_ca_ebb88ce5_setup-noflavor-dev
gpt-build ====> REMOVING empty package globus_simple_ca_ebb88ce5_setup-noflavor-doc
gpt-build ====> REMOVING empty package globus_simple_ca_ebb88ce5_setup-noflavor-pgm_static
gpt-build ====> REMOVING empty package globus_simple_ca_ebb88ce5_setup-noflavor-rtl
globus@cognito:~$ $GLOBUS_LOCATION/sbin/gpt-postinstall
running /usr/local/globus-4.0.1/setup/globus/./setup-ssl-utils.ebb88ce5..
[ Changing to /usr/local/globus-4.0.1/setup/globus/. ]
...
setup-ssl-utils: Complete
..Done
WARNING: The following packages were not set up correctly:
globus_simple_ca_ebb88ce5_setup-noflavor-pgm
Check the package documentation or run postinstall -verbose to see what happened

That installed the package, but the warning is letting us know that root still needs to run the setup script:

globus@cognito:~$ exit
root@cognito:~# source $GLOBUS_LOCATION/etc/globus-user-env.sh
root@cognito:~# $GLOBUS_LOCATION/setup/globus_simple_ca_ebb88ce5_setup/setup-gsi -default
setup-gsi: Configuring GSI security
Making /etc/grid-security...
mkdir /etc/grid-security
Making trusted certs directory: /etc/grid-security/certificates/
mkdir /etc/grid-security/certificates/
Installing /etc/grid-security/certificates//grid-security.conf.ebb88ce5...
Running grid-security-config...
nstalling Globus CA certificate into trusted CA certificate directory...
Installing Globus CA signing policy into trusted CA certificate directory...
setup-gsi: Complete

Now our new machine's security directory looks like our other machine:

root@cognito:~# ls /etc/grid-security/
certificates globus-host-ssl.conf globus-user-ssl.conf grid-security.conf
root@cognito:~# ls /etc/grid-security/certificates/
ebb88ce5.0 globus-user-ssl.conf.ebb88ce5
ebb88ce5.signing_policy grid-security.conf.ebb88ce5
globus-host-ssl.conf.ebb88ce5

Now we need a hostcert for the new machine:

root@cognito:~# grid-cert-request -host `hostname`
The hostname cognito does not appear to be fully qualified.
Do you wish to continue? [n] n
Aborting
...
If you receive no response, contact Globus Simple CA at bacon@choate
root@cognito:~# hostname
cognito

Uh-oh. Our hostname isn't fully qualified, which is going to cause us trouble down the road. If you have this problem, there are several possible solutions. One is to run the hostname command as root to set your FQDN as your hostname. Another possibility is that your /etc/hosts may have a short name listed for your IP address. Let's see what the problem is on cognito:

root@cognito:~# host 140.221.8.109

109.8.221.140.in-addr.arpa domain name pointer cognito.mcs.anl.gov.

If the problem had been in /etc/hosts, I would have fixed it. Here's what a good /etc/hosts line would look like:

140.221.8.109 cognito.mcs.anl.gov cognito

Since reverse lookups work okay, I will just spell out the FQDN by hand in this cert request:

root@cognito:~# grid-cert-request -host cognito.mcs.anl.gov -force
/etc/grid-security/hostcert_request.pem already exists
/etc/grid-security/hostcert.pem already exists
/etc/grid-security/hostkey.pem already exists
...
Your certificate will be mailed to you within two working days.
If you receive no response, contact Globus Simple CA at bacon@choate

The request already existed for "cognito", but the -force overwrote that request with one for "cognito.mcs.anl.gov". Now I need to copy that back to choate and sign it:

root@cognito:~# scp /etc/grid-security/hostcert_request.pem globus@choate:/home/globus

Now I sign it as globus on choate. Remember, that's where I installed the SimpleCA, so that's where I sign it:

globus@choate:/tmp$ grid-ca-sign -in hostcert_request.pem -out out.pem
To sign the request
please enter the password for the CA key:
The new signed certificate is at: /home/globus/.globus/simpleCA//newcerts/03.pem
globus@choate:/tmp$ scp /tmp/out.pem root@cognito:/root/

Root checks his email, then saves the signed cert:

root@cognito:~# cp out.pem /etc/grid-security/hostcert.pem
root@cognito:~# cd /etc/grid-security
root@cognito:/etc/grid-security# cp hostcert.pem containercert.pem
root@cognito:/etc/grid-security# cp hostkey.pem containerkey.pem
root@cognito:/etc/grid-security# chown globus:globus container*.pem
root@cognito:/etc/grid-security# ls -l *.pem
-rw-r--r-- 1 globus globus 2711 2005-11-15 11:14 containercert.pem
-r-------- 1 globus globus 887 2005-11-15 11:15 containerkey.pem
-rw-r--r-- 1 root root 2711 2005-11-15 11:14 hostcert.pem
-rw-r--r-- 1 root root 1405 2005-11-15 11:09 hostcert_request.pem
-r-------- 1 root root 887 2005-11-15 11:09 hostkey.pem

There. Now cognito is setup with host and container certs, and it trusts the CA of my grid. The last step for root is to create a grid-mapfile for myself again:

root@cognito:/etc/grid-security# vim grid-mapfile
root@cognito:/etc/grid-security# cat grid-mapfile
"/O=Grid/OU=GlobusTest/OU=simpleCA-choate.mcs.anl.gov/OU=mcs.anl.gov/CN=Charles Bacon" bacon

Also, user bacon should get a local copy of the usercert:

root@cognito:/etc/grid-security# su – bacon
cognito % scp -r choate:.globus .
Password:
usercert.pem 100% 895 0.9KB/s 00:00
usercert_request.pem 100% 1426 1.4KB/s 00:00
userkey.pem 100% 963 0.9KB/s 00:00

3.4. Setting up your second machine: GridFTP

GridFTP setup on the second machine is identical to the first. I'll just list the commands here, see Section 2.4, “Set up GridFTP” for the file contents, or just copy them from the first machine.

root@cognito:/etc/grid-security# vim /etc/xinetd.d/gridftp
root@cognito:/etc/grid-security# vim /etc/services
root@cognito:/etc/grid-security# /etc/init.d/xinetd reload
Reloading internet superserver configuration: xinetd.

Now we can test it:

root@cognito:/etc/grid-security# su - bacon
bacon@cognito:~$ source $GLOBUS_LOCATION/etc/globus-user-env.sh
bacon@cognito:~$ grid-proxy-init -verify -debug
User Cert File: /home/bacon/.globus/usercert.pem
User Key File: /home/bacon/.globus/userkey.pem
Trusted CA Cert Dir: /etc/grid-security/certificates
Output File: /tmp/x509up_u1817
Your identity: /O=Grid/OU=GlobusTest/OU=simpleCA-choate.mcs.anl.gov/OU=mcs.anl.gov/CN=Charles Bacon
Enter GRID pass phrase for this identity:
Creating proxy ...........++++++++++++
........++++++++++++
Done
Proxy Verify OK
Your proxy is valid until: Tue Nov 15 23:33:37 2005
bacon@cognito:~$ globus-url-copy gsiftp://cognito.mcs.anl.gov/etc/group gsiftp://choate.mcs.anl.gov/tmp/from-cognito

That was a slightly fancier test than I ran on choate. In this case, I did a third-party transfer between two GridFTP servers. It worked, so I have the local and remote security setup correctly.
3.5. Setting up your second machine: Webservices

Setting up the container on the second machine is a lot like the first. I'll list the commands here. See Section 2.5, “Starting the webservices container”, or you can just copy the files from the first machine. First globus creates the start-stop script:

globus@cognito:~$ vim $GLOBUS_LOCATION/start-stop
globus@cognito:~$ chmod +x $GLOBUS_LOCATION/start-stop

Then root creates an init.d script to call it:

root@cognito:~# vim /etc/init.d/globus-4.0.1
root@cognito:~# chmod +x /etc/init.d/globus-4.0.1
root@cognito:/etc/grid-security# /etc/init.d/globus-4.0.1 start
Starting Globus container. PID: 17269

3.6. Setting up your second machine: WS GRAM

For a change of pace, we'll setup GRAM first on the second machine, even though we haven't got a working RFT locally. As with last time, we'll need to setup the sudoers. See Section 2.7, “Setting up WS GRAM” for the sudo contents, or copy the sudoers from the first machine. If you just copy the file, please make sure that you have sudo installed already, and that the permissions are 440.

root@cognito:/etc/grid-security# visudo

Now we can submit a staging job:

bacon@cognito:~$ vim a.rsl
bacon@cognito:~$ cat a.rsl
bacon@cognito:~$ cat a.rsl
my_echo
${GLOBUS_USER_HOME}
Hello
World!
${GLOBUS_USER_HOME}/stdout
${GLOBUS_USER_HOME}/stderr


gsiftp://cognito.mcs.anl.gov:2811/bin/echo
file:///${GLOBUS_USER_HOME}/my_echo




file:///${GLOBUS_USER_HOME}/my_echo


bacon@cognito:~$ globusrun-ws -submit -S -f a.rsl
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:6732f346-5604-11da-9951-0002b3882c16
Termination time: 11/16/2005 18:19 GMT
Current job state: StageIn
Current job state: Active
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
Cleaning up any delegated credentials...Done.
bacon@cognito:~$ cat ~/stdout
Hello World!
bacon@cognito:~$ ls ~/my_echo
ls: /home/bacon/my_echo: No such file or directory

This is an example of a staging job. It copies the /bin/echo command from cognito to my home directory and names it my_echo. Then it runs it with some arguments, and captures the stderr/stdout. One of the neat features here is that it used the RFT service on choate to transfer the file via the GridFTP server on cognito. It's starting to look like a Grid!

If you got an "invalid executable: my_echo" error, make sure you staged the echo command to the machine where you're executing the job. One way to get this error is by running this on cognito, but staging the file to choate instead. The stageIn will work, but the job will fail because you staged to the wrong host.

Also, this won't work if your cognito and choate have different processor architectures. For instance, you can't run an ia64 binary on an ia32 computer. In that case, you could try staging in a shell script instead.

You can get other examples of GRAM RSL files from GRAM usage scenarios.

Also, you can submit from one machine to another using the -F (factory) option. For instance, to submit from cognito to choate:

bacon@cognito:~$ globusrun-ws -F choate -submit -c /bin/true
Submitting job...Done.
Job ID: uuid:0efba320-4780-11dc-bd6b-0007e9d811ce
Termination time: 08/11/2007 20:27 GMT
Current job state: Active
Current job state: CleanUp
Current job state: Done
Destroying job...Done.

Likewise, you could login to choate and submit a job to cognito.