NCSI Parallel/Cluster Workshop - Parallel Protocols

Overview

• Message passing
• Shared memory/SMP
• Process/workload management

PVM

• Parallel Virtual Machine, from Oak Ridge National Laboratory
• Message passing paradigm
• Complete package (rather than a specification ala MPI)
• Language bindings for C, C++, FORTRAN, Perl, and other languages
• Development has mostly ceased although it is a very mature package
• Many books, etc. available that use PVM as the implementation API
• Notion of a virtual machine, users start the virtual machine (pvmd) on all the computers on which the parallel program will run and then start the program which calls the PVM API
• http://www.csm.ornl.gov/pvm

• Parent, hello.c:

  	#include "pvm3.h"
  	
  	main() {
     		int cc, tid, msgtag;
     		char buf[100];
  
  		printf("i'm t%x\n", pvm_mytid());
  
  		cc = pvm_spawn("hello_child", (char**)0, 0, "", 1, &tid);
  
  		if (cc == 1) {
  			msgtag = 1;
  			pvm_recv(tid, msgtag);
  			pvm_upkstr(buf);
  			printf("from t%x: %s\n", tid, buf);
  		} else
  			printf("can't start hello_child\n");
  		
  		pvm_exit();
  	}
  	

• Child, hello_child.c:

  	#include "pvm3.h"
  	
  	main() {
  		int ptid, msgtag;
  		char buf[100];
  		
  		ptid = pvm_parent();
  		
  		strcpy(buf, "hello, world from ");
  		gethostname(buf + strlen(buf), 64);
  		msgtag = 1;
  		pvm_initsend(PvmDataDefault);
  		pvm_pkstr(buf);
  		pvm_send(ptid, msgtag);
  		
  		pvm_exit();
  	}
  	

OpenMP

• Open multi-processing, from the OpenMP industry consortium
• Specification that is supported by many compilers (C, C++ and FORTRAN) which automates data partitioning and process management through compiler directives/pragmas (C, C++) or special comments (FORTRAN) placed in the source code, many compiler vendors support OpenMP
• Shared memory/SMP paradigm, behind the scenes threads are managed by the runtime environment
• Many vendors (e.g.Sun) support similar proprietary mechanisms for their compilers, OpenMP is an attempt to unify and standarize this approach
• www.openmp.org

• Example:

  	program OMP_PRINT
  	
  	implicit none
  	
  	#include "f_hpm.h"
  	
  	integer :: myid, nthreads
  	integer :: OMP_GET_NUM_THREADS, OMP_GET_THREAD_NUM
  	
  	call f_hpminit(0,"hpmfile")
  	
  	!$OMP PARALLEL default(none) private(myid) &
  	!$OMP shared(nthreads)
  	
  	! Determine the number of threads and their id
  	myid = OMP_GET_THREAD_NUM()
  	nthreads = OMP_GET_NUM_THREADS();
  	
  	!$OMP BARRIER
  	
  	call f_hpmtstart(1+myid,"printcounter")
  	
  	print*,'myid = ', myid, ' nthreads ', nthreads
  	
  	! Some actual work is done in parallel here
  	
  	call f_hpmtstop(1+myid)
  	
  	!$OMP END PARALLEL
  	
  	call f_hpmterminate(0)
  	
  	end program OMP_PRINT
  	

SYSV Interprocess Communication

• Operating system API supported by almost every Un*x variant
• Shared memory, pipes, and semaphores
• Notion of a process: private stack, heap, instructions, program counter, and file descriptors; IPC makes it possible for processes to communicate and share data
• Oldest form of parallelism, not used for new projects very often except when particular aspects of the problem require it
• fork() and exec() model
• SYSV IPC over TCP/IP

Threads

• Operating system API supported by almost every Un*x variant, some standardization in the form of POSIX thread libraries
• Shared memory/SMP paradigm
• Notion of a thread: private stack and program counter, shared heap, instructions, and file descriptors; thread libraries provide primatives to support creation and management of threads

JavaSpaces

• Service that provides a high-level tool for creating collaborative and distributed applications in Java
• Persistant object exchange areas through with Java processes coordinate their actions and exchange data
• Remote agents interact with each other indirectly through shared data object spaces, there can be many spaces and clients
• Shared space operations:
  • read - read an entry from a space
  • write - add an entry to a space
  • take - read and remove an entry from a space
  • notify - send a notification if an entry that matches a pattern is added/modified/removed in a space

OpenMosix

Condor

• Workload management system that supports high throughput computing, good for jobs that require large amounts of computational power over a long period of time
• Makes it (relatively) easy to leverage idle cycles on a pool of machines to support long running jobs without interfering with other demands (i.e. interactive use) on those machines
• Queueing mechanism, scheduling policy, priority scheme, resource monitoring, I/O redirection, checkpointing, and job migration
• Very mature technology
• http://www.cs.wisc.edu/condor

How to decide which is best?

• Depends on the compute resources available, the problem, and the software available for your problem space
• Clusters work well with message passing, SMP machines with shared memory models