DIVERSE Adaptable Display System (DADS 1.1)

1. Overview

This section of the DIVERSE Workshop will cover the use of the DADS system written by Andrew Ray and Patrick Shinpaugh. DADS is a Linux cluster solution to drive multiple walls of a CAVE/RAVE system though it can be used on a single desktop or multiple desktops for testing or other purposes. DADS 1.1 includes support for additional environment variables to support DGL.

The DADS system is not entirely stable, though a new version will be in development soon. The new version will (hopefully) improve stability and deal with some issues which reduce the ease of use important in all DIVERSE components. These issues will be discussed later.

2. Usage

Environment Variables

There are several environment variables used with the cluster solution. Some are new and some are existing environment variables from DPF and DTK. All environment variables are set on the console/server machine and will be passed to the clients as needed.

DTK_CLUSTER_CLIENTS (Formerly DPF_CLUSTER_CLIENTS) – a colon separated list of hosts which should all be running the dListener daemon.

DTK_CLUSTER_SERVER (Formerly DPF_CLUSTER_SERVER) – the host name of the console/server which should not be running dListener. This environment variable should be set to the hos name of the cosole/server machine.

DTK_DSO_FILES – From DTK. Use this on the console/server side to specify the DTK DSOs to be loaded on the server.

DPF_DSO_FILES – From DPF. Use this on the console/server side to specify the DTK/DPF DSOs to be loaded on the server.

DTK_CLUSTER_DSO_FILES – Specify the DTK DSOs to be loaded by all clients.

DPF_CLUSTER_DSO_FILES – Specify the DTK/DPF DSOs to be loaded by all clients.

DTK_CLUSTER_DSO_FILES_host – Specify the DTK DSOs to be loaded specifically for host where host is the base or alias of the hostname ( i.e. The host for somehost.example.net would be somehost ).

DPF_CLUSTER_DSO_FILES_host – Specify the DTK/DPF DSOs to be loaded specifically for host where host is the base or alias of the hostname ( i.e. The host for somehost.example.net would be somehost ).

Daemons

dListener
The daemon listens for messages on a shared memory segment. The messages are for starting a program, stopping a program, setting an environment variable, unsetting an environment variable, and checking the status of the listener daemon. The dListener must be running on all client machines or no actions will be taken on that client and the application will lock on the server and clients where it does spawn as the syncing requires communication from all clients.

dRepeat
This daemon will repeat shared memory segments on another named segment. It sleeps between reads and writes to reduce CPU usage and give other processes time.

Applications

dListener-stop
This application stops the dListener on all machines identified in the DTK_CLUSTER_CLIENTS environment variable.

dKiller
This application is used to kill processes on client machines which are no longer functioning properly. The dListener daemon must be running on the client machine for this application to function properly.

dSetEnv
This application is used to set environment variables on client machines which are running the dListener daemon.

dUnsetEnv
This application is used to unset environment variables on client machines which are running the dListener daemon.

dCreateShm
This application is used to create shared memory segments or messaging post offices. It can also be used to create the standard cave shared memory segments head, wand, joystick, and buttons.

dExec
This application can be used to start applications remotely on a client running the dListener daemon.

DSOs

starter
The starter DSO will connect to the message post offices of all clients defined in the DPF_CLUSTER_CLIENTS environment variable and send messages telling all clients running dListener to start the application. The message wille consist of the same command line arguments and will also pass the DPF_CLUSTER_DSO_FILES to the client as DPF_DSO_FILES. The starter DSO is required for the cluster solution to work properly. It is the method by which the client machines running dListener are told to start a program.

death
The death DSO is used to exit the client applications normally when the console/server application has been requested to exit normally. This should be included in the DPF_CLUSTER_DSO_FILES environment variable used on the client machine

navWrite
The navWrite DSO uses a software syncing solution to synchronize the location and head tracker position with clients running the navRead DSO. Must be used in conjunction with the navRead DSO or another navigation DSO implemented to synchronize similarly to the navRead DSO.

navRead
The navRead DSO uses a software syncing solution to retrieve the location and head tracker position with the console/server machine running the navWrite DSO. Must be used on the client in conjunction with a server running the navWrite DSO or another navigation DSO implemented to synchronize similarly to the navWrite DSO.

nvswapbarrier
This DSO provides swap barrier support for NVidia graphics cards supporting framelock and the NV_swap_group extension (i.e. NVidia Quadro FX 3000G). Unfortunately it only works for monoscopic displays. The NVIDIA drivers for the Quadro FX 3000G do not work properly for the swap barrier and they are only available from the PNY Technologies website (http://www.pny.com). Otherwise it would provide synchronization of the swap resulting in displays updating simultaneously even in stereo mode.

dHideCursor
This DSO hides the cursor within the window associated with the application while it is running.

dPerformance
This DSO logs performance to a file called fps.log which it stores in the local /tmp directory. The file contains performance fps information for every 60 frames, and the totals when the application is stopped.

vtCaveClusterGroup
This DSO provides all of the functionality necessary to run the DADS system from the console machine. You should modify the server and client host names to match your CAVE/RAVE system before building this DSO.

vtCaveClusterClientGroup
This DSO provides all of the functionality necessary for the clients to run properly in the cave with the exception of the display. Displays must be specified for each client. Includes xkeyboardMouseInput, caveDTKInput, setHeadView, caveSim, navRead, death, dHideCursor, toggleObjectsGroup, toggleScreenFrame, and debugHeadsUp DSOs.

vtCaveDisplayFront
The standard display provides fullscreen 1280 x 1024 resolution at 0 degree offset from center with stereo support.

vtCaveDisplayFloor
This display provides fullscreen 1280 x 1024 resolution at -90 degree rotation about the x axis with stereo support.

vtCaveDisplayLeft
This display provides fullscreen 1280 x 1024 resolution at 90 degree rotation about the z axis with stereo support.

vtCaveDisplayRight
This display provides fullscreen 1280 x 1024 resolution at -90 degree rotation about the z axis with stereo support.

vtCaveMono*
Each of these DSOs is the same as the vtCave* DSOs above but only provides monoscopic displays.

3.Client/Server Architecture

DADS uses a client/server architecture to control and synchronize multiple displays. A dListener server must be started on each client machine. The starter DSO contacts each machine identified in the DTK_CLUSTER_CLIENTS environment variable and passes the necessary environment variables as well as the command line arguments which initiated the call. It is necessary that the server side also use the navWrite DSO and that the client use the navRead DSO.

The navWrite DSO waits to hear from each client’s navRead. Each navRead sends a message to the navWrite server DSO and then waits to receive the current head, wand, joystick, button, and location shared memory data. This results in a synchronized location, head, and wand position/orientation for each as well as the same data for button and joystick so events will occur on all displays at the same (approximate) moment in time (i.e. events will not be lost).

The death DSO is necessary for client programs to at least attempt to exit normally. When the escape key is pressed for the console program it will exit and send a message to the dListeners of each client which will then send a hangup signal to each process it spawned.

4. Example

An example (for groups of 2 or more people) on the desktop:

One machine is set up as the server. All other machines are set up as clients. Each client must be set up before the the program is run.

CLIENT:

dtk-server –d
dListener --log

SERVER:

dtk-server-d
export DTK_CLUSTER_SERVER=serverhostname
export DTK_CLUSTER_CLIENTS=client:names:separated:by:colons
export DPF_DSO_FILES=desktopGroup:starter:navWrite
export DPF_CLUSTER_DSO_FILES=desktopGroup:death:navRead
diversifly sub.pfb

On the server/console program window press the left mouse button and move the mouse up and down to move forward and back or left and right to rotate left and right. The same action should occur on the To stop the program, from the server machine highlight the program window and press the escape key. The program window on the server should close followed by the program window on each client within several seconds. If this does not occur you will need to open another terminal window and type

killall –9 diversifly

To stop the dListener daemon use the

dListener-stop

command from the server or the client. To stop the dtk-server, use the

dtk-shutdownServer

command. To remove all dtk shared memory files use

dtk-destroySharedMem -r

To remove all semaphores (inter-process communication used for shared memory) that may have built up, use the

rmsem.sh

script. The ordering can be important. If the semaphores are removed prior to stopping the dListener it is possible the dListener will start processes continuously resulting a machine lockup unless you are quick and able to shut it down. Other problems could also occur.

For the CAVE, you can simply use a pre-existing DSO which will handle all of the typical DSOs for you.

export DPF_DSO_FILES=vtCaveClusterGroup
diversifly sub.pfb

To handle cleanup of possible problems a script was written to reset the console and DADS machines. From the command line call

dadsresetall

For full control over the DSOs loaded for each display it is recommended that you create a script.

export DPF_DSO_FILES=desktopCaveEmulateGroup:starter:navWrite:dPerformance
export DPF_CLUSTER_SERVER=console
export DPF_CLUSTER_CLIENTS=dads1:dads2:dads3:dads4
export DPF_CLUSTER_DSO_FILES=vtCaveClusterClientGroup
export DPF_CLUSTER_DSO_FILES_dads1=vtCaveDisplayFront
export DPF_CLUSTER_DSO_FILES_dads2=vtCaveDisplayFloor
export DPF_CLUSTER_DSO_FILES_dads3=vtCaveDisplayRight
export DPF_CLUSTER_DSO_FILES_dads4=vtCaveDisplayLeft
diversifly sub.pfb

5. Issues  -  For those who want to know...

The dListener daemon will on occasion continuously spawn processes possibly locking up the machine and requiring reboot. If this happens carefully type dListener-stop, press the enter key and hope… It is likely you will not see the characters as they are typed and waiting increases the risk that the system will lock. If you are lucky the dListener will stop and it should kill all of the spawned processes at the same time.

Semaphores continuously build up as a result of Performer for Linux (allegedly). There is a maximum number of semaphores which a system can create before it refuses to create more. This mostly happens when the client programs do not exit normally and is more likely to occur on the client machines. Currently the dListener daemon will automatically kill the process after a set time.

On occasion the dListener will fail to stop/kill the process and the machine will end up blocking all subsequent requests to launch a program. Use the dadsresetall script to reset all of the DADS machines. Unfortunately it will only automatically kill diversifly programs. For other programs you will likely have to ssh into the offending machine and kill the process manually.

The dadsresetall script is not 100% effective 100% of the time as mentioned above, but also because the dtk-server on the console machine is owned by nobody and will not always kill the dtk-server or destroy the DTK shared memory. Also, the semaphores are owned by the user who created them and can only be destroyed by the same user or by root.

Variables which might be needed by custom DIVERSE applications or DSOs cannot be easily shared except through DTK shared memory resulting in a greater number of semaphores being created and more likely that a lock will occur sooner. Also, it becomes less transparent to the programmer who will then be required to handle variable sharing specifically for the DADS system if necessary (tedious and/or problematic).

There are other issues but these are the major ones that need to be handled in the next version of DADS.

6. DADS Demonstration and Walkthrough

Switch displays between the SGI and DADS system. (SGI - PRESET 1, DADS - PRESET 2).
Turn on the CAVE wall projectors (DO NOT TURN OFF PROJECTORS - put into and take out of STANDBY mode).
Log into the system.
Set up PATH to access DIVERSE binaries.
Run an example.
See if I can break it... shouldn't be too difficult... and discuss troubleshooting.
If all else fails... contact me...

7. Problems

8. Thanks for attending the DIVERSE Workshop 2004