Created by: Harris Georgiou - mailto:xgeorgiou@yahoo.com
University of Athens - Department of Informatics,
Postgraduate Program (MSc),
Digital Signal Processing & Computer Systems.
This page describes the structure and functionality of the SDGS application, aimed as a tutorial application for the introduction of Remote Method Invocation (RMI) technologies in Java, for the development of a simple distributed group service. All package compiled and source files can be downloaded in one .zip file archive:
- SDGSpack.zip (~20K)
As always, all software is distributed fro free use under the terms of MPL-Mozilla-v1.1 licence. For any comments, questions or bug reports, feel free to contact the author.
The SNS application is an example of using the Java/RMI framework in order to implement a simple distributed group service that runs as a distributed database for records in form of key-value pairs. The application is employed according to the client-server model and the basic function provided is:
- lookup ( keyID )
The lookup function can be used by clients in order to post a request to the server
group and to aquire any available data related to that specific keyID. The main idea
behind this service is the availability of multiple servers in a loosely connected
redundant server group. A client should be able to connect to any one server and post some
data query, activating one or more servers that look in their local DB for results. The
final answer should come from the first server that finds the requested data, or a
negative answer from all active servers in case of negative results.The service is
implemented in a transparent way, so that the client program does not participate in the
distribution of the DB or the synchronization of the multiple server nodes.
The implementation of the SDGS services are based on the RMI framework for remote method calling from the clients. The core issue for the service is the availability of multiple servers to which each client could randomly connect to post his/hers queries to the complete DB. Distribution management of the requests on the DB should be handled either by the clients (sending individual requests to all active servers), or internally by the server group.
Distribution implementation and management was planned on two different distribution models. The first one is related to the connection of the client with one of the active servers by random selection, in order to post the initial request. The entry server runs a lookup on the local DB and tries to resolve the query. If successful, the result is returned immediately to the client and the session is completed, otherwise it forwards the query to all the other servers that has direct accessing references. Second level forwarding is prohibited to avoid network congestion. Finally, the server who received the initial post gathers all the answers from the server group and returns the results to the client, i.e. one possitive answer or one (global) negative answer. In every stage of processing, the client is in direct contact only with the entry server, so in this case the distribution management is an internal issue for the server group.
The second distribution model is related to the distribution management by the client. Each client has updated information on how to get in contact with each member of the group service and randomly decides which one will receive the initial request. In case of negative answer or connection failure, the client re-sends the same request to the next available server, continuing like this until a positive answer is received or all servers report negative results or failure. In this model, each server acts autonomously in all stages, as the choice of node selection is made by the client.
From these two possible approaches, the second one was employed in the final
implementation. Although network traffic volume is roughly the same in both cases, the
first model offers simplicity in the client design versus complexity in the server design
in terms of synchronization, not desirable for scalable distribution. Most importantly,
during processing within the group, the entry server acts like a coordinator in the sense
of master-slave model, a parameter that destroys the very sense of service and data
redundancy within the group. These two factors, along with the fact that in this case the
choice of random selection and communication failures are also a within-group issue to be
resolved, makes this model vulnerable to single-node failures and difficult to scale up to
larger groups. On the other hand, in the second model the complexity of the client program
is weighted against the viability of the service even down to one server node, with little
or no (if the DB is read-only) synchronization overhead on the server side (i.e. easily
scalable). Thus, each server processes the requests posted to itself strictly on local
basis and any possible failure or removal from the group does not affect the intrinsic
functionality of the service. The scaling cost is transferred directly to the clients,
i.e. externally to the service, so that the group may contain any arbitrary number of
server nodes. In order to avoid large overheads for the clients, each client program may
use a localized (limited) list of active server nodes that should be used as entry points,
so that the service becomes a distributed service with localized processing.
For the specific application, the total DB is considered static (no update functions)
and each local DB is transferred to the main memory of each server to the full. Data
organization and management is implemented using the classic model of key-value pairs, and
all queries refer to searches on the key field (keyID) of each record.
The source code of the application is made up from two source packages. The first one refers to the server program and the second to the client program. Shared files between the client and the server programs are originally attached by convention to the server package, as their dependency to the server program is greater than to the client and any update/build process becomes easier.
SDGS Client Package
-------------------
SDGSclient.java main client program
SDGStimeout.java comm setup for clients
(SDGSexception)
(SDGSnotFoundException)
(SDGSkey)
(SDGSvalue)
(SDGSrecord)
(SDGSlist)
(SDGSremote)
(SDGSserver_Stub)
[SDGSlist.cfg] active server list
SDGS Server Package
-------------------
SDGSserver.java main server program
SDGSsettings.java server initialization setup
SDGSlist.java active server list management
SDGSdatabase.java DB management (local)
SDGSkey.java keyID field, DB records
SDGSvalue.java value field, DB records
SDGSrecord.java Db record (SDGSkey+SDGSvalue)
SDGSremote.java server remote interface
SDGSexception.java generic exception
SDGSnotFoundException.java not-found exception
(SDGSserver_Skel)
[SDGSserver.ini] server init file
[SDGSlist.cfg] active server list
[SDGSdatabase.dat] DB content file
[SDGSserver.cfg] current server configuration
The figure above presents the contents of the client and server packages. The files
included in braces [] are data files in text format that are used by the programs. Files
SDGSserver_Stub and SDGSserver_Skel are the stub and skeleton server modules accordingly,
generated automatically by the RMI interfaces compiler. Aside from files SDGSclient and
SDGStimeout related directly only to the client program, all other files are considered
part of the server program package. File SDGSlist.cfg contains the nameIDs of all the
servers in the group and it's management functions are included in class SDGSlist. The
active server list can be global or localized, as described above, and can be included as
a copy or posted in a shared path between the servers and the clients. That same file is
used by the server programs to pick an available nameID for their RMI registry record.
Each client who wishes to contact the service uses the active servers list from file
SDGSlist.cfg and randmly selects one of them. Next, it locates the destination node
(address/port) from the RMI registry using the nameID of that server, to shich it sends
the request. In case of a negative answer of connection failure, the process in repeated
for the remaining nameIDs in the list, until a positive answer is received all servers
report netagive answer or failure.
Each server node runs a mirror copy of the same server program. As each active server should be registered in the RMI registry with a different nameID, there has to be some basic coordination in picking some nameID at startup. Thus, SDGSlist is used to randomly pick a nameID from the list, not previously taken by another server instance, and use it as it's own nameID. Current nameID, current address/port and group configuration is saved by each server in a local file for loggin purposes.
There are two group configurations available. The first one provides startup of multiple server instances on the same node, thus the group practically becomes a multi-process service with multiple copies of the same functionality on the same machine. The second configuration employs startup of multiple server instances on different nodes, using a shared RMI registry service for naming and group management. The exact group configuration, along with the address/port information for the shared registry, is defined through the SDGSserver.ini file and managed by the class SDGSsettings. As with the case of SDGSlist.cfg, SDGSserver.ini file can also be copied or posted on a shared path in order to be accessed by all servers at startup.
For practical reasons, especially in the case of multi-process group configuration, the local files (DB and configuration) of each server instance are stored in separate directories with the according server nameID, inside the initial startup path. Thus, multiple server instances can be started and observed using only one node, as private storage for each instance provides autonomous environment for the servers. In case of true distributed group configuration this setup is not necessary.
The application contains the client and the server program packages. For the corrent compilation of both packages, the following procedure is proposed:
á) Compile the server program:
/svr: javac *.java
â) Create the stub and skeleton modules:
/svr: rmic SDGSserver
ã) Copy all dependencies to the client path:
copy *.class: SDGSexception, SDGSnotFoundException, SDGSkey,SDGSvalue, SDGSrecord, SDGSremote, SDGSlist,SDGSserver_Stub
ä) Compile the client program:
/cli: javac *.java
å) Setup the group configuration:
/svr: vi SDGSserver.ini (reglocal,reghost,regport)
óô) Setup the active servers list:
/svr: vi SDGSlist.cfg (serverID names)
æ) Setup the DB content:
/svr: vi SDGSdatabase.dat (keyID-value pairs)
ç) Setup the private paths for server instances (multi-process configuration only):
/svr: md xxxxx (xxxxx = serverID)
è) Setup all local DBs in private paths (multi-process configuration only):
/svr: vi xxxxx/SDGSdatabase.dat
Due to some technical problems with v1.2 of Sun's Java SDK in Windows platforms, in order for the RMI registry service to function with no problem, rmiregistry executable may have to be copies and executed from the server program path. When the RMI registry service is up and running, the SDGS server program can be started. Local address and port number are displayed and the new SDGSserver.cfg configuration file is created locally for each server instance, in the current path (distributed setup) or in the related private path (multi-process setup). Finally, multiple clients can be initialized and started, using the active servers list and the RMI registry to locate any active server. Each client can select one of these servers to post a request. The session is completed as soon as the first positive answer is received or all servers report negative answers or failure. Any connection problems are dealt with using two levels of timeouts, one for the server list and one for server connections, the exact values of which are defined in class SDGStimeout. Default values for server connections have been established to 5 retries for request phase and 3 retries for reply phase. Between each retries there is a static delay of 100ms plus a random delay of another 0-200ms in order to avoid multpiple collisions with other clients. Finally, the client receives one positive answer from the current server or iterates through the complete list of available servers and receive negative results or failure from all of them. Each search within the (total) distributed DB can be easily traced by the user, even in multi-process (localized) group configuration, by setting the "verbose" parameter to "true" on the client program's configuration file, thus providing detailed status messages regarding the communication between the client and the servers.
Development Platform:
Last updated: 15-Apr-2000.
Harris Georgiou
http://xgeorgio.n3.net
mailto:xgeorgiou@yahoo.com