GeneralComms Documentation

Maintained by: David Austin

1.0 Introduction
1.1 Definitions
1.1.1 Channel
1.1.2 Server
1.1.3 Callback
2.0 GeneralComms Tutorials
2.1 Connecting to Another Process
2.2 Sending to Another Process
2.3 Receiving from Another Process
2.4 Writing a Server (Receiver of Connections)
3.0 GeneralComms and the NameServer
4.0 Connectionless Channels

1.0 Introduction to GeneralComms

The goal of the GeneralComms library is to provide a (fairly) general library for communications. Note that the GeneralComms library does not have the generality (or the overhead) of packages such as ACE. GeneralComms is designed to work on UNIX-like operating systems (stretching that to include QNX) but should be portable to other operating systems.

Note: you do not have to use GeneralComms - it is possible to communicate with any process within DROS using whatever communications library you wish, so long as you obey the message formatting protocol.

1.1 Defintions

1.1.1 Channel

Any form of communications channel where messages can be transmitted. For example, obvious channels are sockets allowing communication between two computers over TCP/IP. However, the channel idea includes two threads within the same process communicating (which hopefully, should be implemented efficiently by just passing pointers). An expanding variety of different communications channels are available.

1.1.2 Server

A class which encapsulates the ability to listen for and accept new connections (whatever the communications channel).

1.2.3 Callbacks

For large, complex software systems, it is necessary to use an event based programming technique. The method for GeneralComms to notify your programs of events (such as receiving a message) is to call a procedure (a callback). There are two methods for callbacks in C++: a static function and a member function of a class.

A static function has the disadvantage that it cannot be associated with a particular instance of a class (hence there is no this). However, the use of member functions makes it difficult to handle events from multiple sources since the same function is called for every event of that type.

Static function callbacks

A static function has the disadvantage that it cannot be associated with a particular instance of a class and so the use of the this variable or any member variables of the class is prohibited. However, one can access the class by passing a pointer to it: that in the example. Also, if the static function is defined as a member of the class then it can access the private variables through a pointer to the class (the variable that in the example).

Here is an example of a static function callback:

class test { public: test(); static void timerHandler(GCEventLoop *loop, long id, test *that) { cerr << "Timer event " << id << " occured\n"; if (that->value == 1) { loop->FinishEventLoop(); } } private: int value; };

Member function callbacks

A base class is defined with an (empty) member function which is called whenever the event occurs. A user can them overload the member function and so their code will be called when the event occurs.

Here is an example of a member function callback:

class MyClass : public GCTimerCallback { public: MyClass(); void timerCallback(GCEventLoop *loop, long id) { cerr << "timer " << id << " occurred!\n"; } };

2.0 GeneralComms Tutorials

2.1 Connecting to Another Process

Connecting to another process is usually quite straightforward. The other process must have a server and must register itself with the NameServer (and so, the NameServer must be running).


  #include "GCConnect.h"
  


  GCChannel *chan;

  chan = Connect(loop, "ServerNameToConnectTo");
  if (chan == NULL) {
    DROS_DEBUG(1, "Failed to connect to ServerNameToConnectTo");
    return -1;
  }

In this example loop is a pointer to the main GCEventLoop. In many cases, you can write GetGlobalResources()->GetLoop() to get this pointer.

There are also a number of other connection methods which can be used if you need more control or don't want to use the NameServer. Look at GCConnect.h for more details.

2.2 Sending to Another Process

Sending to another process is straightforward:


  //First, we must set up the data in the message.  The Packer is used
  //for this.
  char m[sizeof(LogInMsg)];
  int size;

  if (pack_msg_hdr_all(m, sizeof(m), LogIn, LITTLE_ENDIAN_ENCODING, 
 	               GetGlobalResources()->GetNextMsgId()) < 0) {
    DROS_DEBUG(20, "Failed to pack_msg_hdr_all");
    return -1;
  }

  if (pack_msg_data(m, sizeof(m), NULL, 
	  	    LOGIN_FMT_STR, name, passwd) < 0) {
    DROS_DEBUG(20, "Failed to pack_msg_data");
    return -1;
  }

  size = finalise_msg(m, sizeof(m));
  if (size < 0) {
    DROS_DEBUG(20, "Failed to finalise_msg");
    return -1;
  }


  //Now that we have the data, send it off.
  if (chan_->Send(size, m) == size) {
    //Success
    return 0;
  } else {
    DROS_DEBUG(500, "Send error ");
    return -1;
  }

2.3 Receiving from Another Process

Receiving from another process is slightly complex because of the ansynchronous nature of receiving. We cannot predict when our program will receive a message. Therefore, is is necessary to register a handler with the event loop to be called when a message is received.

class MyTest { public: MyTest(); int Setup(GCEventLoop *loop, char *serverName) { chan_ = Connect(loop, serverName); if (chan_ == NULL) { return -1; } loop->AddChannel(chan_); loop->AddChannelEventCallback((ChannelEventHandlerType) HandleReceive, chan_, GC_EVENT_READ, this); return 0; }; /** * This procedure deals with incoming messages. Note that it * must be declared static. */ static int HandleReceive(GCEventLoop *loop, GCServer *s, GCChannel *c, int event, long id, MyTest *that) { unsigned int mtype; unsigned int msgid; if (check_checksum(c->GetBufPtr(), c->DataPending()) < 0) { DROS_DEBUG(2, "Received bogus message, dropping it"); return GC_HANDLER_EXCLUSIVELYMINE; } if (extract_msg_hdr_all(c->GetBufPtr(), c->DataPending(), &mtype, NULL, NULL, &msgid) < 0) { DROS_DEBUG(2, "Failed to extract message type!!, dropping it"); return GC_HANDLER_EXCLUSIVELYMINE; } DROS_DEBUG(1000, "Received message of type " << MessageTypeToString(mtype) << " len = " << c->DataPending()); switch (mtype) { case Ack: DROS_DEBUG(2, "Received an ACK message"); /* Further processing of the message here */ break; default: return GC_HANDLER_NOTMINE; } return GC_HANDLER_EXCLUSIVELYMINE; } private: GCChannel *chan_; };

2.4 Writing a Server (Receiver of Connections)

Writing a server requires a few steps:

Create a server object for the type of connections that you want to receive.
TCPSocketServer s;
Add the server to the event loop and to the list of components (this latter makes sure that the server is set up before the whole system starts running).
loop.AddServer(&s); loop.AddToComponentList(&s);

Add callback routines which get called in the event that someone tries to connect to your server (not always necessary). It is only necessary if you need to respond to the connect event itself i.e. by creating a client class for each new client or to send each new client a welcome message.


int callback2(GCEventLoop *loop, GCServer *s, GCChannel *c, int event, long id,
	      void *userdata) {
  DROS_DEBUG(1, "Channel event " << event << " on fd " << c->GetFd());
  return GC_HANDLER_NONEXCLUSIVELYMINE;
}


int callback(GCEventLoop *loop, GCServer *s, GCChannel **c, int event, long id,
	     void *userdata) {
  DROS_DEBUG(1, "CONNECT EVENT!");
  loop->AddChannelEventCallback((ChannelEventHandlerType) callback2,
				*c, GC_EVENT_READ, NULL);
  return GC_HANDLER_NONEXCLUSIVELYMINE;
}



  s.AddConnectCallback((ConnectEventHandlerType) callback,
		       &s, GC_EVENT_CONNECT, NULL);

Note that callback2 is registered for each of the new connections as they come in. It is also possible to register a single handler for all read events if you wish.

The final step is to makes sure that your server gets registered with the NameServer so that other processes can find it:
s.SetMustBeRegistered("MyServerName");

So, to put it all together, we have:


#include "TCPSocketServer.h"
#include "GCEventLoop.h"
#include "DROSDebug.h"


int callback2(GCEventLoop *loop, GCServer *s, GCChannel *c, int event, long id,
	      void *userdata) {
  DROS_DEBUG(1, "Channel event " << event << " on fd " << c->GetFd());
  return GC_HANDLER_NONEXCLUSIVELYMINE;
}


int callback(GCEventLoop *loop, GCServer *s, GCChannel **c, int event, long id,
	     void *userdata) {
  DROS_DEBUG(1, "CONNECT EVENT!");
  loop->AddChannelEventCallback((ChannelEventHandlerType) callback2,
				*c, GC_EVENT_READ, NULL);
  return GC_HANDLER_NONEXCLUSIVELYMINE;
}

int main() {
  const char *dros_function_name = "main";
  dros_program_name = "TestEventLoop";

  GCEventLoop loop;
  TCPSocketServer s;

  loop.AddServer(&s);
  loop.AddToComponentList(&s);

  s.AddConnectCallback((ConnectEventHandlerType) callback,
		       &s, GC_EVENT_CONNECT, NULL);

  s.SetMustBeRegistered("spam");

  if (loop.EstablishConnections() < 0) {
    DROS_DEBUG(2, "EstablishConnections failed, aborting");
    return -1;
  }

  loop.EventLoop();
  return 0;		
}

3.0 GeneralComms and the NameServer

One of the functions that GeneralComms provides is independence from hard-coded hostnames and addresses for communications. For example instead of saying "connect to host forde.anu.edu.au at port 12345" to find a particular process (say the RobotServer), we can say "connect to the RobotServer". How is this implemented? Well, when everything is working, we don't need to know how things are implemented but, unfortunately, sometimes things will not work and you will need to figure out what has gone wrong. Therefore, this section documents the relationship between GeneralComms and the NameServer.

The NameServer is a server that associates names with addresses. For example, when the RobotServer starts, it locates the NameServer and tells it "hey, I'm the RobotServer, I'm on host forde.anu.edu.au at port 12345...". Then, when another process wants to find the RobotServer, it talks to the NameServer and asks it where the RobotServer is.

4.0 Connectionless Communications

Connectionless communications such as UDP and QNX messages pose particular problems, mainly in terms of working out where the message came from. With a connection based channel, the source of the message is obvious but this is not true for connectionless communications. Thus, quite a deal of effort is spent on implementing this and trying to hide the details. Unfortunately, the details cannot always be hidden and so it pays to understand the underlying mechanisms.

There are two ways to use the connectionless channels: on their own (i.e. when initiating a connection) and with a server (as part of a received connection. In standalone mode, connectionless channels have their own socket which is closed when the connection is closed. In the server case, the server socket is shared amongst all of the channels (i.e. not closed when a channel closes). The server receives all packets and forwards them to the appropriate channel. But how do we know the appropriate channel?

For a connectionless channel, we must preface every message that is sent with a channel id which identifies uniquely the source. The channel id is used by the receiving end to find the correct receiving channel (if no matching channel exists then this must be a new connection).

Finally, how do we tell if connectionless channels close? In short, we can't. Because there is no connection, we can't tell if the other end closes it's socket or otherwise disappears. However, we must close the connectionless channels or they will accumulate and we will run out of resources. The only solution is to close them after a certain time period of inactivity.