Copyright © 2001-2003 Maciej Sobczak
In this tutorial, there are two assumptions:
12340
. This is arbitrary choice.
12341
.
As a first example, you will write your own echo server and client. The echo server will wait for messages from clients and will print them on the screen. Apart from that, the server will also accept a special shutdown message. There will be no data sent back to clients.
First, you will learn how to write simple client. In most distributed systems it is easier to write clients than servers.
The first client in this tutorial sends a message to the server that tells it to shut down. Here it is (file shutdown.c
):
1 #include "yamic.h" 2 #include <stdlib.h> 3 4 int main() 5 { 6 yamiNetInitialize(); 7 8 yamiEasySend("127.0.0.1", 12340, 2, 9 "echo", "shutdown", NULL); 10 11 yamiNetCleanup(); 12 13 return 0; 14 }
Yes, it is that easy.
The first line is necessary to use YAMI in C.
The line with call to yamiEasySend
alone is enough to send a message to the server. There are six parameters used in this function:
"127.0.0.1"
- the server's address. It is a string value and can have the form "comp.company.com" as well.
12340
- the server's port
2
- the level of the message sent. In this example it can be 1
as well (which means ``only strings, please''). In most cases you want it to be Level2, unless you are sending a message to the component that understands only strings.
"echo"
- the name of the destination object. In YAMI, messages are sent to objects which means that there can be many destinations in the same server process.
"shutdown"
- the message name. Later you will see that this message causes the server to shut down.
NULL
- not used now. In the next client example, you will use this parameter to pass some data along with the message.
Note also the yamiNetInitialize
and yamiNetCleanup
functions. They are needed on MS Windows system. If you do not plan to write for Windows, you can safely omit them (they are empty for Unix/Linux) but keep in mind that using them will add to portability of your code.
Interesting? So let's pass some data with the message.
In the previous section, the message had only a name. In many cases it is enough, because it allows the server to make decisions about what action to perform, but there are situations where it would be nice to pass some data together with the message. In YAMI, there are two levels of messages with regard to the data that can be sent:
Both levels are accessible in YAMI Core Library.
There can be many values of different types that can go with the same message at once. All the values are stored in the so-called Parameter Set, which is basically a list of values of chosen type. Such a Parameter Set has to be created before the message is sent and appended to it. It can be even used many times with different messages, but we will not use this feature here.
The client that causes the server to print ``Hello, YAMI!'' message on the screen can look like here (file hello.c
):
1 #include "yamic.h" 2 #include "yamiparams.h" 3 4 int main() 5 { 6 HPARAMSET params; 7 8 yamiNetInitialize(); 9 10 yamiCreateParamSet(¶ms, 1); 11 yamiSetString(params, 0, "Hello, YAMI!"); 12 13 yamiEasySend("127.0.0.1", 12340, 2, 14 "echo", "print", params); 15 16 yamiDestroyParamSet(params); 17 18 yamiNetCleanup(); 19 20 return 0; 21 }
Yes, it is that easy.
The second line is necessary to allow you to use Parameter Set functionality.
The line 6 defines a local variable that is a handle to the Parameter Set object. Later, the Parameter Set is created to contain 1 uninitialized parameter and after that, the parameter is set to be a string of value ``Hello, YAMI!''. As you can see in the next line, the Parameter Set object is used to send the message. Note also that the message name has changed. At the end, the Parameter Set object is destroyed, since it will be no longer needed.
The examples written so far have one thing in common: they send only one message to the server and quit. There is nothing that prevents you from sending millions of messages using the yamiEasySend
function, but in terms of performance it will not be a good idea. The yamiEasySend
function creates a YAMI Agent that is responsible for actually sending a message and destroys it after that. If you want to send many massages, it is a better idea to follow the scheme:
In this section you will write the client according to this scheme. The client itself will be smarter than before, too - it will read the lines of text from its standard input and send each line as a separate message to the echo server. The code is below (file printall.c
):
1 #include "yamic.h" 2 #include "yamiparams.h" 3 #include <stdio.h> 4 #include <string.h> 5 6 #define MAXBUF 1000 7 8 void readline(char *buf) 9 { 10 int i; 11 12 fgets(buf, MAXBUF, stdin); 13 for (i = strlen(buf); i >= 0; --i) 14 if (buf[i] == '\n' || buf[i] == '\r') 15 buf[i] = 0; 16 } 17 18 int main() 19 { 20 HYAMIAGENT agent; 21 HPARAMSET params; 22 char buf[MAXBUF]; 23 24 yamiNetInitialize(); 25 26 yamiCreateAgent(&agent, 12341, NULL); 27 yamiAgentDomainRegister(agent, 28 "echoserver", "127.0.0.1", 12340, 2); 29 30 readline(buf); 31 while (!feof(stdin)) 32 { 33 yamiCreateParamSet(¶ms, 1); 34 yamiSetString(params, 0, buf); 35 36 yamiAgentMsgSend(agent, "echoserver", 37 "echo", "print", params, NULL); 38 39 yamiDestroyParamSet(params); 40 41 readline(buf); 42 } 43 44 yamiDestroyAgent(agent); 45 yamiNetCleanup(); 46 47 return 0; 48 }
The HYAMIAGENT
is a type of handle to the Agent object.
The Agent is created with its own listening port 12341
. This Agent does not use this port, since no information is sent back to the client. Please read about Policies to learn how to create the Agent object without the listening socket.
After the Agent is created, the remote domain is registered in it, like in White Pages Book. The address and port of destination Agent (the one created by the server) is remembered under the name "echoserver"
for later use (the name in the address book is arbitrary).
The program performs a loop where it reads lines of text from standard input. Each line is then sent as a parameter with the message to the server. The last argument to the yamiAgentMsgSend
function allows to retrieve responses from the server. This will be used in later examples.
Finally, we will write the server.
Err... what does this polling mean? First, write the following code (file servera.c
):
1 #include "yamic.h" 2 #include "yamiparams.h" 3 #include <stdio.h> 4 #include <string.h> 5 6 int main() 7 { 8 HYAMIAGENT agent; 9 HINCMSG incoming; 10 HPARAMSET params; 11 const char *msgname; 12 const char *line; 13 14 yamiNetInitialize(); 15 16 yamiCreateAgent(&agent, 12340, NULL); 17 yamiAgentObjectRegister(agent, "echo", 18 polling, NULL, NULL); 19 20 puts("server started"); 21 while (1) 22 { 23 yamiAgentIncomingMsgGetNext(agent, "echo", 24 &incoming, 1); 25 26 yamiAgentIncomingMsgGetMsgName(incoming, 27 &msgname); 28 29 if (strcmp(msgname, "shutdown") == 0) 30 { 31 puts("received the shutdown message"); 32 yamiAgentDestroyIncomingMsg(incoming); 33 break; 34 } 35 else /* if (strcmp(msgname, "print") == 0) */ 36 { 37 yamiAgentIncomingMsgGetParameters(incoming, 38 ¶ms); 39 40 yamiGetStringBuffer(params, 0, &line); 41 puts(line); 42 43 yamiDestroyParamSet(params); 44 yamiAgentDestroyIncomingMsg(incoming); 45 } 46 } 47 48 yamiDestroyAgent(agent); 49 yamiNetCleanup(); 50 51 return 0; 52 }
The HINCMSG
is a type of handle to the incoming message.
After the Agent is created, the polling object is registered in it as a potential target of messages. If any message comes with the object name that is equal to the name of the registered object, the Agent will store the message in a queue, one queue for one object. Later, the server program goes into a loop that... asks the Agent if there are any messages for the echo
object and processes them. This is why the server is polling - it has to ask the Agent for every new message. Moreover, in this example, the server will block waiting for new message if the queue is empty - the last parameter to yamiAgentIncomingMsgGetNext
is non-zero.
When the message is finally retrieved, the server checks its name. If it is "shutdown"
, it breaks the loop and terminates. Otherwise (which means that the message name is "print"
; in real system you would also take some steps if the message has unknown or unacceptable name), the server gets the Parameter Set from the message and prints its first parameter on the standard output.
Note that both the Parameter Set and the incoming message handle are destroyed when not needed.
If there was a polling server, then there must be some alternative. It is called a passive server. Previously, the server had to ask for each incoming message. The advantage was that the server could ask in the most appropriate time if it had some other things to do. However, in some situations it is convenient if it is the Agent who tells the server that there is new message. In this scheme, the server itself is passive - it only waits for invocations from the Agent. It is important to note that the invocations are made from the separate thread that belongs to the Agent object. Let's see the code (file serverb.c
):
1 #include "yamic.h" 2 #include "yamiparams.h" 3 #include "yamisynchro.h" 4 #include <stdio.h> 5 #include <string.h> 6 7 HSEMAPHORE sem; 8 9 void servant(HINCMSG incoming) 10 { 11 HPARAMSET params; 12 const char *msgname; 13 const char *line; 14 15 yamiAgentIncomingMsgGetMsgName(incoming, &msgname); 16 if (strcmp(msgname, "shutdown") == 0) 17 { 18 puts("received the shutdown message"); 19 yamiSemaphoreRelease(sem); 20 } 21 else /* if (strcmp(msgname, "print") == 0) */ 22 { 23 yamiAgentIncomingMsgGetParameters(incoming, 24 ¶ms); 25 26 yamiGetStringBuffer(params, 0, &line); 27 puts(line); 28 29 yamiDestroyParamSet(params); 30 } 31 32 yamiAgentIncomingMsgEat(incoming); 33 } 34 35 int main() 36 { 37 HYAMIAGENT agent; 38 39 yamiNetInitialize(); 40 41 yamiCreateSemaphore(&sem, 0); 42 43 yamiCreateAgent(&agent, 12340, NULL); 44 yamiAgentObjectRegister(agent, "echo", 45 passive_singlethreaded, servant, NULL); 46 47 puts("server started"); 48 49 yamiSemaphoreAcquire(sem); 50 51 /* the process reaches this code */ 52 /* only when the semaphore is released */ 53 54 yamiDestroyAgent(agent); 55 yamiDestroySemaphore(sem); 56 yamiNetCleanup(); 57 58 return 0; 59 }
The yamisynchro.h
file is included so that it will be possible to provide some synchronization between two threads involved here into processing.
The first difference in the main
function is that the object was registered as a passive_singlethreaded
instead of polling
. The last but one parameter to the yamiAgentObjectRegister
function is a pointer to function that will be called by the Agent when there is some new message for the given object. After registering the object, the server process suspends its execution with simple trick - acquire the global semaphore that is initially set to 0. In other words, the server process is going to sleep.
Now, let's go back to the servant
function. The name was chosen because in other distributed infrastructures (notably CORBA) the servant is a piece of software that is responsible for processing messages. This is exactly the purpose of the function. Note that the incoming
handle to the message is already provided. What does the servant
function do? The same as before, with few differences:
incoming
handle is not destroyed - it is a property of the Agent object.
YAMI sends and processes messages asynchronously by default. This means two things:
$ ./printall
(I assume some Unix environment here)
and later type some lines of text by hand. You are not really fast with typing, so the messages (each message contains one line of text) are actually sent to the server before you get your finger off the keyboard. But try this little experiment:
$ ./printall < printall.c
which should sent as many messages to the server as there are lines in the printall.c
file. After reaching end-of-file, the process terminates... which can happen before the sender thread sends anything! Or you will see only few of the lines on the server side. There are three solutions to this problem:
Similar issues arise on the server side. The messages received from the network are automatically stored in the queue. This queue can get overflown (you can increase its maximum size, but again - do not tell anybody). Similarly, the professional solution is to change the dispatching mode to synchronous by switching off dispatching threads in the Agent.
Look at the printsync.c
and servsync.c
files to see how to do it. If you compile them, you can try the test:
$ ./servsync
in one console and (probably on different computer in the network, but you should change the address used in the client source code) in the other:
$ ./printsync < somelongfile
Enjoy.
In the Unix-like systems, the socket can remain open for some time even after the process using it terminated. Try to run the server, shut it down and immediately type:
$ netstat | grep "12340"
You will see some sockets still open in the TIME_WAIT
state. It can take a while until they go away, but during that time you will not be able to restart the server - the system will not allow you to bind to open socket. This can get annoying sometimes, but there's a way around. The servsync.c
file will show you, look at the Agent's Policies.
Many different things can go wrong in the distributed system. Some of them will be your fault, some not. Almost every YAMI function reports success or error condition. In the example programs presented so far there was no error checking - for simplicity and to avoid cluttering the source code. In real programs, you will for sure want to check for error codes. The printsync.c
and servsync.c
files are written with simple, but consistent error checking.
In this second example, you will write the client-server pair where the client asks for something and the server responds to the messages, sending some data back to client. This is probably the most common way of developing distributed systems.
The calculator example consists of:
add
, sub
, mul
and div
, each with two integer parameters. Each message will be replied to, with the appropriate integer result.
I'm sure that after this example you will be able to write distributed systems of arbitrary complexity.
Let's start with easy things. The client (file calcclient.c
) looks like here:
1 #include "yamic.h" 2 #include "yamiparams.h" 3 #include <stdio.h> 4 5 const char *serverhost = "127.0.0.1"; 6 const int serverport = 12340; 7 const int clientport = 12341; 8 const char *domainname = "someDomain"; 9 const char *objectname = "calculator"; 10 11 int main() 12 { 13 HYAMIAGENT agent; 14 HPARAMSET paramset, returnparamset; 15 HMESSAGE hm; 16 enum msgStatus status; 17 int result; 18 19 yamiNetInitialize(); 20 21 yamiCreateAgent(&agent, clientport, NULL); 22 23 yamiAgentDomainRegister(agent, domainname, 24 serverhost, serverport, 2); 25 26 yamiCreateParamSet(¶mset, 2); 27 28 yamiSetInt(paramset, 0, 100); 29 yamiSetInt(paramset, 1, 20); 30 31 yamiAgentMsgSend(agent, domainname, 32 objectname, "add", paramset, &hm); 33 34 yamiDestroyParamSet(paramset); 35 36 yamiAgentMsgWait(hm); 37 38 yamiAgentMsgGetStatus(hm, &status); 39 40 if (status == eReplied) 41 { 42 yamiAgentMsgGetResponse(hm, &returnparamset); 43 44 yamiGetInt(returnparamset, 0, &result); 45 46 yamiDestroyParamSet(returnparamset); 47 48 printf("the result is %d\n", result); 49 } 50 else if (status == eRejected) 51 { 52 puts("the last message was rejected"); 53 } 54 else 55 { 56 puts("no correct reply"); 57 } 58 59 yamiAgentMsgDestroy(hm); 60 yamiDestroyAgent(agent); 61 yamiNetCleanup(); 62 63 return 0; 64 }
The main difference to the previous example is the usage of the last parameter to the yamiAgentMsgSend
and its consequences. The HMESSAGE
is a type of handle to the message token. It allows you to retrieve some information about the message sent to the remote objects. It also allows you to synchronize the client activity with the server - the yamiAgentMsgWait
function allows the client to wait for some change in the message's status.
As you can see, the message is sent with two integer parameters in the Parameter Set (100
and 20
). The message is sent asynchronously, so that you can continue your job without waiting for response and ask for it later. In this example there is nothing interesting to do anyway, so we decide to wait (with the call to yamiAgentMsgWait
) until something interesting happens to the message token. The process wakes up when (for example) the response arrives. The status
of the message is examined to find out if there is a real reply or maybe some network error or something else. If there is a reply, we just retrieve the returning Parameter Set and print its only (first) parameter.
The Calculator example shows also that there are two contexts where Parameter Set can be used:
These two contexts allow to send data in both directions.
Easy? So take also a look at the calcclient2.c
file and play with it - it is an interactive calculator console.
There is one thing to remember, though. The yamiAgentMsgWait
function is your friend, but do not trust him. It may happen that the server crashes in the middle of the computations. Then - the client does not receive any notification and the status of the message is always ePending
. If you call the wait function, you are in troubles (the other solution, sometimes reasonable, is to periodically ask the message token if the response arrived - the client can decide by himself that something went wrong after, say, 10th try). To help you with this problem, the Agent object provides the waker service. The calcclient2.c
file shows how to use it.
The server is presented below (file calcserver.c
):
1 #include "yamic.h" 2 #include "yamiparams.h" 3 #include "yamisynchro.h" 4 #include "yamierrors.h" 5 #include <stdio.h> 6 7 const int serverport = 12340; 8 const char *objectname = "calculator"; 9 10 void servant(HINCMSG incomingmsg) 11 { 12 HPARAMSET paramset, returnparamset; 13 const char *msgname; 14 int val1, val2, result; 15 16 yamiAgentIncomingMsgGetMsgName(incomingmsg, 17 &msgname); 18 19 printf("I have received the message: %s\n", 20 msgname); 21 22 if (strcmp(msgname, "add") && 23 strcmp(msgname, "sub") && 24 strcmp(msgname, "mul") && 25 strcmp(msgname, "div")) 26 { 27 puts("unknown name - rejecting"); 28 yamiAgentIncomingMsgReject(incomingmsg); 29 return; 30 } 31 32 yamiAgentIncomingMsgGetParameters(incomingmsg, 33 ¶mset); 34 35 yamiGetInt(paramset, 0, &val1); 36 yamiGetInt(paramset, 1, &val2); 37 38 yamiDestroyParamSet(paramset); 39 40 if (strcmp(msgname, "add") == 0) 41 { 42 result = val1 + val2; 43 } 44 else if (strcmp(msgname, "sub") == 0) 45 { 46 result = val1 - val2; 47 } 48 else if (strcmp(msgname, "mul") == 0) 49 { 50 result = val1 * val2; 51 } 52 else /* msgname is "div" */ 53 { 54 if (val2 == 0) 55 { 56 puts("dividing by 0 not allowed"); 57 puts("rejecting"); 58 yamiAgentIncomingMsgReject(incomingmsg); 59 return; 60 } 61 else 62 result = val1 / val2; 63 } 64 65 yamiCreateParamSet(&returnparamset, 1); 66 67 yamiSetInt(returnparamset, 0, result); 68 69 yamiAgentIncomingMsgReply(incomingmsg, 70 returnparamset); 71 72 yamiDestroyParamSet(returnparamset); 73 } 74 75 int main() 76 { 77 HYAMIAGENT agent; 78 79 puts("starting the server"); 80 81 yamiNetInitialize(); 82 83 yamiCreateAgent(&agent, serverport, NULL); 84 85 yamiAgentObjectRegister(agent, objectname, 86 passive_singlethreaded, servant, NULL); 87 88 puts("going to sleep..."); 89 yamiSleep(0); 90 91 yamiDestroyAgent(agent); 92 93 yamiNetCleanup(); 94 95 return 0; 96 }
This is a passive server, because it uses the call-back pattern for message dispatching. You have seen it already in the Echo example.
This server has one design quirk. I show you this, because it is sometimes used in servers working in distributed systems (I have seen it in some CORBA examples, really). Namely - after setting up everything in the main
function, the main server thread goes to sleep... and never wakes up. The yamiSleep
function with 0
as a parameter sleeps forever. This means two things:
I do not advocate this style of writing servers, although, as I've pointed out, I have seen it. The two clean solutions that I recommend are:
main
function, call the yamiAgentIncomingMsgGetNext
function on this object instead of going to sleep. The first message that will be received for this special shut-down object will wake you up, so that you can terminate the server in a clean way.
The file calcserver2.c
is the same as presented above calcserver.c
but with error-handling added.
This part of the tutorial is aimed at advanced users, who may wish to experiment a little or who find themselves in unusual situations like the necessity to write software on platforms that do not have threads (like classic Unix or some embedded systems). Basically, you will rarely (if ever) need to write your own event processing.
It is possible to compile the YAMI library without threading. Then, most of the library's functionality is gone and it may seem at first that it is not possible to use YAMI at all. But it is.
In order to correctly process connections coming from remote Agents, the local Agent performs a loop, where new connections are managed and where everything is made. In fact, every thread in the Agent performs its own loop, but the only really necessary is the loop in the receiver module. There are two situations where there may be no separate receiver module:
In both situations the programmer needs to perform the loop by himself, usually in the main part of his program - this applies both to client and server. It is also possible to combine this explicit event processing with other loops that may appear in the crucial part of your programs, like in windows-based GUI applications.
The example programs clientloop.c
and serverloop.c
are similar to the examples presented so far, but their Agents are prepared to work without separate threads and the important event processing is triggered explicitly in the application code. Previously, the event processing was hidden in the receiver module which was running by itself.
The interesting thing to note is that the explicit event processing results in best performance, since everything related to message processing is done in the context of a single thread and most (or all) of the synchronization overhead is avoided. Nice, isn't it?. Of course, this comes at the cost of increased complexity, as usual.
Last few comments:
TIME_WAIT
phenomenon affects clients, too (on Unix-like machines). This is because in the context of the reply (the data sent back to client), the client plays a role of a server to the server that now plays a role of a client... Never mind. You know, what to do.
Enjoy! And, of course, tell your friends about YAMI.