Because of the high performance publish-subscribe message routing features of RTserver, a single RTserver often is sufficiently powerful for small workgroup-sized projects. Large-scale distributed systems for the intranet, extranet, and Internet, however, need the benefit of enterprise-wide publish-subscribe capabilities. Dynamic message routing offers the scalability and flexibility these large systems require.
Dynamic message routing has these features.
When in a WAN environment such as the Internet, TCP/IP has many sophisticated, evolving technologies to ensure reliable delivery of network packets. Internet standards such as Domain Naming Service (DNS) and Open Shortest Path First (OSPF) routing all help TCP/IP scale to the levels it has reached today. As described in Why is GMD Needed?, though, there are gaps in this reliability that any large distributed system must overcome. In addition, there are areas such as publish-subscribe and real-time directory services updating that TCP/IP simply does not address (other products such as DCE and X.500 do not address real-time updates, either).
It might seem redundant for SmartSockets publish-subscribe to offer dynamic message routing when TCP/IP has many seemingly similar dynamic routing features at the network packet level. The global TCP/IP-based Internet is inherently a peer-to-peer system, such as the Web, FTP, email, or Telnet client connecting to the Web, FTP, email, or Telnet server on a specific machine, and does not have the flexible one-to-many publish-subscribe capabilities required for large-scale distributed systems.
Dynamic message routing takes full advantage of TCP/IP’s dynamic routing capabilities and extends them with the power of publish-subscribe. Dynamic message routing is also needed for mixed protocol networks where not every computer supports TCP/IP. TCP/IP can also be used as the backbone, and yet dynamic message routing can deliver messages off to the interconnected networks running Netware or any other protocol.
In addition to routing messages among RTclients, multiple RTservers also dynamically route messages to each other. Multiple RTservers can distribute the load of publish-subscribe message routing. If a project is partitioned so that most of the messages being sent are routed among processes on the same node, then the use of multiple RTservers can reduce the consumption of network bandwidth (processes on the same node can use the non-network local IPC protocol).
The RTclients that are directly connected to an RTserver are called direct RTclients, and, likewise, the RTclients directly connected to other RTservers are called indirect RTclients. Similar terminology is used for direct RTservers and indirect RTservers.
Figure Notes:
Multiple RTservers can form a group that allows SmartSockets projects great flexibility, scalability, and robustness. A group is an arbitrary topology of interconnected RTservers, which is represented with an undirected graph and uses graph algorithms to calculate lowest cost path between two RTservers. Figure 25 shows an example of a group with four RTservers. There can be more than one RTserver group on a network, but each group is self-contained. Just as RTclients in different projects cannot publish messages to each other, RTservers in different groups cannot send messages to each other.
Figure 26 shows a typical RTserver group where some of the communication must occur over a WAN. Rather than have all the RTservers interconnected, RTserver 2 and RTserver 8 act as gateways to the WAN and limit the number of connections occurring over the WAN.
The group approach decreases the number of connections (and thus operating system resources such as socket file descriptors) each RTserver needs, but publish-subscribe message routing for SmartSockets becomes more complex. However, the message routing is handled completely transparently and in real time. The RTservers in the group send messages among themselves to update each other regarding subscribes and unsubscribes. When RTclient connected to RTserver subscribes to a subject, that RTserver is also conceptually subscribing to that subject from the other RTservers. When no RTclients connected to RTserver are subscribing to a subject, then that RTserver does not receive any messages for that subject from other RTservers in the group (this can be overridden for greater efficiency as described in RTserver Subscribes).
The speed at which a message is routed and delivered depends on many factors including how many connections the message has to pass through and also how fast those connections are. For example, local connections on UNIX and MVS are always much faster than TCP/IP network connections. Each RTserver manages the majority of publish-subscribe message routing for its direct RTclients. It is transparent to RTclients how many RTservers are in the group. From the standpoint of RTclient, the group is a virtual RTserver cloud that happens to span several nodes. RTclient simply needs to connect to a single RTserver in the cloud and can then send a message to or receive a message from any RTclients connected to the cloud.
In a large distributed publish-subscribe system, if every RTserver knows about all the RTclients and all subject subscriptions, the system experiences scalability problems due to network chatter that occurs each time the state of the system changes (subscription and RTclient changes). In SmartSockets, RTserver does not have complete knowledge. Instead the RTserver group can be viewed as a distributed real-time database or directory service of publish-subscribe information. The limited information maintained by each RTserver makes SmartSockets publish-subscribe work easily and efficiently, and scale to very large projects. Because the information is distributed, there is no single point of failure in the system.
Just as RTserver only routes messages to RTclients that are subscribed to a subject, RTserver only routes messages to other RTservers that have at least one direct RTclient subscribing to a subject. In general, when RTserver receives a message from another RTserver, it routes the message to its direct RTclients and sometimes also to other specific RTservers. When RTserver receives a message from a direct RTclient, it might route that message to both direct RTclients and other direct RTservers.
The RTserver group uses interior messages to communicate and synchronize with each other. The exterior messages are those routed for RTclients (normal publish-subscribe). The monitoring messages described in Chapter 5, Project Monitoring, have characteristics of both interior and exterior messages. These interior messages use a form of memory-only GMD, complete with acknowledgments and resends, to ensure delivery and ordering so as to prevent corruption of the RTserver distributed database. When there is a topology change in the group of RTservers (due to a new connection being created or an existing connection being destroyed), all appropriate GMD messages are resent in sequence number order to prevent out-of-order delivery and message loss.
RTserver uses lowest cost routing, where each RTserver only knows the first-hop to a destination RTserver, which simplifies accounting. Dijkstra’s shortest-path algorithm is used. Each RTserver runs the same-first hop calculation in parallel. It follows that each RTserver does know about all RTservers in the group.
Figure 27 shows a SmartSockets application with multiple RTservers and RTclients. All RTserver-to-RTserver connections have the default connection cost of one. In Figure 27 a message going from RTclient A to RTclient B would go through one RTserver. A message going from RTclient A to RTclient C travels through two RTservers, RTserver1 and RTserver2. A message going from RTclient A to RTclient D travels through three RTservers, namely RTserver1, RTserver3 and RTserver4.
Routing is dynamic in that it can change at any time. Whenever a new RTserver becomes available or an existing RTserver goes down, routing tables in the RTservers are updated in real time to reflect the new topology. Lowest cost routing can be made even more efficient by changing the relative costs of RTserver to RTserver connections to closely match available bandwidth, load conditions, and so on, and by using RTserver subscribes as shown in the next section.
Figure 28 shows a SmartSockets application with multiple RTservers and RTclients. Two RTserver-to-RTserver connections have non-default connection costs. In Figure 28, there are two possible routes for a message going from RTclient A to RTclient B. The route starting at RTserver1 going through RTserver2 and finally through RTserver4 has a cost of 2+1=3. The alternate route starting at RTserver1 going through RTserver3 and finally through RTserver4 has a cost of 5+1=6. With this configuration, all messages going from RTclient A to RTclient B will be routed along the first route, going from RTserver1 to RTserver2 and on to RTserver 4 because that is the lowest cost route. If the connection between RTserver1 and RTserver2 becomes unavailable, then the messages going from RTclientA to RTclientB will be routed using the RTserver1 to RTserver3 to RTserver4 route.
When a message is routed from one RTserver to several other RTservers, only the minimum necessary copies of messages are sent between the RTservers. That is, if a message is sent to several other RTservers through the same first-hop connection, then the message is marked as needing to be delivered to several RTservers, and one copy, not several, is sent through the proper first hop. This is superior to a flooding algorithm when there are loops in the group topology and the same message could be sent over and over.
SmartSockets provides many mechanisms to allow projects to scale to large numbers of machines. One important technique is to minimize network chatter whenever possible. One method of doing this is through the use of RTserver subscribes, which are specified through the RTserver subscribe command.
This is useful where there is a network bandwidth problem or a usage problem, such as:
The subscribe command can be executed from a command file, from the RTserver command line, or it can be sent to RTserver from another process through a CONTROL message. The general syntax of the subscribe command is:
For example, to subscribe to all subjects underneath /stocks in the stock_trader project, this subscribe command is used:
Additional subscribes are additive. For example, if you issue a command to subscribe to the subject "/stocks/ford", and then issue a second command to subscribe to "/stocks/gm", the RTserver then subscribes to both subjects. There is an associated unsubscribe command to have RTserver unsubscribe from a subject. See subscribe for more details.
RTserver subscribes allow any number of RTclient subscribes to be combined into a single update. This can greatly reduce the amount of network traffic that occurs as RTclients connect and disconnect and as they subscribe and unsubscribe to subjects. The scalable power of SmartSockets publish-subscribe is realized by the power of the hierarchical subject namespace combined with the RTserver subscribes to efficiently cover hundreds or thousands of RTclient subscribes.
For example, consider the simple environment shown in Figure 29. There are 1,000 RTclients connected to a single RTserver, each subscribed to a single subject, such as "/stocks/ford".
.
This adds up to 1,000 subscription messages for the RTserver to send to all other RTservers, which can cause a significant amount of network message traffic if there are many RTservers, or the RTclients are starting/stopping or subscribing/unsubscribing frequently. If this RTserver itself subscribes to "/stocks/...", then the RTserver matches all of the 1,000 RTclient subscribes, and there is only one subscription message, not 1,000 being sent to all other RTservers.
It is important to note that the primary purpose of RTserver subscribes is to increase the efficiency of the distributed system. In all cases, SmartSockets publish-subscribe, GMD, and monitoring all work in any RTserver topology without any manual configuration. Using RTserver subscribes, it is possible that a message may travel all the way to the subscribed RTserver before being discarded if there is no direct RTclient subscribed to the subject.
|
TIBCO SmartSockets™ User’s Guide Software Release 6.8, July 2006 Copyright © TIBCO Software Inc. All rights reserved www.tibco.com |