Building distributed enterprise systems is hard - and tools to help build distributed enterprise systems tend to have a steep learning curve too. So, the kpb327 project has been developed, to provide core middleware, which covers the most frequently needed design patterns in such systems. These include client/server remote procedure calls, order management and message queueing. Key features of kpb327 are, ease of interfacing with other systems, rapid development of integrated business logic, small footprint, and, out-of-the-box redundancy. Importantly, kpb327 doesn't rely on external high-availability hardware or software.

The programs which currently comprise kpb327 are -

flow327 - a fully-resilient order management engine capable of orchestrating concurrent multi-stage orders.

flux327 - this is a point-to-point messaging system which features an extremely easy to use file-based interface, and, guaranteed store-and-forward delivery.

sync327 - provides a resilient infrastructure for routing synchronous client/server RPC calls with load-balancing, on-line upgrade and flow-control features.

The kpb327 programs are built using Erlang, chosen for it's distributed architecture, concurrent processing abilities and unrivalled resilience. Perl is used for the rapid building of business logic. Finally, Java features as an integration layer between the kpb327 programs and other systems.

If you need to transfer data reliably between hosts with ease of use being a primary requirement, then flux327 could be the solution. Using flux327 is as simple as appending a line of text to a file on one host (known as the Inflow node). The line of text will then appear appended to a file on the other host (known as the Outflow node). There's no API necessary - senders and receivers can be as simple as shell-scripts.

Inflow nodes can read from multiple files and then deliver to multiple Outflow nodes - either delivering single lines to multiple Outflow nodes (duplication), or delivering a line to the first available Outflow node in a list. Inflow nodes can also be clustered in pairs. This allows one of the pair to take over the sending of lines which originated on the other of the pair if a node fails. with lines still queued for delivery. Outflow nodes can also receive lines from multiple, separate Inflow nodes. All lines are delivered in a guaranteed manner, with lines held in a persistent, disk-based queue if any destination Outflow node is not contactable.

Inflow nodes have a command-line interface which allows the configuration of destination Outflow nodes to be defined, enabled and disabled on-the-fly.

Flow327 is a resilient, lightweight and fast order management engine. "Orders" consist of one or more tasks, to be executed sequentially, together with key/value data associated with the order. Flow327 is capable of maintaining thousands of concurrently executing orders without compromising performance. Command-line and scripting interfaces are incorporated to allow quick and easy maintenance of the system.

External "worker" processes pick tasks to-do from the Flow327 server - these processes can be written in Erlang, Perl or Java (in fact, any platform which can use tcp sockets). Business logic needed to execute order tasks is contained in the worker process code. When a worker fetches a task to-do, the Flow327 server passes all current order key/value data to that worker. The worker, once it has performed the task, can then either flag the task in the order as complete (updating or adding to the order's key/value dataset at the same time), or flag the task as failed - and in need of operator attention. The Flow327 server maintains an audit log which can be used to provide operators with notification of orders which need attention.

Flow327 also contains a scheduler which can be used to create orders automatically at scheduled times.

One of the most frequently needed functions in a distributed system is for client components to make remote procedure calls (RPC) to server components. Sync327 provides a framework of gateways which link clients to servers in a resilient network. This allows for automated recovery from server failure (hardware or software) and facilitates online software upgrades, without having to take service down. Sync327 also provides immediate feedback to a client if a service (provided by one of more servers) has become overloaded - this avoids problematic queueing of RPC calls.

A Sync327 network consists of one or more Sync327 Gateway Nodes. Servers connect into a gateway and the gateway then advertises the presence of the server. Other gateways listen for the advert broadcasts so that every gateway knows where every server resides and which service the server provides. When a client connects to a gateway and makes an RPC call to a specified service, the gateway selects an available server providing that service and redirects the RPC call to that server. Gateway server lists are updated immediately when a server is brought up, or taken down.

The Sync327 Gateway Node is an Erlang component. Clients and servers are normally written in Java, using the Erlang JInterface API, although clients and servers could also be implemented in Erlang.

The entire kpb327 system - source code and manuals - is available here.