Horses for courses
Clean modularity and predictive scalability
Vs
Fast processing and small memory footprint
As we all know and hopefully agree with the proverb “horses for courses” – there is no single technology that solves all our problems. There are some technologies (specially programming languages) attributed as general-purpose technologies. Even though there is some truth in it – but it actually means they are good for seemingly trivial business and technology problems. In order to solve complex business problems we must divide the problems in multiple layers with very clean separation and harmonize them by neatly tying them together. Choosing the right materials and right glues is the key in achieving success. Success has to be also objectively measured by three fundamental business yardsticks – time to market (TTM – faster is better), total cost of ownership (TCO – lower the better) and return on investment (ROI – improved one). If all the technology decisions and execution strategies are measured objectively against these three yardsticks chances of achieving success becomes much better. To do that some of the tangible steps can be taken as follows. Remember though they are just indicative and not exhaustive list of things – there could be many other peripheral things that need to be taken into consideration while doing this exercise.
- Break the architecture into multiple distinct pieces based on cleanly assigned functionalities.
- Choose the right technology for each separated function. Keep it lightweight – that means don’t choose anything that brings more baggage than desired functionalities.
- Choose the right glues to tie all these distinct pieces together. These “glue” technologies should be chosen very carefully so that it doesn’t bring too much of proprietary styles, doesn’t impose too much of additional works, doesn’t become performance bottleneck, harmonized with the rest of the functional blocks.
- Eliminate paradigm rift between different functional pieces and cut down all static (that also imposes propagation of changes when one part changes) boilerplates.
- Break the architecture into multiple distinct pieces
This is an art and science together with both being equally important. Different scenarios will entail different non-deterministic results. I am not going to discuss this further since it really depends. But do it very carefully and make it crystal clear – that’s a must. Have fun doing it J
- Choose the right technology for each separated function
This is where people bang their heads most and most political and preferential plays happen. With a wrong choice here things may break miserably. So what are the things to consider at the high level to choose a right piece of technology in this area? Here is my list
- Find a clean piece of technology that gives exactly the functionality we are looking at.
- Prefer a library that can be embedded into the rest of the ecosystem to a piece of technology that comes with its own runtime. An additional runtime will bring lots of operational overhead. Avoid that as the rule of thumb. Only bring it if there is absolutely no other option. Data stores, messaging systems, UI servers are exceptions – these are the technologies, which inevitably bring their own runtime, which is ok in most cases. For other functions – prefer libraries to something that comes with large runtime.
- Performance and Scalability consideration: Faster is better, not always. This is another tricky-as-hell area. Concurrency vs parallelism plays an important role here. I personally prefer to make processing as much concurrent as possible to making it parallel. Parallel only when the business problem is really parallel – that is crunching a real big data-set which can not be broken down without breaking the meaning. If a large data-set can be broken down into multiple pieces – do that and process them concurrently. This is the reason Map-reduce is over used in many places where it is not really applicable. Parallelism is very expensive don’t innovate problems for parallel processing if it is not really parallel problem already. In other word don’t use hadoop and anything similar unless you really a MR problem – i.e. a very large data-set that can not be broken down into pieces without losing its meaning. Now if you figured out that adding concurrency to the mix could solve the problem; then choose a simple high horse power technology solution where you can add more horsepower when load is increased. It need not have inherent or in-built clustering support. The individual boxes in the mix work in isolation and they don’t need to communicate between each other – that’s pure concurrency. In a nutshell, if the problem can be modeled as concurrency problem choose simple fast processing capability and then distribute them as independent processing unit over the lose cluster. Simple and fast processing are the keys here. Fast processing will help you minimize the size of the hardware or number of gears and simple is helpful everywhere in order to maintain ease of development and support. In the other hand when our problem is really a parallel problem – that means processing a large data set that may span across multiple machines and processing several parts needs coordination – yield of high horsepower per machine is not as important as linear synchronous scalability. The question becomes if my unbreakable dataset becomes double can be process that at about the same speed if I double the number of machines in the cluster. Since all nodes in this kind of cluster need to communicate with each other, it is a key to figure out how difficult it is to add nodes in the cluster. Data backup, recovery, handling failure all these become very important in this scenario. Parallelism is really difficult. For these reasons we should avoid parallelism as much as we can, if concurrency can solve the same problem avoid parallelism – it might break your back and IT’s back badly. In the case of concurrency, treat the cluster as a set of truly independent nodes then handle failure, backup and everything else at the individual node level – that’s fairly simple – extremely hard in the case of parallelism.
- Choose the right glues to tie all these distinct pieces together
Databases have been the glue for a long time. Its good that they are taking a back seat on that and modern messaging system are taking that place. Like anything else we have to consider whether we need a messaging system or we don’t need it at all. In many cases some inherent capability of programming languages can be used as the glue. Surely it all depends on what kind of problem we are solving. If it is about distributing multiple algorithmic functions across multiple nodes and making them talk to each other – then I will prefer, in most cases, to choose a single programming language and a framework that provides communication framework in the native language way. E.g. if my language of choice is Scala, I will use Akka for this glue. Erlang actor system is erlang is my choice of language. STM is clojure is my choice. The key here is, for algorithmic functional distribution we don’t need to (and should not) bring an alien technology as the glue instead we should choose something that works natively with the choice of technology used to implement that functional blocks. Unfortunately, often time, our problem space is broader than this. We not only have algorithmically/functionally distinct blocks but we also have behaviorally distinct areas too. Like UI, like a third party system that performs very specialized tasks – in those cases a native glue of a single technology wouldn’t work. Even when there is no third-party system involved, we still need use multiple technologies at the behaviorally distinct layers. Horses for courses – you remember, that’s the title of the article. E.g. If I have to process a huge number of events I will probably use Erlang, or Akka or Clojure but if that needs a nice and fairly complex UI – I will use nodeJS serving the UI. Horses for courses again – right thing for right thing. Now when these different technologies come exist and the must communicate with each other we must choose a right “glue” that mediates between these two. At times, there could be multiple glues existing at one time. Here are some very rough principles of choosing the right glue.
- Highly reactive system and eventually consistent: In the case of highly reactive system, e.g. when there is any change in a piece of data UI should immediately update that without refresh, we should to use pure push model. Whenever there is a change anywhere push the changes to all the layers those care. When the data changes are huge we need a very high performance layer that can handle a huge number of pushes. Socket servers – like zeroMQ, NanoMSG, Message Queues – like kafka, rabbitMQ are few examples.
- Highly consistent but not highly reactive: If the system is not reactive, that means there is not need to propagate the changes to any other layer but those layers can selectively choose to pull the data whenever it needs. Search is a perfect example pull model. Avoid polling though – polling is very expensive and nasty. Use push everywhere to push-pull. The difference between polling and pulling is – polling tries to poll new data even when there is no new data or no change, where is pull only pulls when you need some data irrespective of whether there is a change or no change. A distributed data structure service with rich language bindings is the thing to consider while choosing this type of glue. Redis is a great example for this kind.
- Eliminate paradigm rift
Having the big picture, how all the pieces of a system come together is very important. We also need to make sure the multiple pieces are not bringing additional constraints that mandates changes in many other layers when there is a change in one layer. Object Relational Mapping is such an example – a data base change will force change many other layers of the system. In fact anything other than HR or payroll or similarly trivial application where data variety is small and highly structured, RDBMS may not be a good choice. Choose some data stores that has open and standard based data exchange format – like JSON or XML. In that case emission and consumption of data by multiple technologies become is and can be done with lose binding without the necessity of brewing a rigid layer of broker layer.