Technology

Horses for courses

Clean modularity and predictive scalability

Vs

Fast processing and small memory footprint

 

As we all know and hopefully agree with the proverb “horses for courses” – there is no single technology that solves all our problems. There are some technologies (specially programming languages) attributed as general-purpose technologies. Even though there is some truth in it – but it actually means they are good for seemingly trivial business and technology problems. In order to solve complex business problems we must divide the problems in multiple layers with very clean separation and harmonize them by neatly tying them together. Choosing the right materials and right glues is the key in achieving success. Success has to be also objectively measured by three fundamental business yardsticks – time to market (TTM – faster is better), total cost of ownership (TCO – lower the better) and return on investment (ROI – improved one). If all the technology decisions and execution strategies are measured objectively against these three yardsticks chances of achieving success becomes much better. To do that some of the tangible steps can be taken as follows. Remember though they are just indicative and not exhaustive list of things – there could be many other peripheral things that need to be taken into consideration while doing this exercise.

 

1. Break the architecture into multiple distinct pieces based on cleanly assigned functionalities.
2. Choose the right technology for each separated function. Keep it lightweight – that means don’t choose anything that brings more baggage than desired functionalities.
3. Choose the right glues to tie all these distinct pieces together. These “glue” technologies should be chosen very carefully so that it doesn’t bring too much of proprietary styles, doesn’t impose too much of additional works, doesn’t become performance bottleneck, harmonized with the rest of the functional blocks.
4. Eliminate paradigm rift between different functional pieces and cut down all static (that also imposes propagation of changes when one part changes) boilerplates.

 

 

1. Break the architecture into multiple distinct pieces

This is an art and science together with both being equally important. Different scenarios will entail different non-deterministic results. I am not going to discuss this further since it really depends. But do it very carefully and make it crystal clear – that’s a must. Have fun doing it J

1. Choose the right technology for each separated function

This is where people bang their heads most and most political and preferential plays happen. With a wrong choice here things may break miserably. So what are the things to consider at the high level to choose a right piece of technology in this area? Here is my list

• Find a clean piece of technology that gives exactly the functionality we are looking at.
• Prefer a library that can be embedded into the rest of the ecosystem to a piece of technology that comes with its own runtime. An additional runtime will bring lots of operational overhead. Avoid that as the rule of thumb. Only bring it if there is absolutely no other option. Data stores, messaging systems, UI servers are exceptions – these are the technologies, which inevitably bring their own runtime, which is ok in most cases. For other functions – prefer libraries to something that comes with large runtime.
• Performance and Scalability consideration: Faster is better, not always. This is another tricky-as-hell area. Concurrency vs parallelism plays an important role here. I personally prefer to make processing as much concurrent as possible to making it parallel. Parallel only when the business problem is really parallel – that is crunching a real big data-set which can not be broken down without breaking the meaning. If a large data-set can be broken down into multiple pieces – do that and process them concurrently. This is the reason Map-reduce is over used in many places where it is not really applicable. Parallelism is very expensive don’t innovate problems for parallel processing if it is not really parallel problem already. In other word don’t use hadoop and anything similar unless you really a MR problem – i.e. a very large data-set that can not be broken down into pieces without losing its meaning. Now if you figured out that adding concurrency to the mix could solve the problem; then choose a simple high horse power technology solution where you can add more horsepower when load is increased. It need not have inherent or in-built clustering support. The individual boxes in the mix work in isolation and they don’t need to communicate between each other – that’s pure concurrency. In a nutshell, if the problem can be modeled as concurrency problem choose simple fast processing capability and then distribute them as independent processing unit over the lose cluster. Simple and fast processing are the keys here. Fast processing will help you minimize the size of the hardware or number of gears and simple is helpful everywhere in order to maintain ease of development and support. In the other hand when our problem is really a parallel problem – that means processing a large data set that may span across multiple machines and processing several parts needs coordination – yield of high horsepower per machine is not as important as linear synchronous scalability. The question becomes if my unbreakable dataset becomes double can be process that at about the same speed if I double the number of machines in the cluster. Since all nodes in this kind of cluster need to communicate with each other, it is a key to figure out how difficult it is to add nodes in the cluster. Data backup, recovery, handling failure all these become very important in this scenario. Parallelism is really difficult. For these reasons we should avoid parallelism as much as we can, if concurrency can solve the same problem avoid parallelism – it might break your back and IT’s back badly. In the case of concurrency, treat the cluster as a set of truly independent nodes then handle failure, backup and everything else at the individual node level – that’s fairly simple – extremely hard in the case of parallelism.

1. Choose the right glues to tie all these distinct pieces together

Databases have been the glue for a long time. Its good that they are taking a back seat on that and modern messaging system are taking that place. Like anything else we have to consider whether we need a messaging system or we don’t need it at all. In many cases some inherent capability of programming languages can be used as the glue. Surely it all depends on what kind of problem we are solving. If it is about distributing multiple algorithmic functions across multiple nodes and making them talk to each other – then I will prefer, in most cases, to choose a single programming language and a framework that provides communication framework in the native language way. E.g. if my language of choice is Scala, I will use Akka for this glue. Erlang actor system is erlang is my choice of language. STM is clojure is my choice. The key here is, for algorithmic functional distribution we don’t need to (and should not) bring an alien technology as the glue instead we should choose something that works natively with the choice of technology used to implement that functional blocks. Unfortunately, often time, our problem space is broader than this. We not only have algorithmically/functionally distinct blocks but we also have behaviorally distinct areas too. Like UI, like a third party system that performs very specialized tasks – in those cases a native glue of a single technology wouldn’t work. Even when there is no third-party system involved, we still need use multiple technologies at the behaviorally distinct layers. Horses for courses – you remember, that’s the title of the article. E.g. If I have to process a huge number of events I will probably use Erlang, or Akka or Clojure but if that needs a nice and fairly complex UI – I will use nodeJS serving the UI. Horses for courses again – right thing for right thing. Now when these different technologies come exist and the must communicate with each other we must choose a right “glue” that mediates between these two. At times, there could be multiple glues existing at one time. Here are some very rough principles of choosing the right glue.

• Highly reactive system and eventually consistent: In the case of highly reactive system, e.g. when there is any change in a piece of data UI should immediately update that without refresh, we should to use pure push model. Whenever there is a change anywhere push the changes to all the layers those care. When the data changes are huge we need a very high performance layer that can handle a huge number of pushes. Socket servers – like zeroMQ, NanoMSG, Message Queues – like kafka, rabbitMQ are few examples.
• Highly consistent but not highly reactive: If the system is not reactive, that means there is not need to propagate the changes to any other layer but those layers can selectively choose to pull the data whenever it needs. Search is a perfect example pull model. Avoid polling though – polling is very expensive and nasty. Use push everywhere to push-pull.  The difference between polling and pulling is – polling tries to poll new data even when there is no new data or no change, where is pull only pulls when you need some data irrespective of whether there is a change or no change. A distributed data structure service with rich language bindings is the thing to consider while choosing this type of glue. Redis is a great example for this kind.

 

1. Eliminate paradigm rift

Having the big picture, how all the pieces of a system come together is very important. We also need to make sure the multiple pieces are not bringing additional constraints that mandates changes in many other layers when there is a change in one layer. Object Relational Mapping is such an example – a data base change will force change many other layers of the system. In fact anything other than HR or payroll or similarly trivial application where data variety is small and highly structured, RDBMS may not be a good choice. Choose some data stores that has open and standard based data exchange format – like JSON or XML. In that case emission and consumption of data by multiple technologies become is and can be done with lose binding without the necessity of brewing a rigid layer of broker layer.

Adopting Scala: Lean Architecture and Clean Code

This is a blueprint of a book I am writing. Intially started with Manning but then decided to move forward with my own pace. I will probably publish this through kindle self publishing. I also conduct workshops based on the contents here, mostly they are free, I only charge when it is customized and too much of my time is used. Please feel free to comment on the contents and also let me know if yo have similar workshop in mind or anything about technology in general. 

Automation and Assisted Automation

Reactive Automation and Assisted Reactive Automation

 

Reactive automation is a kind of automation that gets triggered when there is a presence of specific pattern of events in the event space; many a times the irregular ones. It is different from the regular automation in a sense that it does not automate the routinely tasks or regular tasks or sequence of tasks but deals with irregular behaviors of an ecosystem to keep the ecosystem functioning as desired and keep itself on its track.

 

Reactive automation triggers an action or a cluster (not always a sequence) of actions upon detection of a “fact” that can be resolved in fully automatic way.

 

Assisted reactive automation or simply assisted automation involves people to take actions to resolve a “fact” or general anomaly and assist the people with knowledge, tools and mechanism.

 

Events and Facts (and cause and effect)

Facts are behavioral observation of an ecosystem at the sufficient macro level over an indefinite period of time. E.g. system is down for last 2 hours is a fact.  In theory, system “was” down at 8:30 AM PT can be treated as an event, but modeling event at that macro level may not be a good idea. System “went” down at 8:00 AM PT is an example of a good event. According to many, including Stanford http://plato.stanford.edu/entries/events/#EveVsFac, every event has associated fact while still they are distinct. A rather complicated example is – system “went“ down at 8:00 AM is an event, and system was or currently is “down” is a fact. Observe the transition or rather discreteness in, what we call, the event, which is not present in, what we call, the fact. For practical reason and simplicity we will further distinguish facts and events; e.g. rather than qualifying “system was down at 8:00 AM” as fact we will qualify “system was down since 8:00 AM for 2 hours” as the fact. Now see the fact is an observable effect of event or events in the event space. Lets take another look at the event we defined “system was down at 8:30 AM” – it’s apparent that it is a piece of sampling of the fact  “system was down since 8:00 AM for 2 hours”. In many acceptable academic theories, it may be correct, but for designing an uncluttered system, we will further distinguish events from merely being sampling of a fact. In that spirit, we will disqualify “system was down at 8:00 (or 8:30) AM” as an event – that’s the slice of the sampling of the fact (the effect). We will rather push the events down as the cause and not the sampling of facts. With that said, “system went down at 8:00 AM” is a good exhibit or indication that contributed to the fact that system was down since 8 AM for 2 hours; it has all theoretical accessories to be qualified as an event like temporal attributes. We can conclude for now saying events are in “exhibit space” and facts are in “conclusion space”. To go further we can call “system is currently down since 8 AM” as “unresolved fact” and “system was down since 8AM for 2 hours” is a “resolved fact”.

 

Conclusion

First major step in reactive automation and assisted reactive automation is about detecting the defined “facts” and deciding an action or series of actions on it. Detection of defined “facts” involves analyzing the “events” in “event space”.  “Conclusion” is the mechanism to detect a “fact”.

 

Below are the mutually non-exclusive kinds of “facts”.

1)   Unresolved and resolved facts – the facts that is currently valid is an unresolved fact and fact that has an end is a resolved fact.

2)   Weighted/Measured facts – Power outage in 10% of the network and power outage in machines 1,3,4 are the examples of weighted of measured facts.

3)   Concrete facts and Undefined Anomaly – concrete facts are well defined and they have well known conclusion (the mechanism to detect the facts) whereas an undefined anomaly is an irregular behavior in the event space. Detecting an undefined anomaly, which we will call just anomaly now onwards for simplicity, is complex and involves complex “conclusion”.

4)   Malignant facts and benign facts – Malignant facts are those that affect the ecosystem from functioning properly or show bad health of the system indicating signs of future troubles and need “corrective measures” or “corrective actions”. Corrective measures and corrective actions are different – we can see detailed differences soon. Benign facts, in the other hand, need no action or just goodwill FYI action like sending a nice report to executives saying how well the system or part of it is functioning. E.g. the newly installed antivirus successfully prevented a new virus attack in the last night.

 

Corrective measures vs. corrective actions

Linguistically the distinction may not be big but for better clarity in system design we will make clear distinction between them. Corrective measure is something that is more permanent in nature than a corrective action.  E.g. installing new A/Cs to solve overheating problem (exhibited as facts) is a corrective measure. Running a forced garbage collection to free up overused memory and replacing the cable chewed up by a rat are examples of “corrective actions”.

 

Some more constructs of reactive automation or assisted reactive automation or just assisted automation:

1)   Monitoring: is the mechanism to provide the ability to analyze any activity in the system generally categorized at multiple different levels. There could be some un-categorized monitoring too.

2)   Monitor Log: logging the different levels of monitoring.

3)   Event Log: explicit log by identified or marked event.

4)   Watch: ability to watch for a special occurrences or events during monitoring.

 

1. Reactive Automation
   1. Self rearrangement and self healing
   2. Assisted Reactive Automation or Assisted Automation
      1. Automation at command (by choice)
      2. Focused knowledge assistance
      3. Crowdsourcing

 

Now the solution – lol I don’t have any readymade since every single situation of automation comes with different context. Using Machine Learning (ML) is a very important factor in this, but definitely not singularly important one. There are many other things that need close consideration like choice of languages used, choice of runtime, organizational behavior, skills, people, many other things. I will write soon about an abstract solution model using some available technologies in today’s open-source world. 

Curl to access Spring Security protected application

Found a post on it - very useful - though it misses 'resources' dir in the standard name - the corrected one is here

curl --data "j_username=&j_password=" http://localhost:8080/yoursecureapp/resources/j_spring_security_check --cookie-jar cookies.txt

Then curl http://localhost:8080/yoursecureapp/secureservice --cookie cookies.txt

The original link - https://bowerstudios.com/node/913

Node JS, Knockout JS, Mongo DB, RabbitMQ - the killer combination

Lets start with the Node JS for the first installment
======================================================
QUICK NOTE: By readers request - browse towards the end to find file upload example using Node and formidable.

Seriously. Look at the below code snippet.

var http = require('http') http.createServer(function(request, response) { response.writeHead(200, {"Content-Type":"text/plain"}) response.write("Hello World") response.end() }).listen(8888)

And you wrote a HTTP server. Its as easy. This is not the end but just the start. This is written in NodeJS. You can find the Node installation guide here. To run this issue this in the terminal node server.js - assuming you wrote the above code in a file called server.js. Then fire a browser and access localhost:8888 - and we will see "Hello world" printed on your browser. Another thing to note in the above code snippet - we passed an anonymous function to anther function - which is an important characteristic of a functional language. Why should I consider NodeJS in the first place? There are so many to choose from. Alright, not all technologies are fit for everything. Node is not fit for everything too. It has its own forte. Java and likes are good in their area Node and likes are good in their territory. Node JS suits well at the frontend server layer - that supplies webpages, handles user interactions, constructs dynamic pages, talks to other system asynchronously. But why we should use Node in these areas? The major essence of the answer lies in the code snippet - Node is event driven. Does it make it fast? No - in raw speed it is not fast, but its being event-driven and asynchronous it provides better responsiveness to the users - does not block calls. This feature in Node is called evented IO - we will examine that later what it really is - and how it works.

IMPATIENTS CAN SKIP THIS SECTION
Now let me take a step back and tell you where I am going and why. The context is fairly complex web application. In web applications we need to consider user interaction, page rendering, occasionally internationalization/localization, form handling, file upload/download, interacting with other systems, information processing, algorithmic processing, etc. Not a single technology is good for all these tasks. In this post we are going to see a good technology stack that works - achieves better results.
IMPATIENTS CAN RESUME HERE
Enough talks. Lets create a RESTful web-service. We will just refactor the the above code. server.js var http = require('http') var url = require('url') function start() { function onRequest(request, response) { var pathname = url.parse(request.url).pathname console.log("Request for pathname " + pathname + " received") response.writeHead(200, {"Content-Type": "text/plain"}) response.write("Hello world from refactored server") response.end() } http.createServer(onRequest).listen(8888) } start() To understand in little deeper Node nitigrities I recommend to start with awesome book The Node Beginner Book by Manuel Kiessling. The examples here are heavily influenced by this book. I refactored codes there in order to make it sometimes efficient and sometimes simple. Lets write one more program called router.js - a special program to handle the incoming requests. We will figure out how to use it in context. function route(pathname) { console.log("request for path - " + pathname + " received through function route") } exports.route = route exports.route makes route (any member - here is a function) available to other programs - remember it. A little refactoring of server.js. var http = require('http') var url = require('url') function start(route) { function onRequest(request, response) { var pathname = url.parse(request.url).pathname route(pathname) response.writeHead(200, {"Content-Type": "text/plain"}) response.write("Hello world from refactored server") response.end() } http.createServer(onRequest).listen(8888) } exports.start = start Now see we exported the method start. That means we want to call it from another program and not executing within server.js.

The difference here is in line 8. We are delegating the processing of pathname to another function route in another program - router.js.

You can just run the server by adding 2 more lines - one in the beginning - var router = require('/router.js') and one in the end - start(router.route). Now run server.js. - node server.js. Point the browser to http://localhost:8888/anything and observe the console.

If you want to make the server.js working without the additional two lines - which is the the right way - add another program - lets name is index.js. As simple as it could be - var server = require ('./server.js') var router = require('./router.js') server.start(router.route) Run node index.js instead of server.js, hit the browser - observer the console.

Cool, router.js so far is in very primitive stage now - lets enhance it. Lets introduce one more program called handler.js. That maps the incoming requests to appropriate functions. var handle = {} handle['/'] = start handle['/start'] = start handle['/upload'] = upload function start() { console.log("method start") } function upload() { console.log("method upload") } exports.handle = handle In order to use this we need to refactor router.js, server.js remains unchanged. The refactored router.js var handler = require('./handler.js') var handle = handler.handle function route(pathname) { console.log("request for path - " + pathname + " received through function route") if(typeof handle[pathname] === 'function') { handle[pathname]() } else { console.log("No request handle found for pathname " + pathname) } } exports.route = route Now you can just run node index.js. Access http://localhost:8888/start, try http://localhost:8888/upload and also try something other than satrt and upload, and observer the console. We have pretty decent HTTP server that has RESTful service capability. Being a heavy Spring 3.1/Roo user - I still can appreciate the rapid development capability using Node - more than rapid development capability - it has some more features that makes it appealing/useful. That is more directness - no big un-understandable convention over configuration theory - it has rather a very astute functional trait - simple and direct programming model. Productivity and great programming model alone can not be the criteria to choose a programming language - we must see what else it brings to us. Non-blocking evented IO is the coolest thing I have seen in a while, which makes user interaction more responsive. Node is not the best in many things - and we must not use it in those areas. Node is not very good in heavy information processing, algorithmic tasks etc. We can of course utilize Node's capability to delegate task in non-blocking way to other programs written in capable languages (may be to Scala or Java for the grandpas - I am uncle at this moment:)) for heavy processing and algorithmic tasks. And Node can be used to handle user interactions, page rendering and and a as top-middleware integration layers - that does the delegation without blocking. BTW - Node has very close connection with C++, and we may want to write some node stuffs in C++ - I have not done this yet myself, if anybody did it, please post your experience here. If we talk in official node dialect - Node is an one big, efficient event loop. To read more about it Mixu's tech blog - Understanding the node.js event loop

FILE UPLOAD EXAMPLE - Using Node JS and Formidable

Lets refactor the code to have a file upload application - I am going to post the the complete code and only code here. It is improved version of what is on the Manuel Kiessling's book.
index.js var server = require ('./server.js') var router = require('./router.js') server.start(router.route)
server.js var http = require('http') var url = require('url') function start(route) { function onRequest(request, response) { var pathname = url.parse(request.url).pathname route(pathname, request, response) } http.createServer(onRequest).listen(8888) } exports.start = start
router.js var handler = require('./handler.js') var handle = handler.handle function route(pathname, request, response) { console.log("request for path - " + pathname + " received through function route") if(typeof handle[pathname] === 'function') { handle[pathname](request, response) } else { console.log("No request handle found for pathname " + pathname) } } exports.route = route
handler.js var handle = {} var querystring = require('querystring'), fs = require('fs'), formidable = require('formidable') handle['/'] = start handle['/start'] = start handle['/upload'] = upload handle['/show'] = show function start(request, response) { console.log("method start") response.writeHead(200, {"Content-Type": "text/html"}) response.write(require('./pages.js').body) response.end() } function upload(request, response) { console.log("method upload has been called") response.writeHead(200, {"Content-Type": "text/html"}) var form = formidable.IncomingForm() var util = require('util'), fields = [], files = [], i = 0 form .on('error', function(err) { console.log("error handled?") response.end("=>Error occured. Did you try to access a post only link without post data?") }) .on('field', function(field, value) { console.log(field, value) fields.push([field, value]) }) .on('file', function(field, file) { console.log(field, file) files.push([field, file]) i += 1 fs.rename(file.path, "/tmp/tttest"+i+".jpg", function(err) { console.log("renamed") if(err) { console.log("error renaming") fs.unlink("/tmp/test.jpg") fs.rename(file.path, "/tmp/tttest"+i+".jpg") } }) response.write("eeee ") }) .on('end', function() { console.log(util.inspect(files)) response.end("something got uploaded") }) form.parse(request) } function show(request, response) { console.log("method show") fs.readFile("/tmp/tttest1.jpg", "binary", function(error, file) { if(error) { response.writeHead(500, {"Content-Type": "text/plain"}) response.write("error" + "\n") response.end() } else { response.writeHead(200, {"Content-Type": "image/jpeg"}) response.write(file, "binary") response.end() } }) } exports.handle = handle
pages.js

var body = ''+
	''+    
	''+''+''+     
	'
'+     
	''+     
	''+     
	'
'+ 
	''+     
	'';

exports.body = body

Just make sure this pages.js is well formatted. All other files should be good but since this one has HTML code into it - I made some deliberate spelling mistakes in tag names - make sure that are right and well formatted - and it will work.

====== Enjoy the Node.JS - rest of the article will be available soon ======

Lets bring the Rabbit here - Welcome RabbitMQ. We just need it to delegate tasks from node to Java 

======= Adding ======

 

 

As the first step to integrating NodeJS and Java using AMQP (RabbitMQ), lets first see how to connect NodeJS and RabbitMQ - both publishing messages and consuming messages. A little bit of change in server.js, have a look.

 

server.js var http = require('http') var url = require('url') var amqp = require('amqp') function start(route) { var connection = amqp.createConnection({ host: 'localhost' }) connection.setMaxListeners(200) connection.on('ready', function() { var exchange = connection.exchange('amq.fanout', {'type': 'fanout', durable: true}, function() { var queue = connection.queue('queueA', {durable: false, exclusive: true}, function() { queue.subscribe(function(msg) { console.log("Message received: " + msg.data.toString()) }) queue.bind('amq.fanout', 'key.b.a') queue.on('queueBindOk', function() { http.createServer(onRequest).listen(8888) }) }) function onRequest(request, response) { var pathname = url.parse(request.url).pathname route(pathname, request, response, exchange) } }) }) } exports.start = start

There are some changes in this code snippet above, lets analyze those chnages minutely before we go to the codes. Line number 11 - thats says connection.setMaxListeners(200) This one I had to use to get rid of one problem I was getting on the node console
(node) warning: possible EventEmitter memory leak detected. 21 listeners added. Use emitter.setMaxListeners() to increase limit

Line number 6 is very straight forward - I just created the AMQP connection where my RabbitMQ was running on localhost on default port. Line number 9, I am doing everything when the connection with the AMQP server (tested with RabbitMQ only) is ready; which may not be the most efficient way as even the HTTP server is being created inside this - that makes HTTP server to depend hard on AMQP server - if AMQP server is not up HTTP server would not be up - bad design but I will let you figure out the best way to tackle it. A little bit of error handling can solve this problem easily. Line number 10-11 - it is very clear that I am creating Exchange and Queue, at line number 13-15 - this code will wait for messages in the defined queue, and if there is any message in the defined queue it will print it on the console. Line number 17 is just the queue binding - remember this is JavaScript and asynchronous so this line may get executed before others in this function. Line number 19-21 I am creating the HTTP server upon successful binding of the queue. Everything else remains same except one more line, line number 25, I added one more parameter to the function route(pathname, request, response, exchange), that is exchange, since this is required to publish a message to the designated queue, we are passing it to the router and then handler so that respective methods can use it in order to publish messages. As simple as that.

Lets look at router.js as the next piece of code, change is very small, only at one place - I am going to talk about that after the code snippet.

router.js var handler = require('./handler.js') var handle = handler.handle function route(pathname, request, response, exchange) { console.log("request for path - " + pathname + " received through function route") if(typeof handle[pathname] === 'function') { handle[pathname](request, response, exchange) } else { console.log("No request handle found for pathname " + pathname) } } exports.route = route

Change is only at line number 5 and line number 9; the new parameter exchange, if you remember I passed it from server.js. This will be used from a method in handler.js to publish messages to the defined queue. Lets see handler.js now.

Notice that I reduced handler.js code for brevity by deleting all file upload related part that was there before. Nevertheless, this code is fully functional, if you want to try this out.

handler.js var handle = {} var querystring = require('querystring'), fs = require('fs'), formidable = require('formidable') handle['/publish'] = publish handle['/messageform'] = messageform function messageform(request, response, exchange) { console.log("method messageform") response.writeHead(200, {"Content-Type": "text/html"}) response.write(require('./messageform.js').body) response.end() } function publish(request, response, exchange) { var form = formidable.IncomingForm() form .on('error', function(err) { console.log("error handled?") response.end("=>Error occured. Did you try to access a post only link without post data?") }) .on('field', function(field, value) { console.log(field, value) exchange.publish("key.a.b", value) }) .on('end', function() { response.end("something good happened") }) form.parse(request) console.log("method publish") response.writeHead(200, {"Content-Type": "text/html"}) response.write(require('./messageform.js').body) response.end() } exports.handle = handle

We are almost done - if you run this project with node index.js and point your browser at http://localhost:8888/messageform, the line number 16 of the above code would be invoked. Now wait a minute you need one more file, thats the final one to demonstrate AMQP communication - that is messageform.js. Look at line number 19 in the above code and you will understand why that is required. Its essentially a HTML snippet. Its very difficult to list HTML file on typepad, let me see how much I can. You will get a clear idea in either way.

messageform.js var body = ''+ ''+     ''+''+''+      '

'+      ''+      ''+      ''+  ''+      ''; exports.body = body

Now once you run the project by using node index.js and point your browser at http://localhost:8888/messageform, you will see the magic. Do not forget to run RabbitMQ on your localhost too.

Node and Java communication - Now we understand how to publish and subscribe messages from NideJS using AMQP server; its just easy to connect Java with NodeJS if we can write Java AMQP publish/subscribe code. Its just easy - spring-amqp also has a nice project to do that. I will write a simple Java program sometimes later if required. Let me know if that is something you guys want.

=================

NodeJS with MongoDB is coming soon - stay tuned. Want it fast - then get ready to pay for it :)

Deep dive into Scala

<<All the examples here are complete and are fully executable>>

This post is being written and published step-by-step. The theme of this entire post will be around why I want to use scala in my enterprise application. The features we will discuss include

• Functional paradigm of scala
• Traits and Mixin 
• Concurrency 
• Notable patterns in scala

For the programmers who just heard and thought enough about scala but did not get a chance or time to dig it in real - I encourage them to have a look into my initial post on scala Dive into Scala  and get started with it in 10 minutes including downloading and installing - I don't guarantee it though if your internet connection is slow and you are using a windows box ;) 

In one sentence why I picked up this language for my next one years primary language for enterprise applications goes like this - Scala beautifully combines object-oriented paradigm with functional paradigm while allowing me to access and leverage the richness of entire java world (true for JVM version of scala). 

In the enterprise application development arena there is no right most language, we should be able to mix and match languages and technologies fluently. Since mainstream scala is a JVM language, it is tuned to that philosophy of mixing and matching with other JVM languages along with our good old Java. In near future I will write on how to use scala, groovy and java simultaneously without compromising on performance and complexity. In fact the intention of mixing and matching is not to have just mixture for sake of it but to use what is best at what. In mix and match our focus will be to improve expressiveness, performance, code compactness. Just 2 complexities are there in this approach; those are - 1) putting write thing at the right place - meaning choosing right language to do get right thing done 2) deep knowledge about multiple languages. Lets stop this philosophy here - lets see more what scala has to offer us.

Quoting directly from official website of scala http://www.scala-lang.org/. But.... but this is pretty "official" - just see what they say - but in this post lets talk about in our own language in our own feeling. 

"Scala is a concise, elegant, type-safe programming language that integrates object-oriented and functional features.

Scala is fully interoperable with Java."

This is seen when you open http://www.scala-lang.org for the first time in the banner. It gets changed though :) 

A little lengthy quotation from scala-lang.org. You know what? even I did not have patience to read these much talked about features in "official" voice. But its good to read and tune in with their way of thinking to really get into it. For the time being read if you can otherwise just look at the bulleted points. We will anyway anyway discuss these in our own way.

"Scala is a modern multi-paradigm programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages.

• Scala is object-oriented: Scala is a pure object-oriented language in the sense that every value is an object. Types and behavior of objects are described by classes and traits. Class abstractions are extended by subclassing and a flexible mixin-based composition mechanism as a clean replacement for multiple inheritance.
• Scala is functional: Scala is also a functional language in the sense that every function is a value. Scala provides a lightweight syntax for defining anonymous functions, it supports higher-order functions, it allows functions to be nested, and supports currying. Scala's case classes and its built-in support for pattern matching model algebraic types used in many functional programming languages.
• Scala is statically typed: Scala is equipped with an expressive type system that enforces statically that abstractions are used in a safe and coherent manner.
• Scala is extensible: The design of Scala acknowledges the fact that in practice, the development of domain-specific applications often requires domain-specific language extensions. Scala provides a unique combination of language mechanisms that make it easy to smoothly add new language constructs in form of libraries:
  • any method may be used as an infix or postfix operator, and
  • closures are constructed automatically depending on the expected type (target typing).
  A joint use of both features facilitates the definition of new statements without extending the syntax and without using macro-like meta-programming facilities.
• Scala interoperates with Java and .NET: Scala is designed to interoperate well with popular programming environments like the Java 2 Runtime Environment (JRE) and the .NET Framework (CLR). In particular, the interaction with mainstream object-oriented languages like Java and C# is as smooth as possible. Scala has the same compilation model (separate compilation, dynamic class loading) like Java and C# and allows access to thousands of high-quality libraries." 

Now Lets see what really these means to us. We are going deeper. After discussing about the applicability of the language - thats the core philosophical essence, we will look into the advanced language features - such as useful, beautiful and fancy syntaxes, annotations, etc. 

Now lets see what does "Scala is functional" mean. Lets see an example before we discuss theory.

object Sum { def sumOfInts(a: Int, b: Int): Int = if (a > b) 0 else a + sumOfInts(a + 1, b) }

This is it. Run it using scala shell or as a scala file. We defined sum of integers between given two numbers in scala. Does not it look like pure algorithmic? Almost the same way we would right the algorithm for it. No unnecessary jugglery of temporary variables. This is the first step towards getting into the functional concept of scala. This is just the tip of the iceberg lets get into the deeper stuffs now.

To reiterate - Scala is functional - we saw a little bit of that now in the line of using it in closer way with the problem domain. Now we will strengthen again the statement "Scala is functional" by saying and showing the functions are objects too in scala. Functions are Objects in scala. Lets have a look into the following routine

object Repeater { def onceInOneSec(cb: () => Unit) { while (true) { cb(); Thread sleep 1000 } } def stringPrinter() { println("print in in given frequency") } def main(args: Array[String]) { onceInOneSec(stringPrinter) } }

We will do two things now. We will see how int he above example a function is treated as object and alongside we will also decode some of the seemingly cryptic syntaxes of scala. Lets go.

In the function main - the arguments in main is pretty similar to that of java - we have a method call onceInOneSec and we - very important - pass another method stringPrinter as argument. It is one of the reason why scala is called functional language. About syntax - I think they are mostly self describing with one special mention is () => Unit it means a function that takes no argument and return nothing. So when we say def onceInOneSec(cb: () => Unit) it means definition of a function that takes a function of type no-argument-nothing-returns as argument where cb is the identifier or the "object" of type function.  Unit is nothing but same as void in C/C++/Java.

Lets modify the code we have written in the following way. object Repeater { def onceInOneSec(cb: () => Unit) { while (true) { cb(); Thread sleep 1000 } } def main(args: Array[String]) { onceInOneSec( () => println("print it in given freq")) } }

In the above example we have eliminated one method stringPrinter. Because it was being called only once. We defined it inline while calling the onceInOneSec function. This method has no name and is know as anonymous method. Cool.. we learned one more thing in scala.

We have seen really basic things so far, but I think these basics were required in the view that scala is pretty new kid to the bigger audience till now. One of my early blog readers commented that "whats deep in this" - he said it correct so far nothing deep, we are at surface and planning to dip deep. One of my favorite scala features is trait. Lets talk about now. 

Trait: people, even the scala-lang.org guys compare it with interface with option to have some implementations into it, where code gets reused. But for me its more than that. It brings a new way of designing and organizing my application - much robust way. Lets see one example of trait before we talk about my philosophical understanding.

trait Flying { def canFly(): Boolean = true def fly() { println("Flying.....") } def hasWings(): Boolean = true def maximumFlyingSpeed(): Int } trait Walking { def canWalk(): Boolean = true def walk() { println("Walking....") } def maximumWalkingSpeed(): Int def hasLegs(): Boolean = true def howManyLegs(): Int } class Dove extends Flying with Walking { def maximumFlyingSpeed(): Int = 30 def howManyLegs(): Int = 2 def maximumWalkingSpeed(): Int = 5 }

We saw an example of trait now. Definitely it looks like abstract class but a class can inherit from multiple traits. So multiple inheritance is possible beyond interface. A good facility indeed, But the key conceptual difference lies somewhere else. The concept helps crispier modeling. Lets think carefully when we think about interface or abstract class we think about parent - we define something that has generic stuffs and then make specialization in concrete classes. The notion of the parent class is a generic real life object itself. This concept forces us to find and model a generic real life object to be parent to the specialized ones. In real life we may not find anything generic and useful together. Also there can be common characteristics in more generic objects unless we break that generic object further - that leads to deeper and complex inheritance tree. Trait at the other hand brings something different in this space - trait can be thought of as characteristics. In the above example instead of defining a generic bird object we defined flying and walking traits that constitutes the real life object Dove, allowing flying and walking to be used independently elsewhere - as you know flying and walking can be other objects characteristics other than a bird. Make sense? 

In a nutshell the concept of trait gives me more instruments towards making my favorite modeling choice - aggregation/composition over inheritance. It really makes a lot of senses to me. 

Mixin: We will now see another powerful application of trait. That is mixin class composition. Very powerful and interesting feature. Lets look at one example. 

abstract class AbstractService { type T def firewallCompatible: Boolean def parameter: T def result: T }

trait Webservice extends AbstractService { def constructIPMessage(f: T => Unit) { f(parameter) } def firewallPingResult(f: T => Unit) { if(firewallCompatible) f(result) } }

class RestfulService extends AbstractService { type T = String def firewallCompatible = true def parameter = "ping the server in DMZ" def result = "fake ping back OK from server in the DMZ" }

object RSTest { def main(args: Array[String]) { class Service extends RestfulService with Webservice val service = new Service service constructIPMessage println service firewallPingResult println } }

If you are using scala> prompt just type
scala>RSTest.main(null) 

Lets understand this in our own way. If you look at the the code in RSTest we are composing a class that has type RestfulService and with the characteristics of Webservice. Note the the terms type and characteristics. Trait describes characteristics where as abstract class or interface describes type (super type). The example of class composition (Service) in the RSTest is called mixin-class composition. One thing to note - in my example there is a high potential of anti-pattern if not used carefully. The example here is very small and limited so the magnitude of the effect of anti-pattern (or whether it is anti-pattern here or not) is not clear enough. But if  trait is tightly coupled with the supertype (abstract class here), the supertype has to be defined very carefully, otherwise the usefulness of the trait will go. Another (non-code) example - lets say we have (super) types like Animal, WildAnimal, DomesticAnimal and characteristics (traits) like Agility, Herbivore, Carnivore. Now If we associate (make dependent) these traits with WildAnimal (what is association? In our example WebService is associated with AbstractService. Do not get confused about association in UML, we are not talking about that) its wrong in the context (anti-pattern) as a trait like agility can well be the trait of a wild animal like cheetah and a domestic animal like horse. So this trait should better be associated with (super) type animal and not (super - next super type) type wild animal. 

However trait can be completely independent of any type. Analogically it is like a range of wide and tele lenses. With wide angle lens your field of view (area of applicability in the case of trait) becomes wider and with tele lens your field of view becomes narrower. Be careful about how much field of view is appropriate for a particular scenario and then define the scope of a trait. Got this? Let me know if you understand otherwise.

==== Its a logical milestone but this post is under enhancement now, I am writing more. Stay tuned ====

Dive into Scala

<<If you know how to write your first program using scala go to Deep dive into Scala >>

You know java, ruby, groovy, python - unlearn those momentarily if you need to learn scala (for that matter any new paradigm). Though smalltalk and haskell knowledge may help. I learned a little bit of haskell to understand what the functional world of languages looks like. Any way scala is an awesome language but without unlearning mainstream languages it may appear seemingly difficult. Lets see one example - a bad old HelloWorld. Before that just download scala from http://www.scala-lang.org/downloads. Pick up appropriate package for your OS, install it, don't forget to make it available in your system path - you know how easy it is; I am not going to explain it here. Now lets say your done - commands scala, scalac are in your system path. Issue scala command from your terminal and you can see the scala shell as below. 

scala>

This is scala shell. Lets write a program directly there. 

scala> print("my name")

and it will display 

my name 

We have got the first dose of our scala - lets now see a real program

scala> object HelloWorld { | def main(args: Array[String]) { | println("Hello World!!!") | } | }

write the above code as it is written here with maintaining the lines from scala prompt. At the end when you press your last "Return/Enter" you see

defined module HelloWorld

And then run it like this from scala prompt

scala> HelloWorld.main(null)

And you will see

HelloWorld!!!

Cool... we saw how to write the first scala program. The difference so far with java or other languages that we defined object directly. Don't worry we can define class too. We will see that next.

scala> class HelloWorld { | def main(args: Array[String]) { | println("HelloWorld!!!") | } | } And you see the message
defined class HelloWorld

Lets run it now
scala>var hw = new HelloWorld
scala>hw.main(null) You see

HelloWorld!!!

on the display

Hello world program done. We should dive deep into the theories of scala. The values it brings, the added advantages over java and other mainstream and contemporary languages. The notion of functional language, the traits and mix-ins, the Actor and concurrency, the compact syntaxes etc. Tune in to my next post - coming in few hours.

One thing - obvious - forgot to mention. You can write the above to code examples in file. If the file name is HelloWorld.scala you can compile it from your terminal shell (not scala shell) using scalac HelloWorld.scala

and then run it from terminal shell like this scala -classpath . HelloWorld

Next read Deep dive into Scala

Spring application testing - an elegant way

Lets have a look the following JUnit test case. Notice the sleek and simple looks of this piece of code. package com.chiradip.rd; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.TestExecutionListeners; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.support.DependencyInjectionTestExecutionListener; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(locations = { "classpath:/META-INF/spring/module-context.xml" }) @TestExecutionListeners(value = DependencyInjectionTestExecutionListener.class) public class SimpleBeanTest { @Autowired private SimpleBean simpleBean; @Test public void testJustAMethod() { simpleBean.justAMethod(); } } As you can see we are testing a spring bean - named SimpleBean. This is autowired. Pretty simple and easy so far. As you can see it is a clutter free test case. SimpleBean.java is a very simple spring bean - just as below. package com.chiradip.rd; import org.springframework.stereotype.Component; @Component public class SimpleBean { public void justAMethod() { System.out.println("test"); } } N.B.: I am just using annotation based bean definition and detection instead of using XML. @Component annotation says SimpleBean is a spring bean. Just for those want to have a look into my spring configuration - it goes below. The above configuration file is given just to show how @Component annotation works. Coming back to the the test case. It looks simple and sexy - and has 3 annotations on the test class "SimpleBeanTest". Most important annotation. as you can guess, is the second one @ContextConfiguration. locations attribute there overrides the default location and specifies the location of the spring configuration file(s) those need to be loaded into context during test. You got it right - it loads the the bean definitions and makes it available in the current context so that the test methods can retrieve those defined beans - including using @Autowired. Lets now see the first annotation @RunWith. "By annotating test classes with @RunWith(SpringJUnit4ClassRunner.class), developers can implement standard JUnit 4.5+ unit and integration tests and simultaneously reap the benefits of the TestContext framework such as support for loading application contexts, dependency injection of test instances, transactional test method execution, and so on." It says it all. Now the 3rd (last) annotation @TestExecutionListeners is optional and not always used. For example in our example its doing nothing. This is very instrumental to control (and/or capture) behavior of the test case under execution. Find more here Springframework 3.0.X testing.  In case you want to look at my pom.xml it goes here below. Note - I used all the artifacts from springsource bundle repository. These are all OSGi-fied. Though OSGi benefits and features are not used in this example - it is ready to be used in OSGi context. Take google's help if you need to know more about OSGi - the future of the the application backbone. 4.0.0 com.chiradip.rd space 1.0-SNAPSHOT jar space http://www.frameworkfactory.org UTF-8 3.0.3.RELEASE org.junit com.springsource.org.junit 4.8.1 test org.springframework org.springframework.core ${spring.version} org.springframework org.springframework.beans ${spring.version} org.springframework org.springframework.context ${spring.version} org.springframework org.springframework.context.support ${spring.version} org.springframework org.springframework.aspects ${spring.version} org.springframework org.springframework.aop ${spring.version} org.springframework org.springframework.instrument ${spring.version} org.springframework org.springframework.expression ${spring.version} org.springframework org.springframework.test ${spring.version} test org.apache.commons com.springsource.org.apache.commons.logging 1.1.1 org.aspectj com.springsource.org.aspectj.runtime 1.6.8.RELEASE org.apache.maven.plugins maven-surefire-plugin **/*Test.java **/Abstract*.java org.junit:com.springsource.org.junit com.springsource.repository.bundles.release SpringSource Enterprise Bundle Repository - SpringSource Bundle Releases http://repository.springsource.com/maven/bundles/release com.springsource.repository.bundles.external SpringSource Enterprise Bundle Repository - External Bundle Releases http://repository.springsource.com/maven/bundles/external

Open framework for Ultra Large Systems

Today every enterprise application that encompasses fairly large number of users and provides services or solutions to the users are ultra large in nature - as it involves many systems, many services, many users, many organizations and many management domains. We define ultra large systems as a system having span into different ownership domains, sizable numbers of processing unit types, large number of users and which provides services to the users or involved parties where the service is constituted in collaboration of different capabilities coming from different ownership domains. 

Now question is what this framework, framework for ultra large systems, would do? Well, there are some common technical "capabilities" required for these systems. Base of every system is the infrastructure of these capabilities. For each solution we build we either create this infrastructure of technical capabilities from ground up, but mostly in an ad-hoc way, or we rely on some proprietary technology stack that controls the entire systems backbone in very closed and intrusive manner. Both are not good in most of the situations. When such infrastructure is built ground up and in ad-hoc basis, we can not envision the technical capabilities distinctly upfront and hence the infrastructure of technical capabilities become unmanageable making the systems unstable in turn. The solution team also spends a lot of time on technology decisions and making all those capabilities working together. Our intention is to take the pain out from solution team and provide them a solid base on which they can start building business focused offerings straight-on. While the focus of a solution team should be on building business focused offerings, they too have constraint of using base systems, or runtime systems, that the organization, for which the system is being built, invested already. Challenge is in two folds here, protecting the investment on base runtime systems and focusing on business focused offerings. Answer to these problem is to build a base that is more or less runtime system independent, and combines the common technical capabilities so that the capabilities can collaboratively work together as and when needed. The target system may also use these capabilities in a-la-carte fashion. But the chosen capabilities are never discrete but always in a harness that makes them integrated and collaborative. We are calling this answer, the base we are discussing about, the framework. Let see the precise definition of the framework.

The Framework: An extensible harness of common technical capabilities found in modern large systems with the following special mentions and constraints.

• No runtime binding - e.g. it should run on any JEE compliant application server irrespective of its vendor.
• A-la-carte model of technical capabilities. Use what you want and discount others. 
• Standard based technical capability binding, making those technical capabilities interchangeable
• Pluggable functional capabilities

Let me now propose what are technologies are going to be used in this framework and then I will explain why my choice is so.

The technologies.

• OSGi - Blueprint Services
• EIP - Spring Integration and Apache Camel
• Spring Batch
• Spring Security
• Java, Scala and Groovy 

I will hold back for some times the explanation of the technology choices. Lets look at now how the common technical capabilities constitutes the harness, the infrastructure of set of common technical capabilities, the framework. Lets see the picture.

=================Picture: The Framework===================