Archive Log files Using Spring Data MongoDB

MongoDB was on my ToDo list for a while. Finally I decided to stop procrastinating and give it a spin. Since simple hello-world app is no fun, I was thinking of some real life use cases of NoSQL database. One simple use case which I could think of was to create a centralised log store. MongoDB already has a Log4j appender using which application logs can be directly written to MongoDB. But archiving the existing logs in MongoDB is something which is not available out of the box. Also sometimes it’s convenient to write application logs to regular log files but use a store like MongoDB for archival purpose.

Being a big fan of Spring Framework, I decided to use SpringData MongoDB module. Now I don’t have to worry about writing all the boilerplate code to deal with MongoDB. Also I use Spring Integration for file polling mechanism. The following blog post elaborates how I use spring-data-mongodb along with spring-integration framework to archive log4j log files in a mongodb store.

Activiti 5 with Spring Integration 2.0 Mail Adapter

Activiti is an open source BPM and workflow system. The first GA release is expected to be out next month i.e. Dec 2010. The roadmap of activiti looks very promising and also involvement of companies like SpringSource and MuleSoft can make it even more interesting. There are couple of good articles and tutorials available on the wiki to help you get started with activiti. To get a feel of the framework, instead of developing a simple hello world app, I thought of integrating activiti with Spring Integration. SpringSource team is working on this integration module and once that is in place some of the boilerplate code from my prototype would be cleaned up. The following blog post demonstrates the use of spring integration mail module with activiti.

jBPM4.4 with Spring3

I’m working on a jBPM prototype since last couple of days. The integration with Spring wasn’t a smooth ride. In my previous projects, I used jBPM3.2. But it seems jBPM4 is pretty much a rewrite. There are couple of few nice additions like service API’ for task and process management, support for java task and many more. And there are few changes which I didn’t like. Till 4.2 the support for JPA was provided, which is taken off starting 4.3 release. Yes definitely you can write some wrappers and still achieve it. But!! Between 4.3 and 4.4 also there are some major API changes. So the whole point is to get the app running, I had to digg into mailing lists and source code. Now that it works, I thought I would write a short blog about it.

Using Spring Integration with Twitter4J to Email Tweets

Recently I was playing around with Spring Integration. To understand any new framework one cannot just rely on documentation. So I created a demo application to try out different features of the framework. Since my main motivation was to understand Spring Integration framework, I wanted to spend minimal efforts in input data generation. So I decided to work with Twitter messages.

Most of you must be familiar with Twitter. Twitter is a service for friends, family, and co–workers to communicate and stay connected through the exchange of quick, frequent messages. There are different clients available to use Twitter. In the sample application, I first read the friends timeline. Then based on the source/client used for tweeting, the tweets are routed to different channels. The tweets originated from “web” are simply logged, while the tweets originated from “Dzone.com” are dispatched using a mail sender. The basic flow of the application is depicted in the following diagram.

Beauty of Spring Batch

Spring source community is coming up with spring batch 2.0 in Q2-2009. Spring batch is the first java based framework for batch processing. I think the decades of experience of Accenture in enterprise batch processing really helped for defining the use cases.

Most of the batch applications need to process high volume business critical transactional data. While doing so some set of non functional requirements (NFR) are sort of mandatory in such applications. These NFRs include performance, scalability, restartability, repeatability. I worked with couple of investment banks and my experience says that such batch applications are developed based on either Messaging model or Multi-threading model. Lot of efforts and time is spent by architects, developers and testers in building this robust infrastructure for batch processing. Also we cannot overlook the cost involved. Whenever you move across projects, you end up creating your own batch processing framework. Sigh!!

Some nice features that are introduced in Spring Batch 2.0 are conditional step execution, finer metadata access control and chunk based processing. To perform chunk based processing, we need to configure the commit-interval in a step. The transaction is committed after number of items specified in commit-interval are processed.

<step id="step1" job-repository="jobRepository" transaction-manager="transactionManager">
      <tasklet reader="itemReader" writer="itemWriter" commit-interval="10"/>
</step>

Given the features provided and use cases handled by Spring batch, it can prove to be the de-facto framework for enterprise batch applications.