Monthly Archives: September 2012

Claimcheck Pattern using Spring Integration and Gemfire

Introduction

For people who are in hurry, here is the code and the steps to execute. Please also refer to my blog on Publisher/Subscriber using Spring Integration and Gemfire as the ESB.

In a Enterprise Service Bus (ESB) architecture, the “Service Bus” will become bottle neck over a period of time. In a typical ESB you write to the Bus and lot of subscribers read from the bus. Gemfire can help ESB to achieve Distributed caching, by virtue of being powerful inmemory distributed caching backbone. Claimcheck pattern is an Enterprise Integration Pattern where in a Publisher component can publish a large payload into Gemfire cache, a GUID key is generated for this payload and placed on the Topic. A subscriber in the Claimcheck pattern will be listening to the Topic for this key and get the payload from Gemfire. The Dataflow diagram is as follows,

Spring Integration Gemfire Claimcheck Pattern

Spring Integration Gemfire Claimcheck Pattern

Claimcheck Pattern using Spring Integration and Gemfire

There are lot of good example from David Turanski in Github to learn Spring Integration with Gemfire. In this prototype, I built a simple replicated region based server based on the idea from basic\replicated-cs example from David’s examples. In order to run this prototype, download and install Activemq and start ActiveMq and create a Topic called “MyTopic”. After that, run the Gemfire Replicated Server as below.

cd gemfire-server
mvn package
mvn exec:java -Dexec.mainClass="org.springframework.data.gemfire.examples.Server"

Now Maven import the claimcheck-subscriber project into STS IDE and start the tomcat instance by clicking “Run on Server” this project. The subscriber will start listening to the messages/GUID in the topic “MyTopic”

Finally publish the XML payload on to the Gemfire cache by executing below command,

cd claimcheck-publisher
mvn test

In the Tomcat server console you will see below message

[Headers={timestamp=1348935448580, id=77480754-a4d8-4bc4-a529-26c9e26a7724, jms_timestamp=1348935447237, jms_redelivered=false, jms_messageId=ID:krishna-PC-62975-1348935446968-1:1:1:1:1}]
29 Sep 2012 12:17:28,702[topicContainer-1] DEBUG: com.goSmarter.gemfire.claimcheckpattern.ServiceActivator - ### PAYLOAD ###<!--?xml version="1.0" encoding="UTF-8"?-->

goSmarter

Conclusion

In this demo, I have demonstrated the claimcheck pattern. In an enterprise ecosystem, this is a basic component of claimcheck pattern. The enhancements on top of this includes various other things including,

  • each subscribers doing partial checkout of payloads from Gemfire.
  • it also need to have a way to work in a cluster environment.
  • Implementing distribution caching

I hope this example helped you.

Spring Integration JMS Mbean configuration

Spring Integration JMS message-driven-channel-adapter has a good MBean support through  DefaultMessageListenerContainer where we can control the flow of messages to the subscribers. In this blog I will demonstrate some of these capabilities. For people in hurry, here is the code @ Github,

  • Download activemq and unzip it
  • run activemq.bat under <activemq-home>/bin folder
  • Go to browser and type http://localhost:8161/admin/queues.jsp, create a new queue “MyQueue”
  • Get the latest code from Github
  • build the war file using “mvn clean package”
  • Deploy the war file in Spring tomcat container
  • “Send to” message to MyQueue in admin console of activemq in the browser
  • You notice the message gets consumed
  • Open jconsole.exe under jdk bin directory
  • Login in to Remote process “localhost:6969”, uid: admin, password: “springsource”
  • Go to MBean tab and expand spring.application/JMSContainer/testQueueContainer/Operation, click on stop operation
  • Again “Send to” message to MyQueue in admin console of activemq in the browser, if you notice, the message is not consumed
  • Go to MBean tab and expand spring.application/JMSContainer/testQueueContainer/Operation, click on start operation
  • Again “Send to” message to MyQueue in admin console of activemq in the browser, if you notice, the message got consumed

Details: Spring Integration JMS Mbean configuration

In a typical enterprise IT production environment, you need to control the message processing by the queue subscribers. Typical scenario would be, if there is a an issue with the subscriber and we want to pause the subscriber to troubleshoot it, we need a mechanism for this as below.

Expose Lifecycle interface of DefaultMessageListenerContainer


<bean id="queue.exporter" class="org.springframework.jmx.export.MBeanExporter">
<property name="beans">
<map>
<entry key="spring.application:type=JMSContainer,name=testQueueContainer" value-ref="queueContainer" />
</map>
</property>
<property name="assembler">
<bean class="org.springframework.jmx.export.assembler.InterfaceBasedMBeanInfoAssembler">
<property name="managedInterfaces">
<value>org.springframework.context.Lifecycle</value>
</property>
</bean>
</property>
</bean>

InterfaceBasedMBeanInfoAssembler class expose an interface of a particular class as a Mbean, in the above example it exposes Lifecycle

Managing JMS subscriber in JConsole

Managing JMS subscriber using JConsole

Managing JMS subscriber using JConsole

When you click on start operation, and click on start button, it will start the subscriber to listen to the message and when you click on stop, it will pause the subscriber.

Conclusion

MBean is a powerful way to expose some of the capabilities of Spring Integration’s various Channel Adapter, so that you can control them from various tools which are MBean aware, like JConsole, Hyperic . We can also expose, Log4J Mbean so that we can change the log level in the production system. More on this in my subsequent blog.

Measuring navigation flow JUnit test in Spring Web flow

Currently I am exploring Spring Web flow. Someone has already written a utility for measuring how well you are testing the Spring Web Flow web application navigation flow using JUnit. In this blog I am extending this utility to work with Sonar. The working project is here and Sonar plugin is here @ Github.

For people in hurry,

  • Get the latest code
  • Build the Spring Web Flow Test Sonar plugin using “mvn package”
  • Deploy sonar plugin jar located in the target folder into Sonar extension location and Start sonar
  • Run “mvn test sonar:sonar” on Spring Web Flow Coverage project
  • Finally you can go to Sonar and see the report as below

Problem Statement: Measuring Spring Web flow test coverage

In any web application there will be a page navigation flow. Frameworks like Spring Web flow, can externalize the page navigation as well as session management between each page in the navigation. In the below example, there is a simple usecase, where in we are presented with a food menu. Based on the menu presented, we can drill down further to findout more or we can cancel if we are not interested.

Details

Based on the usecase there are overall 13 distinct navigation paths in the Spring Web Flow. We can simulate all these paths by first writing the JUnit test and use the library to keep track of all the paths tested. In our test case only 7 paths are covered, out of 13. The sonar report is as below,

Spring WebFlow flow coverage

Utility to measure the navigation flow covered in a web application

We will adopt the Test first approach, by building the JUnit test to test the flow as per this blog.  The psedocode is as below,

private XMLFlowCoveragePathListenerAdapter listener = new XMLFlowCoveragePathListenerAdapter("test-flow");

public void testStartFlow() {
XMLFlowCoveragePathListenerAdapter.setTestName("testStartFlow");
XMLFlowCoveragePathListenerAdapter.iniliatizePath();

//test and assert the flow
}

public void testNextItemInFlow() {
XMLFlowCoveragePathListenerAdapter.setTestName("testNextItemInFlow");
XMLFlowCoveragePathListenerAdapter.iniliatizePath();

//test and assert the flow
}

public void testSecondItemInFlow() {
XMLFlowCoveragePathListenerAdapter.setTestName("testSecondItemInFlow");
XMLFlowCoveragePathListenerAdapter.iniliatizePath();

//test and assert the flow

//FINALLY END THE COVERAGE
XMLFlowCoveragePathListenerAdapter.endCoverage();
}

Conclusion

This utility is in a pre-alpha state. Currently it only show information on how many navigation flows it has covered against the total number as a percentage. There are few enhancements we need to take care of. For example, we need to provide a drilldown on what are the navigation flow that is not covered. We also need to build a threshold, where if it is less than that, it should break the maven build. Ofcourse, we need to cleanup the code.

I feel even in its present form this utility is useful. Feel free to use it.

Spring Drools: Spring Business Rules sample

Drools is a decent business rules engine from JBoss. One of the powerful features of Drools Business Rules Engine is something like a excel decision table. In this blog I will demonstrate how to implement Business Rules in Excel decision table feature using Spring. The working maven based project is @ Github . For people in hurry, you can just run mvn test and you can see what is going on!

This usecase is an example in Drools example, rewritten using Spring. The Pricing Rule decision table demo’s the use of a Excel decision table, in calculating the cost of an insurance policy based on Driver’s age, location, number of prior claims and calculates the base price and the discount.

Drools Business Rules

Business Rules

Spring Dools Excel decision table as per Drools examples

Row 2 of the Decision table indicates the package where all these class that is passed to the Decision table are present.
The decision table take 2 parameters, Driver and Policy if you notice in row 7 both these are indicated. Row 8 consists of the property (getter/setters) of the above objects, for example, age is a property of the Driver. When we pass a Driver object is passed to the rule engine, it will validate each rule and check which rule the Driver object is related to and applies those policies and discount to the Driver object. Let me walk you thru the example. As always, I will start with the JUnit test as below,

public class PricingRuleDTExampleTest {

PricingRuleDTExample ruleRunner;

@Test
public void testExecuteExample() throws Exception {

Driver driver = new Driver();
Policy policy = new Policy();
driver.setAge(30);
driver.setPriorClaims(0);
driver.setLocationRiskProfile("LOW");

assertTrue( ruleRunner.executeExample(driver, policy) == 120);
}
// now create some test data
ksession.execute(Arrays.asList(new Object[] { driver, policy }));

logger.debug("BASE PRICE IS: " + policy.getBasePrice());
logger.debug("DISCOUNT IS: " + policy.getDiscountPercent());

return policy.getBasePrice();

The test will fail because ksession has not been initiated using spring configuration. Spring configuration looks as below,

	<drools-spring:kbase id="kbase">
<drools-spring:resources>
<drools-spring:resource type="DTABLE" source="classpath:rules/org/drools/examples/ExamplePolicyPricing.xls" >
<drools-spring:decisiontable-conf input-type="XLS" worksheet-name="Tables"/>
</drools-spring:resource>
</drools-spring:resources>
</drools-spring:kbase>

<drools-spring:ksession id="ksession" type="stateless" kbase="kbase" />

<bean id="pricingRule" class="com.goSmarter.drools.PricingRuleDTExample">
<property name="ksession" ref="ksession" />
<property name="kbase" ref="kbase" />
</bean>

Now we need to go back and wire the test

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration( { "classpath:price-rule-config.xml" })
public class PricingRuleDTExampleTest {

@Autowired
PricingRuleDTExample ruleRunner;

Your test succeed.

Designing a simple usecase using Test First Approach using Spring

In this blog I will demonstrate designing a usecase with Test First Development approach. Let me explain the usecase first, we need to write an application, where we need to recommend a user some products based on his past purchase history, in which he has similar brands. Get the working code from Github.

If you see in the above usecase, there are 3 objects, User, Item object, and UserItemPurchaseHistory.

Test First Development

As in the test first approach, let us write the test as below,

@Test
public void testUserRelevance() {

ArrayList<Item> productFromApiList = new ArrayList<Item>();

Item Item = new Item();
Item.setName("Ferrari 750 Monza");
Item.setBrand("Ferrari");
productFromApiList.add(Item);

Item = new Item();
Item.setName("Porsche 911 GT3");
Item.setBrand("Porsche");
productFromApiList.add(Item);

Item = new Item();
Item.setName("Cheverlet Corvette");
Item.setBrand("Cheverlet");
productFromApiList.add(Item);

Item = new Item();
Item.setName("Honda City");
Item.setBrand("Honda");
productFromApiList.add(Item);

Item = new Item();
Item.setName("Hynduai Verna");
Item.setBrand("Hynduai");
productFromApiList.add(Item);

HashMap<String, Object> map = new HashMap<String, Object>();

List<Item> productFromApiListOut = productResponseServiceImpl.filterAsPerUserRelevance(productFromApiList, 2);

assertEquals(productFromApiListOut.size(), 1);
assertEquals(productFromApiListOut.get(0).getName(),"Ferrari 750 Monza");
}

Now we need to do database configuration for testing purpose. We will use Spring to configure embedded inmemory database as follows,

<bean id="sqlMapClient">
<property name="configLocation" value="ibatis-config.xml" />
<property name="dataSource" ref="dataSource" />
</bean>

<bean id="ibatisTemplate">
<property name="sqlMapClient" ref="sqlMapClient" />
</bean>

<beans profile="dev">
<jdbc:embedded-database id="dataSource" type="HSQL">
<jdbc:script location="classpath:SourceDB.sql" />
</jdbc:embedded-database>
</beans>

If you notice, there is SourceDB.sql where you have schema design and the seed data. Once you created the config file, you need to wire the same with the JUnit test as below and set the profile,


@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"classpath:applicationContext.xml",
"classpath:test-datasource-config.xml"})
public class FilterAsPerUserFriendsRelevanceTest {

static {
System.setProperty("spring.profiles.active", "dev");
}

The implementation details of  filterAsPerUserRelevance method as below,

public List<Item> filterAsPerUserRelevance(List<Item> productListFilteredAsperUserTaste, int id) {
List<Item> recommendedProducts = getListFromRecommendationEngine(id);

ArrayList<Item> list3 = new ArrayList<Item>();

for (Item productFromApi : productListFilteredAsperUserTaste) {
for (Item item : recommendedProducts) {
if (productFromApi.getBrand().toLowerCase().equals(item.getBrand().toLowerCase())) {
list3.add(productFromApi);
}
}
}
return list3;
}

private List<Item> getListFromRecommendationEngine(int id) {
HashMap<String, Object> map = new HashMap<String, Object>();
map.put("userId", id);

List<UserItemPurchaseHistory> list2 = ibatisTemplate.queryForList("GoSmarter.userItemList", map);

assertEquals(1, list2.size());
assertEquals(list2.get(0).getItemId(), 1);

map = new HashMap<String, Object>();
map.put("itemId", list2.get(0).getItemId());
Item item = (Item) ibatisTemplate.queryForObject("GoSmarter.itemDetails", map);
assertEquals(item.getName(), "FERRARI ENZO");

List<Item> list3 = new ArrayList<Item>();
list3.add(item);
return list3;
}

If you notice we get a input list of the items the user has to choose from. It retrieves the user item purchase history and compared the item brands and return the brands which the user has already purchased and asserts in the JUnit test.

In the test first development, we are testing first and than design the POJO and the associated DB schema and write the business logic and assert the output as per the Seed data.

Once the design is complete, we use a different profile to point to the actual database.

GATE, NLTK: Basic components of Machine Learning (ML) System

Machine Learning Components

I am currently building a Machine Learning system. In this blog I want to captures the elements of a machine learning system.

My definition of a Machine Learning System is to take voice or text inputs from a user and provide relevant information. And over a period of time, learn the user behavior and provides him better information.  Let us hold on to this comment and dissect apart each element.

In the below example, we will consider only text input. Let us also assume that the text input will be a freeflowing English text.

  • As a 1st step, when someone enters a freeflowing text, we need to understand what is the noun, what is the verb, what is the subject and what is the predicate. For doing this we need a Parts of Speech analyzer (POS), for example “I want a Phone”. One of the components of Natural Language Processing (NLP) is POS.
  • For associating relationship between a noun and a number, like “Phone greater than 20 dollers”, we need to run the sentence thru a rule engine. The terminology used for this is Semantic Rule Engine
  • The 3rd aspect is the Ontology, where in each noun needs to translate to a specific product or a place. For example, if someone says “I want a Bike” it should translate as “I want a Bicycle” and it should interpret that the company that manufacture a bicycle is BSA, or a Trac. We typically need to build a Product Ontology
  • Finally if you have buying pattern of a user and his friends in the system, we need a Recommendation Engine to give the user a proper recommendation

2 programming language which have good support of these capabilities are Python and Java. Python has a framework called Natural Language Tool Kit (NLTK). To understand more about NLTK, please refer the Cookbook. Java has a framework called GATE, Stanford NLP and OpenNLP .

For this discussion, we will consider GATE and related frameworks.

In my subsequent blogs I will talk more in details.

Measuring flow channel JUnit test coverage in Spring Integration Flow

Test Coverage

In any enterprise integration project there has to be a Business Workflow that needs to be developed and should have proper test coverage. In one  of our largest Spring Integration project, we had to measure how well the Business workflow has test coverage. Spring Integration (SI) is a lightweight framework that has good unit testing support.

Gary Russell one of the leading Spring Integration Architects helped us in designing this framework. The framework does the following,

  • The framework has a class, you need to inherit this class in your SI to get JUnit Test Coverage test with this class. It will wire tap all the channels and count the total channels tested in the SI flow and calculates how many channels the tests are covering and will derive the coverage ratio.
  • We also implemented a Maven plugin, where in we can set the threshold for the build to succeed based on the coverage ratio
  • This framework also consists of the sonar plugin for channel coverage  (contributed by Gary to Github) for Sonar software quality tool. Once you get the plugin, you need to build the plugin as “mvn package”. Take the jar file from the target and copy it into <Sonar Home>/extensions/plugins.
  • Start Sonar
  • In your SI component, if you do “mvn sonar:sonar” it will, push the quality metrics information into Sonar

The report in Sonar looks as below,

Spring Integration Channel Coverage

Spring Integration Channel Coverage

The codebase for the utility class AbstractChannelCoverageTests is checked in @ github . You can see the test class @ src/test/java/org/springframework/integration/test/coverage/ChannelCoverageTests.java. If you notice, in your SI testclass you need to extend this class as follows,

@ContextConfiguration
@RunWith(SpringJUnit4ClassRunner.class)
public class ChannelCoverageTests extends AbstractChannelCoverageTests {

@Test
void testChannel(){
//testing and asserting the flow
}

It will load a SI config file @ src/test/resources/org/springframework/integration/test/coverage/ChannelCoverageTests-context.xml and load all the channels and will start intercepting for any payload flow and keep count of it.

As per the above diagram, it will indicate how many channels are missed and it helps in making sure you cover those channels as well.