Apache HISE + Apache Camel

Check out this SlideShare Presentation:

Apache HISE + Apache Camel

View more presentations from Rafal Rusin.

jetty webapp osgi way

I will show how to expose simple Jetty OSGi service and use it to register hello world webapp. This app will serve static content from OSGi bundle and implement sample request handler under ServiceMix 4. Code is available here: http://github.com/rafalrusin/jetty-service
Alternative solutions are: using standard OSGi HTTP service http://www.osgi.org/javadoc/r4v42/org/osgi/service/http/HttpService.html to register servlet; and wrap existing WAR application using PAX WEB http://wiki.ops4j.org/display/paxweb/Pax+Web. I won't consider those two, since Jetty itself provides flexible way to handle webapps (including registering servlets). So those two are unnecessary overhead and are less flexible. Anyway those two are usually implemented on top of Jetty.
So the first thing to do is to create Jetty OSGi service. Basicly it will be a Spring Bean exposed to OSGi. Following snippet does the job:

<bean id="jetty-service" class="org.apache.jetty.service.JettyServiceImpl" init-method="init" destroy-method="destroy"/>
<osgi:service id="jetty-service-osgi" ref="jetty-service" interface="org.apache.jetty.service.api.JettyService" />

This will expose jetty-service to OSGi. All other components, which connect to it will wait automaticly until it's registered. Exposed interface has following methods:

public Handler registerApp(String name, Handler handler) throws Exception;
public void unregisterApp(Handler handler) throws Exception;

Those will be invoked to register Hello World application. JettyServiceImpl on the other hand, starts embedded Jetty Server and handles apps registration.

package org.apache.jetty.service;

import org.apache.jetty.service.api.JettyService;
import org.mortbay.jetty.Handler;
import org.mortbay.jetty.Server;
import org.mortbay.jetty.handler.ContextHandler;
import org.mortbay.jetty.handler.ContextHandlerCollection;

public class JettyServiceImpl implements JettyService {
 private Server server;
 private ContextHandlerCollection rootContext;

 public void init() throws Exception {
  server = new Server(8080);
  rootContext = new ContextHandlerCollection();
  server.setHandler(rootContext);
  server.start();
 }

 public void destroy() throws Exception {
  server.stop();
 }

 public Handler registerApp(String name, Handler handler) throws Exception {
  server.stop();
  ContextHandler h = rootContext.addContext("/" + name, name);
  h.setHandler(handler);
  server.start();
  return h;
 }

 public void unregisterApp(Handler handler) throws Exception {
  server.stop();
  rootContext.removeHandler(handler);
  server.start();
 }
}

Next step is to implement sample web app. First, we need to connect jetty-service bean to make it visible in our app.

  <osgi:reference id="jetty-service" interface="org.apache.jetty.service.api.JettyService" bean-name="jetty-service"/>

  <bean class="org.apache.jetty.service.example.HelloWorld" init-method="init" destroy-method="destroy">
    <property name="jettyService" ref="jetty-service"/>
  </bean>

Then, we need to implement sample app.

package org.apache.jetty.service.example;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.jetty.service.api.JettyService;
import org.apache.jetty.service.util.BundleResource;
import org.mortbay.jetty.Handler;
import org.mortbay.jetty.HttpConnection;
import org.mortbay.jetty.Request;
import org.mortbay.jetty.handler.AbstractHandler;
import org.mortbay.jetty.handler.ContextHandler;
import org.mortbay.jetty.handler.ContextHandlerCollection;
import org.mortbay.jetty.handler.ResourceHandler;

public class HelloWorld {

 private JettyService jettyService;
 private Handler registered;

 public void setJettyService(JettyService jettyService) {
  this.jettyService = jettyService;
 }

 public void init() throws Exception {
  ContextHandlerCollection handler = new ContextHandlerCollection();

  handler.addContext("/app", "app").setHandler(new AbstractHandler() {
   public void handle(String target, HttpServletRequest request,
     HttpServletResponse response, int arg3) throws IOException,
     ServletException {
          response.setContentType("text/html");
          response.setStatus(HttpServletResponse.SC_OK);
          response.getWriter().println("<h1>Hello World from Java</h1>" + request.getParameterMap());

          Request base_request = (request instanceof Request) ? (Request)request:HttpConnection.getCurrentConnection().getRequest();
          base_request.setHandled(true);
   }
  });

  ResourceHandler resourceHandler = new ResourceHandler();
  resourceHandler.setBaseResource(new BundleResource(getClass().getResource("/static")));
  ContextHandler contextHandler = handler.addContext("","");
  contextHandler.setHandler(resourceHandler);

  registered = jettyService.registerApp("helloWorld", handler);
 }

 public void destroy() throws Exception {
  jettyService.unregisterApp(registered);
 }
}

Here, we register sample request handler at 'app' sub path and serve static content from jar using BundleResource. Last thing is registering app using jettyService under 'helloWorld' context. So our application will be exposed under http://localhost:8080/helloWorld/ address.
ServiceMix has also so called features. This is the way to collect multiple dependencies under a single name. So we have to create features.xml file, like this:

<features>
    <feature name="jetty-service" version="${project.version}">
        <bundle>mvn:org.apache.jetty.service/service/${project.version}</bundle>
    </feature>

    <feature name="example-jetty-service-helloworld" version="${project.version}">
        <feature version="${project.version}">jetty-service</feature>
        <bundle>mvn:org.apache.jetty.service/example-helloworld/${project.version}</bundle>
    </feature>
</features>

Basicly, we can provide particular dependencies for our project.
Next, we do 'mvn install' on our project and run apache-servicemix-4.2.0-fuse-01-00/bin/servicemix karaf console. On the console, we need to type following commands:

features:addUrl mvn:org.apache.jetty.service/service-karaf/0.1.0-SNAPSHOT/xml/features
features:install example-jetty-service-helloworld
osgi:list

[ 230] [Active     ] [            ] [Started] [   60] Unnamed - org.apache.jetty.service:service:bundle:0.1.0-SNAPSHOT (0.1.0.SNAPSHOT)
[ 231] [Active     ] [            ] [Started] [   60] Unnamed - org.apache.jetty.service:example-helloworld:bundle:0.1.0-SNAPSHOT (0.1.0.SNAPSHOT)

Now, we can enter http://localhost:8080/helloWorld/ to test our app.
And that's it. OSGi and ServiceMix 4 features enable easy way to use dynamic modules in web apps. For example, it's very easy to build simple web framework with loadable components on page (something like mini implementation of Portlets).

Integration Tests for SMX4 with Python

Integration tests and unit tests are important for project quality. Unit tests usually are well suited for developer to verify his changes in runtime. On the other hand, integration tests, are for target user to verify that project's features in the way he interacts with project, work properly. In this article, I will show how to automate integration tests for ServiceMix 4 using SoapUI testrunner and a simple python script.
The idea is to spawn ServiceMix 4 Karaf console and interact with it using python expect library. During this interaction, SoapUI testrunner script is invoked in order to run SoapUI tests.
First, we need to grab SMX4_DIR and SOAPUI_DIR environment variables in our script, like this:

SMX4_DIR=os.getenv("SMX4_DIR")
SOAPUI_DIR=os.getenv("SOAPUI_DIR")

This way, we can invoke later our script using following shell command:

SMX4_DIR=/some/path SOAPUI_DIR=/some/other/path ./our-python-script

Then, we need to spawn ServiceMix 4 console by using python expect library:

import pexpect
import time
import sys
child = pexpect.spawn("bin/servicemix")
child.logfile = sys.stdout
child.expect("karaf.*>")
time.sleep(3)

Here, we set logfile to stdout in order to see our automated interaction with ServiceMix console. Then we need to wait for ServiceMix console command prompt, which would mean console is ready. Additionally, we need to wait a few seconds to avoid problems with running commands too early (which is a kind of small bug in ServiceMix). Then, we can install our features, which we want to test. This example starts Apache HISE test bundle, which loads also Apache HISE engine from dependencies.

child.sendline("features:addUrl mvn:org.apache.hise/hise-karaf/0.3.0-SNAPSHOT/xml/features");
child.expect("karaf.*>")
child.sendline("features:install hise-h2-test-example-osgi")
child.expect("karaf.*>")

Next, we need to wait until the feature is properly started. ServiceMix 4 OSGi container initializes bundles in background, so it's not enough to wait for command prompt to have it started (there doesn't seem to exist a "wait-until-started" console command). So we grep in a loop over installed bundles and see if status is started. In this example, we do 30 retries every second and fail our integration test script after this period, by raising exception.

child.sendline("features:addUrl mvn:org.apache.hise/hise-karaf/0.3.0-SNAPSHOT/xml/features");
rep=0
while True:
    child.sendline("osgi:list|grep -i hise-test-example-osgi")
    l=child.readline()
    l=child.readline()
    if re.match(".*Started", l) != None:
        break
    time.sleep(1)
    child.expect("karaf.*>")
    rep=rep+1
    if rep>30:
        raise Exception("Bundle not installed")

Next, we need to run SoapUI testrunner in order to execute test cases. We need to implement syscall method in order to fail integration tests if SoapUI testrunner completes with fault (non-zero exit code).

import os
def syscall(c):
    if os.system(c) != 0:
        raise Exception("Sys call failed: " + c)

syscall(SOAPUI_DIR + "/bin/testrunner.sh -f results hise-soapui-project.xml")

At the end, we can exit gracefully from ServiceMix console by using shutdown command, like this:

child.sendline("shutdown")

And that's it. Full code of integration test script is available in Apache HISE sources, from Apache repository http://svn.apache.org/repos/asf/incubator/hise/trunk/itest/itest.

Apache ODE 1.3.4 released

Apache ODE 1.3.4 has been released recently. You can grab it from here http://ode.apache.org/getting-ode.html. It has a few new features and a lot of bugfixes.

First of new features is Instance Replayer (http://ode.apache.org/instance-replayer). In short, it's able to restore business process instance by reexecuting every step again based on incoming and outgoing instance communication (it mocks up outgoing requests).
The basic use cases for replayer are:
1. migrate existing log running instances to newest process version given their communication (incoming and outgoing requests)
2. reproduce error scenarios between two instances of ODE (eg. production and development)
Instance Replayer extends management api by 2 operations: replay and getCommunication. In order to do 1, you can invoke following request:

      <pmap:replay>
        <replay>
           <ns:upgradeInstance>1234</ns:upgradeInstance>
        </replay>
     </pmap:replay>

You get a new instance in the newest process version. The old one will be deleted.
In order to do 2, you need to retrieve exchanges from existing instance (or instances) by following request:

     <pmap:getCommunication>
        <getCommunication>
           <ns:iid>1234</ns:iid>
        </getCommunication>
     </pmap:getCommunication>

Then, you can execute replay operation on another ODE installation in order to replicate instance:

     <pmap:replay xmlns:ns="http://www.apache.org/ode/pmapi/types/2006/08/02/">
       <replay>
           <ns:restoreInstance>
           <ns:processType
xmlns:p="http://sample.bpel.org/bpel/sample">p:OnEventCorrelation</ns:processType>
           ... exchanges
           </ns:restoreInstance>
       </replay>
     </pmap:replay>

In order to have time control in bpel process, there is introduced special variable $ode:currentEventDateTime. It's equivalent to $fn:current-dateTime() standard XPath function during live session (while no replaying takes place) and it's set to corresponding time in past during replaying.

The next new feature is SMX4 OSGi bundle distibution. Some detailed information on how to configure it is here: http://ode.apache.org/smx4-osgi-deployment.html. In short, it's possible to deploy ODE and processes as OSGi bundles in ServiceMix4. This simplifies project structure compared to ServiceAssembly and ServiceUnit double zipped packages.

Third feature is small, but may be useful. It makes possible to define Spring properties and use them inside business process. All you need to do is to put beans.xml file in process directory (example is here: http://svn.apache.org/repos/asf/ode/tags/APACHE_ODE_1.3.4/jbi/src/test/resources/SpringPropertiesJbiTest/). You need to define bean of name properties, like this:

<beans>
  <util:properties id="properties">
      <prop key="weirdProp">myWeirdPropValue2</prop>
  </util:properties>
</beans>

ODE will instantiate it and you can use it later, like this:

 <bpws:assign>
   <bpws:copy>
     <bpws:from>ode:process-property('weirdProp')</bpws:from>
     <bpws:to>$output.payload</bpws:to>
   </bpws:copy>
 </bpws:assign>

Git, Beginner to Advanced Survey

Check out this SlideShare Presentation:

Git, Beginner to Advanced Survey

View more presentations from Rafal Rusin.

xquery4j in action

In my previous article, I introduced a wrapper library for Saxon, xquery4j http://github.com/rafalrusin/xquery4j.
Here, I will explain how to use it to create an article generator in Java and XQuery for XHTML, called Article. You can download it here: http://github.com/rafalrusin/Article. It's a simple DSL for article generation.

I think it is something worth noticing, because the whole project took me just a while to implement and has interesting features. Those are:

• embedded code syntax highlighting for a lot of programming languages (using external program highlight),
• creating href entries for links, so you don't need to type URL twice
• it integrates natively with XHTML constructs

This is an example of an input it takes:

<a:article xmlns='http://www.w3.org/1999/xhtml' xmlns:a="urn:article">
<a:l>Some text</a:l>
<a:code lang="xml"><![CDATA[
<someXml/>
]]></a:code>
</a:article>

It generates XHTML output for it, using command

./run <input.xml >output.xhtml

The interesting thing is that XQuery expression for this transformation is very simple to do in Saxon. This is the complete code of it:

declare namespace a="urn:article";
declare default element namespace "http://www.w3.org/1999/xhtml";

declare function a:processLine($l) {
for $i in $l/node()
return
typeswitch ($i)
  case element(a:link, xs:untyped) return <a href="{$i/text()}">{$i/text()}</a>
  default return $i
};

declare function a:articleItem($i) {
typeswitch ($i)
 case element(a:l, xs:untyped) return (a:processLine($i), <br/>)

 case element(a:code, xs:untyped) return
  ( a:highlight($i/text(), $i/@lang)/body/* , <br/>)

 default return "error;"
};

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>a.xml</title>
<link rel="stylesheet" type="text/css" href="highlight.css"/>
</head>
<body>
{
for $i in a:article/*
return
a:articleItem($i)
}
</body>
</html>

Inside this expression, there is bound a:highlight Java function, which takes two strings on input (a code and a language) and returns DOM Node containing XHTML output from highlight command.
Since there is not much trouble with manipulating DOM using xquery4j, we can get as simple solution as this for a:highlight function:

    public static class Mod {
        public static Node highlight(final String code, String lang) throws Exception {
            Validate.notNull(lang);
            final Process p = new ProcessBuilder("highlight", "-X", "--syntax", lang).start();
            Thread t = new Thread(new Runnable() {

                public void run() {
                    try {
                        OutputStream out = p.getOutputStream();
                        IOUtils.write(code, out);
                        out.flush();
                        out.close();
                    } catch (IOException e) {
                        throw new RuntimeException(e);
                    }
                }
            });
            t.start();
            String result = IOUtils.toString(p.getInputStream());
            t.join();
            return DOMUtils.parse(result).getDocumentElement();
        }
    }

Please note that creating a separate thread for feeding input into highlight command is required, since Thread's output queue is limited and potentially might lead to dead lock. So we need to concurrently collect output from spawned Process.
However at the end, when we need to convert a String to DOM and we use xquery4j's DOMUtils.parse(result), so it's a very simple construct.

Embedding XQuery in Java

XQuery is a very powerful language. It can be very useful when you want to do some XML processing in Java.
Let's say you want to create an XML document based on some other XML data. Given something like this:

<employees>
  <employee>
    <name>Fred Jones</name>
    <address location="home">
      <street>900 Aurora Ave.</street>
      <city>Seattle</city>
      <state>WA</state>
      <zip>98115</zip>
    </address>
    <address location="work">
      <street>2011 152nd Avenue NE</street>
      <city>Redmond</city>
      <state>WA</state>
      <zip>98052</zip>
    </address>
    <phone location="work">(425)555-5665</phone>
    <phone location="home">(206)555-5555</phone>
    <phone location="mobile">(206)555-4321</phone>
  </employee>
 </employees>

You want to produce employees' names:

<names>
  <name>Fred Jones</name>
</names>

In XQuery it's just as easy as:

<names>
  {for $name in employees/employee/name/text() return <name>{$name}</name>}
</names>

The most interesting advantage for XQuery over various other methods for generating XML is that XQuery operates natively on XML.
There are tools like JaxB, XmlBeans, which enable strongly typed XML building directly from Java code. But using such approach often requires a lot of Java code to be written, which is not really necessary.
There is also a possibility to use XPath. However it's an inferior solution to XQuery, because XPath doesn't provide a way for building XML documents. It's designed only for nodes selection. On the other hand, XQuery extends XPath, so it supports every construct that XPath does.
Another way to do such processing in Java is to use XSLT. It's the closest approach to XQuery. But the problem with XSLT is that it has its language constructs, like 'for' expressed as XML elements. This makes writing XSLT code much more difficult that XQuery.
XQuery can be seen as a native template language for XML processing.
So the question is: how to evaluate XQuery expressions the best way from Java?
There are some Open Source implementations of XQuery for Java. One of them is inside XmlBeans. However in my opinion the best way is to use Saxon. It's the most mature project for XQuery processing and it's targetted directly for doing that.
However Saxon might be a bit difficult to use directly. At least digging a few interesting features from it took me some time.
So I decided to write a simple class for interfacing Saxon and to provide a few interesting examples of how to use it. That's how xquery4j was born. You can download it from github http://github.com/rafalrusin/xquery4j.
In xquery4j, you can execute XQuery expressions from Java in a simple way:

XQueryEvaluator evaluator = new XQueryEvaluator();
Long result = (Long) evaluator.evaluateExpression("5+5", null).get(0);
Assert.assertEquals(new Long(10), result);

It's possible to bind variables from Java objects, the easy way:

evaluator.bindVariable(QName.valueOf("myVar"), 123);

Sometimes it's also useful to declare Java methods and bind them for XQuery expressions. This is also very simple to do with xquery4j:

public static class MyFunctions {
        public static String myHello(String arg) {
            TestEvaluator te = (TestEvaluator) XQueryEvaluator.contextObjectTL.get();
            te.id++;
            return "hello(" + arg + te.id + ")";
        }
    }

    XQueryEvaluator evaluator = new XQueryEvaluator();
    evaluator.setContextObject(this);
    evaluator.declareJavaClass("http://my.org/employees", MyFunctions.class);
}

This code sets a context object to 'this' and binds all static methods from MyFunctions class to XQuery expressions. So during myHello execution from XQuery, you can easily operate on Java variables from bound context - 'id' in this case.
Here's a way of invoking such bound myHello method from XQuery:

declare namespace my = 'http://my.org/employees'; my:myHello("hello")

xquery4j code contains unit tests, which include examples above.
You can run them by:

mvn package

Feel free to give some feedback on using it.