Saturday, July 5, 2014

Implementing a Distributed Counter Service Using Hazelcast, Jersey 2 and Guice


Modern web services are often required to scale horizontally in order to handle growing load.
Here I present an example distributed Counter service, which uses modern technology stack consisting of Hazelcast, Jersey 2 and Guice.

The sample is based on an excellent example posted by Piersy and can be downloaded from here:
http://github.com/rafalrusin/jersey2-guice-example-with-test

The counter is just a shared value across all participating nodes within a cluster. All nodes can query for data and increase it's value by specified delta. Note that Hazelcast handles synchronization within distributed environment. In order to update the state atomically, we use Hazelcast EntryProcessor.

@Singleton
public class CounterService {

    public int increase(final int delta) {
        return (Integer) map.executeOnKey("counterKey", new CounterEntryProcessor(delta));
    }
}

public class CounterEntryProcessor implements EntryProcessor<String, Integer>, EntryBackupProcessor<String, Integer> {
    private final int delta;

    public CounterEntryProcessor(int delta) {
        this.delta = delta;
    }

    @Override
    public Integer process(Map.Entry<String, Integer> entry) {
        int newValue = entry.getValue() + delta;
        entry.setValue(newValue);
        return newValue;
    }
}

The service is just a Guice Singleton. It has an operation called 'increase', which takes a delta and creates EntryProcessor job for Hazelcast to submit to a node, which owns the value at the time and will update it atomically.
Hazelcast is a library, which implements Java Collections in distributed fashion. It handles replication, cluster membership and distributed locking.

Web service is a simple JAXRS REST Resource. Following code invokes Guice service layer from Resouce implementation:

@RequestScoped
@Path("counter")
public class CounterResource {

    private CounterService service;

    @Inject
    public CounterResource(CounterService service) {
        this.service = service;
    }

    @POST
    @Consumes(MediaType.TEXT_PLAIN)
    @Produces(MediaType.TEXT_PLAIN)
    public String increase(String delta) {
        return "" + service.increase(Integer.parseInt(delta));
    }
}

We also have a test case, which invokes the whole stack and does a request to the service. The whole stack is very lightweight. In runs on Jersey with Embedded Tomcat. The test case completes within a few seconds.

In order to run the example, you need to build distribution first (run dist.sh), then start nodes in separate shells and invoke curl POST requests to manipulate counter state. Here's a sample interaction:

./run.sh 8090
./run.sh 8092
...
Jul 05, 2014 10:20:06 AM com.hazelcast.cluster.ClusterService
INFO: [10.0.0.12]:5702 [dev] [3.2.3]

Members [2] {
        Member [10.0.0.12]:5701
        Member [10.0.0.12]:5702 this
}

Jul 05, 2014 10:20:08 AM com.hazelcast.core.LifecycleService
INFO: [10.0.0.12]:5702 [dev] [3.2.3] Address[10.0.0.12]:5702 is STARTED
...

$ curl -d '1' -H 'Content-Type: text/plain;' http://localhost:8092/webapp/api/counter
22
$ curl -d '1' -H 'Content-Type: text/plain;' http://localhost:8090/webapp/api/counter
23
$ curl -d '5' -H 'Content-Type: text/plain;' http://localhost:8090/webapp/api/counter
28

I think that Hazelcast is a very good step forward into distributed computing. For an every day programmer, it's a set of primitives to manipulate in order to implement a distributed system. It is very easy to integrate it into existing project, which could be either JavaEE or standalone app.

Saturday, June 14, 2014

Running hadoop 2.2.0 wordcount example under Windows


Recently I went to Hadoop Summit in San Jose (http://hadoopsummit.org/san-jose/). The conference was quite interesting (excluding a few boring talks). I found out that HortonWorks is trying to push hard Hadoop into enterprise environment with Hadoop 2.x and Yarn. I love this idea, since there seems to be no good standard for distributed containers in Java these days (forget about JEE clustering).

Surprisingly enough, it looks like Hadoop 2.2.0 is supported natively on Windows, which IMO is a great achievement and is a sign of platform getting more mature.
In this article I show how to run a simple WordCount example in Hadoop 2.2.0 under Windows.

First of all, you need to compile hadoop 2.2.0 distribution, which takes a lot of time (and sometimes tweaking pom files). I uploaded a precompiled version here. You need to edit windows environment variables and add path to bin dir and HADOOP_HOME variable pointing to the dir.

Then you need to format node and run example. Following commands do that:

E:\test>hdfs namenode -format

E:\hadoop-2.2.0\sbin>start-dfs

E:\hadoop-2.2.0\sbin>start-yarn
starting yarn daemons

E:\test>hdfs dfs -mkdir /input

E:\test>hdfs dfs -copyFromLocal file1.txt input

E:\test>hdfs dfs -cat /input/words.txt
...

E:\test>yarn jar E:\hadoop-2.2.0\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.2.0.jar wordcount /input/words.txt /output
14/06/14 14:29:47 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/06/14 14:29:47 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/06/14 14:29:48 INFO input.FileInputFormat: Total input paths to process : 1
...

E:\test>hdfs dfs -cat  /output/part-r-00000
abc1    1
abc2    3
abc3    1

I hope that Hadoop 2.x gains wide adoption in Enterprise Environment, since the industry needs the next gen standard for distributed apps.

Saturday, August 25, 2012

How to create a native Java App


Recently, I stumbled upon JCGO, an interesting project, which translates Java 1.4 code into C.
In this article, I show how to create a native Windows app out of a small Java app.

The Java app I will use is NetCat (https://github.com/rafalrusin/netcat). You can download precompiled executable, netcat.exe, from https://github.com/rafalrusin/netcat/downloads.

So the first step is to download all dependencies. I will use MinGW, MinGW GCC, jcgo-lib-1_14.tar.gz, jcgo-src-1_14.tar.bz2, classpath-0.93 (http://ftp.gnu.org/gnu/classpath/classpath-0.93.tar.gz) and Java sources for the app with dependent libraries: https://github.com/rafalrusin/netcat, commons cli 1.2 (http://commons.apache.org/cli/download_cli.cgi). You need to put all this in the same directory, so it'll have structure like this:

auxbin
classpath-0.93
commons-cli-1.2-src
dlls
goclsp
src
jcgo
jcgo.exe
jcgo.jar
libs
miscsrc
netcat
out
rflg_out
stdpaths.in

Then, you need to run Java to C translator by using command:

jcgo.exe -sourcepath netcat/src -sourcepath commons-cli-1.2-src/src/java netcat.NetCat @stdpaths.in -d out

Initializing...

Analysis pass...
Output pass...
Writing class tables...
Creating main file...
Parsed: 293 java files (2699 KiB). Analyzed: 3067 methods.
Produced: 640 c/h files (3769 KiB).
Contains: 1490 java methods, 4119 normal and 288 indirect calls.
Done conversion in 1 seconds. Total heap size: 36572 KiB.

Next step is to compile it into final executable. Following command does this:

gcc -DJCGO_INET -DJCGO_NOFP -DJCGO_WIN32 -DJCGO_THREADS -I src/include/ -I src/include/boehmgc/ -I src/native/ out/Main.c -o netcat.exe libs/x86/mingw/libgcmt.a -lws2_32

I used some switches, which are suitable for this particular app. For example, by default JCGO doesn't use multithreading or networking. This has to be enabled explicitly. 

And that's it. Now you can try out the app by calling google.com, like this:

$ netcat.exe google.com -p 80
Connecting to google.com port 80
GET
HTTP/1.0 302 Found
Location: http://www.google.pl/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=2f3085ac38771e98:FF=0:TM=1345885031:LM=1345885031:S=8A-IkreMgCogMsey; expires=Mon, 25-Aug-2014 08:57:11 GMT; path=/; domain=.google.com
Set-Cookie: NID=63=O_QZ4bDrzYNiiE0DY8RT-34c_pGt_OZagP3gzrzqCAx_Xo2kO7s9zVrUOx7FVz4TyAEY7Wx9UhglYZSX9UHSdzT7c9mUKzfkJFp5lk5FyfiMIcKITLhgSX4__3QwEYBS; expires=Sun
, 24-Feb-2013 08:57:11 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Sat, 25 Aug 2012 08:57:11 GMT
Server: gws
Content-Length: 218
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.pl/">here</A>.
</BODY></HTML>
^Z

I like the approach of translating Java code into C, because compared to other tools, which generate C++ code, this is more suitable for embedded devices. For example it is possible to generate code for iOS, because Objective C is a superset of C.

One feature I would like to see though is to be able to use reference counting instead of full gc. This is because one of the advantages of C over Java is that it doesn't have GC hangs. So then the programmer would have to make sure there's no cycles in orphaned object structure.

Update: Ivan Maidansky, an author of JCGO, has put some interesting comments regarding this article. In particular, he is aware of some apps in Apple Store, which do this kind of translation. Also, reference counting is discouraged due to multithreading issues. These comments can be found here: https://github.com/ivmai/JCGO/issues/2

Saturday, May 19, 2012

NetCat in Java


Recently, I tried to run netcat under Windows and I failed to compile it. So I decided to write a simple equivalent in Java, which works under all platforms.

NetCat is a very simple and useful tool, which allows to see contents of TCP/IP requests, like HTTP etc., both server and client side. With NetCat it is possible to either set up TCP/IP listener on a port and receive data or send TCP/IP request to remote server.

The code and executable for Java version of NetCat is available on GitHub:
http://github.com/rafalrusin/netcat/downloads

I used Jakarta Commons CLI to handle commandline parameters. The code is as simple as that:



In order to handle stream I/O I implemented a simple StreamTransferer class like this:



The only issue I had was that whenever I closed Java I/O stream using OutputStream.close(), it was closing the whole socket. So I couldn't receive any response back from server. So instead of doing that I had to use Socket shutdownOutput method, like in the code below.



It worked perfectly then.

So it is possible to send HTTP GET requests to google using this tool. In order to do that, you need to connect to google.com and type GET and enter. CTRL+Z and enter is for closing input stream under Windows. CTRL+D is for Linux. The example looks like below:


Saturday, December 17, 2011

Table viewer using JQuery plugin and JEE


DataTables is a JQuery plugin, facilitating building Ajax table editors. In this example, I show how to connect it to JEE backend, which is a simple Servlet. Backend exposes a table, stored as a Java List within Servlet instance. Data is served back in JSON format using Jackson library.

Example is deployed on Google App Engine, datatablesjee.appspot.com. Code is available on GitHub github.com/rafalrusin/datatablesjee.

First, we need to instantiate DataTables plugin within a html page. This is extremely easy, by using code below:

AjaxSource parameter refers to Servlet URI, which handles requests for data. ServerSide argument is set to true, which means that backend will do sorting, filtering and pagination. This allows us to use large tables (>1000 rows) without performance problems on client side.

Now, we need to implement backend. Servlet requires a doGet method, which needs to retrieve parameters sent from client as an Ajax request. Those parameters describe search keyword, starting row and page size of a table.

Then, we need to do filtering and sorting. I used a simple toString + contains methods on a single row in order to do filtering. Sorting is done via custom comparator, which sorts by given column number. Following code does the job:

Last, we need to send JSON response back to client. Here, we use Jackson, which is a very convenient library for manipulating JSON in Java.

iTotalRecords is total number of records, without filtering. iTotalDisplayRecords is number of records after applying filter. aaData is two dimensional array of strings, representing visible table data.

Summing up, I like the idea of Ajax in this form, because client side is not rendered directly by backend (no jsp, etc.). This makes it a detached view, which could be served as static content, from Apache Web Server for example, which is very performant.

Sunday, November 13, 2011

Running Qt4 Examples on Embedded Linux using ARM emulator


In this article I will show how to run Qt4-Embedded Examples on Angstrom Linux using QEMU. The procedure doesn't require any compilation or cross compilation. It uses Angstrom Linux precompiled packages, online image builder, and works both on Windows and Linux.
Qt4 Embedded allows to run Qt applications directly in Linux Framebuffer, bypassing X Windows completely. This is especially important during embedded development, because it allows to save a lot of memory and start up time.

Qt4 has a rich set of examples directly embedded into Qt sources. Below is a few samples of how it looks like:
I will show how to run them.

First, you need to install QEMU. For Windows, the easiest way is to download zipped executables, which I shared here: Qemu-windows-0151. For Linux it's usually apt-get install qemu-system.
Then, we need to build Angstrom image. For those unpatient, I shared a prebuilt image here: angstrom-qt4-embedded. Angstrom has online image builder available here: Angstrom Image Builder. You need to pick console image and download it. The small trick is that you need to download kernel image yourself (from here: kernel-image-2.6.37.2_2.6.37-r4.6_qemuarm.ipk) and unpack it using ar -x kernel-image.ipk command. This is because online image builder doesn't include kernel image for some reason. However this step is not required if you download the image I shared.

Next, you need to start QEMU using kernel image and prebuilt angstrom image. The command looks like this:
qemu-system-arm -M versatilepb -usb -usbdevice wacom-tablet -show-cursor -m 64 -kernel zImage-2.6.37.2 -hda disk.img -append "root=/dev/sda2 rw"
For convenience, I prepared run script, which does that.

Next, you need to login as root and install qt4-embedded using command: opkg install qt4-embedded. This can be again skipped if you use the image I prepared.

In order to run demos, you need to use this command:
qtdemoE -qws
It looks like this:
You can run the other examples from Qt, in standalone mode from /usr/bin/qtopia directory. You need to use similar command app -qws. The command is required to initialize Qt framebuffer. It is possible to run a few executables on the same display. In order to do this, you need to run the first one only with qws parameter. The other apps will connect to it.

Have fun!

Sunday, October 16, 2011

Red Black Tree Visualization using HTML5 Canvas and GWT


I created a sample app, which demonstrates HTML5 Canvas usage from Java through GWT. I think GWT is a great tool for migrating desktop apps into Web nowadays, especially for Java developers. So it's worth to give it a try.

Google has made great progress on migrating desktop apps into Web during last year. They propagated trend for HTML5 support and Javascript JIT compilers among modern browsers. So it's possible to run Quake 2 or CAD software directly in browser at decent speed.

Red Black Tree is a balanced BST tree. Details are described on Wikipedia.
You can run this sample app directly on appspot (it works on iPhone too :-) )
Sample code can be found here: Browse on GitHub.


Let's start from initialization.
First, we need to create Canvas element and add it to HTML. getContext2D is called to obtain drawing context.
We register a timer to redraw frames frequently:

So doUpdate is called every 50 ms and whenever redrawFrame is set to true, it redraws Canvas contents.
In autoplay mode, we call processFrame in while loop. So whenever redraw procedure takes too long, we will process frames without redrawing them. This won't slow down animation on low resources.

Then, we need to draw a tree. We use drawTree procedure, which is recursive and draws nodes along with contents and connections between them:

The best part is that we can do regular Java unit tests on Red Black tree to verify correctness of implementation: