tag:blogger.com,1999:blog-69831415637687326682024-03-18T12:18:51.980-07:00Rafal Rusin's blogThis blog covers topics around programming in various languages, for the industry. It includes my insights based on commercial experience from various projects I worked on. I hope it will provide a simple way of doing complex tasks related to programming, as well as an overview of what's going on in software nowadays.Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-6983141563768732668.post-36550229512684945712019-06-22T18:00:00.000-07:002019-06-22T18:00:26.156-07:00Analyzing MSFT Financial Data using AlphaVantage and Google ColabIn today's world, it's very easy to analyze data sets online, including financial data. This article shows how to compute standard statistical properties of a financial data set, like linear approximation and standard deviation. The example is using python based jupyter notebook, shared through Google Colab.<br />
<br />
Here's a <a href="https://colab.research.google.com/drive/1hefm_0WtYgarEgSAj1VYQpEor9y4ehLE">Jupyter Notebook</a>, which can be used for this exercise.<br />
<br />
First, we start by creating an API key in AlphaVantage.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtMxzcP_TQCxFdO3gKL8lyAuqEHW6SYUZLAMR4I2pKXb7rMTPWVUvGtakOvk-qs3bvmpZo4mTWhM6pMqaRQ0YMQgacloSVi7_0BNXNnu6zS3Kc3idojhAml6wDyUipf7NHXZwwTAnjNX3R/s1600/Screenshot+2019-06-22+at+5.46.45+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="626" data-original-width="1600" height="249" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtMxzcP_TQCxFdO3gKL8lyAuqEHW6SYUZLAMR4I2pKXb7rMTPWVUvGtakOvk-qs3bvmpZo4mTWhM6pMqaRQ0YMQgacloSVi7_0BNXNnu6zS3Kc3idojhAml6wDyUipf7NHXZwwTAnjNX3R/s640/Screenshot+2019-06-22+at+5.46.45+PM.png" width="640" /></a></div>
<br />
<br />
Next, we can display a downloaded financial dataset, which corresponds to MSFT historical data.<br />
<br />
After this step, we run statistical analysis of dataset using Python stats library.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">from scipy import stats</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">slope, intercept, r_value, p_value, std_err = stats.linregress(x, y)</span><br />
<div>
<br /></div>
<br />
Then, we can display the resulting data using Matplotlib. The chart will be displayed in the result window of Jupyter Notebook:<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKwQ3SPGeEZWUdhIO-wLJn50coe9ytfzzvZZbxQ5VaPlL0vdvG8A2yMHr1McQo80LRqic60Da632voYKgVVIE4OZC6BZ7l_UnL-lR4CQZUVCNBvwrNfLutcCPBiP_58C9IDYC_NlFWGTTB/s1600/Screenshot+2019-06-22+at+5.47.15+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="663" data-original-width="839" height="504" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKwQ3SPGeEZWUdhIO-wLJn50coe9ytfzzvZZbxQ5VaPlL0vdvG8A2yMHr1McQo80LRqic60Da632voYKgVVIE4OZC6BZ7l_UnL-lR4CQZUVCNBvwrNfLutcCPBiP_58C9IDYC_NlFWGTTB/s640/Screenshot+2019-06-22+at+5.47.15+PM.png" width="640" /></a></div>
Google Research Colab is a very popular and simple tool for running all sorts of data set analysis tasks. This also includes deep learning using tensorflow library. It is a step forward into simplifying data analysis in the research community.<br />
<br />
<br />Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com15tag:blogger.com,1999:blog-6983141563768732668.post-75784083960373836922017-04-16T15:14:00.001-07:002017-04-16T15:14:33.894-07:00Landing on The Moon using AI Bot <div class="separator" style="clear: both; text-align: left;">
Recently I was playing around with OpenAI Gym and Keras Reinforcement Learning library (keras-rl). I was able to train an AI Agent for a task of landing on the Moon. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
OpenAI Gym provides all sorts of different environments to explore using AI bots. One of them is lunar lander, which I'll focus on here. </div>
<div class="separator" style="clear: both; text-align: left;">
Keras on the other hand is a high level library built on top of TensorFlow (or Theano). It provides mechanisms for constructing deep learning models easily. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Creating a new environment using OpenAI Gym is as easy as this:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "courier new" , "courier" , monospace;">env = gym.make('LunarLander-v2')</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Here's how you can add video recorder of progress during training:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace;">env = gym.wrappers.Monitor(env, </span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace;"> 'recording', </span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace;"> resume=True, </span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace;"> video_callable=lambda count: count % record_video_every == 0</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: "courier new" , "courier" , monospace;"> )</span></div>
<div>
<br />
The model itself is quite simple DQN Agent with LinearAnnealedPolicy. The most important layer is Dense 512 neuron internal layer. It's responsible for understanding of the current situation during landing. Next small, dense layer on top of it is responsible for final decisions related to lunar lander actions (steering the engines).<br />
<br />
Here's how it can be instantiated:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">model = Sequential()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">model.add(Flatten(input_shape=(1,) + env.observation_space.shape))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">model.add(Dense(512))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">model.add(Activation('relu'))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">model.add(Dense(nb_actions))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">model.add(Activation('linear'))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">print(model.summary())</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">memory = SequentialMemory(limit=1000000, window_length=WINDOW_LENGTH)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1., value_min=.1, value_test=.05, </span><span style="font-family: "courier new" , "courier" , monospace;">nb_steps=1000000)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">dqn = DQNAgent(model=model, nb_actions=nb_actions, policy=policy, memory=memory, </span><span style="font-family: "courier new" , "courier" , monospace;">nb_steps_warmup=50000, gamma=.99, target_model_update=10000, </span><span style="font-family: "courier new" , "courier" , monospace;">train_interval=4, delta_clip=1.)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">dqn.compile(Adam(lr=.00025), metrics=['mae'])</span><br />
<div>
<br /></div>
<div>
<br /></div>
<div>
Here's a summary of the model:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Layer (type) Output Shape Param # Connected to 7 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">flatten_1 (Flatten) (None, 8) 0 flatten_input_1[0][0] </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">dense_1 (Dense) (None, 512) 4608 flatten_1[0][0] </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">activation_1 (Activation) (None, 512) 0 dense_1[0][0] </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">dense_2 (Dense) (None, 4) 2052 activation_1[0][0] </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">activation_2 (Activation) (None, 4) 0 dense_2[0][0] </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Total params: 6,660</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Trainable params: 6,660</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Non-trainable params: 0</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<br />
Training can take a lot of time. In this example I used 3.5 million steps.<br />
The outcome gives quite reasonable behavior for lunar lander.<br />
Note that depending on environment, learning can take less time (if environement is not very tricky). In this case I found that difficulties are related to sensitivity of the last phase of landing (touch down). It took some time for the model to figure this part out.<br />
<br />
As a side note, OpenAI provides lunar lander Agent using optimal trajectory. In this example, the Agent polls environment to figure out actions that give highest output of Q function. </div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOgaFneH3UpxXJc1EI_hFpvBKa5iSZJkOGb2Jl8BJWD3__MzjF33mVDypV83FxbfQ98YsixCwCdVBypiY8LBkifLcHV3Y35vmhweIaQCDrTMB87a-DmVCyB39XGPWIkrnZFKvEIs-itbJd/s1600/openaigym.video.0.11382.video000003.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOgaFneH3UpxXJc1EI_hFpvBKa5iSZJkOGb2Jl8BJWD3__MzjF33mVDypV83FxbfQ98YsixCwCdVBypiY8LBkifLcHV3Y35vmhweIaQCDrTMB87a-DmVCyB39XGPWIkrnZFKvEIs-itbJd/s400/openaigym.video.0.11382.video000003.gif" style="border: 1px solid black;" width="400" /></a></div>
<br />Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com38tag:blogger.com,1999:blog-6983141563768732668.post-11761722864157666622017-03-18T17:59:00.000-07:002017-03-18T17:59:52.867-07:00Classifying Dogs vs Cats on a Regular Laptop with 2GB GPU and 90% Accuracy<br />
Machine learning ecosystem has evolved a lot during recent years.<br />
I am amazed that I could run a very sophisticated experiment of classifying dogs vs cats with 90% accuracy on my regular laptop laptop.<br />
It has 2GB NVidia GPU card and 8GB RAM. <br />
Just in 2012 the state of art result of the dogs vs cats classification was 80%.<br />
<br />
I ran it based on an excellent course provided by fast.ai (<a href="http://course.fast.ai/">http://course.fast.ai/</a>).<br />
The competition is organized by Kaggle:<br />
<a href="https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition">https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition</a><br />
<br />
Here's an overview of the approach taken to achieve 90% accuracy.<br />
First, retrieve a publicly available model VGG16, which was prepared by scientists for image recognition competition (for ImageNet). Then remove last layer out of it and replace with Yes / No layer for recognizing cats vs dogs. The remaining layers were set as non trainable. Then run learning process for such model.<br />
<br />
The main libraries used here are Keras with Tensorflow backend.<br />
<br />
Full code is available on fast.ai website. Here in an overview of the most important parts.<br />
Training code:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">import tensorflow as tf</span><br />
<span style="font-family: Courier New, Courier, monospace;">from keras.backend.tensorflow_backend import set_session</span><br />
<span style="font-family: Courier New, Courier, monospace;">config = tf.ConfigProto()</span><br />
<span style="font-family: "Courier New", Courier, monospace;">config.allow_soft_placement=True</span><br />
<span style="font-family: Courier New, Courier, monospace;">config.log_device_placement=True</span><br />
<span style="font-family: Courier New, Courier, monospace;">set_session(tf.Session(config=config))</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"># Import our class, and instantiate</span><br />
<span style="font-family: Courier New, Courier, monospace;">import vgg16; reload(vgg16)</span><br />
<span style="font-family: Courier New, Courier, monospace;">from vgg16 import Vgg16</span><br />
<span style="font-family: Courier New, Courier, monospace;">vgg = Vgg16()</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">batch_size=16</span><br />
<span style="font-family: Courier New, Courier, monospace;">path = "data/dogscats/"</span><br />
<span style="font-family: Courier New, Courier, monospace;">#path = "data/dogscats/sample/"</span><br />
<span style="font-family: Courier New, Courier, monospace;">batches = vgg.get_batches(path+'train', batch_size=batch_size)</span><br />
<span style="font-family: Courier New, Courier, monospace;">val_batches = vgg.get_batches(path+'valid', batch_size=batch_size)</span><br />
<span style="font-family: Courier New, Courier, monospace;">vgg.finetune(batches)</span><br />
<span style="font-family: Courier New, Courier, monospace;">vgg.fit(batches, val_batches, nb_epoch=1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">vgg.model.save('vgg2.h5')</span><br />
<br />
The code uses vgg.finetune call to update the last layer of the model. Here's how it looks like:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">model = self.model</span><br />
<span style="font-family: Courier New, Courier, monospace;"> model.pop()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for layer in model.layers: layer.trainable=False</span><br />
<span style="font-family: Courier New, Courier, monospace;"> model.add(Dense(num, activation='softmax'))</span><br />
<div>
<br /></div>
<div>
Next, it trains model using vgg.fit call and saves result to vgg2.h5 file. </div>
<div>
<br /></div>
<div>
I had to put a few tweaks to the model related to device placement for Tensorflow so it could fit in GPU memory. The last few layers were placed on CPU. Here's the code:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> model = self.model = Sequential()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> model.add(Lambda(vgg_preprocess, input_shape=(3,224,224), output_shape=(3,224,224)))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> with tf.device('/gpu:0'):</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> self.ConvBlock(2, 64)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> self.ConvBlock(2, 128)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> self.ConvBlock(3, 256)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> self.ConvBlock(3, 512)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> self.ConvBlock(3, 512)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> with tf.device('/cpu:0'):</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> model.add(Flatten())</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> self.FCBlock()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> self.FCBlock()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> model.add(Dense(1000, activation='softmax'))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> fname = 'vgg16.h5'</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> model.load_weights(get_file(fname, self.FILE_PATH+fname, cache_subdir='models'))</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Here's the result of a learning process:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">23000/23000 [==============================] - 2103s - loss: 0.5482 - acc: 0.8676 - val_loss: 0.4194 - val_acc: 0.9060</span></div>
</div>
<div>
<br /></div>
<div>
The training process completed in 35 minutes with 90% accuracy on validation set. </div>
<div>
<br /></div>
<div>
I'm very positively surprised that such powerful machine learning tools are available these days and are runnable on regular computers. Moreover the approach presented by fast.ai is very interesting and resembles natural evolution of intelligence by adding new layers. </div>
<div>
<br /></div>
Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com14tag:blogger.com,1999:blog-6983141563768732668.post-48221365903536572902016-03-13T18:08:00.003-07:002019-10-24T03:49:58.139-07:00Learning Sine Function Using Neural Network<br />
Recently I stumbled upon an excellent demo of a 2 layer neural network written by Florian Muellerklein: <a href="https://github.com/FlorianMuellerklein/Machine-Learning">https://github.com/FlorianMuellerklein/Machine-Learning</a>.<br />
<br />
It is written in Python using numpy and focuses on digit recognition based on sklearn dataset.<br />
I decided to play around with it and add visualization for learning process of a sine function.<br />
I used matplotlib for creating an animation. <br />
<br />
The neural network implementation is typical. It uses standard gradient descent procedure with some optimizations like momentum and regularization (also random initialization). <br />
If you are interested in understanding how exactly gradient descent works, I highly recommend an article from Matt Mazur: <a href="http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example">http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example</a>.<br />
<br />
The network architecture I used was following:<br />
<br />
<ul>
<li>1 input neuron (x parameter of sine function)</li>
<li>60 hidden neurons</li>
<li>1 output value (the function result)</li>
</ul>
<br />
Here is a link to source code: <a href="https://github.com/rafalrusin/Machine-Learning/tree/master2">https://github.com/rafalrusin/Machine-Learning/tree/master2</a><br />
<br />
It converges pretty well. Here's an animation showing convergence of sine function during consecutive learning iterations (10 learing iterations per frame):<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/om_TibwdZvw" width="480"></iframe><br />
<br />
Machine Learning is a very fascinating domain that has been emerging rapidly over recent years, mainly in visual object recognition. I hope that it keeps this pace in future (or even exceeds it!).<br />
<br />Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com120tag:blogger.com,1999:blog-6983141563768732668.post-67536731264916076012015-05-16T11:10:00.000-07:002015-05-16T11:10:43.860-07:00Asymptotic Benchmark in JavaMost of the time when we analyze performance of different programs, we use <a href="http://en.wikipedia.org/wiki/Big_O_notation">Big O Notation</a> and run performance tests for a single input size N, which measures execution time.<br />
<br />
Both of these methods have disadvantages.<br />
Big O Notation is theoretical (has no code verification).<br />
Time based performance tests are not very reliable (it's hard to make assertions on them) and don't show the actual asymptotic function behind the programs.<br />
<br />
Here I present a slightly different approach, which is a combination of those 2 techniques.<br />
It uses explicit instruction counting, samples programs for different input sizes and tries to guess asymptotic function behind them.<br />
It also plots the result as a chart using HTML and <a href="https://developers.google.com/chart/interactive/docs/reference">Google Visualization JavaScript Library</a>.<br />
<br />
Note that it's more of a toy for visualization than something one could use in the industry for real world projects.<br />
<br />
First we start by implementing an example program (<a href="http://hibp.ecse.rpi.edu/~connor/education/IEE/lectures/Lecture_5_Binary_Counter.pdf">Binary Counter</a>), which we want to measure. It'll extend a base class Benchmark, which looks like this (full source code is available on github: <a href="https://github.com/rafalrusin/abench">https://github.com/rafalrusin/abench</a>):<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">public abstract class Benchmark {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> private int instructionCount = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> protected void incrementInstructionCount() {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> instructionCount++;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> public void resetInstructionCount() {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> instructionCount = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> public int getInstructionCount() {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return instructionCount;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> protected abstract Object run(int n);</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<div>
<br /></div>
<div>
It has abstact method "run", which takes input size as argument. This will be provided by the benchmark framework for different executions. </div>
<div>
Binary Counter implementation looks like this:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">public class BinaryCounter extends Benchmark {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> @Override</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> protected Object run(int n) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> int[] counter = new int[64];</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for (int i = 0; i < n; i++) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> int p = 0;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> while (counter[p] == 1) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> counter[p++] = 0;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> incrementInstructionCount();</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> counter[p] = 1;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> incrementInstructionCount();</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> return counter;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
</div>
<div>
<br /></div>
<div>
It is a 64 bit binary counter, which starts from 0 and increments values N times by one. Theoretically it should run in O(n) time. We'll verify that. </div>
<div>
<br /></div>
<div>
We need to run it through the framework using following code:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">public class Main {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> public static void main(String[] args) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> BenchmarkRunner benchmarkRunner = new BenchmarkRunner();</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> benchmarkRunner.addFormatter(new TextResultFormatter(new PrintWriter(System.out)));</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> benchmarkRunner.addFormatter(new HtmlResultFormatter(new File("out")));</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for (Benchmark benchmark : new Benchmark[]{</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> new BinaryCounter(),</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> ...</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> ) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> benchmarkRunner.run(benchmark, 1);</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
</div>
<div>
<br /></div>
<div>
Here's the chart generated for Binary Counter along with guessed function behind it:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgX79u_G9FnhN0CmosJFh5ly2SKIJ6jdVIaQV05aLtTnj6hWjSoTh_ciDnNjViDU4-DUJT9USpWLjwrf542CxfbMFw0ghDKGvL3fZJx4Lcci18VmyhVMUgP91_WivHqt7xunIBusBZbJNM7/s1600/b1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgX79u_G9FnhN0CmosJFh5ly2SKIJ6jdVIaQV05aLtTnj6hWjSoTh_ciDnNjViDU4-DUJT9USpWLjwrf542CxfbMFw0ghDKGvL3fZJx4Lcci18VmyhVMUgP91_WivHqt7xunIBusBZbJNM7/s400/b1.png" width="400" /></a></div>
<div>
<br /></div>
<div>
In this case the guessed function is linear, which is what we expected. For input size of 5.6 millions, the instruction count was 11 million.</div>
<div>
<br /></div>
<div>
The guessing process is an iteration over all samples and optimization of <a href="http://en.wikipedia.org/wiki/Mean_squared_error">Mean Squared Error</a>.</div>
<div>
It is implemented in BenchmarkRunner class. </div>
<div>
You can add more functions by extending Function interface. Here's an example of N log N function:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">public class NLogNFunction implements Function {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> @Override</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> public float eval(float x) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> return (float) (Math.log(x) * x);</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
</div>
<div>
<br /></div>
<div>
Here's some more benchmarks: MergeSort (expected asymptotic function is N log N) and NestedLoop (N^2). </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZGjako-58U8aTlU6qPoPezbpYkjYIV6vH61-V_OV4qiQogYnWnVskIjDE18stZ3XxVdLxGtqnffkdlSpq9LyUHXtcd-8I9_bSSapy2WRBGWJwFDzVVWm9ymlzmaPYhlN1Mh5sLqUo_biT/s1600/b2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZGjako-58U8aTlU6qPoPezbpYkjYIV6vH61-V_OV4qiQogYnWnVskIjDE18stZ3XxVdLxGtqnffkdlSpq9LyUHXtcd-8I9_bSSapy2WRBGWJwFDzVVWm9ymlzmaPYhlN1Mh5sLqUo_biT/s400/b2.png" width="400" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWbyCRWcGz7K9DGoTtjSXRRr0MwqhlNZQ5k3oxVl-WCXwNOAKFSYEwnUwJWRGtpmFipVmv2foqBMYcnJlugf461RHFAfcwqwcTB0ct_VSnU4M_z5gWM70-tVHRRxn47tAtgci8SmPPb5zv/s1600/b3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWbyCRWcGz7K9DGoTtjSXRRr0MwqhlNZQ5k3oxVl-WCXwNOAKFSYEwnUwJWRGtpmFipVmv2foqBMYcnJlugf461RHFAfcwqwcTB0ct_VSnU4M_z5gWM70-tVHRRxn47tAtgci8SmPPb5zv/s400/b3.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Implementing reliable performance tests is not a very well solved problem today. I think that the idea of instruction counting is something one can try to use in real world projects. It can be used for testing performance of isolated chunks of code, within unit tests, and can give predictable results. </div>
<div>
<br /></div>
Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com20tag:blogger.com,1999:blog-6983141563768732668.post-47616261248825574392014-11-23T19:52:00.000-08:002014-11-23T19:52:49.524-08:00Handling requests Asynchronously in Java using Jersey 2.13 and Glassfish 4.1<br />
In recent Jersey release 2.x, there is a new API for async request processing. It includes an excellent example server-async-standalone-webapp.<br />
In this article, I'll show how to run it and the results we can get by leveraging async processing.<br />
<br />
First, we need to download Glassfish 4.1 Web profile (<a href="https://glassfish.java.net/download.html">https://glassfish.java.net/download.html</a>).<br />
Then start it using command:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">./asadmin start-domain</span><br />
<br />
Next, we need to download Jersey examples bundle from <a href="https://jersey.java.net/download.html">https://jersey.java.net/download.html</a> and compile <span style="font-family: inherit;">server-async-standalone </span>example by running 'mvn package'.<br />
<br />Then we can deploy it using following command:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">./asadmin deploy <path>/jersey/examples/server-async-standalone/webapp/target/server-async-standalone-webapp.war</span><br />
<br />
Now we are ready to login to Glassfish Admin Console (localhost 4848) and see the status of deployed application:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXjJaSjHs_te1Nv5R_VOi6yl8h1Mmz84oDqvvCYmdpxZ-894ZiXQHSUbUHROmK2rEjObyiI9o8vzqSlmLPJqZtxZgE-kL8RoDMisDqkks1IifFKG9PWq4CQZwAg2a74qf82C1pQ-hoh3DY/s1600/async-deployed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXjJaSjHs_te1Nv5R_VOi6yl8h1Mmz84oDqvvCYmdpxZ-894ZiXQHSUbUHROmK2rEjObyiI9o8vzqSlmLPJqZtxZgE-kL8RoDMisDqkks1IifFKG9PWq4CQZwAg2a74qf82C1pQ-hoh3DY/s1600/async-deployed.png" height="179" width="320" /></a></div>
<br />
<br />
Now we're ready to run a client GUI application, which allows us to run tests against server. Go to '<span style="font-family: inherit;">server-async-standalone/client' and run 'mvn exec:java'. </span><br />
<span style="font-family: inherit;">The application looks like below. I ran sync vs async test on 100 requests and got response times improved from 20 secs to 1.2 secs. </span><br />
<span style="font-family: inherit;"><br /></span>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhikYbbmFFm9suCcrqu16OoriYmnoiz6UH8rY2-YVWU90-dW088R4mkWwkSSKbLU4WZitlG4wtvoOaLSkElVPejmOzpfgLRMlE1GznPoZhG8YLp0oHxhtFSOhhK4nCApWLKThLdP0ntE8WT/s1600/async-100-reqs.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhikYbbmFFm9suCcrqu16OoriYmnoiz6UH8rY2-YVWU90-dW088R4mkWwkSSKbLU4WZitlG4wtvoOaLSkElVPejmOzpfgLRMlE1GznPoZhG8YLp0oHxhtFSOhhK4nCApWLKThLdP0ntE8WT/s1600/async-100-reqs.png" height="320" width="247" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;">Sync</span></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHGqoEbSYtfoRToJFYIS9XjuIfLqRVofMmY8MkmzDfwArkVnlui719C-DIotRF2plDr6kTK2NHFP3JsEEsvihjASGnGLtvtVladrxNqDbPnrf_3IiDEQUITu9D5eA9THgqFBqAj0ijSQ31/s1600/sync-100-reqs.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHGqoEbSYtfoRToJFYIS9XjuIfLqRVofMmY8MkmzDfwArkVnlui719C-DIotRF2plDr6kTK2NHFP3JsEEsvihjASGnGLtvtVladrxNqDbPnrf_3IiDEQUITu9D5eA9THgqFBqAj0ijSQ31/s1600/sync-100-reqs.png" height="320" width="247" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Async</td></tr>
</tbody></table>
<span style="font-family: inherit;"><br /></span>
<br />
<span style="font-family: inherit;">The code is following for sync:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> @GET</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @Path("sync/{echo}")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public String syncEcho(@PathParam("echo") final String echo) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> try {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Thread.sleep(SLEEP_TIME_IN_MILLIS);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } catch (final InterruptedException ex) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> throw new ServiceUnavailableException();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return echo;</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<div>
<br /></div>
<div>
And for async:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">private static final ExecutorService TASK_EXECUTOR = Executors.newCachedThreadPool();</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">...</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> @GET</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> @Path("async/{echo}")</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> public void asyncEcho(@PathParam("echo") final String echo, @Suspended final AsyncResponse ar) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> TASK_EXECUTOR.submit(new Runnable() {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> @Override</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> public void run() {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> try {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Thread.sleep(SLEEP_TIME_IN_MILLIS);</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> } catch (final InterruptedException ex) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> ar.cancel();</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> ar.resume(echo);</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> });</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
</div>
<div>
<br /></div>
<span style="font-family: inherit;">Note that in Async example, the number of background threads is unbounded (TASK_EXECUTOR is a cached thread pool without limit)</span><span style="font-family: inherit;">. So in reality it won't improve the amount of resources (threads) consumed by server in Aync mode. </span><br />
<span style="font-family: inherit;">In Sync mode, the threads hold http executor. The default number of http worker threads in Glassfish is 5. This explains why we get response time around 20 secs for 100 requests. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">"http-listener-1(4)" daemon prio=6 tid=0x000000000cc7d000 nid=0xcf8 waiting on condition [0x00000000110ad000]</span><br />
<span style="font-family: Courier New, Courier, monospace;"> java.lang.Thread.State: TIMED_WAITING (sleeping)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>at java.lang.Thread.sleep(Native Method)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>at org.glassfish.jersey.examples.server.async.LongRunningEchoResource.syncEcho(LongRunningEchoResource.java:77)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>at sun.reflect.GeneratedMethodAccessor437.invoke(Unknown Source)</span><br />
<span style="font-family: Courier New, Courier, monospace;">...</span><br />
<br />
<span style="font-family: inherit;">Note that this is just an example, which illustrates how to enable Async processing. In real world scenarios there can be a third party service, which can be invoked using Jersey Client on the server side. If that service slows down, then the whole http thread pool on the server side can be exhausted and prevent other requests from getting processed. </span><br />
<span style="font-family: inherit;">The suggested solution in this case would be to switch to Async Jersey Client and Async service implementation. </span><br />
<span style="font-family: inherit;"><br /></span>
Async server processing can also help in case where there is a lot of slow clients. For example 20k mobile clients, who send payloads in chunks with 1 second delays. This scenario can easily bring down Sync server side implementations.<br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Async processing is gaining momentum on server side nowadays. There are multiple technologies, </span><br />
<span style="font-family: inherit;">which enable it. Among them are NodeJS, Netty, Akka and Play Framework. Jersey Async processing is one of them. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com1tag:blogger.com,1999:blog-6983141563768732668.post-46041632889860834782014-07-05T11:12:00.000-07:002014-07-05T11:12:58.702-07:00Implementing a Distributed Counter Service Using Hazelcast, Jersey 2 and GuiceModern web services are often required to scale horizontally in order to handle growing load.<br />
Here I present an example distributed Counter service, which uses modern technology stack consisting of Hazelcast, Jersey 2 and Guice.<br />
<br />
The sample is based on an excellent example posted by Piersy and can be downloaded from here:<br />
<a href="http://github.com/rafalrusin/jersey2-guice-example-with-test">http://github.com/rafalrusin/jersey2-guice-example-with-test</a><br />
<div>
<br /></div>
The counter is just a shared value across all participating nodes within a cluster. All nodes can query for data and increase it's value by specified delta. Note that Hazelcast handles synchronization within distributed environment. In order to update the state atomically, we use Hazelcast EntryProcessor.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">@Singleton</span><br />
<span style="font-family: Courier New, Courier, monospace;">public class CounterService {</span><br />
<div>
<br /></div>
<span style="font-family: Courier New, Courier, monospace;"> public int increase(final int delta) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return (Integer) map.executeOnKey("counterKey", new CounterEntryProcessor(delta));</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<div>
}</div>
<br />
<span style="font-family: Courier New, Courier, monospace;">public class CounterEntryProcessor implements EntryProcessor<String, Integer>, EntryBackupProcessor<String, Integer> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> private final int delta;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> public CounterEntryProcessor(int delta) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> this.delta = delta;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> @Override</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public Integer process(Map.Entry<String, Integer> entry) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int newValue = entry.getValue() + delta;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> entry.setValue(newValue);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return newValue;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<br />
The service is just a Guice Singleton. It has an operation called 'increase', which takes a delta and creates EntryProcessor job for Hazelcast to submit to a node, which owns the value at the time and will update it atomically.<br />
Hazelcast is a library, which implements Java Collections in distributed fashion. It handles replication, cluster membership and distributed locking.<br />
<br />
Web service is a simple JAXRS REST Resource. Following code invokes Guice service layer from Resouce implementation:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">@RequestScoped</span><br />
<span style="font-family: Courier New, Courier, monospace;">@Path("counter")</span><br />
<span style="font-family: Courier New, Courier, monospace;">public class CounterResource {</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> private CounterService service;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> @Inject</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public CounterResource(CounterService service) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> this.service = service;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: 'Courier New', Courier, monospace;"> @POST</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @Consumes(MediaType.TEXT_PLAIN)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @Produces(MediaType.TEXT_PLAIN)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public String increase(String delta) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return "" + service.increase(Integer.parseInt(delta));</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
We also have a test case, which invokes the whole stack and does a request to the service. The whole stack is very lightweight. In runs on Jersey with Embedded Tomcat. The test case completes within a few seconds.<br />
<br />
In order to run the example, you need to build distribution first (run dist.sh), then start nodes in separate shells and invoke curl POST requests to manipulate counter state. Here's a sample interaction:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">./run.sh 8090</span><br />
<span style="font-family: Courier New, Courier, monospace;">./run.sh 8092</span><br />
<span style="font-family: Courier New, Courier, monospace;">...</span><br />
<span style="font-family: Courier New, Courier, monospace;">Jul 05, 2014 10:20:06 AM com.hazelcast.cluster.ClusterService</span><br />
<span style="font-family: Courier New, Courier, monospace;">INFO: [10.0.0.12]:5702 [dev] [3.2.3]</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">Members [2] {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Member [10.0.0.12]:5701</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Member [10.0.0.12]:5702 this</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">Jul 05, 2014 10:20:08 AM com.hazelcast.core.LifecycleService</span><br />
<span style="font-family: Courier New, Courier, monospace;">INFO: [10.0.0.12]:5702 [dev] [3.2.3] Address[10.0.0.12]:5702 is STARTED</span><br />
<span style="font-family: Courier New, Courier, monospace;">...</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">$ curl -d '1' -H 'Content-Type: text/plain;' http://localhost:8092/webapp/api/counter</span><br />
<span style="font-family: Courier New, Courier, monospace;">22</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">$ curl -d '1' -H 'Content-Type: text/plain;' http://localhost:8090/webapp/api/counter</span><br />
<span style="font-family: Courier New, Courier, monospace;">23</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">$ curl -d '5' -H 'Content-Type: text/plain;' http://localhost:8090/webapp/api/counter</span><br />
<span style="font-family: Courier New, Courier, monospace;">28</span><br />
<br />
I think that Hazelcast is a very good step forward into distributed computing. For an every day programmer, it's a set of primitives to manipulate in order to implement a distributed system. It is very easy to integrate it into existing project, which could be either JavaEE or standalone app.<br />
<br />Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com113tag:blogger.com,1999:blog-6983141563768732668.post-91461671470744962152014-06-14T15:43:00.000-07:002014-06-14T15:43:27.801-07:00Running hadoop 2.2.0 wordcount example under WindowsRecently I went to Hadoop Summit in San Jose (<a href="http://hadoopsummit.org/san-jose/">http://hadoopsummit.org/san-jose/</a>). The conference was quite interesting (excluding a few boring talks). I found out that HortonWorks is trying to push hard Hadoop into enterprise environment with Hadoop 2.x and Yarn. I love this idea, since there seems to be no good standard for distributed containers in Java these days (forget about JEE clustering).<br />
<br />
Surprisingly enough, it looks like Hadoop 2.2.0 is supported natively on Windows, which IMO is a great achievement and is a sign of platform getting more mature.<br />
In this article I show how to run a simple WordCount example in Hadoop 2.2.0 under Windows.<br />
<br />
First of all, you need to compile hadoop 2.2.0 distribution, which takes a lot of time (and sometimes tweaking pom files). I uploaded a precompiled version <a href="https://github.com/rafalrusin/hadoop/blob/master/hadoop-2.2.0-windows.tar.gz">here</a>. You need to edit windows environment variables and add path to bin dir and HADOOP_HOME variable pointing to the dir.<br />
<br />
Then you need to format node and run example. Following commands do that:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">E:\test>hdfs namenode -format</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">E:\hadoop-2.2.0\sbin>start-dfs</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">E:\hadoop-2.2.0\sbin>start-yarn</span><br />
<span style="font-family: Courier New, Courier, monospace;">starting yarn daemons</span><br />
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">E:\test>hdfs dfs -mkdir /input</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<span style="font-family: Courier New, Courier, monospace;">E:\test>hdfs dfs -copyFromLocal file1.txt input</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">E:\test>hdfs dfs -cat /input/words.txt</span><br />
<span style="font-family: Courier New, Courier, monospace;">...</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">E:\test>yarn jar E:\hadoop-2.2.0\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.2.0.jar wordcount /input/words.txt /output</span><br />
<span style="font-family: Courier New, Courier, monospace;">14/06/14 14:29:47 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id</span><br />
<span style="font-family: Courier New, Courier, monospace;">14/06/14 14:29:47 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=</span><br />
<span style="font-family: Courier New, Courier, monospace;">14/06/14 14:29:48 INFO input.FileInputFormat: Total input paths to process : 1</span><br />
<span style="font-family: Courier New, Courier, monospace;">...</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">E:\test>hdfs dfs -cat /output/part-r-00000</span><br />
<span style="font-family: Courier New, Courier, monospace;">abc1 1</span><br />
<span style="font-family: Courier New, Courier, monospace;">abc2 3</span><br />
<span style="font-family: Courier New, Courier, monospace;">abc3 1</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
I hope that Hadoop 2.x gains wide adoption in Enterprise Environment, since the industry needs the next gen standard for distributed apps.Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com142tag:blogger.com,1999:blog-6983141563768732668.post-32515030834760056092012-08-25T02:49:00.000-07:002012-08-26T01:01:18.401-07:00How to create a native Java AppRecently, I stumbled upon <a href="https://github.com/ivmai/JCGO">JCGO</a>, an interesting project, which translates Java 1.4 code into C.<br />
In this article, I show how to create a native Windows app out of a small Java app.<br />
<br />
The Java app I will use is NetCat (<a href="https://github.com/rafalrusin/netcat">https://github.com/rafalrusin/netcat</a>). You can download precompiled executable, netcat.exe, from <a href="https://github.com/rafalrusin/netcat/downloads">https://github.com/rafalrusin/netcat/downloads</a>.<br />
<br />
So the first step is to download all dependencies. I will use MinGW, MinGW GCC, jcgo-lib-1_14.tar.gz, jcgo-src-1_14.tar.bz2, classpath-0.93 (http://ftp.gnu.org/gnu/classpath/classpath-0.93.tar.gz) and Java sources for the app with dependent libraries: <a href="https://github.com/rafalrusin/netcat">https://github.com/rafalrusin/netcat</a>, commons cli 1.2 (<a href="http://commons.apache.org/cli/download_cli.cgi">http://commons.apache.org/cli/download_cli.cgi</a>). You need to put all this in the same directory, so it'll have structure like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">auxbin</span><br />
<span style="font-family: Courier New, Courier, monospace;">classpath-0.93</span><br />
<span style="font-family: Courier New, Courier, monospace;">commons-cli-1.2-src</span><br />
<span style="font-family: Courier New, Courier, monospace;">dlls</span><br />
<span style="font-family: Courier New, Courier, monospace;">goclsp</span><br />
<span style="font-family: Courier New, Courier, monospace;">src</span><br />
<span style="font-family: Courier New, Courier, monospace;">jcgo</span><br />
<span style="font-family: Courier New, Courier, monospace;">jcgo.exe</span><br />
<span style="font-family: Courier New, Courier, monospace;">jcgo.jar</span><br />
<span style="font-family: Courier New, Courier, monospace;">libs</span><br />
<span style="font-family: Courier New, Courier, monospace;">miscsrc</span><br />
<span style="font-family: Courier New, Courier, monospace;">netcat</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">out</span><br />
<span style="font-family: Courier New, Courier, monospace;">rflg_out</span><br />
<span style="font-family: Courier New, Courier, monospace;">stdpaths.in</span><br />
<div>
<br /></div>
<div>
Then, you need to run Java to C translator by using command:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">jcgo.exe -sourcepath netcat/src -sourcepath commons-cli-1.2-src/src/java netcat.NetCat @stdpaths.in -d out</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"></span><br />
<div>
<span style="font-family: Courier New, Courier, monospace;">Initializing...</span></div>
<span style="font-family: Courier New, Courier, monospace;">
</span>
<br />
<div>
<span style="font-family: Courier New, Courier, monospace;">Analysis pass...</span></div>
<span style="font-family: Courier New, Courier, monospace;">
</span>
<div>
<span style="font-family: Courier New, Courier, monospace;">Output pass...</span></div>
<span style="font-family: Courier New, Courier, monospace;">
<div>
Writing class tables...</div>
<div>
Creating main file...</div>
<div>
Parsed: 293 java files (2699 KiB). Analyzed: 3067 methods.</div>
<div>
Produced: 640 c/h files (3769 KiB).</div>
<div>
Contains: 1490 java methods, 4119 normal and 288 indirect calls.</div>
<div>
Done conversion in 1 seconds. Total heap size: 36572 KiB.</div>
</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">Next step is to compile it into final executable. Following command does this:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">gcc -DJCGO_INET -DJCGO_NOFP -DJCGO_WIN32 -DJCGO_THREADS -I src/include/ -I src/include/boehmgc/ -I src/native/ out/Main.c -o netcat.exe libs/x86/mingw/libgcmt.a -lws2_32</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">I used some switches, which are suitable for this particular app. For example, by default JCGO doesn't use multithreading or networking. This has to be enabled explicitly. </span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">And that's it. Now you can try out the app by calling google.com, like this:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ netcat.exe google.com -p 80</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Connecting to google.com port 80</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">GET</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">HTTP/1.0 302 Found</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Location: http://www.google.pl/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Cache-Control: private</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Content-Type: text/html; charset=UTF-8</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Set-Cookie: PREF=ID=2f3085ac38771e98:FF=0:TM=1345885031:LM=1345885031:S=8A-IkreMgCogMsey; expires=Mon, 25-Aug-2014 08:57:11 GMT; path=/; domain=.google.com</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Set-Cookie: NID=63=O_QZ4bDrzYNiiE0DY8RT-34c_pGt_OZagP3gzrzqCAx_Xo2kO7s9zVrUOx7FVz4TyAEY7Wx9UhglYZSX9UHSdzT7c9mUKzfkJFp5lk5FyfiMIcKITLhgSX4__3QwEYBS; expires=Sun</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">, 24-Feb-2013 08:57:11 GMT; path=/; domain=.google.com; HttpOnly</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Date: Sat, 25 Aug 2012 08:57:11 GMT</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Server: gws</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Content-Length: 218</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">X-XSS-Protection: 1; mode=block</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">X-Frame-Options: SAMEORIGIN</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><TITLE>302 Moved</TITLE></HEAD><BODY></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><H1>302 Moved</H1></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">The document has moved</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><A HREF="http://www.google.pl/">here</A>.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"></BODY></HTML></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">^Z</span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
I like the approach of translating Java code into C, because compared to other tools, which generate C++ code, this is more suitable for embedded devices. For example it is possible to generate code for iOS, because Objective C is a superset of C.<br />
<br />
One feature I would like to see though is to be able to use reference counting instead of full gc. This is because one of the advantages of C over Java is that it doesn't have GC hangs. So then the programmer would have to make sure there's no cycles in orphaned object structure.<br />
<br />
Update: Ivan Maidansky, an author of JCGO, has put some interesting comments regarding this article. In particular, he is aware of some apps in Apple Store, which do this kind of translation. Also, reference counting is discouraged due to multithreading issues. These comments can be found here: <a href="https://github.com/ivmai/JCGO/issues/2">https://github.com/ivmai/JCGO/issues/2</a></div>
Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com6tag:blogger.com,1999:blog-6983141563768732668.post-38841789578679058132012-05-19T12:21:00.000-07:002012-05-19T12:30:40.342-07:00NetCat in JavaRecently, I tried to run netcat under Windows and I failed to compile it. So I decided to write a simple equivalent in Java, which works under all platforms. <br />
<br />
NetCat is a very simple and useful tool, which allows to see contents of TCP/IP requests, like HTTP etc., both server and client side. With NetCat it is possible to either set up TCP/IP listener on a port and receive data or send TCP/IP request to remote server.<br />
<br />
The code and executable for Java version of NetCat is available on GitHub: <br />
<a href="http://github.com/rafalrusin/netcat/downloads">http://github.com/rafalrusin/netcat/downloads</a><br />
<br />
I used Jakarta Commons CLI to handle commandline parameters. The code is as simple as that:<br />
<br />
<script src="https://gist.github.com/2732001.js?file=netcat_main.java"></script><br />
<br />
In order to handle stream I/O I implemented a simple StreamTransferer class like this:<br />
<br />
<script src="https://gist.github.com/2732001.js?file=StreamTransferer.java"></script><br />
<br />
The only issue I had was that whenever I closed Java I/O stream using OutputStream.close(), it was closing the whole socket. So I couldn't receive any response back from server. So instead of doing that I had to use Socket shutdownOutput method, like in the code below.<br />
<br />
<script src="https://gist.github.com/2732001.js?file=ShutdownOutput.java"></script><br />
<br />
It worked perfectly then. <br />
<br />
So it is possible to send HTTP GET requests to google using this tool. In order to do that, you need to connect to google.com and type GET and enter. CTRL+Z and enter is for closing input stream under Windows. CTRL+D is for Linux. The example looks like below:<br />
<br />
<script src="https://gist.github.com/2732001.js?file=connect_to_google"></script><br />Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com27tag:blogger.com,1999:blog-6983141563768732668.post-21391073907011484462011-12-17T15:51:00.000-08:002011-12-17T15:51:22.543-08:00Table viewer using JQuery plugin and JEE<a href="http://datatables.net">DataTables</a> is a <a href="http://jquery.com">JQuery</a> plugin, facilitating building Ajax table editors. In this example, I show how to connect it to JEE backend, which is a simple Servlet. Backend exposes a table, stored as a Java List within Servlet instance. Data is served back in JSON format using <a href="http://jackson.codehaus.org">Jackson library</a>.<br/>
<br/>
Example is deployed on Google App Engine, <a href="http://datatablesjee.appspot.com">datatablesjee.appspot.com</a>.
Code is available on GitHub <a href="http://github.com/rafalrusin/datatablesjee">github.com/rafalrusin/datatablesjee</a>. <br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsHuTVdvuWv6QHIgNMQK-_nHpWs5YyagmtveafEpoJlClHASFD7uyqac784SqaVsKOqsSd7g3es9_kIn3DLso3B-Xcb_tqHd8pmR19M1yvIAijkKanOBO81zFSR9bg5SHWf99KbrWQuEtI/s1600/datatablesjee.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="288" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsHuTVdvuWv6QHIgNMQK-_nHpWs5YyagmtveafEpoJlClHASFD7uyqac784SqaVsKOqsSd7g3es9_kIn3DLso3B-Xcb_tqHd8pmR19M1yvIAijkKanOBO81zFSR9bg5SHWf99KbrWQuEtI/s400/datatablesjee.png" /></a></div>
<br/>
First, we need to instantiate DataTables plugin within a html page.
This is extremely easy, by using code below:<br/>
<br/>
<script src="https://gist.github.com/1491712.js?file=instantiate.js"></script>
AjaxSource parameter refers to Servlet URI, which handles requests for data.
ServerSide argument is set to true, which means that backend will do sorting, filtering
and pagination. This allows us to use large tables (>1000 rows) without performance problems
on client side. <br/>
<br/>
Now, we need to implement backend.
Servlet requires a doGet method, which needs to retrieve parameters sent from client
as an Ajax request. Those parameters describe search keyword, starting row and page size
of a table. <br/>
<br/>
<script src="https://gist.github.com/1491712.js?file=init.java"></script>
Then, we need to do filtering and sorting. I used a simple toString + contains methods
on a single row in order to do filtering. Sorting is done via custom comparator, which
sorts by given column number. Following code does the job: <br/>
<br/>
<script src="https://gist.github.com/1491712.js?file=filtering.java"></script>
Last, we need to send JSON response back to client. Here, we use Jackson,
which is a very convenient library for manipulating JSON in Java. <br/>
<br/>
<script src="https://gist.github.com/1491712.js?file=result.java"></script>
iTotalRecords is total number of records, without filtering.
iTotalDisplayRecords is number of records after applying filter.
aaData is two dimensional array of strings, representing visible table data. <br/>
<br/>
Summing up, I like the idea of Ajax in this form, because client side is
not rendered directly by backend (no jsp, etc.).
This makes it a detached view, which could be served as static content, from Apache Web Server for example, which is very performant.Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com1tag:blogger.com,1999:blog-6983141563768732668.post-70012681977169396802011-11-13T03:01:00.000-08:002011-11-13T03:01:23.950-08:00Running Qt4 Examples on Embedded Linux using ARM emulatorIn this article I will show how to run Qt4-Embedded Examples on Angstrom Linux using QEMU. The procedure doesn't require any compilation or cross compilation. It uses Angstrom Linux precompiled packages, online image builder, and works both on Windows and Linux. <br/>
Qt4 Embedded allows to run Qt applications directly in Linux Framebuffer, bypassing X Windows completely. This is especially important during embedded development, because it allows to save a lot of memory and start up time. <br/>
<br/>
Qt4 has a rich set of examples directly embedded into Qt sources. Below is a few samples of how it looks like: <br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ_YmEC7Kmwv06KPOvv4Tcabd6b2c1M97a72cP-4SgXflRGWCmR18TT4pbEYQORhshg_114plaZMWmHWFdxw6-KavRTQZHAvhu6Izp6mYqAEDz0PYOw37WnIbtKB-Uaz2NQConvkNMwlVm/s1600/path-stroking.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="315" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ_YmEC7Kmwv06KPOvv4Tcabd6b2c1M97a72cP-4SgXflRGWCmR18TT4pbEYQORhshg_114plaZMWmHWFdxw6-KavRTQZHAvhu6Izp6mYqAEDz0PYOw37WnIbtKB-Uaz2NQConvkNMwlVm/s400/path-stroking.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhr3gnm0weWkcl-OIrcnQZQ3xBYyvicKxv1mk1mJ4Kx9hqeK6zlXBFCNYP-jMFjVORbI2-KeVzGll4E8umQQ_lBiA40C1sIztpuwNWnm4mGet_xCEBThTGurPO_6ZpezGwxyaIO4QijxEGp/s1600/tetrix.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="315" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhr3gnm0weWkcl-OIrcnQZQ3xBYyvicKxv1mk1mJ4Kx9hqeK6zlXBFCNYP-jMFjVORbI2-KeVzGll4E8umQQ_lBiA40C1sIztpuwNWnm4mGet_xCEBThTGurPO_6ZpezGwxyaIO4QijxEGp/s400/tetrix.png" /></a></div>
I will show how to run them.<br/>
<br/>
First, you need to install QEMU. For Windows, the easiest way is to download zipped executables,
which I shared here: <a href="http://www.4shared.com/folder/hjxINbny/embedded.html">Qemu-windows-0151</a>. For Linux it's usually <b>apt-get install qemu-system</b>. <br/>
Then, we need to build Angstrom image. For those unpatient, I shared a prebuilt image here: <a href="http://www.4shared.com/folder/hjxINbny/embedded.html">angstrom-qt4-embedded</a>.
Angstrom has online image builder available here: <a href="http://narcissus.angstrom-distribution.org">Angstrom Image Builder</a>. You need to pick console image and download it. The small trick is that you need to download kernel image yourself (from here: <a href="http://feeds.angstrom-distribution.org/feeds/unstable/ipk/glibc/armv5te/machine/qemuarm/kernel-image-2.6.37.2_2.6.37-r4.6_qemuarm.ipk">kernel-image-2.6.37.2_2.6.37-r4.6_qemuarm.ipk</a>) and unpack it using <b>ar -x kernel-image.ipk</b> command. This is because online image builder doesn't include kernel image for some reason. However this step is not required if you download the image I shared. <br/>
<br/>
Next, you need to start QEMU using kernel image and prebuilt angstrom image. The command looks like this:<br/>
<b>
qemu-system-arm -M versatilepb -usb -usbdevice wacom-tablet -show-cursor -m 64 -kernel zImage-2.6.37.2 -hda disk.img -append "root=/dev/sda2 rw"</b><br/>
For convenience, I prepared <b>run</b> script, which does that. <br/>
<br/>
Next, you need to login as root and install qt4-embedded using command:
<b>opkg install qt4-embedded</b>. This can be again skipped if you use the image I prepared.
</br>
<br/>
In order to run demos, you need to use this command:<br/>
<b>qtdemoE -qws</b><br/>
It looks like this:
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7WVXVIIvKvFWsI7SxkLnIR_TV_rRbW_T5mYg0xsOqO-0gPjyvMeW9iZl3or8yabqOH3nXrhSEdn_oXpwipitFC20Hn21cdLYJYuB4C3kQz0oqefDwRtN9r88VC_qLFE1pg3EiU1KLqQak/s1600/angstrom-console.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="472" width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7WVXVIIvKvFWsI7SxkLnIR_TV_rRbW_T5mYg0xsOqO-0gPjyvMeW9iZl3or8yabqOH3nXrhSEdn_oXpwipitFC20Hn21cdLYJYuB4C3kQz0oqefDwRtN9r88VC_qLFE1pg3EiU1KLqQak/s400/angstrom-console.png" /></a></div>
You can run the other examples from Qt, in standalone mode from <b>/usr/bin/qtopia</b> directory. You need to use similar command <b>app -qws</b>.
The command is required to initialize Qt framebuffer.
It is possible to run a few executables on the same display. In order to do this, you need to run the first one only with <b>qws</b> parameter. The other apps will connect to it.<br/>
<br/>
Have fun!<br/>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com4tag:blogger.com,1999:blog-6983141563768732668.post-62132269937973262252011-10-16T05:50:00.000-07:002011-10-16T05:56:44.486-07:00Red Black Tree Visualization using HTML5 Canvas and GWTI created a sample app, which demonstrates HTML5 Canvas usage from Java through GWT.
I think GWT is a great tool for migrating desktop apps into Web nowadays, especially
for Java developers. So it's worth to give it a try. <br/>
<br/>
Google has made great progress on <i>migrating desktop apps into Web</i> during last year.
They propagated trend for HTML5 support and Javascript JIT compilers among modern browsers.
So it's possible to run Quake 2 or CAD software directly in browser at decent speed. <br/>
<br/>
<a href="http://en.wikipedia.org/wiki/Red%E2%80%93black_tree">Red Black Tree</a> is a balanced BST tree. Details are described on Wikipedia. <br/>
You can <a href="http://rbvis123.appspot.com">run this sample app directly on appspot</a> (it works on iPhone too :-) )<br/>
Sample code can be found here:
<a href="http://github.com/rafalrusin/rbvis">Browse on GitHub</a>.<br/>
<br/>
<img src="http://sites.google.com/site/rrusin999/syntax/rbvis.png" width="600" height="224"/>
<br/>
Let's start from initialization.<br/>
First, we need to create Canvas element and add it to HTML.
getContext2D is called to obtain drawing context.<br/>
We register a timer to redraw frames frequently: <br/>
<script src="https://gist.github.com/1290819.js?file=init.java"></script><br/>
So doUpdate is called every 50 ms and whenever redrawFrame is set to true, it redraws
Canvas contents. <br/>
In autoplay mode, we call processFrame in while loop. So whenever redraw procedure takes too long, we will process frames without redrawing them. This won't slow down animation on low resources. <br/>
<script src="https://gist.github.com/1290819.js?file=processFrame.java"></script><br/>
Then, we need to draw a tree.
We use drawTree procedure, which is recursive and draws nodes along with contents and connections between them: <br/>
<script src="https://gist.github.com/1290819.js?file=drawTree.java"></script><br/>
The best part is that we can do regular Java unit tests on Red Black tree to verify
correctness of implementation:<br/>
<script src="https://gist.github.com/1290819.js?file=test.java"></script><br/>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com6tag:blogger.com,1999:blog-6983141563768732668.post-74770849794094001982011-10-03T12:55:00.000-07:002011-10-03T13:06:09.503-07:00Grammar parser in C++Recently I stumbled upon implementing a simple parser in C++. The task is very classic, however I couldn't find any good resources on the web to help me out. <br/>
I tried different tools (including ANTLR), but finally the easiest way I found was bison + flex.
It's unbelievable that this technology from 1989 is still actively developed.
Latest stable release is from May 14, 2011.
Moreover, many important projects make use of it. Among them are Ruby, PHP, Google Go, Bash shell.<br/>
So I decided to create a minimalistic example, which works from scratch.<br/>
I published the code on github <a href="http://github.com/rafalrusin/calculator">Calculator</a>, so you can check it out.<br/>
The whole example is 88 lines long and evaluates common expressions, like 2+2*2-13*(7+19/2).<br/>
Let's start with lexer. In flex, you need to define regular expressions, which produce tokens. Such tokens are later processed by a scanner. So we have to define calculator.lex, like this: <br/>
<script src="https://gist.github.com/1259974.js?file=calculator.lex"></script><br/>
Flex will generate yylex() function, which we can call later to produce tokens. <br/>
Next, we need to create a scanner (calculator.y), which specifies a grammar. It's simple like that:<br/>
<script src="https://gist.github.com/1259974.js?file=calculator.y"></script><br/>
Here, we specify types for all tokens using C/C++ union like structure.
Variable $$ is used to store result of particular reductions. <br/>
Additionally, we need to specify %left precedence for +, -, *, / operators to resolve shift / reduce conflicts between them. <br/>
And that's basically it. We have a working expression parser. <br/>
I implemented it in a way, that executable takes a file name containing expressions as an argument.<br/>
So you can try ./calculator input.txt to see the result.Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com1tag:blogger.com,1999:blog-6983141563768732668.post-11471890986934727362011-06-12T15:20:00.000-07:002011-06-12T15:20:38.582-07:00Visualizing GIS data in JavaFX 2.0 beta using GeoToolsGeographic data mostly comprises of polygon coordinates sets along with attributes, like country or city name, etc. This is quite easy to visualize in JavaFX, which supports rendering for SVG paths. <br/>
In the article, I show how to read such GIS data from ESRI type database files using open source
library <a href="http://geotools.org">GeoTools</a>. <br/>
The data itself comes for free from <a href="http://www.naturalearthdata.com">www.naturalearthdata.com</a>.<br/>
Sample code can be found here:
<a href="http://github.com/rafalrusin/geotools-fx-test">Browse on GitHub</a>.<br/>
<br/>
<img src="http://sites.google.com/site/rrusin999/syntax/geotools-fx-test.png" width="569" height="442"/>
<br/>
<br/>
GIS data usually comes in form of SHP and DBF files. In order to read it, we use GeoTools
parser. Following code iterates over so called "features" from within data files
and retrieves name attribute and shape geometry. <br/><br/>
<script src="https://gist.github.com/1022034.js?file=iterating.java"></script><br/>
Next, we need to create JavaFX polygons for each feature from iteration.
Small note here. Each feature may comprise of multiple polygons. For example "United States"
shape may contain separate polygon for Alaska. So we need additional loop to generate
such polygons.<br/>
In order to create a polygon in JavaFX, we use Path class along with MoveTo and LineTo path elements. Following snippet does the job. <br/><br/>
<script src="https://gist.github.com/1022034.js?file=polygons.java"></script><br/>
The remaining part is to implement zoom and panning functionality. This is fairly easy
in JavaFX. We can use translate and scale properties from main Group shape.
Panning functionality is handled using following snippet:<br/><br/>
<script src="https://gist.github.com/1022034.js?file=panning.java"></script><br/>
Zoom is coded this way:<br/><br/>
<script src="https://gist.github.com/1022034.js?file=zoom.java"></script><br/>
That's it. Now we have basic GIS data viewer in JavaFX 2.Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com5tag:blogger.com,1999:blog-6983141563768732668.post-43368818010828931682011-03-22T16:03:00.000-07:002011-03-22T16:03:36.739-07:00Implementing graph editor in JavaFX using MVC like approachJavaFX has very good SVG support, embedded into language runtime. This makes it interesting choice for implementing custom UI components, which includes graph editors. <br/>
In this article, I show how to create a simple graph editor, using MVC like approach and XML serialization via XStream. <br/>
<br/>
Example can be found here:
<a href="http://sites.google.com/site/rrusin999/syntax/netflow.jnlp">Launch JNLP</a>,
<a href="http://github.com/rafalrusin/javafx">Browse on GitHub</a>.<br/>
Application brings graph editing functionality and maximum network flow computation using <a href="http://en.wikipedia.org/wiki/Ford%E2%80%93Fulkerson_algorithm">Ford Fulkerson algorithm</a> <br/>
<br/>
<img src="http://sites.google.com/site/rrusin999/syntax/netflow.png" width="569" height="442"/>
<br/>
<br/>
Architecture is divided into Model, View and Controller. I made a slight upgrade to classic understanding of MVC. Classic View was responsible for displaying data only. Here, View consists also of UI parts, which include editable labels or combo boxes.
Also, in Classic MVC, Model was responsible for refreshing View after Model changes by sending events to registered views. Here, Controller is responsible for refreshing View after Model changes and Model is just a plain POJO structure. I found such approach easier to implement. <br/>
<br/>
<b>Model</b> consists of Three Java classes (non JavaFX), which represent structure of a graph: MNode, MShape, MConnection. Separating those classes from UI gives benefit of easier serialization. In this case using XStream::toXML(model) does the job. Sample output is like this: <br/><br/>
<script src="https://gist.github.com/882258.js?file=model.xml"></script>
<br/>
<b>View</b> consists of corresponding UI implementations for Model elements, which are UINode, UILine, UIShape. Here, UINode is connected to MNode through model property. This is <a href="http://en.wikipedia.org/wiki/Bridge_pattern">Bridge Pattern</a> like approach for splitting class hierarchy of nodes into two. <br/>
UINode classes refer to controller to perform user input actions, like delete node. <br/>
<br/>
<b>Controller</b> implements user action logic. This includes add, delete, and drag node. It is also responsible for refreshing UI after model changes. In order to do that easily, it uses Weak Hash Map, which keys are Model nodes and values are UI Nodes. Update function is like this:<br/><br/>
<script src="https://gist.github.com/882258.js?file=netflow.fx"></script>
Because of Weak Hash Map, removing nodes from Model leads to removing corresponding UI elements from map automaticly. <br/>
<br/>
JavaFX SVG support and Layouts make it easy to render quite good looking nodes and connections. <br/>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com12tag:blogger.com,1999:blog-6983141563768732668.post-63917231764611259732011-03-07T13:42:00.000-08:002011-03-07T13:56:55.046-08:00Drawing arrows in JavaFXSome time in the past, I was wondering what's the easiest solution for drawing arrow
connections between shapes. The problem boils down to computing boundary point for given shape,
which intersects with connecting line. <br/>
The solution is not so difficult when we consider polygon shapes. But it becomes more difficult considering curves and far more difficult for generic SVG shapes. <br/>
In this article, I show a simple way to find such boundary points for generic SVG shapes, utilizing JavaFX "contains" method and simple bisection algorithm. <br/>
Following application does the job:
<a href="http://sites.google.com/site/rrusin999/syntax/arrows.jnlp">Launch JNLP</a>,
<a href="http://github.com/rafalrusin/javafx">Browse on GitHub</a>.<br/>
Application allows to drag shapes to track shape boundaries.<br/>
<img src="http://sites.google.com/site/rrusin999/syntax/arrows.png" width="410" height="314"/>
<br/>
So, "contains" method in JavaFX allows us to poll any 2D point for intersection with SVG shape.
Next, we need to assume, that we know a point inside a shape, from which every ray has exactly one intersection. This includes star-like shapes and ellipses. Next, we can use simple bisection algorithm to find such boundary point. We cut off once length between two points is too small. Following snippet does the job:
<script src="http://gist.github.com/859239.js?file=bisect.fx">
</script>
Next, we need to draw arrows on the ends. In order to do that, we can use JavaFX Affine Transform, which is matrix transformation for SVG path. We need to compute transformation vectors for the matrix. This sounds scarry, but in fact is relatively easy. First vector in matrix is (targetPoint - sourcePoint) / length. Second one is perpendicular to it.
Following code applies transformation:
<script src="http://gist.github.com/859239.js?file=affine">
</script>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com3tag:blogger.com,1999:blog-6983141563768732668.post-4350423122197691792010-10-23T15:10:00.001-07:002010-10-23T15:10:29.228-07:00Apache HISE + Apache CamelCheck out this SlideShare Presentation: <div style="width:425px" id="__ss_5540260"><strong style="display:block;margin:12px 0 4px"><a href="http://www.slideshare.net/rrusin/apache-hise-apache-camel" title="Apache HISE + Apache Camel">Apache HISE + Apache Camel</a></strong><object id="__sse5540260" width="425" height="355"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=hise-101023164306-phpapp02&stripped_title=apache-hise-apache-camel&userName=rrusin" /><param name="allowFullScreen" value="true"/><param name="allowScriptAccess" value="always"/><embed name="__sse5540260" src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=hise-101023164306-phpapp02&stripped_title=apache-hise-apache-camel&userName=rrusin" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><div style="padding:5px 0 12px">View more <a href="http://www.slideshare.net/">presentations</a> from <a href="http://www.slideshare.net/rrusin">Rafal Rusin</a>.</div></div>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com3tag:blogger.com,1999:blog-6983141563768732668.post-32141209695795148372010-07-18T16:06:00.000-07:002010-07-18T16:06:26.001-07:00jetty webapp osgi wayI will show how to expose simple Jetty OSGi service and use it to register hello world webapp.
This app will serve static content from OSGi bundle and implement sample request handler under ServiceMix 4.
Code is available here: <a href="http://github.com/rafalrusin/jetty-service">http://github.com/rafalrusin/jetty-service</a>
<br/>
Alternative solutions are: using standard OSGi HTTP service <a href="http://www.osgi.org/javadoc/r4v42/org/osgi/service/http/HttpService.html">http://www.osgi.org/javadoc/r4v42/org/osgi/service/http/HttpService.html</a>
to register servlet; and wrap existing WAR application using PAX WEB <a href="http://wiki.ops4j.org/display/paxweb/Pax+Web">http://wiki.ops4j.org/display/paxweb/Pax+Web</a>.
I won't consider those two, since Jetty itself provides flexible way to handle webapps (including registering servlets). So those two are unnecessary overhead and are less flexible. Anyway those two are usually implemented on top of Jetty.
<br/>
So the first thing to do is to create Jetty OSGi service. Basicly it will be a Spring Bean exposed to OSGi.
Following snippet does the job:
<pre class="hl">
<span class="hl kwa"><bean</span> <span class="hl kwb">id</span>=<span class="hl str">"jetty-service"</span> <span class="hl kwb">class</span>=<span class="hl str">"org.apache.jetty.service.JettyServiceImpl"</span> <span class="hl kwb">init-method</span>=<span class="hl str">"init"</span> <span class="hl kwb">destroy-method</span>=<span class="hl str">"destroy"</span><span class="hl kwa">/></span>
<span class="hl kwa"><osgi:service</span> <span class="hl kwb">id</span>=<span class="hl str">"jetty-service-osgi"</span> <span class="hl kwb">ref</span>=<span class="hl str">"jetty-service"</span> <span class="hl kwb">interface</span>=<span class="hl str">"org.apache.jetty.service.api.JettyService"</span> <span class="hl kwa">/></span>
</pre><br/>
This will expose jetty-service to OSGi. All other components, which connect to it will wait automaticly until it's registered.
Exposed interface has following methods:
<pre class="hl">
<span class="hl kwa">public</span> <span class="hl kwc">Handler</span> <span class="hl kwd">registerApp</span><span class="hl sym">(</span><span class="hl kwc">String</span> name<span class="hl sym">,</span> <span class="hl kwc">Handler</span> handler<span class="hl sym">)</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span><span class="hl sym">;</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">unregisterApp</span><span class="hl sym">(</span><span class="hl kwc">Handler</span> handler<span class="hl sym">)</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span><span class="hl sym">;</span>
</pre><br/>
Those will be invoked to register Hello World application.
JettyServiceImpl on the other hand, starts embedded Jetty Server and handles apps registration.
<pre class="hl">
<span class="hl kwa">package</span> org<span class="hl sym">.</span>apache<span class="hl sym">.</span>jetty<span class="hl sym">.</span>service<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>apache<span class="hl sym">.</span>jetty<span class="hl sym">.</span>service<span class="hl sym">.</span>api<span class="hl sym">.</span>JettyService<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span><span class="hl kwc">Handler</span><span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>Server<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>handler<span class="hl sym">.</span>ContextHandler<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>handler<span class="hl sym">.</span>ContextHandlerCollection<span class="hl sym">;</span>
<span class="hl kwa">public class</span> JettyServiceImpl <span class="hl kwa">implements</span> JettyService <span class="hl sym">{</span>
<span class="hl kwa">private</span> Server server<span class="hl sym">;</span>
<span class="hl kwa">private</span> ContextHandlerCollection rootContext<span class="hl sym">;</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">init</span><span class="hl sym">()</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span> <span class="hl sym">{</span>
server <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwd">Server</span><span class="hl sym">(</span><span class="hl num">8080</span><span class="hl sym">);</span>
rootContext <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwd">ContextHandlerCollection</span><span class="hl sym">();</span>
server<span class="hl sym">.</span><span class="hl kwd">setHandler</span><span class="hl sym">(</span>rootContext<span class="hl sym">);</span>
server<span class="hl sym">.</span><span class="hl kwd">start</span><span class="hl sym">();</span>
<span class="hl sym">}</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">destroy</span><span class="hl sym">()</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span> <span class="hl sym">{</span>
server<span class="hl sym">.</span><span class="hl kwd">stop</span><span class="hl sym">();</span>
<span class="hl sym">}</span>
<span class="hl kwa">public</span> <span class="hl kwc">Handler</span> <span class="hl kwd">registerApp</span><span class="hl sym">(</span><span class="hl kwc">String</span> name<span class="hl sym">,</span> <span class="hl kwc">Handler</span> handler<span class="hl sym">)</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span> <span class="hl sym">{</span>
server<span class="hl sym">.</span><span class="hl kwd">stop</span><span class="hl sym">();</span>
ContextHandler h <span class="hl sym">=</span> rootContext<span class="hl sym">.</span><span class="hl kwd">addContext</span><span class="hl sym">(</span><span class="hl str">"/"</span> <span class="hl sym">+</span> name<span class="hl sym">,</span> name<span class="hl sym">);</span>
h<span class="hl sym">.</span><span class="hl kwd">setHandler</span><span class="hl sym">(</span>handler<span class="hl sym">);</span>
server<span class="hl sym">.</span><span class="hl kwd">start</span><span class="hl sym">();</span>
<span class="hl kwa">return</span> h<span class="hl sym">;</span>
<span class="hl sym">}</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">unregisterApp</span><span class="hl sym">(</span><span class="hl kwc">Handler</span> handler<span class="hl sym">)</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span> <span class="hl sym">{</span>
server<span class="hl sym">.</span><span class="hl kwd">stop</span><span class="hl sym">();</span>
rootContext<span class="hl sym">.</span><span class="hl kwd">removeHandler</span><span class="hl sym">(</span>handler<span class="hl sym">);</span>
server<span class="hl sym">.</span><span class="hl kwd">start</span><span class="hl sym">();</span>
<span class="hl sym">}</span>
<span class="hl sym">}</span>
</pre><br/>
<br/>
Next step is to implement sample web app.
First, we need to connect jetty-service bean to make it visible in our app.
<pre class="hl">
<span class="hl kwa"><osgi:reference</span> <span class="hl kwb">id</span>=<span class="hl str">"jetty-service"</span> <span class="hl kwb">interface</span>=<span class="hl str">"org.apache.jetty.service.api.JettyService"</span> <span class="hl kwb">bean-name</span>=<span class="hl str">"jetty-service"</span><span class="hl kwa">/></span>
<span class="hl kwa"><bean</span> <span class="hl kwb">class</span>=<span class="hl str">"org.apache.jetty.service.example.HelloWorld"</span> <span class="hl kwb">init-method</span>=<span class="hl str">"init"</span> <span class="hl kwb">destroy-method</span>=<span class="hl str">"destroy"</span><span class="hl kwa">></span>
<span class="hl kwa"><property</span> <span class="hl kwb">name</span>=<span class="hl str">"jettyService"</span> <span class="hl kwb">ref</span>=<span class="hl str">"jetty-service"</span><span class="hl kwa">/></span>
<span class="hl kwa"></bean></span>
</pre><br/>
Then, we need to implement sample app.
<pre class="hl">
<span class="hl kwa">package</span> org<span class="hl sym">.</span>apache<span class="hl sym">.</span>jetty<span class="hl sym">.</span>service<span class="hl sym">.</span>example<span class="hl sym">;</span>
<span class="hl kwa">import</span> java<span class="hl sym">.</span>io<span class="hl sym">.</span><span class="hl kwc">IOException</span><span class="hl sym">;</span>
<span class="hl kwa">import</span> javax<span class="hl sym">.</span>servlet<span class="hl sym">.</span>ServletException<span class="hl sym">;</span>
<span class="hl kwa">import</span> javax<span class="hl sym">.</span>servlet<span class="hl sym">.</span>http<span class="hl sym">.</span>HttpServletRequest<span class="hl sym">;</span>
<span class="hl kwa">import</span> javax<span class="hl sym">.</span>servlet<span class="hl sym">.</span>http<span class="hl sym">.</span>HttpServletResponse<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>apache<span class="hl sym">.</span>jetty<span class="hl sym">.</span>service<span class="hl sym">.</span>api<span class="hl sym">.</span>JettyService<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>apache<span class="hl sym">.</span>jetty<span class="hl sym">.</span>service<span class="hl sym">.</span>util<span class="hl sym">.</span>BundleResource<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span><span class="hl kwc">Handler</span><span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>HttpConnection<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span><span class="hl kwc">Request</span><span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>handler<span class="hl sym">.</span>AbstractHandler<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>handler<span class="hl sym">.</span>ContextHandler<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>handler<span class="hl sym">.</span>ContextHandlerCollection<span class="hl sym">;</span>
<span class="hl kwa">import</span> org<span class="hl sym">.</span>mortbay<span class="hl sym">.</span>jetty<span class="hl sym">.</span>handler<span class="hl sym">.</span>ResourceHandler<span class="hl sym">;</span>
<span class="hl kwa">public class</span> HelloWorld <span class="hl sym">{</span>
<span class="hl kwa">private</span> JettyService jettyService<span class="hl sym">;</span>
<span class="hl kwa">private</span> <span class="hl kwc">Handler</span> registered<span class="hl sym">;</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">setJettyService</span><span class="hl sym">(</span>JettyService jettyService<span class="hl sym">) {</span>
<span class="hl kwa">this</span><span class="hl sym">.</span>jettyService <span class="hl sym">=</span> jettyService<span class="hl sym">;</span>
<span class="hl sym">}</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">init</span><span class="hl sym">()</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span> <span class="hl sym">{</span>
ContextHandlerCollection handler <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwd">ContextHandlerCollection</span><span class="hl sym">();</span>
handler<span class="hl sym">.</span><span class="hl kwd">addContext</span><span class="hl sym">(</span><span class="hl str">"/app"</span><span class="hl sym">,</span> <span class="hl str">"app"</span><span class="hl sym">).</span><span class="hl kwd">setHandler</span><span class="hl sym">(</span><span class="hl kwa">new</span> <span class="hl kwd">AbstractHandler</span><span class="hl sym">() {</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">handle</span><span class="hl sym">(</span><span class="hl kwc">String</span> target<span class="hl sym">,</span> HttpServletRequest request<span class="hl sym">,</span>
HttpServletResponse response<span class="hl sym">,</span> <span class="hl kwb">int</span> arg3<span class="hl sym">)</span> <span class="hl kwa">throws</span> <span class="hl kwc">IOException</span><span class="hl sym">,</span>
ServletException <span class="hl sym">{</span>
response<span class="hl sym">.</span><span class="hl kwd">setContentType</span><span class="hl sym">(</span><span class="hl str">"text/html"</span><span class="hl sym">);</span>
response<span class="hl sym">.</span><span class="hl kwd">setStatus</span><span class="hl sym">(</span>HttpServletResponse<span class="hl sym">.</span>SC_OK<span class="hl sym">);</span>
response<span class="hl sym">.</span><span class="hl kwd">getWriter</span><span class="hl sym">().</span><span class="hl kwd">println</span><span class="hl sym">(</span><span class="hl str">"<h1>Hello World from Java</h1>"</span> <span class="hl sym">+</span> request<span class="hl sym">.</span><span class="hl kwd">getParameterMap</span><span class="hl sym">());</span>
<span class="hl kwc">Request</span> base_request <span class="hl sym">= (</span>request <span class="hl kwa">instanceof</span> <span class="hl kwc">Request</span><span class="hl sym">)</span> ? <span class="hl sym">(</span><span class="hl kwc">Request</span><span class="hl sym">)</span>request<span class="hl sym">:</span>HttpConnection<span class="hl sym">.</span><span class="hl kwd">getCurrentConnection</span><span class="hl sym">().</span><span class="hl kwd">getRequest</span><span class="hl sym">();</span>
base_request<span class="hl sym">.</span><span class="hl kwd">setHandled</span><span class="hl sym">(</span>true<span class="hl sym">);</span>
<span class="hl sym">}</span>
<span class="hl sym">});</span>
ResourceHandler resourceHandler <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwd">ResourceHandler</span><span class="hl sym">();</span>
resourceHandler<span class="hl sym">.</span><span class="hl kwd">setBaseResource</span><span class="hl sym">(</span><span class="hl kwa">new</span> <span class="hl kwd">BundleResource</span><span class="hl sym">(</span><span class="hl kwd">getClass</span><span class="hl sym">().</span><span class="hl kwd">getResource</span><span class="hl sym">(</span><span class="hl str">"/static"</span><span class="hl sym">)));</span>
ContextHandler contextHandler <span class="hl sym">=</span> handler<span class="hl sym">.</span><span class="hl kwd">addContext</span><span class="hl sym">(</span><span class="hl str">""</span><span class="hl sym">,</span><span class="hl str">""</span><span class="hl sym">);</span>
contextHandler<span class="hl sym">.</span><span class="hl kwd">setHandler</span><span class="hl sym">(</span>resourceHandler<span class="hl sym">);</span>
registered <span class="hl sym">=</span> jettyService<span class="hl sym">.</span><span class="hl kwd">registerApp</span><span class="hl sym">(</span><span class="hl str">"helloWorld"</span><span class="hl sym">,</span> handler<span class="hl sym">);</span>
<span class="hl sym">}</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">destroy</span><span class="hl sym">()</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span> <span class="hl sym">{</span>
jettyService<span class="hl sym">.</span><span class="hl kwd">unregisterApp</span><span class="hl sym">(</span>registered<span class="hl sym">);</span>
<span class="hl sym">}</span>
<span class="hl sym">}</span>
</pre><br/>
Here, we register sample request handler at 'app' sub path and serve static content from jar using BundleResource.
Last thing is registering app using jettyService under 'helloWorld' context. So our application will be exposed under
<a href="http://localhost:8080/helloWorld/">http://localhost:8080/helloWorld/</a> address.
<br/>
ServiceMix has also so called features. This is the way to collect multiple dependencies under a single name. So we have to create features.xml file, like this:
<pre class="hl">
<span class="hl sym"><</span>features<span class="hl sym">></span>
<span class="hl sym"><</span>feature name<span class="hl sym">=</span><span class="hl str">"jetty-service"</span> version<span class="hl sym">=</span><span class="hl str">"${project.version}"</span><span class="hl sym">></span>
<span class="hl sym"><</span>bundle<span class="hl sym">></span>mvn<span class="hl sym">:</span>org<span class="hl sym">.</span>apache<span class="hl sym">.</span>jetty<span class="hl sym">.</span>service<span class="hl sym">/</span>service<span class="hl sym">/</span>$<span class="hl sym">{</span>project<span class="hl sym">.</span>version<span class="hl sym">}</</span>bundle<span class="hl sym">></span>
<span class="hl sym"></</span>feature<span class="hl sym">></span>
<span class="hl sym"><</span>feature name<span class="hl sym">=</span><span class="hl str">"example-jetty-service-helloworld"</span> version<span class="hl sym">=</span><span class="hl str">"${project.version}"</span><span class="hl sym">></span>
<span class="hl sym"><</span>feature version<span class="hl sym">=</span><span class="hl str">"${project.version}"</span><span class="hl sym">></span>jetty<span class="hl sym">-</span>service<span class="hl sym"></</span>feature<span class="hl sym">></span>
<span class="hl sym"><</span>bundle<span class="hl sym">></span>mvn<span class="hl sym">:</span>org<span class="hl sym">.</span>apache<span class="hl sym">.</span>jetty<span class="hl sym">.</span>service<span class="hl sym">/</span>example<span class="hl sym">-</span>helloworld<span class="hl sym">/</span>$<span class="hl sym">{</span>project<span class="hl sym">.</span>version<span class="hl sym">}</</span>bundle<span class="hl sym">></span>
<span class="hl sym"></</span>feature<span class="hl sym">></span>
<span class="hl sym"></</span>features<span class="hl sym">></span>
</pre><br/>
Basicly, we can provide particular dependencies for our project.
<br/>
Next, we do 'mvn install' on our project and run apache-servicemix-4.2.0-fuse-01-00/bin/servicemix karaf console.
On the console, we need to type following commands:
<pre class="hl">
features:addUrl mvn:org.apache.jetty.service/service-karaf/0.1.0-SNAPSHOT/xml/features
features:install example-jetty-service-helloworld
osgi:list
[ 230] [Active ] [ ] [Started] [ 60] Unnamed - org.apache.jetty.service:service:bundle:0.1.0-SNAPSHOT (0.1.0.SNAPSHOT)
[ 231] [Active ] [ ] [Started] [ 60] Unnamed - org.apache.jetty.service:example-helloworld:bundle:0.1.0-SNAPSHOT (0.1.0.SNAPSHOT)</pre><br/>
Now, we can enter <a href="http://localhost:8080/helloWorld/">http://localhost:8080/helloWorld/</a> to test our app.
<br/>
And that's it.
OSGi and ServiceMix 4 features enable easy way to use dynamic modules in web apps. For example, it's very
easy to build simple web framework with loadable components on page (something like mini implementation of Portlets).
<br/>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com5tag:blogger.com,1999:blog-6983141563768732668.post-8593463652982057612010-07-10T09:54:00.000-07:002010-07-10T09:54:02.066-07:00Integration Tests for SMX4 with PythonIntegration tests and unit tests are important for project quality.
Unit tests usually are well suited for developer to verify his changes in runtime.
On the other hand, integration tests, are for target user to verify that project's features in the way he interacts with project,
work properly.
In this article, I will show how to automate integration tests for ServiceMix 4 using SoapUI testrunner and a simple python script.
<br/>
The idea is to spawn ServiceMix 4 Karaf console and interact with it using python expect library. During this interaction,
SoapUI testrunner script is invoked in order to run SoapUI tests.
<br/>
First, we need to grab SMX4_DIR and SOAPUI_DIR environment variables in our script, like this:
<pre class="hl">
SMX4_DIR<span class="hl sym">=</span>os<span class="hl sym">.</span><span class="hl kwd">getenv</span><span class="hl sym">(</span><span class="hl str">"SMX4_DIR"</span><span class="hl sym">)</span>
SOAPUI_DIR<span class="hl sym">=</span>os<span class="hl sym">.</span><span class="hl kwd">getenv</span><span class="hl sym">(</span><span class="hl str">"SOAPUI_DIR"</span><span class="hl sym">)</span>
</pre><br/>
This way, we can invoke later our script using following shell command:
<pre class="hl">
SMX4_DIR<span class="hl sym">=/</span>some<span class="hl sym">/</span>path SOAPUI_DIR<span class="hl sym">=/</span>some<span class="hl sym">/</span>other<span class="hl sym">/</span>path .<span class="hl sym">/</span>our<span class="hl sym">-</span>python<span class="hl sym">-</span><span class="hl kwc">script</span>
</pre><br/>
Then, we need to spawn ServiceMix 4 console by using python expect library:
<pre class="hl">
<span class="hl kwa">import</span> pexpect
<span class="hl kwa">import</span> time
<span class="hl kwa">import</span> sys
child <span class="hl sym">=</span> pexpect<span class="hl sym">.</span><span class="hl kwd">spawn</span><span class="hl sym">(</span><span class="hl str">"bin/servicemix"</span><span class="hl sym">)</span>
child<span class="hl sym">.</span>logfile <span class="hl sym">=</span> sys<span class="hl sym">.</span>stdout
child<span class="hl sym">.</span><span class="hl kwd">expect</span><span class="hl sym">(</span><span class="hl str">"karaf.*>"</span><span class="hl sym">)</span>
time<span class="hl sym">.</span><span class="hl kwd">sleep</span><span class="hl sym">(</span><span class="hl num">3</span><span class="hl sym">)</span>
</pre><br/>
Here, we set logfile to stdout in order to see our automated interaction with ServiceMix console.
Then we need to wait for ServiceMix console command prompt, which would mean console is ready.
Additionally, we need to wait a few seconds to avoid problems with running commands too early (which is a kind of small bug in ServiceMix).
Then, we can install our features, which we want to test. This example starts Apache HISE test bundle, which loads also Apache HISE engine from dependencies.
<pre class="hl">
child<span class="hl sym">.</span><span class="hl kwd">sendline</span><span class="hl sym">(</span><span class="hl str">"features:addUrl mvn:org.apache.hise/hise-karaf/0.3.0-SNAPSHOT/xml/features"</span><span class="hl sym">);</span>
child<span class="hl sym">.</span><span class="hl kwd">expect</span><span class="hl sym">(</span><span class="hl str">"karaf.*>"</span><span class="hl sym">)</span>
child<span class="hl sym">.</span><span class="hl kwd">sendline</span><span class="hl sym">(</span><span class="hl str">"features:install hise-h2-test-example-osgi"</span><span class="hl sym">)</span>
child<span class="hl sym">.</span><span class="hl kwd">expect</span><span class="hl sym">(</span><span class="hl str">"karaf.*>"</span><span class="hl sym">)</span>
</pre><br/>
Next, we need to wait until the feature is properly started.
ServiceMix 4 OSGi container initializes bundles in background, so it's not enough to wait for command prompt to have it started (there doesn't seem to exist a "wait-until-started" console command).
So we grep in a loop over installed bundles and see if status is started.
In this example, we do 30 retries every second and fail our integration test script after this period, by raising exception.
<pre class="hl">
child<span class="hl sym">.</span><span class="hl kwd">sendline</span><span class="hl sym">(</span><span class="hl str">"features:addUrl mvn:org.apache.hise/hise-karaf/0.3.0-SNAPSHOT/xml/features"</span><span class="hl sym">);</span>
rep<span class="hl sym">=</span><span class="hl num">0</span>
<span class="hl kwa">while True</span><span class="hl sym">:</span>
child<span class="hl sym">.</span><span class="hl kwd">sendline</span><span class="hl sym">(</span><span class="hl str">"osgi:list|grep -i hise-test-example-osgi"</span><span class="hl sym">)</span>
l<span class="hl sym">=</span>child<span class="hl sym">.</span><span class="hl kwd">readline</span><span class="hl sym">()</span>
l<span class="hl sym">=</span>child<span class="hl sym">.</span><span class="hl kwd">readline</span><span class="hl sym">()</span>
<span class="hl kwa">if</span> re<span class="hl sym">.</span><span class="hl kwd">match</span><span class="hl sym">(</span><span class="hl str">".*Started"</span><span class="hl sym">,</span> l<span class="hl sym">) !=</span> <span class="hl kwa">None</span><span class="hl sym">:</span>
<span class="hl kwa">break</span>
time<span class="hl sym">.</span><span class="hl kwd">sleep</span><span class="hl sym">(</span><span class="hl num">1</span><span class="hl sym">)</span>
child<span class="hl sym">.</span><span class="hl kwd">expect</span><span class="hl sym">(</span><span class="hl str">"karaf.*>"</span><span class="hl sym">)</span>
rep<span class="hl sym">=</span>rep<span class="hl sym">+</span><span class="hl num">1</span>
<span class="hl kwa">if</span> rep<span class="hl sym">></span><span class="hl num">30</span><span class="hl sym">:</span>
<span class="hl kwa">raise</span> <span class="hl kwc">Exception</span><span class="hl sym">(</span><span class="hl str">"Bundle not installed"</span><span class="hl sym">)</span>
</pre><br/>
Next, we need to run SoapUI testrunner in order to execute test cases. We need to implement syscall method in order to fail integration tests if SoapUI testrunner completes with fault (non-zero exit code).
<pre class="hl">
<span class="hl kwa">import</span> os
<span class="hl kwa">def</span> <span class="hl kwd">syscall</span><span class="hl sym">(</span>c<span class="hl sym">):</span>
<span class="hl kwa">if</span> os<span class="hl sym">.</span><span class="hl kwd">system</span><span class="hl sym">(</span>c<span class="hl sym">) !=</span> <span class="hl num">0</span><span class="hl sym">:</span>
<span class="hl kwa">raise</span> <span class="hl kwc">Exception</span><span class="hl sym">(</span><span class="hl str">"Sys call failed: "</span> <span class="hl sym">+</span> c<span class="hl sym">)</span>
<span class="hl kwd">syscall</span><span class="hl sym">(</span>SOAPUI_DIR <span class="hl sym">+</span> <span class="hl str">"/bin/testrunner.sh -f results hise-soapui-project.xml"</span><span class="hl sym">)</span>
</pre><br/>
At the end, we can exit gracefully from ServiceMix console by using shutdown command, like this:
<pre class="hl">
child<span class="hl sym">.</span><span class="hl kwd">sendline</span><span class="hl sym">(</span><span class="hl str">"shutdown"</span><span class="hl sym">)</span>
</pre><br/>
<br/>
And that's it.
Full code of integration test script is available in Apache HISE sources, from Apache repository <a href="http://svn.apache.org/repos/asf/incubator/hise/trunk/itest/itest">http://svn.apache.org/repos/asf/incubator/hise/trunk/itest/itest</a>.Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com3tag:blogger.com,1999:blog-6983141563768732668.post-35704159658368814222010-06-24T17:47:00.000-07:002010-06-24T17:47:07.303-07:00Apache ODE 1.3.4 releasedApache ODE 1.3.4 has been released recently. You can grab it from here <a href="http://ode.apache.org/getting-ode.html">http://ode.apache.org/getting-ode.html</a>.
It has a few new features and a lot of bugfixes.
<br/>
<br/>
First of new features is Instance Replayer (<a href="http://ode.apache.org/instance-replayer">http://ode.apache.org/instance-replayer</a>).
In short, it's able to restore business process instance by reexecuting every step again based on incoming and outgoing instance communication (it mocks up outgoing requests).
<br/>
The basic use cases for replayer are:
<br/>
1. migrate existing log running instances to newest process version
given their communication (incoming and outgoing requests)
<br/>
2. reproduce error scenarios between two instances of ODE (eg.
production and development)
<br/>
Instance Replayer extends management api by 2 operations: replay and getCommunication.
In order to do 1, you can invoke following request:
<pre class="hl">
<span class="hl kwa"><pmap:replay></span>
<span class="hl kwa"><replay></span>
<span class="hl kwa"><ns:upgradeInstance></span><span class="hl num">1234</span><span class="hl kwa"></ns:upgradeInstance></span>
<span class="hl kwa"></replay></span>
<span class="hl kwa"></pmap:replay></span>
</pre><br/>
<br/>
You get a new instance in the newest process version. The old one will be deleted.
<br/>
In order to do 2, you need to retrieve exchanges from existing instance (or instances) by following request:
<pre class="hl">
<span class="hl kwa"><pmap:getCommunication></span>
<span class="hl kwa"><getCommunication></span>
<span class="hl kwa"><ns:iid></span><span class="hl num">1234</span><span class="hl kwa"></ns:iid></span>
<span class="hl kwa"></getCommunication></span>
<span class="hl kwa"></pmap:getCommunication></span>
</pre><br/>
<br/>
Then, you can execute replay operation on another ODE installation in order to replicate instance:
<pre class="hl">
<span class="hl kwa"><pmap:replay</span> xmlns:<span class="hl kwb">ns</span>=<span class="hl str">"http://www.apache.org/ode/pmapi/types/2006/08/02/"</span><span class="hl kwa">></span>
<span class="hl kwa"><replay></span>
<span class="hl kwa"><ns:restoreInstance></span>
<span class="hl kwa"><ns:processType</span>
xmlns:<span class="hl kwb">p</span>=<span class="hl str">"http://sample.bpel.org/bpel/sample"</span><span class="hl kwa">></span>p:OnEventCorrelation<span class="hl kwa"></ns:processType></span>
... exchanges
<span class="hl kwa"></ns:restoreInstance></span>
<span class="hl kwa"></replay></span>
<span class="hl kwa"></pmap:replay></span>
</pre><br/>
<br/>
In order to have time control in bpel process, there is introduced special variable
$ode:currentEventDateTime. It's equivalent to $fn:current-dateTime() standard XPath function
during live session (while no replaying takes place) and it's set to corresponding time in past during
replaying.
<br/>
<br/>
The next new feature is SMX4 OSGi bundle distibution. Some detailed information on how to configure it is here: <a href="http://ode.apache.org/smx4-osgi-deployment.html">http://ode.apache.org/smx4-osgi-deployment.html</a>.
In short, it's possible to deploy ODE and processes as OSGi bundles in ServiceMix4. This simplifies project structure compared to ServiceAssembly and ServiceUnit double zipped packages.
<br/>
<br/>
Third feature is small, but may be useful. It makes possible to define Spring properties and use them inside business process.
All you need to do is to put beans.xml file in process directory (example is here: <a href="http://svn.apache.org/repos/asf/ode/tags/APACHE_ODE_1.3.4/jbi/src/test/resources/SpringPropertiesJbiTest/">http://svn.apache.org/repos/asf/ode/tags/APACHE_ODE_1.3.4/jbi/src/test/resources/SpringPropertiesJbiTest/</a>).
You need to define bean of name properties, like this:
<pre class="hl">
<span class="hl kwa"><beans></span>
<span class="hl kwa"><util:properties</span> <span class="hl kwb">id</span>=<span class="hl str">"properties"</span><span class="hl kwa">></span>
<span class="hl kwa"><prop</span> <span class="hl kwb">key</span>=<span class="hl str">"weirdProp"</span><span class="hl kwa">></span>myWeirdPropValue2<span class="hl kwa"></prop></span>
<span class="hl kwa"></util:properties></span>
<span class="hl kwa"></beans></span>
</pre><br/>
ODE will instantiate it and you can use it later, like this:
<pre class="hl">
<span class="hl kwa"><bpws:assign></span>
<span class="hl kwa"><bpws:copy></span>
<span class="hl kwa"><bpws:from></span>ode:process-property('weirdProp')<span class="hl kwa"></bpws:from></span>
<span class="hl kwa"><bpws:to></span>$output.payload<span class="hl kwa"></bpws:to></span>
<span class="hl kwa"></bpws:copy></span>
<span class="hl kwa"></bpws:assign></span>
</pre><br/>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com2tag:blogger.com,1999:blog-6983141563768732668.post-66618707717247709612010-02-23T07:37:00.001-08:002010-02-23T07:37:02.579-08:00Git, Beginner to Advanced SurveyCheck out this SlideShare Presentation: <div style="width:425px;text-align:left" id="__ss_3255219"><a style="font:14px Helvetica,Arial,Sans-serif;display:block;margin:12px 0 3px 0;text-decoration:underline;" href="http://www.slideshare.net/rrusin/git-beginner-to-advanced-survey" title="Git, Beginner to Advanced Survey">Git, Beginner to Advanced Survey</a><object style="margin:0px" width="425" height="355"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=git-100223061944-phpapp01&stripped_title=git-beginner-to-advanced-survey" /><param name="allowFullScreen" value="true"/><param name="allowScriptAccess" value="always"/><embed src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=git-100223061944-phpapp01&stripped_title=git-beginner-to-advanced-survey" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><div style="font-size:11px;font-family:tahoma,arial;height:26px;padding-top:2px;">View more <a style="text-decoration:underline;" href="http://www.slideshare.net/">presentations</a> from <a style="text-decoration:underline;" href="http://www.slideshare.net/rrusin">Rafal Rusin</a>.</div></div>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com1tag:blogger.com,1999:blog-6983141563768732668.post-41642306521278533882010-01-19T14:11:00.000-08:002010-02-19T03:52:31.654-08:00xquery4j in actionIn my previous article, I introduced a wrapper library for Saxon, xquery4j <a href="http://github.com/rafalrusin/xquery4j">http://github.com/rafalrusin/xquery4j</a>.<br/>Here, I will explain how to use it to create an article generator in Java and XQuery for XHTML, called Article. You can download it here: <a href="http://github.com/rafalrusin/Article">http://github.com/rafalrusin/Article</a>. It's a simple DSL for article generation.<br/><br />
I think it is something worth noticing, because the whole project took me just a while to implement and has interesting features. Those are:<br />
<ul xmlns:a="urn:article"> <li>embedded code syntax highlighting for a lot of programming languages (using external program highlight), </li>
<li>creating href entries for links, so you don't need to type URL twice</li>
<li>it integrates natively with XHTML constructs</li>
</ul> <br/>This is an example of an input it takes:<br/><pre class="hl"><span class="hl kwa"><a:article</span> xmlns='http://www.w3.org/<span class="hl num">1999</span>/xhtml' xmlns:<span class="hl kwb">a</span>=<span class="hl str">"urn:article"</span><span class="hl kwa">></span>
<span class="hl kwa"><a:l></span>Some text<span class="hl kwa"></a:l></span>
<span class="hl kwa"><a:code</span> <span class="hl kwb">lang</span>=<span class="hl str">"xml"</span><span class="hl kwa">></span><![CDATA[
<span class="hl kwa"><someXml/></span>
]]<span class="hl kwa">></a:code></span>
<span class="hl kwa"></a:article></span>
</pre><br/>It generates XHTML output for it, using command<br/><pre class="hl">.<span class="hl sym">/</span>run <span class="hl sym"><</span>input.xml <span class="hl sym">></span>output.xhtml
</pre><br/>The interesting thing is that XQuery expression for this transformation is very simple to do in Saxon. This is the complete code of it:<br/><pre class="hl">declare namespace <span class="hl kwb">a</span>=<span class="hl str">"urn:article"</span>;
declare default element namespace <span class="hl str">"http://www.w3.org/1999/xhtml"</span>;
declare function a:processLine($l) {
for $i in $l/node()
return
typeswitch ($i)
case element(a:link, xs:untyped) return <span class="hl kwa"><a</span> <span class="hl kwb">href</span>=<span class="hl str">"{$i/text()}"</span><span class="hl kwa">></span>{$i/text()}<span class="hl kwa"></a></span>
default return $i
};
declare function a:articleItem($i) {
typeswitch ($i)
case element(a:l, xs:untyped) return (a:processLine($i), <span class="hl kwa"><br/></span>)
case element(a:code, xs:untyped) return
( a:highlight($i/text(), $i/@lang)/body/* , <span class="hl kwa"><br/></span>)
default return <span class="hl str">"error;"</span>
};
<span class="hl kwa"><html</span> <span class="hl kwb">xmlns</span>=<span class="hl str">"http://www.w3.org/1999/xhtml"</span><span class="hl kwa">></span>
<span class="hl kwa"><head></span>
<span class="hl kwa"><title></span>a.xml<span class="hl kwa"></title></span>
<span class="hl kwa"><link</span> <span class="hl kwb">rel</span>=<span class="hl str">"stylesheet"</span> <span class="hl kwb">type</span>=<span class="hl str">"text/css"</span> <span class="hl kwb">href</span>=<span class="hl str">"highlight.css"</span><span class="hl kwa">/></span>
<span class="hl kwa"></head></span>
<span class="hl kwa"><body></span>
{
for $i in a:article/*
return
a:articleItem($i)
}
<span class="hl kwa"></body></span>
<span class="hl kwa"></html></span>
</pre><br/>Inside this expression, there is bound a:highlight Java function, which takes two strings on input (a code and a language) and returns DOM Node containing XHTML output from highlight command.<br/>Since there is not much trouble with manipulating DOM using xquery4j, we can get as simple solution as this for a:highlight function:<br/><pre class="hl"> <span class="hl kwa">public static class</span> Mod <span class="hl sym">{</span>
<span class="hl kwa">public static</span> <span class="hl kwc">Node</span> <span class="hl kwd">highlight</span><span class="hl sym">(</span><span class="hl kwa">final</span> <span class="hl kwc">String</span> code<span class="hl sym">,</span> <span class="hl kwc">String</span> lang<span class="hl sym">)</span> <span class="hl kwa">throws</span> <span class="hl kwc">Exception</span> <span class="hl sym">{</span>
Validate<span class="hl sym">.</span><span class="hl kwd">notNull</span><span class="hl sym">(</span>lang<span class="hl sym">);</span>
<span class="hl kwa">final</span> <span class="hl kwc">Process</span> p <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwc">ProcessBuilder</span><span class="hl sym">(</span><span class="hl str">"highlight"</span><span class="hl sym">,</span> <span class="hl str">"-X"</span><span class="hl sym">,</span> <span class="hl str">"--syntax"</span><span class="hl sym">,</span> lang<span class="hl sym">).</span><span class="hl kwd">start</span><span class="hl sym">();</span>
<span class="hl kwc">Thread</span> t <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwc">Thread</span><span class="hl sym">(</span><span class="hl kwa">new</span> <span class="hl kwc">Runnable</span><span class="hl sym">() {</span>
<span class="hl kwa">public</span> <span class="hl kwb">void</span> <span class="hl kwd">run</span><span class="hl sym">() {</span>
<span class="hl kwa">try</span> <span class="hl sym">{</span>
<span class="hl kwc">OutputStream</span> out <span class="hl sym">=</span> p<span class="hl sym">.</span><span class="hl kwd">getOutputStream</span><span class="hl sym">();</span>
IOUtils<span class="hl sym">.</span><span class="hl kwd">write</span><span class="hl sym">(</span>code<span class="hl sym">,</span> out<span class="hl sym">);</span>
out<span class="hl sym">.</span><span class="hl kwd">flush</span><span class="hl sym">();</span>
out<span class="hl sym">.</span><span class="hl kwd">close</span><span class="hl sym">();</span>
<span class="hl sym">}</span> <span class="hl kwa">catch</span> <span class="hl sym">(</span><span class="hl kwc">IOException</span> e<span class="hl sym">) {</span>
<span class="hl kwa">throw new</span> <span class="hl kwc">RuntimeException</span><span class="hl sym">(</span>e<span class="hl sym">);</span>
<span class="hl sym">}</span>
<span class="hl sym">}</span>
<span class="hl sym">});</span>
t<span class="hl sym">.</span><span class="hl kwd">start</span><span class="hl sym">();</span>
<span class="hl kwc">String</span> result <span class="hl sym">=</span> IOUtils<span class="hl sym">.</span><span class="hl kwd">toString</span><span class="hl sym">(</span>p<span class="hl sym">.</span><span class="hl kwd">getInputStream</span><span class="hl sym">());</span>
t<span class="hl sym">.</span><span class="hl kwd">join</span><span class="hl sym">();</span>
<span class="hl kwa">return</span> DOMUtils<span class="hl sym">.</span><span class="hl kwd">parse</span><span class="hl sym">(</span>result<span class="hl sym">).</span><span class="hl kwd">getDocumentElement</span><span class="hl sym">();</span>
<span class="hl sym">}</span>
<span class="hl sym">}</span>
</pre><br/>Please note that creating a separate thread for feeding input into highlight command is required, since Thread's output queue is limited and potentially might lead to dead lock. So we need to concurrently collect output from spawned Process.<br/>However at the end, when we need to convert a String to DOM and we use xquery4j's DOMUtils.parse(result), so it's a very simple construct.<br/>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com4tag:blogger.com,1999:blog-6983141563768732668.post-75945501662464671712010-01-16T14:11:00.000-08:002010-01-18T13:36:24.247-08:00Embedding XQuery in JavaXQuery is a very powerful language. It can be very useful when you want to do some XML processing in Java.<br/>Let's say you want to create an XML document based on some other XML data. Given something like this:<br/><pre class="hl"><span class="hl kwa"><employees></span>
<span class="hl kwa"><employee></span>
<span class="hl kwa"><name></span>Fred Jones<span class="hl kwa"></name></span>
<span class="hl kwa"><address</span> <span class="hl kwb">location</span>=<span class="hl str">"home"</span><span class="hl kwa">></span>
<span class="hl kwa"><street></span><span class="hl num">900</span> Aurora Ave.<span class="hl kwa"></street></span>
<span class="hl kwa"><city></span>Seattle<span class="hl kwa"></city></span>
<span class="hl kwa"><state></span>WA<span class="hl kwa"></state></span>
<span class="hl kwa"><zip></span><span class="hl num">98115</span><span class="hl kwa"></zip></span>
<span class="hl kwa"></address></span>
<span class="hl kwa"><address</span> <span class="hl kwb">location</span>=<span class="hl str">"work"</span><span class="hl kwa">></span>
<span class="hl kwa"><street></span><span class="hl num">2011 152</span>nd Avenue NE<span class="hl kwa"></street></span>
<span class="hl kwa"><city></span>Redmond<span class="hl kwa"></city></span>
<span class="hl kwa"><state></span>WA<span class="hl kwa"></state></span>
<span class="hl kwa"><zip></span><span class="hl num">98052</span><span class="hl kwa"></zip></span>
<span class="hl kwa"></address></span>
<span class="hl kwa"><phone</span> <span class="hl kwb">location</span>=<span class="hl str">"work"</span><span class="hl kwa">></span>(<span class="hl num">425</span>)<span class="hl num">555</span>-<span class="hl num">5665</span><span class="hl kwa"></phone></span>
<span class="hl kwa"><phone</span> <span class="hl kwb">location</span>=<span class="hl str">"home"</span><span class="hl kwa">></span>(<span class="hl num">206</span>)<span class="hl num">555</span>-<span class="hl num">5555</span><span class="hl kwa"></phone></span>
<span class="hl kwa"><phone</span> <span class="hl kwb">location</span>=<span class="hl str">"mobile"</span><span class="hl kwa">></span>(<span class="hl num">206</span>)<span class="hl num">555</span>-<span class="hl num">4321</span><span class="hl kwa"></phone></span>
<span class="hl kwa"></employee></span>
<span class="hl kwa"></employees></span>
</pre><br/>You want to produce employees' names:<br/><pre class="hl"><span class="hl kwa"><names></span>
<span class="hl kwa"><name></span>Fred Jones<span class="hl kwa"></name></span>
<span class="hl kwa"></names></span>
</pre><br/>In XQuery it's just as easy as:<br/><pre class="hl"><span class="hl kwa"><names></span>
{for $name in employees/employee/name/text() return <span class="hl kwa"><name></span>{$name}<span class="hl kwa"></name></span>}
<span class="hl kwa"></names></span>
</pre><br/>The most interesting advantage for XQuery over various other methods for generating XML is that XQuery operates natively on XML.<br/>There are tools like JaxB, XmlBeans, which enable strongly typed XML building directly from Java code. But using such approach often requires a lot of Java code to be written, which is not really necessary.<br/>There is also a possibility to use XPath. However it's an inferior solution to XQuery, because XPath doesn't provide a way for building XML documents. It's designed only for nodes selection. On the other hand, XQuery extends XPath, so it supports every construct that XPath does.<br/>Another way to do such processing in Java is to use XSLT. It's the closest approach to XQuery. But the problem with XSLT is that it has its language constructs, like 'for' expressed as XML elements. This makes writing XSLT code much more difficult that XQuery.<br/>XQuery can be seen as a native template language for XML processing.<br/>So the question is: how to evaluate XQuery expressions the best way from Java?<br/>There are some Open Source implementations of XQuery for Java. One of them is inside XmlBeans. However in my opinion the best way is to use Saxon. It's the most mature project for XQuery processing and it's targetted directly for doing that.<br/>However Saxon might be a bit difficult to use directly. At least digging a few interesting features from it took me some time.<br/>So I decided to write a simple class for interfacing Saxon and to provide a few interesting examples of how to use it. That's how xquery4j was born. You can download it from github <a href="http://github.com/rafalrusin/xquery4j" xmlns:a="urn:article">http://github.com/rafalrusin/xquery4j</a>.<br/>In xquery4j, you can execute XQuery expressions from Java in a simple way:<br/><pre class="hl">XQueryEvaluator evaluator <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwd">XQueryEvaluator</span><span class="hl sym">();</span>
<span class="hl kwc">Long</span> result <span class="hl sym">= (</span><span class="hl kwc">Long</span><span class="hl sym">)</span> evaluator<span class="hl sym">.</span><span class="hl kwd">evaluateExpression</span><span class="hl sym">(</span><span class="hl str">"5+5"</span><span class="hl sym">,</span> null<span class="hl sym">).</span><span class="hl kwd">get</span><span class="hl sym">(</span><span class="hl num">0</span><span class="hl sym">);</span>
Assert<span class="hl sym">.</span><span class="hl kwd">assertEquals</span><span class="hl sym">(</span><span class="hl kwa">new</span> <span class="hl kwc">Long</span><span class="hl sym">(</span><span class="hl num">10</span><span class="hl sym">),</span> result<span class="hl sym">);</span>
</pre><br/>It's possible to bind variables from Java objects, the easy way:<br/><pre class="hl">evaluator<span class="hl sym">.</span><span class="hl kwd">bindVariable</span><span class="hl sym">(</span><span class="hl kwc">QName</span><span class="hl sym">.</span><span class="hl kwd">valueOf</span><span class="hl sym">(</span><span class="hl str">"myVar"</span><span class="hl sym">),</span> <span class="hl num">123</span><span class="hl sym">);</span>
</pre><br/>Sometimes it's also useful to declare Java methods and bind them for XQuery expressions. This is also very simple to do with xquery4j:<br/><pre class="hl"><span class="hl kwa">public static class</span> MyFunctions <span class="hl sym">{</span>
<span class="hl kwa">public static</span> <span class="hl kwc">String</span> <span class="hl kwd">myHello</span><span class="hl sym">(</span><span class="hl kwc">String</span> arg<span class="hl sym">) {</span>
TestEvaluator te <span class="hl sym">= (</span>TestEvaluator<span class="hl sym">)</span> XQueryEvaluator<span class="hl sym">.</span>contextObjectTL<span class="hl sym">.</span><span class="hl kwd">get</span><span class="hl sym">();</span>
te<span class="hl sym">.</span>id<span class="hl sym">++;</span>
<span class="hl kwa">return</span> <span class="hl str">"hello("</span> <span class="hl sym">+</span> arg <span class="hl sym">+</span> te<span class="hl sym">.</span>id <span class="hl sym">+</span> <span class="hl str">")"</span><span class="hl sym">;</span>
<span class="hl sym">}</span>
<span class="hl sym">}</span>
XQueryEvaluator evaluator <span class="hl sym">=</span> <span class="hl kwa">new</span> <span class="hl kwd">XQueryEvaluator</span><span class="hl sym">();</span>
evaluator<span class="hl sym">.</span><span class="hl kwd">setContextObject</span><span class="hl sym">(</span><span class="hl kwa">this</span><span class="hl sym">);</span>
evaluator<span class="hl sym">.</span><span class="hl kwd">declareJavaClass</span><span class="hl sym">(</span><span class="hl str">"http://my.org/employees"</span><span class="hl sym">,</span> MyFunctions<span class="hl sym">.</span><span class="hl kwa">class</span><span class="hl sym">);</span>
<span class="hl sym">}</span>
</pre><br/>This code sets a context object to 'this' and binds all static methods from MyFunctions class to XQuery expressions. So during myHello execution from XQuery, you can easily operate on Java variables from bound context - 'id' in this case.<br />
Here's a way of invoking such bound myHello method from XQuery:<br/><pre class="hl">declare namespace my = 'http://my.org/employees'; my:myHello(<span class="hl str">"hello"</span>)
</pre><br/>xquery4j code contains unit tests, which include examples above.<br/>You can run them by:<br/><pre class="hl">mvn package
</pre><br/>Feel free to give some feedback on using it.<br/>Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com85tag:blogger.com,1999:blog-6983141563768732668.post-57676794883269613762009-12-08T12:53:00.000-08:002009-12-08T12:53:20.744-08:00ODEO new releaseRecently, I released a new version 1.1.34 of ODEO (it's ODE tuned for Oracle and ServiceMix).<br />
You can check details (and downloads) here: <a href="http://top.touk.pl/confluence/display/top/ODEO">http://top.touk.pl/confluence/display/top/ODEO.</a><br />
This version contains yet another set of fixes, which drive your BPEL engine more robust and useful for production use.<br />
Recent improvements include: faster redeployment for high-load installations, priorities for jobs and digests for compiled processes (which enable substitution of xpaths in processes and have old instances continued in new version).<br />
There are also some bug fixes and user experience improvements, so you are welcome to check it out!Rafal Rusinhttp://www.blogger.com/profile/09196932088834314947noreply@blogger.com15