Profile cover photo
Profile photo
Pulasthi Supun Wickramasinghe
717 followers
717 followers
About
Pulasthi's posts

Post has attachment
Shell script for Tree structured copying to copy large data files to large number of nodes with scp
Sometimes you need to copy a large file to number of remote hosts. I recently had a similar situation where i had to copy a 56GB data file to around 30 compute nodes in an HPC cluster. And i did not have the option to copy it to the shared disk (since it wa...

Post has attachment
Setting up Heron Cluster with Apache Aurora Locally
In this post we will be looking at how we can setup Heron steam processing engine in Apache Aurora in our local machine. Oh Boy this is going to be a long post :D. I am doing this on Ubuntu 14.04 and these steps should be similar to any Linux machine. Heron...

Post has attachment

Post has attachment
Getting started with Heron in Ubuntu 14.04
I was trying to get started with Heron which is a stream processing engine from twitter and faced some problems when trying to do the initial setup on Ubuntu. I am using Ubuntu 14.04 so not these problems might not happen in other Ubuntu versions. The steps...

Post has attachment
Apache Hadoop detailed word count example from scratch
In this post we will look at how to create and run a word count program in Apache Hadoop. In order to make it easy for a beginner we will cover most of the setup steps as well. Please note that this blog entry is for Linux based environment. I am running Ub...

Post has attachment
Reading and Writing Binary files in java - Converting Endianness ( between Big-endian byte order and Little-endian byte order)
In this post we will look into working with binary files in java. The end goal of the post will be to create simple java application that reads a binary file and write it back to a different binary file. The difference between the two files would be the byt...

Post has attachment

Post has attachment
Access Apache Spark Web UI when cluster is running on closed port server machines
When you have a Apache spark cluster running on a server were ports are closed you cannot simply access the Spark master web UI by :8080. The solution to this is to use SSH Tunnels. Which is pretty straight forward. Note: You can checkout my blog post on ho...

Post has attachment
Connecting the Real World and World of Computing with a Complete Cloud-Centric IOT Model
IOT or Internet of Things is creating a lot of buzz in the computer world these days. And i also started to like IOT and read about it a bit. I wrote an article about IOT and a cloud centric model that would allow achieve a sustainable IOT architecture In t...

Post has attachment
Simple log analysis with Apache Spark Java API
In this post we will try to redo the sample that we did in my previous post Simple log analysis with Apache Spark , using the Spark JAVA api and since i am more accustomed to maven we will create a simple maven project to accomplish this task. For this post...
Wait while more posts are being loaded