OS161 on Ubuntu 10.10

I was trying to get Ubuntu 10.10 to work with OS161. It was a big struggle but the main trick of getting it to work was having gcc 3.4. To install gcc 3.4 you would use the following command

wget http://ge.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-3.4/libstdc++6-dev_3.4.6-6ubuntu3_i386.deb &&
wget http://ge.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-3.4/gcc-3.4_3.4.6-8ubuntu2_i386.deb &&
wget http://ge.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-3.4/gcc-3.4-base_3.4.6-8ubuntu2_i386.deb &&
sudo dpkg -i *.deb &&

You would then probably like to change your default gcc to 3.4 by doing the following command

cd /usr/bin/
sudo ln -s gcc-3.4 gcc

Then you can get the source of os161 and follow the nice step by step instructions by clicking here.

Hope this lets you spent less time configuring and more time hacking.

Express with Jade on node.js

Over the weekend, I was playing around with node.js – a javascript implementation of an event I/O handler. Node.js is an event based framework that allows developer to easily build scalable network programs. The framework runs on the javascript V8 engine and was influenced by by Ruby’s Event Machine and Python’s Twisted. Node.js breaks the barrier between server side language and client side language.

Node.js is very easy to use, node.js came with a neat package manager like python’s easy_install and ruby’s gem. The packages that I played around with was express – a web framework, and Jade– a template engine. Below, I will show you how to combine the two packages Jade and Express to make a web app in 5 minutes.
first, make sure you have node.js, npm, express and jade installed on your machine.

Then create a file (call it example.js) using any editor you wish, and paste the code below.

example.js

var express = require('express')
var jade = require('jade')
app = express.createServer();
app.get('/',function(req,res){
var local_var = "I am a local var";
jade.renderFile('index.jade', {locals:{local_var:local_var}} ,function(err,html){
res.send(html);
});
});
app.listen();
console.log("Express server is started on port %s", app.address().port);
The first two lines imports the two packages that we are going to use and gives them each a name space.

app = express.createServer();
This allows us to have a handle on a concrete server

app.get('/',function(req,res){ .....});
defines a call back on our server. Every time the url path ‘/’ of our server is queried,
the function(req,res){….} will be called.

jade.renderFile('index.jade', {locals:{local_var:local_var}} ,function(err,html){
res.send(html);
});
renders the file ‘index.jade’ and pass the local variables defined in the hash {locals:{local_var:local_var}} to the template for proper variable substitution, in our case local_var (defined as “I am a local var”).
res.send(html) then sends the html that was rendered back to the client.

Our template file will look like the below. Paste it into a file in the same directory as example.js, and name this file index.jade.

index.jade
p= local_var
The above basically says that the paragraph will contain the value of what ever is defined in the variable ‘local_var’.

runnning the app

$ node example.js
Express server is started on port <port>


testing if it works
curl localhost:<port>

the above should give you

<p>I am a local var</p>

 

CASCON 2010

CASCON 2010 has been a fruitful one, it was there which I had my abstract published at ACM, I chaired for a conference for the first time. I had prepare weeks on my topic – which was clearly not enough and I was still as nervous as hell during the presentation.  My presentation was on cache-oblivious algorithms. “A cache-oblivious algorithm (or cache-transcendent algorithm) is an algorithm designed to use the CPU cache without having the size of the cache (or the length of the cache lines, etcetera) as an explicit parameter.”  as described by wikipedia. To present a topic, I had to understood the cache-oblivious algos completely, and it was not easy at all, since cache-oblivious algorithms are fairly new concepts. I was learning most of the material off this paper and 2 lectures videos from MIT, some of the algorithms were trivial, while others tooks days to understand the basic. The evaluation of my presentation not all positive, some people thought it was too deep into the algorithms. I hope to learn from this experience and improve my presentation skill for future workshops; for whatever conference it might be.

Using finger with awk, finding out lab account info of your teammates in case you forgot

On one evening during my third year at U of T, I was working on an CSC343(intro to databases) assignment with my partner. When the assignment was finally completed I was overjoyed, with the assignment due in less than an hour, my partner put me in charge of submitting the assignment as he had to leave immediately and run some errands. When I was ready to submit the assignment electronically on a online app, the app asked me for my partner’s lab account name. I was in total panick, because I forgot his account name! I took a DEEP breath and tried to call my partner on his cell phone, but he didn’t pick up. “FML, my partner’s gonna kille me” I thought . I then messaged a friend and asked for consultation and possibly prayers. Instead of prayers however, she gave me a bash shell script that uses for loops, finger, /etc/pwd and grep. The script was magix, because it located my partner’s account info, and to this day, I still think she is l33t.

Now a year later, I wondered if there were a simpler way to find out the account information of a classmate by name (or part of his/her name) without using shell scripts. And here is what I came up with

export $REGEX=<name>
cat /etc/passwd | awk -F : '{print $1}' |xargs finger -l|awk '/Name: [a-zA-Z]*'$REGEX'[a-zA-Z]*/{print $0;getline;print}'

The command might look daunting at first, but if you break it down, it is really simple.
cat /etc/passwd displays various information of all the user accounts on system, it usually looks like this

jack:x:1000:1000:jack,,,:/home/jack:/bin/bash
riak:x:114:123:Riak Data Store,,,:/var/lib/riak:/bin/bash
mysql:x:115:124:MySQL Server,,,:/nonexistent:/bin/false

for more information on /etc/passwod visit here

awk -F : {print $1}

The above command basically says that for each line, print the first ($1) element delimited by the symbol “:”. In our case it will display all of the user account name.

jack
riak
mysql

The above would be an example of user accounts.

xargs finger -l

finger allows us to find more information on a given account name, including the user’s name. This is pretty essential.

awk '/Name: [a-zA-Z]*'$REGEX'[a-zA-Z]*/{print $0;getline;print}'

prints the line that contain the values that are defined by $REGEX.

So this little trick command allows the user to find account information based on the user’s registered user name, I hope this is useful to you, and I hope my WTF moment doesn’t happen to you.

Linux networking/ip notes

Networking can be a pain, and in linux there are several config files that I must remember where they are and what they do. Note Ubuntu is a bit different.

/etc/sysconfig/network
For various configuration like hostname, gateway, and/or the type of protocol to use.
The below is a summary of possible field and their values, thanks to this faq

NETWORKING=answer, where answer is yes or no -Configure networking or not to configure networking.
FORWARD_IPV4=answer, where answer is yes or no -Perform IP forwarding or not to perform IP forwarding.
HOSTNAME=hostname, where hostname is the hostname of your server.
GATEWAY=gwip, where gwip is the IP address of the remote network gateway -if available.
GATEWAYDEV=gwdev, where gwdev is the device name eth# you use to access the remote gateway.
an example network file can be

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=jack-machine
GATEWAY=10.11.0.1


/etc/sysconfig/network-scripts/ifcfg-eth0
This file contains the configuration for a specific network device (in our case it is ifcfg-eth0)

DEVICE=devicename, where devicename is the name of the physical network device.
IPADDR=ipaddr, where ipaddr is the IP address.
NETMASK=netmask, where netmask is the netmask IP value.
NETWORK=network, where network is the network IP address.
BROADCAST=broadcast, where broadcast is the broadcast IP address.
ONBOOT=answer, where answer is yes or no. Do the interface need to be active or inactive at boot time.
BOOTPROTO=proto, where proto is one of the following :
none - No boot-time protocol should be used.
bootp - The bootp now pump protocol should be used.
dhcp - The dhcp protocol should be used.
USERCTL=answer, where answer is one of the following:
yes - Non-root users are allowed to control this device.
no - Only the super-user root is allowed to control this device.

an example network-script file can be

DEVICE=eth0
IPADDR=208.164.186.1
NETMASK=255.255.255.0
NETWORK=208.164.186.0
BROADCAST=208.164.186.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

/etc/resolv.conf
Contains information on the dns that this machine is using
An example file would be

search example.com
;the name server that the resolver should query.
nameserver 10.11.0.5

A special note on the search parameter; it is the list of domains that we will search by default. for example if we nslookup on svn, it will actually lookup for svn.example.com. if we didn’t have the search rule below, it would just search for svn and give a not found response.

Ubuntu 10.10 Checking battery state…

A few days ago I had to reinstall by ubuntu on my T-400. After the re installation, it required me to upgrade my video driver using a proprietary software. After I restarted my computer, the machine gets stuck at the boot stage, displaying the message “Checking battery state….”

I attempted at searching on Google for ways to remedy this my problem, but no luck. I was frustrated and figured that I might be able to figure something out if I had a terminal. I went to tty1 by pressing ctrl + alt + F1, I then removed my graphics configuration by issuing the following command
sudo rm /etc/X11/xorg.conf
I then issued a reboot
sudo reboot
and everything was fine and I could login into my machine normally.

Using awk to remove python2.7

Today I wanted to remove python 2.7, and I didn’t really want to do it manually by doing sudo rm on individual python2.7 file/folder. So instead I used whereis, xargs, and grep to help me. I used the following command 

whereis python | awk 'BEGIN{FS=" "}{for(i=1;i<NF;i++) print $i}' | grep python2.7 |xargs sudo rm -fr

Lets walk this through step by step.

>whereis python

Finds all python files that are in the standard directory, for example in my machine it produced
python2: /usr/bin/python2.7 /usr/bin/python2.6-config /usr/bin/python2.6 /etc/python2.7 /etc/python2.6 /usr/lib/python2.7 /usr/lib/python2.6 /usr/local/bin/python2.7 /usr/local/bin/python2.7-config /usr/local/lib/python2.7

We want to split this line by white spaces; this is where the awk expression comes along.
awk 'BEGIN{FS=" "}{for(i=1;i<NF;i++) print $i}

The above uses awk to split white space delimited strings into seperate lines, the FS is a special awk’s built-in variable that determines the delimited string, in our case it is a space. The NF stands for the total number of elements.
I got the following on my machine.

/usr/bin/python2.7
/usr/bin/python2.6-config
/usr/bin/python2.6
/etc/python2.7
/etc/python2.6
/usr/lib/python2.7
/usr/lib/python2.6
/usr/local/bin/python2.7
/usr/local/bin/python2.7-config
/usr/local/lib/python2.7
/usr/local/lib/python2.6

then with some help of grep and xargs I was able to successfully remove python2.7 from my system with precision.

grep python2.7 |xargs sudo rm -fr

This is only a little taste of awk’s power, it fully supports regular expression and makes command line scripting a breeze.
For example if you wanted to kill all processes that contains the word java, then you can just do

ps ax | grep java| awk '/java/{print $2;}' |xargs kill -9

especially useful for killing multiple java processes that hangs.
I for one have just begin to use awk, and I see it as a essential tool for all scripters in linux environment.

vim regex

Ever wonder just how useful vim REALLY is? Well, the substitution command in VIM makes it really useful, it supports all sorts of regex, and makes editing very easy and fast assuming that the user knows what he/she is doing. Vim is one of those tools that takes more thinking than doing at the beginning. For example it took me 30 minutes to surround lines from 24 to 35 with =>” and “.  To give you a better picture of my problem look at the below example.

line containing contents foo bar will become =>”foo bar” for arbitrary number of lines.

There are several ways of doing this. One of the ways is by using the Visual block mode to select lines 24 to 35 and press I to insert =>” at the beginning of the lines. Then again select the lines and press A to append “ to the end of the lines. But I didn’t think this was a satisfactory solution to my problem. What if there were 50 lines that needs to be encapsulated instead of 11 lines? (visual block selection can be a pain sometimes)

So we can use subsitution command :

:24,35s/^.*$/=>"&"/

to solve our problem.

What the command says is that from lines 24 to 35 we are going to substitute the from beginning of the line to the end of the line (denote this as X), with the X but with =>” before it and “ after it. The & symbol represents the contents which we just matched (in this case it is X).

Here is another useful substitution command you might use

:23s/[a-z]*_*[a-z]*\s*/=>"&",\r/g

The above command takes a line (i.e line 23) of multiple words for example:

23: foo bar lulz cat

to

=>"foo",

=>"bar",

=>"lulz",

=>"cats",

this is specifically useful if you want to construct a large php array of multiple keys and values. What I do is I first type out the words that will be the values then use the above command, then type in the corresponding key for each value. e.g.

array(

1=>"foo",

2=>"bar",

3=>"lulz",

4=>"cats"

);

I hope the above will help you constructing large php arrays as it did for me.

HBase

Over the weekend I was fooling around with HBase. HBase is a distributed database, you would have one master node (name node) set up and multiple regional nodes set up to offer high availability, and performance; the downside is that HBase is not a relational database (that means no foreign keys and the other nice features that comes databases like mySQL). Hbase has its own unique way of storing data that is not too different from google’s BigTable. To understand how data is stored in HBase one can refer to http://wiki.apache.org/hadoop/Hbase/HbaseArchitecture. (Although this is kind of out of date, you can still get a general picture).

HBase requires HDFS to be set up, the instruction for the whole process can be found at http://wiki.apache.org/hadoop/QuickStart.

HBase came with several interfaces: shell, restful, thrift and java. Below I will describe my experiences with some of the interfaces and my opinion on others.

Shell

HBase provides a JRUBY IRB-based shell that is very similar to the standard SQL shell. Starting the shell is simple, go to the bin directory of your Hbase, and do ./hbase shell. The shell is really well designed and allows a user to do table and row operations with minimal effort. To create a table in the shell one would only have to do:

create 'table1','f1','f2','f3'

The above command will create a table with the name ‘table1’ and with the column families ‘f1’, ‘f2’, and ‘f3’.

To insert data into hbase

put 'table1','row1','f1:col9', 'value', 'timestamp'

The above command will create an row with the key ‘row1’ in the table ‘table1’. ‘f1:col9’ is the specific column which the data “value” is to be put.  As you can see this is a lot easier than the standard sql statements to create tables and insert data.

Pro: Easy to use

Cons: None

For a more detailed guide on the HBase shell visit http://wiki.apache.org/hadoop/Hbase/Shell.

Rest

The restful interface that HBase provides is quite nice in theory though it is still in alpha development stage. The HBase people calls the restful interface the “Stargate” interface. The Stargate interface runs on the Jetty server by default, but they provide a .war that allows the user to switch from Jetty into the apache tomcat server. There are a lot of discussions on tomcat vs Jetty online. The main lesson I got out of those discussions was that Jetty is more light weight than tomcat, but tomcat provides performance advantages (One would need to use JProfiler to confirm this). A lot of people tend to prefer restful interfaces because they are so easy to access and use, in fact many of the hbase adapter libraries will use restful interface bindings to communicate with HBase.

If you want the current restful docs specific to your version you’d have to go to http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/stargate/package-summary.html#package_description

replace the 0.20.6 with the version of your HBase. (to find the version of your hbase, simply go to the shell and type ‘version’).

Stargate can accept and return three types of data (depending on the header): xml, json, and protobufs(binary). It is good to mention that xml is the most supported data type. Another thing about the restful interface is that the columns, rows, and data are encoded in base64, so for every get and put you have to know what to encode and decode appropriately.

There are tons of posts on the HBase mailing list which talks about the (the lack of) performance that the restful interface provides. Some say that the restful interface is suitable for services which have many clients posting and reading data at a low rate.

In general I am not too happy with the restful interface that HBase provides.

Pro: Ease of access

Cons: Everything else

Thrift

I admit that I haven’t used thrift with HBase and some says that the performance is terrific. I just didn’t like the fact that I’d have to install 64mb of C++ libraries just to communicate with HBase.

Pros: Performance

Cons: Client needs to install thrift

Java

If you are doing development in Java, then kudos to you, this is probably the interface that you want to use. It has the best performance compared to all other interfaces and is the most supported interface.

Pros: Speed, community support

Cons: Java only

Thanks for reading my long rambling, the material of the posts  are only of my opinion, experience and knowledge, please correct me if I am wrong.

Hello world!

Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!