Kristian Lunde

www.klunde.net

Installing Cassandra and Thrift on OSX

with 4 comments

Note that this installation description was written for Cassandra 0.5 and may not be correct for the current releases.

Cassandra is a NoSQL distributed database developed by Facebook, it is built to handle huge amounts of data and to perform CRUD operations quickly. The Cassandra site’s strap line says:

“The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo’s fully distributed design and Bigtable’s ColumnFamily-based data model.

Thrift is also developed by Facebook and is a software framework for service development and is used as an interface to Cassandra. The Thrift page site says:

“Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.”

Both Cassandra and Thrift are apache incubator projects.

Installing Cassandra

1. Download cassandra

Download from: http://incubator.apache.org/cassandra/.

2. Create and set the correct paths in the storage-conf.xml

You can find the storage-conf.xml file in your cassandra root directory /conf

My configuration file settings in storage-conf.xml:

  1.   <CommitLogDirectory>/Users/kristianlunde/tmp/cassandra-log/commitlog</CommitLogDirectory>
  2.   <DataFileDirectories>
  3.       <DataFileDirectory>/Users/kristianlunde/workspaces/mysapient/cassandra/data</DataFileDirectory>
  4.   </DataFileDirectories>
  5.   <CalloutLocation>/Users/kristianlunde/workspaces/mysapient/cassandra/callouts</CalloutLocation>
  6.   <BootstrapFileDirectory>/Users/kristianlunde/workspaces/mysapient/cassandra/bootstrap</BootstrapFileDirectory>
  7.   <StagingFileDirectory>/Users/kristianlunde/workspaces/mysapient/cassandra/staging</StagingFileDirectory>

Notice: You have to create all these directories for cassandra to run properly.

3. Set a log directory in the log4j.properties file

This file is found in the same directory as the storage-conf.xml

4. Check that you are running java 6 as default

  1.  java -version

If you are running an earlier version of java you will have to change your version. Java 6 should already be installed on your mac if you keep your os in sync with the automatic updates from apple. You can change your java version by using the “Java Settings” application located in your /Application/Utilities directory.

5. Starting Cassandra

You should be ready to go now, navigate to the root directory of your cassandra installation and start cassandra by typing:

  1. bin/cassandra -f

If you dont see any error messages cassandra is probably running as it should, so it is time to test it out.

Cassandra comes with a CLI interface which allowes you to do simple queries to the database. Notice that the CLI interface is not not as powerful as the thrift interface. You can for instance not execute get queries in Super Columns, those queries will create a java exception.

To test the CLI interface, run the following command from the cassandra root directory:

  1. ./bin/cassandra-cli –host localhost –port 9160

Inserting values to the keyspace:

  1.  cassandra> set Keyspace1.Standard1['blog-post']['name'] = 'Installing Cassandra and Thrift OSX'
  2.  Value inserted.
  3.  cassandra> set Keyspace1.Standard1['blog-post']['author'] = 'Kristian Lunde'
  4.  Value inserted.

Retrieving data from the keyspace:

cassandra> get Keyspace1.Standard1['blog-post']
  1.    (column=name, value=Installing Cassandra and Thrift OSX; timestamp=1258748376097)
  2.    (column=author, value=Kristian Lunde; timestamp=1258748405486)
  3.  Returned 2 rows.
  4.  cassandra>

Installing Thrift

Update: I found this manual after I had installed thrift: http://wiki.apache.org/thrift/ThriftInstallationMacOSX, using this install guide will probably fix the issues I had with compiling thrift.

1. Download Thrift

Download from http://incubator.apache.org/thrift/download/ and extract it.

2. Check that you have installed the following:

  • g++ 3.3.5+
  • Runtime libraries for lex and yacc might be needed for the compiler.
  • boost 1.33.1+ (1.34.0 for building all tests) http://www.boost.org/.

I had to install boost manually:

  1. sudo port install boost

Notice: the boost installation might take a while, It took about 5 – 10 minutes on my Macbook PRO (2.53GHz).

You can see the full requirements for thrift at http://wiki.apache.org/thrift/ThriftRequirements.

3. Start the installation

  1.  kristian-lundes-macbook-pro:thrift kristianlunde$ ./bootstrap.sh
  2.  configure.ac:26: installing `./missing'
  3. configure.ac:26: installing `./install-sh'
  4.  compiler/cpp/Makefile.am: installing `./depcomp'
  5. configure.ac: installing `./ylwrap'
  6.  kristian-lundes-macbook-pro:thrift
  1. ./configure

This ended up in an error message for me:

  1.  ./configure: line 20722: syntax error near unexpected token `MONO,'
  2. ./configure: line 20722: `  PKG_CHECK_MODULES(MONO, mono >= 2.0.0, net_3_5=yes, net_3_5=no)'

To fix this I had to copy my pkg.m4 file from /opt/local/share/aclocal/pkg.m4 to my thrift/aclocal directory.

Navigate to your thrift root directory:

  1.  cp /opt/local/share/aclocal/pkg.m4 aclocal

Thanks to http://aaronspotlatch.appspot.com/archive/Jul-2008 and
http://qslack.com/post/thrift-macosx-104 for pointing me in the right direction.

You should now be ready to run make

  1. make

and

  1. sudo make install

You should now be able to run thrift on your mac.

  1. thrift

You should now be ready to build your amazing application with Cassandra if both your installation of Cassandra and Thrift were successful.

I will try to post another blog post shortly on using Cassandra, Thrift and PHP. Stay tuned.

Resources

Written by Kristian Lunde

November 20th, 2009 at 9:39 pm

Posted in Mac,NoSQL,web

Tagged with ,

Klunde.net moved server

with 2 comments

Tonight (Friday 06.11.2009) klunde.net was moved from the servetheworld.net server to my new server provider Webfaction.co.uk. The move seemed to go painlessly, even the email accounts seem to work properly.

Klunde.net has been with servetheworld for approx. 5 years now. Webfaction had a much better solution to offer than Servetheworld and it was therefore an easy choice.

Farewell Servetheworld.

Written by Kristian Lunde

November 6th, 2009 at 11:39 pm

Posted in Misc

Recurring payments with Paypal and PHP

without comments

I have spent the past few days working with online payment solutions for mysapient. Yesterday I came across a bit of a troublesome issue when setting up recurring payments with paypal. There is no secret that the quality of the paypal documentation varies, so I had some troubles finding any proper information about this issue. Luckily there was a lot of other developers struggling with this same issue.

A quick google search gives you a lot of results about this topic and it can take you a while to get through them all, but at least here is the forum post that helped me getting this solved:

http://www.pdncommunity.com/pdn/board/message?board.id=nvp&view=by_date_ascending&message.id=5463

It seems like the setExpressCheckout needs the following parameters to work properly:

  1. [L_BILLINGTYPE0] => RecurringPayments
  2. [L_BILLINGAGREEMENTDESCRIPTION0] => Basic Plan

It is also important to add the “desc” field to the createRecurringPaymentsProfile call.

  1. [DESC] => Basic Plan

Written by Kristian Lunde

October 2nd, 2009 at 9:59 am

Posted in Misc

Amazon Frenzy

without comments

The other day I had a frenzy at amazon and ordered quite a few books I have had on my shopping list. The books are of course all computer and web related. The books I ordered was:

I am halfway through the building scalable web sites and it is really good, even if you are a seasoned web developer I think you can learn quite a bit from it. I have also started to read the MySQL book and can’t wait to learn more about database replication, that is something I really want to find out more about. The “Don’t make me think” book will hopefully assist me on building more user friendly web sites, and hopefully the javascript book(still in the mail) will help me to brush up my js skills.

The Cocoa book was on sale, and you never know when you have to do a bit of programming for a mac :P

Written by Kristian Lunde

September 24th, 2009 at 8:29 pm

Merging code bases

without comments

Yesterday I had the “pleasure” of merging two code bases of the same application. The code bases had been developed in two different parts of the world, but shared a common foundation. I got access to a development version of the code base a few weeks ago, and the final delivery of the application was done late last week. No version control system was shared between myself and the contractor which made the merge a bit more cumbersome. Unfortunately I could not wait for the contractor to finish the development before I started to add features and bug fixes to the application; this eventually resulted in two separate versions of the code base.

I was aware that this merge would going to happen from the very start so some precautions were taken before I started my own branch of the source code.

1. I separated all new features out in separate directories and added symbolic links to these in the existing code base. This worked very well and we had no problem at all adding the new features to the final delivery.

2. I tried to be very careful and keep track of all the bug fixes and changes done to the original code.

Trouble

I realized that we would have trouble with the final merge not long after I received the first development version, it was cluttered with bugs and issues which made it impossible to even run it in my development version. To get the application up and running I had to make a bunch of changes to the code.

In addition to the initial bugs I soon realized that the front end of the application (read: html and css) was a complete mess. The site was not browser compliant, nothing validated and it was impossible to go through. These issues would not be resolved by the contractor and since the project was on the clock it needed resolving as quickly as possible. This ended up in a complete rebuild of the front end which modified 300+ files.

1st Attempt – Failure

1. Created a git branch of my development code
2. Added the final code from the contractor to the branch

This ended up in a complete mess, nothing working, a complete mess.

2nd Attempt – Success

When the final delivery from the contractor came through the two code bases was in completely different states, most of the bug fixes I had was still needed.

1. Created a git branch of my code base
3. Found the differences from the initial development version we got and the final delivery

  1. diff -qr dev final | grep -v -e 'DS_Store' -e 'Thumbs' | sort > changes.txt

Where dev is the directory of the untouched development version we got access to, and final is the directory of the final delivery. This resulted in a complete list (changes.txt) of files which differed between the two original versions. It also identified files that was obsolete in and new files that was added. An example of the content in the changes.txt can be seen below.

  1. Files dev/view/file-1.php and final/view/file-1.php differ
  2. Files dev/view/file-2.php and final/view/file-2.php differ
  3. Only in dev/css: css-1.css
  4. Only in dev/css: css-2.css
  5. Only in dev/css: css-3.css
  6. Only in final/css: css.css
  7. Only in final/: file-a.php

4. Once I had this overview I added all the new files from the final version into my git branch.
5. Updated all the files I knew had not been changed. I had a list of all the core files which had been changed.
6. I manually had to go through all the core files that had been changed and compare them with the files from the final delivery.
7. Remove all old files which only were present in the development delivery.
8. Manual comparison of all the front end files, updating and merging these files by hand. Diff can not be used here since the entire front end has changed, and it would only result in a complete difference, still there might have been changes that I needed to incorporate with the new front end.

I still have approx 150 front end files to compare, it is time consuming and frustrating labor, but it seems to be the only way to do it. I keep testing the application while doing the update and so far all of the changes and updates has been successful.

The positive flip of this is that I get a good overview of the code and understanding of the application when I have to go through much of the code from the final delivery.

I might not have chosen the best solution and I would love to hear your approach if you have done similar things or have an opinion about it.

Written by Kristian Lunde

September 16th, 2009 at 7:22 am

Retweet vs TweetMeme WordPress plugins

without comments

I decided earlier this week that I should add a Retweet plugin to my blog; so I googled “Retweet wordpress”, which gave me two good results the Retweet plugin and the TweetMeme plugin. I thought I would install and try both of them before I decided which one I should go for.

Retweet

I installed the Retweet plugin first, this had to be installed, I had to add a custom field to each post and I had to add some PHP code to one of the template files. Once this was done I had to style the link myself.

TweetMeme

The TweetMeme plugin however had to be installed and then it needed to be configured from a separate settings page. Once that was done it was added to all of my posts and it even showed how many retweets each post had.

Conclusion

When I need a plugin for wordpress I really do not want to do more than install it and do some configuration on a settings page. It is way to much hassle to modify template files and adding custom fields to the posts.

Guess which plugin I chose :P

Written by Kristian Lunde

September 10th, 2009 at 7:15 am

Posted in web,Wordpress

Hello mySapient

with 2 comments

It has been a couple of weeks since my last update and a lot has happened since then. I was very much in doubt of what I should do in my last blog post. I had the choice of starting up my own company or to join a company. I chose to go for the latter one and join a company called mySapient, I have already spent a week with them and I am really looking forward to work with these guys. mySapient is a relatively new company, which are developing 3D online games, I am however not hired to do 3D development but rather to work with their web applications.  We are at the moment 4 game developers and me, doing the web stuff. It is great fun and it is really interesting to learn about online gaming and 3D games.

So, why did I choose to go for mySapient? I spoke so warmly about freelancing and setting up my own company. Well, it was no easy choice to make, I had met a some really good freelancers which I could have had the chance to work with, they had this spark and enthusiasm, and I was really tempted to join forces with them. Still there are so many uncertainties running your own company or working freelance, and having a family to support I chose to go down the safe route with a set income.  It was however not just the safety of working in mySapient that lead me to them, all the employees I met had the energy and spark which I am looking for. I also find it very interesting to work with people in a very different sector of computing than myself. I learn so much, and look forward to learn about gaming and their development processes. We have already discovered several practices that are common both for the web world and the gaming world, interesting stuff.

I would like to thank all of you for your support and assistance, it has been highly appreciated.

Written by Kristian Lunde

August 8th, 2009 at 2:46 pm

Posted in Real Life

Moving on – looking into the future

without comments

I resigned from my position at Orange Bus this week as you probably already know, and I thought I would keep you updated on what I am up to now. This has been a very hectic week, I have had several interviews and talked to a lot of interesting people. I have pretty much been running in meetings, talking, sending and answering emails and using twitter. Twitter has proven to be a brilliant tool I am surprised by all the response I got through twitter when I posted the message about my resignation.

At the moment I am looking into two possible permanent jobs, both seem interesting but I am not at all sure that this is the way I want to go. What I have discovered the past few days is that there is a lot of good ideas out there, these ideas need to be developed, they need technical assistance and a lot of them need more than just the technical assistance. They need partnerships, they need to team up with someone and work together.

I have also seen that the freelancers in Newcastle are piled down in work, they have so much to do that they don’t know how to get it all done in time. These things have caught my attention, all the freelancers I have spoken to and all the people with the great ideas have this spark, glow and enthusiasm I have talked so much about. They love what they are doing and they really believe in their ideas or products. They are the kind of people I would like to work with, and I am at the moment looking into starting up my own company. I know there is enough work out there, and I believe that I can make a difference and help and assist some of these people.

Walking a path without knowing where it ends up is very interesting and I am sure this will lead me to the right place. I look forward to the coming weeks and I am loving every minute of this.

Written by Kristian Lunde

July 25th, 2009 at 7:14 pm

Posted in Real Life

Going our separate ways

with 3 comments

Yesterday (21 July 2009) I chose to resign from my position in Orange Bus. It was a very difficult decision to take, because I like the company and I like the people working there. I do however believe I took the right decision, it was something I had been thinking about for quite a while.

The most frequent question I have been asked the last 24 hours is:

Why did you resign from Orange Bus?…. I thought Orange Bus was a good place to work.

First of all I would like to say that YES!! Orange Bus is a good place to work. I have nothing bad to say about Orange Bus, they are a innovative and experienced web agency, have a brilliant team, and provide their customers with smart and solid web solutions.

My reason for resigning is much more personal, I have the last 4 months been seeing my enthusiasm and “spark” disappear bit by bit. This has scared me tremendously, because without my “spark” without that enthusiasm I am no longer a good developer and my creativity are not what it should be without it the “spark”. I spent a while trying to figure out why this was happening, and it took me some time before I found the problem.

I did not do what I am best at and likes to do the most, designing the architectural layout of software and develop in a way I enjoy. Drupal which is the main development platform in Orange Bus is a very good CMS (Content Management System) and I hugely respect it and the people behind it. I do however feel that this is not a platform I am very comfortable with. This is purely a personal preference.

To resign from a job without having a new job might seem idiotic in the current economical climate, but I honestly believe this was the best to do both for myself and for Orange Bus, neither of us would benefit from me staying. I am confident that I will find a new job and I look forward to talking it on. I have already had a few interviews and talked with some interesting people. I have a few ideas I would really like to see put to life, so I might set up my own company back in Norway. If and when that happens I will make sure to let you know. For now I will stay in Newcastle and see what happens.

There is no hard feelings between myself and Orange Bus, I have had a very good time there and I have learned a lot, I wish them all the best in the future.

Written by Kristian Lunde

July 22nd, 2009 at 5:30 pm

Posted in Real Life

Javascript snippets

with one comment

Lately I have been doing some javascript development and I have written a couple of functions which I find really useful. I almost find the code snippets to be used on all sites I work on so I thought I would share them with you. Both the snippets are written using the jQuery library.

Opening urls in a separate window/tab in a xhtml strict environment

The target attribute is not a valid attribute in xhtml strict and your site will fail to validate if you use it, so to get around this you can use replace your target attribute with the rel attribute. Adding the rel=”external” attribute on your anchors that points to an external source and adding some javascript solves the problem.

  1. <a href="http://www.klunde.net" rel="external" title="Klunde.net">Kristian Lunde</a>
  1. function externalLinks() {
  2.         $("a[rel='external']").each(function() {
  3.   $(this).attr('target', '_blank');
  4.         });
  5. };

The code here is pretty self explaining, but to describe it briefly, the javascript function must be invoked once the DOM is loaded (see code further down on the page). The function loops through all anchors that have the rel=”external” attribute and add the attribute target=”_blank”. I guess this is a bit of a hack, but it works and it keeps the clients happy ;)

Creating spam proof mailto: anchors

It is a well known fact that if you add your email address on a site in a plain mailto anchor you will be flooded by spam after a while. You can easily avoid this by adding a little bit of javascript on your site.

It works simply by printing out the email address as plain text replacing the @ with a ” AT ” and add the email addresses in a div or which ever element you prefer and add a set class to that tag. Then you run a small snippet of javascript that find all the elements with that class, I have chosen the class name “email”. The script replaces the content in all the elements with the email class with the proper html anchor. This can not be picked up by web crawlers and it displays the links properly for the user. All browsers without javascript support / enabled will of course only see the post AT somesite.com.

  1. <div class="email">post AT  somesite.com</div>
  1. function createMailTo() {
  2.         var c_email_field = $('.email');
  3.  if($(c_email_field).length > 0) {
  4.   $(c_email_field).each(function()
  5.   {
  6.    var email = $(this).html().replace(' AT ', '@');
  7.    $(this).html('<a href="mailto:' + email + '" rel="nofollow" title="' + email + '">' + email + '</a>');
  8.   });
  9.  };
  10. };

Running this script would transform the code into:

  1. <a href="mailto:post@somesite.com" rel="nofollow" title="post@somesite.com">post@somesite.com</a>

Putting it all together

I have but together a simple example to illustrate the functions in action with all code and functionality. You should be able to copy this code save it as a .html file and run it in your browser.

  1. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
  2. <html>
  3. <head>
  4.   <title>Javascript snippets</title>
  5. </head>
  6. <body>
  7.  
  8. <a href="http://www.klunde.net" rel="external" title="Klunde.net">Kristian Lunde</a>
  9. <div class="email">post AT somesite.com</div>
  10.  
  11. <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
  12. <script type="text/javascript">
  13.  
  14. function externalLinks() {
  15.         $("a[rel='external']").each(function() {
  16.   $(this).attr('target', '_blank');
  17.         });
  18. };
  19.  
  20. function createMailTo() {
  21.         var c_email_field = $('.email');
  22.  if($(c_email_field).length > 0) {
  23.   $(c_email_field).each(function()
  24.   {
  25.    var email = $(this).html().replace(' AT ', '@');
  26.    $(this).html('<a href="mailto:' + email + '" rel="nofollow" title="' + email + '">' + email + '</a>');
  27.   });
  28.  };
  29. };
  30.  
  31. $(function() {
  32.         externalLinks();
  33.         createMailTo();
  34. });
  35. </script>
  36. </body>
  37. </html>

Written by Kristian Lunde

June 18th, 2009 at 10:55 pm

Posted in web,web development

Tagged with , ,

Get Adobe Flash player