Kristian Lunde

I encountered a rather strange issue on one of MySapients CentOS 5 servers the other day, the server seemed to run fine, without any issues what so ever, but it seem to run very low on memory. In fact it only had 150MB memory available of a total 4GB.

This obviously caused me a bit of a worry, but after some digging around I found the answer, linux is borrowing unused memory to do disk caching. Memory is automatically reassigned when the server needs to allocate memory to other services, in other words there is nothing to worry about.

There is a good site about this describing this functionality in more detail, you can read it at:

http://www.linuxatemyram.com/

Yesterday I installed the openbook plugin for wordpress on this site. My intentions was to build a “My bookshelf” page similar to what Mats Lindh and Trond Huso have on their sites.

I installed the plugin without any problems but the layout was a bit nasty. I don’t know if that was because of the theme I am running on this site or if the openbook markup was messed up. Anyway I rewrote the html and css for the openbooks template on its configuration page and added some custom css to the site to get it working.

This is my openbook template:

This is the css I added:

At last I added this custom html in the “my bookshelf” page:

You can see the result here: My Bookshelf .

This is a great article describing super columns and the general structure of cassandra. A must read for NoSQL newbies.

http://arin.me/code/wtf-is-a-supercolumn-cassandra-data-model

Cassandra is a NoSQL distributed database developed by Facebook, it is built to handle huge amounts of data and to perform CRUD operations quickly. The Cassandra site’s strap line says:

“The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo’s fully distributed design and Bigtable’s ColumnFamily-based data model.

Thrift is also developed by Facebook and is a software framework for service development and is used as an interface to Cassandra. The Thrift page site says:

“Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.”

Both Cassandra and Thrift are apache incubator projects.

Installing Cassandra

1. Download cassandra

Download from: http://incubator.apache.org/cassandra/.

2. Create and set the correct paths in the storage-conf.xml

You can find the storage-conf.xml file in your cassandra root directory /conf

My configuration file settings in storage-conf.xml:

Notice: You have to create all these directories for cassandra to run properly.

3. Set a log directory in the log4j.properties file

This file is found in the same directory as the storage-conf.xml

4. Check that you are running java 6 as default

If you are running an earlier version of java you will have to change your version. Java 6 should already be installed on your mac if you keep your os in sync with the automatic updates from apple. You can change your java version by using the “Java Settings” application located in your /Application/Utilities directory.

5. Starting Cassandra

You should be ready to go now, navigate to the root directory of your cassandra installation and start cassandra by typing:

If you dont see any error messages cassandra is probably running as it should, so it is time to test it out.

Cassandra comes with a CLI interface which allowes you to do simple queries to the database. Notice that the CLI interface is not not as powerful as the thrift interface. You can for instance not execute get queries in Super Columns, those queries will create a java exception.

To test the CLI interface, run the following command from the cassandra root directory:

Inserting values to the keyspace:

Retrieving data from the keyspace:

Installing Thrift

Update: I found this manual after I had installed thrift: http://wiki.apache.org/thrift/ThriftInstallationMacOSX, using this install guide will probably fix the issues I had with compiling thrift.

1. Download Thrift

Download from http://incubator.apache.org/thrift/download/ and extract it.

2. Check that you have installed the following:

• g++ 3.3.5+
• Runtime libraries for lex and yacc might be needed for the compiler.
• boost 1.33.1+ (1.34.0 for building all tests) http://www.boost.org/.

I had to install boost manually:

Notice: the boost installation might take a while, It took about 5 – 10 minutes on my Macbook PRO (2.53GHz).

You can see the full requirements for thrift at http://wiki.apache.org/thrift/ThriftRequirements.

3. Start the installation

This ended up in an error message for me:

To fix this I had to copy my pkg.m4 file from /opt/local/share/aclocal/pkg.m4 to my thrift/aclocal directory.

Navigate to your thrift root directory:

Thanks to http://aaronspotlatch.appspot.com/archive/Jul-2008 and
http://qslack.com/post/thrift-macosx-104 for pointing me in the right direction.

You should now be ready to run make

and

You should now be able to run thrift on your mac.

You should now be ready to build your amazing application with Cassandra if both your installation of Cassandra and Thrift were successful.

I will try to post another blog post shortly on using Cassandra, Thrift and PHP. Stay tuned.

Resources

• The Cassandra Project
• The Cassandra Wiki
• The Thrift Wiki
• http://jetfar.com/installing-cassandra-and-thrift-on-snow-leopard-a-quick-start-guide/
• Setting up a Cassandra cluster
• http://www.onemanclapping.org/2009/11/building-cassandra-thrift-bindings-on.html

Tonight (Friday 06.11.2009) klunde.net was moved from the servetheworld.net server to my new server provider Webfaction.co.uk. The move seemed to go painlessly, even the email accounts seem to work properly.

Klunde.net has been with servetheworld for approx. 5 years now. Webfaction had a much better solution to offer than Servetheworld and it was therefore an easy choice.

Farewell Servetheworld.

I have spent the past few days working with online payment solutions for mysapient. Yesterday I came across a bit of a troublesome issue when setting up recurring payments with paypal. There is no secret that the quality of the paypal documentation varies, so I had some troubles finding any proper information about this issue. Luckily there was a lot of other developers struggling with this same issue.

A quick google search gives you a lot of results about this topic and it can take you a while to get through them all, but at least here is the forum post that helped me getting this solved:

http://www.pdncommunity.com/pdn/board/message?board.id=nvp&view=by_date_ascending&message.id=5463

It seems like the setExpressCheckout needs the following parameters to work properly:

It is also important to add the “desc” field to the createRecurringPaymentsProfile call.

The other day I had a frenzy at amazon and ordered quite a few books I have had on my shopping list. The books are of course all computer and web related. The books I ordered was:

• Building Scalable Web Sites by Cal Henderson
• High Performance MySQL
• JavaScript: The Good Parts by Douglas Crockford
• Don’t Make Me Think!: A Common Sense Approach to Web Usability by Steve Krug
• Cocoa Programming for Mac OS X by Aaron Hillegass

I am halfway through the building scalable web sites and it is really good, even if you are a seasoned web developer I think you can learn quite a bit from it. I have also started to read the MySQL book and can’t wait to learn more about database replication, that is something I really want to find out more about. The “Don’t make me think” book will hopefully assist me on building more user friendly web sites, and hopefully the javascript book(still in the mail) will help me to brush up my js skills.

The Cocoa book was on sale, and you never know when you have to do a bit of programming for a mac

Yesterday I had the “pleasure” of merging two code bases of the same application. The code bases had been developed in two different parts of the world, but shared a common foundation. I got access to a development version of the code base a few weeks ago, and the final delivery of the application was done late last week. No version control system was shared between myself and the contractor which made the merge a bit more cumbersome. Unfortunately I could not wait for the contractor to finish the development before I started to add features and bug fixes to the application; this eventually resulted in two separate versions of the code base.

I was aware that this merge would going to happen from the very start so some precautions were taken before I started my own branch of the source code.

1. I separated all new features out in separate directories and added symbolic links to these in the existing code base. This worked very well and we had no problem at all adding the new features to the final delivery.

2. I tried to be very careful and keep track of all the bug fixes and changes done to the original code.

Trouble

I realized that we would have trouble with the final merge not long after I received the first development version, it was cluttered with bugs and issues which made it impossible to even run it in my development version. To get the application up and running I had to make a bunch of changes to the code.

In addition to the initial bugs I soon realized that the front end of the application (read: html and css) was a complete mess. The site was not browser compliant, nothing validated and it was impossible to go through. These issues would not be resolved by the contractor and since the project was on the clock it needed resolving as quickly as possible. This ended up in a complete rebuild of the front end which modified 300+ files.

1st Attempt – Failure

1. Created a git branch of my development code
2. Added the final code from the contractor to the branch

This ended up in a complete mess, nothing working, a complete mess.

2nd Attempt – Success

When the final delivery from the contractor came through the two code bases was in completely different states, most of the bug fixes I had was still needed.

1. Created a git branch of my code base
3. Found the differences from the initial development version we got and the final delivery

Where dev is the directory of the untouched development version we got access to, and final is the directory of the final delivery. This resulted in a complete list (changes.txt) of files which differed between the two original versions. It also identified files that was obsolete in and new files that was added. An example of the content in the changes.txt can be seen below.

4. Once I had this overview I added all the new files from the final version into my git branch.
5. Updated all the files I knew had not been changed. I had a list of all the core files which had been changed.
6. I manually had to go through all the core files that had been changed and compare them with the files from the final delivery.
7. Remove all old files which only were present in the development delivery.
8. Manual comparison of all the front end files, updating and merging these files by hand. Diff can not be used here since the entire front end has changed, and it would only result in a complete difference, still there might have been changes that I needed to incorporate with the new front end.

I still have approx 150 front end files to compare, it is time consuming and frustrating labor, but it seems to be the only way to do it. I keep testing the application while doing the update and so far all of the changes and updates has been successful.

The positive flip of this is that I get a good overview of the code and understanding of the application when I have to go through much of the code from the final delivery.

I might not have chosen the best solution and I would love to hear your approach if you have done similar things or have an opinion about it.

I decided earlier this week that I should add a Retweet plugin to my blog; so I googled “Retweet wordpress”, which gave me two good results the Retweet plugin and the TweetMeme plugin. I thought I would install and try both of them before I decided which one I should go for.

Retweet

I installed the Retweet plugin first, this had to be installed, I had to add a custom field to each post and I had to add some PHP code to one of the template files. Once this was done I had to style the link myself.

TweetMeme

The TweetMeme plugin however had to be installed and then it needed to be configured from a separate settings page. Once that was done it was added to all of my posts and it even showed how many retweets each post had.

Conclusion

When I need a plugin for wordpress I really do not want to do more than install it and do some configuration on a settings page. It is way to much hassle to modify template files and adding custom fields to the posts.

Guess which plugin I chose

It has been a couple of weeks since my last update and a lot has happened since then. I was very much in doubt of what I should do in my last blog post. I had the choice of starting up my own company or to join a company. I chose to go for the latter one and join a company called mySapient, I have already spent a week with them and I am really looking forward to work with these guys. mySapient is a relatively new company, which are developing 3D online games, I am however not hired to do 3D development but rather to work with their web applications.  We are at the moment 4 game developers and me, doing the web stuff. It is great fun and it is really interesting to learn about online gaming and 3D games.

So, why did I choose to go for mySapient? I spoke so warmly about freelancing and setting up my own company. Well, it was no easy choice to make, I had met a some really good freelancers which I could have had the chance to work with, they had this spark and enthusiasm, and I was really tempted to join forces with them. Still there are so many uncertainties running your own company or working freelance, and having a family to support I chose to go down the safe route with a set income.  It was however not just the safety of working in mySapient that lead me to them, all the employees I met had the energy and spark which I am looking for. I also find it very interesting to work with people in a very different sector of computing than myself. I learn so much, and look forward to learn about gaming and their development processes. We have already discovered several practices that are common both for the web world and the gaming world, interesting stuff.

I would like to thank all of you for your support and assistance, it has been highly appreciated.