Getting the Cache Right: Part 1
So recently a client called up with regards to their website. They were worried about a increase of traffic to their site and wanted some reassurance that their site would still work. One of the action points was to first work out the bottle necks in the site. The site is an openCMS tomcat application using MySQL as its database. After tweaking both tomcat and MySQL we managed to get the transaction rate up to around 33 a second with a average time of just over 2 seconds. This was not good. Enter Varnish! Varnish is a caching daemon which sits in front of apache and caches pages. I have not used it much but from what I see it is POWERFUL!
To set up varnish and apache on the same box I have to tweet the apache config a little. To start with I had to make it bind to localhost port 8080 to allow varnish to run on port 80. To do this I changed a little line in the httpd.conf (for Red Hat/CentOS). The line I changed was from this:
1 | Listen xx.xx.xx.xx:80 |
to:
1 | Listen 127.0.0.1:8080 |
After a quick restart apache is now running on localhost:8080. Now we are ready to set up varnish. To do this I first installed the software following the instructions on the wiki. Once done I then created a new vcl file. Varnish config is in its own language called VCL (Varnish Configuration Language) more info can be found here about it. Now that I created a config file (lets call it example.com.vcl) I then had to define the backend server which varnish would use to get any requests. To do this I put in the following code:
1 2 3 4 | backend live { .host = "127.0.0.1"; .port = "8080"; } |
This tells varnish that it should go to the backend at 127.0.0.1 on port 8080 for any requests. You can have multiple backends and varnish can also check to make sure the backends are healthy but I did not require this feature so I cannot comment on it.
Now that I have set up the backend I needed to make a few small changes to the /etc/sysconfig/varnish file which give the daemon its defaults when starting up. The following are things I had to change:
This allows varnish to sit on port 80.
This tells the daemon what config file to use
This puts the cached pages into the memory and allocated is 1G. I also had to comment out VARNISH_STORAGE_FILE and VARNISH_STORAGE_SIZE.
After completing this I could start varnish and the website sprang to life!
Currently varnish will not cache anything but in my next post I shall explain some of the simple rules that we have set up and also how I came to work out what to cache.
Popping All The Way
One of the more interesting thing that happened today, had to be when one of my co-worker came to me with a query about their email. If they logged into webmail, there was no email, if he checked on his desktop using Thunderbird, there was no email, but his laptop did have email. In particular it was only his inbox which wasn't showing up. What was the problem? Well when the user set up their Thunderbird client he had set the account to be a POP3 account and to remove messages from the server...
It is a fairly easy mistake to make. So how do you sort this? Dragging and dropping? Well no, luckily our Kerio MailServer backups up every night. Unfortunately this did mean a little bit of downtime for our mail server mid-way through the day (this couldn't be helped). The command to restore the users inbox was fairly straight forward and required not a lot of thought. Firstly (since I run Kerio on a Linux server I had to change into the /opt/kerio/mailserver directory to find the kmsrecover file. Here is the recovery command (by the way, I changed the users name to mine to protect him):
./kmsrecover -v -d kraya.co.uk -u richard.annand -f "INBOX" /opt/kerio/mailserver/store/backup
When I checked the Kerio manual for how to do this, it suggests to point it at a particular backup file. But just pointing it at the directory lets Kerio work it all our for you. SIMPLE! After that, turned the mailserver back on and there we go, emails back.
A BIG Adobe Problem
Readers of Stephen Ramsay's blog may have recently read his post about Adobe. To sum up, one of our clients used over 70GB of data which was defiantly above there normal usage. The reason? Adobe. If you'd like to read more visit Stephen's blog.
I decided that I would discuss the technical side of how exactly we blocked this from happening. I never realised how simple it actually is to block a website using squid. Firstly I should mention that it is important to get the rules in the EXACT order. I made the mistake of having:
acl bad url_regex -i '/etc/squid/squid-block.acl'
acl our_networks src 192.168.100.0/24 127.0.0.1http_access allow our_networks
http_access deny bad
http_access deny all
This of course meant that 'our_network' was allowed to access any website (as it comes before on the deny allow list). So to fix this I changed it to:
acl bad url_regex -i '/etc/squid/squid-block.acl'
acl our_networks src 192.168.100.0/24 127.0.0.1http_access deny bad
http_access allow our_networks
http_access deny all
This meant that the 'bad' list was above the network thus being denied first. But now what? Squid has these settings but no rules. So this is where the file /etc/squid/squid-block.acl comes into play! This file contains the website that you wish to ban. So this file for me contained:
ardownload.adobe.com
And that's it, the website is blocked! Simple isn't it.
I'd like to thank freesoftwaremazine.com for the help article.
Another blog coming soon!
Ciao!
Twitter Update
So I have decided to split up my twitter. I now have two. If you would like to follow me and my work tweets follow richarda_work.
New blog post coming soon.
Ciao!
Upgrades, Upgrades, Upgrades!
Well this week has been a week of upgrades! The Kraya blogs, SugarCRM, Ubuntu Desktops and Kerio MailServers.
So firstly, it was time that the Kraya blogs were upgraded, one of the main reasons was so Krish could use the Arras theme (which by the way is looking SWEET! Check it out http://krish.blog.kraya.co.uk). Another big thing that was to happen was for the blogs to change from being .net addresses to .co.uk addresses. My first plan of attack was to move a copy of the blogs to another server (which turned out to be its new home in the end). After rsyncing the files and dumping the database I dived right in and though I'd see if I could just upgrade the site, well it kind of worked...I managed to logging and change the blogs to .co.uk but after that I couldn't do anything. So after some suggestions from other people in the office I took the database dump and searched and replaced everyone blog.kraya.net occurrence with blog.kraya.co.uk. After putting the database back up I could get into the admin panel. Erhh but ye there was a problem...a very big problem...none of the username were recognised. So after some head scratching and lots of IRC chat, I found out that it's not nessesary to upgrade version by version, you can infact go straight to the latest release! This made it so much easier. So after getting the latest WordPress MU and some playing around with files we had lift off! One thing to note, after installing the latest version I had forgotten to pull in the themes, so this meant that the blogs would not display. To sort this you can either select a new theme or simply bring the themes accross. I also took this opportunity to add in a few plugins like Digg Digg which now gives us the great new sharing features.
Well I think thats enough for today, I shall continue this next week.
Ciao!
Time Machine?
Right, as some of you may know I have recently purchased a MacBook Pro. I am loving it. But I now have a small problem. I am trying to use time machine to backup all those important files which I keep. Simple? Well not when you are trying to back up to a Western Digital My Book World Edition.
I used this drive with my Windows systems no problem. But I cannot use this drive as a disk for time machine. I tried to use iTimeMachine to allow me to do this, but unfortunately whenever a backup tries to run it fails.
Am I doomed? Will I have to go out and spend a rediculous amount of money on Apple’s time capsule? Lets hope not. Even although the time capsule does look rather slick, at a cool £229.00 for the 500GB version and £379.00 for the 1TB version I’d rather stay far, FAR away from it.
iTunes – The Saga Continues
Right, so heres a wee update. Basicly I have three computers, one iPod, one iPhone and a Apple TV box. I want all of them to have the same iTunes music libraries. Should be simple, right? Well so far it ain't been all flowers and roses. If you want to catch-up on everything that I've done so far then look back at some of my other posts. But anyway now that bits out the way here's what's happened so far.
So I finally decided to bite the bullet and get a external HDD to store my iTunes music (grr). £40 Iomega 320GB portable external HDD from Tesco Direct (Bargain?). So now that I have my nice new drive I decided to consolidate all my iTunes music library back into C:\Users\Richard\Music instead of having a separate partition. Once I'd done that using iTunes magic I then downloaded a program called Allways Sync. It's a nice little application does exactly what it says on the tin really and I cant complain. So i sync'd my entire music library to my new drive. The first time it obviously took a little time but after the first go it hasn't been to bad. So now that it's on the drive, now what? Well I installed the same application on my other PC's and sync'd the music library to iTunes' default directory. After it was done, woohoo, my music library was now on another machine.
So now I have my music libraries on different machines, my Apple TV box sync'd to my desktop at home and my iPod and iPhone sync'd to my laptop. Happy days! I think it's fair to say, this chapter is finished.
The Battle Is Over…
I have to say my battle with iTunes is now over and I lost 
In the end I decided that it would be best for me to just sync my iPod from one computer (my laptop) and use TuneRanger to sync my music library to another computer. TuneRanger DID work for syncing my iPod but it didnt being accross any of my ratings and my auto playlists within iTunes. I am now in the process of syncing my iTunes library on my laptop to my desktop and im HOPING this is going to work.
Anyway I'll keep you up to date with what happens.
New Fancy Tools
Right so I was getting a bit fed up with having to sign into the web front to post my blog entrys, so I decided to look for something that allowed me to do my blog posts from both my iPhone and my desktop machines that I have. To which I found Qumana. It seems pritty good at first sight and I'll be giving it a go over the next few weeks as I continue to blog my heart out both here and on my personal blog.
What I like about Qumana is that it gives the end users a ver simple GUI which just makes it so nice to blog. It still allows me to add in categories and tags. Its also nice that it has a spacific button for adding in ad's and html coding. Problem is that I now have a add at the bottom of each post but oh well.
Anyway more about this later when I get to properly test it out.
Tags: Richard, Richard Annand, Qumana, iPhone, desktop client, blogging client, GUI
Powered by Qumana
My Battle With iTunes…
So this weekend I decided that it was time to clean out my desktop PC and re-install Windows on a duel-boot with XP Pro 64-bit and Vista Business 64-bit (so one day I can upgrade if I wanted to). The problems all started after I re-installed both my OS's. I had my 40GB iTunes library to put back in place. Sounds simple? Well it hasn't been. The problem that I had was that on my old install the partition which stored all my music was the D drive. So when I reinstalled my OS's D is taken up my the other OS (e.g. Vista partition is D in XP). And allas Vista dosnt allow you to change the system volume letters. I really didn't want to replace my entire music library as I had all my ratings etc. So my first plan was to use a piece of software called TuneRanger to sync my iPod back to my desktop. This is a good idea, in theory. Turns out my iPod was a little corrupt so not all my music was getting transfered.
I decided since I couldn't find anything on the net to help me with this to write it all up. So heres what I did to solve the problem...
First i needed to release the letter D from the system drive. To do this I ran regedit in Vista and navigated to HKEY_LOCAL_MACHINE > SYSTEM > MountedDevices. I then right clicked on the entry called "\DosDevices\D:" and renamed it "\DosDevices\Y:" (since Y was free). I then went into computer manager and changed the drive letter for my original music drive (M:) to the D which was now avalible. Once doing this I restarted the machine and logged back in. SUCCESS! The drive letter had changed. Now for the tricky bit! I navigated to the Music folder in my user home folder and move the iTunes folder to my desktop (just incase things went a bit pear shapped) and copied all the iTunes library files into a new iTunes folder. I opened iTunes and TADA! all my music was there and worked. Problem was, now everything was pointing at the D drive. To solve this i went into the preferences in iTunes and changed the default libary folder to the M drive. The final step was to go to File > Library > Consolidate Library. This then moved all my music from the D to the new M drive (the default locations for iTunes to keep my music organised). I must point out here that iTunes organises all my music for me, if this option wasn't on them this wouldn't have worked.
So now I'm sitting here waiting for TuneRanger to copy all my music accross to my iPod. I decided to use TuneRanger instead of iTunes for syncing as this then allows me to plug my iPod into any computer without it trying to re-sync with someone else's library.
Next step: Now I want to sycn my Library accross to two other computers which I have and also to my work PC. This could be fun...
