I went accross a french blog post some weeks ago on SEO analytics with Google Analytics, which is in my opinion one the biggest bomb I read regarding a Google Analytcs technique in the last 6 months. I decided to translate that post on my blog to tell you about it, and share this secret with the english speaking community.
As most of us know, this is impossible to track search engine bots with GA as the tool get data through Javascript tags and the only way to track bots is with Log files. Those 2 french guys found a solution, which will give you the opportunity to see on Google Analytics, which Spiders visited your site, and which pages they crawled!
You first need to create a new profile within your Google Analytics account. The profile has to be for a new domain (see below). You can give to the profile the name you want although ROBOTS.yoursite.com is in my opinion a good way to keep things clear within your GA account.
You then need to download this folder, which contains 3 PHP file, and change 3 small things within the config.php file which are:
- Adding the Google Analytics ID of the profile you have just created.
- Adding you domain name
- Adding the Hash of your domain, which you can find looking at the value of your _utma cookie on your website (see pic below).
The next step is to upload the folder “analytics” on the ftp server of your site and include the “analytics” folder on each page of your site (the footer is the best option). You can include it as below:
<?php include_once( dirname(__FILE__).’analytics/analytics.php’); ?>
That’s it! That is the only thing to do. I manage to implement this in 30 minutes, without having strong technical experience. The Outcome is very interesting, as you can then see which bots visited your website.
When you click on the bot name, you will be able to see which pages have been crawled by this specific spider.
And finally, if you click in one of the page, you will be able to see which machine visited it (IP).
I’m not the real author of this post, so please visit the real guys who made it, and give them some nice deserved comments. Web analytics blog AND SEO blog.
