This page describes some ways to extract search engine hits from a websites log files. Extracting Hits from Apache Log Files To extract just the Googlebot hits on the site using the GNU/Linux terminal, try this: grep 'Googlebot\/' access.log > googlebot_access.log That will write the Googlebot hits to a new logfile called googlebot_access.log. You can also pipe that output into another command, for example to extract only the URLs that Googlebot is requesting:
- Using log files for SEO analysis is a great way to uncover issues that you may have otherwise missed. This is because, unlike third party spiders, they allow you to see exactly how Googlebot is crawling a site. If you’re an SEO professional looking to carry out your own log file analysis, then the chances are you’ll have to request the files through your own, or your clients, dev team.