Access Websites as a bot

UPDATE: This is NOT working now.

Google bot and yahoo bot are the general terms for their automated web crawling service that is linked to Google and yahoo search engines. Bots sends out requests to web-pages that use a Google or Yahoo Bot user agent. This specific user agent is used for several purposes including identification and restrictions.

Web masters can for instance filter out Bot from their website statistics to get a better picture of how many real users visit the site in a given time. Some web-masters and services on the other hand try to cheat by allowing Bots access to all of their contents while they display a registration or buy page to users who want to access the same information.
That’s not allowed according to Google’s terms of use but some web-masters do it nevertheless.
Some users had now the idea to pose as Google Bot to access the information without buying or registering first.

Be The Bot helps you in this. It contains a form where a web address can be entered. The user can select to act as Google Bot or Yahoo Bot. The requested URL will then be displayed on the same screen.

Have you ever been googleing something, and you see exactly what you need in the preview, but when you click the link it doesn't show you what you want to see?
This is because the owners of the site are trying to trick you to do something like registering etc,. When Google crawls the site, it gives header “Google-bot”. The programmers of the site just makes a check for the header.

This works on all pages that allow Google Bot or Yahoo Bot complete access to their website but block visitors by asking to do some other thing.