This little guide will teach you pretty much everything about Robots.txt in simple words.

What is it?

Its basically a text file which you upload to the root of your hosting account/server and it tells search engine spiders which page to crawl and which one's they should not.

Why use this?

You don't have to. Only use this when you don't want some of your pages indexed or even the whole site. Like a private site. That's when you would use it.

How does it look like?

Here's a typical example of a robots.txt

How do I get it to work?

User-agent: * <-- Is telling which crawlers should keep out. You can add a specific crawler or apply it to all of them by using an asterisk, *.

Disallow: <-- Its telling which directories or files they should not crawl. If you don't want them to crawl your site then use a forward slash, /.

Well you first need to know which pages you don't want the spiders to crawl. From there is should be pretty simple.

Let me know if you need help.

-th13rteen

__________________
WebRampage! - Your crazy little guide to the web

Thanks for this again dude.. I really need this...

__________________

Nice thanks for its