HLRnet.com: Technologie: Internet: Robots.txt

Robots.txt

• Met het bestand robots.txt (te plaatsen in de root van je website) kan je een deel van je website afschermen van spiders, of slechte robots (die bv. op zoek zijn naar e-mailadressen).
• Bjoernsworld (DE)
• How to keep bad robots, spiders and web crawlers away (Fleiner)
• How to Set Up a robots.txt to Control Search Engine Spiders (The Site Wizard)
• Jasoft: robots that visit your website
• robotstxt.org
• Robots.txt information
• Robotcop: punish robots that don't respect robots.txt
• Sitepoint: the problem with ROBOTS.TXT
• Taming the beast: robots that read
• Open source 
• <nothing to list for the time being>
• See also
  • ACAP (Automated Content Access Protocol) is the new open standard that enables website owners to express terms under which crawlers (also known as robots or spiders) and others are allowed access to and use of their website contents.