The Robots.txt protocol, also called the ârobots exclusion standardâ is designed to lock out web spiders from accessing part of a website. It is a security or privacy measure, the equivalent of hanging a âKeep Outâ sign on your door.This proto…