Generally in case of small websites webmasters are under false assumption that they do not require to create a robot.txt file. But it is required. First of all let us define the robot.txt file. Even prior to it we need to define what a web robot is. A web…