But not every website, or every part of each website, is intended for general consumption or for the search engines to pick up on and publisize, so a special "Meta Tag" (see example below) was developed for the search engines to spot and act on. Typically, this tag would tell them to not index the page, although there are other applications as well.
For Example: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">The search engine programs are set up to look for this tag, and also to see if there exists a file called ROBOTS.TXT which could also contain instructions for the indexing programs. Most of the time this file is not found but the webserver where your website resides does faithfully make a note of the fact that an unsuccessful attempt was made to retrieve the file.. And as many of the standards used today are based on the early terms and methodologies, the name of the file persists.
So, while instances of the missing file Robots.txt will appear in your statistics, this is nothing that you should get too alarmed about. We could make the issue disappear by placing an empty robots.txt out there as spider fodder, but then you'd be losing a useful piece of information. Noticing that the file is called for means that the spiders are visiting. No calls for robots.txt would be an indication of low search engine indexing activity and may be a signal for some search engine registration work to be stepped up.