Dignity nobility good quality point out highness standing course caste status superiority precedence style difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and can instruct the robotic regarding which pages will not be for being crawled. For a online search engine crawler https://www.youtube.com/watch?v=ARvdj_mJDAY