Dignity nobility quality point out highness standing class caste standing superiority priority style difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and will instruct the robotic as to which pages aren't to be crawled. As being a internet search engine crawler may https://www.youtube.com/watch?v=ZPZ0MrJuEfM