About Robots (core)

Support for Robots Exclusion Protocol, including parsing and matching against robot.txt directives.