robots.txt matching the way RFC 9309 and Google define it: most specific rule wins, * and $ wildcards. Fixes urllib.robotparser first-match bug. No dependencies.
-
Updated
Sep 29, 2026 - Python
#
robotparser
Here is 1 public repository matching this topic...
Add this topic to your repo
To associate your repository with the robotparser topic, visit your repo's landing page and select "manage topics."