robotstxt

Robots.txt exclusion protocol implementation for Go language

uncategorizedRed Team

Official docs ↗Download ↗Source repo ↗

Platforms: Linux · Last verified September 6, 2026

This package contains a robots.txt exclusion protocol implementation for Go language (golang).

Only use this tool against systems you own or are explicitly authorized to test — see the disclaimer.

Getting started

Install on Kali Linux with `sudo apt install robotstxt`. See the official Kali tool page and upstream homepage linked above for full usage and configuration details.

Commonly preinstalled on

Kali Linux

Debian-based distribution built for penetration testing and security auditing