urldedupe

urldedupe is a c++ tool to quickly pass in a list of URLs and get back a list of deduplicated (unique) URL and query string combination.

Only use this tool against systems you own or are explicitly authorized to test — see the disclaimer.