Go to file

Brian Buller 197750425d Smarter URL checking It should handle subdomains appropriately now		2015-08-20 09:16:02 -05:00
.gitignore	Init commit	2015-07-29 07:43:21 -05:00
LICENSE	Init commit	2015-07-29 07:43:21 -05:00
README.md	Whoop, those are tags	2015-08-15 10:39:08 -05:00
fullscrape.go	Smarter URL checking	2015-08-20 09:16:02 -05:00

README.md

fullscrape

A Full Website Scraper

Usage:

fullscrape url out-dir [-d=depth] [-x]

Arguments:

url: The url that you want to start the scrape at. out-dir: The output directory. If it doesn't exist, it will be created. d: The depth of the scrape. Defaults to no limit. x: Cross Domain scraping. This could get dangerous, the internet's a big place.

Sometime I may add the ability for fullscrape to automatically make non-relative links relative. Because that seems useful.