Using Yahoo Site Explorer service for crawling other web sites
The POC proves once again that Web2.0 technologies open new ways of attacking Web infrastructure. Keep in mind that this spider is ultra fast. It does only several connects in order to obtain the entire directory structure of the targeted website. Also, keep in mind that it will take less then 5 minutes to make it equipped with the latest AJAX exploits. Therefore, I am not responsible for your actions.
I am planning to write a follow up post on how we can make basic client-side XSS scanner on the top of this spider, so stay tuned.
Link: proof of concept spider