The site crawl provides us a wealth of knowledge, but to just take this audit to another stage, we have to talk to the search engines. Regrettably, engines like google do not like to provide unrestricted access to their servers so we are going to just should accept the https://www.youtube.com/watch?v=EVeTP0aV8gY