hours after Google site, has collected 2 pages of new sites, followed by the search included the home page in 24 hours. The new station so far has been born for more than 30 hours, while Shanghai has not released the home of love. Because the search mechanism and rules are not the same, at present, Google’s new collection efficiency is awesome, love Shanghai slow.
, a spider crawl between
process can be seen from the above, Google spider is polite and pay attention to etiquette, will be strange new owner repeatedly knocked on the door clearance in every action after the reference robots rules for page crawl, reminds me of the PLA’s three discipline eight note, like a here. Love Shanghai spider ignore robots, door repeatedly bite home more than 10 eloquence to let go, let a person feel very brutal violence, eating spiders continue to eat home page, the love of spiders in Shanghai robots files are completely ignored.
In March 1st
share is the new sites in Google and love Shanghai grab and included differences in the hope that through this case, to study Shanghai dragon friends and new friends to learn some reference and inspiration, let you know more about Google love Shanghai crawl rules and Collection >
started a new website to make a type of clothing, in the local debugging after the upload space. This website is a new domain name, whether from space and space domain name and website content, is the day of registration and purchase, I observed the whole process of specific love Shanghai and Google spiders crawl and included new sites. For a new website, love Shanghai and Google have many differences in the content of the website and included the crawling process, mainly in the following aspects:
I always love to observe the spider whereabouts, known as the "Spider Man" in many webmaster group, this is the source of my habits and hobbies. This new chance is definitely not to be missed. The content of the website upload after the first few portal forum post reply lead spider, a few hours after the access log has been produced, love Shanghai spider faster, before Google came to the site. Through the log view, Google spider is very interesting, repeated crawling robots file 10 times after carefully grab the home page, while after the climb from the robots side to crawl the site inside pages. Love of spiders in Shanghai seems to be more anxious, my website is made robots file, but Shanghai does not love crawling this file directly to grab the web page, and very hungry and crazy, in 1 minutes, repeated crawling home 20 times, then go on to grab the page. Return status code 20000, said all grab success. Interested friends can go to my website crawling log yesterday, grabbed a new station, there is a very good reference value for the law and character of the spiders.
two, the website included differences in