The role of Robots.txt and method of use
he can’t climb to the spider doesn’t mean he can’t grab.
I need to grab the
love experience: jingyan.baidu贵族宝贝
we have to drop "the overall quality of our many irresistible factors.
2, shielding some of the duplicate pages.
we only those with common features page shielding. The spider can’t climb to the page, it will display a 404 page, it will form a dead link.
shield shield refers to let search engine put out. Shanghai love force will be relatively slow, like an old man, he is already an old man, he reflected particularly slow, Google quickly. Google as long as you submit, he immediately grab 24 hours in force, you see the shield blocked pages directly in Google Adsense management tools. But love not love Shanghai, Shanghai reflects the very slow.
1, shielding some content to the page.
as we search for a word, he will also appear such content. There is no search results page [page], error pages, page 404 (if you do). You will find more and more things we want to shield. And these things we have to deal with.
Robots is a website and search engine of the third party agreement (early for the protection of privacy).
Explanation: ShanghaiEffect of
address, a source address provided on the line. For example, a page inside of me, I don’t care a wrong address, but this link does not exist, the spider can’t grab, display code > 404
page, DZ forum comments such as abbreviated version (ative), WordPress RSS, the repeat of the page we need for processing. Some of the duplicate pages is we have to do, this is no way to do.
registration page: /register_p.jsp
can grab to address and can crawl into 2 concepts.
3, shielding some dead links page.
message page…… [/ sign up page forum page content page] sign is
, the origin of Robots.txt
login page: /login_c.jsp
regardless of our website forums, blogs, shopping mall, enterprises are good, now we need to deal with both of these pages and operation, we can only shield, can not be deleted. Our aim is to be able to achieve a function for users to search, we understand better.