返回浏览器检测 · iPhone 手机真假检测 · 搜索引擎蜘蛛记录 · 今日UA记录 · 未知UA记录 · 更新日志

搜索引擎蜘蛛代码大全

搜索引擎蜘蛛与网络爬虫记录

 

本页面用来自然搜集来自搜索引擎抓取蜘蛛的 User-Agent 代码,当然有时候会发现除了百度、搜狗和360搜索之外,有很多没有见过的蜘蛛,本页暂时没有将蜘蛛代码进行排序,只是按照抓取的顺序自然收集和展示。

相关文章:
CCBot/2.0 (http://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.4.2661.102 Safari/537.36; 360Spider
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Scrapy/1.4.0 (+http://scrapy.org)
Scrapy/0.22.2 (+http://scrapy.org)
Scrapy/0.24.6 (+http://scrapy.org)
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
YisouSpider
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
Mozilla/5.0 (compatible; Renren Share Slurp 1.0; http://share.renren.com/share/help/webmaster)
Auto Spider 1.0
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Chouti Spider Beta
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
Baiduspider
Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
BaiduSpider
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
Twitterbot/1.0
SinaWeiboBot
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.2.2661.102 Safari/537.36; 360Spider
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0(compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
elefent/1.2 (web crawler; crawler.elefent.net; webmaster at elefent dot net)
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; IDBot/1.1; +http://www.id-search.xyz/bot.html)
Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
netEstate NE Crawler (+http://www.website-datenbank.de/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - web-crawlers@hubspot.com
Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
Mozilla/5.0 (compatible; ShaoziSpider/1.0;)
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Baidu-YunGuanCe-SLABot(ce.baidu.com)
Barkrowler/0.7 (+http://www.exensa.com/crawl)