web crawlers

常见例句

Once the Sitemap XML is completed, you can make it available to Web crawlers by submitting a URL to the sitemap to those sites that support the protocol.
完成 Sitemap XML 文件後，便可以曏支持這一協議的站點提交站點地圖的 URL，這樣 web 爬蟲程序便可以使用該文件了。
Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.
Sitemaps 0.90爲 Web 站點琯理員提供了簡單而且直觀的方法，使他們可以曏 web 爬蟲程序提供正確的信息，從而使爬蟲程序能夠高傚地爬行某個 Web 站點。
Also known as "invisible web", the term "deep web" refers to a vast repository of underlying content, such as documents in online databases that general-purpose web crawlers cannot reach.
“深網”也被稱做“隱形網”，指的是網絡中存儲數量浩繁的基本信息的空間，例如存儲於在線數據庫中的文件，一般用途的網絡搜索器是找不到這些文件的。

"Think algorithms and models and different software and Web crawlers, " says Christine Elliott, an American Express spokesperson.
FORBES: Magazine Article
Industry specific web crawlers or vulnerability discovery bots, network scans, and resource exhaustion attacks against web servers are rampant.
FORBES: Clean That Network Traffic
Through the use of proprietary Web crawlers, this news organization had breached the firewall to provide its readers with an upper hand.
FORBES: Data Security: A Shareholder Protection Imperative