Location:
Search - crawler
Search list
Description: 网页抓取器又叫网络机器人(Robot)、网络爬行者、网络蜘蛛。网络机器人（Web Robot），也称网络蜘蛛(Spider)，漫游者（Wanderer）和爬虫（Crawler），是指某个能以人类无法达到的速度不断重复执行某项任务的自动程序。他们能自动漫游与Web站点，在Web上按某种策略自动进行远程数据的检索和获取，并产生本地索引，产生本地数据库，提供查询接口，共搜索引擎调用。-web crawling robots- known network (Robot), Web crawling, spider network. Network Robot (Web Robot), also called network spider (Spider), rovers (Wanderer) and reptiles (Crawler), is a human can not reach the speed of repeated execution of a mandate automatic procedures. They can automatically roaming and Web site on the Web strategy by some automatic remote data access and retrieval, Index and produce local, have local database, which provides interfaces for a total of search engine called.
Platform: |
Size: 20480 |
Author: shengping |
Hits:
Description: 一个很好的搜索引擎爬行器程序，想了解搜索引擎原理的朋友可以看看这个。-a good search engine crawling with procedures that to understand the principles of search engine you can look at this.
Platform: |
Size: 16788480 |
Author: zhaomin |
Hits:
Description: 基于atmel公司的mega16单片机做的机器人控制程序，机器人采用坦克履带式小车，有越障，追踪，寻迹等功能-atmel based company mega16 SCM done robot control procedures, Robot used tanks crawler Dolly, the more obstacles, tracking, tracing, and other functions
Platform: |
Size: 39936 |
Author: 朱宇 |
Hits:
Description: combine Focused Crawler
Platform: |
Size: 831488 |
Author: 金红 |
Hits:
Description: 1、锁定某个主题抓取；
2、能够产生日志文本文件，格式为：时间戳(timestamp)、URL；
3、抓取某一URL时最多允许建立2个连接（注意：本地作网页解析的线程数则不限）
4、遵守文明蜘蛛规则：必须分析robots.txt文件和meta tag有无限制；一个线程抓完一个网页后要sleep 2秒钟；
5、能对HTML网页进行解析，提取出链接URL，能判别提取的URL是否已处理过，不重复解析已crawl过的网页；
6、能够对spider/crawler程序的一些基本参数进行设置，包括：抓取深度(depth)、种子URL等；
7、使用User-agent向服务器表明自己的身份；
8、产生抓取统计信息：包括抓取速度、抓取完成所需时间、抓取网页总数；重要变量和所有类、方法加注释；
9、请遵守编程规范，如类、方法、文件等的命名规范，
10、可选：GUI图形用户界面、web界面，通过界面管理spider/crawler，包括启停、URL增删等
-1, the ability to lock a particular theme crawls; 2, can produce log text file format : timestamp (timestamp), the URL; 3. crawls up a URL to allow for the establishment of two connecting (Note : local website for a few analytical thread is not limited) 4, abide by the rules of civilized spiders : to be analyzed robots.txt file and meta tag unrestricted; End grasp a thread after a website to sleep two seconds; 5, capable of HTML pages for analysis, Links to extract URL, the extract can judge whether the URL have been processed. Analysis has not repeat crawl over the web; 6. to the spider/crawler some of the basic procedures for setting up parameters, including : Grasp depth (depth), seeds URL; 7. use User-agent to the server to identify themselves; 8, crawls produce statistical informati
Platform: |
Size: 1911808 |
Author: |
Hits:
Description: 爬虫设计，认真阅读对您的文件包热后写出具体功能
Platform: |
Size: 11264 |
Author: 望类 |
Hits:
Description: 这是一个WEB CRAWLER程序，能下载同一网站上的所有网页-This is a WEB CRAWLER procedures, can download the same site all pages
Platform: |
Size: 3072 |
Author: xut |
Hits:
Description: A web crawler (also known as a web spider or web robot) is a program or automated script
which browses the in a methodical, automated manner. Other less frequently used names for
web crawlers are ants, automatic indexers, bots, and worms (Kobayashi and Takeda, 2000).来源。-A web crawler (also known as a web spider or web robot) is a program or automated scriptwhich browses the in a methodical, automated manner. Other less frequently used names forweb crawlers are ants, automatic indexers, bots, and worms (Kobayashi and Takeda , 2000). source.
Platform: |
Size: 218112 |
Author: sun |
Hits:
Description: C#编写的网页内容抓取工具，多线程采集，效率很高-C# Prepared crawler web content, multi-threaded collection, high efficiency
Platform: |
Size: 279552 |
Author: youdechun |
Hits:
Description: 一个简单的在互联网上抓包的程序，仅供大家参考-A simple Internet capture procedures, for your reference
Platform: |
Size: 2197504 |
Author: ahsm |
Hits:
Description: lucene 是java 的版的搜索引擎公共模块， 本人使用此模块， 已经开发实现了网页的抓取。 -is java version of Lucene search engine public module, I use this module, has developed a web crawler.
Platform: |
Size: 395264 |
Author: chenbaoji |
Hits:
Description: Heritrix: Internet Archive Web Crawler
The archive-crawler project is building a flexible, extensible, robust, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accesible content.
Platform: |
Size: 3096576 |
Author: gaoquan |
Hits:
Description: 功能：
根据指定的网址，下载网页，并分析其中的URL继续下载，并将网页主要内容存为本地文件
为搜索引擎的索引的建立提供原材料 -Function: In accordance with the specified URL, download page and analyze the URL to download the main page and save it as a local file for search engine indexing of the establishment of the provision of raw materials
Platform: |
Size: 41984 |
Author: veryha |
Hits:
Description: 1.Hyper Estraier是一个用C语言开发的全文检索引擎,他是由一位日本人开发的.工程注册在sourceforge.net(http://hyperestraier.sourceforge.net).
2.Hyper的特性:
高速度,高稳定性,高可扩展性…(这可都是有原因的,不是瞎吹)
P2P架构(可译为端到端的,不是咱们下大片用的p2p)
自带Web Crawler
文档权重排序
良好的多字节支持(想一想，它是由日本人开发的….)
简单实用的API(我看了一遍，真是个个都实用,我能看懂的，也就算简单了)
短语,正则表达式搜索(这个有点过了,不带这个,不是好的Full text Search Engine?)
结构化文档搜索能力(大概就是指可以自行给文档加上一堆属性并搜索这些属性吧?这个我没有实验)
Platform: |
Size: 649216 |
Author: gengbin |
Hits:
Description: C++写的网络爬虫程序，可以正确爬下网页内容-C++ Write network reptiles procedures, you can climb down the right Web content
Platform: |
Size: 1613824 |
Author: ly |
Hits:
Description: 基于Lucene的网页生成工具,对于有网页爬行器从网络上下载下来的网页库，本软件可以对他们进行网页索引生成，生成网页索引是搜索引擎设计中核心的部分之一。也称网页预处理子系统。本程序用的是基于lucene而设计的。-Lucene-based web page generation tool, for Crawler has pages downloaded from the web page database, the software can index their web pages to generate, generate web pages search engine index is part of the design of one of the core. Also known as pre-processing subsystem website. This procedure used is based on the Lucene designed.
Platform: |
Size: 3340288 |
Author: 纯哲 |
Hits:
Description: 用java写的一个网络蜘蛛，他可以从指定的URL开始解析抓取网页上的URL，对于抓取到的URL自动分成站内外URL，并可以设置抓取的深度。-Using java to write a Web Spider, he can from the specified URL to start crawling on the page to resolve URL, the URL for the crawler to automatically divided into stations inside and outside the URL, and can set the crawling depth.
Platform: |
Size: 5120 |
Author: 纯哲 |
Hits:
Description: 用C#實現HTML剖析的功能，可以用於瀏覽器及Web Crawler的開發-With C# Achieve HTML parsing functions, can be used in browsers and Web Crawler Development
Platform: |
Size: 79872 |
Author: gagaclub |
Hits:
Description: java写的crawler，看看看不懂，大家一起研究一下吧！-java wrote crawler, can not read to see if we can work together to look at it!
Platform: |
Size: 702464 |
Author: 刘双 |
Hits:
Description: A WebSpider or crawler is an automated program that follows links on websites and calls a WebRobot to handle the contents of each link. -A WebSpider or crawler is an automated program that follows links on websites and calls a WebRobot to handle the contents of each link.
Platform: |
Size: 54272 |
Author: king |
Hits:
«
1
23
4
5
6
7
8
9
10
...
50
»