Description: High concurrency, heavyweight software support a distributed crawler written in pure Go, located in the Internet data collection, provide a crawler tool only needs to pay attention to the rules of custom and powerful to have Go or JS programming based on the people.
It supports a single server and client three operating modes, including Web, GUI, three command line interface rules simple and flexible, batch task and rich output mode (mysql/mongodb/csv/excel), a large number of Demo shared and it also supports the horizontal and vertical two grasping mode, supports analog login and task suspend, cancel a series of advanced features.
To Search:
File list (Check if you may need any files):
pholcus-1.2.0
.............\.gitignore
.............\app
.............\...\aid
.............\...\...\history
.............\...\...\.......\failure.go
.............\...\...\.......\history.go
.............\...\...\.......\success.go
.............\...\...\proxy
.............\...\...\.....\host.go
.............\...\...\.....\proxy.go
.............\...\app.go
.............\...\crawler
.............\...\.......\crawler.go
.............\...\.......\crawlerpool.go
.............\...\.......\spiderqueue.go
.............\...\distribute
.............\...\..........\interface.go
.............\...\..........\master_api.go
.............\...\..........\slave_api.go
.............\...\..........\task.go
.............\...\..........\taskjar.go
.............\...\downloader
.............\...\..........\downloader.go
.............\...\..........\downloader_surfer.go
.............\...\..........\request
.............\...\..........\.......\request.go
.............\...\..........\.......\request_test.go
.............\...\..........\.......\temp.go
.............\...\..........\surfer
.............\...\..........\......\agent
.............\...\..........\......\.....\agent.go
.............\...\..........\......\.....\agent_bsd.go
.............\...\..........\......\.....\agent_linux.go
.............\...\..........\......\.....\agent_windows.go
.............\...\..........\......\param.go
.............\...\..........\......\phantom.go
.............\...\..........\......\request.go
.............\...\..........\......\surf.go
.............\...\..........\......\surfer.go
.............\...\..........\......\util.go
.............\...\pipeline
.............\...\........\collector
.............\...\........\.........\collector.go
.............\...\........\.........\data
.............\...\........\.........\....\data.go
.............\...\........\.........\docker.go
.............\...\........\.........\output_csv.go
.............\...\........\.........\output_data.go
.............\...\........\.........\output_excel.go
.............\...\........\.........\output_file.go
.............\...\........\.........\output_kafka.go
.............\...\........\.........\output_mgo.go
.............\...\........\.........\output_mysql.go
.............\...\........\.........\output_util.go
.............\...\........\output.go
.............\...\........\pipeline.go
.............\...\scheduler
.............\...\.........\matrix.go
.............\...\.........\scheduler.go
.............\...\spider
.............\...\......\common
.............\...\......\......\common.go
.............\...\......\......\form.go
.............\...\......\context.go
.............\...\......\parsejs.go
.............\...\......\species.go
.............\...\......\spider.go
.............\...\......\timer.go
.............\...\......\timer_test.go
.............\cmd
.............\...\pholcus-cmd.go
.............\common
.............\......\bytes
.............\......\.....\bytes.go
.............\......\config
.............\......\......\config.go
.............\......\......\fake.go
.............\......\......\ini.go
.............\......\......\ini_test.go
.............\......\......\json.go
.............\......\......\json_test.go
.............\......\gc
.............\......\..\gc.go
.............\......\kafka
.............\......\.....\kafka.go
.............\......\mahonia
.............\......\.......\8bit.go
.............\......\.......\ASCII.go
.............\......\.......\big5-data.go
.............\......\.......\big5.go
.............\......\.......\charset.go
.............\......\.......\convert_string.go
.............\......\.......\cp51932.go
.............\......\.......\entity.go
.............\......\.......\entity_data.go
.............\......\.......\euc-jp.go
.............\......\.......\euc-kr-data.go
.............\......\.......\euc-kr.go
.............\......\.......\fallback.go
Download