web crawlers, support robot.txt

JavaCrawler
...........\.classpath
...........\.DS_Store
__MACOSX
........\JavaCrawler
........\...........\._.DS_Store
JavaCrawler\.project
...........\.settings
...........\.........\org.eclipse.jdt.core.prefs
...........\bin
...........\...\Controller
...........\...\..........\BFSCrawler$1.class
...........\...\..........\BFSCrawler.class
...........\...\..........\GetWebsitePage.class
...........\...\..........\HtmlParser$1.class
...........\...\..........\HtmlParser.class
...........\...\Main.class
...........\...\Models
...........\...\......\LinkFilter.class
...........\...\......\MyQueue.class
...........\...\PracHttpParser.class
...........\DownloadFiles
...........\.............\.DS_Store
__MACOSX\JavaCrawler\DownloadFiles
........\...........\.............\._.DS_Store
JavaCrawler\DownloadFiles\duke.edu.html
...........\.............\today.duke.edu_2014_09_sept11memorial.html
...........\.............\today.duke.edu_alumni-features.html
...........\.............\today.duke.edu_alumni.html
...........\.............\today.duke.edu_topic_alumni#other.html
...........\.............\today.duke.edu_topic_alumni.html
...........\.............\Xinran_Luo_urls
...........\Jars
...........\....\commons-codec-1.3.jar
...........\....\commons-httpclient-3.1.jar
...........\....\commons-lang.jar
...........\....\commons-logging-1.1.1.jar
...........\....\htmllexer.jar
...........\....\htmlparser.jar
...........\....\je-4.0.92.jar
...........\MyQueue
...........\src
...........\...\Controller
...........\...\..........\BFSCrawler.java
...........\...\..........\GetWebsitePage.java
...........\...\..........\HtmlParser.java
...........\...\Main.java
...........\...\Models
...........\...\......\LinkFilter.java
...........\...\......\MyQueue.java
...........\...\PracHttpParser.java

• We are an exchange download platform that only provides communication channels. The downloaded content comes from the internet. Except for download issues, please Google on your own.
• The downloaded content is provided for members to upload. If it unintentionally infringes on your copyright, please contact us.
• Please use Winrar for decompression tools
• If download fail, Try it againg or Feedback to us.
• If downloaded content did not match the introduction, Feedback to us，Confirm and will be refund.
• Before downloading, you can inquire through the uploaded person information

Nothing．