View source for NLP tab

The NLP tab contains the visone Natural Language Processing functionality which can be used to create networks from text written in a natural language (usually English).

The NLP tab is not shown by default. In order for it to be enabled, visone must be started with the command line option
 -e de.visone.nlp.NLPExtension
If texts with long sentences are to be processed, it is furthermore desirable to add <code>-Xmx1g</code> to prevent visone from running out of memory during network creation.

In addition, the files <code>englishPCFG.ser.gz</code> and <code>left3words-wsj-0-18.tagger</code> should be downloaded from the [http://nlp.stanford.edu/ Stanford Natural Language Processing Group's website] and placed in the visone working directory. This will allow ''centering resonance analysis'' and ''word net analysis'' to work without explicitly specifying a parser or tagger file (see below).

== centering resonance analysis ==
This performs a [[CRA]] analysis of the input text.
* '''quick layout''' does a quick layout after network creation, which may take some time for huge networks.
* The '''stopword filter''' can either use the builtin list of stopwords (common function words that can be assumed to carry no meaning for the topic of the text), or a file containing one lowercase stopword per line, with empty lines and comments (starting with <code>#</code>) ignored. Also, a minimum word length can be set, so that all words shorter than this length are always ignored.
* A comma-separated list of '''center tags''' can be specified. These are the node names as used by the parser, eg. NP (noun phrase) or VP (verb phrase). Phrases can be nested, such that an NP contains more NPs. If '''take centers from bottom level''' is selected, the inner ones are used, else the top one, thus producing a much denser network.
* The '''parser file''' must be specified, however the default path points to <code>englishPCFG.ser.gz</code> in the visone working directory, so if the file is placed there as recommended, nothing should need to be selected.

== word net analysis ==
Performs a [[WNA]] analysis of the input text.
* For WNA, either a '''window size''' must be set, or if '''use co-occurence''' is selected, all words within a sentence are connected.
* WNA can filter according to POS (part-of-speech) tags. The default tag set selects nouns, verbs, adjectives, adverbs and foreign words. Alternatively, an external file with one tag per line can be specified; in this case, empty lines and comments (starting with <code>#</code>) are ignored. The default tagger uses the [ftp://ftp.cis.upenn.edu/pub/treebank/doc/tagguide.ps.gz Penn Treebank Tagset].
* The '''tagger file''' must be specified, however the default path points to <code>left3words-wsj-0-18.tagger</code> in the visone working directory, so if the file is placed there as recommended, nothing should need to be selected.

See the [[#centering resonance analysis|centering resonance analysis section]] for the '''quick layout''' and '''stopword filter''' options.

== crawl link structure ==
This function captures the link structure of a website into a network. One node is created for each page, and a (directed) edge for every link from one page to another. Each node will have the following attributes:
* '''url''': the URL of the page
* '''text''': the text content of the page (if enabled)
Each edge will have an attribute named '''linktext''' containing the text of the hyperlink represented by that edge. In addition, if a link was found but not followed, it will be marked as unconfirmed.

All that is required to use the crawler is the '''url''' field, which must be filled with the URL of the page at which the crawler should start, such as <code><nowiki>http://visone.info/wiki/index.php/Main_Page</nowiki></code> for the visone wiki. After crawling has finished, the network does ''not'' get layouted to allow handling of extremely large graphs, thus all nodes will be show lying on a single point. Click the quick layout button to see the network structure.

For finer control, the following options can be adjusted:
* '''browser''': The browser that visone will masquerade as, to avoid being blocked. Select one from the list, or paste any user agent string.
* '''max depth''': The maximum length of a chain of links that will be followed. For example, 0 will not follow any links, 1 will follow all links on the start page, and 2 will follow all links on the start page and on pages referenced by the start page.
* '''page limit''': The maximum number of pages that will be ''downloaded''. This is ''not'' a limit on the number of nodes created, because nodes are created for every page that is found, even if it is never downloaded.
* '''same host only''': If selected, links to different hosts will not be followed. For example, if a page on <code>visone.info</code> links to another page on <code>visone.info</code>, this link will be followed, but a link to <code>google.com</code> will not.
* '''store text''': If selected, the text content of every page that is downloaded will be stored in the '''text''' attribute of its node.

=== crawler expert options ===
* '''crawl rule selection''': This option decides which parts of the webpage will be considered textual content.
** '''standard''' uses a default rule that should work for most websites.
** '''domain specific''' selects the rule based on the URL, so that for example only the actual text of Wikipedia articles is extracted. The name of the rule file is generated from the URL, for example, the rule file for <code><nowiki>http://www.informatik.uni-konstanz.de/algo</nowiki></code> is <code>algo.www.informatik.uni-konstanz.de.rule</code>. Some rules, such as the one for Wikipedia, are already built in.
** '''specified''' uses the rule selected in the '''crawl rule''' dropdown.
* '''cookies''': If the website requires a login, paste the HTTP <code>Cookie:</code> header value here. (Obtaining this value might be complicated, though.)
* '''ignore errors''': If this option is ''not'' selected, any error encountered during crawling, such as the common ''page not found'', will terminate the crawler. Unchecking is only recommended if there are known to be no dead links.
* '''mean delay''': Delay after a successful page download, in milliseconds.
* '''cool down after error''': Delay after a failed page download, in milliseconds.
* '''respect nofollow''': If selected, visone will observe the [http://en.wikipedia.org/wiki/Nofollow nofollow] specification, and not follow any links such marked.

== crawl text ==
The purpose of this function is to download the textual content of a singe webpage and store it in a file that can be used as an input file for the text analysis methods discussed above.

Use '''output file''' to select the name of the file to (over-)write. For the '''url''', '''browser''' and expert options, see [[#crawl link structure|above]].