Das Korpus wurde weiterhin mit Methoden der Computerlinguistik aufbereitet, insbesondere unter Verwendung des <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Python Natural Language Toolkits</hi> (NLTK<ref xmlns="http://www.tei-c.org/ns/1.0" target="ftn2" n="2" />). Die Verarbeitung des Korpus umfasst die grundlegende Lemmatisierung mit dem <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">WordNet-Lemmatizer</hi> (Teil des NLTK) und eine Stoppwortbereinigung (NLTK-Stoppwortliste für Englisch mit eigener Erweiterung<ref xmlns="http://www.tei-c.org/ns/1.0" target="ftn3" n="3" />) sowie die Wortartenannotation mithilfe des <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Stanford Log-linear Part-of-Speech-Taggers</hi>. <?biblio ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.3115/1073445.1073478", "abstract" : "We present a new part-of-speech tagger that demonstrates the following ideas: (i) explicit use of both preceding and following tag contexts via a dependency network representation, (ii) broad use of lexical features, including jointly conditioning on multiple consecutive words, (iii) effective use of priors in conditional loglinear models, and (iv) /ufb01ne-grained modeling of unknown word features. Using these ideas together, the resulting tagger gives a 97.24% accuracy on the Penn Treebank WSJ, an error reduction of 4.4% on the best previous single automatically learned tagging result", "author" : [ { "dropping-particle" : "", "family" : "Toutanova", "given" : "Kristina", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Klein", "given" : "Dan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Manning", "given" : "Christopher D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1 (NAACL '03),", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2003" ] ] }, "note" : "NULL", "page" : "252-259", "title" : "Feature-rich part-of-speech tagging with a cyclic dependency network", "type" : "article-journal" }, "uris" : [ "http://www.mendeley.com/documents/?uuid=52a7cd62-5322-3e6b-b133-3b9229721a06" ] } ], "mendeley" : { "formattedCitation" : "(Toutanova et al. 2003)", "plainTextFormattedCitation" : "(Toutanova et al. 2003)", "previouslyFormattedCitation" : "(Toutanova et al. 2003)" }, "properties" : { "noteIndex" : 0 }, "schema" : "https://github.com/citation-style-language/schema/raw/master/csl-citation.json" }?> Da Dylan in seinen Texten häufig umgangssprachliche Formulierungen wie etwa verkürzte Gerundformen (bspw. „savin“, „swimmin“) verwendet, wurde für den POS-Tagger ein Modell verwendet, welches auf der Grundlage von Twitter-Texten trainiert wurde und gute Ergebnisse für Texte mit nicht-standardisiertem Vokabular und Slang liefert. <?biblio ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISSN" : "13138502", "abstract" : "Part-of-speech information is a pre-requisite in many NLP algorithms. However, Twitter text is difficult to part-of-speech tag: it is noisy, with linguistic errors and idiosyncratic style. We present a detailed error analysis of existing taggers, motivating a series of tagger augmentations which are demonstrated to improve performance. We identify and evaluate techniques for improving English part-of-speech tagging performance in this genre. Further, we present a novel approach to system combination for the case where available taggers use different tagsets, based on voteconstrained bootstrapping with unlabeled data. Coupled with assigning prior probabilities to some tokens and handling of unknown words and slang, we reach 88.7% tagging accuracy (90.5% on development data). This is a new high in PTB-compatible tweet part-of-speech tagging, reducing token error by 26.8% and sentence error by 12.2%. The model, training data and tools are made available.", "author" : [ { "dropping-particle" : "", "family" : "Derczynski", "given" : "Leon", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ritter", "given" : "Alan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Clark", "given" : "Sam", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bontcheva", "given" : "Kalina", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Proceedings of the Recent Advances in Natural Language Processing", "id" : "ITEM-1", "issue" : "September", "issued" : { "date-parts" : [ [ "2013" ] ] }, "note" : "NULL", "page" : "198-206", "title" : "Twitter part-of-speech tagging for all: Overcoming sparse and noisy data", "type" : "article-journal" }, "uris" : [ "http://www.mendeley.com/documents/?uuid=da458a49-791e-3108-b4c5-fecd4eff7ef3" ] } ], "mendeley" : { "formattedCitation" : "(Derczynski et al. 2013)", "plainTextFormattedCitation" : "(Derczynski et al. 2013)", "previouslyFormattedCitation" : "(Derczynski et al. 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "https://github.com/citation-style-language/schema/raw/master/csl-citation.json" }?>