urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:	edition	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1	body	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.1	div1	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.1.1	p	Fiction books are weakly represented in the German Reference Corpus DeReKo (Kupietz et al. 2010, 2018). This is primarily due to the tremendously higher costs associated with licensing and converting raw fiction data into TEI-encoded XML format, compared to newspaper articles (Kupietz et al. 2014, p. 2). In the following, we discuss how we addressed these challenges and successfully created a large extensible corpus of recent fiction books.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.2	div1	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.2.1	p	Linguistics and literary studies both face the challenge that their research data is affected by third-party rights. Obtaining transferable, uniform licenses for German fiction books is particularly costly as no licensing models for non-expressive use (previously also called “non-consumptive use”, see Kamocki 2018) of entire texts as primary research data are generally established. Moreover, individual author permissions are often required, since the use of texts as research data is not covered by standard licensing agreements between authors and publishers.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.2.2	p	Our solution to this challenge consists of two main pillars: The first leverages § 14 of the German National Library Act (DNBG) requiring submission of all digitally published media works to the DNB. The second pillar is part of the general strategy adopted for DeReKo to address legal issues through infrastructural means, following Jim Gray's (2003) famous principle: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">If the data cannot move, put the computation near the data </hi>(Kupietz, Diewald, and Margaretha 2022, 163ff) <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">. </hi>In this application case, our approach is to leave the full texts in the DNB and make them accessible via an instance of the corpus analysis platform KorAP (Bański et al. 2012) running within DNB. This ensures that the data can be made available more openly than under the text and data mining exception, <hi xmlns="http://www.tei-c.org/ns/1.0" rend="sup"><ref n="1" target="ftn1" /> </hi> and be used as effectively as possible for linguistic and related research purposes, but within the limits set by copyright law. The main restriction is that the context size of each search result is limited to a maximum of 50 words.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.3	div1	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.3.1	p	To address the second challenge, and to make conversion into high-quality TEI-XML encoded corpora feasible, we ignored PDF ebooks and limited our focus to the 273,976 books available in the XML-based EPUB format and drew a 10% random sample from these as a first step, stratified by year of publication, resulting in a sample of 26,091 ebooks. As a second step, on the occasion of the 20th anniversary of the German Book Prize in October 2024, we added all 362 digitally available longlisted titles from the past two decades to the corpus, in cooperation with the German Publishers and Booksellers Association. To convert the data to the TEI I5 format (Lüngen and Sperberg-McQueen 2012) used by DeReKo, we applied XSLT 3.0 stylesheets in three passes, via the Saxon XSLT processor and GNU Make, using the DNB SRU API to retrieve consistent metadata, a heuristic genre classifier based on this, and a MALLET (McCallum 2002) based implementation of the standard DeReKo topic domain classifier (Weiß 2005; Klosa, Lüngen, and Kupietz 2012). In subsequent steps, the TEI-XML data was converted to KorAP-XML format and annotated for POS and lemma using the TreeTagger (Schmid 1994), for POS and morphosyntactic properties using MarMoT (Mueller et al. 2013), and dependencies using MaltParser (Nivre et al. 2007). These tools were selected due to their good balance between accuracy and performance. <hi xmlns="http://www.tei-c.org/ns/1.0" rend="sup"><ref n="2" target="ftn2" /> </hi> If required for specific applications, additional annotation layers can be added later, since KorAP supports any number of them. <hi xmlns="http://www.tei-c.org/ns/1.0" rend="sup"><ref n="3" target="ftn3" /> </hi>
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.3.2	p	The entire conversion and annotation process was completed in 48 hours on a Linux server with 96 cores and 1.5 TB of RAM. The composition of the resulting corpus, categorized by genres and publication years, is presented in Figure 1. Genre classifications were derived from the DNB metadata using string matching heuristics. <hi xmlns="http://www.tei-c.org/ns/1.0" rend="sup"><ref n="4" target="ftn4" /> </hi> It is important to note that the relative proportions and the ‘representativeness’ or ‘balance’ of strata are not relevant in the case of DeLiKo@DNB. Instead, only the minimum absolute sizes of the strata are important, as users are invited to define their own virtual subcorpora. This allows them to create stratified, task-specific subsamples based on metadata constraints (see Kupietz et al. 2010; Kupietz 2016 for a detailed account of this <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">primordial sample </hi>approach).
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.3.3	p	<figure xmlns="http://www.tei-c.org/ns/1.0"> <graphic url="Pictures/82059be6af9e6d70f09ff3ccb59b18eb.png" /> <head>Figure 1: DeLiKo@DNB composition by publication year and genre (log scaled)</head> </figure>
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.4	div1	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.4.1	p	DeLiKo@DNB, currently comprising 2.02 billion words, is freely accessible through the website <ref xmlns="http://www.tei-c.org/ns/1.0" target="https://korap.dnb.de/"> <hi rend="color(#1155cc)underline">https://korap.dnb.de</hi> </ref> (see Figure 1), leveraging the full range of KorAP features, such as flexible metadata-based definition of virtual subcorpora, multiple annotation layers, and complex annotation search in six supported query languages, simplified by a query assistant and query-by-match functionalities. The corpus is also accessible via the KorAP API, supported by libraries for R <hi xmlns="http://www.tei-c.org/ns/1.0" rend="sup"><ref n="5" target="ftn5" /> </hi> and Python <hi xmlns="http://www.tei-c.org/ns/1.0" rend="sup"><ref n="6" target="ftn6" /> </hi> (Kupietz, Diewald, and Margaretha 2022).
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.5	div1	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.5.1	p	We aim to regularly expand the corpus by incorporating newly published books. Additionally, we plan to enhance the search and analysis capabilities by integrating advancements from the long-term KorAP project, including updates to the user interface and client libraries. Further additions, improvements, and extensions, concerning e.g. the addition of books, text classifications, or annotation layers, will be driven by the demands of the user communities engaging with DeLiKo@DNB.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:1.5.2	p	<figure xmlns="http://www.tei-c.org/ns/1.0"> <graphic url="Pictures/bc38e9233a3e4d02f9c97410ec9d337e.png" /> <head>Figure 2: KorAP query in DeLiKo@DNB for the lemma 'Herz' (heart) followed by a verb and an adjective, in a virtual corpus containing only horror novels, published between 2010 and 2019</head> </figure>
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2	back	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.1	notes	<note xmlns="http://www.tei-c.org/ns/1.0" xml:id="ftn1" rend="footnote text" n="1">The German TDM implementation (§ 60d UrhG) allows only “to make the corpus available to the public for a specifically limited circle of persons for their joint scientific research“. For the DNB's holdings, this is only possible on the DNB's premises.</note><note xmlns="http://www.tei-c.org/ns/1.0" xml:id="ftn2" rend="footnote text" n="2">The source code of the conversion pipeline is available on GitHub at </note><note xmlns="http://www.tei-c.org/ns/1.0" xml:id="ftn3" rend="footnote text" n="3">For the advantages of maximising recall or precision via multiple classifiers, instead of trying to increase accuracy with just one classifier, see Belica et al. (2011).</note> <note xmlns="http://www.tei-c.org/ns/1.0" xml:id="ftn4" rend="footnote text" n="4">See the genre table in https://github.com/KorAP/epub2korap/blob/main/xslt/epub2i5.xsl </note> <note xmlns="http://www.tei-c.org/ns/1.0" xml:id="ftn5" rend="footnote text" n="5">See https://cran.r-project.org/package=RKorAPClient. An example R script that produces Figure 1 can be found here https://github.com/KorAP/epub2korap/blob/main/scripts/DeLiKoCompositionChart.R. </note><note xmlns="http://www.tei-c.org/ns/1.0" xml:id="ftn6" rend="footnote text" n="6"> https://pypi.org/project/KorAPClient/ </note>
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2	bibliogr	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1	listBibl	
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.1	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Bański, Piotr, Peter M. Fischer, Elena Frick, Erik Ketzan, Marc Kupietz, Carsten Schnober, Oliver Schonefeld, and Andreas Witt</hi>. 2012. “The New IDS Corpus Analysis Platform: Challenges and Prospects.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)</hi>, 2905–11. Istanbul, Turkey: European Language Resources Association (ELRA). <ref xmlns="http://www.tei-c.org/ns/1.0" target="http://www.lrec-conf.org/proceedings/lrec2012/pdf/789_Paper.pdf"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="http://www.lrec-conf.org/proceedings/lrec2012/pdf/789_Paper.pdf" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.2	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Belica, Cyril, Marc Kupietz, Harald Lüngen, and Andreas Witt</hi>. 2011. “The Morphosyntactic Annotation of DeReKo: Interpretation, Opportunities and Pitfalls.” In Grammar & Corpora. Third International Conference, Mannheim, Sept., 22-24 2009, 451–69. Tübingen: Narr. <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://nbn-resolving.org/urn:nbn:de:bsz:mh39-44890" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.3	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Gray, Jim</hi>. 2003. “Distributed Computing Economics.” MSR-TR-2003-24. Microsoft Research.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.4	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Kamocki, Paweł</hi>. 2018. “The Argument for ‘Non-Consumptive Use’ in the EU: How Copyright Could Be Redefined to Allow Text and Data Mining.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Intellectual Property Perspectives on the Regulation of New Technologies</hi>, 237–58. Edward Elgar Publishing. <ref xmlns="http://www.tei-c.org/ns/1.0" target="https://doi.org/10.4337/9781786436382.00016"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://doi.org/10.4337/9781786436382.00016" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.5	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Klosa, Annette, Marc Kupietz, and Harald Lüngen</hi>. 2012. “Zum Nutzen von Korpusauszeichnungen für die Lexikographie.” <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Lexicographica</hi> 28:71–97.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.6	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Kupietz, Marc</hi>. 2016. “Constructing a Corpus.” In The Oxford Handbook of Lexicography, edited by Philip Durkin, 62–75. Oxford: OUP. <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://doi.org/10.1093/oxfordhb/9780199691630.013.5" />
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.7	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Kupietz, Marc, Cyril Belica, Holger Keibel, and Andreas Witt</hi>. 2010. “The German Reference Corpus DeReKo: A Primordial Sample for Linguistic Research.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)</hi>. Valletta, Malta: European Language Resources Association (ELRA). <ref xmlns="http://www.tei-c.org/ns/1.0" target="http://www.lrec-conf.org/proceedings/lrec2010/pdf/414_Paper.pdf"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="http://www.lrec-conf.org/proceedings/lrec2010/pdf/414_Paper.pdf" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.8	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Kupietz, Marc, Nils Diewald, and Eliza Margaretha</hi>. 2022. “Building Paths to Corpus Data: A Multi-Level Least Effort and Maximum Return Approach.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">CLARIN. The Infrastructure for Language Resources.</hi>, edited by Darja Fišer and Andreas Witt. Berlin: deGruyter. <ref xmlns="http://www.tei-c.org/ns/1.0" target="https://doi.org/10.1515/9783110767377-007"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://doi.org/10.1515/9783110767377-007" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.9	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Kupietz, Marc, Harald Lüngen, Piotr Bański, and Cyril Belica</hi>. 2014. “Maximizing the Potential of Very Large Corpora.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Proceedings of the LREC-2014-Workshop Challenges in the Management of Large Corpora (CMLC2)</hi>, edited by Marc Kupietz, Hanno Biber, Harald Lüngen, Piotr Bański, Evelyn Breiteneder, Karlheinz Mörth, Andreas Witt, and Jani Takhsha, ELRA, 1–6. Reykjavik, Iceland. <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://nbn-resolving.org/urn:nbn:de:bsz:mh39-31634" />
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.10	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Kupietz, Marc, Harald Lüngen, Paweł Kamocki, and Andreas Witt</hi>. 2018. “The German Reference Corpus DeReKo: New Developments – New Opportunities.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)</hi>. Miyazaki, Japan: European Language Resources Association (ELRA). <ref xmlns="http://www.tei-c.org/ns/1.0" target="https://www.aclweb.org/anthology/L18-1689"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://www.aclweb.org/anthology/L18-1689" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.11	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Lüngen, Harald, and Christopher M. Sperberg-McQueen</hi>. 2012. “A TEI P5 Document Grammar for the IDS Text Model.” <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Journal of the Text Encoding Initiative</hi>, no. 3, 1–18.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.12	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">McCallum, Andrew Kachites</hi>. 2002. “MALLET: A Machine Learning for Language Toolkit.” <ref xmlns="http://www.tei-c.org/ns/1.0" target="http://mallet.cs.umass.edu/"> </ref> <ref xmlns="http://www.tei-c.org/ns/1.0" target="http://mallet.cs.umass.edu/"> <hi rend="color(#1155cc)underline">http://mallet.cs.umass.edu</hi> </ref>.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.13	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Mueller, Thomas, Helmut Schmid, and Hinrich Schütze</hi>. 2013. “Efficient Higher-Order CRFs for Morphological Tagging.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing</hi>, edited by David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, 322–32. Seattle, Washington, USA: Association for Computational Linguistics. <ref xmlns="http://www.tei-c.org/ns/1.0" target="https://aclanthology.org/D13-1032"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://aclanthology.org/D13-1032" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.14	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Nivre, Joakim, Johan Hall, Jens Nilsson, Atanas Chanev, Gülşen Eryiğit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi</hi>. 2007. “MaltParser: A Language-Independent System for Data-Driven Dependency Parsing.” <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Natural Language Engineering</hi> 13 (June): 95–135. <ref xmlns="http://www.tei-c.org/ns/1.0" target="https://doi.org/10.1017/S1351324906004505"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://doi.org/10.1017/S1351324906004505" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.15	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Schmid, Helmut</hi>. 1994. “Probabilistic Part-of-Speech Tagging Using Decision Trees.” In <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">Proceedings of International Conference on New Methods in Language Processing</hi>, 44–49. Manchester, UK. <ref xmlns="http://www.tei-c.org/ns/1.0" target="https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/tree-tagger1.pdf"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/tree-tagger1.pdf" />.
urn:cts:dhd:2025.kupietz_marc_national_library_as_corpus_introducing_deliko_:2.2.1.16	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">Weiß, Christian</hi>. 2005. “Die Thematische Erschließung von Sprachkorpora.” <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">OPAL - Online publizierte Arbeiten zur Linguistik</hi>, OPAL - Online publizierte Arbeiten zur Linguistik - 2005,1, 2005 (1). <ref xmlns="http://www.tei-c.org/ns/1.0" target="http://pub.ids-mannheim.de/laufend/opal/pdf/opal2005-1.pdf"> </ref> <ptr xmlns="http://www.tei-c.org/ns/1.0" target="http://pub.ids-mannheim.de/laufend/opal/pdf/opal2005-1.pdf" />.