urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:	edition	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1	body	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.1	div	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.1.1	p	for more than a decade, republican magazines and newspapers have been collected by institutes and projects now joined in the centre for asian and transcultural studies (cats) at heidelberg university. our platform “early chinese periodicals online” (ecpo, https://uni-heidelberg.de/ecpo), provides open access to more than 300.000 digital images and their metadata, cf. arnold and hessel (2020) . since the material consists mostly of image scans, the project ran a number of experiments to explore possible approaches towards full text generation (arnold, 2021). for newspapers printed in latin scripts much has changed since rose holley commented item “use the ‘training’ facility (artificial intelligence) in the ocr software” with “not viable for cost effective mass scale digitization” and noted “do not pursue” in her list of “potential methods of improving ocr accuracy suggested by andp team” (holley, 2009, table 2, item 9). today, when researchers write that “transforming [historical newspapers] into machine-readable data by means of ocr poses some major challenges” they do that while they introduce their own ocr pipeline (holley, 2009).
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.1.2	p	unfortunately, these approaches cannot just be adopted to historical chinese newspapers. as we have shown, especially complex layout and resulting difficulties of reliable automatic page segmentation have so far prevented full text generation of these newspapers even within china (arnold, 2021; arnold, forthcoming; arnold et al., forthcoming). in this long abstract we present the first results from a systematic approach towards full text extraction from a republican china newspaper ( 1). our basis is a small corpus for which text ground truth exists. we present our character segmentation method which produces about 90.000 images of characters. based on the hypothesis that pre-training on extensive amounts of suitably augmented character images will increase the ocr accuracy for evaluation on real-life character image data, we generate synthetic training data. we then compare the ocr recognition results and show that a combination of synthetic and real characters produces best results. finally, we propose a method that makes use of a masked language model for ocr error correction.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.1.3	p	note: we will treat single rectangular text blocks (fig. 2) as given and proceed from here to present effective methods for generating a data set later used to train an ocr model. we show that pre-training on artificially created training data can significantly improve ocr accuracy. due to the limited scope of the presented experiments, this approach is still limited in terms of retrieved glyph size, image quality and font style, hence the model is not necessarily directly applicable to other historical chinese documents.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.2	div1	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.2.1	p	our corpus consists of 9.385 scanned folds from the entertainment newspaper jing bao <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">晶報</hi> (the crystal), published 03.03.1919–23.05.1940 (fig. 1). the double-keyed text ground truth comprises all april 1939 issues (40 folds, ~245.000 characters). aside from text blocks and their headings, it also contains mastheads, advertisements and marginalia, however, the methods presented below will solely focus on “header-less” text blocks of uniform font-size.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3	div1	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.1	p	due to chinese characters’ nearly squared appearance, it is common to find resulting text blocks implicitly displaying a grid layout (see fig. 2). deviation from the grid usually appears when additional characters had to be squeezed into one column or because of inaccurate printing. in order for the method described below to work, we manually sort out any text blocks that don’t adhere to the grid layout and then extract the corresponding ground truth section for every crop.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.2	p	after adaptive binarization (kernel size: 125 px) we calculate horizontal and vertical projection profiles, cf. fan et al. (1998). to perform deskewing, we find an angle <hi xmlns="http://www.tei-c.org/ns/1.0" rend="math">α</hi> with <hi xmlns="http://www.tei-c.org/ns/1.0" rend="math">α</hi> <hi xmlns="http://www.tei-c.org/ns/1.0" rend="math">∈</hi> [-2.0,-1.5,...,2.0] such that rotating the image by <hi xmlns="http://www.tei-c.org/ns/1.0" rend="math">α</hi> maximizes
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.3	p	where w and h are the width and height of the image, c<hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic subscript">i</hi> is the number of black pixels in the <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">i</hi>-th column (= the corresponding value of the vertical projection profile) and l<hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic subscript">j</hi> in the <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">j</hi>-th line.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.4	p	after deskewing, we cut the gray-scale, non-binarized original text block image into single character images along separators defined by the following heuristic:
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.5	p	(1) use the valleys of the vertical projection profile to define separators between the columns.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.6	p	(2) use the valleys of the horizontal (global) projection profile to define separators between the lines.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.7	p	(3) for every column, produce another (local) projection profile. if a local separator lies within 7px distance of a global separator defined by (2), discard the global separator and only use the local separator; else only use the global separator.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.8	p	the positions of the valleys are obtained by scipy.signal.find_peaks using a minimum distance of (1) 22, (2) 20 and (3) 14.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.9	p	for normalization and contrast enhancing the following method is used:
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.10	p	finally, the resulting fields can be easily mapped to the ground truth text. indentations have to be manually marked, and since the cnn (cf. section 3.) requires squared images as input, we add white padding to transform the rectangular character images into square ones.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.3.11	p	this method entirely relies on correct annotation. while we can easily detect errors like missing lines, this is harder for missing or extra characters within a line (checking the line length), and basically impossible for typos or swapped characters. to avoid such mistakes we can only double-check annotations, otherwise they lower recognition accuracy.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.4	div1	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.4.1	p	the method described in the section above yields a total of 92.039 character images (47.986 train + 21.676 dev + 22.377 test). due to the zipfian distribution, we additionally present the following table:
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.4.2	p	motivated by the low quantity of training samples for higher x, we generate additional synthetic training data and propose the following research hypothesis:
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.4.3	p	pre-training on extensive amounts of suitably augmented character images will increase the ocr accuracy for evaluation on real-life character image data.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.4.4	p	with the goal of imitating the real-life character images with artificial training data, we apply the following, partly randomized (in b., e2.2, f., g., and h.) augmentations to glyph images extracted from various fonts:
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.4.5	p	since ultimately, the classes used for ocr are unicode points, the question arises which code points to synthesize additional training data from. we employ the simple heuristic of using all of the glyphs featured in the ground truth, and adding any missing ones from the 4000 most frequent characters of a representative corpus. furthermore, inconsistencies caused by han-unification have to be solved. for example, the image data features <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">靑</hi> instead of <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">青</hi> and <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">淸</hi> instead of <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">清</hi> (all different code points), however only one code point exists for every other character containing <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">靑</hi>/ <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">青</hi> as a component ( <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">請</hi>, <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">情</hi>, <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">靜</hi>, …). while <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">值</hi> and <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">値</hi> (the latter being the variant used in our image data) have different code points, their right component itself ( <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">直</hi>) is han-unified, etc. we decide to always use the most accurate code point as long as it’s not part of the cjk compatibility ideographs block (u+f900...u+faff), so e.g. <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">令</hi> (u+4ee4) is used instead of <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">令</hi> (u+f9a8), even though the latter might appear more accurate, depending on the font. generally, we find that the character variants printed in our image data to be visually closer to the japanese standard (e.g. the components <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">爫</hi> and <hi xmlns="http://www.tei-c.org/ns/1.0" rend="chinese">⺬</hi>), so we choose several japanese fonts for training data generation.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.5	div1	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.5.1	p	we decide on using a googlenet cnn architecture (szegedy et al., 2015), slightly modified to take 1-channel inputs instead of rgb-images. this has proven to be effective regarding both printed and handwritten chinese character recognition, e.g. zhong et al. (2015) and xu et al. (2018). training on different character image sets, we obtain the following top-k accuracies on the real-life validation set for <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">k</hi> <hi xmlns="http://www.tei-c.org/ns/1.0" rend="math">∈</hi>{1,...,10}:
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.5.2	p	we also find that the selection of the fonts by which variants (i.e. mainland chinese, taiwanese, japanese, korean) it was designed for is largely negligible, i.e. a taiwanese font may score higher than a japanese font, even though the latter features glyph variants closer to those found in our data. this is probably because the percentage of characters with regional variants is relatively small, and also implies that the characters’ stroke length and distance as well as small variations in the size of single character components is more relevant to the ocr accuracy when evaluating on real-life character images.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.5.3	p	interestingly, while there is a huge difference in performance after training on synthetic vs. real data, the human eye is barely able to differentiate between even a big selection of synthetic and real character images if presented next to each other (cf. fig. 5).
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.6	div1	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.6.1	p	finally, we aim to improve top-1 accuracy values by using language models to find the correct character among the second to k-th prediction. as can be seen in the table in section 3, there is a significant jump from top-1 to top-2 accuracy, meaning that for wrong predictions the gold character is often predicted in the second position.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.6.2	p	inspired by wang et al. (2019), we propose a method that identifies characters likely to be incorrect: let x1 and x2 denote the logit scores of the top 2 candidates output by the ocr model. now we set a threshold t for the difference between x1 and x2. any ocr prediction where x1 − x2 &lt; t is treated as likely to be incorrect and is passed on to the correction step. this step works by having a pre-trained bert model re-predict the character from the top k ocr candidates. systematically testing for different combinations of t and k (with t <hi xmlns="http://www.tei-c.org/ns/1.0" rend="math">∈</hi> [0,0.5, . . . ,10] and k <hi xmlns="http://www.tei-c.org/ns/1.0" rend="math">∈</hi> [0,1, . . . ,18]), we settle with t = 2.5 and k = 7, where we attain the following final results:
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:1.6.3	p	as becomes evident, the presented post-processing method reduces the error by 18.1% (dev. set) / 16.1 % (test set).
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2	back	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1	bibliogr	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1	listBibl	
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.1	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">arnold, matthias</hi> (2021): <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">ground truth, neural networks, ocr: towards full text of republican china newspapers</hi>. https://tinyurl.com/ecpo-intro [letzter zugriff 15. juli 2021].
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.2	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">arnold, matthias</hi> (forthcoming): "multilingual research projects: challenges for making use of standards, authority files, and character recognition", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">digital studies / le champ numérique.</hi>
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.3	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">arnold, matthias / hessel, lena</hi> (2020): "transforming data silos into knowledge: early chinese periodicals online (ecpo)", in: heuveline, vincent / gebhart, fabian / mohammadianbisheh, nina (hrsg.): <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic" xml:space="preserve"> e-science-tage 2019: data to knowledge</hi>. heidelberg: heibooks. s. 95–109. 10.11588/heibooks.598.c8420.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.4	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">arnold, matthias / paterson, duncan / xie, jia</hi> (forthcoming): "procedural challenges: machine learning tasks for ocr of historical cjk newspapers", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">international journal of digital humanities.</hi>
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.5	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">fan, kuo-chin / wang, liang-shen / tu, yin-tien</hi> (1998): "classification of machine-printed and handwritten texts using character block layout variance", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">pattern recognition</hi> 31, s. 1275–1284.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.6	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">holley, rose</hi> (2009): "how good can it get? analysing and improving ocr accuracy in large scale historic newspaper digitisation programs", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">d-lib magazine</hi> 10.1045/march2009-holley.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.7	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">liebl, bernhard / burghardt, manuel</hi> (2020): "from historical newspapers to machine-readable data: the origami ocr pipeline", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">proceedings of the workshop on computational humanities research (chr 2020).</hi> amsterdam, the netherlands. s. 351–373. (= ceur workshop proceedings). http://ceur-ws.org/vol-2723/long20.pdf [letzter zugriff 15. juli 2021].
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.8	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">sung, doris / sun, liying / arnold, matthias</hi> (2014): "the birth of a database of historical periodicals: chinese women’s magazines in the late qing and early republican period", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic" xml:space="preserve">tulsa studies in women’s literature </hi>33, s. 227–237.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.9	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">szegedy, christian et al</hi>. (2015): "going deeper with convolutions", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">2015 ieee conference on computer vision and pattern recognition (cvpr).</hi> boston, ma. s. 1–9. 10.1109/cvpr.2015.7298594.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.10	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">wang, hsiang-an / liu, pin-ting</hi> (2019): "towards a higher accuracy of optical character recognition of chinese rare books in making use of text model", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">proceedings of the 3rd international conference on digital access to textual cultural heritage.</hi> new york, ny, usa: association for computing machinery. s. 15–18. (= datech2019). 10.1145/3322905.3322922.
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.11	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">xu, xin et al.</hi> (2018): "chinese characters recognition from screen-rendered images using inception deep learning architecture", in: zeng, bing et al. (hg.): <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">advances in multimedia information processing – pcm 2017.</hi> cham: springer international publishing. s. 722–732. (= lecture notes in computer science).
urn:cts:dhd:2022.arnold_matthias_building_and_improving_an_ocr_classifier_for:2.1.1.12	bibl	<hi xmlns="http://www.tei-c.org/ns/1.0" rend="bold">zhong, zhuoyao / jin, lianwen / xie, zecheng</hi> (2015): "high performance offline handwritten chinese character recognition using googlenet and directional feature maps", in: <hi xmlns="http://www.tei-c.org/ns/1.0" rend="italic">2015 13th international conference on document analysis and recognition (icdar).</hi> tunis, tunisia. s. 846–850. 10.1109/icdar.2015.7333881.