{"id":35755,"date":"2021-06-08T10:06:48","date_gmt":"2021-06-08T10:06:48","guid":{"rendered":"https:\/\/wp-staging.amberscript.com\/?p=35755"},"modified":"2023-06-09T10:25:42","modified_gmt":"2023-06-09T08:25:42","slug":"dutch-punctuation-model","status":"publish","type":"post","link":"https:\/\/wp-staging.amberscript.com\/en\/blog\/dutch-punctuation-model\/","title":{"rendered":"Dutch punctuation model"},"content":{"rendered":"\n

Grammatical aspects like gerund, prepositions, and basic grammar rules play an important role in most known languages. Have you ever thought about the fact that punctuation also plays a critical part?
Punctuation matters in language. It implies the correct arrangement of small, sometimes hardly noticeable marks in the appropriate places to indicate the exact length and the meaning of the sentence.
In the following text, we’ll take a closer look into the AI punctuation model we have developed for our Dutch language speech recognition model.<\/em><\/p>\n\n\n\n

The role of punctuation in language<\/h2>\n\n\n\n

Punctuation is an integral part of written text and helps in making text intelligible and coherent. The absence of punctuation hampers readability and can make texts incomprehensible. Furthermore, punctuation marks reduce ambiguity. Consider this example where a comma can completely alter the meaning of a sentence:<\/p>\n\n\n\n

\n

\u201cMost of the time travellers worry about their luggage\u201d<\/em><\/p>\n<\/blockquote>\n\n\n\n

vs<\/p>\n\n\n\n

\n

\u201cMost of the time, travellers worry about their luggage\u201d<\/em><\/p>\n<\/blockquote>\n\n\n\n

Missing punctuation can also lead to awkward sentences, as in this classic example:<\/p>\n\n\n\n

\n

\u201cI find inspiration in cooking my family and my dog\u201d<\/em><\/p>\n<\/blockquote>\n\n\n\n

Punctuation in speech-to-text<\/h2>\n\n\n\n

Therefore, speech-to-text systems must include punctuation when they produce a transcript. Typical automatic speech recognition (ASR) systems, however, do not output punctuation marks since they don\u2019t have a spoken form. Furthermore, the generated transcript is composed of only lowercase words, making it difficult to understand. A properly punctuated transcript also aids in the automatic generation of subtitles for videos.<\/p>\n\n\n\n

This problem can be solved by incorporating a separate punctuation model that can automatically add punctuation to the output from an ASR model. It can be cast as a natural language processing (NLP) problem where the goal is to predict the punctuation mark (or the lack thereof) for every word in a transcript.<\/p>\n\n\n\n

Language models<\/h2>\n\n\n\n

Deep learning has witnessed tremendous progress in the last few years, fuelled by the increase in computational power. The field of NLP was taken by storm by the introduction of BERT<\/a> in 2018. Developed by Google AI, BERT is a large language model based on the transformer architecture<\/a>. It was touted as NLP\u2019s ImageNet moment<\/a>, referring to how ImageNet steered progress in representation learning from images in the field of computer vision. BERT is a marked improvement over earlier language representation models such as GloVe<\/a> embeddings, and contextual representations such as ELMo<\/a>.<\/p>\n\n\n\n

For an intuitive explanation of how BERT works, refer to this excellent blog post by Jay Alammar<\/a>. In simple terms, it is trained on raw texts in a self-supervised manner, i.e., without human annotations. Specifically, it is trained on two tasks \u2014 masked language modeling and next sentence prediction. At the end of the training, the model is said to be \u201cpre-trained\u201d and captures the semantics of language with its word and sentence representations. A pre-trained BERT can then be fine-tuned on a downstream NLP task. When it was published, BERT produced state-of-the-art results after fine-tuning on a range of NLP tasks, including natural language inference (NLI), question answering, etc.<\/p>\n\n\n\n

Punctuation model at Amberscript<\/h2>\n\n\n\n

At Amberscript, we develop custom ASR models, one of them for Dutch. As noted before, the transcripts produced by the model lack any punctuation marks. Currently, there are no open-source punctuation models available that are specific to the Dutch language. Therefore, we developed a punctuation model based on BERT to automatically add the following punctuation marks \u2014 question mark, period, exclamation mark, comma, colon, and semicolon. Other punctuation marks that occur in pairs, such as quotation marks and parentheses are much more difficult to determine solely based on the text.<\/p>\n\n\n\t

\n\t\t\n\t
\n\n