Better Punctuation Prediction with Hierarchical Phrase-Based Translation

Transcription

1 Better Punctuation Prediction with Hierarchical Phrase-Based Translation Stephan Peitz, Markus Freitag and Hermann Ney IWSLT 2014, Lake Tahoe, CA December 4th, 2014 Human Language Technology and Pattern Recognition Lehrstuhl für Informatik 6 Computer Science Department RWTH Aachen University, Germany S. Peitz et al. Punctuation Prediction with Hiero 1 IWSLT December 4th, 2014

2 Outline Introduction Modeling Punctuation Prediction as Machine Translation Hierarchical Phrase-based Translation Experimental Evaluation Conclusion S. Peitz et al. Punctuation Prediction with Hiero 2 IWSLT December 4th, 2014

3 Introduction Spoken language translation (SLT) Automatic speech recognition (ASR) Machine translation (MT) In speech punctuation is not made explicitly ASR systems provide an output without punctuation marks MT systems are trained on data with proper punctuation Reintroduce punctuation marks with monolingual translation Translate from unpunctuated text to text with punctuation Based on phrase-based translation In this work Use hierarchical instead of phrase-based translation Investigation of the optimization criterion S. Peitz et al. Punctuation Prediction with Hiero 3 IWSLT December 4th, 2014

4 Introduction Monolingual translation system More features besides the language model Scaling factors can be tuned Phrase-based translation Sequence of words are translated at once Local contextual information is preserved Useful to predict punctuation depending of its surrounding words (e.g. commas) Limitation: dependencies beyond the local context Hierarchical phrase-based translation Discontinuous phrases with gaps Capture long-range dependencies between words and punctuation marks S. Peitz et al. Punctuation Prediction with Hiero 4 IWSLT December 4th, 2014

5 Modeling Punctuation Prediction as Machine Translation Translation Model Extract from a pseudo-bilingual corpus Take monolingual corpus as source and target text Create monotone alignment Remove punctuation marks from the source text Punctuation marks in the target sentences become unaligned? da du machst was? da du machst was was machst du da? was machst du da S. Peitz et al. Punctuation Prediction with Hiero 5 IWSLT December 4th, 2014

6 Modeling Punctuation Prediction as Machine Translation Optimization Remove punctuation marks from a development set Use the original development set as reference Tune scaling factors with MERT [Och 03] Prediction performance is measured with the F 1 -Score Use F α -Score rather than BLEU as a more suitable optimization criterion F α = (1 + α) (precision recall) α precision + recall By varying α, more emphasis can be put on recall or precision S. Peitz et al. Punctuation Prediction with Hiero 6 IWSLT December 4th, 2014

7 Modeling Punctuation Prediction as Machine Translation Language model Trained on monolingual corpora with proper punctuation Decoding Translate from unpunctuated text to text with punctuation Monotone, no reordering model is necessary In this work Perform prediction before the actual translation Final machine translation system has not to be retrained [Ma & Tinsley + 08, Peitz & Freitag + 11] S. Peitz et al. Punctuation Prediction with Hiero 7 IWSLT December 4th, 2014

8 Hierarchical Phrase-based Translation Allow discontinuous phrases with gaps Obtain phrases from word-aligned bilingual training data Sub-phrases within a phrase are replaced by a generic non-terminal X Maximum of two gaps per rule X über X 0 hinausgehen X 1, go beyond X 0 X 1 Reordering is modelled implicitly Formalized as a synchronous context-free grammar (SCFG) Speaking of rules rather than phrases [Chiang 05] S. Peitz et al. Punctuation Prediction with Hiero 8 IWSLT December 4th, 2014

9 Punctuation Prediction based on Hierarchical Translation Aim: model dependencies between words and punctuation marks e.g. relationship between question word ( was ) and question mark X was X 0, was X 0? X machst du X 0, machst du X 0? Restrictions Performing monotone translation Reordering is not necessary Rules with one non-terminal maximum is sufficient S. Peitz et al. Punctuation Prediction with Hiero 9 IWSLT December 4th, 2014

10 Additional Extraction Heuristic? da du machst was was machst du da X was machst du da, was machst du da S. Peitz et al. Punctuation Prediction with Hiero 10 IWSLT December 4th, 2014

11 Additional Extraction Heuristic? da du machst was was machst du da X was machst du da, was machst du da? S. Peitz et al. Punctuation Prediction with Hiero 11 IWSLT December 4th, 2014

12 Additional Extraction Heuristic? da X~0 was was X~0 X was machst du da, was machst du da? X machst du da, machst du da X was X 0, was X 0? S. Peitz et al. Punctuation Prediction with Hiero 12 IWSLT December 4th, 2014

13 Experimental Evaluation Evaluation of prediction performance Removed punctuation from provided development and test sets (manual transcriptions, no ASR errors) Measurement: Precision, Recall and F 1 -Score Optimization criteria: BLEU and F α -Score with α = {1, 2, 3, 4} Phrase-based (PBT) vs. hierarchical translation (HPBT) Comparison against HIDDEN-NGRAM [Stolcke 02] Evaluated on the IWSLT 2014 translation tasks German English and English French Translation models were trained on indomain data Language model was trained on all available data S. Peitz et al. Punctuation Prediction with Hiero 13 IWSLT December 4th, 2014

14 Prediction Results From unpunctuated German text to German with punctuation marks system tuned on Prec. Rec. F 1 PBT BLEU F F F F HPBT BLEU F F F F HIDDEN-NGRAM HIDDEN-NGRAM outperforms PBT in terms of F 1 HPBT tuned on F 2 works best S. Peitz et al. Punctuation Prediction with Hiero 14 IWSLT December 4th, 2014

15 Analysis Were hierarchical rules used in the decoding process? system tuned on lexical rules hierarchical rules PBT BLEU PBT F HPBT F All applied hierarchical rules introduced punctuation marks S. Peitz et al. Punctuation Prediction with Hiero 15 IWSLT December 4th, 2014

16 Analysis Input "was machst du nur" PBT "was machst du nur." Applied phrases was machst du, was machst du nur, nur. HPBT "was machst du nur?" Applied rules X was, was X machst du X 0, machst du X 0? X nur, nur S. Peitz et al. Punctuation Prediction with Hiero 16 IWSLT December 4th, 2014

17 Impact on Translation Quality Translation tasks: English French German English Tested on enriched manual and automatic transcription Applied baseline phrase-based MT systems trained on all available data Measurement: BLEU S. Peitz et al. Punctuation Prediction with Hiero 17 IWSLT December 4th, 2014

18 Impact on Translation Quality German English WER of automatic transcription: 21.6% transcription manual automatic system tuned on Prec. Rec. F 1 BLEU BLEU PBT BLEU PBT F HPBT F HIDDEN-NGRAM correct punctuation Prediction using HPBT seems to help Only small improvement on automatic transcription S. Peitz et al. Punctuation Prediction with Hiero 18 IWSLT December 4th, 2014

19 Impact on Translation Quality English French WER of automatic transcription: 16.7% transcription manual automatic system tuned on Prec. Rec. F 1 BLEU BLEU PBT BLEU PBT F HPBT F HIDDEN-NGRAM correct punctuation Prediction using monolingual MT systems works best Mixed results on automatic transcription S. Peitz et al. Punctuation Prediction with Hiero 19 IWSLT December 4th, 2014

20 Conclusion Punctuation prediction based hierarchical translation Capture long-range dependencies between words punctuation marks Improvements in terms of Precision, Recall and F 1 -Score Small impact on translation quality Use F α -Score as optimization criterion Future work Investigate features operating on sentence level Enrich grammar with syntactical information S. Peitz et al. Punctuation Prediction with Hiero 20 IWSLT December 4th, 2014

21 Thank you for your attention Stephan Peitz S. Peitz et al. Punctuation Prediction with Hiero 21 IWSLT December 4th, 2014

22 References [Chiang 05] D. Chiang: A Hierarchical Phrase-Based Model for Statistical Machine Translation. pp , Ann Arbor, Michigan, June [Ma & Tinsley + 08] Y. Ma, J. Tinsley, H. Hassan, J. Du, A. Way: Exploiting Alignment Techniques in MaTrEx: the DCU Machine Translation System for IWSLT08. In Proc. of the International Workshop on Spoken Language Translation, pp , Hawaii, USA, [Och 03] F.J. Och: Minimum Error Rate Training in Statistical Machine Translation. pp , Sapporo, Japan, July [Peitz & Freitag + 11] S. Peitz, M. Freitag, A. Mauser, H. Ney: Modeling Punctuation Prediction as Machine Translation. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA, Dec [Stolcke 02] A. Stolcke: SRILM-An extensible language modeling toolkit. In In Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP 2002), pp , S. Peitz et al. Punctuation Prediction with Hiero 22 IWSLT December 4th, 2014

23 The Blackslide GoBack