We investigate the use of Amazon Mechanical Turk for the creation of translations from English to Haitian Creole. The intention is to produce a bilingual corpus for Statistical Machine Translation. In several experiments we offer varying amounts of money for the translation tasks. The current results show that there is no clear correlation between pay and the translation quality. Almost all translations show a significant overlap with online translation tools which indicates that the workers did often not translate the sentences themselves.
Our group is currently involved in the development of an English↔Haitian Creole translation system for use in the earthquake region of Haiti. One of the current tasks is the rapid production of a bilingual English↔Haitian Creole in-domain medical dialogue corpus to be able to train a Statistical Machine Translation system. Some native Haitian Creole speakers volunteered to help with translations and we also intend to use professional translators to support the effort.
Amazon’s Mechanical Turk (AMT) is an interesting alternative here as it would be cheaper than using professional translators. This is especially relevant for an English↔Haitian Creole translation system as the commercial potential is probably limited.
One of the main concerns with using AMT for NLP tasks, especially translation is the quality of the resulting data and the availability of workers with Haitian Creole knowledge.
The experiments in this paper address these concerns and evaluate the translations produced by Amazon Mechanical Turk compared with professionals and unpaid volunteers. We investigate the overall quality of the produced translations and compare the translations done at different pay rates.
2.1 Experimental setup
The data that was used for the initial translation experiments consisted of 100 lines of short English sentences from typical medical dialogues (see example sentences in Table 1).
Do you have drinking water?
Do you require medical assistance?
When can I start traveling again?
I need something for sunburn.
The ankle is swollen.
Has there been any change in your appetite?
I've had a heart attack before.
I am going to give you an injection.
Have you had any surgeries before?
I'm allergic to certain medicines.
Table 1: Example sentences
Overall the 100 sentences consisted of 430 words. Only 100 sentences were initially used as it was unclear if any translations to Haitian Creole could be produced using Amazon Mechanical Turk. It was also not possible to locate a professional translator for Haitian Creole yet. Results for professional translations will be included in the final paper.
Volunteer translations for all 100 sentences were collected from http://translation4haiti.org. This was setup during crisiscamp (http://crisiscommons.org) to allow Haitian Creole native speakers to contribute and collect translations.
2.2 Producing translations with AMT