Evaluating Translations Produced By Amazon Mechanical Turk

1940 words - 8 pages

Abstract
We investigate the use of Amazon Mechanical Turk for the creation of translations from English to Haitian Creole. The intention is to produce a bilingual corpus for Statistical Machine Translation. In several experiments we offer varying amounts of money for the translation tasks. The current results show that there is no clear correlation between pay and the translation quality. Almost all translations show a significant overlap with online translation tools which indicates that the workers did often not translate the sentences themselves.
1 Introduction
Our group is currently involved in the development of an English↔Haitian Creole translation system for use in the earthquake region of Haiti. One of the current tasks is the rapid production of a bilingual English↔Haitian Creole in-domain medical dialogue corpus to be able to train a Statistical Machine Translation system. Some native Haitian Creole speakers volunteered to help with translations and we also intend to use professional translators to support the effort.
Amazon’s Mechanical Turk (AMT) is an interesting alternative here as it would be cheaper than using professional translators. This is especially relevant for an English↔Haitian Creole translation system as the commercial potential is probably limited.
One of the main concerns with using AMT for NLP tasks, especially translation is the quality of the resulting data and the availability of workers with Haitian Creole knowledge.
The experiments in this paper address these concerns and evaluate the translations produced by Amazon Mechanical Turk compared with professionals and unpaid volunteers. We investigate the overall quality of the produced translations and compare the translations done at different pay rates.
2 Experiments
2.1 Experimental setup
The data that was used for the initial translation experiments consisted of 100 lines of short English sentences from typical medical dialogues (see example sentences in Table 1).

Do you have drinking water?
Do you require medical assistance?
When can I start traveling again?
I need something for sunburn.
The ankle is swollen.
Has there been any change in your appetite?
I've had a heart attack before.
I am going to give you an injection.
Have you had any surgeries before?
I'm allergic to certain medicines.
Table 1: Example sentences

Overall the 100 sentences consisted of 430 words. Only 100 sentences were initially used as it was unclear if any translations to Haitian Creole could be produced using Amazon Mechanical Turk. It was also not possible to locate a professional translator for Haitian Creole yet. Results for professional translations will be included in the final paper.
Volunteer translations for all 100 sentences were collected from http://translation4haiti.org. This was setup during crisiscamp (http://crisiscommons.org) to allow Haitian Creole native speakers to contribute and collect translations.
2.2 Producing translations with AMT
...

Find Another Essay On Evaluating Translations Produced by Amazon Mechanical Turk

External & Internal Analysis of Amazon during 1997. Example of a Strategic Management in the New Economy.

2445 words - 10 pages weakness.Distribution and Operation channels are not fully in controlled by the firm. From a functional analysis standpoint, Amazon has one large glaring flaw in that it relies heavily on shipping channels to get their product to the customer. Since this is handled by an outside company, a large portion of the quality of service is taken out of Amazon's hands. The shipping company can be late, lose the customer's package or have any other number of

Empirical Data Results Essay

1233 words - 5 pages made a contract of $29 million with Aptara for data and content conversion. Furthermore, Aptara has earned a huge popularity for transferring the children’s books being produced by Candlewick press and Walker books to Amazon Kindle. Code Mantra: Code Mantra is an American based company which is located in Philadelphia and it was established in 2002. It provides very unique services for e-book production such as asset changing facility

An Analysis of the Epic Poem, Beowulf - Origin and Evolution of Beowulf

1659 words - 7 pages glorified Pagan tale by using ambiguous imagery such as the unclear "fire's embrace" (36).  In evaluating these words, a reader must decide if the speaker is alluding to the fires of Hell or the fires of Pagan sacrifices.  Although the speaker tries to fuse his conflicting religious views with the story of Beowulf, when the excerpt is juxtaposed against the entire Pagan text the startling contrast affirms my conclusion that this excerpt is a Christian

Continental Margins

673 words - 3 pages Continental Margins What is weathering and erosion? Weathering is basically the breakdown and alteration of rocks and minerals at or close to the Earth’s surface into products that are more in equilibrium with the conditions found in its environment. Erosion is “the removal of weathered rocks by moving water, wind, ice, or gravity” (Thomas and Turk). These two processes together help make the Earth’s surface how it is today. Weathering is a

Destruction of the Amazon Rainforest

2051 words - 8 pages discovery of oil the amount of land used for growing crops increase by 350 percent and two million hectares of land was cleared from cattle. After this point, the conversion of natural ecosystems into farms and ranches spread into neighboring countries and continues to at a rate of two percent per year. Indigenous groups such as the Yanomamö and Kayapo have been living in the Amazon for thousands of years, slowly accumulating a detailed knowledge of

Evaluation of Deforestation in the Brazilian Amazon

2577 words - 10 pages ). Secondly, as has been said Amazon has rich biodiversity, which could extinct because of logging and burning. It is estimated that about 20 million years ago, Amazon rainforest started to exist, where the sustainable environment was established and almost all species of flora and fauna are connected to each other (Hoorn et al. 2010). Even indigenous people have been trying to keep sustainability of Amazon by experimenting their home place for about 5

Cloud Computing Defined

2241 words - 9 pages cloud-based services which included storage, computation and even human intelligence through the Amazon Mechanical Turk. In 2006, Amazon then launched its Elastic Compute cloud (EC2), as a commercial web service that allowed small companies and individuals to rent computers to run their own computer applications. An additional milestone achievement came in 2009 when Web 2.0 hit its big mark and Google and other software companies started to

Traveling in the Amazon

2509 words - 10 pages the Amazon River, while enjoying a small trip to a single country or touring the whole river from beginning to the end. It is also for grandparents who want to surprise their grandchildren with an unforgettable geography lesson that your grandchildren will treasure forever. This is for those who want to understand the mysticism developed by a place that has remained virtually unchanged for thousands of years. This journey is also for ladies that

Deforestation as a Global Issue

2101 words - 8 pages government for cattle ranching as the land in the Amazon is extremely cheap for these billion dollar companies. Commercial logging is cutting trees for sale as timber or pulp. Logging can occur selectively so only the economically valuable species are cut or by clear-cutting all the trees. Commercial logging uses heavy machinery, such as bulldozers and trucks to transport the logs to processing factories and for export

E-Business Analysis Paper

6616 words - 26 pages lanzado varios programas y productos los cuales son exclusivos de la compañía, como Amazon Mechanical Turk, Amapedia, Amapedia, Amazon MP3, Amazon Vine, Amazon Kindle etc. En adición Amazon provee herramientas para que sus clientes expresen su conformidad con los productos adquiridos a través de ellos y a su vez pueden ser evaluados los productos por futuros clientes.iTunes Antes de comenzar a comunicar el look and

Imperialism in the middle east

1341 words - 5 pages impossible to colonize. --Pan-Slavism: Slavic nationalist movement. Favored and patronized by Russian government because it diverted attention from domestic problems and inadequacies. Slavs in Ottoman empire = willing to use Pan-Slavism as means for combatting their Turk rul ers. Revolution broke out in Bulgaria and Bosnia in 1875,76, and Russia declared war on Turkey in 1877. --British = again willing to fight the Russians: had to protect the Suez

Similar Essays

Bezos Case Study

7622 words - 30 pages management report with references. Therefore, please feel free to use headings and bullet points.Good luckDr Stuart PaulSeptember, 2014Inside Amazon's Idea Machine: How Bezos Decodes Customers by George AndersComment NowFollow Comments Following Comments Unfollow Comments This story appears in the April 23rd, 2012 issue of FORBES magazine. A few months ago Amazon reached what its founder and CEO Jeff Bezos demurely tells me was "an interesting

The Mystery Of Media Ideal Body Image: Influences On People’s Appearance Self Esteem

2416 words - 10 pages behavior among Japanese female and male teenagers. Journal Of Communication, 59(2), 279-295. doi:10.1111/j.1460-2466.2009.01416.x Lyubomirsky, S. & Ross, L. (1997). Hedonic consequences of social comparison: A contrast of happy and urihappy people. Journal of Personality and Social Psychology, 73, 1141-1151. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411

Field Day Group And Women Victimization

1013 words - 4 pages more important issues of nationalism. The Field Day group which is supposed to “rethink ideas about the nation, literature, politics and culture” fails to include the issues of women in its agenda. Onkey argues that Field Day simply ignores the creative works produced by women and topics of women, sexuality and gender are absent in most literary discourse. Female writers are also marginalized since “of over 300 writers included covering 1500 years

The Use Of Alternative Energy Essay

1718 words - 7 pages rotation of turbine blades is transferred into electrical currents by means of an electrical generator. Wind turbines are constructed in numbers on wind farms. Wind farms are able to generate electrical currents on a large scale. National electrical grids use wind farms as well as small individual turbines used for providing electricity to isolated locations or individual homes (Turk & Bensel, 2011). According to Journal for Nature Conservation In