This website uses cookies to ensure you have the best experience. Learn more

Learning Orthographic Structure With Sequential Generative Neural Networks

1785 words - 8 pages

In this paper we modeled the process of learning orthographic structure from sequences of letters using a recently proposed generative neural network, the Recurrent Temporal Restricted Boltzmann Machine (RTRBM; Sutskever et al., 2008). We showed that this sequential network is able to learn the structure of English monosyllables in a completely unsupervised fashion, by only trying to accurately reproduce input wordforms presented one letter at a time.
We first demonstrated that the RTRBM successfully learned graphotactics by testing its performance on a prediction task, where the initial part of a word was given as a context and the network predicted the probability distribution of the following letter. The RTRBM yielded a prediction performance comparable to that of a simple recurrent network (SRN), which is the most widely used connectionist architecture to model sequential data. We also compared the RTRBM with other popular (non-connectionist) probabilistic generative models, that is Hidden Markov Models and n-grams, which constitute the state-of-the-art in several sequence learning tasks but do not provide any insight in terms of the underlying neural computation. We then assessed the generative ability of the considered models, by letting them to autonomously produce sequences of letters and measuring their well-formedness. In particular, we calculated their averaged length, orthographic neighborhood and constrained bigram and trigram frequencies, and we used these indicators to compare the quality of generated pseudowords with that of two pseudoword generators used in psycholinguistic studies, namely the ARC nonwords database (Rastle et al., 2002) and Wuggy (Keuleers & Brysbaert, 2010). We found that the RTRBM produced very high quality pseudowords, thus confirming that it correctly learned the orthographic structure of English monosyllables. In this regard, it is worth noting that the results of our model should readily extend to other alphabetic languages.
In order to allow autonomous generation of sequences with the SRN, which is inherently deterministic and input-driven (i.e., bottom-up), we had to extend its basic formulation. We therefore presented a stochastic variant of SRNs, in which the activations of the output units are first normalized in order to be treated as conditional probability distributions. An external stochastic process is then used to sample the next element of the sequence, which is fed back as input to the network at the following timestep. Interestingly, the pattern of results obtained with the RTRBM was very similar to that obtained with the stochastic variant of SRNs. On the one hand, this is not surprising because both connectionist models try to predict the next element of a sequence by learning conditional probabilities from the training data. Indeed, there is a tight formal relationship between probabilistic graphical models and recursive neural networks (Baldi & Rosen-Zvi, 2005). However, the two...

Find Another Essay On Learning Orthographic Structure with Sequential Generative Neural Networks

The Main Characteristics and Basic Differences of Generative Grammar and Usage-Based Approaches

551 words - 3 pages . Overall, Chomsky's universal grammar is a principle of the language faculty embedded in the human mind and generative grammar is based on the principles of syntactic rules. However, usage-based approaches build upon the theory of language learning with input, mental representation, frequency, and categorization. Due to their dissimilar theories and approaches towards language acquisition, they have sided against each other rather than to work together. Works Cited Herbst, Thomas (2010). English Linguistics: A Coursebook for Students of English. Berlin: De Gruyter Mouton.

Effects of Prior Knowledge on Generative Tasks

2963 words - 12 pages condition than in the other two (Marsh, Ward, et al., 1999). Therefore, the evidence suggests that those who were presented with a strict set of word structure retained that structure in their creation of new words. These results appear to mirror those of other experiments testing the effects of examples and constraints on creativity. While the constraining effects on creativity when provided examples just prior to the generative task

Artificial Neural Networks

1124 words - 4 pages artificial neural networks. Before understanding the complex structure of an artificial neural network, a rudimentary knowledge of an organic neural network is essential. The human brain consists of over billions of neurons interconnected by trillions of synapses. Neurons exchange electrical impulses through the synapses attained from other neurons or from the senses. When something novel is experienced, these neurons create new connections

Data Mining Techniques for Customer Relationship Management

2184 words - 9 pages network technology. As described by Mitra and Acharya (2003), neural networks models implement an important pattern recognition system. The main benefit of this technology is that “learning” is done through exampling the discriminating characteristics among various pattern classes, automatically determining inherent relationships in a data environment which subsequently reduces the error. The system developed by NeovistaSolutions enabled the

Neural Networks in Investments

2692 words - 11 pages have also connections between neurons of the same layer. A selection of important Neural Networks are as follows: i.) Perceptron: The Perceptron was first introduced by F. Rosenblatt in 1958. It is a very simple neural net type with two neuron layers that accepts only binary input and output values (0 or 1). The learning process is supervised (A specific type of a learning algorithm. The output (pattern) of the net is compared with a target

Machine Learning

2808 words - 11 pages regression tree techniques become popular during the 1980s (Brieman et al. 1984) and with the help of powerful computers and large digital data resources they became the backbone of the modern machine learning. In 1990s neural networks and decision trees were adapted to the larger fields of data mining and statistics. Today Machine learning is a mathematically rigorous discipline that encompasses sophisticated modeling, optimization, and learning

Voting Based Neural Network: Extreme Learning Machine

1057 words - 5 pages -3062 5. Feng G, Huang G-B, Lin Q, Gay R (2009) Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning. IEEE Trans Neural Netw 20(8):1352-1357 6. Yang Y, Wang Y, Yuan X (2012) Bidirectional Extreme Learning Machine for Regression Problem and Its Learning Effectiveness. IEEE Trans Neural Netw 23(9):1498-1505 7. Lan Y, Soh YC, Huang G-B (2009) Ensemble of online sequential extreme learning machine

Linear and Non-linear Quantitative Structure – Activity Relationship Models on Indole Substitution Patterns as Inhibitors of HIV-1 Attachment

2221 words - 9 pages and GA-ANN models, respectively (N=40). Keywords: Genetic algorithm; artificial neural network; multiple linear regressions; HIV; Quantitative Structure – Activity Relationship; 1. Introduction The process of human immunodeficiency virus-1( HIV-1) entry into host cells, offers considerable potential for therapeutic intervention, with viral entry proceeding through multiple sequential steps involving attachment

Artificial Neural Network for non-Linear Dynamic Process of a Cyclone Scrubber

1469 words - 6 pages neural network architecture have been proposed. One pre- dominant architecture is feed-forward neural network (FFNN). In this study, the three-layer feed-forward neural network (3-FFNN) is used as shown in Figure 2. A more detailed description of 3-FFNN will be found in many textbooks that have been published on neural networks. Figure 2 The structure of three layer feed- forward neural network (3-FFNN) Many studies have proved the

A Novel Neuro-fuzzy Classification Technique

1002 words - 5 pages different aspects of uncertainties or incompleteness about real life situations. In a fuzzy system the features are associated with a degree of membership to different classes. Both NNs and fuzzy systems are very adaptable in estimating the input-output relationships. Neural networks deal with numeric and quantitative data while fuzzy systems can handle symbolic and qualitative information. Therefore it would be nice to combine these two approaches so

Literacy Acquisition of Bilingual Learners

1652 words - 7 pages materials and resources included in their learning environment to enable their progression and association between their home language and English. If a bilingual child is to flourish in the classroom, with their knowledge of the English language being extended, then they need the experiences of hearing and using language with a vast range of purposes (Smyth, 2003). In order to wholly participate across the curriculum, bilingual pupils need

Similar Essays

An Hmm Based Pre Training Approach For Sequential Data

2103 words - 9 pages the fine-tuning phase. To this aim, a simpler probabilistic model is first trained on the real dataset. We chose to use HMMs because they represent a sound, powerful and efficient framework for modeling sequential data, and despite their linear formulation they can reach state-of-the-art results in many practical prob- lems, including music modeling [5]. Learning the structure of polyphonic music with HMMs was challenging due to the exponential

Study On The Rbf Neural Network Approach To Quick Cost Estimate Of Construction Projects

5512 words - 22 pages acting upon the initial values and network structure (Gori, 1992). Based on the preferable characteristics including distributed storage, parallel processing and self-adaptive learning, RBF neural network can realize the non-linear mapping between input layer and output layer directly with sample data. There is no need to establish precise equations. In recent years RBF neural network has begun to be applied in various fields including

Neural Network Concept In Artificial Intelligence

1918 words - 8 pages . Generally speaking, neural networks are specified by 1) node (processing unit) characteristics, 2) network topology and, 3) learning paradigm. A neural network structure is based on: Node characteristics The nodes of a directed network are called processing units. The state of the unit represents the potential of a neuron. It can also be called activation. The state of a neuron is affected by its previous state, the total accumulated input signals, and

Neural Networks Essay

1326 words - 5 pages with an intellect comparable to that of a human. This pursuit of an artificial intelligence has had a long history. Several different approaches have been attempted as a result of this goal. In particular, the study of neural networks has evolved from this pursuit for an intelligent machine. The field of neural networks involves a new approach to computing that uses mathematical structures with the ability to learn (Zsolutions). These methods