Evolving Weights and Transfer Functions in Feed Forward Neural Networks

M.Annunziato¹, I.Bertini¹, M.Lucchetti², S.Pizzuti^1,3
1ENEA – Energy, New technologies, Environment Agency
‘Casaccia’ R.C.,  Via Anguillarese 301, 00060 Rome Italy
Phone: +39-0630484411, Fax: +39-0630484811

email:{mauro.annunziato, ilaria.bertini, stefano.pizzuti}@casaccia.enea.it
²University of Rome “La Sapienza”

Dept. of Computer and Systems Science, Via Eudossiana 18, 00184 Rome Italy

Phone: +39-0644585938, Fax: +39-0644585367

email: lucchetti@dis.uniroma1.it

³CS - Communication Systems s.r.l.

Piazza della Repubblica 32, Milan Italy

Keywords: Evolutionary Neural Networks, Evolutionary Algorithms, Feed Forward Neural Networks, Smart Adaptive Systems

 

Considerable research on the off-line evolution of Artificial Neural Networks (ANN) using Evolutionary Algorithms has been carried out in recent years giving rise to a new branch of ANN known as Evolutionary Neural Networks. In this context most of the research has concentrated on the evolution of weights and topological structures but relatively little has been done on the evolution of node transfer functions and the simultaneous evolution of both weights and node transfer functions. Often the transfer function of each node in the architecture has been assumed to be fixed and predefined by human experts but the transfer function has been shown to be an important part of an ANN architecture which has significant impact on ANN's performance. Moreover the transfer function is often assumed to be the same for all the nodes in an ANN, at least for all the nodes in the same layer. In this way the application of EAs to the evolution of node transfer functions to get the optimal network’s architecture seems to be a promising research field.  

The goal of this work is to show that in general Evolutionary Computation (EC) is a good approach to solve the problem we are facing here. We show different evolutionary algorithms applied to the simultaneous off-line evolution of weights and transfer functions of feed-forward neural networks. Experimentation has been carried out to classical benchmarks when weights and both weights and transfer function are evolved and a comparison of the proposed evolutionary methods with classical methodologies (the back-propagation algorithm) are shown. All the networks’ topologies have been chosen to be highly inadequate to solve the problems with BP and fixed sigmoidal transfer functions because we want to show that the proposed methodologies applied in this work can find good solutions in the situations where traditional techniques fail. 

We conducted tests with three different types of evolutionary algorithms we carried out. They are all inspired by the artificial life metaphor and their basic principle consists in leaving the system a certain degree of freedom in order to develop an emergent behaviour by combining genetics with other peculiar aspects of life (interaction, competition, co-operation, food quest, etc.). In the proposed algorithms each individual represents a feed forward neural network in competition with the others by means of the proper fitness, which depends on the capability of reconstructing the training database. The genotype is composed by the synaptic weights and the transfer functions and reproduction operators simultaneously evolve them.

Results are very promising and show the effectiveness of the addressed evolutionary methodologies to solve the problem of simultaneously finding the optimal weights and transfer functions of a neural network. In particular a significant performance enhancement is achieved when EC algorithms are applied to simultaneously evolve both weights and transfer functions. This result is extremely remarkable and shows the EC capabilities to simultaneously and effectively train and design neural networks. In this situation EC is an approach which provide useful tools to solve problems which BP alone cannot and which are often left to human expertise. Moreover results stressed the importance of transfer functions in neural networks.