Advanced supervised learning in multi-layer perceptrons — From backpropagation to adaptive learning algorithms

doi:10.1016/0920-5489(94)90017-5

Computer Standards & Interfaces

Volume 16, Issue 3, July 1994, Pages 265-278

https://doi.org/10.1016/0920-5489(94)90017-5 Get rights and content

Abstract

Since the presentation of the backpropagation algorithm [1] a vast variety of improvements of the technique for training the weights in a feed-forward neural network have been proposed. The following article introduces the concept of supervised learning in multi-layer perceptrons based on the technique of gradient descent. Some problems and drawbacks of the original backpropagation learning procedure are discussed, eventually leading to the development of more sophisticated techniques

This article concentrates on adaptive learning strategies. Some of the most popular learning algorithms are described and discussed according to their classification in terms of global and local adaptation strategies.

The behavior of several learning procedures on some popular benchmark problems is reported, thereby illuminating convergence, robustness, and scaling properties of the respective algorithms.

References (12)

M.F Moller
A scaled conjugate gradient algorithm for fast supervised learning
Neural Networks
(1993)
F.M Silva et al.
Speeding up backpropagation
D.E Rumelhart et al.
Learning internal representations by error propagation
R Salomon
Improved convergence rate of backpropagation with dynamic adaptation of the learning rate
W Schiffmann et al.
Optimization of the backpropagation algorithm for training multilayer perceptrons
J Herz et al.
Introduction to the Theory of Neural Computation
(1991)

There are more references available in the full text version of this article.

Cited by (417)

Nacre-like block lattice metamaterials with targeted phononic band gap and mechanical properties
2024, Journal of the Mechanical Behavior of Biomedical Materials
The extraordinary quasi-static mechanical properties of nacre-like composite metamaterials, such as high specific strength, stiffness, and toughness, are due to the periodic arrangement of two distinct phases in a “brick and mortar” structure. It is also theorized that the hierarchical periodic structure of nacre structures can provide wider band gaps at different frequency scales. However, the function of hierarchy in the dynamic behavior of metamaterials is largely unknown, and most current investigations are focused on a single objective and specialized applications. Nature, on the other hand, appears to develop systems that represent a trade-off between multiple objectives, such as stiffness, fatigue resistance, and wave attenuation. Given the wide range of design options available to these systems, a multidisciplinary strategy combining diverse objectives may be a useful opportunity provided by bioinspired artificial systems. This paper describes a class of hierarchically-architected block lattice metamaterials with simultaneous wave filtering and enhanced mechanical properties, using deep learning based on artificial neural networks (ANN), to overcome the shortcomings of traditional design methods for forward prediction, parameter design, and topology design of block lattice metamaterial. Our approach uses ANN to efficiently describe the complicated interactions between nacre geometry and its attributes, and then use the Bayesian optimization technique to determine the optimal geometry constants that match the given fitness requirements. We numerically demonstrate that complete band gaps, that is attributed to the coupling effects of local resonances and Bragg scattering, exist. The coupling effects are naturally influenced by the topological arrangements of the continuous structures and the mechanical characteristics of the component phases. We also demonstrate how we can tune the frequency of the complete band gap by modifying the geometrical configurations and volume fraction distribution of the metamaterials. This research contributes to the development of mechanically robust block lattice metamaterials and lenses capable of controlling acoustic and elastic waves in hostile settings.
Machine learning regression algorithms for generating chemical element maps from X-ray fluorescence data of paintings
2024, Chemometrics and Intelligent Laboratory Systems
Generating chemical element maps of paintings from X-ray fluorescence (XRF) data is a very valuable tool for the scientific community of conservators and art historians. Hand-held XRF scanners are cheap and easily portable but their use provides scans with a few data, so additional analytical tools are needed to obtain reliable chemical element maps from them. Recently, the software tool SmART_Scan was released, which uses an algorithm based on the minimum hypercube distance (MHD) to compute this kind of maps. In this paper, we propose a new methodology to address this problem by using machine learning algorithms for regression as alternative and more accurate techniques than MHD. We tested MHD versus eight machine learning regression algorithms on two paintings with different features. Our results showed that machine learning algorithms Random Forest and kNN significantly outperformed MHD in Mean Squared Error (MSE) and coefficient of determination (R²) for all the experiments. When using experts’ data and a hold-out validation, kNN was the best-ranked algorithm. Random Forest was the best-ranked algorithm when cross-validation was used. We did not find significant differences in average MSE nor in R² between kNN and Random Forest, so we can conclude that Random Forest is the best-suited algorithm for computing chemical element maps of paintings from XRF data.
Identification of critical SARS-CoV-2 amino acids associated with COVID-19 hospitalization rate using machine learning and statistical modeling: An observational study in the United States
2023, Infection, Genetics and Evolution
The COVID-19 pandemic has put many medical systems on the verge of collapse in the last two years. Virus mutation was one of the important factors affecting the COVID-19 infection severity and hospitalizations. Although over ten thousand SARS-CoV-2 mutations being reported since the beginning of the COVID-19 pandemic, only a small percentage of mutations are likely to affect the virus phenotype and change its severity. Finding out which amino acids have the greatest impact on COVID-19 hospitalization rate is an important research question.
This observational study used the COVID-19 case hospitalization ratio (CHR) to represent the virus severity related with hospitalization. The database is based on the daily state-level epidemiological and genomic sequential data in the United States from the Alpha wave to the first Omicron wave. The critical amino acids that mostly affected the CHR were determined by using four types of models including extreme gradient boosting decision trees (XGBoost), artificial neural networks (ANNs), logistic regression and Lasso regression models.
The XGBoost, ANN, logistic regression, and Lasso regression models all produce excellent results (mean square error for all state-level models does not exceed 0.0008 using the testing dataset). Based on the rank of importance of all covariates, the critical amino acids most affecting the CHR were identified, including T19, L24, P25, P26, A27, A67, H69, V70, T95, G142, V143, Y145, E156, F157, N211, L212, V213, R214, D215, G339, R346, S373, L452, S477, T478, E484, N501, A570, P681, and T716.
This study identified critical amino acids that are most likely to affect the hospitalization rate, allowing public health workers to monitor these highly risky amino acids and raise an alarm immediately when more severe mutations occur. Furthermore, the methodology and results may be extended to other regions.
A data-driven approach to estimating post-discovery parameters of unexplored oilfields
2023, Petroleum
Consider a typical situation where an investor is considering acquiring an unexplored oilfield. The oilfield has undergone a preliminary geological and geophysical study in which pre-discovery data such as lithology, depth, depositional system, diagenetic overprint, structural compartmentalization, and trap type are available. In this situation, investors usually estimate production rates using a volumetric approach. A more accurate estimation of production rates can be obtained using analytical methods, which require additional data such as net pay, porosity, oil formation volume factor, permeability, viscosity, and pressure. We call these data post-discovery parameters because they are only available after discovery through exploration drilling. A data-driven approach to estimating post-discovery parameters of an unexplored oilfield is developed based on its pre-discovery data by learning from proven reservoir data. Using the Gaussian mixture model, and a data-driven reservoir typology based on the joint probability distribution of post-discovery parameters is established. We came up with 12 reservoir types. Subsequently, an artificial neural network classification model with the resilient backpropagation algorithm is used to find relationships between pre-discovery data and reservoir types. Based on k-fold cross-validation with k = 10, the accuracy of the classification model is stable with an average of 87.9%. With our approach, an investor considering acquiring an unexplored oilfield can classify the oilfield's reservoir into a particular type and estimate its post-discovery parameters' joint probability distribution. The investor can incorporate this information into a valuation model to calculate the production rates more accurately, estimate the oilfield's value and risk, and make an informed acquisition decision accordingly.
CoRe optimizer: an all-in-one solution for machine learning
2024, Machine Learning: Science and Technology
Deep learning in water protection of resources, environment, and ecology: achievement and challenges
2024, Environmental Science and Pollution Research

View all citing articles on Scopus

View full text

PaperAdvanced supervised learning in multi-layer perceptrons — From backpropagation to adaptive learning algorithms

Abstract

Neural Networks

Learning internal representations by error propagation

Improved convergence rate of backpropagation with dynamic adaptation of the learning rate

Optimization of the backpropagation algorithm for training multilayer perceptrons

Introduction to the Theory of Neural Computation

Paper
Advanced supervised learning in multi-layer perceptrons — From backpropagation to adaptive learning algorithms