A Correspondence between Normalization Strategies in Artificial and Biological Neural Networks.

Shen, Yang, Wang, Julia, Navlakha, Saket (August 2021) A Correspondence between Normalization Strategies in Artificial and Biological Neural Networks. Neural Computation. pp. 1-25. ISSN 0899-7667

Abstract

A fundamental challenge at the interface of machine learning and neuroscience is to uncover computational principles that are shared between artificial and biological neural networks. In deep learning, normalization methods such as batch normalization, weight normalization, and their many variants help to stabilize hidden unit activity and accelerate network training, and these methods have been called one of the most important recent innovations for optimizing deep networks. In the brain, homeostatic plasticity represents a set of mechanisms that also stabilize and normalize network activity to lie within certain ranges, and these mechanisms are critical for maintaining normal brain function. In this article, we discuss parallels between artificial and biological normalization methods at four spatial scales: normalization of a single neuron's activity, normalization of synaptic weights of a neuron, normalization of a layer of neurons, and normalization of a network of neurons. We argue that both types of methods are functionally equivalent-that is, both push activation patterns of hidden units toward a homeostatic state, where all neurons are equally used-and we argue that such representations can improve coding capacity, discrimination, and regularization. As a proof of concept, we develop an algorithm, inspired by a neural normalization technique called synaptic scaling, and show that this algorithm performs competitively against existing normalization methods on several data sets. Overall, we hope this bidirectional connection will inspire neuroscientists and machine learners in three ways: to uncover new normalization algorithms based on established neurobiological principles; to help quantify the trade-offs of different homeostatic plasticity mechanisms used in the brain; and to offer insights about how stability may not hinder, but may actually promote, plasticity.

Item Type:	Paper
Subjects:	bioinformatics bioinformatics > computational biology > algorithms organs, tissues, organelles, cell types and functions > cell types and functions > cell types organs, tissues, organelles, cell types and functions > cell types and functions > cell types organs, tissues, organelles, cell types and functions > cell types and functions > cell types organs, tissues, organelles, cell types and functions > cell types and functions bioinformatics > computational biology bioinformatics > computational biology > algorithms > machine learning organs, tissues, organelles, cell types and functions > tissues types and functions > neural networks organs, tissues, organelles, cell types and functions > cell types and functions > cell types > neurons organs, tissues, organelles, cell types and functions > cell types and functions > cell types > neurons organs, tissues, organelles, cell types and functions > cell types and functions > cell types > neurons organs, tissues, organelles, cell types and functions
CSHL Authors:	Shen, Yang Navlakha, Saket Wang, Julia
Communities:	CSHL labs > Engel lab CSHL labs > Navlakha lab School of Biological Sciences > Publications
SWORD Depositor:	CSHL Elements
Depositing User:	CSHL Elements
Date:	30 August 2021
Date Deposited:	08 Sep 2021 13:28
Last Modified:	25 Jan 2024 16:42
PMCID:	PMC8662716
URI:	https://repository.cshl.edu/id/eprint/40346

Actions (login required)

Administrator's edit/view item