query: I find that the primary constraint in modern artificial intelligence applications is the computational resources expended on multilayer perceptrons (MLP) within neural networks. To enhance the efficiency of MLP, I am seeking a method to distill MLP by eliminating non-essential weights. To accomplish this, I require a theoretical understanding of the weights in shallow MLPs and an empirical application of these theories to deeper neural networks. Furthermore, I need a metric that takes into account both the computational resources saved and the performance loss resulting from the distillation. I understand that the concept of sparsity could be beneficial in representing the number of edges in a graph, but I need guidance on how to apply this concept to MLPs, which can be viewed as a specific type of graph.

positive:
In this work, we firstly apply the Train-Tensor (TT) networks to construct a
compact representation of the classical Multilayer Perceptron, representing a
reduction of up to 95% of the coefficients. A comparative analysis between
tensor model and standard multilayer neural networks is also carried out in the
context of prediction of the Mackey-Glass noisy chaotic time series and NASDAQ
index. We show that the weights of a multidimensional regression model can be
learned by means of TT network and the optimization of TT weights is a more
robust to the impact of coefficient initialization and hyper-parameter setting.
Furthermore, an efficient algorithm based on alternating least squares has been
proposed for approximating the weights in TT-format with a reduction of
computational calculus, providing a much faster convergence than the well-known
adaptive learning-method algorithms, widely applied for optimizing neural
networks.

positive:
The holy grail in deep neural network research is porting the memory- and
computation-intensive network models on embedded platforms with a minimal
compromise in model accuracy. To this end, we propose a novel approach, termed
as Variational Student, where we reap the benefits of compressibility of the
knowledge distillation (KD) framework, and sparsity inducing abilities of
variational inference (VI) techniques. Essentially, we build a sparse student
network, whose sparsity is induced by the variational parameters found via
optimizing a loss function based on VI, leveraging the knowledge learnt by an
accurate but complex pre-trained teacher network. Further, for sparsity
enhancement, we also employ a Block Sparse Regularizer on a concatenated tensor
of teacher and student network weights. We demonstrate that the marriage of KD
and the VI techniques inherits compression properties from the KD framework,
and enhances levels of sparsity from the VI approach, with minimal compromise
in the model accuracy. We benchmark our results on LeNet MLP and VGGNet (CNN)
and illustrate a memory footprint reduction of 64x and 213x on these MLP and
CNN variants, respectively, without a need to retrain the teacher network.
Furthermore, in the low data regime, we observed that our method outperforms
state-of-the-art Bayesian techniques in terms of accuracy.

positive:
Deep neural networks (DNNs) have achieved extraordinary success in numerous
areas. However, to attain this success, DNNs often carry a large number of
weight parameters, leading to heavy costs of memory and computation resources.
Overfitting is also likely to happen in such network when the training data are
insufficient. These shortcomings severely hinder the application of DNNs in
resource-constrained platforms. In fact, many network weights are known to be
redundant and can be removed from the network without much loss of performance.
To this end, we introduce a new non-convex integrated transformed $\ell_1$
regularizer to promote sparsity for DNNs, which removes both redundant
connections and unnecessary neurons simultaneously. To be specific, we apply
the transformed $\ell_1$ to the matrix space of network weights and utilize it
to remove redundant connections. Besides, group sparsity is also employed as an
auxiliary to remove unnecessary neurons. An efficient stochastic proximal
gradient algorithm is presented to solve the new model at the same time. To the
best of our knowledge, this is the first work to utilize a non-convex
regularizer in sparse optimization based method to promote sparsity for DNNs.
Experiments on several public datasets demonstrate the effectiveness of the
proposed method.

negative:
Neural networks are the pinnacle of Artificial Intelligence, as in recent
years we witnessed many novel architectures, learning and optimization
techniques for deep learning. Capitalizing on the fact that neural networks
inherently constitute multipartite graphs among neuron layers, we aim to
analyze directly their structure to extract meaningful information that can
improve the learning process. To our knowledge graph mining techniques for
enhancing learning in neural networks have not been thoroughly investigated. In
this paper we propose an adapted version of the k-core structure for the
complete weighted multipartite graph extracted from a deep learning
architecture. As a multipartite graph is a combination of bipartite graphs,
that are in turn the incidence graphs of hypergraphs, we design k-hypercore
decomposition, the hypergraph analogue of k-core degeneracy. We applied
k-hypercore to several neural network architectures, more specifically to
convolutional neural networks and multilayer perceptrons for image recognition
tasks after a very short pretraining. Then we used the information provided by
the hypercore numbers of the neurons to re-initialize the weights of the neural
network, thus biasing the gradient optimization scheme. Extensive experiments
proved that k-hypercore outperforms the state-of-the-art initialization
methods.

negative:
Deep neural networks are over-parameterized, which implies that the number of
parameters are much larger than the number of samples used to train the
network. Even in such a regime deep architectures do not overfit. This
phenomenon is an active area of research and many theories have been proposed
trying to understand this peculiar observation. These include the Vapnik
Chervonenkis (VC) dimension bounds and Rademacher complexity bounds which show
that the capacity of the network is characterized by the norm of weights rather
than the number of parameters. However, the effect of input noise on these
measures for shallow and deep architectures has not been studied. In this
paper, we analyze the effects of various regularization schemes on the
complexity of a neural network which we characterize with the loss, $L_2$ norm
of the weights, Rademacher complexities (Directly Approximately Regularizing
Complexity-DARC1), VC dimension based Low Complexity Neural Network (LCNN) when
subject to varying degrees of Gaussian input noise. We show that $L_2$
regularization leads to a simpler hypothesis class and better generalization
followed by DARC1 regularizer, both for shallow as well as deeper
architectures. Jacobian regularizer works well for shallow architectures with
high level of input noises. Spectral normalization attains highest test set
accuracies both for shallow and deeper architectures. We also show that Dropout
alone does not perform well in presence of input noise. Finally, we show that
deeper architectures are robust to input noise as opposed to their shallow
counterparts.

negative:
Tabular datasets are the last "unconquered castle" for deep learning, with
traditional ML methods like Gradient-Boosted Decision Trees still performing
strongly even against recent specialized neural architectures. In this paper,
we hypothesize that the key to boosting the performance of neural networks lies
in rethinking the joint and simultaneous application of a large set of modern
regularization techniques. As a result, we propose regularizing plain
Multilayer Perceptron (MLP) networks by searching for the optimal
combination/cocktail of 13 regularization techniques for each dataset using a
joint optimization over the decision on which regularizers to apply and their
subsidiary hyperparameters. We empirically assess the impact of these
regularization cocktails for MLPs in a large-scale empirical study comprising
40 tabular datasets and demonstrate that (i) well-regularized plain MLPs
significantly outperform recent state-of-the-art specialized neural network
architectures, and (ii) they even outperform strong traditional ML methods,
such as XGBoost.

negative:
We initiate the first empirical study on the use of MLP architectures for
vision-and-language (VL) fusion. Through extensive experiments on 5 VL tasks
and 5 robust VQA benchmarks, we find that: (i) Without pre-training, using MLPs
for multimodal fusion has a noticeable performance gap compared to
transformers; (ii) However, VL pre-training can help close the performance gap;
(iii) Instead of heavy multi-head attention, adding tiny one-head attention to
MLPs is sufficient to achieve comparable performance to transformers. Moreover,
we also find that the performance gap between MLPs and transformers is not
widened when being evaluated on the harder robust VQA benchmarks, suggesting
using MLPs for VL fusion can generalize roughly to a similar degree as using
transformers. These results hint that MLPs can effectively learn to align
vision and text features extracted from lower-level encoders without heavy
reliance on self-attention. Based on this, we ask an even bolder question: can
we have an all-MLP architecture for VL modeling, where both VL fusion and the
vision encoder are replaced with MLPs? Our result shows that an all-MLP VL
model is sub-optimal compared to state-of-the-art full-featured VL models when
both of them get pre-trained. However, pre-training an all-MLP can surprisingly
achieve a better average score than full-featured transformer models without
pre-training. This indicates the potential of large-scale pre-training of
MLP-like architectures for VL modeling and inspires the future research
direction on simplifying well-established VL modeling with less inductive
design bias. Our code is publicly available at:
https://github.com/easonnie/mlp-vil

negative:
Deep neural networks have achieved increasingly accurate results on a wide
variety of complex tasks. However, much of this improvement is due to the
growing use and availability of computational resources (e.g use of GPUs, more
layers, more parameters, etc). Most state-of-the-art deep networks, despite
performing well, over-parameterize approximate functions and take a significant
amount of time to train. With increased focus on deploying deep neural networks
on resource constrained devices like smart phones, there has been a push to
evaluate why these models are so resource hungry and how they can be made more
efficient. This work evaluates and compares three distinct methods for deep
model compression and acceleration: weight pruning, low rank factorization, and
knowledge distillation. Comparisons on VGG nets trained on CIFAR10 show that
each of the models on their own are effective, but that the true power lies in
combining them. We show that by combining pruning and knowledge distillation
methods we can create a compressed network 85 times smaller than the original,
all while retaining 96% of the original model's accuracy.

negative:
In recent years, deep neural networks (DNNs) have been applied to various
machine leaning tasks, including image recognition, speech recognition, and
machine translation. However, large DNN models are needed to achieve
state-of-the-art performance, exceeding the capabilities of edge devices. Model
reduction is thus needed for practical use. In this paper, we point out that
deep learning automatically induces group sparsity of weights, in which all
weights connected to an output channel (node) are zero, when training DNNs
under the following three conditions: (1) rectified-linear-unit (ReLU)
activations, (2) an $L_2$-regularized objective function, and (3) the Adam
optimizer. Next, we analyze this behavior both theoretically and
experimentally, and propose a simple model reduction method: eliminate the zero
weights after training the DNN. In experiments on MNIST and CIFAR-10 datasets,
we demonstrate the sparsity with various training setups. Finally, we show that
our method can efficiently reduce the model size and performs well relative to
methods that use a sparsity-inducing regularizer.

negative:
The growing energy and performance costs of deep learning have driven the
community to reduce the size of neural networks by selectively pruning
components. Similarly to their biological counterparts, sparse networks
generalize just as well, if not better than, the original dense networks.
Sparsity can reduce the memory footprint of regular networks to fit mobile
devices, as well as shorten training time for ever growing networks. In this
paper, we survey prior work on sparsity in deep learning and provide an
extensive tutorial of sparsification for both inference and training. We
describe approaches to remove and add elements of neural networks, different
training strategies to achieve model sparsity, and mechanisms to exploit
sparsity in practice. Our work distills ideas from more than 300 research
papers and provides guidance to practitioners who wish to utilize sparsity
today, as well as to researchers whose goal is to push the frontier forward. We
include the necessary background on mathematical methods in sparsification,
describe phenomena such as early structure adaptation, the intricate relations
between sparsity and the training process, and show techniques for achieving
acceleration on real hardware. We also define a metric of pruned parameter
efficiency that could serve as a baseline for comparison of different sparse
networks. We close by speculating on how sparsity can improve future workloads
and outline major open problems in the field.

negative:
While machine learning is traditionally a resource intensive task, embedded
systems, autonomous navigation and the vision of the Internet-of-Things fuel
the interest in resource efficient approaches. These approaches require a
carefully chosen trade-off between performance and resource consumption in
terms of computation and energy. On top of this, it is crucial to treat
uncertainty in a consistent manner in all but the simplest applications of
machine learning systems. In particular, a desideratum for any real-world
system is to be robust in the presence of outliers and corrupted data, as well
as being `aware' of its limits, i.e.\ the system should maintain and provide an
uncertainty estimate over its own predictions. These complex demands are among
the major challenges in current machine learning research and key to ensure a
smooth transition of machine learning technology into every day's applications.
In this article, we provide an overview of the current state of the art of
machine learning techniques facilitating these real-world requirements. First
we provide a comprehensive review of resource-efficiency in deep neural
networks with focus on techniques for model size reduction, compression and
reduced precision. These techniques can be applied during training or as
post-processing and are widely used to reduce both computational complexity and
memory footprint. As most (practical) neural networks are limited in their ways
to treat uncertainty, we contrast them with probabilistic graphical models,
which readily serve these desiderata by means of probabilistic inference. In
that way, we provide an extensive overview of the current state-of-the-art of
robust and efficient machine learning for real-world systems.

negative:
Mixture-of-Experts (MoE), a conditional computation architecture, achieved
promising performance by scaling local module (i.e. feed-forward network) of
transformer. However, scaling the cross-token module (i.e. self-attention) is
challenging due to the unstable training. This work proposes Sparse-MLP, an
all-MLP model which applies sparsely-activated MLPs to cross-token modeling.
Specifically, in each Sparse block of our all-MLP model, we apply two stages of
MoE layers: one with MLP experts mixing information within channels along image
patch dimension, the other with MLP experts mixing information within patches
along the channel dimension. In addition, by proposing importance-score routing
strategy for MoE and redesigning the image representation shape, we further
improve our model's computational efficiency. Experimentally, we are more
computation-efficient than Vision Transformers with comparable accuracy. Also,
our models can outperform MLP-Mixer by 2.5\% on ImageNet Top-1 accuracy with
fewer parameters and computational cost. On downstream tasks, i.e. Cifar10 and
Cifar100, our models can still achieve better performance than baselines.

negative:
This work proposes a domain-informed neural network architecture for
experimental particle physics, using particle interaction localization with the
time-projection chamber (TPC) technology for dark matter research as an example
application. A key feature of the signals generated within the TPC is that they
allow localization of particle interactions through a process called
reconstruction. While multilayer perceptrons (MLPs) have emerged as a leading
contender for reconstruction in TPCs, such a black-box approach does not
reflect prior knowledge of the underlying scientific processes. This paper
looks anew at neural network-based interaction localization and encodes prior
detector knowledge, in terms of both signal characteristics and detector
geometry, into the feature encoding and the output layers of a multilayer
neural network. The resulting Domain-informed Neural Network (DiNN) limits the
receptive fields of the neurons in the initial feature encoding layers in order
to account for the spatially localized nature of the signals produced within
the TPC. This aspect of the DiNN, which has similarities with the emerging area
of graph neural networks in that the neurons in the initial layers only connect
to a handful of neurons in their succeeding layer, significantly reduces the
number of parameters in the network in comparison to an MLP. In addition, in
order to account for the detector geometry, the output layers of the network
are modified using two geometric transformations to ensure the DiNN produces
localizations within the interior of the detector. The end result is a neural
network architecture that has 60% fewer parameters than an MLP, but that still
achieves similar localization performance and provides a path to future
architectural developments with improved performance because of their ability
to encode additional domain knowledge into the architecture.

negative:
Understanding the fundamental principles behind the massive success of neural
networks is one of the most important open questions in deep learning. However,
due to the highly complex nature of the problem, progress has been relatively
slow. In this note, through the lens of infinite-width networks, a.k.a. neural
kernels, we present one such principle resulting from hierarchical localities.
It is well-known that the eigenstructure of infinite-width multilayer
perceptrons (MLPs) depends solely on the concept frequency, which measures the
order of interactions. We show that the topologies from deep convolutional
networks (CNNs) restructure the associated eigenspaces into finer subspaces. In
addition to frequency, the new structure also depends on the concept space,
which measures the spatial distance among nonlinear interaction terms. The
resulting fine-grained eigenstructure dramatically improves the network's
learnability, empowering them to simultaneously model a much richer class of
interactions, including Long-Range-Low-Frequency interactions,
Short-Range-High-Frequency interactions, and various interpolations and
extrapolations in-between. Additionally, model scaling can improve the
resolutions of interpolations and extrapolations and, therefore, the network's
learnability. Finally, we prove a sharp characterization of the generalization
error for infinite-width CNNs of any depth in the high-dimensional setting. Two
corollaries follow: (1) infinite-width deep CNNs can break the curse of
dimensionality without losing their expressivity, and (2) scaling improves
performance in both the finite and infinite data regimes.

negative:
We study how neural networks trained by gradient descent extrapolate, i.e.,
what they learn outside the support of the training distribution. Previous
works report mixed empirical results when extrapolating with neural networks:
while feedforward neural networks, a.k.a. multilayer perceptrons (MLPs), do not
extrapolate well in certain simple tasks, Graph Neural Networks (GNNs) --
structured networks with MLP modules -- have shown some success in more complex
tasks. Working towards a theoretical explanation, we identify conditions under
which MLPs and GNNs extrapolate well. First, we quantify the observation that
ReLU MLPs quickly converge to linear functions along any direction from the
origin, which implies that ReLU MLPs do not extrapolate most nonlinear
functions. But, they can provably learn a linear target function when the
training distribution is sufficiently "diverse". Second, in connection to
analyzing the successes and limitations of GNNs, these results suggest a
hypothesis for which we provide theoretical and empirical evidence: the success
of GNNs in extrapolating algorithmic tasks to new data (e.g., larger graphs or
edge weights) relies on encoding task-specific non-linearities in the
architecture or features. Our theoretical analysis builds on a connection of
over-parameterized networks to the neural tangent kernel. Empirically, our
theory holds across different training settings.

negative:
We propose a neural network based approach for extracting models from dynamic
data using ordinary and partial differential equations. In particular, given a
time-series or spatio-temporal dataset, we seek to identify an accurate
governing system which respects the intrinsic differential structure. The
unknown governing model is parameterized by using both (shallow) multilayer
perceptrons and nonlinear differential terms, in order to incorporate relevant
correlations between spatio-temporal samples. We demonstrate the approach on
several examples where the data is sampled from various dynamical systems and
give a comparison to recurrent networks and other data-discovery methods. In
addition, we show that for MNIST and Fashion MNIST, our approach lowers the
parameter cost as compared to other deep neural networks.

negative:
Recently, sparse training methods have started to be established as a de
facto approach for training and inference efficiency in artificial neural
networks. Yet, this efficiency is just in theory. In practice, everyone uses a
binary mask to simulate sparsity since the typical deep learning software and
hardware are optimized for dense matrix operations. In this paper, we take an
orthogonal approach, and we show that we can train truly sparse neural networks
to harvest their full potential. To achieve this goal, we introduce three novel
contributions, specially designed for sparse neural networks: (1) a parallel
training algorithm and its corresponding sparse implementation from scratch,
(2) an activation function with non-trainable parameters to favour the gradient
flow, and (3) a hidden neurons importance metric to eliminate redundancies. All
in one, we are able to break the record and to train the largest neural network
ever trained in terms of representational power -- reaching the bat brain size.
The results show that our approach has state-of-the-art performance while
opening the path for an environmentally friendly artificial intelligence era.

negative:
Long training times of deep neural networks are a bottleneck in machine
learning research. The major impediment to fast training is the quadratic
growth of both memory and compute requirements of dense and convolutional
layers with respect to their information bandwidth. Recently, training `a
priori' sparse networks has been proposed as a method for allowing layers to
retain high information bandwidth, while keeping memory and compute low.
However, the choice of which sparse topology should be used in these networks
is unclear. In this work, we provide a theoretical foundation for the choice of
intra-layer topology. First, we derive a new sparse neural network
initialization scheme that allows us to explore the space of very deep sparse
networks. Next, we evaluate several topologies and show that seemingly similar
topologies can often have a large difference in attainable accuracy. To explain
these differences, we develop a data-free heuristic that can evaluate a
topology independently from the dataset the network will be trained on. We then
derive a set of requirements that make a good topology, and arrive at a single
topology that satisfies all of them.

negative:
Deep Neural Networks are highly over-parameterized and the size of the neural
networks can be reduced significantly after training without any decrease in
performance. One can clearly see this phenomenon in a wide range of
architectures trained for various problems. Weight/channel pruning,
distillation, quantization, matrix factorization are some of the main methods
one can use to remove the redundancy to come up with smaller and faster models.
  This work starts with a short informative chapter, where we motivate the
pruning idea and provide the necessary notation. In the second chapter, we
compare various saliency scores in the context of parameter pruning. Using the
insights obtained from this comparison and stating the problems it brings we
motivate why pruning units instead of the individual parameters might be a
better idea. We propose some set of definitions to quantify and analyze units
that don't learn and create any useful information. We propose an efficient way
for detecting dead units and use it to select which units to prune. We get 5x
model size reduction through unit-wise pruning on MNIST.

negative:
Deep learning based models are relatively large, and it is hard to deploy
such models on resource-limited devices such as mobile phones and embedded
devices. One possible solution is knowledge distillation whereby a smaller
model (student model) is trained by utilizing the information from a larger
model (teacher model). In this paper, we present a survey of knowledge
distillation techniques applied to deep learning models. To compare the
performances of different techniques, we propose a new metric called
distillation metric. Distillation metric compares different knowledge
distillation algorithms based on sizes and accuracy scores. Based on the
survey, some interesting conclusions are drawn and presented in this paper.

negative:
Recently, click-through rate (CTR) prediction models have evolved from
shallow methods to deep neural networks. Most deep CTR models follow an
Embedding\&MLP paradigm, that is, first mapping discrete id features, e.g. user
visited items, into low dimensional vectors with an embedding module, then
learn a multi-layer perception (MLP) to fit the target. In this way, embedding
module performs as the representative learning and plays a key role in the
model performance. However, in many real-world applications, deep CTR model
often suffers from poor generalization performance, which is mostly due to the
learning of embedding parameters. In this paper, we model user behavior using
an interest delay model, study carefully the embedding mechanism, and obtain
two important results: (i) We theoretically prove that small aggregation radius
of embedding vectors of items which belongs to a same user interest domain will
result in good generalization performance of deep CTR model. (ii) Following our
theoretical analysis, we design a new embedding structure named res-embedding.
In res-embedding module, embedding vector of each item is the sum of two
components: (i) a central embedding vector calculated from an item-based
interest graph (ii) a residual embedding vector with its scale to be relatively
small. Empirical evaluation on several public datasets demonstrates the
effectiveness of the proposed res-embedding structure, which brings significant
improvement on the model performance.

negative:
Among interpretable machine learning methods, the class of Generalised
Additive Neural Networks (GANNs) is referred to as Self-Explaining Neural
Networks (SENN) because of the linear dependence on explicit functions of the
inputs. In binary classification this shows the precise weight that each input
contributes towards the logit. The nomogram is a graphical representation of
these weights. We show that functions of individual and pairs of variables can
be derived from a functional Analysis of Variance (ANOVA) representation,
enabling an efficient feature selection to be carried by application of the
logistic Lasso. This process infers the structure of GANNs which otherwise
needs to be predefined. As this method is particularly suited for tabular data,
it starts by fitting a generic flexible model, in this case a Multi-layer
Perceptron (MLP) to which the ANOVA decomposition is applied. This has the
further advantage that the resulting GANN can be replicated as a SENN, enabling
further refinement of the univariate and bivariate component functions to take
place. The component functions are partial responses hence the SENN is a
partial response network. The Partial Response Network (PRN) is equally as
transparent as a traditional logistic regression model, but capable of
non-linear classification with comparable or superior performance to the
original MLP. In other words, the PRN is a fully interpretable representation
of the MLP, at the level of univariate and bivariate effects. The performance
of the PRN is shown to be competitive for benchmark data, against
state-of-the-art machine learning methods including GBM, SVM and Random
Forests. It is also compared with spline-based Sparse Additive Models (SAM)
showing that a semi-parametric representation of the GAM as a neural network
can be as effective as the SAM though less constrained by the need to set
spline nodes.

negative:
Deep learning has redefined the field of artificial intelligence (AI) thanks
to the rise of artificial neural networks, which are architectures inspired by
their neurological counterpart in the brain. Through the years, this dualism
between AI and neuroscience has brought immense benefits to both fields,
allowing neural networks to be used in dozens of applications. These networks
use an efficient implementation of reverse differentiation, called
backpropagation (BP). This algorithm, however, is often criticized for its
biological implausibility (e.g., lack of local update rules for the
parameters). Therefore, biologically plausible learning methods that rely on
predictive coding (PC), a framework for describing information processing in
the brain, are increasingly studied. Recent works prove that these methods can
approximate BP up to a certain margin on multilayer perceptrons (MLPs), and
asymptotically on any other complex model, and that zero-divergence inference
learning (Z-IL), a variant of PC, is able to exactly implement BP on MLPs.
However, the recent literature shows also that there is no biologically
plausible method yet that can exactly replicate the weight update of BP on
complex models. To fill this gap, in this paper, we generalize (PC and) Z-IL by
directly defining them on computational graphs, and show that it can perform
exact reverse differentiation. What results is the first biologically plausible
algorithm that is equivalent to BP in the way of updating parameters on any
neural network, providing a bridge between the interdisciplinary research of
neuroscience and deep learning.

negative:
The state-of-the-art deep neural networks (DNNs) have significant
computational and data management requirements. The size of both training data
and models continue to increase. Sparsification and pruning methods are shown
to be effective in removing a large fraction of connections in DNNs. The
resulting sparse networks present unique challenges to further improve the
computational efficiency of training and inference in deep learning. Both the
feedforward (inference) and backpropagation steps in stochastic gradient
descent (SGD) algorithm for training sparse DNNs involve consecutive sparse
matrix-vector multiplications (SpMVs). We first introduce a distributed-memory
parallel SpMV-based solution for the SGD algorithm to improve its scalability.
The parallelization approach is based on row-wise partitioning of weight
matrices that represent neuron connections between consecutive layers. We then
propose a novel hypergraph model for partitioning weight matrices to reduce the
total communication volume and ensure computational load-balance among
processors. Experiments performed on sparse DNNs demonstrate that the proposed
solution is highly efficient and scalable. By utilizing the proposed matrix
partitioning scheme, the performance of our solution is further improved
significantly.

negative:
To put a state-of-the-art neural network to practical use, it is necessary to
design a model that has a good trade-off between the resource consumption and
performance on the test set. Many researchers and engineers are developing
methods that enable training or designing a model more efficiently. Developing
an efficient model includes several strategies such as network architecture
search, pruning, quantization, knowledge distillation, utilizing cheap
convolution, regularization, and also includes any craft that leads to a better
performance-resource trade-off. When combining these technologies together, it
would be ideal if one source of performance improvement does not conflict with
others. We call this property as the orthogonality in model efficiency. In this
paper, we focus on knowledge distillation and demonstrate that knowledge
distillation methods are orthogonal to other efficiency-enhancing methods both
analytically and empirically. Analytically, we claim that knowledge
distillation functions analogous to a ensemble method, bootstrap aggregating.
This analytical explanation is provided from the perspective of implicit data
augmentation property of knowledge distillation. Empirically, we verify
knowledge distillation as a powerful apparatus for practical deployment of
efficient neural network, and also introduce ways to integrate it with other
methods effectively.

negative:
Pruning coupled with learning aims to optimize the neural network (NN)
structure for solving specific problems. This optimization can be used for
various purposes: to prevent overfitting, to save resources for implementation
and training, to provide explainability of the trained NN, and many others. The
minimal structure that cannot be pruned further is not unique. Ensemble of
minimal structures can be used as a committee of intellectual agents that
solves problems by voting. Each minimal NN presents an "empirical knowledge"
about the problem and can be verbalized. The non-uniqueness of such knowledge
extracted from data is an important property of data-driven Artificial
Intelligence (AI). In this work, we review an approach to pruning based on the
principle: What controls training should control pruning. This principle is
expected to work both for artificial NN and for selection and modification of
important synaptic contacts in brain. In back-propagation artificial NN
learning is controlled by the gradient of loss functions. Therefore, the first
order sensitivity indicators are used for pruning and the algorithms based on
these indicators are reviewed. The notion of logically transparent NN was
introduced. The approach was illustrated on the problem of political
forecasting: predicting the results of the US presidential election. Eight
minimal NN were produced that give different forecasting algorithms. The
non-uniqueness of solution can be utilised by creation of expert panels
(committee). Another use of NN pluralism is to identify areas of input signals
where further data collection is most useful. In Conclusion, we discuss the
possible future of widely advertised XAI program.

negative:
Deep learning is finding its way into the embedded world with applications
such as autonomous driving, smart sensors and aug- mented reality. However, the
computation of deep neural networks is demanding in energy, compute power and
memory. Various approaches have been investigated to reduce the necessary
resources, one of which is to leverage the sparsity occurring in deep neural
networks due to the high levels of redundancy in the network parameters. It has
been shown that sparsity can be promoted specifically and the achieved sparsity
can be very high. But in many cases the methods are evaluated on rather small
topologies. It is not clear if the results transfer onto deeper topologies. In
this paper, the TensorQuant toolbox has been extended to offer a platform to
investigate sparsity, especially in deeper models. Several practical relevant
topologies for varying classification problem sizes are investigated to show
the differences in sparsity for activations, weights and gradients.

negative:
Neural network models are widely used in solving many challenging problems,
such as computer vision, personalized recommendation, and natural language
processing. Those models are very computationally intensive and reach the
hardware limit of the existing server and IoT devices. Thus, finding better
model architectures with much less amount of computation while maximally
preserving the accuracy is a popular research topic. Among various mechanisms
that aim to reduce the computation complexity, identifying the zero values in
the model weights and in the activations to avoid computing them is a promising
direction.
  In this paper, we summarize the current status of the research on the
computation of sparse neural networks, from the perspective of the sparse
algorithms, the software frameworks, and the hardware accelerations. We observe
that the search for the sparse structure can be a general methodology for
high-quality model explorations, in addition to a strategy for high-efficiency
model execution. We discuss the model accuracy influenced by the number of
weight parameters and the structure of the model. The corresponding models are
called to be located in the weight dominated and structure dominated regions,
respectively. We show that for practically complicated problems, it is more
beneficial to search large and sparse models in the weight dominated region. In
order to achieve the goal, new approaches are required to search for proper
sparse structures, and new sparse training hardware needs to be developed to
facilitate fast iterations of sparse models.

negative:
The success of convolutional neural networks (CNNs) in various applications
is accompanied by a significant increase in computation and parameter storage
costs. Recent efforts to reduce these overheads involve pruning and compressing
the weights of various layers while at the same time aiming to not sacrifice
performance. In this paper, we propose a novel criterion for CNN pruning
inspired by neural network interpretability: The most relevant units, i.e.
weights or filters, are automatically found using their relevance scores
obtained from concepts of explainable AI (XAI). By exploring this idea, we
connect the lines of interpretability and model compression research. We show
that our proposed method can efficiently prune CNN models in transfer-learning
setups in which networks pre-trained on large corpora are adapted to
specialized tasks. The method is evaluated on a broad range of computer vision
datasets. Notably, our novel criterion is not only competitive or better
compared to state-of-the-art pruning criteria when successive retraining is
performed, but clearly outperforms these previous criteria in the
resource-constrained application scenario in which the data of the task to be
transferred to is very scarce and one chooses to refrain from fine-tuning. Our
method is able to compress the model iteratively while maintaining or even
improving accuracy. At the same time, it has a computational cost in the order
of gradient computation and is comparatively simple to apply without the need
for tuning hyperparameters for pruning.

negative:
To address the large model size and intensive computation requirement of deep
neural networks (DNNs), weight pruning techniques have been proposed and
generally fall into two categories, i.e., static regularization-based pruning
and dynamic regularization-based pruning. However, the former method currently
suffers either complex workloads or accuracy degradation, while the latter one
takes a long time to tune the parameters to achieve the desired pruning rate
without accuracy loss. In this paper, we propose a unified DNN weight pruning
framework with dynamically updated regularization terms bounded by the
designated constraint, which can generate both non-structured sparsity and
different kinds of structured sparsity. We also extend our method to an
integrated framework for the combination of different DNN compression tasks.

negative:
Multilayer-perceptrons (MLP) are known to struggle with learning functions of
high-frequencies, and in particular cases with wide frequency bands. We present
a spatially adaptive progressive encoding (SAPE) scheme for input signals of
MLP networks, which enables them to better fit a wide range of frequencies
without sacrificing training stability or requiring any domain specific
preprocessing. SAPE gradually unmasks signal components with increasing
frequencies as a function of time and space. The progressive exposure of
frequencies is monitored by a feedback loop throughout the neural optimization
process, allowing changes to propagate at different rates among local spatial
portions of the signal space. We demonstrate the advantage of SAPE on a variety
of domains and applications, including regression of low dimensional signals
and images, representation learning of occupancy networks, and a geometric task
of mesh transfer between 3D shapes.

negative:
We consider infinitely-wide multi-layer perceptrons (MLPs) which are limits
of standard deep feed-forward neural networks. We assume that, for each layer,
the weights of an MLP are initialized with i.i.d. samples from either a
light-tailed (finite variance) or heavy-tailed distribution in the domain of
attraction of a symmetric $\alpha$-stable distribution, where $\alpha\in(0,2]$
may depend on the layer. For the bias terms of the layer, we assume i.i.d.
initializations with a symmetric $\alpha$-stable distribution having the same
$\alpha$ parameter of that layer. We then extend a recent result of Favaro,
Fortini, and Peluchetti (2020), to show that the vector of pre-activation
values at all nodes of a given hidden layer converges in the limit, under a
suitable scaling, to a vector of i.i.d. random variables with symmetric
$\alpha$-stable distributions.

negative:
We show that passing input points through a simple Fourier feature mapping
enables a multilayer perceptron (MLP) to learn high-frequency functions in
low-dimensional problem domains. These results shed light on recent advances in
computer vision and graphics that achieve state-of-the-art results by using
MLPs to represent complex 3D objects and scenes. Using tools from the neural
tangent kernel (NTK) literature, we show that a standard MLP fails to learn
high frequencies both in theory and in practice. To overcome this spectral
bias, we use a Fourier feature mapping to transform the effective NTK into a
stationary kernel with a tunable bandwidth. We suggest an approach for
selecting problem-specific Fourier features that greatly improves the
performance of MLPs for low-dimensional regression tasks relevant to the
computer vision and graphics communities.

negative:
Interpretability of neural networks and their underlying theoretical
behaviour remain an open field of study even after the great success of their
practical applications, particularly with the emergence of deep learning. In
this work, NN2Poly is proposed: a theoretical approach to obtain an explicit
polynomial model that provides an accurate representation of an already trained
fully-connected feed-forward artificial neural network (a multilayer perceptron
or MLP). This approach extends a previous idea proposed in the literature,
which was limited to single hidden layer networks, to work with arbitrarily
deep MLPs in both regression and classification tasks. The objective of this
paper is to achieve this by using a Taylor expansion on the activation
function, at each layer, and then using several combinatorial properties to
calculate the coefficients of the desired polynomials. Discussion is presented
on the main computational challenges of this method, and the way to overcome
them by imposing certain constraints during the training phase. Finally,
simulation experiments as well as an application to a real data set are
presented to demonstrate the effectiveness of the proposed method.

negative:
Through the success of deep learning in various domains, artificial neural
networks are currently among the most used artificial intelligence methods.
Taking inspiration from the network properties of biological neural networks
(e.g. sparsity, scale-freeness), we argue that (contrary to general practice)
artificial neural networks, too, should not have fully-connected layers. Here
we propose sparse evolutionary training of artificial neural networks, an
algorithm which evolves an initial sparse topology (Erd\H{o}s-R\'enyi random
graph) of two consecutive layers of neurons into a scale-free topology, during
learning. Our method replaces artificial neural networks fully-connected layers
with sparse ones before training, reducing quadratically the number of
parameters, with no decrease in accuracy. We demonstrate our claims on
restricted Boltzmann machines, multi-layer perceptrons, and convolutional
neural networks for unsupervised and supervised learning on 15 datasets. Our
approach has the potential to enable artificial neural networks to scale up
beyond what is currently possible.

negative:
Deep Learning using the eponymous deep neural networks (DNNs) has become an
attractive approach towards various data-based problems of theoretical physics
in the past decade. There has been a clear trend to deeper architectures
containing increasingly more powerful and involved layers. Contrarily, Taylor
coefficients of DNNs still appear mainly in the light of interpretability
studies, where they are computed at most to first order. However, especially in
theoretical physics numerous problems benefit from accessing higher orders, as
well. This gap motivates a general formulation of neural network (NN) Taylor
expansions. Restricting our analysis to multilayer perceptrons (MLPs) and
introducing quantities we refer to as propagators and vertices, both depending
on the MLP's weights and biases, we establish a graph-theoretical approach.
Similarly to Feynman rules in quantum field theories, we can systematically
assign diagrams containing propagators and vertices to the corresponding
partial derivative. Examining this approach for S-wave scattering lengths of
shallow potentials, we observe NNs to adapt their derivatives mainly to the
leading order of the target function's Taylor expansion. To circumvent this
problem, we propose an iterative NN perturbation theory. During each iteration
we eliminate the leading order, such that the next-to-leading order can be
faithfully learned during the subsequent iteration. After performing two
iterations, we find that the first- and second-order Born terms are correctly
adapted during the respective iterations. Finally, we combine both results to
find a proxy that acts as a machine-learned second-order Born approximation.

negative:
From the perspective of expressive power, this work compares multi-layer
Graph Neural Networks (GNNs) with a simplified alternative that we call
Graph-Augmented Multi-Layer Perceptrons (GA-MLPs), which first augments node
features with certain multi-hop operators on the graph and then applies an MLP
in a node-wise fashion. From the perspective of graph isomorphism testing, we
show both theoretically and numerically that GA-MLPs with suitable operators
can distinguish almost all non-isomorphic graphs, just like the
Weifeiler-Lehman (WL) test. However, by viewing them as node-level functions
and examining the equivalence classes they induce on rooted graphs, we prove a
separation in expressive power between GA-MLPs and GNNs that grows
exponentially in depth. In particular, unlike GNNs, GA-MLPs are unable to count
the number of attributed walks. We also demonstrate via community detection
experiments that GA-MLPs can be limited by their choice of operator family, as
compared to GNNs with higher flexibility in learning.

negative:
This paper aims to provide a selective survey about knowledge
distillation(KD) framework for researchers and practitioners to take advantage
of it for developing new optimized models in the deep neural network field. To
this end, we give a brief overview of knowledge distillation and some related
works including learning using privileged information(LUPI) and generalized
distillation(GD). Even though knowledge distillation based on the
teacher-student architecture was initially devised as a model compression
technique, it has found versatile applications over various frameworks.
  In this paper, we review the characteristics of knowledge distillation from
the hypothesis that the three important ingredients of knowledge distillation
are distilled knowledge and loss,teacher-student paradigm, and the distillation
process. In addition, we survey the versatility of the knowledge distillation
by studying its direct applications and its usage in combination with other
deep learning paradigms. Finally we present some future works in knowledge
distillation including explainable knowledge distillation where the analytical
analysis of the performance gain is studied and the self-supervised learning
which is a hot research topic in deep learning community.

negative:
Hypernetworks are neural networks that generate weights for another neural
network. We formulate the hypernetwork training objective as a compromise
between accuracy and diversity, where the diversity takes into account trivial
symmetry transformations of the target network. We explain how this simple
formulation generalizes variational inference. We use multi-layered perceptrons
to form the mapping from the low dimensional input random vector to the high
dimensional weight space, and demonstrate how to reduce the number of
parameters in this mapping by parameter sharing. We perform experiments and
show that the generated weights are diverse and lie on a non-trivial manifold.

negative:
A closed-form solution exists in two-class linear discriminant analysis
(LDA), which discriminates two Gaussian-distributed classes in a
multi-dimensional feature space. In this work, we interpret the multilayer
perceptron (MLP) as a generalization of a two-class LDA system so that it can
handle an input composed by multiple Gaussian modalities belonging to multiple
classes. Besides input layer $l_{in}$ and output layer $l_{out}$, the MLP of
interest consists of two intermediate layers, $l_1$ and $l_2$. We propose a
feedforward design that has three stages: 1) from $l_{in}$ to $l_1$: half-space
partitionings accomplished by multiple parallel LDAs, 2) from $l_1$ to $l_2$:
subspace isolation where one Gaussian modality is represented by one neuron, 3)
from $l_2$ to $l_{out}$: class-wise subspace mergence, where each Gaussian
modality is connected to its target class. Through this process, we present an
automatic MLP design that can specify the network architecture (i.e., the layer
number and the neuron number at a layer) and all filter weights in a
feedforward one-pass fashion. This design can be generalized to an arbitrary
distribution by leveraging the Gaussian mixture model (GMM). Experiments are
conducted to compare the performance of the traditional backpropagation-based
MLP (BP-MLP) and the new feedforward MLP (FF-MLP).

negative:
This paper proposes a set of new error criteria and learning approaches,
Adaptive Normalized Risk-Averting Training (ANRAT), to attack the non-convex
optimization problem in training deep neural networks (DNNs). Theoretically, we
demonstrate its effectiveness on global and local convexity lower-bounded by
the standard $L_p$-norm error. By analyzing the gradient on the convexity index
$\lambda$, we explain the reason why to learn $\lambda$ adaptively using
gradient descent works. In practice, we show how this method improves training
of deep neural networks to solve visual recognition tasks on the MNIST and
CIFAR-10 datasets. Without using pretraining or other tricks, we obtain results
comparable or superior to those reported in recent literature on the same tasks
using standard ConvNets + MSE/cross entropy. Performance on deep/shallow
multilayer perceptrons and Denoised Auto-encoders is also explored. ANRAT can
be combined with other quasi-Newton training methods, innovative network
variants, regularization techniques and other specific tricks in DNNs. Other
than unsupervised pretraining, it provides a new perspective to address the
non-convex optimization problem in DNNs.

negative:
The power law has been observed in the degree distributions of many
biological neural networks. Sparse deep neural networks, which learn an
economical representation from the data, resemble biological neural networks in
many ways. In this paper, we study if these artificial networks also exhibit
properties of the power law. Experimental results on two popular deep learning
models, namely, multilayer perceptrons and convolutional neural networks, are
affirmative. The power law is also naturally related to preferential
attachment. To study the dynamical properties of deep networks in continual
learning, we propose an internal preferential attachment model to explain how
the network topology evolves. Experimental results show that with the arrival
of a new task, the new connections made follow this preferential attachment
process.

negative:
A fundamental task for artificial intelligence is learning. Deep Neural
Networks have proven to cope perfectly with all learning paradigms, i.e.
supervised, unsupervised, and reinforcement learning. Nevertheless, traditional
deep learning approaches make use of cloud computing facilities and do not
scale well to autonomous agents with low computational resources. Even in the
cloud, they suffer from computational and memory limitations, and they cannot
be used to model adequately large physical worlds for agents which assume
networks with billions of neurons. These issues are addressed in the last few
years by the emerging topic of sparse training, which trains sparse networks
from scratch. This paper discusses sparse training state-of-the-art, its
challenges and limitations while introducing a couple of new theoretical
research directions which has the potential of alleviating sparse training
limitations to push deep learning scalability well beyond its current
boundaries. Nevertheless, the theoretical advancements impact in complex
multi-agents settings is discussed from a real-world perspective, using the
smart grid case study.

negative:
Graph Neural Networks (GNNs) have already been widely applied in various
graph mining tasks. However, they suffer from the shallow architecture issue,
which is the key impediment that hinders the model performance improvement.
Although several relevant approaches have been proposed, none of the existing
studies provides an in-depth understanding of the root causes of performance
degradation in deep GNNs. In this paper, we conduct the first systematic
experimental evaluation to present the fundamental limitations of shallow
architectures. Based on the experimental results, we answer the following two
essential questions: (1) what actually leads to the compromised performance of
deep GNNs; (2) when we need and how to build deep GNNs. The answers to the
above questions provide empirical insights and guidelines for researchers to
design deep and well-performed GNNs. To show the effectiveness of our proposed
guidelines, we present Deep Graph Multi-Layer Perceptron (DGMLP), a powerful
approach (a paradigm in its own right) that helps guide deep GNN designs.
Experimental results demonstrate three advantages of DGMLP: 1) high accuracy --
it achieves state-of-the-art node classification performance on various
datasets; 2) high flexibility -- it can flexibly choose different propagation
and transformation depths according to graph size and sparsity; 3) high
scalability and efficiency -- it supports fast training on large-scale graphs.
Our code is available in https://github.com/zwt233/DGMLP.

negative:
Deep Neural Networks (DNNs) are the key to the state-of-the-art machine
vision, sensor fusion and audio/video signal processing. Unfortunately, their
computation complexity and tight resource constraints on the Edge make them
hard to leverage on mobile, embedded and IoT devices. Due to great diversity of
Edge devices, DNN designers have to take into account the hardware platform and
application requirements during network training. In this work we introduce
pruning via matrix pivoting as a way to improve network pruning by compromising
between the design flexibility of architecture-oblivious and performance
efficiency of architecture-aware pruning, the two dominant techniques for
obtaining resource-efficient DNNs. We also describe local and global network
optimization techniques for efficient implementation of the resulting pruned
networks. In combination, the proposed pruning and implementation result in
close to linear speed up with the reduction of network coefficients during
pruning.

negative:
Many efforts have been made to facilitate natural language processing tasks
with pre-trained language models (LMs), and brought significant improvements to
various applications. To fully leverage the nearly unlimited corpora and
capture linguistic information of multifarious levels, large-size LMs are
required; but for a specific task, only parts of these information are useful.
Such large-sized LMs, even in the inference stage, may cause heavy computation
workloads, making them too time-consuming for large-scale applications. Here we
propose to compress bulky LMs while preserving useful information with regard
to a specific task. As different layers of the model keep different
information, we develop a layer selection method for model pruning using
sparsity-inducing regularization. By introducing the dense connectivity, we can
detach any layer without affecting others, and stretch shallow and wide LMs to
be deep and narrow. In model training, LMs are learned with layer-wise dropouts
for better robustness. Experiments on two benchmark datasets demonstrate the
effectiveness of our method.

negative:
Convolutional Neural Networks (CNNs) are the go-to model for computer vision.
Recently, attention-based networks, such as the Vision Transformer, have also
become popular. In this paper we show that while convolutions and attention are
both sufficient for good performance, neither of them are necessary. We present
MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs).
MLP-Mixer contains two types of layers: one with MLPs applied independently to
image patches (i.e. "mixing" the per-location features), and one with MLPs
applied across patches (i.e. "mixing" spatial information). When trained on
large datasets, or with modern regularization schemes, MLP-Mixer attains
competitive scores on image classification benchmarks, with pre-training and
inference cost comparable to state-of-the-art models. We hope that these
results spark further research beyond the realms of well established CNNs and
Transformers.

negative:
Federated learning is an emerging technique used to prevent the leakage of
private information. Unlike centralized learning that needs to collect data
from users and store them collectively on a cloud server, federated learning
makes it possible to learn a global model while the data are distributed on the
users' devices. However, compared with the traditional centralized approach,
the federated setting consumes considerable communication resources of the
clients, which is indispensable for updating global models and prevents this
technique from being widely used. In this paper, we aim to optimize the
structure of the neural network models in federated learning using a
multi-objective evolutionary algorithm to simultaneously minimize the
communication costs and the global model test errors. A scalable method for
encoding network connectivity is adapted to federated learning to enhance the
efficiency in evolving deep neural networks. Experimental results on both
multilayer perceptrons and convolutional neural networks indicate that the
proposed optimization method is able to find optimized neural network models
that can not only significantly reduce communication costs but also improve the
learning performance of federated learning compared with the standard fully
connected neural networks.

negative:
In the context of learning to map an input $I$ to a function
$h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an
embedding-based method, which learns a fixed function in which $I$ is encoded
as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x)
= q(x,e(I))$, and (ii) hypernetworks, in which the weights $\theta_I$ of the
function $h_I(x) = g(x;\theta_I)$ are given by a hypernetwork $f$ as
$\theta_I=f(I)$. In this paper, we define the property of modularity as the
ability to effectively learn a different function for each input instance $I$.
For this purpose, we adopt an expressivity perspective of this property and
extend the theory of Devore et al. 1996 and provide a lower bound on the
complexity (number of trainable parameters) of neural networks as function
approximators, by eliminating the requirements for the approximation method to
be robust. Our results are then used to compare the complexities of $q$ and
$g$, showing that under certain conditions and when letting the functions $e$
and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of
magnitude. This sheds light on the modularity of hypernetworks in comparison
with the embedding-based method. Besides, we show that for a structured target
function, the overall number of trainable parameters in a hypernetwork is
smaller by orders of magnitude than the number of trainable parameters of a
standard neural network and an embedding method.

negative:
The construction of a multilayer perceptron (MLP) as a piecewise low-order
polynomial approximator using a signal processing approach is presented in this
work. The constructed MLP contains one input, one intermediate and one output
layers. Its construction includes the specification of neuron numbers and all
filter weights. Through the construction, a one-to-one correspondence between
the approximation of an MLP and that of a piecewise low-order polynomial is
established. Comparison between piecewise polynomial and MLP approximations is
made. Since the approximation capability of piecewise low-order polynomials is
well understood, our findings shed light on the universal approximation
capability of an MLP.

negative:
In this paper, we propose a probabilistic representation of MultiLayer
Perceptrons (MLPs) to improve the information-theoretic interpretability. Above
all, we demonstrate that the activations being i.i.d. is not valid for all the
hidden layers of MLPs, thus the existing mutual information estimators based on
non-parametric inference methods, e.g., empirical distributions and Kernel
Density Estimate (KDE), are invalid for measuring the information flow in MLPs.
Moreover, we introduce explicit probabilistic explanations for MLPs: (i) we
define the probability space (Omega_F, t, P_F) for a fully connected layer f
and demonstrate the great effect of an activation function on the probability
measure P_F ; (ii) we prove the entire architecture of MLPs as a Gibbs
distribution P; and (iii) the back-propagation aims to optimize the sample
space Omega_F of all the fully connected layers of MLPs for learning an optimal
Gibbs distribution P* to express the statistical connection between the input
and the label. Based on the probabilistic explanations for MLPs, we improve the
information-theoretic interpretability of MLPs in three aspects: (i) the random
variable of f is discrete and the corresponding entropy is finite; (ii) the
information bottleneck theory cannot correctly explain the information flow in
MLPs if we take into account the back-propagation; and (iii) we propose novel
information-theoretic explanations for the generalization of MLPs. Finally, we
demonstrate the proposed probabilistic representation and information-theoretic
explanations for MLPs in a synthetic dataset and benchmark datasets.

negative:
The history of deep learning has shown that human-designed problem-specific
networks can greatly improve the classification performance of general neural
models. In most practical cases, however, choosing the optimal architecture for
a given task remains a challenging problem. Recent architecture-search methods
are able to automatically build neural models with strong performance but fail
to fully appreciate the interaction between neural architecture and weights.
This work investigates the problem of disentangling the role of the neural
structure and its edge weights, by showing that well-trained architectures may
not need any link-specific fine-tuning of the weights. We compare the
performance of such weight-free networks (in our case these are binary networks
with {0, 1}-valued weights) with random, weight-agnostic, pruned and standard
fully connected networks. To find the optimal weight-agnostic network, we use a
novel and computationally efficient method that translates the hard
architecture-search problem into a feasible optimization problem.More
specifically, we look at the optimal task-specific architectures as the optimal
configuration of binary networks with {0, 1}-valued weights, which can be found
through an approximate gradient descent strategy. Theoretical convergence
guarantees of the proposed algorithm are obtained by bounding the error in the
gradient approximation and its practical performance is evaluated on two
real-world data sets. For measuring the structural similarities between
different architectures, we use a novel spectral approach that allows us to
underline the intrinsic differences between real-valued networks and
weight-free architectures.

negative:
The aim of this paper is to propose an application of mutual
information-based ensemble methods to the analysis and classification of heart
beats associated with different types of Arrhythmia. Models of multilayer
perceptrons, support vector machines, and radial basis function neural networks
were trained and tested using the MIT-BIH arrhythmia database. This research
brings a focus to an ensemble method that, to our knowledge, is a novel
application in the area of ECG Arrhythmia detection. The proposed classifier
ensemble method showed improved performance, relative to either majority voting
classifier integration or to individual classifier performance. The overall
ensemble accuracy was 98.25%.

negative:
{\em Hypernetworks} are architectures that produce the weights of a
task-specific {\em primary network}. A notable application of hypernetworks in
the recent literature involves learning to output functional representations.
In these scenarios, the hypernetwork learns a representation corresponding to
the weights of a shallow MLP, which typically encodes shape or image
information. While such representations have seen considerable success in
practice, they remain lacking in the theoretical guarantees in the wide regime
of the standard architectures. In this work, we study wide over-parameterized
hypernetworks. We show that unlike typical architectures, infinitely wide
hypernetworks do not guarantee convergence to a global minima under gradient
descent. We further show that convexity can be achieved by increasing the
dimensionality of the hypernetwork's output, to represent wide MLPs. In the
dually infinite-width regime, we identify the functional priors of these
architectures by deriving their corresponding GP and NTK kernels, the latter of
which we refer to as the {\em hyperkernel}. As part of this study, we make a
mathematical contribution by deriving tight bounds on high order Taylor
expansion terms of standard fully connected ReLU networks.

negative:
We propose an end-to-end model based on convolutional and recurrent neural
networks for speech enhancement. Our model is purely data-driven and does not
make any assumptions about the type or the stationarity of the noise. In
contrast to existing methods that use multilayer perceptrons (MLPs), we employ
both convolutional and recurrent neural network architectures. Thus, our
approach allows us to exploit local structures in both the frequency and
temporal domains. By incorporating prior knowledge of speech signals into the
design of model structures, we build a model that is more data-efficient and
achieves better generalization on both seen and unseen noise. Based on
experiments with synthetic data, we demonstrate that our model outperforms
existing methods, improving PESQ by up to 0.6 on seen noise and 0.64 on unseen
noise.

negative:
Embedding based models have been the state of the art in collaborative
filtering for over a decade. Traditionally, the dot product or higher order
equivalents have been used to combine two or more embeddings, e.g., most
notably in matrix factorization. In recent years, it was suggested to replace
the dot product with a learned similarity e.g. using a multilayer perceptron
(MLP). This approach is often referred to as neural collaborative filtering
(NCF). In this work, we revisit the experiments of the NCF paper that
popularized learned similarities using MLPs. First, we show that with a proper
hyperparameter selection, a simple dot product substantially outperforms the
proposed learned similarities. Second, while a MLP can in theory approximate
any function, we show that it is non-trivial to learn a dot product with an
MLP. Finally, we discuss practical issues that arise when applying MLP based
similarities and show that MLPs are too costly to use for item recommendation
in production environments while dot products allow to apply very efficient
retrieval algorithms. We conclude that MLPs should be used with care as
embedding combiner and that dot products might be a better default choice.

negative:
How to leverage cross-document interactions to improve ranking performance is
an important topic in information retrieval (IR) research. However, this topic
has not been well-studied in the learning-to-rank setting and most of the
existing work still treats each document independently while scoring. The
recent development of deep learning shows strength in modeling complex
relationships across sequences and sets. It thus motivates us to study how to
leverage cross-document interactions for learning-to-rank in the deep learning
framework. In this paper, we formally define the permutation-equivariance
requirement for a scoring function that captures cross-document interactions.
We then propose a self-attention based document interaction network and show
that it satisfies the permutation-equivariant requirement, and can generate
scores for document sets of varying sizes. Our proposed methods can
automatically learn to capture document interactions without any auxiliary
information, and can scale across large document sets. We conduct experiments
on three ranking datasets: the benchmark Web30k, a Gmail search, and a Google
Drive Quick Access dataset. Experimental results show that our proposed methods
are both more effective and efficient than baselines.

negative:
Artificial Neural Networks(ANN) has been phenomenally successful on various
pattern recognition tasks. However, the design of neural networks rely heavily
on the experience and intuitions of individual developers. In this article, the
author introduces a mathematical structure called MLP algebra on the set of all
Multilayer Perceptron Neural Networks(MLP), which can serve as a guiding
principle to build MLPs accommodating to the particular data sets, and to build
complex MLPs from simpler ones.

negative:
We develop a mathematically rigorous framework for multilayer neural networks
in the mean field regime. As the network's widths increase, the network's
learning trajectory is shown to be well captured by a meaningful and
dynamically nonlinear limit (the \textit{mean field} limit), which is
characterized by a system of ODEs. Our framework applies to a broad range of
network architectures, learning dynamics and network initializations. Central
to the framework is the new idea of a \textit{neuronal embedding}, which
comprises of a non-evolving probability space that allows to embed neural
networks of arbitrary widths.
  Using our framework, we prove several properties of large-width multilayer
neural networks. Firstly we show that independent and identically distributed
initializations cause strong degeneracy effects on the network's learning
trajectory when the network's depth is at least four. Secondly we obtain
several global convergence guarantees for feedforward multilayer networks under
a number of different setups. These include two-layer and three-layer networks
with independent and identically distributed initializations, and multilayer
networks of arbitrary depths with a special type of correlated initializations
that is motivated by the new concept of \textit{bidirectional diversity}.
Unlike previous works that rely on convexity, our results admit non-convex
losses and hinge on a certain universal approximation property, which is a
distinctive feature of infinite-width neural networks and is shown to hold
throughout the training process. Aside from being the first known results for
global convergence of multilayer networks in the mean field regime, they
demonstrate flexibility of our framework and incorporate several new ideas and
insights that depart from the conventional convex optimization wisdom.

negative:
Successful sequential recommendation systems rely on accurately capturing the
user's short-term and long-term interest. Although Transformer-based models
achieved state-of-the-art performance in the sequential recommendation task,
they generally require quadratic memory and time complexity to the sequence
length, making it difficult to extract the long-term interest of users. On the
other hand, Multi-Layer Perceptrons (MLP)-based models, renowned for their
linear memory and time complexity, have recently shown competitive results
compared to Transformer in various tasks. Given the availability of a massive
amount of the user's behavior history, the linear memory and time complexity of
MLP-based models make them a promising alternative to explore in the sequential
recommendation task. To this end, we adopted MLP-based models in sequential
recommendation but consistently observed that MLP-based methods obtain lower
performance than those of Transformer despite their computational benefits.
From experiments, we observed that introducing explicit high-order interactions
to MLP layers mitigates such performance gap. In response, we propose the
Multi-Order Interaction (MOI) layer, which is capable of expressing an
arbitrary order of interactions within the inputs while maintaining the memory
and time complexity of the MLP layer. By replacing the MLP layer with the MOI
layer, our model was able to achieve comparable performance with
Transformer-based models while retaining the MLP-based models' computational
benefits.

negative:
We propose a Label Propagation based algorithm for weakly supervised text
classification. We construct a graph where each document is represented by a
node and edge weights represent similarities among the documents. Additionally,
we discover underlying topics using Latent Dirichlet Allocation (LDA) and
enrich the document graph by including the topics in the form of additional
nodes. The edge weights between a topic and a text document represent level of
"affinity" between them. Our approach does not require document level
labelling, instead it expects manual labels only for topic nodes. This
significantly minimizes the level of supervision needed as only a few topics
are observed to be enough for achieving sufficiently high accuracy. The Label
Propagation Algorithm is employed on this enriched graph to propagate labels
among the nodes. Our approach combines the advantages of Label Propagation
(through document-document similarities) and Topic Modelling (for minimal but
smart supervision). We demonstrate the effectiveness of our approach on various
datasets and compare with state-of-the-art weakly supervised text
classification approaches.

negative:
Document classification is a challenging task with important applications.
The deep learning approaches to the problem have gained much attention
recently. Despite the progress, the proposed models do not incorporate the
knowledge of the document structure in the architecture efficiently and not
take into account the contexting importance of words and sentences. In this
paper, we propose a new approach based on a combination of convolutional neural
networks, gated recurrent units, and attention mechanisms for document
classification tasks. The main contribution of this work is the use of
convolution layers to extract more meaningful, generalizable and abstract
features by the hierarchical representation. The proposed method in this paper
improves the results of the current attention-based approaches for document
classification.

negative:
We study the task of semi-supervised learning on multilayer graphs by taking
into account both labeled and unlabeled observations together with the
information encoded by each individual graph layer. We propose a regularizer
based on the generalized matrix mean, which is a one-parameter family of matrix
means that includes the arithmetic, geometric and harmonic means as particular
cases. We analyze it in expectation under a Multilayer Stochastic Block Model
and verify numerically that it outperforms state of the art methods. Moreover,
we introduce a matrix-free numerical scheme based on contour integral
quadratures and Krylov subspace solvers that scales to large sparse multilayer
graphs.

negative:
The question answering system can answer questions from various fields and
forms with deep neural networks, but it still lacks effective ways when facing
multiple evidences. We introduce a new model called SRQA, which means Synthetic
Reader for Factoid Question Answering. This model enhances the question
answering system in the multi-document scenario from three aspects: model
structure, optimization goal, and training method, corresponding to Multilayer
Attention (MA), Cross Evidence (CE), and Adversarial Training (AT)
respectively. First, we propose a multilayer attention network to obtain a
better representation of the evidences. The multilayer attention mechanism
conducts interaction between the question and the passage within each layer,
making the token representation of evidences in each layer takes the
requirement of the question into account. Second, we design a cross evidence
strategy to choose the answer span within more evidences. We improve the
optimization goal, considering all the answers' locations in multiple evidences
as training targets, which leads the model to reason among multiple evidences.
Third, adversarial training is employed to high-level variables besides the
word embedding in our model. A new normalization method is also proposed for
adversarial perturbations so that we can jointly add perturbations to several
target variables. As an effective regularization method, adversarial training
enhances the model's ability to process noisy data. Combining these three
strategies, we enhance the contextual representation and locating ability of
our model, which could synthetically extract the answer span from several
evidences. We perform SRQA on the WebQA dataset, and experiments show that our
model outperforms the state-of-the-art models (the best fuzzy score of our
model is up to 78.56%, with an improvement of about 2%).

negative:
We propose a method for simultaneously detecting shared and unshared
communities in heterogeneous multilayer weighted and undirected networks. The
multilayer network is assumed to follow a generative probabilistic model that
takes into account the similarities and dissimilarities between the
communities. We make use of a variational Bayes approach for jointly inferring
the shared and unshared hidden communities from multilayer network
observations. We show that our approach outperforms state-of-the-art algorithms
in detecting disparate (shared and private) communities on synthetic data as
well as on real genome-wide fibroblast proliferation dataset.

negative:
Multilayer graphs are commonly used for representing different relations
between entities and handling heterogeneous data processing tasks. New
challenges arise in multilayer graph clustering for assigning clusters to a
common multilayer node set and for combining information from each layer. This
paper presents a theoretical framework for multilayer spectral graph clustering
of the nodes via convex layer aggregation. Under a novel multilayer signal plus
noise model, we provide a phase transition analysis that establishes the
existence of a critical value on the noise level that permits reliable cluster
separation. The analysis also specifies analytical upper and lower bounds on
the critical value, where the bounds become exact when the clusters have
identical sizes. Numerical experiments on synthetic multilayer graphs are
conducted to validate the phase transition analysis and study the effect of
layer weights and noise levels on clustering reliability.

negative:
We investigate the representation power of graph neural networks in the
semi-supervised node classification task under heterophily or low homophily,
i.e., in networks where connected nodes may have different class labels and
dissimilar features. Many popular GNNs fail to generalize to this setting, and
are even outperformed by models that ignore the graph structure (e.g.,
multilayer perceptrons). Motivated by this limitation, we identify a set of key
designs -- ego- and neighbor-embedding separation, higher-order neighborhoods,
and combination of intermediate representations -- that boost learning from the
graph structure under heterophily. We combine them into a graph neural network,
H2GCN, which we use as the base method to empirically evaluate the
effectiveness of the identified designs. Going beyond the traditional
benchmarks with strong homophily, our empirical analysis shows that the
identified designs increase the accuracy of GNNs by up to 40% and 27% over
models without them on synthetic and real networks with heterophily,
respectively, and yield competitive performance under homophily.

negative:
Like most learning algorithms, the multilayer perceptrons (MLP) is designed
to learn a vector of parameters from data. However, in certain scenarios we are
interested in learning structured parameters (predictions) in the form of
symmetric positive definite matrices. Here, we introduce a variant of the MLP,
referred to as the matrix MLP, that is specialized at learning symmetric
positive definite matrices. We also present an application of the model within
the context of the variational autoencoder (VAE). Our formulation of the VAE
extends the vanilla formulation to the cases where the recognition and the
generative networks can be from the parametric family of distributions with
dense covariance matrices. Two specific examples are discussed in more detail:
the dense covariance Gaussian and its generalization, the power exponential
distribution. Our new developments are illustrated using both synthetic and
real data.

negative:
The main objective of eXplainable Artificial Intelligence (XAI) is to provide
effective explanations for black-box classifiers. The existing literature lists
many desirable properties for explanations to be useful, but there is no
consensus on how to quantitatively evaluate explanations in practice. Moreover,
explanations are typically used only to inspect black-box models, and the
proactive use of explanations as a decision support is generally overlooked.
Among the many approaches to XAI, a widely adopted paradigm is Local Linear
Explanations - with LIME and SHAP emerging as state-of-the-art methods. We show
that these methods are plagued by many defects including unstable explanations,
divergence of actual implementations from the promised theoretical properties,
and explanations for the wrong label. This highlights the need to have standard
and unbiased evaluation procedures for Local Linear Explanations in the XAI
field. In this paper we address the problem of identifying a clear and
unambiguous set of metrics for the evaluation of Local Linear Explanations.
This set includes both existing and novel metrics defined specifically for this
class of explanations. All metrics have been included in an open Python
framework, named LEAF. The purpose of LEAF is to provide a reference for end
users to evaluate explanations in a standardised and unbiased way, and to guide
researchers towards developing improved explainable techniques.

negative:
Many recent developments in network analysis have focused on multilayer
networks, which one can use to encode time-dependent interactions, multiple
types of interactions, and other complications that arise in complex systems.
Like their monolayer counterparts, multilayer networks in applications often
have mesoscale features, such as community structure. A prominent type of
method for inferring such structures is the employment of multilayer stochastic
block models (SBMs). A common (but {potentially} inadequate) assumption of
these models is the sampling of edges in different layers independently,
conditioned on the community labels of the nodes. In this paper, we relax this
assumption of independence by incorporating edge correlations into an SBM-like
model. We derive maximum-likelihood estimates of the key parameters of our
model, and we propose a measure of layer correlation that reflects the
similarity between connectivity patterns in different layers. Finally, we
explain how to use correlated models for edge "prediction" (i.e., inference) in
multilayer networks. By taking into account edge correlations, prediction
accuracy improves both in synthetic networks and in a temporal network of
shoppers who are connected to previously-purchased grocery products.

negative:
For the past ten years, CNN has reigned supreme in the world of computer
vision, but recently, Transformer has been on the rise. However, the quadratic
computational cost of self-attention has become a serious problem in practice
applications. There has been much research on architectures without CNN and
self-attention in this context. In particular, MLP-Mixer is a simple
architecture designed using MLPs and hit an accuracy comparable to the Vision
Transformer. However, the only inductive bias in this architecture is the
embedding of tokens. This leaves open the possibility of incorporating a
non-convolutional (or non-local) inductive bias into the architecture, so we
used two simple ideas to incorporate inductive bias into the MLP-Mixer while
taking advantage of its ability to capture global correlations. A way is to
divide the token-mixing block vertically and horizontally. Another way is to
make spatial correlations denser among some channels of token-mixing. With this
approach, we were able to improve the accuracy of the MLP-Mixer while reducing
its parameters and computational complexity. The small model that is RaftMLP-S
is comparable to the state-of-the-art global MLP-based model in terms of
parameters and efficiency per calculation. In addition, we tackled the problem
of fixed input image resolution for global MLP-based models by utilizing
bicubic interpolation. We demonstrated that these models could be applied as
the backbone of architectures for downstream tasks such as object detection.
However, it did not have significant performance and mentioned the need for
MLP-specific architectures for downstream tasks for global MLP-based models.
The source code in PyTorch version is available at
\url{https://github.com/okojoalg/raft-mlp}.

negative:
High-level applications, such as machine learning, are evolving from simple
models based on multilayer perceptrons for simple image recognition to much
deeper and more complex neural networks for self-driving vehicle control
systems.The rapid increase in the consumption of memory and computational
resources by these models demands the use of multi-core parallel systems to
scale the execution of the complex emerging applications that depend on them.
However, parallel programs running on high-performance computers often suffer
from data communication bottlenecks, limited memory bandwidth, and
synchronization overhead due to irregular critical sections. In this paper, we
propose a framework to reduce the data communication and improve the
scalability and performance of these applications in multi-core systems. We
design a vertex cut framework for partitioning LLVM IR graphs into clusters
while taking into consideration the data communication and workload balance
among clusters. First, we construct LLVM graphs by compiling high-level
programs into LLVM IR, instrumenting code to obtain the execution order of
basic blocks and the execution time for each memory operation, and analyze data
dependencies in dynamic LLVM traces. Next, we formulate the problem as Weight
Balanced $p$-way Vertex Cut, and propose a generic and flexible framework,
wherein four different greedy algorithms are proposed for solving this problem.
Lastly, we propose a memory-centric run-time mapping of the linear time
complexity to map clusters generated from the vertex cut algorithms onto a
multi-core platform. We conclude that our best algorithm, WB-Libra, provides
performance improvements of 1.56x and 1.86x over existing state-of-the-art
approaches for 8 and 1024 clusters running on a multi-core platform,
respectively.

negative:
This paper is about a machine learning approach based on the multilinear
projection of an unknown function (or probability distribution) to be estimated
towards a linear (or multilinear) dimensional space E'. The proposal transforms
the problem of predicting the target of an observation x into a problem of
determining a consensus among the k nearest neighbors of x's image within the
dimensional space E'. The algorithms that concretize it allow both regression
and binary classification. Implementations carried out using Scala/Spark and
assessed on a dozen LIBSVM datasets have demonstrated improvements in
prediction accuracies in comparison with other prediction algorithms
implemented within Spark MLLib such as multilayer perceptrons, logistic
regression classifiers and random forests.

negative:
Image denoising can be described as the problem of mapping from a noisy image
to a noise-free image. The best currently available denoising methods
approximate this mapping with cleverly engineered algorithms. In this work we
attempt to learn this mapping directly with plain multi layer perceptrons (MLP)
applied to image patches. We will show that by training on large image
databases we are able to outperform the current state-of-the-art image
denoising methods. In addition, our method achieves results that are superior
to one type of theoretical bound and goes a large way toward closing the gap
with a second type of theoretical bound. Our approach is easily adapted to less
extensively studied types of noise, such as mixed Poisson-Gaussian noise, JPEG
artifacts, salt-and-pepper noise and noise resembling stripes, for which we
achieve excellent results as well. We will show that combining a block-matching
procedure with MLPs can further improve the results on certain images. In a
second paper, we detail the training trade-offs and the inner mechanisms of our
MLPs.

negative:
We present here a new model and algorithm which performs an efficient Natural
gradient descent for Multilayer Perceptrons. Natural gradient descent was
originally proposed from a point of view of information geometry, and it
performs the steepest descent updates on manifolds in a Riemannian space. In
particular, we extend an approach taken by the "Whitened neural networks"
model. We make the whitening process not only in feed-forward direction as in
the original model, but also in the back-propagation phase. Its efficacy is
shown by an application of this "Bidirectional whitened neural networks" model
to a handwritten character recognition data (MNIST data).

negative:
We present a novel hierarchical approach to multi-class classification which
is generic in that it can be applied to different classification models (e.g.,
support vector machines, perceptrons), and makes no explicit assumptions about
the probabilistic structure of the problem as it is usually done in multi-class
classification. By adding a cascade of additional classifiers, each of which
receives the previous classifier's output in addition to regular input data,
the approach harnesses unused information that manifests itself in the form of,
e.g., correlations between predicted classes. Using multilayer perceptrons as a
classification model, we demonstrate the validity of this approach by testing
it on a complex ten-class 3D gesture recognition task.

negative:
We proposed a novel multilayer correlated topic model (MCTM) to analyze how
the main ideas inherit and vary between a document and its different segments,
which helps understand an article's structure. The variational
expectation-maximization (EM) algorithm was derived to estimate the posterior
and parameters in MCTM. We introduced two potential applications of MCTM,
including the paragraph-level document analysis and market basket data
analysis. The effectiveness of MCTM in understanding the document structure has
been verified by the great predictive performance on held-out documents and
intuitive visualization. We also showed that MCTM could successfully capture
customers' popular shopping patterns in the market basket analysis.

negative:
This paper presents a new state-of-the-art for document image classification
and retrieval, using features learned by deep convolutional neural networks
(CNNs). In object and scene analysis, deep neural nets are capable of learning
a hierarchical chain of abstraction from pixel inputs to concise and
descriptive representations. The current work explores this capacity in the
realm of document analysis, and confirms that this representation strategy is
superior to a variety of popular hand-crafted alternatives. Experiments also
show that (i) features extracted from CNNs are robust to compression, (ii) CNNs
trained on non-document images transfer well to document analysis tasks, and
(iii) enforcing region-specific feature-learning is unnecessary given
sufficient training data. This work also makes available a new labelled subset
of the IIT-CDIP collection, containing 400,000 document images across 16
categories, useful for training new CNNs for document analysis.

negative:
Most of the textual information available to us are temporally variable. In a
world where information is dynamic, time-stamping them is a very important
task. Documents are a good source of information and are used for many tasks
like, sentiment analysis, classification of reviews etc. The knowledge of
creation date of documents facilitates several tasks like summarization, event
extraction, temporally focused information extraction etc. Unfortunately, for
most of the documents on the web, the time-stamp meta-data is either erroneous
or missing. Thus document dating is a challenging problem which requires
inference over the temporal structure of the document alongside the contextual
information of the document. Prior document dating systems have largely relied
on handcrafted features while ignoring such document-internal structures. In
this paper we propose NeuralDater, a Graph Convolutional Network (GCN) based
document dating approach which jointly exploits syntactic and temporal graph
structures of document in a principled way. We also pointed out some
limitations of NeuralDater and tried to utilize both context and temporal
information in documents in a more flexible and intuitive manner proposing AD3:
Attentive Deep Document Dater, an attention-based document dating system. To
the best of our knowledge these are the first application of deep learning
methods for the task. Through extensive experiments on real-world datasets, we
find that our models significantly outperforms state-of-the-art baselines by a
significant margin.

negative:
This paper shows that a long chain of perceptrons (that is, a multilayer
perceptron, or MLP, with many hidden layers of width one) can be a universal
classifier. The classification procedure is not necessarily computationally
efficient, but the technique throws some light on the kind of computations
possible with narrow and deep MLPs.

negative:
The adoption of intelligent systems with Artificial Neural Networks (ANNs)
embedded in hardware for real-time applications currently faces a growing
demand in fields like the Internet of Things (IoT) and Machine to Machine
(M2M). However, the application of ANNs in this type of system poses a
significant challenge due to the high computational power required to process
its basic operations. This paper aims to show an implementation strategy of a
Multilayer Perceptron (MLP) type neural network, in a microcontroller (a
low-cost, low-power platform). A modular matrix-based MLP with the full
classification process was implemented, and also the backpropagation training
in the microcontroller. The testing and validation were performed through
Hardware in the Loop (HIL) of the Mean Squared Error (MSE) of the training
process, classification result, and the processing time of each implementation
module. The results revealed a linear relationship between the values of the
hyperparameters and the processing time required for classification, also the
processing time concurs with the required time for many applications on the
fields mentioned above. These findings show that this implementation strategy
and this platform can be applied successfully on real-time applications that
require the capabilities of ANNs.

negative:
Artificial Neural Networks have emerged as an important tool for
classification and have been widely used to classify a non-linear separable
pattern. The most popular artificial neural networks model is a Multilayer
Perceptron (MLP) as it is able to perform classification task with significant
success. However due to the complexity of MLP structure and also problems such
as local minima trapping, over fitting and weight interference have made neural
network training difficult. Thus, the easy way to avoid these problems is to
remove the hidden layers. This paper presents the ability of Functional Link
Neural Network (FLNN) to overcome the complexity structure of MLP by using
single layer architecture and propose an Artificial Bee Colony (ABC)
optimization for training the FLNN. The proposed technique is expected to
provide better learning scheme for a classifier in order to get more accurate
classification result

negative:
One of the major prerequisites for any deep learning approach is the
availability of large-scale training data. When dealing with scanned document
images in real world scenarios, the principal information of its content is
stored in the layout itself. In this work, we have proposed an automated deep
generative model using Graph Neural Networks (GNNs) to generate synthetic data
with highly variable and plausible document layouts that can be used to train
document interpretation systems, in this case, specially in digital mailroom
applications. It is also the first graph-based approach for document layout
generation task experimented on administrative document images, in this case,
invoices.

negative:
Aiming at drastic speedup for point-feature embeddings at test time, we
propose a new framework that uses a pair of multi-layer perceptrons (MLP) and a
lookup table (LUT) to transform point-coordinate inputs into high-dimensional
features. When compared with PointNet's feature embedding part realized by MLP
that requires millions of dot products, the proposed framework at test time
requires no such layers of matrix-vector products but requires only looking up
the nearest entities from the tabulated MLP followed by interpolation, defined
over discrete inputs on a 3D lattice that is substantially arranged
irregularly. We call this framework LUTI-MLP: LUT Interpolation ML that
provides a way to train end-to-end irregularly tabulated MLP coupled to a LUT
in a specific manner without the need for any approximation at test time.
LUTI-MLP also provides significant speedup for Jacobian computation of the
embedding function wrt global pose coordinate on Lie algebra $\mathfrak{se}(3)$
at test time, which could be used for point-set registration problems. After
extensive evaluation using the ModelNet40, we confirmed that the LUTI-MLP even
with a small (e.g., $4^3$) lattice yields performance comparable to that of the
MLP while achieving significant speedup: $100\times$ for the embedding,
$12\times$ for the approximate Jacobian, and $860\times$ for the canonical
Jacobian.

negative:
Perceptrons are neuronal devices capable of fully discriminating linearly
separable classes. Although straightforward to implement and train, their
applicability is usually hindered by non-trivial requirements imposed by
real-world classification problems. Therefore, several approaches, such as
kernel perceptrons, have been conceived to counteract such difficulties. In
this paper, we investigate an enhanced perceptron model based on the notion of
contrastive biclusters. From this perspective, a good discriminative bicluster
comprises a subset of data instances belonging to one class that show high
coherence across a subset of features and high differentiation from nearest
instances of the other class under the same features (referred to as its
contrastive bicluster). Upon each local subspace associated with a pair of
contrastive biclusters a perceptron is trained and the model with highest area
under the receiver operating characteristic curve (AUC) value is selected as
the final classifier. Experiments conducted on a range of data sets, including
those related to a difficult biosignal classification problem, show that the
proposed variant can be indeed very useful, prevailing in most of the cases
upon standard and kernel perceptrons in terms of accuracy and AUC measures.

negative:
In this paper, we focus on a typical two-phase phenomenon in the learning of
multi-layer perceptrons (MLPs), and we aim to explain the reason for the
decrease of feature diversity in the first phase. Specifically, people find
that, in the training of MLPs, the training loss does not decrease
significantly until the second phase. To this end, we further explore the
reason why the diversity of features over different samples keeps decreasing in
the first phase, which hurts the optimization of MLPs. We explain such a
phenomenon in terms of the learning dynamics of MLPs. Furthermore, we
theoretically explain why four typical operations can alleviate the decrease of
the feature diversity.

negative:
Nowadays, neural networks are widely used in many applications as artificial
intelligence models for learning tasks. Since typically neural networks process
a very large amount of data, it is convenient to formulate them within the
mean-field and kinetic theory. In this work we focus on a particular class of
neural networks, i.e. the residual neural networks, assuming that each layer is
characterized by the same number of neurons $N$, which is fixed by the
dimension of the data. This assumption allows to interpret the residual neural
network as a time-discretized ordinary differential equation, in analogy with
neural differential equations. The mean-field description is then obtained in
the limit of infinitely many input data. This leads to a Vlasov-type partial
differential equation which describes the evolution of the distribution of the
input data. We analyze steady states and sensitivity with respect to the
parameters of the network, namely the weights and the bias. In the simple
setting of a linear activation function and one-dimensional input data, the
study of the moments provides insights on the choice of the parameters of the
network. Furthermore, a modification of the microscopic dynamics, inspired by
stochastic residual neural networks, leads to a Fokker-Planck formulation of
the network, in which the concept of network training is replaced by the task
of fitting distributions. The performed analysis is validated by artificial
numerical simulations. In particular, results on classification and regression
problems are presented.

negative:
The ability to identify applications based on the network data they generate
could be a valuable tool for cyber defense. We report on a machine learning
technique capable of using netflow-like features to predict the application
that generated the traffic. In our experiments, we used ground-truth labels
obtained from host-based sensors deployed in a large enterprise environment; we
applied random forests and multilayer perceptrons to the tasks of browser vs.
non-browser identification, browser fingerprinting, and process name
prediction. For each of these tasks, we demonstrate how machine learning models
can achieve high classification accuracy using only netflow-like features as
the basis for classification.

negative:
Recently, visual Transformer (ViT) and its following works abandon the
convolution and exploit the self-attention operation, attaining a comparable or
even higher accuracy than CNNs. More recently, MLP-Mixer abandons both the
convolution and the self-attention operation, proposing an architecture
containing only MLP layers. To achieve cross-patch communications, it devises
an additional token-mixing MLP besides the channel-mixing MLP. It achieves
promising results when training on an extremely large-scale dataset. But it
cannot achieve as outstanding performance as its CNN and ViT counterparts when
training on medium-scale datasets such as ImageNet1K and ImageNet21K. The
performance drop of MLP-Mixer motivates us to rethink the token-mixing MLP. We
discover that the token-mixing MLP is a variant of the depthwise convolution
with a global reception field and spatial-specific configuration. But the
global reception field and the spatial-specific property make token-mixing MLP
prone to over-fitting. In this paper, we propose a novel pure MLP architecture,
spatial-shift MLP (S$^2$-MLP). Different from MLP-Mixer, our S$^2$-MLP only
contains channel-mixing MLP. We utilize a spatial-shift operation for
communications between patches. It has a local reception field and is
spatial-agnostic. It is parameter-free and efficient for computation. The
proposed S$^2$-MLP attains higher recognition accuracy than MLP-Mixer when
training on ImageNet-1K dataset. Meanwhile, S$^2$-MLP accomplishes as excellent
performance as ViT on ImageNet-1K dataset with considerably simpler
architecture and fewer FLOPs and parameters.

negative:
The light scattering of multilayer nanoparticles can be solved by Maxwell
equations. However, it is difficult to solve the inverse design of multilayer
nanoparticles by using the traditional trial-and-error method. Here, we present
a method for forward simulation and inverse design of multilayer nanoparticles.
We combine the global search ability of genetic algorithm with the local search
ability of neural network. First, the genetic algorithm is used to find a
suitable solution, and then the neural network is used to fine-tune it. Due to
the non-unique relationship between physical structures and optical responses,
we first train a forward neural network, and then it is applied to the inverse
design of multilayer nanoparticles. Not only here, this method can easily be
extended to predict and find the best design parameters for other optical
structures.

negative:
In this work, a region-based Deep Convolutional Neural Network framework is
proposed for document structure learning. The contribution of this work
involves efficient training of region based classifiers and effective
ensembling for document image classification. A primary level of `inter-domain'
transfer learning is used by exporting weights from a pre-trained VGG16
architecture on the ImageNet dataset to train a document classifier on whole
document images. Exploiting the nature of region based influence modelling, a
secondary level of `intra-domain' transfer learning is used for rapid training
of deep learning models for image segments. Finally, stacked generalization
based ensembling is utilized for combining the predictions of the base deep
neural network models. The proposed method achieves state-of-the-art accuracy
of 92.2% on the popular RVL-CDIP document image dataset, exceeding benchmarks
set by existing algorithms.

negative:
Metric learning aims to learn a distance metric such that semantically
similar instances are pulled together while dissimilar instances are pushed
away. Many existing methods consider maximizing or at least constraining a
distance margin in the feature space that separates similar and dissimilar
pairs of instances to guarantee their generalization ability. In this paper, we
advocate imposing an adversarial margin in the input space so as to improve the
generalization and robustness of metric learning algorithms. We first show
that, the adversarial margin, defined as the distance between training
instances and their closest adversarial examples in the input space, takes
account of both the distance margin in the feature space and the correlation
between the metric and triplet constraints. Next, to enhance robustness to
instance perturbation, we propose to enlarge the adversarial margin through
minimizing a derived novel loss function termed the perturbation loss. The
proposed loss can be viewed as a data-dependent regularizer and easily plugged
into any existing metric learning methods. Finally, we show that the enlarged
margin is beneficial to the generalization ability by using the theoretical
technique of algorithmic robustness. Experimental results on 16 datasets
demonstrate the superiority of the proposed method over existing
state-of-the-art methods in both discrimination accuracy and robustness against
possible noise.

negative:
In this work we use Recurrent Neural Networks and Multilayer Perceptrons to
predict NYSE, NASDAQ and AMEX stock prices from historical data. We experiment
with different architectures and compare data normalization techniques. Then,
we leverage those findings to question the efficient-market hypothesis through
a formal statistical test.

negative:
Modern entity linking systems rely on large collections of documents
specifically annotated for the task (e.g., AIDA CoNLL). In contrast, we propose
an approach which exploits only naturally occurring information: unlabeled
documents and Wikipedia. Our approach consists of two stages. First, we
construct a high recall list of candidate entities for each mention in an
unlabeled document. Second, we use the candidate lists as weak supervision to
constrain our document-level entity linking model. The model treats entities as
latent variables and, when estimated on a collection of unlabelled texts,
learns to choose entities relying both on local context of each mention and on
coherence with other entities in the document. The resulting approach rivals
fully-supervised state-of-the-art systems on standard test sets. It also
approaches their performance in the very challenging setting: when tested on a
test set sampled from the data used to estimate the supervised systems. By
comparing to Wikipedia-only training of our model, we demonstrate that modeling
unlabeled documents is beneficial.

negative:
Recent approaches in literature have exploited the multi-modal information in
documents (text, layout, image) to serve specific downstream document tasks.
However, they are limited by their - (i) inability to learn cross-modal
representations across text, layout and image dimensions for documents and (ii)
inability to process multi-page documents. Pre-training techniques have been
shown in Natural Language Processing (NLP) domain to learn generic textual
representations from large unlabelled datasets, applicable to various
downstream NLP tasks. In this paper, we propose a multi-task learning-based
framework that utilizes a combination of self-supervised and supervised
pre-training tasks to learn a generic document representation applicable to
various downstream document tasks. Specifically, we introduce Document Topic
Modelling and Document Shuffle Prediction as novel pre-training tasks to learn
rich image representations along with the text and layout representations for
documents. We utilize the Longformer network architecture as the backbone to
encode the multi-modal information from multi-page documents in an end-to-end
fashion. We showcase the applicability of our pre-training framework on a
variety of different real-world document tasks such as document classification,
document information extraction, and document retrieval. We evaluate our
framework on different standard document datasets and conduct exhaustive
experiments to compare performance against various ablations of our framework
and state-of-the-art baselines.

negative:
Multilayer graphs are commonly used for representing different relations
between entities and handling heterogeneous data processing tasks. Non-standard
multilayer graph clustering methods are needed for assigning clusters to a
common multilayer node set and for combining information from each layer. This
paper presents a multilayer spectral graph clustering (SGC) framework that
performs convex layer aggregation. Under a multilayer signal plus noise model,
we provide a phase transition analysis of clustering reliability. Moreover, we
use the phase transition criterion to propose a multilayer iterative model
order selection algorithm (MIMOSA) for multilayer SGC, which features automated
cluster assignment and layer weight adaptation, and provides statistical
clustering reliability guarantees. Numerical simulations on synthetic
multilayer graphs verify the phase transition analysis, and experiments on
real-world multilayer graphs show that MIMOSA is competitive or better than
other clustering methods.

negative:
Multilayer perceptron (MLP) is a class of networks composed of multiple
layers of perceptrons, and it is essentially a mathematical function. Based on
MLP, we develop a new numerical method to find the extrema of functionals. As
demonstrations, we present our solutions in three physic scenes. Ideally, the
same method is applicable to any cases where the objective curve/surface can be
fitted by second-order differentiable functions. This method can also be
extended to cases where there are a finite number of non-differentiable (but
continuous) points/surfaces.

negative:
We are interested in the widespread problem of clustering documents and
finding topics in large collections of written documents in the presence of
metadata and hyperlinks. To tackle the challenge of accounting for these
different types of datasets, we propose a novel framework based on Multilayer
Networks and Stochastic Block Models. The main innovation of our approach over
other techniques is that it applies the same non-parametric probabilistic
framework to the different sources of datasets simultaneously. The key
difference to other multilayer complex networks is the strong unbalance between
the layers, with the average degree of different node types scaling differently
with system size. We show that the latter observation is due to generic
properties of text, such as Heaps' law, and strongly affects the inference of
communities. We present and discuss the performance of our method in different
datasets (hundreds of Wikipedia documents, thousands of scientific papers, and
thousands of E-mails) showing that taking into account multiple types of
information provides a more nuanced view on topic- and document-clusters and
increases the ability to predict missing links.

negative:
Policy distillation in deep reinforcement learning provides an effective way
to transfer control policies from a larger network to a smaller untrained
network without a significant degradation in performance. However, policy
distillation is underexplored in deep reinforcement learning, and existing
approaches are computationally inefficient, resulting in a long distillation
time. In addition, the effectiveness of the distillation process is still
limited to the model capacity. We propose a new distillation mechanism, called
real-time policy distillation, in which training the teacher model and
distilling the policy to the student model occur simultaneously. Accordingly,
the teacher's latest policy is transferred to the student model in real time.
This reduces the distillation time to half the original time or even less and
also makes it possible for extremely small student models to learn skills at
the expert level. We evaluated the proposed algorithm in the Atari 2600 domain.
The results show that our approach can achieve full distillation in most games,
even with compression ratios up to 1.7%.

negative:
We propose a new framework for estimating generative models via an
adversarial process, in which we simultaneously train two models: a generative
model G that captures the data distribution, and a discriminative model D that
estimates the probability that a sample came from the training data rather than
G. The training procedure for G is to maximize the probability of D making a
mistake. This framework corresponds to a minimax two-player game. In the space
of arbitrary functions G and D, a unique solution exists, with G recovering the
training data distribution and D equal to 1/2 everywhere. In the case where G
and D are defined by multilayer perceptrons, the entire system can be trained
with backpropagation. There is no need for any Markov chains or unrolled
approximate inference networks during either training or generation of samples.
Experiments demonstrate the potential of the framework through qualitative and
quantitative evaluation of the generated samples.

negative:
Rapid increase of digitized document give birth to high demand of document
image retrieval. While conventional document image retrieval approaches depend
on complex OCR-based text recognition and text similarity detection, this paper
proposes a new content-based approach, in which more attention is paid to
features extraction and fusion. In the proposed approach, multiple features of
document images are extracted by different CNN models. After that, the
extracted CNN features are reduced and fused into weighted average feature.
Finally, the document images are ranked based on feature similarity to a
provided query image. Experimental procedure is performed on a group of
document images that transformed from academic papers, which contain both
English and Chinese document, the results show that the proposed approach has
good ability to retrieve document images with similar text content, and the
fusion of CNN features can effectively improve the retrieval accuracy.

negative:
Neural Networks (NNs) can be used to solve Ordinary and Partial Differential
Equations (ODEs and PDEs) by redefining the question as an optimization
problem. The objective function to be optimized is the sum of the squares of
the PDE to be solved and of the initial/boundary conditions. A feed forward NN
is trained to minimise this loss function evaluated on a set of collocation
points sampled from the domain where the problem is defined. A compact and
smooth solution, that only depends on the weights of the trained NN, is then
obtained. This approach is often referred to as PINN, from Physics Informed
Neural Network~\cite{raissi2017physics_1, raissi2017physics_2}. Despite the
success of the PINN approach in solving various classes of PDEs, an
implementation of this idea that is capable of solving a large class of ODEs
and PDEs with good accuracy and without the need to finely tune the
hyperparameters of the network, is not available yet. In this paper, we
introduce a new implementation of this concept - called dNNsolve - that makes
use of dual Neural Networks to solve ODEs/PDEs. These include: i) sine and
sigmoidal activation functions, that provide a more efficient basis to capture
both secular and periodic patterns in the solutions; ii) a newly designed
architecture, that makes it easy for the the NN to approximate the solution
using the basis functions mentioned above. We show that dNNsolve is capable of
solving a broad range of ODEs/PDEs in 1, 2 and 3 spacetime dimensions, without
the need of hyperparameter fine-tuning.

negative:
The significance of heating load (HL) accurate approximation is the primary
motivation of this research to distinguish the most efficient predictive model
among several neural-metaheuristic models. The proposed models are through
synthesizing multi-layer perceptron network (MLP) with ant lion optimization
(ALO), biogeography-based optimization (BBO), dragonfly algorithm (DA),
evolutionary strategy (ES), invasive weed optimization (IWO), and league
champion optimization (LCA) hybrid algorithms. Each ensemble is optimized in
terms of the operating population. Accordingly, the ALO-MLP, BBO-MLP, DA-MLP,
ES-MLP, IWO-MLP, and LCA-MLP presented their best performance for population
sizes of 350, 400, 200, 500, 50, and 300, respectively. The comparison was
carried out by implementing a ranking system. Based on the obtained overall
scores (OSs), the BBO (OS = 36) featured as the most capable optimization
technique, followed by ALO (OS = 27) and ES (OS = 20). Due to the efficient
performance of these algorithms, the corresponding MLPs can be promising
substitutes for traditional methods used for HL analysis.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.