query: In the field of robotic manipulation, I have consistently grappled with the challenge of deploying robots in unfamiliar and evolving environments. These environments could range from outdoor settings to busy traffic roads or even battlefields. My objective is to enable my robot to operate efficiently in these new surroundings, acquiring and applying new knowledge to better adapt to the environment. My current approach is based on the premise that a diverse environment offers a variety of rewards. Consequently, my robotic agent should be capable of learning to optimize a diverse set of rewards. Moreover, the agent should be able to swiftly adapt when the rewards (or objectives) change, especially when the environment remains constant. Ultimately, this system should empower robots to learn from and adapt to changes in their environment and tasks. This would involve incrementally constructing models based on their own experiences, thereby enhancing their overall performance in real-world scenarios.

positive:
One of the great promises of robot learning systems is that they will be able
to learn from their mistakes and continuously adapt to ever-changing
environments. Despite this potential, most of the robot learning systems today
are deployed as a fixed policy and they are not being adapted after their
deployment. Can we efficiently adapt previously learned behaviors to new
environments, objects and percepts in the real world? In this paper, we present
a method and empirical evidence towards a robot learning framework that
facilitates continuous adaption. In particular, we demonstrate how to adapt
vision-based robotic manipulation policies to new variations by fine-tuning via
off-policy reinforcement learning, including changes in background, object
shape and appearance, lighting conditions, and robot morphology. Further, this
adaptation uses less than 0.2% of the data necessary to learn the task from
scratch. We find that our approach of adapting pre-trained policies leads to
substantial performance gains over the course of fine-tuning, and that
pre-training via RL is essential: training from scratch or adapting from
supervised ImageNet features are both unsuccessful with such small amounts of
data. We also find that these positive results hold in a limited continual
learning setting, in which we repeatedly fine-tune a single lineage of policies
using data from a succession of new tasks. Our empirical conclusions are
consistently supported by experiments on simulated manipulation tasks, and by
52 unique fine-tuning experiments on a real robotic grasping system pre-trained
on 580,000 grasps.

positive:
AI systems that learn through reward feedback about the actions they take are
increasingly deployed in domains that have significant impact on our daily
life. However, in many cases the online rewards should not be the only guiding
criteria, as there are additional constraints and/or priorities imposed by
regulations, values, preferences, or ethical principles. We detail a novel
online agent that learns a set of behavioral constraints by observation and
uses these learned constraints as a guide when making decisions in an online
setting while still being reactive to reward feedback. To define this agent, we
propose to adopt a novel extension to the classical contextual multi-armed
bandit setting and we provide a new algorithm called Behavior Constrained
Thompson Sampling (BCTS) that allows for online learning while obeying
exogenous constraints. Our agent learns a constrained policy that implements
the observed behavioral constraints demonstrated by a teacher agent, and then
uses this constrained policy to guide the reward-based online exploration and
exploitation. We characterize the upper bound on the expected regret of the
contextual bandit algorithm that underlies our agent and provide a case study
with real world data in two application domains. Our experiments show that the
designed agent is able to act within the set of behavior constraints without
significantly degrading its overall reward performance.

positive:
Learning algorithms are enabling robots to solve increasingly challenging
real-world tasks. These approaches often rely on demonstrations and reproduce
the behavior shown. Unexpected changes in the environment may require using
different behaviors to achieve the same effect, for instance to reach and grasp
an object in changing clutter. An emerging paradigm addressing this robustness
issue is to learn a diverse set of successful behaviors for a given task, from
which a robot can select the most suitable policy when faced with a new
environment. In this paper, we explore a novel realization of this vision by
learning a generative model over policies. Rather than learning a single
policy, or a small fixed repertoire, our generative model for policies
compactly encodes an unbounded number of policies and allows novel controller
variants to be sampled. Leveraging our generative policy network, a robot can
sample novel behaviors until it finds one that works for a new environment. We
demonstrate this idea with an application of robust ball-throwing in the
presence of obstacles. We show that this approach achieves a greater diversity
of behaviors than an existing evolutionary approach, while maintaining good
efficacy of sampled behaviors, allowing a Baxter robot to hit targets more
often when ball throwing in the presence of obstacles.

positive:
The success of reinforcement learning for real world robotics has been, in
many cases limited to instrumented laboratory scenarios, often requiring
arduous human effort and oversight to enable continuous learning. In this work,
we discuss the elements that are needed for a robotic learning system that can
continually and autonomously improve with data collected in the real world. We
propose a particular instantiation of such a system, using dexterous
manipulation as our case study. Subsequently, we investigate a number of
challenges that come up when learning without instrumentation. In such
settings, learning must be feasible without manually designed resets, using
only on-board perception, and without hand-engineered reward functions. We
propose simple and scalable solutions to these challenges, and then demonstrate
the efficacy of our proposed system on a set of dexterous robotic manipulation
tasks, providing an in-depth analysis of the challenges associated with this
learning paradigm. We demonstrate that our complete system can learn without
any human intervention, acquiring a variety of vision-based skills with a
real-world three-fingered hand. Results and videos can be found at
https://sites.google.com/view/realworld-rl/

positive:
While reinforcement learning provides an appealing formalism for learning
individual skills, a general-purpose robotic system must be able to master an
extensive repertoire of behaviors. Instead of learning a large collection of
skills individually, can we instead enable a robot to propose and practice its
own behaviors automatically, learning about the affordances and behaviors that
it can perform in its environment, such that it can then repurpose this
knowledge once a new task is commanded by the user? In this paper, we study
this question in the context of self-supervised goal-conditioned reinforcement
learning. A central challenge in this learning regime is the problem of goal
setting: in order to practice useful skills, the robot must be able to
autonomously set goals that are feasible but diverse. When the robot's
environment and available objects vary, as they do in most open-world settings,
the robot must propose to itself only those goals that it can accomplish in its
present setting with the objects that are at hand. Previous work only studies
self-supervised goal-conditioned RL in a single-environment setting, where goal
proposals come from the robot's past experience or a generative model are
sufficient. In more diverse settings, this frequently leads to impossible goals
and, as we show experimentally, prevents effective learning. We propose a
conditional goal-setting model that aims to propose goals that are feasible
from the robot's current state. We demonstrate that this enables
self-supervised goal-conditioned off-policy learning with raw image
observations in the real world, enabling a robot to manipulate a variety of
objects and generalize to new objects that were not seen during training.

positive:
In order to be effective general purpose machines in real world environments,
robots not only will need to adapt their existing manipulation skills to new
circumstances, they will need to acquire entirely new skills on-the-fly. A
great promise of continual learning is to endow robots with this ability, by
using their accumulated knowledge and experience from prior skills. We take a
fresh look at this problem, by considering a setting in which the robot is
limited to storing that knowledge and experience only in the form of learned
skill policies. We show that storing skill policies, careful pre-training, and
appropriately choosing when to transfer those skill policies is sufficient to
build a continual learner in the context of robotic manipulation. We analyze
which conditions are needed to transfer skills in the challenging Meta-World
simulation benchmark. Using this analysis, we introduce a pair-wise metric
relating skills that allows us to predict the effectiveness of skill transfer
between tasks, and use it to reduce the problem of continual learning to
curriculum selection. Given an appropriate curriculum, we show how to
continually acquire robotic manipulation skills without forgetting, and using
far fewer samples than needed to train them from scratch.

positive:
We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in
the context of Reinforcement Learning (RL). SAC-X enables learning of complex
behaviors - from scratch - in the presence of multiple sparse reward signals.
To this end, the agent is equipped with a set of general auxiliary tasks, that
it attempts to learn simultaneously via off-policy RL. The key idea behind our
method is that active (learned) scheduling and execution of auxiliary policies
allows the agent to efficiently explore its environment - enabling it to excel
at sparse reward RL. Our experiments in several challenging robotic
manipulation settings demonstrate the power of our approach.

positive:
A core challenge for an autonomous agent acting in the real world is to adapt
its repertoire of skills to cope with its noisy perception and dynamics. To
scale learning of skills to long-horizon tasks, robots should be able to learn
and later refine their skills in a structured manner through trajectories
rather than making instantaneous decisions individually at each time step. To
this end, we propose the Soft Actor-Critic Gaussian Mixture Model (SAC-GMM), a
novel hybrid approach that learns robot skills through a dynamical system and
adapts the learned skills in their own trajectory distribution space through
interactions with the environment. Our approach combines classical robotics
techniques of learning from demonstration with the deep reinforcement learning
framework and exploits their complementary nature. We show that our method
utilizes sensors solely available during the execution of preliminarily learned
skills to extract relevant features that lead to faster skill refinement.
Extensive evaluations in both simulation and real-world environments
demonstrate the effectiveness of our method in refining robot skills by
leveraging physical interactions, high-dimensional sensory data, and sparse
task completion rewards. Videos, code, and pre-trained models are available at
http://sac-gmm.cs.uni-freiburg.de.

positive:
Learning predictive models from interaction with the world allows an agent,
such as a robot, to learn about how the world works, and then use this learned
model to plan coordinated sequences of actions to bring about desired outcomes.
However, learning a model that captures the dynamics of complex skills
represents a major challenge: if the agent needs a good model to perform these
skills, it might never be able to collect the experience on its own that is
required to learn these delicate and complex behaviors. Instead, we can imagine
augmenting the training set with observational data of other agents, such as
humans. Such data is likely more plentiful, but represents a different
embodiment. For example, videos of humans might show a robot how to use a tool,
but (i) are not annotated with suitable robot actions, and (ii) contain a
systematic distributional shift due to the embodiment differences between
humans and robots. We address the first challenge by formulating the
corresponding graphical model and treating the action as an observed variable
for the interaction data and an unobserved variable for the observation data,
and the second challenge by using a domain-dependent prior. In addition to
interaction data, our method is able to leverage videos of passive observations
in a driving dataset and a dataset of robotic manipulation videos. A robotic
planning agent equipped with our method can learn to use tools in a tabletop
robotic manipulation setting by observing humans without ever seeing a robotic
video of tool use.

positive:
For many real-world robotics applications, robots need to continually adapt
and learn new concepts. Further, robots need to learn through limited data
because of scarcity of labeled data in the real-world environments. To this
end, my research focuses on developing robots that continually learn in dynamic
unseen environments/scenarios, learn from limited human supervision, remember
previously learned knowledge and use that knowledge to learn new concepts. I
develop machine learning models that not only produce State-of-the-results on
benchmark datasets but also allow robots to learn new objects and scenes in
unconstrained environments which lead to a variety of novel robotics
applications.

positive:
We are motivated by the goal of generalist robots that can complete a wide
range of tasks across many environments. Critical to this is the robot's
ability to acquire some metric of task success or reward, which is necessary
for reinforcement learning, planning, or knowing when to ask for help. For a
general-purpose robot operating in the real world, this reward function must
also be able to generalize broadly across environments, tasks, and objects,
while depending only on on-board sensor observations (e.g. RGB images). While
deep learning on large and diverse datasets has shown promise as a path towards
such generalization in computer vision and natural language, collecting high
quality datasets of robotic interaction at scale remains an open challenge. In
contrast, "in-the-wild" videos of humans (e.g. YouTube) contain an extensive
collection of people doing interesting tasks across a diverse range of
settings. In this work, we propose a simple approach, Domain-agnostic Video
Discriminator (DVD), that learns multitask reward functions by training a
discriminator to classify whether two videos are performing the same task, and
can generalize by virtue of learning from a small amount of robot data with a
broad dataset of human videos. We find that by leveraging diverse human
datasets, this reward function (a) can generalize zero shot to unseen
environments, (b) generalize zero shot to unseen tasks, and (c) can be combined
with visual model predictive control to solve robotic manipulation tasks on a
real WidowX200 robot in an unseen environment from a single human demo.

positive:
Exploration of indoor environments has recently experienced a significant
interest, also thanks to the introduction of deep neural agents built in a
hierarchical fashion and trained with Deep Reinforcement Learning (DRL) on
simulated environments. Current state-of-the-art methods employ a dense
extrinsic reward that requires the complete a priori knowledge of the layout of
the training environment to learn an effective exploration policy. However,
such information is expensive to gather in terms of time and resources. In this
work, we propose to train the model with a purely intrinsic reward signal to
guide exploration, which is based on the impact of the robot's actions on its
internal representation of the environment. So far, impact-based rewards have
been employed for simple tasks and in procedurally generated synthetic
environments with countable states. Since the number of states observable by
the agent in realistic indoor environments is non-countable, we include a
neural-based density model and replace the traditional count-based
regularization with an estimated pseudo-count of previously visited states. The
proposed exploration approach outperforms DRL-based competitors relying on
intrinsic rewards and surpasses the agents trained with a dense extrinsic
reward computed with the environment layouts. We also show that a robot
equipped with the proposed approach seamlessly adapts to point-goal navigation
and real-world deployment.

positive:
While reinforcement learning algorithms can learn effective policies for
complex tasks, these policies are often brittle to even minor task variations,
especially when variations are not explicitly provided during training. One
natural approach to this problem is to train agents with manually specified
variation in the training task or environment. However, this may be infeasible
in practical situations, either because making perturbations is not possible,
or because it is unclear how to choose suitable perturbation strategies without
sacrificing performance. The key insight of this work is that learning diverse
behaviors for accomplishing a task can directly lead to behavior that
generalizes to varying environments, without needing to perform explicit
perturbations during training. By identifying multiple solutions for the task
in a single environment during training, our approach can generalize to new
situations by abandoning solutions that are no longer effective and adopting
those that are. We theoretically characterize a robustness set of environments
that arises from our algorithm and empirically find that our diversity-driven
approach can extrapolate to various changes in the environment and task.

positive:
Training robots for operation in the real world is a complex, time consuming
and potentially expensive task. Despite significant success of reinforcement
learning in games and simulations, research in real robot applications has not
been able to match similar progress. While sample complexity can be reduced by
training policies in simulation, such policies can perform sub-optimally on the
real platform given imperfect calibration of model dynamics. We present an
approach -- supplemental to fine tuning on the real robot -- to further benefit
from parallel access to a simulator during training and reduce sample
requirements on the real robot. The developed approach harnesses auxiliary
rewards to guide the exploration for the real world agent based on the
proficiency of the agent in simulation and vice versa. In this context, we
demonstrate empirically that the reciprocal alignment for both agents provides
further benefit as the agent in simulation can adjust to optimize its behaviour
for states commonly visited by the real-world agent.

positive:
Reinforcement learning has been applied to a wide variety of robotics
problems, but most of such applications involve collecting data from scratch
for each new task. Since the amount of robot data we can collect for any single
task is limited by time and cost considerations, the learned behavior is
typically narrow: the policy can only execute the task in a handful of
scenarios that it was trained on. What if there was a way to incorporate a
large amount of prior data, either from previously solved tasks or from
unsupervised or undirected environment interaction, to extend and generalize
learned behaviors? While most prior work on extending robotic skills using
pre-collected data focuses on building explicit hierarchies or skill
decompositions, we show in this paper that we can reuse prior data to extend
new skills simply through dynamic programming. We show that even when the prior
data does not actually succeed at solving the new task, it can still be
utilized for learning a better policy, by providing the agent with a broader
understanding of the mechanics of its environment. We demonstrate the
effectiveness of our approach by chaining together several behaviors seen in
prior datasets for solving a new task, with our hardest experimental setting
involving composing four robotic skills in a row: picking, placing, drawer
opening, and grasping, where a +1/0 sparse reward is provided only on task
completion. We train our policies in an end-to-end fashion, mapping
high-dimensional image observations to low-level robot control commands, and
present results in both simulated and real world domains. Additional materials
and source code can be found on our project website:
https://sites.google.com/view/cog-rl

positive:
This paper contributes a preliminary report on the advantages and
disadvantages of incorporating simultaneous human control and feedback signals
in the training of a reinforcement learning robotic agent. While robotic
human-machine interfaces have become increasingly complex in both form and
function, control remains challenging for users. This has resulted in an
increasing gap between user control approaches and the number of robotic motors
which can be controlled. One way to address this gap is to shift some autonomy
to the robot. Semi-autonomous actions of the robotic agent can then be shaped
by human feedback, simplifying user control. Most prior work on agent shaping
by humans has incorporated training with feedback, or has included indirect
control signals. By contrast, in this paper we explore how a human can provide
concurrent feedback signals and real-time myoelectric control signals to train
a robot's actor-critic reinforcement learning control system. Using both a
physical and a simulated robotic system, we compare training performance on a
simple movement task when reward is derived from the environment, when reward
is provided by the human, and combinations of these two approaches. Our results
indicate that some benefit can be gained with the inclusion of human generated
feedback.

positive:
In most real world scenarios, a policy trained by reinforcement learning in
one environment needs to be deployed in another, potentially quite different
environment. However, generalization across different environments is known to
be hard. A natural solution would be to keep training after deployment in the
new environment, but this cannot be done if the new environment offers no
reward signal. Our work explores the use of self-supervision to allow the
policy to continue training after deployment without using any rewards. While
previous methods explicitly anticipate changes in the new environment, we
assume no prior knowledge of those changes yet still obtain significant
improvements. Empirical evaluations are performed on diverse simulation
environments from DeepMind Control suite and ViZDoom, as well as real robotic
manipulation tasks in continuously changing environments, taking observations
from an uncalibrated camera. Our method improves generalization in 31 out of 36
environments across various tasks and outperforms domain randomization on a
majority of environments.

positive:
Humans learn all their life long. They accumulate knowledge from a sequence
of learning experiences and remember the essential concepts without forgetting
what they have learned previously. Artificial neural networks struggle to learn
similarly. They often rely on data rigorously preprocessed to learn solutions
to specific problems such as classification or regression. In particular, they
forget their past learning experiences if trained on new ones. Therefore,
artificial neural networks are often inept to deal with real-life settings such
as an autonomous-robot that has to learn on-line to adapt to new situations and
overcome new problems without forgetting its past learning-experiences.
Continual learning (CL) is a branch of machine learning addressing this type of
problem. Continual algorithms are designed to accumulate and improve knowledge
in a curriculum of learning-experiences without forgetting. In this thesis, we
propose to explore continual algorithms with replay processes. Replay processes
gather together rehearsal methods and generative replay methods. Generative
Replay consists of regenerating past learning experiences with a generative
model to remember them. Rehearsal consists of saving a core-set of samples from
past learning experiences to rehearse them later. The replay processes make
possible a compromise between optimizing the current learning objective and the
past ones enabling learning without forgetting in sequences of tasks settings.
We show that they are very promising methods for continual learning. Notably,
they enable the re-evaluation of past data with new knowledge and the
confrontation of data from different learning-experiences. We demonstrate their
ability to learn continually through unsupervised learning, supervised learning
and reinforcement learning tasks.

positive:
Reinforcement learning (RL) algorithms find applications in inventory
control, recommender systems, vehicular traffic management, cloud computing and
robotics. The real-world complications of many tasks arising in these domains
makes them difficult to solve with the basic assumptions underlying classical
RL algorithms. RL agents in these applications often need to react and adapt to
changing operating conditions. A significant part of research on single-agent
RL techniques focuses on developing algorithms when the underlying assumption
of stationary environment model is relaxed. This paper provides a survey of RL
methods developed for handling dynamically varying environment models. The goal
of methods not limited by the stationarity assumption is to help autonomous
agents adapt to varying operating conditions. This is possible either by
minimizing the rewards lost during learning by RL agent or by finding a
suitable policy for the RL agent which leads to efficient operation of the
underlying system. A representative collection of these algorithms is discussed
in detail in this work along with their categorization and their relative
merits and demerits. Additionally we also review works which are tailored to
application domains. Finally, we discuss future enhancements for this field.

positive:
Live fire creates a dynamic, rapidly changing environment that presents a
worthy challenge for deep learning and artificial intelligence methodologies to
assist firefighters with scene comprehension in maintaining their situational
awareness, tracking and relay of important features necessary for key decisions
as they tackle these catastrophic events. We propose a deep Q-learning based
agent who is immune to stress induced disorientation and anxiety and thus able
to make clear decisions for navigation based on the observed and stored facts
in live fire environments. As a proof of concept, we imitate structural fire in
a gaming engine called Unreal Engine which enables the interaction of the agent
with the environment. The agent is trained with a deep Q-learning algorithm
based on a set of rewards and penalties as per its actions on the environment.
We exploit experience replay to accelerate the learning process and augment the
learning of the agent with human-derived experiences. The agent trained under
this deep Q-learning approach outperforms agents trained through alternative
path planning systems and demonstrates this methodology as a promising
foundation on which to build a path planning navigation assistant capable of
safely guiding fire fighters through live fire environments.

positive:
In this paper we consider the problem of robot navigation in simple maze-like
environments where the robot has to rely on its onboard sensors to perform the
navigation task. In particular, we are interested in solutions to this problem
that do not require localization, mapping or planning. Additionally, we require
that our solution can quickly adapt to new situations (e.g., changing
navigation goals and environments). To meet these criteria we frame this
problem as a sequence of related reinforcement learning tasks. We propose a
successor feature based deep reinforcement learning algorithm that can learn to
transfer knowledge from previously mastered navigation tasks to new problem
instances. Our algorithm substantially decreases the required learning time
after the first task instance has been solved, which makes it easily adaptable
to changing environments. We validate our method in both simulated and real
robot experiments with a Robotino and compare it to a set of baseline methods
including classical planning-based navigation.

positive:
Deep reinforcement learning (RL) has emerged as a promising approach for
autonomously acquiring complex behaviors from low level sensor observations.
Although a large portion of deep RL research has focused on applications in
video games and simulated control, which does not connect with the constraints
of learning in real environments, deep RL has also demonstrated promise in
enabling physical robots to learn complex skills in the real world. At the same
time,real world robotics provides an appealing domain for evaluating such
algorithms, as it connects directly to how humans learn; as an embodied agent
in the real world. Learning to perceive and move in the real world presents
numerous challenges, some of which are easier to address than others, and some
of which are often not considered in RL research that focuses only on simulated
domains. In this review article, we present a number of case studies involving
robotic deep RL. Building off of these case studies, we discuss commonly
perceived challenges in deep RL and how they have been addressed in these
works. We also provide an overview of other outstanding challenges, many of
which are unique to the real-world robotics setting and are not often the focus
of mainstream RL research. Our goal is to provide a resource both for
roboticists and machine learning researchers who are interested in furthering
the progress of deep RL in the real world.

positive:
For a robot to learn a good policy, it often requires expensive equipment
(such as sophisticated sensors) and a prepared training environment conducive
to learning. However, it is seldom possible to perfectly equip robots for
economic reasons, nor to guarantee ideal learning conditions, when deployed in
real-life environments. A solution would be to prepare the robot in the lab
environment, when all necessary material is available to learn a good policy.
After training in the lab, the robot should be able to get by without the
expensive equipment that used to be available to it, and yet still be
guaranteed to perform well on the field. The transition between the lab
(source) and the real-world environment (target) is related to transfer
learning, where the state-space between the source and target tasks differ. We
tackle a simulated task with continuous states and discrete actions presenting
this challenge, using Bootstrapped Dual Policy Iteration, a model-free
actor-critic reinforcement learning algorithm, and Policy Shaping.
Specifically, we train a BDPI agent, embodied by a virtual robot performing a
task in the V-Rep simulator, sensing its environment through several proximity
sensors. The resulting policy is then used by a second agent learning the same
task in the same environment, but with camera images as input. The goal is to
obtain a policy able to perform the task relying on merely camera images.

negative:
If a robotic agent wants to exploit symbolic planning techniques to achieve
some goal, it must be able to properly ground an abstract planning domain in
the environment in which it operates. However, if the environment is initially
unknown by the agent, the agent needs to explore it and discover the salient
aspects of the environment needed to reach its goals. Namely, the agent has to
discover: (i) the objects present in the environment, (ii) the properties of
these objects and their relations, and finally (iii) how abstract actions can
be successfully executed. The paper proposes a framework that aims to
accomplish the aforementioned perspective for an agent that perceives the
environment partially and subjectively, through real value sensors (e.g., GPS,
and on-board camera) and can operate in the environment through low level
actuators (e.g., move forward of 20 cm). We evaluate the proposed architecture
in photo-realistic simulated environments, where the sensors are RGB-D on-board
camera, GPS and compass, and low level actions include movements,
grasping/releasing objects, and manipulating objects. The agent is placed in an
unknown environment and asked to find objects of a certain type, place an
object on top of another, close or open an object of a certain type. We compare
our approach with the state of the art methods on object goal navigation based
on reinforcement learning, showing better performances.

negative:
Robots that are trained to perform a task in a fixed environment often fail
when facing unexpected changes to the environment due to a lack of exploration.
We propose a principled way to adapt the policy for better exploration in
changing sparse-reward environments. Unlike previous works which explicitly
model environmental changes, we analyze the relationship between the value
function and the optimal exploration for a Gaussian-parameterized policy and
show that our theory leads to an effective strategy for adjusting the variance
of the policy, enabling fast adapt to changes in a variety of sparse-reward
environments.

negative:
Reinforcement Learning AI commonly uses reward/penalty signals that are
objective and explicit in an environment -- e.g. game score, completion time,
etc. -- in order to learn the optimal strategy for task performance. However,
Human-AI interaction for such AI agents should include additional reinforcement
that is implicit and subjective -- e.g. human preferences for certain AI
behavior -- in order to adapt the AI behavior to idiosyncratic human
preferences. Such adaptations would mirror naturally occurring processes that
increase trust and comfort during social interactions. Here, we show how a
hybrid brain-computer-interface (hBCI), which detects an individual's level of
interest in objects/events in a virtual environment, can be used to adapt the
behavior of a Deep Reinforcement Learning AI agent that is controlling a
virtual autonomous vehicle. Specifically, we show that the AI learns a driving
strategy that maintains a safe distance from a lead vehicle, and most novelly,
preferentially slows the vehicle when the human passengers of the vehicle
encounter objects of interest. This adaptation affords an additional 20\%
viewing time for subjectively interesting objects. This is the first
demonstration of how an hBCI can be used to provide implicit reinforcement to
an AI agent in a way that incorporates user preferences into the control
system.

negative:
Learning a policy capable of moving an agent between any two states in the
environment is important for many robotics problems involving navigation and
manipulation. Due to the sparsity of rewards in such tasks, applying
reinforcement learning in these scenarios can be challenging. Common approaches
for tackling this problem include reward engineering with auxiliary rewards,
requiring domain-specific knowledge or changing the objective.
  In this work, we introduce a method based on region-growing that allows
learning in an environment with any pair of initial and goal states. Our
algorithm first learns how to move between nearby states and then increases the
difficulty of the start-goal transitions as the agent's performance improves.
This approach creates an efficient curriculum for learning the objective
behavior of reaching any goal from any initial state. In addition, we describe
a method to adaptively adjust expansion of the growing region that allows
automatic adjustment of the key exploration hyperparameter to environments with
different requirements. We evaluate our approach on a set of simulated
navigation and manipulation tasks, where we demonstrate that our algorithm can
efficiently learn a policy in the presence of sparse rewards.

negative:
Autonomous robots may be able to adapt their behavior in response to changes
in the environment. This is useful, for example, to efficiently handle limited
resources or to respond appropriately to unexpected events such as faults. The
architecture of a self-adaptive robot is complex because it should include
automatic mechanisms to dynamically configure the elements that control robot
behaviors. To facilitate the construction of this type of architectures, it is
useful to have general solutions in the form of software tools that may be
applicable to different robotic systems. This paper presents an original
algorithm to dynamically configure the control architecture, which is
applicable to the development of self-adaptive autonomous robots. This
algorithm uses a constraint-based configuration approach to decide which basic
robot behaviors should be activated in response to both reactive and
deliberative events. The algorithm uses specific search heuristics and
initialization procedures to achieve the performance required by robotic
systems. The solution has been implemented as a software development tool
called Behavior Coordinator CBC (Constraint-Based Configuration), which is
based on ROS and open source, available to the general public. This tool has
been successfully used for building multiple applications of autonomous aerial
robots.

negative:
One of the central challenges faced by a reinforcement learning (RL) agent is
to effectively learn a (near-)optimal policy in environments with large state
spaces having sparse and noisy feedback signals. In real-world applications, an
expert with additional domain knowledge can help in speeding up the learning
process via \emph{shaping the environment}, i.e., making the environment more
learner-friendly. A popular paradigm in literature is \emph{potential-based
reward shaping}, where the environment's reward function is augmented with
additional local rewards using a potential function. However, the applicability
of potential-based reward shaping is limited in settings where (i) the state
space is very large, and it is challenging to compute an appropriate potential
function, (ii) the feedback signals are noisy, and even with shaped rewards the
agent could be trapped in local optima, and (iii) changing the rewards alone is
not sufficient, and effective shaping requires changing the dynamics. We
address these limitations of potential-based shaping methods and propose a
novel framework of \emph{environment shaping using state abstraction}. Our key
idea is to compress the environment's large state space with noisy signals to
an abstracted space, and to use this abstraction in creating smoother and more
effective feedback signals for the agent. We study the theoretical
underpinnings of our abstraction-based environment shaping, and show that the
agent's policy learnt in the shaped environment preserves near-optimal behavior
in the original environment.

negative:
It is a long-standing challenge to enable an intelligent agent to learn in
one environment and generalize to an unseen environment without further data
collection and finetuning. In this paper, we consider a zero shot
generalization problem setup that complies with biological intelligent agents'
learning and generalization processes. The agent is first presented with
previous experiences in the training environment, along with task description
in the form of trajectory-level sparse rewards. Later when it is placed in the
new testing environment, it is asked to perform the task without any
interaction with the testing environment. We find this setting natural for
biological creatures and at the same time, challenging for previous methods.
Behavior cloning, state-of-art RL along with other zero-shot learning methods
perform poorly on this benchmark. Given a set of experiences in the training
environment, our method learns a neural function that decomposes the sparse
reward into particular regions in a contingency-aware observation as a per step
reward. Based on such decomposed rewards, we further learn a dynamics model and
use Model Predictive Control (MPC) to obtain a policy. Since the rewards are
decomposed to finer-granularity observations, they are naturally generalizable
to new environments that are composed of similar basic elements. We demonstrate
our method on a wide range of environments, including a classic video game --
Super Mario Bros, as well as a robotic continuous control task. Please refer to
the project page for more visualized results.

negative:
Deep reinforcement learning agents have achieved state-of-the-art results by
directly maximising cumulative reward. However, environments contain a much
wider variety of possible training signals. In this paper, we introduce an
agent that also maximises many other pseudo-reward functions simultaneously by
reinforcement learning. All of these tasks share a common representation that,
like unsupervised learning, continues to develop in the absence of extrinsic
rewards. We also introduce a novel mechanism for focusing this representation
upon extrinsic rewards, so that learning can rapidly adapt to the most relevant
aspects of the actual task. Our agent significantly outperforms the previous
state-of-the-art on Atari, averaging 880\% expert human performance, and a
challenging suite of first-person, three-dimensional \emph{Labyrinth} tasks
leading to a mean speedup in learning of 10$\times$ and averaging 87\% expert
human performance on Labyrinth.

negative:
The main novelty of the proposed approach is that it allows a robot to learn
an end-to-end policy which can adapt to changes in the environment during
execution. While goal conditioning of policies has been studied in the RL
literature, such approaches are not easily extended to cases where the robot's
goal can change during execution. This is something that humans are naturally
able to do. However, it is difficult for robots to learn such reflexes (i.e.,
to naturally respond to dynamic environments), especially when the goal
location is not explicitly provided to the robot, and instead needs to be
perceived through a vision sensor. In the current work, we present a method
that can achieve such behavior by combining traditional kinematic planning,
deep learning, and deep reinforcement learning in a synergistic fashion to
generalize to arbitrary environments. We demonstrate the proposed approach for
several reaching and pick-and-place tasks in simulation, as well as on a real
system of a 6-DoF industrial manipulator. A video describing our work could be
found \url{https://youtu.be/hE-Ew59GRPQ}.

negative:
Successfully navigating a complex environment to obtain a desired outcome is
a difficult task, that up to recently was believed to be capable only by
humans. This perception has been broken down over time, especially with the
introduction of deep reinforcement learning, which has greatly increased the
difficulty of tasks that can be automated. However, for traditional
reinforcement learning agents this requires an environment to be able to
provide frequent extrinsic rewards, which are not known or accessible for many
real-world environments. This project aims to explore and contrast existing
reinforcement learning solutions that circumnavigate the difficulties of an
environment that provide sparse rewards. Different reinforcement solutions will
be implemented over a several video game environments with varying difficulty
and varying frequency of rewards, as to properly investigate the applicability
of these solutions. This project introduces a novel reinforcement learning
solution by combining aspects of two existing state of the art sparse reward
solutions, curiosity driven exploration and unsupervised auxiliary tasks.

negative:
A grand goal in AI is to build a robot that can accurately navigate based on
natural language instructions, which requires the agent to perceive the scene,
understand and ground language, and act in the real-world environment. One key
challenge here is to learn to navigate in new environments that are unseen
during training. Most of the existing approaches perform dramatically worse in
unseen environments as compared to seen ones. In this paper, we present a
generalizable navigational agent. Our agent is trained in two stages. The first
stage is training via mixed imitation and reinforcement learning, combining the
benefits from both off-policy and on-policy optimization. The second stage is
fine-tuning via newly-introduced 'unseen' triplets (environment, path,
instruction). To generate these unseen triplets, we propose a simple but
effective 'environmental dropout' method to mimic unseen environments, which
overcomes the problem of limited seen environment variability. Next, we apply
semi-supervised learning (via back-translation) on these dropped-out
environments to generate new paths and instructions. Empirically, we show that
our agent is substantially better at generalizability when fine-tuned with
these triplets, outperforming the state-of-art approaches by a large margin on
the private unseen test set of the Room-to-Room task, and achieving the top
rank on the leaderboard.

negative:
Developing agents that can quickly adapt their behavior to new tasks remains
a challenge. Meta-learning has been applied to this problem, but previous
methods require either specifying a reward function which can be tedious or
providing demonstrations which can be inefficient. In this paper, we
investigate if, and how, a "helper" agent can be trained to interactively adapt
their behavior to maximize the reward of another agent, whom we call the
"prime" agent, without observing their reward or receiving explicit
demonstrations. To this end, we propose to meta-learn a helper agent along with
a prime agent, who, during training, observes the reward function and serves as
a surrogate for a human prime. We introduce a distribution of multi-agent
cooperative foraging tasks, in which only the prime agent knows the objects
that should be collected. We demonstrate that, from the emerged physical
communication, the trained helper rapidly infers and collects the correct
objects.

negative:
The objective of a reinforcement learning agent is to behave so as to
maximise the sum of a suitable scalar function of state: the reward. These
rewards are typically given and immutable. In this paper, we instead consider
the proposition that the reward function itself can be a good locus of learned
knowledge. To investigate this, we propose a scalable meta-gradient framework
for learning useful intrinsic reward functions across multiple lifetimes of
experience. Through several proof-of-concept experiments, we show that it is
feasible to learn and capture knowledge about long-term exploration and
exploitation into a reward function. Furthermore, we show that unlike policy
transfer methods that capture "how" the agent should behave, the learned reward
functions can generalise to other kinds of agents and to changes in the
dynamics of the environment by capturing "what" the agent should strive to do.

negative:
Designing a good reward function is essential to robot planning and
reinforcement learning, but it can also be challenging and frustrating. The
reward needs to work across multiple different environments, and that often
requires many iterations of tuning. We introduce a novel divide-and-conquer
approach that enables the designer to specify a reward separately for each
environment. By treating these separate reward functions as observations about
the underlying true reward, we derive an approach to infer a common reward
across all environments. We conduct user studies in an abstract grid world
domain and in a motion planning domain for a 7-DOF manipulator that measure
user effort and solution quality. We show that our method is faster, easier to
use, and produces a higher quality solution than the typical method of
designing a reward jointly across all environments. We additionally conduct a
series of experiments that measure the sensitivity of these results to
different properties of the reward design task, such as the number of
environments, the number of feasible solutions per environment, and the
fraction of the total features that vary within each environment. We find that
independent reward design outperforms the standard, joint, reward design
process but works best when the design problem can be divided into simpler
subproblems.

negative:
Deep Reinforcement Learning (DRL) has been successfully used to solve
different challenges, e.g. complex board and computer games, recently. However,
solving real-world robotics tasks with DRL seems to be a more difficult
challenge. The desired approach would be to train the agent in a simulator and
transfer it to the real world. Still, models trained in a simulator tend to
perform poorly in real-world environments due to the differences. In this
paper, we present a DRL-based algorithm that is capable of performing
autonomous robot control using Deep Q-Networks (DQN). In our approach, the
agent is trained in a simulated environment and it is able to navigate both in
a simulated and real-world environment. The method is evaluated in the
Duckietown environment, where the agent has to follow the lane based on a
monocular camera input. The trained agent is able to run on limited hardware
resources and its performance is comparable to state-of-the-art approaches.

negative:
To acquire a new skill, humans learn better and faster if a tutor, based on
their current knowledge level, informs them of how much attention they should
pay to particular content or practice problems. Similarly, a machine learning
model could potentially be trained better with a scorer that "adapts" to its
current learning state and estimates the importance of each training data
instance. Training such an adaptive scorer efficiently is a challenging
problem; in order to precisely quantify the effect of a data instance at a
given time during the training, it is typically necessary to first complete the
entire training process. To efficiently optimize data usage, we propose a
reinforcement learning approach called Differentiable Data Selection (DDS). In
DDS, we formulate a scorer network as a learnable function of the training
data, which can be efficiently updated along with the main model being trained.
Specifically, DDS updates the scorer with an intuitive reward signal: it should
up-weigh the data that has a similar gradient with a dev set upon which we
would finally like to perform well. Without significant computing overhead, DDS
delivers strong and consistent improvements over several strong baselines on
two very different tasks of machine translation and image classification.

negative:
To achieve general artificial intelligence, reinforcement learning (RL)
agents should learn not only to optimize returns for one specific task but also
to constantly build more complex skills and scaffold their knowledge about the
world, without forgetting what has already been learned. In this paper, we
discuss the desired characteristics of environments that can support the
training and evaluation of lifelong reinforcement learning agents, review
existing environments from this perspective, and propose recommendations for
devising suitable environments in the future.

negative:
We propose a novel approach to address one aspect of the non-stationarity
problem in multi-agent reinforcement learning (RL), where the other agents may
alter their policies due to environment changes during execution. This violates
the Markov assumption that governs most single-agent RL methods and is one of
the key challenges in multi-agent RL. To tackle this, we propose to train
multiple policies for each agent and postpone the selection of the best policy
at execution time. Specifically, we model the environment non-stationarity with
a finite set of scenarios and train policies fitting each scenario. In addition
to multiple policies, each agent also learns a policy predictor to determine
which policy is the best with its local information. By doing so, each agent is
able to adapt its policy when the environment changes and consequentially the
other agents alter their policies during execution. We empirically evaluated
our method on a variety of common benchmark problems proposed for multi-agent
deep RL in the literature. Our experimental results show that the agents
trained by our algorithm have better adaptiveness in changing environments and
outperform the state-of-the-art methods in all the tested environments.

negative:
In many sequential decision making tasks, it is challenging to design reward
functions that help an RL agent efficiently learn behavior that is considered
good by the agent designer. A number of different formulations of the
reward-design problem, or close variants thereof, have been proposed in the
literature. In this paper we build on the Optimal Rewards Framework of Singh
et.al. that defines the optimal intrinsic reward function as one that when used
by an RL agent achieves behavior that optimizes the task-specifying or
extrinsic reward function. Previous work in this framework has shown how good
intrinsic reward functions can be learned for lookahead search based planning
agents. Whether it is possible to learn intrinsic reward functions for learning
agents remains an open problem. In this paper we derive a novel algorithm for
learning intrinsic rewards for policy-gradient based learning agents. We
compare the performance of an augmented agent that uses our algorithm to
provide additive intrinsic rewards to an A2C-based policy learner (for Atari
games) and a PPO-based policy learner (for Mujoco domains) with a baseline
agent that uses the same policy learners but with only extrinsic rewards. Our
results show improved performance on most but not all of the domains.

negative:
Recent work has shown that deep reinforcement-learning agents can learn to
follow language-like instructions from infrequent environment rewards. However,
this places on environment designers the onus of designing language-conditional
reward functions which may not be easily or tractably implemented as the
complexity of the environment and the language scales. To overcome this
limitation, we present a framework within which instruction-conditional RL
agents are trained using rewards obtained not from the environment, but from
reward models which are jointly trained from expert examples. As reward models
improve, they learn to accurately reward agents for completing tasks for
environment configurations---and for instructions---not present amongst the
expert data. This framework effectively separates the representation of what
instructions require from how they can be executed. In a simple grid world, it
enables an agent to learn a range of commands requiring interaction with blocks
and understanding of spatial relations and underspecified abstract
arrangements. We further show the method allows our agent to adapt to changes
in the environment without requiring new expert examples.

negative:
Learning reward functions from data is a promising path towards achieving
scalable Reinforcement Learning (RL) for robotics. However, a major challenge
in training agents from learned reward models is that the agent can learn to
exploit errors in the reward model to achieve high reward behaviors that do not
correspond to the intended task. These reward delusions can lead to unintended
and even dangerous behaviors. On the other hand, adversarial imitation learning
frameworks tend to suffer the opposite problem, where the discriminator learns
to trivially distinguish agent and expert behavior, resulting in reward models
that produce low reward signal regardless of the input state. In this paper, we
connect these two classes of reward learning methods to positive-unlabeled (PU)
learning, and we show that by applying a large-scale PU learning algorithm to
the reward learning problem, we can address both the reward under- and
over-estimation problems simultaneously. Our approach drastically improves both
GAIL and supervised reward learning, without any additional assumptions.

negative:
Reinforcement learning is concerned with identifying reward-maximizing
behaviour policies in environments that are initially unknown. State-of-the-art
reinforcement learning approaches, such as deep Q-networks, are model-free and
learn to act effectively across a wide range of environments such as Atari
games, but require huge amounts of data. Model-based techniques are more
data-efficient, but need to acquire explicit knowledge about the environment.
  In this paper, we take a step towards using model-based techniques in
environments with a high-dimensional visual state space by demonstrating that
it is possible to learn system dynamics and the reward structure jointly. Our
contribution is to extend a recently developed deep neural network for video
frame prediction in Atari games to enable reward prediction as well. To this
end, we phrase a joint optimization problem for minimizing both video frame and
reward reconstruction loss, and adapt network parameters accordingly. Empirical
evaluations on five Atari games demonstrate accurate cumulative reward
prediction of up to 200 frames. We consider these results as opening up
important directions for model-based reinforcement learning in complex,
initially unknown environments.

negative:
Agents of general intelligence deployed in real-world scenarios must adapt to
ever-changing environmental conditions. While such adaptive agents may leverage
engineered knowledge, they will require the capacity to construct and evaluate
knowledge themselves from their own experience in a bottom-up, constructivist
fashion. This position paper builds on the idea of encoding knowledge as
temporally extended predictions through the use of general value functions.
Prior work has focused on learning predictions about externally derived signals
about a task or environment (e.g. battery level, joint position). Here we
advocate that the agent should also predict internally generated signals
regarding its own learning process - for example, an agent's confidence in its
learned predictions. Finally, we suggest how such information would be
beneficial in creating an introspective agent that is able to learn to make
good decisions in a complex, changing world.

negative:
Many reinforcement-learning researchers treat the reward function as a part
of the environment, meaning that the agent can only know the reward of a state
if it encounters that state in a trial run. However, we argue that this is an
unnecessary limitation and instead, the reward function should be provided to
the learning algorithm. The advantage is that the algorithm can then use the
reward function to check the reward for states that the agent hasn't even
encountered yet. In addition, the algorithm can simultaneously learn policies
for multiple reward functions. For each state, the algorithm would calculate
the reward using each of the reward functions and add the rewards to its
experience replay dataset. The Hindsight Experience Replay algorithm developed
by Andrychowicz et al. (2017) does just this, and learns to generalize across a
distribution of sparse, goal-based rewards. We extend this algorithm to
linearly-weighted, multi-objective rewards and learn a single policy that can
generalize across all linear combinations of the multi-objective reward.
Whereas other multi-objective algorithms teach the Q-function to generalize
across the reward weights, our algorithm enables the policy to generalize, and
can thus be used with continuous actions.

negative:
Reinforcement learning usually uses the feedback rewards of environmental to
train agents. But the rewards in the actual environment are sparse, and even
some environments will not rewards. Most of the current methods are difficult
to get good performance in sparse reward or non-reward environments. Although
using shaped rewards is effective when solving sparse reward tasks, it is
limited to specific problems and learning is also susceptible to local optima.
We propose a model-free method that does not rely on environmental rewards to
solve the problem of sparse rewards in the general environment. Our method use
the minimum number of transitions between states as the distance to replace the
rewards of environmental, and proposes a goal-distance gradient to achieve
policy improvement. We also introduce a bridge point planning method based on
the characteristics of our method to improve exploration efficiency, thereby
solving more complex tasks. Experiments show that our method performs better on
sparse reward and local optimal problems in complex environments than previous
work.

negative:
Reinforcement learning with sparse rewards is still an open challenge.
Classic methods rely on getting feedback via extrinsic rewards to train the
agent, and in situations where this occurs very rarely the agent learns slowly
or cannot learn at all. Similarly, if the agent receives also rewards that
create suboptimal modes of the objective function, it will likely prematurely
stop exploring. More recent methods add auxiliary intrinsic rewards to
encourage exploration. However, auxiliary rewards lead to a non-stationary
target for the Q-function. In this paper, we present a novel approach that (1)
plans exploration actions far into the future by using a long-term visitation
count, and (2) decouples exploration and exploitation by learning a separate
function assessing the exploration value of the actions. Contrary to existing
methods which use models of reward and dynamics, our approach is off-policy and
model-free. We further propose new tabular environments for benchmarking
exploration in reinforcement learning. Empirical results on classic and novel
benchmarks show that the proposed approach outperforms existing methods in
environments with sparse rewards, especially in the presence of rewards that
create suboptimal modes of the objective function. Results also suggest that
our approach scales gracefully with the size of the environment. Source code is
available at https://github.com/sparisi/visit-value-explore

negative:
Humans and animals show remarkable flexibility in adjusting their behaviour
when their goals, or rewards in the environment change. While such flexibility
is a hallmark of intelligent behaviour, these multi-task scenarios remain an
important challenge for machine learning algorithms and neurobiological models
alike. We investigated two approaches that could enable this flexibility:
factorized representations, which abstract away general aspects of a task from
those prone to change, and nonparametric, memory-based approaches, which can
provide a principled way of using similarity to past experiences to guide
current behaviour. In particular, we combine the successor representation (SR)
that factors the value of actions into expected outcomes and corresponding
rewards with evaluating task similarity through clustering the space of reward
functions. The proposed algorithm inverts a generative model over tasks, and
dynamically samples from a flexible number of distinct SR maps while
accumulating evidence about the current task context through amortized
inference. It improves SR's transfer capabilities and outperforms competing
algorithms and baselines in settings with both known and unsignalled rewards
changes. Further, as a neurobiological model of spatial coding in the
hippocampus, it explains important signatures of this representation, such as
the "flickering" behaviour of hippocampal maps, and trajectory-dependent place
cells (so-called splitter cells) and their dynamics. We thus provide a novel
algorithmic approach for multi-task learning, as well as a common normative
framework that links together these different characteristics of the brain's
spatial representation.

negative:
An agent learning through interactions should balance its action selection
process between probing the environment to discover new rewards and using the
information acquired in the past to adopt useful behaviour. This trade-off is
usually obtained by perturbing either the agent's actions (e.g., e-greedy or
Gibbs sampling) or the agent's parameters (e.g., NoisyNet), or by modifying the
reward it receives (e.g., exploration bonus, intrinsic motivation, or
hand-shaped rewards). Here, we adopt a disruptive but simple and generic
perspective, where we explicitly disentangle exploration and exploitation.
Different losses are optimized in parallel, one of them coming from the true
objective (maximizing cumulative rewards from the environment) and others being
related to exploration. Every loss is used in turn to learn a policy that
generates transitions, all shared in a single replay buffer. Off-policy methods
are then applied to these transitions to optimize each loss. We showcase our
approach on a hard-exploration environment, show its sample-efficiency and
robustness, and discuss further implications.

negative:
Transfer in reinforcement learning refers to the notion that generalization
should occur not only within a task but also across tasks. We propose a
transfer framework for the scenario where the reward function changes between
tasks but the environment's dynamics remain the same. Our approach rests on two
key ideas: "successor features", a value function representation that decouples
the dynamics of the environment from the rewards, and "generalized policy
improvement", a generalization of dynamic programming's policy improvement
operation that considers a set of policies rather than a single one. Put
together, the two ideas lead to an approach that integrates seamlessly within
the reinforcement learning framework and allows the free exchange of
information across tasks. The proposed method also provides performance
guarantees for the transferred policy even before any learning has taken place.
We derive two theorems that set our approach in firm theoretical ground and
present experiments that show that it successfully promotes transfer in
practice, significantly outperforming alternative methods in a sequence of
navigation tasks and in the control of a simulated robotic arm.

negative:
Learning robust value functions given raw observations and rewards is now
possible with model-free and model-based deep reinforcement learning
algorithms. There is a third alternative, called Successor Representations
(SR), which decomposes the value function into two components -- a reward
predictor and a successor map. The successor map represents the expected future
state occupancy from any given state and the reward predictor maps states to
scalar rewards. The value function of a state can be computed as the inner
product between the successor map and the reward weights. In this paper, we
present DSR, which generalizes SR within an end-to-end deep reinforcement
learning framework. DSR has several appealing properties including: increased
sensitivity to distal reward changes due to factorization of reward and world
dynamics, and the ability to extract bottleneck states (subgoals) given
successor maps trained under a random policy. We show the efficacy of our
approach on two diverse environments given raw pixel observations -- simple
grid-world domains (MazeBase) and the Doom game engine.

negative:
Predictions and predictive knowledge have seen recent success in improving
not only robot control but also other applications ranging from industrial
process control to rehabilitation. A property that makes these predictive
approaches well suited for robotics is that they can be learned online and
incrementally through interaction with the environment. However, a remaining
challenge for many prediction-learning approaches is an appropriate choice of
prediction-learning parameters, especially parameters that control the
magnitude of a learning machine's updates to its predictions (the learning rate
or step size). To begin to address this challenge, we examine the use of online
step-size adaptation using a sensor-rich robotic arm. Our method of choice,
Temporal-Difference Incremental Delta-Bar-Delta (TIDBD), learns and adapts step
sizes on a feature level; importantly, TIDBD allows step-size tuning and
representation learning to occur at the same time. We show that TIDBD is a
practical alternative for classic Temporal-Difference (TD) learning via an
extensive parameter search. Both approaches perform comparably in terms of
predicting future aspects of a robotic data stream. Furthermore, the use of a
step-size adaptation method like TIDBD appears to allow a system to
automatically detect and characterize common sensor failures in a robotic
application. Together, these results promise to improve the ability of robotic
devices to learn from interactions with their environments in a robust way,
providing key capabilities for autonomous agents and robots.

negative:
The ability to transfer skills across tasks has the potential to scale up
reinforcement learning (RL) agents to environments currently out of reach.
Recently, a framework based on two ideas, successor features (SFs) and
generalised policy improvement (GPI), has been introduced as a principled way
of transferring skills. In this paper we extend the SFs & GPI framework in two
ways. One of the basic assumptions underlying the original formulation of SFs &
GPI is that rewards for all tasks of interest can be computed as linear
combinations of a fixed set of features. We relax this constraint and show that
the theoretical guarantees supporting the framework can be extended to any set
of tasks that only differ in the reward function. Our second contribution is to
show that one can use the reward functions themselves as features for future
tasks, without any loss of expressiveness, thus removing the need to specify a
set of features beforehand. This makes it possible to combine SFs & GPI with
deep learning in a more stable way. We empirically verify this claim on a
complex 3D environment where observations are images from a first-person
perspective. We show that the transfer promoted by SFs & GPI leads to very good
policies on unseen tasks almost instantaneously. We also describe how to learn
policies specialised to the new tasks in a way that allows them to be added to
the agent's set of skills, and thus be reused in the future.

negative:
Robots are extending their presence in domestic environments every day, being
more common to see them carrying out tasks in home scenarios. In the future,
robots are expected to increasingly perform more complex tasks and, therefore,
be able to acquire experience from different sources as quickly as possible. A
plausible approach to address this issue is interactive feedback, where a
trainer advises a learner on which actions should be taken from specific states
to speed up the learning process. Moreover, deep reinforcement learning has
been recently widely utilized in robotics to learn the environment and acquire
new skills autonomously. However, an open issue when using deep reinforcement
learning is the excessive time needed to learn a task from raw input images. In
this work, we propose a deep reinforcement learning approach with interactive
feedback to learn a domestic task in a human-robot scenario. We compare three
different learning methods using a simulated robotic arm for the task of
organizing different objects; the proposed methods are (i) deep reinforcement
learning (DeepRL); (ii) interactive deep reinforcement learning using a
previously trained artificial agent as an advisor (agent-IDeepRL); and (iii)
interactive deep reinforcement learning using a human advisor (human-IDeepRL).
We demonstrate that interactive approaches provide advantages for the learning
process. The obtained results show that a learner agent, using either
agent-IDeepRL or human-IDeepRL, completes the given task earlier and has fewer
mistakes compared to the autonomous DeepRL approach.

negative:
In this paper, we propose an AI-FML robotic agent for student learning
behavior ontology construction which can be applied in English speaking and
listening domain. The AI-FML robotic agent with the ontology contains the
perception intelligence, computational intelligence, and cognition intelligence
for analyzing student learning behavior. In addition, there are three
intelligent agents, including a perception agent, a computational agent, and a
cognition agent in the AI-FML robotic agent. We deploy the perception agent and
the cognition agent on the robot Kebbi Air. Moreover, the computational agent
with the Deep Neural Network (DNN) model is performed in the cloud and can
communicate with the perception agent and cognition agent via the Internet. The
proposed AI-FML robotic agent is applied in Taiwan and tested in Japan. The
experimental results show that the agents can be utilized in the human and
machine co-learning model for the future education.

negative:
In many real-world scenarios where extrinsic rewards to the agent are
extremely sparse, curiosity has emerged as a useful concept providing intrinsic
rewards that enable the agent to explore its environment and acquire
information to achieve its goals. Despite their strong performance on many
sparse-reward tasks, existing curiosity approaches rely on an overly holistic
view of state transitions, and do not allow for a structured understanding of
specific aspects of the environment. In this paper, we formulate curiosity
based on grounded question answering by encouraging the agent to ask questions
about the environment and be curious when the answers to these questions
change. We show that natural language questions encourage the agent to uncover
specific knowledge about their environment such as the physical properties of
objects as well as their spatial relationships with other objects, which serve
as valuable curiosity rewards to solve sparse-reward tasks more efficiently.

negative:
Long-term temporal credit assignment is an important challenge in deep
reinforcement learning (RL). It refers to the ability of the agent to attribute
actions to consequences that may occur after a long time interval. Existing
policy-gradient and Q-learning algorithms typically rely on dense environmental
rewards that provide rich short-term supervision and help with credit
assignment. However, they struggle to solve tasks with delays between an action
and the corresponding rewarding feedback. To make credit assignment easier,
recent works have proposed algorithms to learn dense "guidance" rewards that
could be used in place of the sparse or delayed environmental rewards. This
paper is in the same vein -- starting with a surrogate RL objective that
involves smoothing in the trajectory-space, we arrive at a new algorithm for
learning guidance rewards. We show that the guidance rewards have an intuitive
interpretation, and can be obtained without training any additional neural
networks. Due to the ease of integration, we use the guidance rewards in a few
popular algorithms (Q-learning, Actor-Critic, Distributional-RL) and present
results in single-agent and multi-agent tasks that elucidate the benefit of our
approach when the environmental rewards are sparse or delayed.

negative:
In the real world, robots with embodiment face various issues such as dynamic
continuous changes of the environment and input/output disturbances. The key to
solving these issues can be found in daily life; people `do actions associated
with sensing' and `dynamically change their plans when necessary'. We propose
the use of a new concept, enabling robots to do these two things, for
autonomously controlling mobile robots. We implemented our concept to make two
experiments under static/dynamic environments. The results of these experiments
show that our idea provides a way to adapt to dynamic changes of the
environment in the real world.

negative:
Much of the current work on reinforcement learning studies episodic settings,
where the agent is reset between trials to an initial state distribution, often
with well-shaped reward functions. Non-episodic settings, where the agent must
learn through continuous interaction with the world without resets, and where
the agent receives only delayed and sparse reward signals, is substantially
more difficult, but arguably more realistic considering real-world environments
do not present the learner with a convenient "reset mechanism" and easy reward
shaping. In this paper, instead of studying algorithmic improvements that can
address such non-episodic and sparse reward settings, we instead study the
kinds of environment properties that can make learning under such conditions
easier. Understanding how properties of the environment impact the performance
of reinforcement learning agents can help us to structure our tasks in ways
that make learning tractable. We first discuss what we term "environment
shaping" -- modifications to the environment that provide an alternative to
reward shaping, and may be easier to implement. We then discuss an even simpler
property that we refer to as "dynamism," which describes the degree to which
the environment changes independent of the agent's actions and can be measured
by environment transition entropy. Surprisingly, we find that even this
property can substantially alleviate the challenges associated with
non-episodic RL in sparse reward settings. We provide an empirical evaluation
on a set of new tasks focused on non-episodic learning with sparse rewards.
Through this study, we hope to shift the focus of the community towards
analyzing how properties of the environment can affect learning and the
ultimate type of behavior that is learned via RL.

negative:
Robot learning is a very promising topic for the future of automation and
machine intelligence. Future robots should be able to autonomously acquire
skills, learn to represent their environment, and interact with it. While these
topics have been explored in simulation, real-world robot learning research
seems to be still limited. This is due to the additional challenges encountered
in the real-world, such as noisy sensors and actuators, safe exploration,
non-stationary dynamics, autonomous environment resetting as well as the cost
of running experiments for long periods of time. Unless we develop scalable
solutions to these problems, learning complex tasks involving hand-eye
coordination and rich contacts will remain an untouched vision that is only
feasible in controlled lab environments. We propose dual-arm settings as
platforms for robot learning. Such settings enable safe data collection for
acquiring manipulation skills as well as training perception modules in a
robot-supervised manner. They also ease the processes of resetting the
environment. Furthermore, adversarial learning could potentially boost the
generalization capability of robot learning methods by maximizing the
exploration based on game-theoretic objectives while ensuring safety based on
collaborative task spaces. In this paper, we will discuss the potential
benefits of this setup as well as the challenges and research directions that
can be pursued.

negative:
Reinforcement learning (RL), particularly in sparse reward settings, often
requires prohibitively large numbers of interactions with the environment,
thereby limiting its applicability to complex problems. To address this,
several prior approaches have used natural language to guide the agent's
exploration. However, these approaches typically operate on structured
representations of the environment, and/or assume some structure in the natural
language commands. In this work, we propose a model that directly maps pixels
to rewards, given a free-form natural language description of the task, which
can then be used for policy learning. Our experiments on the Meta-World robot
manipulation domain show that language-based rewards significantly improves the
sample efficiency of policy learning, both in sparse and dense reward settings.

negative:
Recent reinforcement learning (RL) approaches have shown strong performance
in complex domains such as Atari games, but are often highly sample
inefficient. A common approach to reduce interaction time with the environment
is to use reward shaping, which involves carefully designing reward functions
that provide the agent intermediate rewards for progress towards the goal.
However, designing appropriate shaping rewards is known to be difficult as well
as time-consuming. In this work, we address this problem by using natural
language instructions to perform reward shaping. We propose the LanguagE-Action
Reward Network (LEARN), a framework that maps free-form natural language
instructions to intermediate rewards based on actions taken by the agent. These
intermediate language-based rewards can seamlessly be integrated into any
standard reinforcement learning algorithm. We experiment with Montezuma's
Revenge from the Atari Learning Environment, a popular benchmark in RL. Our
experiments on a diverse set of 15 tasks demonstrate that, for the same number
of interactions with the environment, language-based rewards lead to successful
completion of the task 60% more often on average, compared to learning without
language.

negative:
The aim of my Ph.D. thesis concerns Reasoning in Highly Reactive
Environments. As reasoning in highly reactive environments, we identify the
setting in which a knowledge-based agent, with given goals, is deployed in an
environment subject to repeated, sudden and possibly unknown changes. This is
for instance the typical setting in which, e.g., artificial agents for
video-games (the so called "bots"), cleaning robots, bomb clearing robots, and
so on are deployed. In all these settings one can follow the classical approach
in which the operations of the agent are distinguished in "sensing" the
environment with proper interface devices, "thinking", and then behaving
accordingly using proper actuators. In order to operate in an highly reactive
environment, an artificial agent needs to be: 1. Responsive -> The agent must
be able to react repeatedly and in a reasonable amount of time; 2. Elastic ->
The agent must stay reactive also under varying workload; 3. Resilient -> The
agent must stay responsive also in case of internal failure or failure of one
of the programmed actions in the environment. Nowadays, thanks to new
technologies in the field of Artificial Intelligence, it is already technically
possible to create AI agents that are able to operate in reactive environments.
Nevertheless, several issues stay unsolved, and are subject of ongoing
research.

negative:
Imitation learning is an effective approach for autonomous systems to acquire
control policies when an explicit reward function is unavailable, using
supervision provided as demonstrations from an expert, typically a human
operator. However, standard imitation learning methods assume that the agent
receives examples of observation-action tuples that could be provided, for
instance, to a supervised learning algorithm. This stands in contrast to how
humans and animals imitate: we observe another person performing some behavior
and then figure out which actions will realize that behavior, compensating for
changes in viewpoint, surroundings, object positions and types, and other
factors. We term this kind of imitation learning "imitation-from-observation,"
and propose an imitation learning method based on video prediction with context
translation and deep reinforcement learning. This lifts the assumption in
imitation learning that the demonstration should consist of observations in the
same environment configuration, and enables a variety of interesting
applications, including learning robotic skills that involve tool use simply by
observing videos of human tool use. Our experimental results show the
effectiveness of our approach in learning a wide range of real-world robotic
tasks modeled after common household chores from videos of a human
demonstrator, including sweeping, ladling almonds, pushing objects as well as a
number of tasks in simulation.

negative:
Solving tasks with sparse rewards is one of the most important challenges in
reinforcement learning. In the single-agent setting, this challenge is
addressed by introducing intrinsic rewards that motivate agents to explore
unseen regions of their state spaces; however, applying these techniques
naively to the multi-agent setting results in agents exploring independently,
without any coordination among themselves. Exploration in cooperative
multi-agent settings can be accelerated and improved if agents coordinate their
exploration. In this paper we introduce a framework for designing intrinsic
rewards which consider what other agents have explored such that the agents can
coordinate. Then, we develop an approach for learning how to dynamically select
between several exploration modalities to maximize extrinsic rewards.
Concretely, we formulate the approach as a hierarchical policy where a
high-level controller selects among sets of policies trained on diverse
intrinsic rewards and the low-level controllers learn the action policies of
all agents under these specific rewards. We demonstrate the effectiveness of
the proposed approach in cooperative domains with sparse rewards where
state-of-the-art methods fail and challenging multi-stage tasks that
necessitate changing modes of coordination.

negative:
There are situations in which an agent should receive rewards only after
having accomplished a series of previous tasks. In other words, the reward that
the agent receives is non-Markovian. One natural and quite general way to
represent history-dependent rewards is via a Mealy machine; a finite state
automaton that produces output sequences (rewards in our case) from input
sequences (state/action observations in our case). In our formal setting, we
consider a Markov decision process (MDP) that models the dynamic of the
environment in which the agent evolves and a Mealy machine synchronised with
this MDP to formalise the non-Markovian reward function. While the MDP is known
by the agent, the reward function is unknown from the agent and must be learnt.
  Learning non-Markov reward functions is a challenge. Our approach to overcome
this challenging problem is a careful combination of the Angluin's L* active
learning algorithm to learn finite automata, testing techniques for
establishing conformance of finite model hypothesis and optimisation techniques
for computing optimal strategies in Markovian (immediate) reward MDPs. We also
show how our framework can be combined with classical heuristics such as Monte
Carlo Tree Search. We illustrate our algorithms and a preliminary
implementation on two typical examples for AI.

negative:
To solve its task, a robot needs to have the ability to interpret its
perceptions. In vision, this interpretation is particularly difficult and
relies on the understanding of the structure of the scene, at least to the
extent of its task and sensorimotor abilities. A robot with the ability to
build and adapt this interpretation process according to its own tasks and
capabilities would push away the limits of what robots can achieve in a non
controlled environment. A solution is to provide the robot with processes to
build such representations that are not specific to an environment or a
situation. A lot of works focus on objects segmentation, recognition and
manipulation. Defining an object solely on the basis of its visual appearance
is challenging given the wide range of possible objects and environments.
Therefore, current works make simplifying assumptions about the structure of a
scene. Such assumptions reduce the adaptivity of the object extraction process
to the environments in which the assumption holds. To limit such assumptions,
we introduce an exploration method aimed at identifying moveable elements in a
scene without considering the concept of object. By using the interactive
perception framework, we aim at bootstrapping the acquisition process of a
representation of the environment with a minimum of context specific
assumptions. The robotic system builds a perceptual map called relevance map
which indicates the moveable parts of the current scene. A classifier is
trained online to predict the category of each region (moveable or
non-moveable). It is also used to select a region with which to interact, with
the goal of minimizing the uncertainty of the classification. A specific
classifier is introduced to fit these needs: the collaborative mixture models
classifier. The method is tested on a set of scenarios of increasing
complexity, using both simulations and a PR2 robot.

negative:
Deep reinforcement learning primarily focuses on learning behavior, usually
overlooking the fact that an agent's function is largely determined by form.
So, how should one go about finding a morphology fit for solving tasks in a
given environment? Current approaches that co-adapt morphology and behavior use
a specific task's reward as a signal for morphology optimization. However, this
often requires expensive policy optimization and results in task-dependent
morphologies that are not built to generalize. In this work, we propose a new
approach, Task-Agnostic Morphology Evolution (TAME), to alleviate both of these
issues. Without any task or reward specification, TAME evolves morphologies by
only applying randomly sampled action primitives on a population of agents.
This is accomplished using an information-theoretic objective that efficiently
ranks agents by their ability to reach diverse states in the environment and
the causality of their actions. Finally, we empirically demonstrate that across
2D, 3D, and manipulation environments TAME can evolve morphologies that match
the multi-task performance of those learned with task supervised algorithms.
Our code and videos can be found at
https://sites.google.com/view/task-agnostic-evolution.

negative:
Deep Reinforcement Learning has been shown to be very successful in complex
games, e.g. Atari or Go. These games have clearly defined rules, and hence
allow simulation. In many practical applications, however, interactions with
the environment are costly and a good simulator of the environment is not
available. Further, as environments differ by application, the optimal
inductive bias (architecture, hyperparameters, etc.) of a reinforcement agent
depends on the application. In this work, we propose a multi-arm bandit
framework that selects from a set of different reinforcement learning agents to
choose the one with the best inductive bias. To alleviate the problem of sparse
rewards, the reinforcement learning agents are augmented with surrogate
rewards. This helps the bandit framework to select the best agents early, since
these rewards are smoother and less sparse than the environment reward. The
bandit has the double objective of maximizing the reward while the agents are
learning and selecting the best agent after a finite number of learning steps.
Our experimental results on standard environments show that the proposed
framework is able to consistently select the optimal agent after a finite
number of steps, while collecting more cumulative reward compared to selecting
a sub-optimal architecture or uniformly alternating between different agents.

negative:
A fundamental challenge in multiagent reinforcement learning is to learn
beneficial behaviors in a shared environment with other simultaneously learning
agents. In particular, each agent perceives the environment as effectively
non-stationary due to the changing policies of other agents. Moreover, each
agent is itself constantly learning, leading to natural non-stationarity in the
distribution of experiences encountered. In this paper, we propose a novel
meta-multiagent policy gradient theorem that directly accounts for the
non-stationary policy dynamics inherent to multiagent learning settings. This
is achieved by modeling our gradient updates to consider both an agent's own
non-stationary policy dynamics and the non-stationary policy dynamics of other
agents in the environment. We show that our theoretically grounded approach
provides a general solution to the multiagent learning problem, which
inherently comprises all key aspects of previous state of the art approaches on
this topic. We test our method on a diverse suite of multiagent benchmarks and
demonstrate a more efficient ability to adapt to new agents as they learn than
baseline methods across the full spectrum of mixed incentive, competitive, and
cooperative domains.

negative:
In typical reinforcement learning (RL), the environment is assumed given and
the goal of the learning is to identify an optimal policy for the agent taking
actions through its interactions with the environment. In this paper, we extend
this setting by considering the environment is not given, but controllable and
learnable through its interaction with the agent at the same time. This
extension is motivated by environment design scenarios in the real-world,
including game design, shopping space design and traffic signal design.
Theoretically, we find a dual Markov decision process (MDP) w.r.t. the
environment to that w.r.t. the agent, and derive a policy gradient solution to
optimizing the parametrized environment. Furthermore, discontinuous
environments are addressed by a proposed general generative framework. Our
experiments on a Maze game design task show the effectiveness of the proposed
algorithms in generating diverse and challenging Mazes against various agent
settings.

negative:
The development of intelligent machines is one of the biggest unsolved
challenges in computer science. In this paper, we propose some fundamental
properties these machines should have, focusing in particular on communication
and learning. We discuss a simple environment that could be used to
incrementally teach a machine the basics of natural-language-based
communication, as a prerequisite to more complex interaction with human users.
We also present some conjectures on the sort of algorithms the machine should
support in order to profitably learn from the environment.

negative:
Learning to control an environment without hand-crafted rewards or expert
data remains challenging and is at the frontier of reinforcement learning
research. We present an unsupervised learning algorithm to train agents to
achieve perceptually-specified goals using only a stream of observations and
actions. Our agent simultaneously learns a goal-conditioned policy and a goal
achievement reward function that measures how similar a state is to the goal
state. This dual optimization leads to a co-operative game, giving rise to a
learned reward function that reflects similarity in controllable aspects of the
environment instead of distance in the space of observations. We demonstrate
the efficacy of our agent to learn, in an unsupervised manner, to reach a
diverse set of goals on three domains -- Atari, the DeepMind Control Suite and
DeepMind Lab.

negative:
One question central to Reinforcement Learning is how to learn a feature
representation that supports algorithm scaling and re-use of learned
information from different tasks. Successor Features approach this problem by
learning a feature representation that satisfies a temporal constraint. We
present an implementation of an approach that decouples the feature
representation from the reward function, making it suitable for transferring
knowledge between domains. We then assess the advantages and limitations of
using Successor Features for transfer.

negative:
Reinforcement learning is a promising approach to developing hard-to-engineer
adaptive solutions for complex and diverse robotic tasks. However, learning
with real-world robots is often unreliable and difficult, which resulted in
their low adoption in reinforcement learning research. This difficulty is
worsened by the lack of guidelines for setting up learning tasks with robots.
In this work, we develop a learning task with a UR5 robotic arm to bring to
light some key elements of a task setup and study their contributions to the
challenges with robots. We find that learning performance can be highly
sensitive to the setup, and thus oversights and omissions in setup details can
make effective learning, reproducibility, and fair comparison hard. Our study
suggests some mitigating steps to help future experimenters avoid difficulties
and pitfalls. We show that highly reliable and repeatable experiments can be
performed in our setup, indicating the possibility of reinforcement learning
research extensively based on real-world robots.

negative:
In this work we create agents that can perform well beyond a single,
individual task, that exhibit much wider generalisation of behaviour to a
massive, rich space of challenges. We define a universe of tasks within an
environment domain and demonstrate the ability to train agents that are
generally capable across this vast space and beyond. The environment is
natively multi-agent, spanning the continuum of competitive, cooperative, and
independent games, which are situated within procedurally generated physical 3D
worlds. The resulting space is exceptionally diverse in terms of the challenges
posed to agents, and as such, even measuring the learning progress of an agent
is an open research problem. We propose an iterative notion of improvement
between successive generations of agents, rather than seeking to maximise a
singular objective, allowing us to quantify progress despite tasks being
incomparable in terms of achievable rewards. We show that through constructing
an open-ended learning process, which dynamically changes the training task
distributions and training objectives such that the agent never stops learning,
we achieve consistent learning of new behaviours. The resulting agent is able
to score reward in every one of our humanly solvable evaluation levels, with
behaviour generalising to many held-out points in the universe of tasks.
Examples of this zero-shot generalisation include good performance on Hide and
Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks
we characterise the behaviour of our agent, and find interesting emergent
heuristic behaviours such as trial-and-error experimentation, simple tool use,
option switching, and cooperation. Finally, we demonstrate that the general
capabilities of this agent could unlock larger scale transfer of behaviour
through cheap finetuning.

negative:
An important goal of research in Deep Reinforcement Learning in mobile
robotics is to train agents capable of solving complex tasks, which require a
high level of scene understanding and reasoning from an egocentric perspective.
When trained from simulations, optimal environments should satisfy a currently
unobtainable combination of high-fidelity photographic observations, massive
amounts of different environment configurations and fast simulation speeds. In
this paper we argue that research on training agents capable of complex
reasoning can be simplified by decoupling from the requirement of high fidelity
photographic observations. We present a suite of tasks requiring complex
reasoning and exploration in continuous, partially observable 3D environments.
The objective is to provide challenging scenarios and a robust baseline agent
architecture that can be trained on mid-range consumer hardware in under 24h.
Our scenarios combine two key advantages: (i) they are based on a simple but
highly efficient 3D environment (ViZDoom) which allows high speed simulation
(12000fps); (ii) the scenarios provide the user with a range of difficulty
settings, in order to identify the limitations of current state of the art
algorithms and network architectures. We aim to increase accessibility to the
field of Deep-RL by providing baselines for challenging scenarios where new
ideas can be iterated on quickly. We argue that the community should be able to
address challenging problems in reasoning of mobile agents without the need for
a large compute infrastructure.

negative:
Despite the potential of reinforcement learning (RL) for building
general-purpose robotic systems, training RL agents to solve robotics tasks
still remains challenging due to the difficulty of exploration in purely
continuous action spaces. Addressing this problem is an active area of research
with the majority of focus on improving RL methods via better optimization or
more efficient exploration. An alternate but important component to consider
improving is the interface of the RL algorithm with the robot. In this work, we
manually specify a library of robot action primitives (RAPS), parameterized
with arguments that are learned by an RL policy. These parameterized primitives
are expressive, simple to implement, enable efficient exploration and can be
transferred across robots, tasks and environments. We perform a thorough
empirical study across challenging tasks in three distinct domains with image
input and a sparse terminal reward. We find that our simple change to the
action interface substantially improves both the learning efficiency and task
performance irrespective of the underlying RL algorithm, significantly
outperforming prior methods which learn skills from offline expert data. Code
and videos at https://mihdalal.github.io/raps/

negative:
A practical approach to robot reinforcement learning is to first collect a
large batch of real or simulated robot interaction data, using some data
collection policy, and then learn from this data to perform various tasks,
using offline learning algorithms. Previous work focused on manually designing
the data collection policy, and on tasks where suitable policies can easily be
designed, such as random picking policies for collecting data about object
grasping. For more complex tasks, however, it may be difficult to find a data
collection policy that explores the environment effectively, and produces data
that is diverse enough for the downstream task. In this work, we propose that
data collection policies should actively explore the environment to collect
diverse data. In particular, we develop a simple-yet-effective goal-conditioned
reinforcement-learning method that actively focuses data collection on novel
observations, thereby collecting a diverse data-set. We evaluate our method on
simulated robot manipulation tasks with visual inputs and show that the
improved diversity of active data collection leads to significant improvements
in the downstream learning tasks.

negative:
While recent progress in deep reinforcement learning has enabled robots to
learn complex behaviors, tasks with long horizons and sparse rewards remain an
ongoing challenge. In this work, we propose an effective reward shaping method
through predictive coding to tackle sparse reward problems. By learning
predictive representations offline and using these representations for reward
shaping, we gain access to reward signals that understand the structure and
dynamics of the environment. In particular, our method achieves better learning
by providing reward signals that 1) understand environment dynamics 2)
emphasize on features most useful for learning 3) resist noise in learned
representations through reward accumulation. We demonstrate the usefulness of
this approach in different domains ranging from robotic manipulation to
navigation, and we show that reward signals produced through predictive coding
are as effective for learning as hand-crafted rewards.

negative:
Reinforcement learning is a promising method to accomplish robotic control
tasks. The task of playing musical instruments is, however, largely unexplored
because it involves the challenge of achieving sequential goals - melodies -
that have a temporal dimension. In this paper, we address robotic musicianship
by introducing a temporal extension to goal-conditioned reinforcement learning:
Time-dependent goals. We demonstrate that these can be used to train a robotic
musician to play the theremin instrument. We train the robotic agent in
simulation and transfer the acquired policy to a real-world robotic
thereminist. Supplemental video: https://youtu.be/jvC9mPzdQN4

negative:
Mastering robotic manipulation skills through reinforcement learning (RL)
typically requires the design of shaped reward functions. Recent developments
in this area have demonstrated that using sparse rewards, i.e. rewarding the
agent only when the task has been successfully completed, can lead to better
policies. However, state-action space exploration is more difficult in this
case. Recent RL approaches to learning with sparse rewards have leveraged
high-quality human demonstrations for the task, but these can be costly, time
consuming or even impossible to obtain. In this paper, we propose a novel and
effective approach that does not require human demonstrations. We observe that
every robotic manipulation task could be seen as involving a locomotion task
from the perspective of the object being manipulated, i.e. the object could
learn how to reach a target state on its own. In order to exploit this idea, we
introduce a framework whereby an object locomotion policy is initially obtained
using a realistic physics simulator. This policy is then used to generate
auxiliary rewards, called simulated locomotion demonstration rewards (SLDRs),
which enable us to learn the robot manipulation policy. The proposed approach
has been evaluated on 13 tasks of increasing complexity, and can achieve higher
success rate and faster learning rates compared to alternative algorithms.
SLDRs are especially beneficial for tasks like multi-object stacking and
non-rigid object manipulation.

negative:
The application of reinforcement learning (RL) in robotic control is still
limited in the environments with sparse and delayed rewards. In this paper, we
propose a practical self-imitation learning method named Self-Imitation
Learning with Constant Reward (SILCR). Instead of requiring hand-defined
immediate rewards from environments, our method assigns the immediate rewards
at each timestep with constant values according to their final episodic
rewards. In this way, even if the dense rewards from environments are
unavailable, every action taken by the agents would be guided properly. We
demonstrate the effectiveness of our method in some challenging continuous
robotics control tasks in MuJoCo simulation and the results show that our
method significantly outperforms the alternative methods in tasks with sparse
and delayed rewards. Even compared with alternatives with dense rewards
available, our method achieves competitive performance. The ablation
experiments also show the stability and reproducibility of our method.

negative:
In recent years, a variety of tasks have been accomplished by deep
reinforcement learning (DRL). However, when applying DRL to tasks in a
real-world environment, designing an appropriate reward is difficult. Rewards
obtained via actual hardware sensors may include noise, misinterpretation, or
failed observations. The learning instability caused by these unstable signals
is a problem that remains to be solved in DRL. In this work, we propose an
approach that extends existing DRL models by adding a subtask to directly
estimate the variance contained in the reward signal. The model then takes the
feature map learned by the subtask in a critic network and sends it to the
actor network. This enables stable learning that is robust to the effects of
potential noise. The results of experiments in the Atari game domain with
unstable reward signals show that our method stabilizes training convergence.
We also discuss the extensibility of the model by visualizing feature maps.
This approach has the potential to make DRL more practical for use in noisy,
real-world scenarios.

negative:
Transfer learning is an important new subfield of multiagent reinforcement
learning that aims to help an agent learn about a problem by using knowledge
that it has gained solving another problem, or by using knowledge that is
communicated to it by an agent who already knows the problem. This is useful
when one wishes to change the architecture or learning algorithm of an agent
(so that the new knowledge need not be built "from scratch"), when new agents
are frequently introduced to the environment with no knowledge, or when an
agent must adapt to similar but different problems. Great progress has been
made in the agent-to-agent case using the Teacher/Student framework proposed by
(Torrey and Taylor 2013). However, that approach requires that learning from a
teacher be treated differently from learning in every other reinforcement
learning context. In this paper, I propose a method which allows the
teacher/student framework to be applied in a way that fits directly and
naturally into the more general reinforcement learning framework by integrating
the teacher feedback into the reward signal received by the learning agent. I
show that this approach can significantly improve the rate of learning for an
agent playing a one-player stochastic game; I give examples of potential
pitfalls of the approach; and I propose further areas of research building on
this framework.

negative:
Humans continually expand their learned knowledge to new domains and learn
new concepts without any interference with past learned experiences. In
contrast, machine learning models perform poorly in a continual learning
setting, where input data distribution changes over time. Inspired by the
nervous system learning mechanisms, we develop a computational model that
enables a deep neural network to learn new concepts and expand its learned
knowledge to new domains incrementally in a continual learning setting. We rely
on the Parallel Distributed Processing theory to encode abstract concepts in an
embedding space in terms of a multimodal distribution. This embedding space is
modeled by internal data representations in a hidden network layer. We also
leverage the Complementary Learning Systems theory to equip the model with a
memory mechanism to overcome catastrophic forgetting through implementing
pseudo-rehearsal. Our model can generate pseudo-data points for experience
replay and accumulate new experiences to past learned experiences without
causing cross-task interference.

negative:
A common approach to solving physical reasoning tasks is to train a value
learner on example tasks. A limitation of such an approach is that it requires
learning about object dynamics solely from reward values assigned to the final
state of a rollout of the environment. This study aims to address this
limitation by augmenting the reward value with self-supervised signals about
object dynamics. Specifically, we train the model to characterize the
similarity of two environment rollouts, jointly with predicting the outcome of
the reasoning task. This similarity can be defined as a distance measure
between the trajectory of objects in the two rollouts, or learned directly from
pixels using a contrastive formulation. Empirically, we find that this approach
leads to substantial performance improvements on the PHYRE benchmark for
physical reasoning (Bakhtin et al., 2019), establishing a new state-of-the-art.

negative:
We study the problem of designing autonomous agents that can learn to
cooperate effectively with a potentially suboptimal partner while having no
access to the joint reward function. This problem is modeled as a cooperative
episodic two-agent Markov decision process. We assume control over only the
first of the two agents in a Stackelberg formulation of the game, where the
second agent is acting so as to maximise expected utility given the first
agent's policy. How should the first agent act in order to learn the joint
reward function as quickly as possible and so that the joint policy is as close
to optimal as possible? We analyse how knowledge about the reward function can
be gained in this interactive two-agent scenario. We show that when the
learning agent's policies have a significant effect on the transition function,
the reward function can be learned efficiently.

negative:
Reinforcement learning algorithms can train agents that solve problems in
complex, interesting environments. Normally, the complexity of the trained
agent is closely related to the complexity of the environment. This suggests
that a highly capable agent requires a complex environment for training. In
this paper, we point out that a competitive multi-agent environment trained
with self-play can produce behaviors that are far more complex than the
environment itself. We also point out that such environments come with a
natural curriculum, because for any skill level, an environment full of agents
of this level will have the right level of difficulty. This work introduces
several competitive multi-agent environments where agents compete in a 3D world
with simulated physics. The trained agents learn a wide variety of complex and
interesting skills, even though the environment themselves are relatively
simple. The skills include behaviors such as running, blocking, ducking,
tackling, fooling opponents, kicking, and defending using both arms and legs. A
highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX

negative:
A key problem in structured output prediction is direct optimization of the
task reward function that matters for test evaluation. This paper presents a
simple and computationally efficient approach to incorporate task reward into a
maximum likelihood framework. By establishing a link between the log-likelihood
and expected reward objectives, we show that an optimal regularized expected
reward is achieved when the conditional distribution of the outputs given the
inputs is proportional to their exponentiated scaled rewards. Accordingly, we
present a framework to smooth the predictive probability of the outputs using
their corresponding rewards. We optimize the conditional log-probability of
augmented outputs that are sampled proportionally to their exponentiated scaled
rewards. Experiments on neural sequence to sequence models for speech
recognition and machine translation show notable improvements over a maximum
likelihood baseline by using reward augmented maximum likelihood (RAML), where
the rewards are defined as the negative edit distance between the outputs and
the ground truth labels.

negative:
Many real-world robotic operations that involve high-dimensional humanoid
robots require fast-reaction to plan disturbances and probabilistic guarantees
over collision risks, whereas most probabilistic motion planning approaches
developed for car-like robots can not be directly applied to high-dimensional
robots. In this paper, we present probabilistic Chekov (p-Chekov), a
fast-reactive motion planning system that can provide safety guarantees for
high-dimensional robots suffering from process noises and observation noises.
Leveraging recent advances in machine learning as well as our previous work in
deterministic motion planning that integrated trajectory optimization into a
sparse roadmap framework, p-Chekov demonstrates its superiority in terms of
collision avoidance ability and planning speed in high-dimensional robotic
motion planning tasks in complex environments without the convexification of
obstacles. Comprehensive theoretical and empirical analysis provided in this
paper shows that p-Chekov can effectively satisfy user-specified chance
constraints over collision risk in practical robotic manipulation tasks.

negative:
For a multi-robot system equipped with heterogeneous capabilities, this paper
presents a mechanism to allocate robots to tasks in a resilient manner when
anomalous environmental conditions such as weather events or adversarial
attacks affect the performance of robots within the tasks. Our primary
objective is to ensure that each task is assigned the requisite level of
resources, measured as the aggregated capabilities of the robots allocated to
the task. By keeping track of task performance deviations under external
perturbations, our framework quantifies the extent to which robot capabilities
(e.g., visual sensing or aerial mobility) are affected by environmental
conditions. This enables an optimization-based framework to flexibly reallocate
robots to tasks based on the most degraded capabilities within each task. In
the face of resource limitations and adverse environmental conditions, our
algorithm minimally relaxes the resource constraints corresponding to some
tasks, thus exhibiting a graceful degradation of performance. Simulated
experiments in a multi-robot coverage and target tracking scenario demonstrate
the efficacy of the proposed approach.

negative:
Mobile robots, performing long-term manipulation activities in human
environments, have to perceive a wide variety of objects possessing very
different visual characteristics and need to reliably keep track of these
throughout the execution of a task. In order to be efficient, robot perception
capabilities need to go beyond what is currently perceivable and should be able
to answer queries about both current and past scenes. In this paper we
investigate a perception system for long-term robot manipulation that keeps
track of the changing environment and builds a representation of the perceived
world. Specifically we introduce an amortized component that spreads perception
tasks throughout the execution cycle. The resulting query driven perception
system asynchronously integrates results from logged images into a symbolic and
numeric (what we call sub-symbolic) representation that forms the perceptual
belief state of the robot.

negative:
We describe a simple scheme that allows an agent to learn about its
environment in an unsupervised manner. Our scheme pits two versions of the same
agent, Alice and Bob, against one another. Alice proposes a task for Bob to
complete; and then Bob attempts to complete the task. In this work we will
focus on two kinds of environments: (nearly) reversible environments and
environments that can be reset. Alice will "propose" the task by doing a
sequence of actions and then Bob must undo or repeat them, respectively. Via an
appropriate reward structure, Alice and Bob automatically generate a curriculum
of exploration, enabling unsupervised training of the agent. When Bob is
deployed on an RL task within the environment, this unsupervised training
reduces the number of supervised episodes needed to learn, and in some cases
converges to a higher reward.

negative:
Inverse reinforcement learning attempts to reconstruct the reward function in
a Markov decision problem, using observations of agent actions. As already
observed in Russell [1998] the problem is ill-posed, and the reward function is
not identifiable, even under the presence of perfect information about optimal
behavior. We provide a resolution to this non-identifiability for problems with
entropy regularization. For a given environment, we fully characterize the
reward functions leading to a given policy and demonstrate that, given
demonstrations of actions for the same reward under two distinct discount
factors, or under sufficiently different environments, the unobserved reward
can be recovered up to a constant. We also give general necessary and
sufficient conditions for reconstruction of time-homogeneous rewards on finite
horizons, and for action-independent rewards, generalizing recent results of
Kim et al. [2021] and Fu et al. [2018].

negative:
It has been postulated that a good representation is one that disentangles
the underlying explanatory factors of variation. However, it remains an open
question what kind of training framework could potentially achieve that.
Whereas most previous work focuses on the static setting (e.g., with images),
we postulate that some of the causal factors could be discovered if the learner
is allowed to interact with its environment. The agent can experiment with
different actions and observe their effects. More specifically, we hypothesize
that some of these factors correspond to aspects of the environment which are
independently controllable, i.e., that there exists a policy and a learnable
feature for each such aspect of the environment, such that this policy can
yield changes in that feature with minimal changes to other features that
explain the statistical variations in the observed data. We propose a specific
objective function to find such factors, and verify experimentally that it can
indeed disentangle independently controllable aspects of the environment
without any extrinsic reward signal.

negative:
Robotic grasping is a crucial area of research as it can result in the
acceleration of the automation of several Industries utilizing robots ranging
from manufacturing to healthcare. Reinforcement learning is the field of study
where an agent learns a policy to execute an action by exploring and exploiting
rewards from an environment. Reinforcement learning can thus be used by the
agent to learn how to execute a certain task, in our case grasping an object.
We have used the Pick and Place environment provided by OpenAI's Gym to
engineer rewards. Hindsight Experience Replay (HER) has shown promising results
with problems having a sparse reward. In the default configuration of the
OpenAI baseline and environment the reward function is calculated using the
distance between the target location and the robot end-effector. By weighting
the cost based on the distance of the end-effector from the goal in the x,y and
z-axes we were able to almost halve the learning time compared to the baselines
provided by OpenAI, an intuitive strategy that further reduced learning time.
In this project, we were also able to introduce certain user desired
trajectories in the learnt policies (city-block / Manhattan trajectories). This
helps us understand that by engineering the rewards we can tune the agent to
learn policies in a certain way even if it might not be the most optimal but is
the desired manner.

negative:
In cooperative multi-agent reinforcement learning (MARL), how to design a
suitable reward signal to accelerate learning and stabilize convergence is a
critical problem. The global reward signal assigns the same global reward to
all agents without distinguishing their contributions, while the local reward
signal provides different local rewards to each agent based solely on
individual behavior. Both of the two reward assignment approaches have some
shortcomings: the former might encourage lazy agents, while the latter might
produce selfish agents.
  In this paper, we study reward design problem in cooperative MARL based on
packet routing environments. Firstly, we show that the above two reward signals
are prone to produce suboptimal policies. Then, inspired by some observations
and considerations, we design some mixed reward signals, which are
off-the-shelf to learn better policies. Finally, we turn the mixed reward
signals into the adaptive counterparts, which achieve best results in our
experiments. Other reward signals are also discussed in this paper. As reward
design is a very fundamental problem in RL and especially in MARL, we hope that
MARL researchers can rethink the rewards used in their systems.

negative:
Legg and Hutter, as well as subsequent authors, considered intelligent agents
through the lens of interaction with reward-giving environments, attempting to
assign numeric intelligence measures to such agents, with the guiding principle
that a more intelligent agent should gain higher rewards from environments in
some aggregate sense. In this paper, we consider a related question: rather
than measure numeric intelligence of one Legg- Hutter agent, how can we compare
the relative intelligence of two Legg-Hutter agents? We propose an elegant
answer based on the following insight: we can view Legg-Hutter agents as
candidates in an election, whose voters are environments, letting each
environment vote (via its rewards) which agent (if either) is more intelligent.
This leads to an abstract family of comparators simple enough that we can prove
some structural theorems about them. It is an open question whether these
structural theorems apply to more practical intelligence measures.

negative:
With the incremental development of robotic platforms to automate the manual
processes, path planning has become a critical domain with or without the
knowledge of the indoor and outdoor environment. The algorithms can be
intelligent or pre-structured and should optimally reach the destination
efficiently. The major challenge in this domain is to find a path which is free
from static obstacles as well as dynamic obstacles. In this paper, a
methodology is proposed with the implementation details of the robotic platform
to cover the critical key points and to arrive at the original key point in a
dynamic environment. The main computation is happening inside a Raspberry Pi B+
module, and compass, wheel encoders and ultrasonic sensors were used in the
implementation for the localization of the robot to relevant key points.

negative:
Our world is open-ended, non-stationary, and constantly evolving; thus what
we talk about and how we talk about it change over time. This inherent dynamic
nature of language contrasts with the current static language modelling
paradigm, which trains and evaluates models on utterances from overlapping time
periods. Despite impressive recent progress, we demonstrate that Transformer-XL
language models perform worse in the realistic setup of predicting future
utterances from beyond their training period, and that model performance
becomes increasingly worse with time. We find that, while increasing model size
alone -- a key driver behind recent progress -- does not solve this problem,
having models that continually update their knowledge with new information can
indeed mitigate this performance degradation over time. Hence, given the
compilation of ever-larger language modelling datasets, combined with the
growing list of language-model-based NLP applications that require up-to-date
factual knowledge about the world, we argue that now is the right time to
rethink the static way in which we currently train and evaluate our language
models, and develop adaptive language models that can remain up-to-date with
respect to our ever-changing and non-stationary world. We publicly release our
dynamic, streaming language modelling benchmarks for WMT and arXiv to
facilitate language model evaluation that takes temporal dynamics into account.

negative:
This paper describes an application of reinforcement learning to the mention
detection task. We define a novel action-based formulation for the mention
detection task, in which a model can flexibly revise past labeling decisions by
grouping together tokens and assigning partial mention labels. We devise a
method to create mention-level episodes and we train a model by rewarding
correctly labeled complete mentions, irrespective of the inner structure
created. The model yields results which are on par with a competitive
supervised counterpart while being more flexible in terms of achieving targeted
behavior through reward modeling and generating internal mention structure,
especially on longer mentions.

negative:
Reinforcement Learning (RL) agents require the specification of a reward
signal for learning behaviours. However, introduction of corrupt or stochastic
rewards can yield high variance in learning. Such corruption may be a direct
result of goal misspecification, randomness in the reward signal, or
correlation of the reward with external factors that are not known to the
agent. Corruption or stochasticity of the reward signal can be especially
problematic in robotics, where goal specification can be particularly difficult
for complex tasks. While many variance reduction techniques have been studied
to improve the robustness of the RL process, handling such stochastic or
corrupted reward structures remains difficult. As an alternative for handling
this scenario in model-free RL methods, we suggest using an estimator for both
rewards and value functions. We demonstrate that this improves performance
under corrupted stochastic rewards in both the tabular and non-linear function
approximation settings for a variety of noise types and environments. The use
of reward estimation is a robust and easy-to-implement improvement for handling
corrupted reward signals in model-free RL.

negative:
This paper discusses recent research progress in robotic grasping and
manipulation in the light of the latest Robotic Grasping and Manipulation
Competitions (RGMCs). We first provide an overview of past benchmarks and
competitions related to the robotics manipulation field. Then, we discuss the
methodology behind designing the manipulation tasks in RGMCs. We provide a
detailed analysis of key challenges for each task and identify the most
difficult aspects based on the competing teams' performance in recent years. We
believe that such an analysis is insightful to determine the future research
directions for the robotic manipulation domain.

negative:
We update the method of describing and assessing the process of the study of
an abstract environment by a system, proposed earlier. We do not model any
biological cognition mechanisms and consider the system as an agent equipped
with an information processor (or a group of such agents), which makes a move
in the environment, consumes information supplied by the environment, and gives
out the next move (hence, the process is considered as a game). The system
moves in an unknown environment and should recognize new objects located in it.
In this case, the system should build comprehensive images of visible things
and memorize them if necessary (and it should also choose the current goal
set). The main problems here are object recognition, and the informational
reward rating in the game. Thus, the main novelty of the paper is a new method
of evaluating the amount of visual information about the object as the reward.
In such a system, we suggest using a minimally pre-trained neural network to be
responsible for the recognition: at first, we train the network only for
Biederman geons (geometrical primitives). The geons are generated
programmatically and we demonstrate that such a trained network recognizes
geons in real objects quite well. We also offer to generate, procedurally, new
objects from geon schemes (geon combinations in images) obtained from the
environment and to store them in a database. In this case, we do not obtain new
information about an object (i.e., our reward is maximal, thus the game and the
object cognition process stop) when we stop getting new schemes of this kind.
These schemes are generated from geons connected with the object. In the case
of a possibly known item, the informational reward is maximal when we have no
more detection uncertainty for any of the objects.

negative:
Recently we see a rising number of methods in the field of eXplainable
Artificial Intelligence. To our surprise, their development is driven by model
developers rather than a study of needs for human end users. The analysis of
needs, if done, takes the form of an A/B test rather than a study of open
questions. To answer the question "What would a human operator like to ask the
ML model?" we propose a conversational system explaining decisions of the
predictive model. In this experiment, we developed a chatbot called dr_ant to
talk about machine learning model trained to predict survival odds on Titanic.
People can talk with dr_ant about different aspects of the model to understand
the rationale behind its predictions. Having collected a corpus of 1000+
dialogues, we analyse the most common types of questions that users would like
to ask. To our knowledge, it is the first study which uses a conversational
system to collect the needs of human operators from the interactive and
iterative dialogue explorations of a predictive model.

negative:
We present the process of transitioning machine learning models to the
TensorFlow framework at a large scale in an online advertising ecosystem. In
this talk we address the key challenges we faced and describe how we
successfully tackled them; notably, implementing the models in TF and serving
them efficiently with low latency using various optimization techniques.

negative:
For the past few decades, man has been trying to create an intelligent
computer which can talk and respond like he can. The task of creating a system
that can talk like a human being is the primary objective of Automatic Speech
Recognition. Various Speech Recognition techniques have been developed in
theory and have been applied in practice. This paper discusses the problems
that have been encountered in developing Speech Recognition, the techniques
that have been applied to automate the task, and a representation of the core
problems of present day Speech Recognition by using Fuzzy Mathematics.

negative:
This paper presents a plausible reasoning system to illustrate some broad
issues in knowledge representation: dualities between different reasoning
forms, the difficulty of unifying complementary reasoning styles, and the
approximate nature of plausible reasoning. These issues have a common
underlying theme: there should be an underlying belief calculus of which the
many different reasoning forms are special cases, sometimes approximate. The
system presented allows reasoning about defaults, likelihood, necessity and
possibility in a manner similar to the earlier work of Adams. The system is
based on the belief calculus of subjective Bayesian probability which itself is
based on a few simple assumptions about how belief should be manipulated.
Approximations, semantics, consistency and consequence results are presented
for the system. While this puts these often discussed plausible reasoning forms
on a probabilistic footing, useful application to practical problems remains an
issue.

negative:
This paper presents an acoustic echo canceler based on a U-Net convolutional
neural network for single-talk and double-talk scenarios. U-Net networks have
previously been used in the audio processing area for source separation
problems because of their ability to reproduce the finest details of audio
signals, but to our knowledge, this is the first time they have been used for
acoustic echo cancellation (AEC). The U-Net hyperparameters have been optimized
to obtain the best AEC performance, but using a reduced number of parameters to
meet a latency restriction of 40 ms. The training and testing of our model have
been carried out within the framework of the 'ICASSP 2021 AEC Challenge'
organized by Microsoft. We have trained the optimized U-Net model with a
synthetic dataset only (S-U-Net) and with a synthetic dataset and the
single-talk set of a real dataset (SR-U-Net), both datasets were released for
the challenge. The S-U-Net model presented better results for double-talk
scenarios, thus their inferred near-end signals from the blind testset were
submitted to the challenge. Our canceler ranked 12th among 17 teams, and 5th
among 10 academia teams, obtaining an overall mean opinion score of 3.57.

negative:
Two interpretations about syllogistic statements are described in this paper.
One is the so-called set-based interpretation, which assumes that quantified
statements and syllogisms talk about quantity-relationships between sets. The
other one, the so-called conditional interpretation, assumes that quantified
propositions talk about conditional propositions and how strong are the links
between the antecedent and the consequent. Both interpretations are compared
attending to three different questions (existential import, singular statements
and non-proportional quantifiers) from the point of view of their impact on the
further development of this type of reasoning.

negative:
There is knowledge. There is belief. And there is tacit agreement.' 'We may
talk about objects. We may talk about attributes of the objects. Or we may talk
both about objects and their attributes.' This work inspects tacit agreements
on assumptions about the relation between objects and their attributes, and
studies a way of expressing them, presenting as the result what we term gradual
logic in which the sense of truth gradually shifts. It extends classical logic
instances with a new logical connective capturing the object-attribute
relation. A formal semantics is presented. Decidability is proved. Para-
consistent/epistemic/conditional/intensional/description/combined logics are
compared.