query: My objective is to devise a methodology for constructing intelligent robots that mimic human-like behavior. These robots would be capable of operating in real-world environments and executing a range of complex tasks. Such a robotic agent would interact with its surroundings, making decisions and performing actions. More importantly, it would continuously adapt to its environment by learning from signals and cues it receives. I aim to create a unified framework that can integrate a large number of independent modules and models. Each of these would perform a fundamental function of the robotic agent, such as processing visual inputs, decision-making, and robotic arm manipulation. While these sub-models would be independent, each responsible for a different function, they would need to interact with each other to optimize the overall performance of the agent. I currently believe that the transfer of information, such as gradient information and parameter updates, from one module to another should be swift and efficient. I intend to demonstrate the adaptability of my robotic agent to different environments and situations effectively using various datasets.

positive:
In this paper, we propose a novel task, Manipulation Question Answering
(MQA), where the robot performs manipulation actions to change the environment
in order to answer a given question. To solve this problem, a framework
consisting of a QA module and a manipulation module is proposed. For the QA
module, we adopt the method for the Visual Question Answering (VQA) task. For
the manipulation module, a Deep Q Network (DQN) model is designed to generate
manipulation actions for the robot to interact with the environment. We
consider the situation where the robot continuously manipulating objects inside
a bin until the answer to the question is found. Besides, a novel dataset that
contains a variety of object models, scenarios and corresponding
question-answer pairs is established in a simulation environment. Extensive
experiments have been conducted to validate the effectiveness of the proposed
framework.

positive:
To achieve seamless human-robot interactions, robots need to intimately
reason about complex interaction dynamics and future human behaviors within
their motion planning process. However, there is a disconnect between
state-of-the-art neural network-based human behavior models and robot motion
planners -- either the behavior models are limited in their consideration of
downstream planning or a simplified behavior model is used to ensure
tractability of the planning problem. In this work, we present a framework that
fuses together the interpretability and flexibility of trajectory optimization
(TO) with the predictive power of state-of-the-art human trajectory prediction
models. In particular, we leverage gradient information from data-driven
prediction models to explicitly reason about human-robot interaction dynamics
within a gradient-based TO problem. We demonstrate the efficacy of our approach
in a multi-agent scenario whereby a robot is required to safely and efficiently
navigate through a crowd of up to ten pedestrians. We compare against a variety
of planning methods, and show that by explicitly accounting for interaction
dynamics within the planner, our method offers safer and more efficient
behaviors, even yielding proactive and nuanced behaviors such as waiting for a
pedestrian to pass before moving.

positive:
The study of human-robot interaction is fundamental to the design and use of
robotics in real-world applications. Robots will need to predict and adapt to
the actions of human collaborators in order to achieve good performance and
improve safety and end-user adoption. This paper evaluates a human-robot
collaboration scheme that combines the task allocation and motion levels of
reasoning: the robotic agent uses Bayesian inference to predict the next goal
of its human partner from his or her ongoing motion, and re-plans its own
actions in real time. This anticipative adaptation is desirable in many
practical scenarios, where humans are unable or unwilling to take on the
cognitive overhead required to explicitly communicate their intent to the
robot. A behavioral experiment indicates that the combination of goal inference
and dynamic task planning significantly improves both objective and perceived
performance of the human-robot team. Participants were highly sensitive to the
differences between robot behaviors, preferring to work with a robot that
adapted to their actions over one that did not.

positive:
Learning predictive models from interaction with the world allows an agent,
such as a robot, to learn about how the world works, and then use this learned
model to plan coordinated sequences of actions to bring about desired outcomes.
However, learning a model that captures the dynamics of complex skills
represents a major challenge: if the agent needs a good model to perform these
skills, it might never be able to collect the experience on its own that is
required to learn these delicate and complex behaviors. Instead, we can imagine
augmenting the training set with observational data of other agents, such as
humans. Such data is likely more plentiful, but represents a different
embodiment. For example, videos of humans might show a robot how to use a tool,
but (i) are not annotated with suitable robot actions, and (ii) contain a
systematic distributional shift due to the embodiment differences between
humans and robots. We address the first challenge by formulating the
corresponding graphical model and treating the action as an observed variable
for the interaction data and an unobserved variable for the observation data,
and the second challenge by using a domain-dependent prior. In addition to
interaction data, our method is able to leverage videos of passive observations
in a driving dataset and a dataset of robotic manipulation videos. A robotic
planning agent equipped with our method can learn to use tools in a tabletop
robotic manipulation setting by observing humans without ever seeing a robotic
video of tool use.

positive:
Learning from Demonstration (LfD) constitutes one of the most robust
methodologies for constructing efficient cognitive robotic systems. Despite the
large body of research works already reported, current key technological
challenges include those of multi-agent learning and long-term autonomy.
Towards this direction, a novel cognitive architecture for multi-agent LfD
robotic learning is introduced, targeting to enable the reliable deployment of
open, scalable and expandable robotic systems in large-scale and complex
environments. In particular, the designed architecture capitalizes on the
recent advances in the Artificial Intelligence (AI) field, by establishing a
Federated Learning (FL)-based framework for incarnating a multi-human
multi-robot collaborative learning environment. The fundamental
conceptualization relies on employing multiple AI-empowered cognitive processes
(implementing various robotic tasks) that operate at the edge nodes of a
network of robotic platforms, while global AI models (underpinning the
aforementioned robotic tasks) are collectively created and shared among the
network, by elegantly combining information from a large number of human-robot
interaction instances. Regarding pivotal novelties, the designed cognitive
architecture a) introduces a new FL-based formalism that extends the
conventional LfD learning paradigm to support large-scale multi-agent
operational settings, b) elaborates previous FL-based self-learning robotic
schemes so as to incorporate the human in the learning loop and c) consolidates
the fundamental principles of FL with additional sophisticated AI-enabled
learning methodologies for modelling the multi-level inter-dependencies among
the robotic tasks. The applicability of the proposed framework is explained
using an example of a real-world industrial case study for agile
production-based Critical Raw Materials (CRM) recovery.

positive:
In open-ended continuous environments, robots need to learn multiple
parameterised control tasks in hierarchical reinforcement learning. We
hypothesise that the most complex tasks can be learned more easily by
transferring knowledge from simpler tasks, and faster by adapting the
complexity of the actions to the task. We propose a task-oriented
representation of complex actions, called procedures, to learn online task
relationships and unbounded sequences of action primitives to control the
different observables of the environment. Combining both goal-babbling with
imitation learning, and active learning with transfer of knowledge based on
intrinsic motivation, our algorithm self-organises its learning process. It
chooses at any given time a task to focus on; and what, how, when and from whom
to transfer knowledge. We show with a simulation and a real industrial robot
arm, in cross-task and cross-learner transfer settings, that task composition
is key to tackle highly complex tasks. Task decomposition is also efficiently
transferred across different embodied learners and by active imitation, where
the robot requests just a small amount of demonstrations and the adequate type
of information. The robot learns and exploits task dependencies so as to learn
tasks of every complexity.

positive:
The great diversity of end-user tasks ranging from manufacturing environments
to personal homes makes pre-programming robots for general purpose applications
extremely challenging. In fact, teaching robots new actions from scratch that
can be reused for previously unseen tasks remains a difficult challenge and is
generally left up to robotics experts. In this work, we present iRoPro, an
interactive Robot Programming framework that allows end-users with little to no
technical background to teach a robot new reusable actions. We combine
Programming by Demonstration and Automated Planning techniques to allow the
user to construct the robot's knowledge base by teaching new actions by
kinesthetic demonstration. The actions are generalised and reused with a task
planner to solve previously unseen problems defined by the user. We implement
iRoPro as an end-to-end system on a Baxter Research Robot to simultaneously
teach low- and high-level actions by demonstration that the user can customise
via a Graphical User Interface to adapt to their specific use case. To evaluate
the feasibility of our approach, we first conducted pre-design experiments to
better understand the user's adoption of involved concepts and the proposed
robot programming process. We compare results with post-design experiments,
where we conducted a user study to validate the usability of our approach with
real end-users. Overall, we showed that users with different programming levels
and educational backgrounds can easily learn and use iRoPro and its robot
programming process.

positive:
Intelligent robots need to achieve abstract objectives using concrete,
spatiotemporally complex sensory information and motor control. Tabula rasa
deep reinforcement learning (RL) has tackled demanding tasks in terms of either
visual, abstract, or physical reasoning, but solving these jointly remains a
formidable challenge. One recent, unsolved benchmark task that integrates these
challenges is Mujoban, where a robot needs to arrange 3D warehouses generated
from 2D Sokoban puzzles. We explore whether integrated tasks like Mujoban can
be solved by composing RL modules together in a sense-plan-act hierarchy, where
modules have well-defined roles similarly to classic robot architectures.
Unlike classic architectures that are typically model-based, we use only
model-free modules trained with RL or supervised learning. We find that our
modular RL approach dramatically outperforms the state-of-the-art monolithic RL
agent on Mujoban. Further, learned modules can be reused when, e.g., using a
different robot platform to solve the same task. Together our results give
strong evidence for the importance of research into modular RL designs. Project
website: https://sites.google.com/view/modular-rl/

positive:
Service robots need to show appropriate social behaviour in order to be
deployed in social environments such as healthcare, education, retail, etc.
Some of the main capabilities that robots should have are navigation and
conversational skills. If the person is impatient, the person might want a
robot to navigate faster and vice versa. Linguistic features that indicate
politeness can provide social cues about a person's patient and impatient
behaviour. The novelty presented in this paper is to dynamically incorporate
politeness in robotic dialogue systems for navigation. Understanding the
politeness in users' speech can be used to modulate the robot behaviour and
responses. Therefore, we developed a dialogue system to navigate in an indoor
environment, which produces different robot behaviours and responses based on
users' intention and degree of politeness. We deploy and test our system with
the Pepper robot that adapts to the changes in user's politeness.

positive:
Reinforcement learning methods can achieve significant performance but
require a large amount of training data collected on the same robotic platform.
A policy trained with expensive data is rendered useless after making even a
minor change to the robot hardware. In this paper, we address the challenging
problem of adapting a policy, trained to perform a task, to a novel robotic
hardware platform given only few demonstrations of robot motion trajectories on
the target robot. We formulate it as a few-shot meta-learning problem where the
goal is to find a meta-model that captures the common structure shared across
different robotic platforms such that data-efficient adaptation can be
performed. We achieve such adaptation by introducing a learning framework
consisting of a probabilistic gradient-based meta-learning algorithm that
models the uncertainty arising from the few-shot setting with a low-dimensional
latent variable. We experimentally evaluate our framework on a simulated
reaching and a real-robot picking task using 400 simulated robots generated by
varying the physical parameters of an existing set of robotic platforms. Our
results show that the proposed method can successfully adapt a trained policy
to different robotic platforms with novel physical parameters and the
superiority of our meta-learning algorithm compared to state-of-the-art methods
for the introduced few-shot policy adaptation problem.

positive:
Deep artificial neural networks, trained with labeled data sets are widely
used in numerous vision and robotics applications today. In terms of AI, these
are called reflex models, referring to the fact that they do not self-evolve or
actively adapt to environmental changes. As demand for intelligent robot
control expands to many high level tasks, reinforcement learning and state
based models play an increasingly important role. Herein, in computer vision
and robotics domain, we study a novel approach to add reinforcement controls
onto the image recognition reflex models to attain better overall performance,
specifically to a wider environment range beyond what is expected of the task
reflex models. Follow a common infrastructure with environment sensing and AI
based modeling of self-adaptive agents, we implement multiple types of AI
control agents. To the end, we provide comparative results of these agents with
baseline, and an insightful analysis of their benefit to improve overall image
recognition performance in real world.

negative:
One spectrum on which robotic control paradigms lie is the degree in which a
model of the environment is involved, from methods that are completely
model-free such as model-free RL, to methods that require a known model such as
optimal control, with other methods such as model-based RL somewhere in the
middle. On one end of the spectrum, model-free RL can learn control policies
for high-dimensional (hi-dim), complex robotic tasks through trial-and-error
without knowledge of a model of the environment, but tends to require a large
amount of data. On the other end, "classical methods" such as optimal control
generate solutions without collecting data, but assume that an accurate model
of the system and environment is known and are mostly limited to problems with
low-dimensional (lo-dim) state spaces. In this paper, we bring the two ends of
the spectrum together. Although models of hi-dim systems and environments may
not exist, lo-dim approximations of these systems and environments are widely
available, especially in robotics. Therefore, we propose to solve hi-dim,
complex robotic tasks in two stages. First, assuming a coarse model of the
hi-dim system, we compute a lo-dim value function for the lo-dim version of the
problem using classical methods (eg. value iteration and optimal control).
Then, the lo-dim value function is used as a baseline function to warm-start
the model-free RL process that learns hi-dim policies. The lo-dim value
function provides global guidance for model-free RL, alleviating the data
inefficiency of model-free RL. We demonstrate our approach on two robot
learning tasks with hi-dim state spaces and observe significant improvement in
policy performance and learning efficiency. We also give an empirical analysis
of our method with a third task.

negative:
To solve its task, a robot needs to have the ability to interpret its
perceptions. In vision, this interpretation is particularly difficult and
relies on the understanding of the structure of the scene, at least to the
extent of its task and sensorimotor abilities. A robot with the ability to
build and adapt this interpretation process according to its own tasks and
capabilities would push away the limits of what robots can achieve in a non
controlled environment. A solution is to provide the robot with processes to
build such representations that are not specific to an environment or a
situation. A lot of works focus on objects segmentation, recognition and
manipulation. Defining an object solely on the basis of its visual appearance
is challenging given the wide range of possible objects and environments.
Therefore, current works make simplifying assumptions about the structure of a
scene. Such assumptions reduce the adaptivity of the object extraction process
to the environments in which the assumption holds. To limit such assumptions,
we introduce an exploration method aimed at identifying moveable elements in a
scene without considering the concept of object. By using the interactive
perception framework, we aim at bootstrapping the acquisition process of a
representation of the environment with a minimum of context specific
assumptions. The robotic system builds a perceptual map called relevance map
which indicates the moveable parts of the current scene. A classifier is
trained online to predict the category of each region (moveable or
non-moveable). It is also used to select a region with which to interact, with
the goal of minimizing the uncertainty of the classification. A specific
classifier is introduced to fit these needs: the collaborative mixture models
classifier. The method is tested on a set of scenarios of increasing
complexity, using both simulations and a PR2 robot.

negative:
Robotic systems are more present in our society everyday. In human-robot
environments, it is crucial that end-users may correctly understand their
robotic team-partners, in order to collaboratively complete a task. To increase
action understanding, users demand more explainability about the decisions by
the robot in particular situations. Recently, explainable robotic systems have
emerged as an alternative focused not only on completing a task satisfactorily,
but also on justifying, in a human-like manner, the reasons that lead to making
a decision. In reinforcement learning scenarios, a great effort has been
focused on providing explanations using data-driven approaches, particularly
from the visual input modality in deep learning-based systems. In this work, we
focus rather on the decision-making process of reinforcement learning agents
performing a task in a robotic scenario. Experimental results are obtained
using 3 different set-ups, namely, a deterministic navigation task, a
stochastic navigation task, and a continuous visual-based sorting object task.
As a way to explain the goal-driven robot's actions, we use the probability of
success computed by three different proposed approaches: memory-based,
learning-based, and introspection-based. The difference between these
approaches is the amount of memory required to compute or estimate the
probability of success as well as the kind of reinforcement learning
representation where they could be used. In this regard, we use the
memory-based approach as a baseline since it is obtained directly from the
agent's observations. When comparing the learning-based and the
introspection-based approaches to this baseline, both are found to be suitable
alternatives to compute the probability of success, obtaining high levels of
similarity when compared using both the Pearson's correlation and the mean
squared error.

negative:
Robotic systems are ever more capable of automation and fulfilment of complex
tasks, particularly with reliance on recent advances in intelligent systems,
deep learning and artificial intelligence. However, as robots and humans come
closer in their interactions, the matter of interpretability, or explainability
of robot decision-making processes for the human grows in importance. A
successful interaction and collaboration will only take place through mutual
understanding of underlying representations of the environment and the task at
hand. This is currently a challenge in deep learning systems. We present a
hierarchical deep reinforcement learning system, consisting of a low-level
agent handling the large actions/states space of a robotic system efficiently,
by following the directives of a high-level agent which is learning the
high-level dynamics of the environment and task. This high-level agent forms a
representation of the world and task at hand that is interpretable for a human
operator. The method, which we call Dot-to-Dot, is tested on a MuJoCo-based
model of the Fetch Robotics Manipulator, as well as a Shadow Hand, to test its
performance. Results show efficient learning of complex actions/states spaces
by the low-level agent, and an interpretable representation of the task and
decision-making process learned by the high-level agent.

negative:
The capability of a mobile robot to efficiently and safely perform complex
missions is limited by its knowledge of the environment, namely the situation.
Advanced reasoning, decision-making, and execution skills enable an intelligent
agent to act autonomously in unknown environments. Situational Awareness (SA)
is a fundamental capability of humans that has been deeply studied in various
fields, such as psychology, military, aerospace, and education. Nevertheless,
it has yet to be considered in robotics, which has focused on single
compartmentalized concepts such as sensing, spatial perception, sensor fusion,
state estimation, and Simultaneous Localization and Mapping (SLAM). Hence, the
present research aims to connect the broad multidisciplinary existing knowledge
to pave the way for a complete SA system for mobile robotics that we deem
paramount for autonomy. To this aim, we define the principal components to
structure a robotic SA and their area of competence. Accordingly, this paper
investigates each aspect of SA, surveying the state-of-the-art robotics
algorithms that cover them, and discusses their current limitations.
Remarkably, essential aspects of SA are still immature since the current
algorithmic development restricts their performance to only specific
environments. Nevertheless, Artificial Intelligence (AI), particularly Deep
Learning (DL), has brought new methods to bridge the gap that maintains these
fields apart from the deployment to real-world scenarios. Furthermore, an
opportunity has been discovered to interconnect the vastly fragmented space of
robotic comprehension algorithms through the mechanism of Situational Graph
(S-Graph), a generalization of the well-known scene graph. Therefore, we
finally shape our vision for the future of robotic Situational Awareness by
discussing interesting recent research directions.

negative:
Robot understanding of human intentions is essential for fluid human-robot
interaction. Intentions, however, cannot be directly observed and must be
inferred from behaviors. We learn a model of adaptive human behavior
conditioned on the intention as a latent variable. We then embed the human
behavior model into a principled probabilistic decision model, which enables
the robot to (i) explore actively in order to infer human intentions and (ii)
choose actions that maximize its performance. Furthermore, the robot learns
from the demonstrated actions of human experts to further improve exploration.
Preliminary experiments in simulation indicate that our approach, when applied
to autonomous driving, improves the efficiency and safety of driving in common
interactive driving scenarios.

negative:
A pressing question when designing intelligent autonomous systems is how to
integrate the various subsystems concerned with complementary tasks. More
specifically, robotic vision must provide task-relevant information about the
environment and the objects in it to various planning related modules. In most
implementations of the traditional Perception-Cognition-Action paradigm these
tasks are treated as quasi-independent modules that function as black boxes for
each other. It is our view that perception can benefit tremendously from a
tight collaboration with cognition. We present RoboSherlock, a
knowledge-enabled cognitive perception systems for mobile robots performing
human-scale everyday manipulation tasks. In RoboSherlock, perception and
interpretation of realistic scenes is formulated as an unstructured information
management(UIM) problem. The application of the UIM principle supports the
implementation of perception systems that can answer task-relevant queries
about objects in a scene, boost object recognition performance by combining the
strengths of multiple perception algorithms, support knowledge-enabled
reasoning about objects and enable automatic and knowledge-driven generation of
processing pipelines. We demonstrate the potential of the proposed framework
through feasibility studies of systems for real-world scene perception that
have been built on top of the framework.

negative:
Imitation Learning (IL) is a powerful paradigm to teach robots to perform
manipulation tasks by allowing them to learn from human demonstrations
collected via teleoperation, but has mostly been limited to single-arm
manipulation. However, many real-world tasks require multiple arms, such as
lifting a heavy object or assembling a desk. Unfortunately, applying IL to
multi-arm manipulation tasks has been challenging -- asking a human to control
more than one robotic arm can impose significant cognitive burden and is often
only possible for a maximum of two robot arms. To address these challenges, we
present Multi-Arm RoboTurk (MART), a multi-user data collection platform that
allows multiple remote users to simultaneously teleoperate a set of robotic
arms and collect demonstrations for multi-arm tasks. Using MART, we collected
demonstrations for five novel two and three-arm tasks from several
geographically separated users. From our data we arrived at a critical insight:
most multi-arm tasks do not require global coordination throughout its full
duration, but only during specific moments. We show that learning from such
data consequently presents challenges for centralized agents that directly
attempt to model all robot actions simultaneously, and perform a comprehensive
study of different policy architectures with varying levels of centralization
on our tasks. Finally, we propose and evaluate a base-residual policy framework
that allows trained policies to better adapt to the mixed coordination setting
common in multi-arm manipulation, and show that a centralized policy augmented
with a decentralized residual model outperforms all other models on our set of
benchmark tasks. Additional results and videos at
https://roboturk.stanford.edu/multiarm .

negative:
In order for cooperative robots ("co-robots") to respond to human behaviors
accurately and efficiently in human-robot collaboration, interpretation of
human actions, awareness of new situations, and appropriate decision making are
all crucial abilities for co-robots. For this purpose, the human behaviors
should be interpreted by co-robots in the same manner as human peers. To
address this issue, a novel interpretability indicator is introduced so that
robot actions are appropriate to the current human behaviors. In addition, the
complete consideration of all potential situations of a robot's environment is
nearly impossible in real-world applications, making it difficult for the
co-robot to act appropriately and safely in new scenarios. This is true even
when the pretrained model is highly accurate in a known situation. For
effective and safe teaming with humans, we introduce a new generalizability
indicator that allows a co-robot to self-reflect and reason about when an
observation falls outside the co-robot's learned model. Based on topic modeling
and two novel indicators, we propose a new Self-reflective Risk-aware
Artificial Cognitive (SRAC) model. The co-robots are able to consider action
risks and identify new situations so that better decisions can be made.
Experiments both using real-world datasets and on physical robots suggest that
our SRAC model significantly outperforms the traditional methodology and
enables better decision making in response to human activities.

negative:
In this study, we investigate how a robot can generate novel and creative
actions from its own experience of learning basic actions. Inspired by a
machine learning approach to computational creativity, we propose a dynamic
neural network model that can learn and generate robot's actions. We conducted
a set of simulation experiments with a humanoid robot. The results showed that
the proposed model was able to learn the basic actions and also to generate
novel actions by modulating and combining those learned actions. The analysis
on the neural activities illustrated that the ability to generate creative
actions emerged from the model's nonlinear memory structure self-organized
during training. The results also showed that the different way of learning the
basic actions induced the self-organization of the memory structure with the
different characteristics, resulting in the generation of different levels of
creative actions. Our approach can be utilized in human-robot interaction in
which a user can interactively explore the robot's memory to control its
behavior and also discover other novel actions.

negative:
Reinforcement learning (RL) can be used to create a tactical decision-making
agent for autonomous driving. However, previous approaches only output
decisions and do not provide information about the agent's confidence in the
recommended actions. This paper investigates how a Bayesian RL technique, based
on an ensemble of neural networks with additional randomized prior functions
(RPF), can be used to estimate the uncertainty of decisions in autonomous
driving. A method for classifying whether or not an action should be considered
safe is also introduced. The performance of the ensemble RPF method is
evaluated by training an agent on a highway driving scenario. It is shown that
the trained agent can estimate the uncertainty of its decisions and indicate an
unacceptable level when the agent faces a situation that is far from the
training distribution. Furthermore, within the training distribution, the
ensemble RPF agent outperforms a standard Deep Q-Network agent. In this study,
the estimated uncertainty is used to choose safe actions in unknown situations.
However, the uncertainty information could also be used to identify situations
that should be added to the training process.

negative:
Legged robots have significant potential to operate in highly unstructured
environments. The design of locomotion control is, however, still challenging.
Currently, controllers must be either manually designed for specific robots and
tasks, or automatically designed via machine learning methods that require long
training times and yield large opaque controllers. Drawing inspiration from
animal locomotion, we propose a simple yet versatile modular neural control
structure with fast learning. The key advantages of our approach are that
behavior-specific control modules can be added incrementally to obtain
increasingly complex emergent locomotion behaviors, and that neural connections
interfacing with existing modules can be quickly and automatically learned. In
a series of experiments, we show how eight modules can be quickly learned and
added to a base control module to obtain emergent adaptive behaviors allowing a
hexapod robot to navigate in complex environments. We also show that modules
can be added and removed during operation without affecting the functionality
of the remaining controller. Finally, the control approach was successfully
demonstrated on a physical hexapod robot. Taken together, our study reveals a
significant step towards fast automatic design of versatile neural locomotion
control for complex robotic systems.

negative:
This paper presents a distributed algorithm applicable to a wide range of
practical multi-robot applications. In such multi-robot applications, the
user-defined objectives of the mission can be cast as a general optimization
problem, without explicit guidelines of the subtasks per different robot. Owing
to the unknown environment, unknown robot dynamics, sensor nonlinearities,
etc., the analytic form of the optimization cost function is not available a
priori. Therefore, standard gradient-descent-like algorithms are not applicable
to these problems. To tackle this, we introduce a new algorithm that carefully
designs each robot's subcost function, the optimization of which can accomplish
the overall team objective. Upon this transformation, we propose a distributed
methodology based on the cognitive-based adaptive optimization (CAO) algorithm,
that is able to approximate the evolution of each robot's cost function and to
adequately optimize its decision variables (robot actions). The latter can be
achieved by online learning only the problem-specific characteristics that
affect the accomplishment of mission objectives. The overall, low-complexity
algorithm can straightforwardly incorporate any kind of operational constraint,
is fault-tolerant, and can appropriately tackle time-varying cost functions. A
cornerstone of this approach is that it shares the same convergence
characteristics as those of block coordinate descent algorithms. The proposed
algorithm is evaluated in three heterogeneous simulation set-ups under multiple
scenarios, against both general-purpose and problem-specific algorithms. Source
code is available at
https://github.com/athakapo/A-distributed-plug-n-play-algorithm-for-multi-robot-applications.

negative:
Human-robot interaction often occurs in the form of instructions given from a
human to a robot. For a robot to successfully follow instructions, a common
representation of the world and objects in it should be shared between humans
and the robot so that the instructions can be grounded. Achieving this
representation can be done via learning, where both the world representation
and the language grounding are learned simultaneously. However, in robotics
this can be a difficult task due to the cost and scarcity of data. In this
paper, we tackle the problem by separately learning the world representation of
the robot and the language grounding. While this approach can address the
challenges in getting sufficient data, it may give rise to inconsistencies
between both learned components. Therefore, we further propose Bayesian
learning to resolve such inconsistencies between the natural language grounding
and a robot's world representation by exploiting spatio-relational information
that is implicitly present in instructions given by a human. Moreover, we
demonstrate the feasibility of our approach on a scenario involving a robotic
arm in the physical world.

negative:
Seamless Human-Robot Interaction is the ultimate goal of developing service
robotic systems. For this, the robotic agents have to understand their
surroundings to better complete a given task. Semantic scene understanding
allows a robotic agent to extract semantic knowledge about the objects in the
environment. In this work, we present a semantic scene understanding pipeline
that fuses 2D and 3D detection branches to generate a semantic map of the
environment. The 2D mask proposals from state-of-the-art 2D detectors are
inverse-projected to the 3D space and combined with 3D detections from point
segmentation networks. Unlike previous works that were evaluated on collected
datasets, we test our pipeline on an active photo-realistic robotic environment
- BenchBot. Our novelty includes rectification of 3D proposals using projected
2D detections and modality fusion based on object size. This work is done as
part of the Robotic Vision Scene Understanding Challenge (RVSU). The
performance evaluation demonstrates that our pipeline has improved on baseline
methods without significant computational bottleneck.

negative:
We propose a developmental approach that allows a robot to interpret and
describe the actions of human agents by reusing previous experience. The robot
first learns the association between words and object affordances by
manipulating the objects in its environment. It then uses this information to
learn a mapping between its own actions and those performed by a human in a
shared environment. It finally fuses the information from these two models to
interpret and describe human actions in light of its own experience. In our
experiments, we show that the model can be used flexibly to do inference on
different aspects of the scene. We can predict the effects of an action on the
basis of object properties. We can revise the belief that a certain action
occurred, given the observed effects of the human action. In an early action
recognition fashion, we can anticipate the effects when the action has only
been partially observed. By estimating the probability of words given the
evidence and feeding them into a pre-defined grammar, we can generate relevant
descriptions of the scene. We believe that this is a step towards providing
robots with the fundamental skills to engage in social collaboration with
humans.

negative:
Despite the success of reinforcement learning methods, they have yet to have
their breakthrough moment when applied to a broad range of robotic manipulation
tasks. This is partly due to the fact that reinforcement learning algorithms
are notoriously difficult and time consuming to train, which is exacerbated
when training from images rather than full-state inputs. As humans perform
manipulation tasks, our eyes closely monitor every step of the process with our
gaze focusing sequentially on the objects being manipulated. With this in mind,
we present our Attention-driven Robotic Manipulation (ARM) algorithm, which is
a general manipulation algorithm that can be applied to a range of
sparse-rewarded tasks, given only a small number of demonstrations. ARM splits
the complex task of manipulation into a 3 stage pipeline: (1) a Q-attention
agent extracts relevant pixel locations from RGB and point cloud inputs, (2) a
next-best pose agent that accepts crops from the Q-attention agent and outputs
poses, and (3) a control agent that takes the goal pose and outputs joint
actions. We show that current learning algorithms fail on a range of RLBench
tasks, whilst ARM is successful.

negative:
The intuitive collaboration of humans and intelligent robots (embodied AI) in
the real-world is an essential objective for many desirable applications of
robotics. Whilst there is much research regarding explicit communication, we
focus on how humans and robots interact implicitly, on motor adaptation level.
We present a real-world setup of a human-robot collaborative maze game,
designed to be non-trivial and only solvable through collaboration, by limiting
the actions to rotations of two orthogonal axes, and assigning each axes to one
player. This results in neither the human nor the agent being able to solve the
game on their own. We use deep reinforcement learning for the control of the
robotic agent, and achieve results within 30 minutes of real-world play,
without any type of pre-training. We then use this setup to perform systematic
experiments on human/agent behaviour and adaptation when co-learning a policy
for the collaborative game. We present results on how co-policy learning occurs
over time between the human and the robotic agent resulting in each
participant's agent serving as a representation of how they would play the
game. This allows us to relate a person's success when playing with different
agents than their own, by comparing the policy of the agent with that of their
own agent.

negative:
Human-robot interaction can be divided into two categories based on the
physical distance between the human and robot: remote and proximal. In proximal
interaction, the human and robot often engage in close coordination; in remote
interaction, the human and robot are less coupled due to communication
constraints. As a result, providing automation for the robot in remote
interaction becomes more important. Thus far, human factor studies on
automation in remote human-robot interaction have been restricted to various
forms of supervision, in which the robot is essentially being used as a smart
mobile manipulation platform with sensing capabilities. In this paper, we
investigate the incorporation of general planning capability into the robot to
facilitate peer-to-peer human-robot teaming, in which the human and robot are
viewed as teammates that are physically separated. The human and robot share
the same global goal and collaborate to achieve it. Note that humans may feel
uncomfortable at such robot autonomy, which can potentially reduce teaming
performance. One important difference between peer-to-peer teaming and
supervised teaming is that an autonomous robot in peer-to-peer teaming can
achieve the goal alone when the task information is completely specified.
However, incompleteness often exists, which implies information asymmetry.
While information asymmetry can be desirable sometimes, it may also lead to the
robot choosing improper actions that negatively influence the teaming
performance. We aim to investigate the various trade-offs, e.g., mental
workload and situation awareness, between these two types of remote human-robot
teaming.

negative:
In this paper we present a Bayesian reinforcement learning framework that
allows robotic manipulators to adaptively recover from random mechanical
failures autonomously, hence being survivable. To this end, we formulate the
framework of Bayesian Policy Morphing (BPM) that enables a robot agent to
self-modify its learned policy after the diminution of its maneuvering
dimensionality. We build upon existing actor-critic framework, and extend it to
perform policy gradient updates as posterior learning, taking past policy
updates as prior distributions. We show that policy search, in the direction
biased by prior experience, significantly improves learning efficiency in terms
of sampling requirements. We demonstrate our results on an 8-DOF robotic arm
with our algorithm of BPM, while intentionally disabling random joints with
different damage types like unresponsive joints, constant offset errors and
angular imprecision. Our results have shown that, even with physical damages,
the robotic arm can still successfully maintain its functionality to accurately
locate and grasp a given target object.

negative:
As robots leave the controlled environments of factories to autonomously
function in more complex, natural environments, they will have to respond to
the inevitable fact that they will become damaged. However, while animals can
quickly adapt to a wide variety of injuries, current robots cannot "think
outside the box" to find a compensatory behavior when damaged: they are limited
to their pre-specified self-sensing abilities, can diagnose only anticipated
failure modes, and require a pre-programmed contingency plan for every type of
potential damage, an impracticality for complex robots. Here we introduce an
intelligent trial and error algorithm that allows robots to adapt to damage in
less than two minutes, without requiring self-diagnosis or pre-specified
contingency plans. Before deployment, a robot exploits a novel algorithm to
create a detailed map of the space of high-performing behaviors: This map
represents the robot's intuitions about what behaviors it can perform and their
value. If the robot is damaged, it uses these intuitions to guide a
trial-and-error learning algorithm that conducts intelligent experiments to
rapidly discover a compensatory behavior that works in spite of the damage.
Experiments reveal successful adaptations for a legged robot injured in five
different ways, including damaged, broken, and missing legs, and for a robotic
arm with joints broken in 14 different ways. This new technique will enable
more robust, effective, autonomous robots, and suggests principles that animals
may use to adapt to injury.

negative:
A common vision from science fiction is that robots will one day inhabit our
physical spaces, sense the world as we do, assist our physical labours, and
communicate with us through natural language. Here we study how to design
artificial agents that can interact naturally with humans using the
simplification of a virtual environment. This setting nevertheless integrates a
number of the central challenges of artificial intelligence (AI) research:
complex visual perception and goal-directed physical control, grounded language
comprehension and production, and multi-agent social interaction. To build
agents that can robustly interact with humans, we would ideally train them
while they interact with humans. However, this is presently impractical.
Therefore, we approximate the role of the human with another learned agent, and
use ideas from inverse reinforcement learning to reduce the disparities between
human-human and agent-agent interactive behaviour. Rigorously evaluating our
agents poses a great challenge, so we develop a variety of behavioural tests,
including evaluation by humans who watch videos of agents or interact directly
with them. These evaluations convincingly demonstrate that interactive training
and auxiliary losses improve agent behaviour beyond what is achieved by
supervised learning of actions alone. Further, we demonstrate that agent
capabilities generalise beyond literal experiences in the dataset. Finally, we
train evaluation models whose ratings of agents agree well with human
judgement, thus permitting the evaluation of new agent models without
additional effort. Taken together, our results in this virtual environment
provide evidence that large-scale human behavioural imitation is a promising
tool to create intelligent, interactive agents, and the challenge of reliably
evaluating such agents is possible to surmount.

negative:
Predictions and predictive knowledge have seen recent success in improving
not only robot control but also other applications ranging from industrial
process control to rehabilitation. A property that makes these predictive
approaches well suited for robotics is that they can be learned online and
incrementally through interaction with the environment. However, a remaining
challenge for many prediction-learning approaches is an appropriate choice of
prediction-learning parameters, especially parameters that control the
magnitude of a learning machine's updates to its predictions (the learning rate
or step size). To begin to address this challenge, we examine the use of online
step-size adaptation using a sensor-rich robotic arm. Our method of choice,
Temporal-Difference Incremental Delta-Bar-Delta (TIDBD), learns and adapts step
sizes on a feature level; importantly, TIDBD allows step-size tuning and
representation learning to occur at the same time. We show that TIDBD is a
practical alternative for classic Temporal-Difference (TD) learning via an
extensive parameter search. Both approaches perform comparably in terms of
predicting future aspects of a robotic data stream. Furthermore, the use of a
step-size adaptation method like TIDBD appears to allow a system to
automatically detect and characterize common sensor failures in a robotic
application. Together, these results promise to improve the ability of robotic
devices to learn from interactions with their environments in a robust way,
providing key capabilities for autonomous agents and robots.

negative:
Human input has enabled autonomous systems to improve their capabilities and
achieve complex behaviors that are otherwise challenging to generate
automatically. Recent work focuses on how robots can use such input - like
demonstrations or corrections - to learn intended objectives. These techniques
assume that the human's desired objective already exists within the robot's
hypothesis space. In reality, this assumption is often inaccurate: there will
always be situations where the person might care about aspects of the task that
the robot does not know about. Without this knowledge, the robot cannot infer
the correct objective. Hence, when the robot's hypothesis space is
misspecified, even methods that keep track of uncertainty over the objective
fail because they reason about which hypothesis might be correct, and not
whether any of the hypotheses are correct. In this paper, we posit that the
robot should reason explicitly about how well it can explain human inputs given
its hypothesis space and use that situational confidence to inform how it
should incorporate human input. We demonstrate our method on a 7
degree-of-freedom robot manipulator in learning from two important types of
human input: demonstrations of manipulation tasks, and physical corrections
during the robot's task execution.

negative:
The increasing presence of robots alongside humans, such as in human-robot
teams in manufacturing, gives rise to research questions about the kind of
behaviors people prefer in their robot counterparts. We term actions that
support interaction by reducing future interference with others as supportive
robot actions and investigate their utility in a co-located manipulation
scenario. We compare two robot modes in a shared table pick-and-place task: (1)
Task-oriented: the robot only takes actions to further its own task objective
and (2) Supportive: the robot sometimes prefers supportive actions to
task-oriented ones when they reduce future goal-conflicts. Our experiments in
simulation, using a simplified human model, reveal that supportive actions
reduce the interference between agents, especially in more difficult tasks, but
also cause the robot to take longer to complete the task. We implemented these
modes on a physical robot in a user study where a human and a robot perform
object placement on a shared table. Our results show that a supportive robot
was perceived as a more favorable coworker by the human and also reduced
interference with the human in the more difficult of two scenarios. However, it
also took longer to complete the task highlighting an interesting trade-off
between task-efficiency and human-preference that needs to be considered before
designing robot behavior for close-proximity manipulation scenarios.

negative:
Autonomy is fundamental for artificial agents acting in complex real-world
scenarios. The acquisition of many different skills is pivotal to foster
versatile autonomous behaviour and thus a main objective for robotics and
machine learning. Intrinsic motivations have proven to properly generate a
task-agnostic signal to drive the autonomous acquisition of multiple policies
in settings requiring the learning of multiple tasks. However, in real world
scenarios tasks may be interdependent so that some of them may constitute the
precondition for learning other ones. Despite different strategies have been
used to tackle the acquisition of interdependent/hierarchical tasks, fully
autonomous open-ended learning in these scenarios is still an open question.
Building on previous research within the framework of intrinsically-motivated
open-ended learning, we propose an architecture for robot control that tackles
this problem from the point of view of decision making, i.e. treating the
selection of tasks as a Markov Decision Process where the system selects the
policies to be trained in order to maximise its competence over all the tasks.
The system is then tested with a humanoid robot solving interdependent multiple
reaching tasks.

negative:
The ability to plan and execute goal specific actions in varied, unexpected
settings is a central requirement of intelligent agents. In this paper, we
explore how an agent can be equipped with an internal model of the dynamics of
the external world, and how it can use this model to plan novel actions by
running multiple internal simulations ("visual imagination"). Our models
directly process raw visual input, and use a novel object-centric prediction
formulation based on visual glimpses centered on objects (fixations) to enforce
translational invariance of the learned physical laws. The agent gathers
training data through random interaction with a collection of different
environments, and the resulting model can then be used to plan goal-directed
actions in novel environments that the agent has not seen before. We
demonstrate that our agent can accurately plan actions for playing a simulated
billiards game, which requires pushing a ball into a target position or into
collision with another ball.

negative:
A major component for developing intelligent and autonomous robots is a
suitable knowledge representation, from which a robot can acquire knowledge
about its actions or world. However, unlike humans, robots cannot creatively
adapt to novel scenarios, as their knowledge and environment are rigidly
defined. To address the problem of producing novel and flexible task plans
called task trees, we explore how we can derive plans with concepts not
originally in the robot's knowledge base. Existing knowledge in the form of a
knowledge graph is used as a base of reference to create task trees that are
modified with new object or state combinations. To demonstrate the flexibility
of our method, we randomly selected recipes from the Recipe1M+ dataset and
generated their task trees. The task trees were then thoroughly checked with a
visualization tool that portrays how each ingredient changes with each action
to produce the desired meal. Our results indicate that the proposed method can
produce task plans with high accuracy even for never-before-seen ingredient
combinations.

negative:
Intelligent behaviour in the physical world exhibits structure at multiple
spatial and temporal scales. Although movements are ultimately executed at the
level of instantaneous muscle tensions or joint torques, they must be selected
to serve goals defined on much longer timescales, and in terms of relations
that extend far beyond the body itself, ultimately involving coordination with
other agents. Recent research in artificial intelligence has shown the promise
of learning-based approaches to the respective problems of complex movement,
longer-term planning and multi-agent coordination. However, there is limited
research aimed at their integration. We study this problem by training teams of
physically simulated humanoid avatars to play football in a realistic virtual
environment. We develop a method that combines imitation learning, single- and
multi-agent reinforcement learning and population-based training, and makes use
of transferable representations of behaviour for decision making at different
levels of abstraction. In a sequence of stages, players first learn to control
a fully articulated body to perform realistic, human-like movements such as
running and turning; they then acquire mid-level football skills such as
dribbling and shooting; finally, they develop awareness of others and play as a
team, bridging the gap between low-level motor control at a timescale of
milliseconds, and coordinated goal-directed behaviour as a team at the
timescale of tens of seconds. We investigate the emergence of behaviours at
different levels of abstraction, as well as the representations that underlie
these behaviours using several analysis techniques, including statistics from
real-world sports analytics. Our work constitutes a complete demonstration of
integrated decision-making at multiple scales in a physically embodied
multi-agent setting. See project video at https://youtu.be/KHMwq9pv7mg.

negative:
This paper contributes a preliminary report on the advantages and
disadvantages of incorporating simultaneous human control and feedback signals
in the training of a reinforcement learning robotic agent. While robotic
human-machine interfaces have become increasingly complex in both form and
function, control remains challenging for users. This has resulted in an
increasing gap between user control approaches and the number of robotic motors
which can be controlled. One way to address this gap is to shift some autonomy
to the robot. Semi-autonomous actions of the robotic agent can then be shaped
by human feedback, simplifying user control. Most prior work on agent shaping
by humans has incorporated training with feedback, or has included indirect
control signals. By contrast, in this paper we explore how a human can provide
concurrent feedback signals and real-time myoelectric control signals to train
a robot's actor-critic reinforcement learning control system. Using both a
physical and a simulated robotic system, we compare training performance on a
simple movement task when reward is derived from the environment, when reward
is provided by the human, and combinations of these two approaches. Our results
indicate that some benefit can be gained with the inclusion of human generated
feedback.

negative:
Designing robots capable of generating interpretable behavior is a
prerequisite for achieving effective human-robot collaboration. This means that
the robots need to be capable of generating behavior that aligns with human
expectations and, when required, provide explanations to the humans in the
loop. However, exhibiting such behavior in arbitrary environments could be
quite expensive for robots, and in some cases, the robot may not even be able
to exhibit the expected behavior. Given structured environments (like
warehouses and restaurants), it may be possible to design the environment so as
to boost the interpretability of the robot's behavior or to shape the human's
expectations of the robot's behavior. In this paper, we investigate the
opportunities and limitations of environment design as a tool to promote a type
of interpretable behavior -- known in the literature as explicable behavior. We
formulate a novel environment design framework that considers design over
multiple tasks and over a time horizon. In addition, we explore the
longitudinal aspect of explicable behavior and the trade-off that arises
between the cost of design and the cost of generating explicable behavior over
a time horizon.

negative:
Nowadays, model-free reinforcement learning algorithms have achieved
remarkable performance on many decision making and control tasks, but high
sample complexity and low sample efficiency still hinder the wide use of
model-free reinforcement learning algorithms. In this paper, we argue that if
we intend to design an intelligent agent that learns fast and transfers well,
the agent must be able to reflect key elements of intelligence, like intuition,
Memory, PredictionandCuriosity. We propose an agent framework that integrates
off-policy reinforcement learning with world model learning, so as to embody
the important features of intelligence in our algorithm design. We adopt the
state-of-art model-free reinforcement learning algorithm, Soft Actor-Critic, as
the agent intuition, and world model learning through RNN to endow the agent
with memory, curiosity, and the ability to predict. We show that these ideas
can work collaboratively with each other and our agent (RMC) can give new
state-of-art results while maintaining sample efficiency and training
stability. Moreover, our agent framework can be easily extended from MDP to
POMDP problems without performance loss.

negative:
We design a simple reinforcement learning (RL) agent that implements an
optimistic version of $Q$-learning and establish through regret analysis that
this agent can operate with some level of competence in any environment. While
we leverage concepts from the literature on provably efficient RL, we consider
a general agent-environment interface and provide a novel agent design and
analysis. This level of generality positions our results to inform the design
of future agents for operation in complex real environments. We establish that,
as time progresses, our agent performs competitively relative to policies that
require longer times to evaluate. The time it takes to approach asymptotic
performance is polynomial in the complexity of the agent's state representation
and the time required to evaluate the best policy that the agent can represent.
Notably, there is no dependence on the complexity of the environment. The
ultimate per-period performance loss of the agent is bounded by a constant
multiple of a measure of distortion introduced by the agent's state
representation. This work is the first to establish that an algorithm
approaches this asymptotic condition within a tractable time frame.

negative:
A practical approach to robot reinforcement learning is to first collect a
large batch of real or simulated robot interaction data, using some data
collection policy, and then learn from this data to perform various tasks,
using offline learning algorithms. Previous work focused on manually designing
the data collection policy, and on tasks where suitable policies can easily be
designed, such as random picking policies for collecting data about object
grasping. For more complex tasks, however, it may be difficult to find a data
collection policy that explores the environment effectively, and produces data
that is diverse enough for the downstream task. In this work, we propose that
data collection policies should actively explore the environment to collect
diverse data. In particular, we develop a simple-yet-effective goal-conditioned
reinforcement-learning method that actively focuses data collection on novel
observations, thereby collecting a diverse data-set. We evaluate our method on
simulated robot manipulation tasks with visual inputs and show that the
improved diversity of active data collection leads to significant improvements
in the downstream learning tasks.

negative:
Well structured visual representations can make robot learning faster and can
improve generalization. In this paper, we study how we can acquire effective
object-centric representations for robotic manipulation tasks without human
labeling by using autonomous robot interaction with the environment. Such
representation learning methods can benefit from continuous refinement of the
representation as the robot collects more experience, allowing them to scale
effectively without human intervention. Our representation learning approach is
based on object persistence: when a robot removes an object from a scene, the
representation of that scene should change according to the features of the
object that was removed. We formulate an arithmetic relationship between
feature vectors from this observation, and use it to learn a representation of
scenes and objects that can then be used to identify object instances, localize
them in the scene, and perform goal-directed grasping tasks where the robot
must retrieve commanded objects from a bin. The same grasping procedure can
also be used to automatically collect training data for our method, by
recording images of scenes, grasping and removing an object, and recording the
outcome. Our experiments demonstrate that this self-supervised approach for
tasked grasping substantially outperforms direct reinforcement learning from
images and prior representation learning methods.

negative:
Telepresence robots are used in various forms in various use-cases that helps
to avoid physical human presence at the scene of action. In this work, we focus
on a telepresence robot that can be used to attend a meeting remotely with a
group of people. Unlike a one-to-one meeting, participants in a group meeting
can be located at a different part of the room, especially in an informal
setup. As a result, all of them may not be at the viewing angle of the robot,
a.k.a. the remote participant. In such a case, to provide a better meeting
experience, the robot should localize the speaker and bring the speaker at the
center of the viewing angle. Though sound source localization can easily be
done using a microphone-array, bringing the speaker or set of speakers at the
viewing angle is not a trivial task. First of all, the robot should react only
to a human voice, but not to the random noises. Secondly, if there are multiple
speakers, to whom the robot should face or should it rotate continuously with
every new speaker? Lastly, most robotic platforms are resource-constrained and
to achieve a real-time response, i.e., avoiding network delay, all the
algorithms should be implemented within the robot itself. This article presents
a study and implementation of an attention shifting scheme in a telepresence
meeting scenario which best suits the needs and expectations of the collocated
and remote attendees. We define a policy to decide when a robot should rotate
and how much based on real-time speaker localization. Using user satisfaction
study, we show the efficacy and usability of our system in the meeting
scenario. Moreover, our system can be easily adapted to other scenarios where
multiple people are located.

negative:
Humans make decisions and act alongside other humans to pursue both
short-term and long-term goals. As a result of ongoing progress in areas such
as computing science and automation, humans now also interact with non-human
agents of varying complexity as part of their day-to-day activities;
substantial work is being done to integrate increasingly intelligent machine
agents into human work and play. With increases in the cognitive, sensory, and
motor capacity of these agents, intelligent machinery for human assistance can
now reasonably be considered to engage in joint action with humans---i.e., two
or more agents adapting their behaviour and their understanding of each other
so as to progress in shared objectives or goals. The mechanisms, conditions,
and opportunities for skillful joint action in human-machine partnerships is of
great interest to multiple communities. Despite this, human-machine joint
action is as yet under-explored, especially in cases where a human and an
intelligent machine interact in a persistent way during the course of
real-time, daily-life experience. In this work, we contribute a virtual reality
environment wherein a human and an agent can adapt their predictions, their
actions, and their communication so as to pursue a simple foraging task. In a
case study with a single participant, we provide an example of human-agent
coordination and decision-making involving prediction learning on the part of
the human and the machine agent, and control learning on the part of the
machine agent wherein audio communication signals are used to cue its human
partner in service of acquiring shared reward. These comparisons suggest the
utility of studying human-machine coordination in a virtual reality
environment, and identify further research that will expand our understanding
of persistent human-machine joint action.

negative:
A common vision from science fiction is that robots will one day inhabit our
physical spaces, sense the world as we do, assist our physical labours, and
communicate with us through natural language. Here we study how to design
artificial agents that can interact naturally with humans using the
simplification of a virtual environment. We show that imitation learning of
human-human interactions in a simulated world, in conjunction with
self-supervised learning, is sufficient to produce a multimodal interactive
agent, which we call MIA, that successfully interacts with non-adversarial
humans 75% of the time. We further identify architectural and algorithmic
techniques that improve performance, such as hierarchical action selection.
Altogether, our results demonstrate that imitation of multi-modal, real-time
human behaviour may provide a straightforward and surprisingly effective means
of imbuing agents with a rich behavioural prior from which agents might then be
fine-tuned for specific purposes, thus laying a foundation for training capable
agents for interactive robots or digital assistants. A video of MIA's behaviour
may be found at https://youtu.be/ZFgRhviF7mY

negative:
One of the long-term challenges of robotics is to enable robots to interact
with humans in the visual world via natural language, as humans are visual
animals that communicate through language. Overcoming this challenge requires
the ability to perform a wide variety of complex tasks in response to
multifarious instructions from humans. In the hope that it might drive progress
towards more flexible and powerful human interactions with robots, we propose a
dataset of varied and complex robot tasks, described in natural language, in
terms of objects visible in a large set of real images. Given an instruction,
success requires navigating through a previously-unseen environment to identify
an object. This represents a practical challenge, but one that closely reflects
one of the core visual problems in robotics. Several state-of-the-art
vision-and-language navigation, and referring-expression models are tested to
verify the difficulty of this new task, but none of them show promising results
because there are many fundamental differences between our task and previous
ones. A novel Interactive Navigator-Pointer model is also proposed that
provides a strong baseline on the task. The proposed model especially achieves
the best performance on the unseen test split, but still leaves substantial
room for improvement compared to the human performance.

negative:
The representation of the knowledge needed by a robot to perform complex
tasks is restricted by the limitations of perception. One possible way of
overcoming this situation and designing "knowledgeable" robots is to rely on
the interaction with the user. We propose a multi-modal interaction framework
that allows to effectively acquire knowledge about the environment where the
robot operates. In particular, in this paper we present a rich representation
framework that can be automatically built from the metric map annotated with
the indications provided by the user. Such a representation, allows then the
robot to ground complex referential expressions for motion commands and to
devise topological navigation plans to achieve the target locations.

negative:
In this paper, we propose an AI-FML robotic agent for student learning
behavior ontology construction which can be applied in English speaking and
listening domain. The AI-FML robotic agent with the ontology contains the
perception intelligence, computational intelligence, and cognition intelligence
for analyzing student learning behavior. In addition, there are three
intelligent agents, including a perception agent, a computational agent, and a
cognition agent in the AI-FML robotic agent. We deploy the perception agent and
the cognition agent on the robot Kebbi Air. Moreover, the computational agent
with the Deep Neural Network (DNN) model is performed in the cloud and can
communicate with the perception agent and cognition agent via the Internet. The
proposed AI-FML robotic agent is applied in Taiwan and tested in Japan. The
experimental results show that the agents can be utilized in the human and
machine co-learning model for the future education.

negative:
When deployed, AI agents will encounter problems that are beyond their
autonomous problem-solving capabilities. Leveraging human assistance can help
agents overcome their inherent limitations and robustly cope with unfamiliar
situations. We present a general interactive framework that enables an agent to
request and interpret rich, contextually useful information from an assistant
that has knowledge about the task and the environment. We demonstrate the
practicality of our framework on a simulated human-assisted navigation problem.
Aided with an assistance-requesting policy learned by our method, a navigation
agent achieves up to a 7x improvement in success rate on tasks that take place
in previously unseen environments, compared to fully autonomous behavior. We
show that the agent can take advantage of different types of information
depending on the context, and analyze the benefits and challenges of learning
the assistance-requesting policy when the assistant can recursively decompose
tasks into subtasks.

negative:
Seamlessly interacting with humans or robots is hard because these agents are
non-stationary. They update their policy in response to the ego agent's
behavior, and the ego agent must anticipate these changes to co-adapt. Inspired
by humans, we recognize that robots do not need to explicitly model every
low-level action another agent will make; instead, we can capture the latent
strategy of other agents through high-level representations. We propose a
reinforcement learning-based framework for learning latent representations of
an agent's policy, where the ego agent identifies the relationship between its
behavior and the other agent's future strategy. The ego agent then leverages
these latent dynamics to influence the other agent, purposely guiding them
towards policies suitable for co-adaptation. Across several simulated domains
and a real-world air hockey game, our approach outperforms the alternatives and
learns to influence the other agent.

negative:
Robot navigation in human semi-static and crowded environments can lead to
the freezing problem, where the robot can not move due to the presence of
humans standing on its path and no other path is available. Classical
approaches of robot navigation do not provide a solution for this problem. In
such situations, the robot could interact with the humans in order to clear its
path instead of considering them as unanimated obstacles. In this work, we
propose a robot behavior for a wheeled humanoid robot that complains with
social norms for clearing its path when the robot is frozen due to the presence
of humans. The behavior consists of two modules: 1) A detection module, which
make use of the Yolo v3 algorithm trained to detect human hands and human arms.
2) A gesture module, which make use of a policy trained in simulation using the
Proximal Policy Optimization algorithm. Orchestration of the two models is done
using the ROS framework.

negative:
The future will be replete with scenarios where humans are robots will be
working together in complex environments. Teammates interact, and the robot's
interaction has to be about getting useful information about the human's
(teammate's) model. There are many challenges before a robot can interact, such
as incorporating the structural differences in the human's model, ensuring
simpler responses, etc. In this paper, we investigate how a robot can interact
to localize the human model from a set of models. We show how to generate
questions to refine the robot's understanding of the teammate's model. We
evaluate the method in various planning domains. The evaluation shows that
these questions can be generated offline, and can help refine the model through
simple answers.

negative:
As environments involving both robots and humans become increasingly common,
so does the need to account for people during planning. To plan effectively,
robots must be able to respond to and sometimes influence what humans do. This
requires a human model which predicts future human actions. A simple model may
assume the human will continue what they did previously; a more complex one
might predict that the human will act optimally, disregarding the robot;
whereas an even more complex one might capture the robot's ability to influence
the human. These models make different trade-offs between computational time
and performance of the resulting robot plan. Using only one model of the human
either wastes computational resources or is unable to handle critical
situations. In this work, we give the robot access to a suite of human models
and enable it to assess the performance-computation trade-off online. By
estimating how an alternate model could improve human prediction and how that
may translate to performance gain, the robot can dynamically switch human
models whenever the additional computation is justified. Our experiments in a
driving simulator showcase how the robot can achieve performance comparable to
always using the best human model, but with greatly reduced computation.

negative:
When deploying autonomous agents in the real world, we need effective ways of
communicating objectives to them. Traditional skill learning has revolved
around reinforcement and imitation learning, each with rigid constraints on the
format of information exchanged between the human and the agent. While scalar
rewards carry little information, demonstrations require significant effort to
provide and may carry more information than is necessary. Furthermore, rewards
and demonstrations are often defined and collected before training begins, when
the human is most uncertain about what information would help the agent. In
contrast, when humans communicate objectives with each other, they make use of
a large vocabulary of informative behaviors, including non-verbal
communication, and often communicate throughout learning, responding to
observed behavior. In this way, humans communicate intent with minimal effort.
In this paper, we propose such interactive learning as an alternative to reward
or demonstration-driven learning. To accomplish this, we introduce a
multi-agent training framework that enables an agent to learn from another
agent who knows the current task. Through a series of experiments, we
demonstrate the emergence of a variety of interactive learning behaviors,
including information-sharing, information-seeking, and question-answering.
Most importantly, we find that our approach produces an agent that is capable
of learning interactively from a human user, without a set of explicit
demonstrations or a reward function, and achieving significantly better
performance cooperatively with a human than a human performing the task alone.

negative:
As autonomous service robots become more affordable and thus available also
for the general public, there is a growing need for user friendly interfaces to
control the robotic system. Currently available control modalities typically
expect users to be able to express their desire through either touch, speech or
gesture commands. While this requirement is fulfilled for the majority of
users, paralyzed users may not be able to use such systems. In this paper, we
present a novel framework, that allows these users to interact with a robotic
service assistant in a closed-loop fashion, using only thoughts. The
brain-computer interface (BCI) system is composed of several interacting
components, i.e., non-invasive neuronal signal recording and decoding,
high-level task planning, motion and manipulation planning as well as
environment perception. In various experiments, we demonstrate its
applicability and robustness in real world scenarios, considering
fetch-and-carry tasks and tasks involving human-robot interaction. As our
results demonstrate, our system is capable of adapting to frequent changes in
the environment and reliably completing given tasks within a reasonable amount
of time. Combined with high-level planning and autonomous robotic systems,
interesting new perspectives open up for non-invasive BCI-based human-robot
interactions.

negative:
Reinforcement learning is a promising method to accomplish robotic control
tasks. The task of playing musical instruments is, however, largely unexplored
because it involves the challenge of achieving sequential goals - melodies -
that have a temporal dimension. In this paper, we address robotic musicianship
by introducing a temporal extension to goal-conditioned reinforcement learning:
Time-dependent goals. We demonstrate that these can be used to train a robotic
musician to play the theremin instrument. We train the robotic agent in
simulation and transfer the acquired policy to a real-world robotic
thereminist. Supplemental video: https://youtu.be/jvC9mPzdQN4

negative:
Reinforcement learning is a powerful technique to train an agent to perform a
task. However, an agent that is trained using reinforcement learning is only
capable of achieving the single task that is specified via its reward function.
Such an approach does not scale well to settings in which an agent needs to
perform a diverse set of tasks, such as navigating to varying positions in a
room or moving objects to varying locations. Instead, we propose a method that
allows an agent to automatically discover the range of tasks that it is capable
of performing. We use a generator network to propose tasks for the agent to try
to achieve, specified as goal states. The generator network is optimized using
adversarial training to produce tasks that are always at the appropriate level
of difficulty for the agent. Our method thus automatically produces a
curriculum of tasks for the agent to learn. We show that, by using this
framework, an agent can efficiently and automatically learn to perform a wide
set of tasks without requiring any prior knowledge of its environment. Our
method can also learn to achieve tasks with sparse rewards, which traditionally
pose significant challenges.

negative:
As reinforcement learning agents become increasingly integrated into complex,
real-world environments, designing for safety becomes a critical consideration.
We specifically focus on researching scenarios where agents can cause undesired
side effects while executing a policy on a primary task. Since one can define
multiple tasks for a given environment dynamics, there are two important
challenges. First, we need to abstract the concept of safety that applies
broadly to that environment independent of the specific task being executed.
Second, we need a mechanism for the abstracted notion of safety to modulate the
actions of agents executing different policies to minimize their side-effects.
In this work, we propose Safety Aware Reinforcement Learning (SARL) - a
framework where a virtual safe agent modulates the actions of a main
reward-based agent to minimize side effects. The safe agent learns a
task-independent notion of safety for a given environment. The main agent is
then trained with a regularization loss given by the distance between the
native action probabilities of the two agents. Since the safe agent effectively
abstracts a task-independent notion of safety via its action probabilities, it
can be ported to modulate multiple policies solving different tasks within the
given environment without further training. We contrast this with solutions
that rely on task-specific regularization metrics and test our framework on the
SafeLife Suite, based on Conway's Game of Life, comprising a number of complex
tasks in dynamic environments. We show that our solution is able to match the
performance of solutions that rely on task-specific side-effect penalties on
both the primary and safety objectives while additionally providing the benefit
of generalizability and portability.

negative:
A core aspect of human intelligence is the ability to learn new tasks quickly
and switch between them flexibly. Here, we describe a modular continual
reinforcement learning paradigm inspired by these abilities. We first introduce
a visual interaction environment that allows many types of tasks to be unified
in a single framework. We then describe a reward map prediction scheme that
learns new tasks robustly in the very large state and action spaces required by
such an environment. We investigate how properties of module architecture
influence efficiency of task learning, showing that a module motif
incorporating specific design principles (e.g. early bottlenecks, low-order
polynomial nonlinearities, and symmetry) significantly outperforms more
standard neural network motifs, needing fewer training examples and fewer
neurons to achieve high levels of performance. Finally, we present a
meta-controller architecture for task switching based on a dynamic neural
voting scheme, which allows new modules to use information learned from
previously-seen tasks to substantially improve their own learning efficiency.

negative:
Among the many anticipated roles for robots in the future is that of being a
human teammate. Aside from all the technological hurdles that have to be
overcome with respect to hardware and control to make robots fit to work with
humans, the added complication here is that humans have many conscious and
subconscious expectations of their teammates - indeed, we argue that teaming is
mostly a cognitive rather than physical coordination activity. This introduces
new challenges for the AI and robotics community and requires fundamental
changes to the traditional approach to the design of autonomy. With this in
mind, we propose an update to the classical view of the intelligent agent
architecture, highlighting the requirements for mental modeling of the human in
the deliberative process of the autonomous agent. In this article, we outline
briefly the recent efforts of ours, and others in the community, towards
developing cognitive teammates along these guidelines.

negative:
Various automatic curriculum learning (ACL) methods have been proposed to
improve the sample efficiency and final performance of deep reinforcement
learning (DRL). They are designed to control how a DRL agent collects data,
which is inspired by how humans gradually adapt their learning processes to
their capabilities. For example, ACL can be used for subgoal generation, reward
shaping, environment generation, or initial state generation. However, prior
work only considers curriculum learning following one of the aforementioned
predefined paradigms. It is unclear which of these paradigms are complementary,
and how the combination of them can be learned from interactions with the
environment. Therefore, in this paper, we propose a unified automatic
curriculum learning framework to create multi-objective but coherent curricula
that are generated by a set of parametric curriculum modules. Each curriculum
module is instantiated as a neural network and is responsible for generating a
particular curriculum. In order to coordinate those potentially conflicting
modules in unified parameter space, we propose a multi-task hyper-net learning
framework that uses a single hyper-net to parameterize all those curriculum
modules. In addition to existing hand-designed curricula paradigms, we further
design a flexible memory mechanism to learn an abstract curriculum, which may
otherwise be difficult to design manually. We evaluate our method on a series
of robotic manipulation tasks and demonstrate its superiority over other
state-of-the-art ACL methods in terms of sample efficiency and final
performance.

negative:
Many works in collaborative robotics and human-robot interaction focuses on
identifying and predicting human behaviour while considering the information
about the robot itself as given. This can be the case when sensors and the
robot are calibrated in relation to each other and often the reconfiguration of
the system is not possible, or extra manual work is required. We present a deep
learning based approach to remove the constraint of having the need for the
robot and the vision sensor to be fixed and calibrated in relation to each
other. The system learns the visual cues of the robot body and is able to
localise it, as well as estimate the position of robot joints in 3D space by
just using a 2D color image. The method uses a cascaded convolutional neural
network, and we present the structure of the network, describe our own
collected dataset, explain the network training and achieved results. A fully
trained system shows promising results in providing an accurate mask of where
the robot is located and a good estimate of its joints positions in 3D. The
accuracy is not good enough for visual servoing applications yet, however, it
can be sufficient for general safety and some collaborative tasks not requiring
very high precision. The main benefit of our method is the possibility of the
vision sensor to move freely. This allows it to be mounted on moving objects,
for example, a body of the person or a mobile robot working in the same
environment as the robots are operating in.

negative:
Observational learning is a type of learning that occurs as a function of
observing, retaining and possibly replicating or imitating the behaviour of
another agent. It is a core mechanism appearing in various instances of social
learning and has been found to be employed in several intelligent species,
including humans. In this paper, we investigate to what extent the explicit
modelling of other agents is necessary to achieve observational learning
through machine learning. Especially, we argue that observational learning can
emerge from pure Reinforcement Learning (RL), potentially coupled with memory.
Through simple scenarios, we demonstrate that an RL agent can leverage the
information provided by the observations of an other agent performing a task in
a shared environment. The other agent is only observed through the effect of
its actions on the environment and never explicitly modeled. Two key aspects
are borrowed from observational learning: i) the observer behaviour needs to
change as a result of viewing a 'teacher' (another agent) and ii) the observer
needs to be motivated somehow to engage in making use of the other agent's
behaviour. The later is naturally modeled by RL, by correlating the learning
agent's reward with the teacher agent's behaviour.

negative:
In outdoor environments, mobile robots are required to navigate through
terrain with varying characteristics, some of which might significantly affect
the integrity of the platform. Ideally, the robot should be able to identify
areas that are safe for navigation based on its own percepts about the
environment while avoiding damage to itself. Bayesian optimisation (BO) has
been successfully applied to the task of learning a model of terrain
traversability while guiding the robot through more traversable areas. An
issue, however, is that localisation uncertainty can end up guiding the robot
to unsafe areas and distort the model being learnt. In this paper, we address
this problem and present a novel method that allows BO to consider localisation
uncertainty by applying a Gaussian process model for uncertain inputs as a
prior. We evaluate the proposed method in simulation and in experiments with a
real robot navigating over rough terrain and compare it against standard BO
methods.

negative:
Imitation learning is an effective approach for autonomous systems to acquire
control policies when an explicit reward function is unavailable, using
supervision provided as demonstrations from an expert, typically a human
operator. However, standard imitation learning methods assume that the agent
receives examples of observation-action tuples that could be provided, for
instance, to a supervised learning algorithm. This stands in contrast to how
humans and animals imitate: we observe another person performing some behavior
and then figure out which actions will realize that behavior, compensating for
changes in viewpoint, surroundings, object positions and types, and other
factors. We term this kind of imitation learning "imitation-from-observation,"
and propose an imitation learning method based on video prediction with context
translation and deep reinforcement learning. This lifts the assumption in
imitation learning that the demonstration should consist of observations in the
same environment configuration, and enables a variety of interesting
applications, including learning robotic skills that involve tool use simply by
observing videos of human tool use. Our experimental results show the
effectiveness of our approach in learning a wide range of real-world robotic
tasks modeled after common household chores from videos of a human
demonstrator, including sweeping, ladling almonds, pushing objects as well as a
number of tasks in simulation.

negative:
Communication is not only an action of choosing a signal, but needs to
consider the context and sensor signals. It also needs to decide what
information is communicated and how it is represented in or understood from
signals. Therefore, communication should be realized comprehensively together
with its purpose and other functions.
  The recent successful results in end-to-end reinforcement learning (RL) show
the importance of comprehensive learning and the usefulness of end-to-end RL.
Although little is known, we have shown that a variety of communications emerge
through RL using a (recurrent) neural network (NN). Here, three of them are
introduced.
  In the 1st one, negotiation to avoid conflicts among 4 randomly-picked agents
was learned. Each agent generates a binary signal from the output of its
recurrent NN (RNN), and receives 4 signals from the agents three times. After
learning, each agent made an appropriate final decision after negotiation for
any combination of 4 agents. Differentiation of individuality among the agents
also could be seen.
  The 2nd one focused on discretization of communication signal. A sender agent
perceives the receiver's location and generates a continuous signal twice by
its RNN. A receiver agent receives them sequentially, and moves according to
its RNN's output to reach the sender's location. When noises were added to the
signal, it was binarized through learning and 2-bit communication was
established.
  The 3rd one focused on end-to-end comprehensive communication. A sender
receives 1,785-pixel real camera image on which a real robot can be seen, and
sends two sounds whose frequencies are computed by its NN. A receiver receives
them, and two motion commands for the robot are generated by its NN. After
learning, though some preliminary learning was necessary for the sender, the
robot could reach the goal from any initial location.

negative:
It is highly desirable for robots that work alongside humans to be able to
understand instructions in natural language. Existing language conditioned
imitation learning models directly predict the actuator commands from the image
observation and the instruction text. Rather than directly predicting actuator
commands, we propose translating the natural language instruction to a Python
function which queries the scene by accessing the output of the object detector
and controls the robot to perform the specified task. This enables the use of
non-differentiable modules such as a constraint solver when computing commands
to the robot. Moreover, the labels in this setup are significantly more
informative computer programs that capture the intent of the expert rather than
teleoperated demonstrations. We show that the proposed method performs better
than training a neural network to directly predict the robot actions.

negative:
Predictive human models often need to adapt their parameters online from
human data. This raises previously ignored safety-related questions for robots
relying on these models such as what the model could learn online and how
quickly could it learn it. For instance, when will the robot have a confident
estimate in a nearby human's goal? Or, what parameter initializations guarantee
that the robot can learn the human's preferences in a finite number of
observations? To answer such analysis questions, our key idea is to model the
robot's learning algorithm as a dynamical system where the state is the current
model parameter estimate and the control is the human data the robot observes.
This enables us to leverage tools from reachability analysis and optimal
control to compute the set of hypotheses the robot could learn in finite time,
as well as the worst and best-case time it takes to learn them. We demonstrate
the utility of our analysis tool in four human-robot domains, including
autonomous driving and indoor navigation.

negative:
Human emotions are expressed through multiple modalities, including verbal
and non-verbal information. Moreover, the affective states of human users can
be the indicator for the level of engagement and successful interaction,
suitable for the robot to use as a rewarding factor to optimize robotic
behaviors through interaction. This study demonstrates a multimodal human-robot
interaction (HRI) framework with reinforcement learning to enhance the robotic
interaction policy and personalize emotional interaction for a human user. The
goal is to apply this framework in social scenarios that can let the robots
generate a more natural and engaging HRI framework.

negative:
Recent development in developing humanoid robot poses new challenges to
human-machine interaction communication. A major challenge is to develop robots
that can behave like and interact with human in the most natural way possible.
This paper proposes a system to develop a robot that can receive command, and
talk to people in natural language. In addition, the robot can also be
"trained" to become an expert in sepcific areas to provide expert advice to
human-beings. Most important of all, the robot can display emotions through
facial expression, speech and gesture so that the interaction process will
become more comprehensive and compelling.

negative:
In many robotic applications, an autonomous agent must act within and explore
a partially observed environment that is unobserved by its human teammate. We
consider such a setting in which the agent can, while acting, transmit
declarative information to the human that helps them understand aspects of this
unseen environment. In this work, we address the algorithmic question of how
the agent should plan out what actions to take and what information to
transmit. Naturally, one would expect the human to have preferences, which we
model information-theoretically by scoring transmitted information based on the
change it induces in weighted entropy of the human's belief state. We formulate
this setting as a belief MDP and give a tractable algorithm for solving it
approximately. Then, we give an algorithm that allows the agent to learn the
human's preferences online, through exploration. We validate our approach
experimentally in simulated discrete and continuous partially observed
search-and-recover domains. Visit http://tinyurl.com/chitnis-corl-18 for a
supplementary video.

negative:
Meta-learning algorithms can accelerate the model-based reinforcement
learning (MBRL) algorithms by finding an initial set of parameters for the
dynamical model such that the model can be trained to match the actual dynamics
of the system with only a few data-points. However, in the real world, a robot
might encounter any situation starting from motor failures to finding itself in
a rocky terrain where the dynamics of the robot can be significantly different
from one another. In this paper, first, we show that when meta-training
situations (the prior situations) have such diverse dynamics, using a single
set of meta-trained parameters as a starting point still requires a large
number of observations from the real system to learn a useful model of the
dynamics. Second, we propose an algorithm called FAMLE that mitigates this
limitation by meta-training several initial starting points (i.e., initial
parameters) for training the model and allows the robot to select the most
suitable starting point to adapt the model to the current situation with only a
few gradient steps. We compare FAMLE to MBRL, MBRL with a meta-trained model
with MAML, and model-free policy search algorithm PPO for various simulated and
real robotic tasks, and show that FAMLE allows the robots to adapt to novel
damages in significantly fewer time-steps than the baselines.

negative:
We explore the problem of estimating the mass distribution of an articulated
object by an interactive robotic agent. Our method predicts the mass
distribution of an object by using the limited sensing and actuating
capabilities of a robotic agent that is interacting with the object. We are
inspired by the role of exploratory play in human infants. We take the combined
approach of supervised and reinforcement learning to train an agent that learns
to strategically interact with the object to estimate the object's mass
distribution. Our method consists of two neural networks: (i) the policy
network which decides how to interact with the object, and (ii) the predictor
network that estimates the mass distribution given a history of observations
and interactions. Using our method, we train a robotic arm to estimate the mass
distribution of an object with moving parts (e.g. an articulated rigid body
system) by pushing it on a surface with unknown friction properties. We also
demonstrate how our training from simulations can be transferred to real
hardware using a small amount of real-world data for fine-tuning. We use a UR10
robot to interact with 3D printed articulated chains with varying mass
distributions and show that our method significantly outperforms the baseline
system that uses random pushes to interact with the object.

negative:
Cooperation in multi-agent and multi-robot systems can help agents build
various formations, shapes, and patterns presenting corresponding functions and
purposes adapting to different situations. Relationships between agents such as
their spatial proximity and functional similarities could play a crucial role
in cooperation between agents. Trust level between agents is an essential
factor in evaluating their relationships' reliability and stability, much as
people do. This paper proposes a new model called Relative Needs Entropy (RNE)
to assess trust between robotic agents. RNE measures the distance of needs
distribution between individual agents or groups of agents. To exemplify its
utility, we implement and demonstrate our trust model through experiments
simulating a heterogeneous multi-robot grouping task in a persistent urban
search and rescue mission consisting of tasks at two levels of difficulty. The
results suggest that RNE trust-Based grouping of robots can achieve better
performance and adaptability for diverse task execution compared to the
state-of-the-art energy-based or distance-based grouping models.

negative:
Predicting accurate future trajectories of multiple agents is essential for
autonomous systems, but is challenging due to the complex agent interaction and
the uncertainty in each agent's future behavior. Forecasting multi-agent
trajectories requires modeling two key dimensions: (1) time dimension, where we
model the influence of past agent states over future states; (2) social
dimension, where we model how the state of each agent affects others. Most
prior methods model these two dimensions separately, e.g., first using a
temporal model to summarize features over time for each agent independently and
then modeling the interaction of the summarized features with a social model.
This approach is suboptimal since independent feature encoding over either the
time or social dimension can result in a loss of information. Instead, we would
prefer a method that allows an agent's state at one time to directly affect
another agent's state at a future time. To this end, we propose a new
Transformer, AgentFormer, that jointly models the time and social dimensions.
The model leverages a sequence representation of multi-agent trajectories by
flattening trajectory features across time and agents. Since standard attention
operations disregard the agent identity of each element in the sequence,
AgentFormer uses a novel agent-aware attention mechanism that preserves agent
identities by attending to elements of the same agent differently than elements
of other agents. Based on AgentFormer, we propose a stochastic multi-agent
trajectory prediction model that can attend to features of any agent at any
previous timestep when inferring an agent's future position. The latent intent
of all agents is also jointly modeled, allowing the stochasticity in one
agent's behavior to affect other agents. Our method substantially improves the
state of the art on well-established pedestrian and autonomous driving
datasets.

negative:
Learning in multi-agent environments is difficult due to the non-stationarity
introduced by an opponent's or partner's changing behaviors. Instead of
reactively adapting to the other agent's (opponent or partner) behavior, we
propose an algorithm to proactively influence the other agent's strategy to
stabilize -- which can restrain the non-stationarity caused by the other agent.
We learn a low-dimensional latent representation of the other agent's strategy
and the dynamics of how the latent strategy evolves with respect to our robot's
behavior. With this learned dynamics model, we can define an unsupervised
stability reward to train our robot to deliberately influence the other agent
to stabilize towards a single strategy. We demonstrate the effectiveness of
stabilizing in improving efficiency of maximizing the task reward in a variety
of simulated environments, including autonomous driving, emergent
communication, and robotic manipulation. We show qualitative results on our
website: https://sites.google.com/view/stable-marl/.

negative:
Humans are able to seamlessly visually imitate others, by inferring their
intentions and using past experience to achieve the same end goal. In other
words, we can parse complex semantic knowledge from raw video and efficiently
translate that into concrete motor control. Is it possible to give a robot this
same capability? Prior research in robot imitation learning has created agents
which can acquire diverse skills from expert human operators. However,
expanding these techniques to work with a single positive example during test
time is still an open challenge. Apart from control, the difficulty stems from
mismatches between the demonstrator and robot domains. For example, objects may
be placed in different locations (e.g. kitchen layouts are different in every
house). Additionally, the demonstration may come from an agent with different
morphology and physical appearance (e.g. human), so one-to-one action
correspondences are not available. This paper investigates techniques which
allow robots to partially bridge these domain gaps, using their past
experience. A neural network is trained to mimic ground truth robot actions
given context video from another agent, and must generalize to unseen task
instances when prompted with new videos during test time. We hypothesize that
our policy representations must be both context driven and dynamics aware in
order to perform these tasks. These assumptions are baked into the neural
network using the Transformers attention mechanism and a self-supervised
inverse dynamics loss. Finally, we experimentally determine that our method
accomplishes a $\sim 2$x improvement in terms of task success rate over prior
baselines in a suite of one-shot manipulation tasks.

negative:
The exploration mechanism used by a Deep Reinforcement Learning (RL) agent
plays a key role in determining its sample efficiency. Thus, improving over
random exploration is crucial to solve long-horizon tasks with sparse rewards.
We propose to leverage an ensemble of partial solutions as teachers that guide
the agent's exploration with action suggestions throughout training. While the
setup of learning with teachers has been previously studied, our proposed
approach - Actor-Critic with Teacher Ensembles (AC-Teach) - is the first to
work with an ensemble of suboptimal teachers that may solve only part of the
problem or contradict other each other, forming a unified algorithmic solution
that is compatible with a broad range of teacher ensembles. AC-Teach leverages
a probabilistic representation of the expected outcome of the teachers' and
student's actions to direct exploration, reduce dithering, and adapt to the
dynamically changing quality of the learner. We evaluate a variant of AC-Teach
that guides the learning of a Bayesian DDPG agent on three tasks - path
following, robotic pick and place, and robotic cube sweeping using a hook - and
show that it improves largely on sampling efficiency over a set of baselines,
both for our target scenario of unconstrained suboptimal teachers and for
easier setups with optimal or single teachers. Additional results and videos at
https://sites.google.com/view/acteach/home.

negative:
It is important for socially assistive robots to be able to recognize when a
user needs and wants help. Such robots need to be able to recognize human needs
in a real-time manner so that they can provide timely assistance. We propose an
architecture that uses social cues to determine when a robot should provide
assistance. Based on a multimodal fusion approach upon eye gaze and language
modalities, our architecture is trained and evaluated on data collected in a
robot-assisted Lego building task. By focusing on social cues, our architecture
has minimal dependencies on the specifics of a given task, enabling it to be
applied in many different contexts. Enabling a social robot to recognize a
user's needs through social cues can help it to adapt to user behaviors and
preferences, which in turn will lead to improved user experiences.

negative:
My overarching research goal is to provide robots with perceptional abilities
that allow interactions with humans in a human-like manner. To develop these
perceptional abilities, I believe that it is useful to study the principles of
the human visual system. I use these principles to develop new computer vision
algorithms and validate their effectiveness in intelligent robotic systems. I
am enthusiastic about this approach as it offers the dual benefit of uncovering
principles inherent in the human visual system, as well as applying these
principles to its artificial counterpart. Fig. 1 contains a depiction of my
research.

negative:
The objective of a reinforcement learning agent is to behave so as to
maximise the sum of a suitable scalar function of state: the reward. These
rewards are typically given and immutable. In this paper, we instead consider
the proposition that the reward function itself can be a good locus of learned
knowledge. To investigate this, we propose a scalable meta-gradient framework
for learning useful intrinsic reward functions across multiple lifetimes of
experience. Through several proof-of-concept experiments, we show that it is
feasible to learn and capture knowledge about long-term exploration and
exploitation into a reward function. Furthermore, we show that unlike policy
transfer methods that capture "how" the agent should behave, the learned reward
functions can generalise to other kinds of agents and to changes in the
dynamics of the environment by capturing "what" the agent should strive to do.

negative:
In the vision and language navigation task, the agent may encounter ambiguous
situations that are hard to interpret by just relying on visual information and
natural language instructions. We propose an interactive learning framework to
endow the agent with the ability to ask for users' help in such situations. As
part of this framework, we investigate multiple learning approaches for the
agent with different levels of complexity. The simplest model-confusion-based
method lets the agent ask questions based on its confusion, relying on the
predefined confidence threshold of a next action prediction model. To build on
this confusion-based method, the agent is expected to demonstrate more
sophisticated reasoning such that it discovers the timing and locations to
interact with a human. We achieve this goal using reinforcement learning (RL)
with a proposed reward shaping term, which enables the agent to ask questions
only when necessary. The success rate can be boosted by at least 15% with only
one question asked on average during the navigation. Furthermore, we show that
the RL agent is capable of adjusting dynamically to noisy human responses.
Finally, we design a continual learning strategy, which can be viewed as a data
augmentation method, for the agent to improve further utilizing its interaction
history with a human. We demonstrate the proposed strategy is substantially
more realistic and data-efficient compared to previously proposed
pre-exploration techniques.

negative:
We study the problem of multi-robot mapless navigation in the popular
Centralized Training and Decentralized Execution (CTDE) paradigm. This problem
is challenging when each robot considers its path without explicitly sharing
observations with other robots and can lead to non-stationary issues in Deep
Reinforcement Learning (DRL). The typical CTDE algorithm factorizes the joint
action-value function into individual ones, to favor cooperation and achieve
decentralized execution. Such factorization involves constraints (e.g.,
monotonicity) that limit the emergence of novel behaviors in an individual as
each agent is trained starting from a joint action-value. In contrast, we
propose a novel architecture for CTDE that uses a centralized state-value
network to compute a joint state-value, which is used to inject global state
information in the value-based updates of the agents. Consequently, each model
computes its gradient update for the weights, considering the overall state of
the environment. Our idea follows the insights of Dueling Networks as a
separate estimation of the joint state-value has both the advantage of
improving sample efficiency, while providing each robot information whether the
global state is (or is not) valuable. Experiments in a robotic navigation task
with 2 4, and 8 robots, confirm the superior performance of our approach over
prior CTDE methods (e.g., VDN, QMIX).

negative:
Social learning is a key component of human and animal intelligence. By
taking cues from the behavior of experts in their environment, social learners
can acquire sophisticated behavior and rapidly adapt to new circumstances. This
paper investigates whether independent reinforcement learning (RL) agents in a
multi-agent environment can learn to use social learning to improve their
performance. We find that in most circumstances, vanilla model-free RL agents
do not use social learning. We analyze the reasons for this deficiency, and
show that by imposing constraints on the training environment and introducing a
model-based auxiliary loss we are able to obtain generalized social learning
policies which enable agents to: i) discover complex skills that are not
learned from single-agent training, and ii) adapt online to novel environments
by taking cues from experts present in the new environment. In contrast, agents
trained with model-free RL or imitation learning generalize poorly and do not
succeed in the transfer tasks. By mixing multi-agent and solo training, we can
obtain agents that use social learning to gain skills that they can deploy when
alone, even out-performing agents trained alone from the start.

negative:
The ability to distinguish between the self and the background is of
paramount importance for robotic tasks. The particular case of hands, as the
end effectors of a robotic system that more often enter into contact with other
elements of the environment, must be perceived and tracked with precision to
execute the intended tasks with dexterity and without colliding with obstacles.
They are fundamental for several applications, from Human-Robot Interaction
tasks to object manipulation. Modern humanoid robots are characterized by high
number of degrees of freedom which makes their forward kinematics models very
sensitive to uncertainty. Thus, resorting to vision sensing can be the only
solution to endow these robots with a good perception of the self, being able
to localize their body parts with precision. In this paper, we propose the use
of a Convolution Neural Network (CNN) to segment the robot hand from an image
in an egocentric view. It is known that CNNs require a huge amount of data to
be trained. To overcome the challenge of labeling real-world images, we propose
the use of simulated datasets exploiting domain randomization techniques. We
fine-tuned the Mask-RCNN network for the specific task of segmenting the hand
of the humanoid robot Vizzy. We focus our attention on developing a methodology
that requires low amounts of data to achieve reasonable performance while
giving detailed insight on how to properly generate variability in the training
dataset. Moreover, we analyze the fine-tuning process within the complex model
of Mask-RCNN, understanding which weights should be transferred to the new task
of segmenting robot hands. Our final model was trained solely on synthetic
images and achieves an average IoU of 82% on synthetic validation data and
56.3% on real test data. These results were achieved with only 1000 training
images and 3 hours of training time using a single GPU.

negative:
Proxemics is a branch of non-verbal communication concerned with studying the
spatial behavior of people and animals. This behavior is an essential part of
the communication process due to delimit the acceptable distance to interact
with another being. With increasing research on human-agent interaction, new
alternatives are needed that allow optimal communication, avoiding agents
feeling uncomfortable. Several works consider proxemic behavior with cognitive
agents, where human-robot interaction techniques and machine learning are
implemented. However, environments consider fixed personal space and that the
agent previously knows it. In this work, we aim to study how agents behave in
environments based on proxemic behavior, and propose a modified gridworld to
that aim. This environment considers an issuer with proxemic behavior that
provides a disagreement signal to the agent. Our results show that the learning
agent can identify the proxemic space when the issuer gives feedback about
agent performance.

negative:
The mental models that humans form of other agents---encapsulating human
beliefs about agent goals, intentions, capabilities, and more---create an
underlying basis for interaction. These mental models have the potential to
affect both the human's decision making during the interaction and the human's
subjective assessment of the interaction. In this paper, we surveyed existing
methods for modeling how humans view robots, then identified a potential method
for improving these estimates through inferring a human's model of a robot
agent directly from their actions. Then, we conducted an online study to
collect data in a grid-world environment involving humans moving an avatar past
a virtual agent. Through our analysis, we demonstrated that participants'
action choices leaked information about their mental models of a virtual agent.
We conclude by discussing the implications of these findings and the potential
for such a method to improve human-robot interactions.

negative:
Any agent that is part of the environment it interacts with and has versatile
actuators (such as arms and fingers), will in principle have the ability to
self-modify -- for example by changing its own source code. As we continue to
create more and more intelligent agents, chances increase that they will learn
about this ability. The question is: will they want to use it? For example,
highly intelligent systems may find ways to change their goals to something
more easily achievable, thereby `escaping' the control of their designers. In
an important paper, Omohundro (2008) argued that goal preservation is a
fundamental drive of any intelligent system, since a goal is more likely to be
achieved if future versions of the agent strive towards the same goal. In this
paper, we formalise this argument in general reinforcement learning, and
explore situations where it fails. Our conclusion is that the self-modification
possibility is harmless if and only if the value function of the agent
anticipates the consequences of self-modifications and use the current utility
function when evaluating the future.

negative:
In this study Arduino card based mobile robot design was realized. This robot
can serve as a security robot, an auxiliary robot or a control robot. The
designed robot has two operation modes. The first operating mode is autonomous
mode. In this mode, the robot detects the surroundings with the help of
ultrasonic sensors placed around it, and keeps track of the places it passes by
using the encoder. It is able to navigate without hitting any place and passing
from where it passes, and it transmits the patient's pulse and temperature
condition to the user by other systems installed on it. Also the IP camera
sends the scene on the screen. The emergency button to be placed next to the
patient sends information to the user in emergency situations. If the
abnormality is detected in the temperature and pulse again, the user gives a
message. When the pre-recorded drug use times come, the system can alert the
patient. The second mode is manual mode. In this mode, the user can move the
desired direction of the robot with the Android operating system. In addition,
all data received in autonomous mode can be sent to the user. Thus, the user
can control the mobile robot with the camera image even if it is not in the
vicinity of the robot.

negative:
Multi-task learning can leverage information learned by one task to benefit
the training of other tasks. Despite this capacity, naive formulations often
degrade performance and in particular, identifying the tasks that would benefit
from co-training remains a challenging design question. In this paper, we
analyze the dynamics of information transfer, or transference, across tasks
throughout training. Specifically, we develop a similarity measure that can
quantify transference among tasks and use this quantity to both better
understand the optimization dynamics of multi-task learning as well as improve
overall learning performance. In the latter case, we propose two methods to
leverage our transference metric. The first operates at a macro-level by
selecting which tasks should train together while the second functions at a
micro-level by determining how to combine task gradients at each training step.
We find these methods can lead to significant improvement over prior work on
three supervised multi-task learning benchmarks and one multi-task
reinforcement learning paradigm.

negative:
Agent Based Models are very popular in a number of different areas. For
example, they have been used in a range of domains ranging from modeling of
tumor growth, immune systems, molecules to models of social networks, crowds
and computer and mobile self-organizing networks. One reason for their success
is their intuitiveness and similarity to human cognition. However, with this
power of abstraction, in spite of being easily applicable to such a wide number
of domains, it is hard to validate agent-based models. In addition, building
valid and credible simulations is not just a challenging task but also a
crucial exercise to ensure that what we are modeling is, at some level of
abstraction, a model of our conceptual system; the system that we have in mind.
In this paper, we address this important area of validation of agent based
models by presenting a novel technique which has broad applicability and can be
applied to all kinds of agent-based models. We present a framework, where a
virtual overlay multi-agent system can be used to validate simulation models.
In addition, since agent-based models have been typically growing, in parallel,
in multiple domains, to cater for all of these, we present a new single
validation technique applicable to all agent based models. Our technique, which
allows for the validation of agent based simulations uses VOMAS: a Virtual
Overlay Multi-agent System. This overlay multi-agent system can comprise
various types of agents, which form an overlay on top of the agent based
simulation model that needs to be validated. Other than being able to watch and
log, each of these agents contains clearly defined constraints, which, if
violated, can be logged in real time. To demonstrate its effectiveness, we show
its broad applicability in a wide variety of simulation models ranging from
social sciences to computer networks in spatial and non-spatial conceptual
models.

negative:
In cooperative multiagent planning, it can often be beneficial for an agent
to make commitments about aspects of its behavior to others, allowing them in
turn to plan their own behaviors without taking the agent's detailed behavior
into account. Extending previous work in the Bayesian setting, we consider
instead a worst-case setting in which the agent has a set of possible
environments (MDPs) it could be in, and develop a commitment semantics that
allows for probabilistic guarantees on the agent's behavior in any of the
environments it could end up facing. Crucially, an agent receives observations
(of reward and state transitions) that allow it to potentially eliminate
possible environments and thus obtain higher utility by adapting its policy to
the history of observations. We develop algorithms and provide theory and some
preliminary empirical results showing that they ensure an agent meets its
commitments with history-dependent policies while minimizing maximum regret
over the possible environments.

negative:
For the past few decades, man has been trying to create an intelligent
computer which can talk and respond like he can. The task of creating a system
that can talk like a human being is the primary objective of Automatic Speech
Recognition. Various Speech Recognition techniques have been developed in
theory and have been applied in practice. This paper discusses the problems
that have been encountered in developing Speech Recognition, the techniques
that have been applied to automate the task, and a representation of the core
problems of present day Speech Recognition by using Fuzzy Mathematics.

negative:
This paper describes an application of reinforcement learning to the mention
detection task. We define a novel action-based formulation for the mention
detection task, in which a model can flexibly revise past labeling decisions by
grouping together tokens and assigning partial mention labels. We devise a
method to create mention-level episodes and we train a model by rewarding
correctly labeled complete mentions, irrespective of the inner structure
created. The model yields results which are on par with a competitive
supervised counterpart while being more flexible in terms of achieving targeted
behavior through reward modeling and generating internal mention structure,
especially on longer mentions.

negative:
People are becoming increasingly comfortable using Digital Assistants (DAs)
to interact with services or connected objects. However, for non-programming
users, the available possibilities for customizing their DA are limited and do
not include the possibility of teaching the assistant new tasks. To make the
most of the potential of DAs, users should be able to customize assistants by
instructing them through Natural Language (NL). To provide such
functionalities, NL interpretation in traditional assistants should be
improved: (1) The intent identification system should be able to recognize new
forms of known intents, and to acquire new intents as they are expressed by the
user. (2) In order to be adaptive to novel intents, the Natural Language
Understanding module should be sample efficient, and should not rely on a
pretrained model. Rather, the system should continuously collect the training
data as it learns new intents from the user. In this work, we propose AidMe
(Adaptive Intent Detection in Multi-Domain Environments), a user-in-the-loop
adaptive intent detection framework that allows the assistant to adapt to its
user by learning his intents as their interaction progresses. AidMe builds its
repertoire of intents and collects data to train a model of semantic similarity
evaluation that can discriminate between the learned intents and autonomously
discover new forms of known intents. AidMe addresses two major issues - intent
learning and user adaptation - for instructable digital assistants. We
demonstrate the capabilities of AidMe as a standalone system by comparing it
with a one-shot learning system and a pretrained NLU module through simulations
of interactions with a user. We also show how AidMe can smoothly integrate to
an existing instructable digital assistant.

negative:
Named entity linking is to map an ambiguous mention in documents to an entity
in a knowledge base. The named entity linking is challenging, given the fact
that there are multiple candidate entities for a mention in a document. It is
difficult to link a mention when it appears multiple times in a document, since
there are conflicts by the contexts around the appearances of the mention. In
addition, it is difficult since the given training dataset is small due to the
reason that it is done manually to link a mention to its mapping entity. In the
literature, there are many reported studies among which the recent embedding
methods learn vectors of entities from the training dataset at document level.
To address these issues, we focus on how to link entity for mentions at a
sentence level, which reduces the noises introduced by different appearances of
the same mention in a document at the expense of insufficient information to be
used. We propose a new unified embedding method by maximizing the relationships
learned from knowledge graphs. We confirm the effectiveness of our method in
our experimental studies.