query: I am seeking to develop a robotic agent capable of comprehending user input and completing assigned tasks. I believe the agent's functionality can be separated into two segments: understanding commands and executing actions. First, I require a model that can interpret user commands, whether written or verbal, and translate them into a list of real-world actions. I believe a language model would be beneficial at this stage. However, I need advice on how to strike a balance between the size of the language model and the accuracy of its interpretation of the input. Additionally, I require a reinforcement learning framework for the action execution phase. However, I've noticed that users often provide detailed yet suboptimal instructions. Therefore, I am in search of a reward function that incorporates social psychology to enhance service quality. I anticipate an algorithm that empowers the agent to assess user input and carry out the most efficient actions while not straying too far from the original input.

positive:
Robots operating alongside humans in diverse, stochastic environments must be
able to accurately interpret natural language commands. These instructions
often fall into one of two categories: those that specify a goal condition or
target state, and those that specify explicit actions, or how to perform a
given task. Recent approaches have used reward functions as a semantic
representation of goal-based commands, which allows for the use of a
state-of-the-art planner to find a policy for the given task. However, these
reward functions cannot be directly used to represent action-oriented commands.
We introduce a new hybrid approach, the Deep Recurrent Action-Goal Grounding
Network (DRAGGN), for task grounding and execution that handles natural
language from either category as input, and generalizes to unseen environments.
Our robot-simulation results demonstrate that a system successfully
interpreting both goal-oriented and action-oriented task specifications brings
us closer to robust natural language understanding for human-robot interaction.

positive:
We are increasingly surrounded by artificially intelligent technology that
takes decisions and executes actions on our behalf. This creates a pressing
need for general means to communicate with, instruct and guide artificial
agents, with human language the most compelling means for such communication.
To achieve this in a scalable fashion, agents must be able to relate language
to the world and to actions; that is, their understanding of language must be
grounded and embodied. However, learning grounded language is a notoriously
challenging problem in artificial intelligence research. Here we present an
agent that learns to interpret language in a simulated 3D environment where it
is rewarded for the successful execution of written instructions. Trained via a
combination of reinforcement and unsupervised learning, and beginning with
minimal prior knowledge, the agent learns to relate linguistic symbols to
emergent perceptual representations of its physical surroundings and to
pertinent sequences of actions. The agent's comprehension of language extends
beyond its prior experience, enabling it to apply familiar language to
unfamiliar situations and to interpret entirely novel instructions. Moreover,
the speed with which this agent learns new words increases as its semantic
knowledge grows. This facility for generalising and bootstrapping semantic
knowledge indicates the potential of the present approach for reconciling
ambiguous natural language with the complexity of the physical world.

positive:
Recent work has described neural-network-based agents that are trained with
reinforcement learning (RL) to execute language-like commands in simulated
worlds, as a step towards an intelligent agent or robot that can be instructed
by human users. However, the optimisation of multi-goal motor policies via deep
RL from scratch requires many episodes of experience. Consequently,
instruction-following with deep RL typically involves language generated from
templates (by an environment simulator), which does not reflect the varied or
ambiguous expressions of real users. Here, we propose a conceptually simple
method for training instruction-following agents with deep RL that are robust
to natural human instructions. By applying our method with a state-of-the-art
pre-trained text-based language model (BERT), on tasks requiring agents to
identify and position everyday objects relative to other objects in a
naturalistic 3D simulated room, we demonstrate substantially-above-chance
zero-shot transfer from synthetic template commands to natural instructions
given by humans. Our approach is a general recipe for training any deep
RL-based system to interface with human users, and bridges the gap between two
research directions of notable recent success: agent-centric motor behavior and
text-based representation learning.

positive:
It is highly desirable for robots that work alongside humans to be able to
understand instructions in natural language. Existing language conditioned
imitation learning models directly predict the actuator commands from the image
observation and the instruction text. Rather than directly predicting actuator
commands, we propose translating the natural language instruction to a Python
function which queries the scene by accessing the output of the object detector
and controls the robot to perform the specified task. This enables the use of
non-differentiable modules such as a constraint solver when computing commands
to the robot. Moreover, the labels in this setup are significantly more
informative computer programs that capture the intent of the expert rather than
teleoperated demonstrations. We show that the proposed method performs better
than training a neural network to directly predict the robot actions.

positive:
Interactive reinforcement learning agents use human feedback or instruction
to help them learn in complex environments. Often, this feedback comes in the
form of a discrete signal that is either positive or negative. While
informative, this information can be difficult to generalize on its own. In
this work, we explore how natural language advice can be used to provide a
richer feedback signal to a reinforcement learning agent by extending policy
shaping, a well-known Interactive reinforcement learning technique. Usually
policy shaping employs a human feedback policy to help an agent to learn more
about how to achieve its goal. In our case, we replace this human feedback
policy with policy generated based on natural language advice. We aim to
inspect if the generated natural language reasoning provides support to a deep
reinforcement learning agent to decide its actions successfully in any given
environment. So, we design our model with three networks: first one is the
experience driven, next is the advice generator and third one is the advice
driven. While the experience driven reinforcement learning agent chooses its
actions being influenced by the environmental reward, the advice driven neural
network with generated feedback by the advice generator for any new state
selects its actions to assist the reinforcement learning agent to better policy
shaping.

positive:
Our goal is to create an interactive natural language interface that
efficiently and reliably learns from users to complete tasks in simulated
robotics settings. We introduce a neural semantic parsing system that learns
new high-level abstractions through decomposition: users interactively teach
the system by breaking down high-level utterances describing novel behavior
into low-level steps that it can understand. Unfortunately, existing methods
either rely on grammars which parse sentences with limited flexibility, or
neural sequence-to-sequence models that do not learn efficiently or reliably
from individual examples. Our approach bridges this gap, demonstrating the
flexibility of modern neural systems, as well as the one-shot reliable
generalization of grammar-based methods. Our crowdsourced interactive
experiments suggest that over time, users complete complex tasks more
efficiently while using our system by leveraging what they just taught. At the
same time, getting users to trust the system enough to be incentivized to teach
high-level utterances is still an ongoing challenge. We end with a discussion
of some of the obstacles we need to overcome to fully realize the potential of
the interactive paradigm.

positive:
The recently proposed ALFRED challenge task aims for a virtual robotic agent
to complete complex multi-step everyday tasks in a virtual home environment
from high-level natural language directives, such as "put a hot piece of bread
on a plate". Currently, the best-performing models are able to complete less
than 5% of these tasks successfully. In this work we focus on modeling the
translation problem of converting natural language directives into detailed
multi-step sequences of actions that accomplish those goals in the virtual
environment. We empirically demonstrate that it is possible to generate gold
multi-step plans from language directives alone without any visual input in 26%
of unseen cases. When a small amount of visual information is incorporated,
namely the starting location in the virtual environment, our best-performing
GPT-2 model successfully generates gold command sequences in 58% of cases. Our
results suggest that contextualized language models may provide strong visual
semantic planning modules for grounded virtual agents.

positive:
People are becoming increasingly comfortable using Digital Assistants (DAs)
to interact with services or connected objects. However, for non-programming
users, the available possibilities for customizing their DA are limited and do
not include the possibility of teaching the assistant new tasks. To make the
most of the potential of DAs, users should be able to customize assistants by
instructing them through Natural Language (NL). To provide such
functionalities, NL interpretation in traditional assistants should be
improved: (1) The intent identification system should be able to recognize new
forms of known intents, and to acquire new intents as they are expressed by the
user. (2) In order to be adaptive to novel intents, the Natural Language
Understanding module should be sample efficient, and should not rely on a
pretrained model. Rather, the system should continuously collect the training
data as it learns new intents from the user. In this work, we propose AidMe
(Adaptive Intent Detection in Multi-Domain Environments), a user-in-the-loop
adaptive intent detection framework that allows the assistant to adapt to its
user by learning his intents as their interaction progresses. AidMe builds its
repertoire of intents and collects data to train a model of semantic similarity
evaluation that can discriminate between the learned intents and autonomously
discover new forms of known intents. AidMe addresses two major issues - intent
learning and user adaptation - for instructable digital assistants. We
demonstrate the capabilities of AidMe as a standalone system by comparing it
with a one-shot learning system and a pretrained NLU module through simulations
of interactions with a user. We also show how AidMe can smoothly integrate to
an existing instructable digital assistant.

positive:
Wheelchair-mounted robotic arms (and other assistive robots) should help
their users perform everyday tasks. One way robots can provide this assistance
is shared autonomy. Within shared autonomy, both the human and robot maintain
control over the robot's motion: as the robot becomes confident it understands
what the human wants, it increasingly intervenes to automate the task. But how
does the robot know what tasks the human may want to perform in the first
place? Today's shared autonomy approaches often rely on prior knowledge: for
example, the robot must know the set of possible human goals a priori. In the
long-term, however, this prior knowledge will inevitably break down -- sooner
or later the human will reach for a goal that the robot did not expect. In this
paper we propose a learning approach to shared autonomy that takes advantage of
repeated interactions. Learning to assist humans would be impossible if they
performed completely different tasks at every interaction: but our insight is
that users living with physical disabilities repeat important tasks on a daily
basis (e.g., opening the fridge, making coffee, and having dinner). We
introduce an algorithm that exploits these repeated interactions to recognize
the human's task, replicate similar demonstrations, and return control when
unsure. As the human repeatedly works with this robot, our approach continually
learns to assist tasks that were never specified beforehand: these tasks
include both discrete goals (e.g., reaching a cup) and continuous skills (e.g.,
opening a drawer). Across simulations and an in-person user study, we
demonstrate that robots leveraging our approach match existing shared autonomy
methods for known goals, and outperform imitation learning baselines on new
tasks. See videos here: https://youtu.be/Plh4t3wQeIA

positive:
We propose a developmental approach that allows a robot to interpret and
describe the actions of human agents by reusing previous experience. The robot
first learns the association between words and object affordances by
manipulating the objects in its environment. It then uses this information to
learn a mapping between its own actions and those performed by a human in a
shared environment. It finally fuses the information from these two models to
interpret and describe human actions in light of its own experience. In our
experiments, we show that the model can be used flexibly to do inference on
different aspects of the scene. We can predict the effects of an action on the
basis of object properties. We can revise the belief that a certain action
occurred, given the observed effects of the human action. In an early action
recognition fashion, we can anticipate the effects when the action has only
been partially observed. By estimating the probability of words given the
evidence and feeding them into a pre-defined grammar, we can generate relevant
descriptions of the scene. We believe that this is a step towards providing
robots with the fundamental skills to engage in social collaboration with
humans.

positive:
Behavioral skills or policies for autonomous agents are conventionally
learned from reward functions, via reinforcement learning, or from
demonstrations, via imitation learning. However, both modes of task
specification have their disadvantages: reward functions require manual
engineering, while demonstrations require a human expert to be able to actually
perform the task in order to generate the demonstration. Instruction following
from natural language instructions provides an appealing alternative: in the
same way that we can specify goals to other humans simply by speaking or
writing, we would like to be able to specify tasks for our machines. However, a
single instruction may be insufficient to fully communicate our intent or, even
if it is, may be insufficient for an autonomous agent to actually understand
how to perform the desired task. In this work, we propose an interactive
formulation of the task specification problem, where iterative language
corrections are provided to an autonomous agent, guiding it in acquiring the
desired skill. Our proposed language-guided policy learning algorithm can
integrate an instruction and a sequence of corrections to acquire new skills
very quickly. In our experiments, we show that this method can enable a policy
to follow instructions and corrections for simulated navigation and
manipulation tasks, substantially outperforming direct, non-interactive
instruction following.

positive:
Humans generally use natural language to communicate task requirements to
each other. Ideally, natural language should also be usable for communicating
goals to autonomous machines (e.g., robots) to minimize friction in task
specification. However, understanding and mapping natural language goals to
sequences of states and actions is challenging. Specifically, existing work
along these lines has encountered difficulty in generalizing learned policies
to new natural language goals and environments. In this paper, we propose a
novel adversarial inverse reinforcement learning algorithm to learn a
language-conditioned policy and reward function. To improve generalization of
the learned policy and reward function, we use a variational goal generator to
relabel trajectories and sample diverse goals during training. Our algorithm
outperforms multiple baselines by a large margin on a vision-based natural
language instruction following dataset (Room-2-Room), demonstrating a promising
advance in enabling the use of natural language instructions in specifying
agent goals.

positive:
Natural language understanding for robotics can require substantial domain-
and platform-specific engineering. For example, for mobile robots to
pick-and-place objects in an environment to satisfy human commands, we can
specify the language humans use to issue such commands, and connect concept
words like red can to physical object properties. One way to alleviate this
engineering for a new domain is to enable robots in human environments to adapt
dynamically---continually learning new language constructions and perceptual
concepts. In this work, we present an end-to-end pipeline for translating
natural language commands to discrete robot actions, and use clarification
dialogs to jointly improve language parsing and concept grounding. We train and
evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we
transfer the learned agent to a physical robot platform to demonstrate it in
the real world.

positive:
In this paper we present a framework for the teleoperation of pick-and-place
tasks. We define a shared control policy that allows to blend between direct
user control and autonomous control based on user intent inference. One of the
main challenges in shared autonomy systems is to define the arbitration
function, which decides when to let the autonomous agent take over. In this
work, we propose a model and training method to learn the arbitration function.
Our model is based on a recurrent neural network that takes as input the state,
intent prediction scores and user command to produce an arbitration between
user and robot commands. This work extends our previous work on differentiable
policies for shared autonomy. Differentiability of the policy is desirable to
further train the shared autonomy system end-to-end. In this work we propose
training of the arbitration function by using data from user performing the
task with shared control. We present initial results by teleoperating a gripper
in a virtual environment using pre-trained motion generation and intent
prediction. We compare our data aggregation training procedure to a handcrafted
arbitration function. Our preliminary results show the efficacy of the approach
and shed light on limitations that we believe demonstrate the need for user
adaptability in shared autonomy systems.

positive:
Recent successes combine reinforcement learning algorithms and deep neural
networks, despite reinforcement learning not being widely applied to robotics
and real world scenarios. This can be attributed to the fact that current
state-of-the-art, end-to-end reinforcement learning approaches still require
thousands or millions of data samples to converge to a satisfactory policy and
are subject to catastrophic failures during training. Conversely, in real world
scenarios and after just a few data samples, humans are able to either provide
demonstrations of the task, intervene to prevent catastrophic actions, or
simply evaluate if the policy is performing correctly. This research
investigates how to integrate these human interaction modalities to the
reinforcement learning loop, increasing sample efficiency and enabling
real-time reinforcement learning in robotics and real world scenarios. This
novel theoretical foundation is called Cycle-of-Learning, a reference to how
different human interaction modalities, namely, task demonstration,
intervention, and evaluation, are cycled and combined to reinforcement learning
algorithms. Results presented in this work show that the reward signal that is
learned based upon human interaction accelerates the rate of learning of
reinforcement learning algorithms and that learning from a combination of human
demonstrations and interventions is faster and more sample efficient when
compared to traditional supervised learning algorithms. Finally,
Cycle-of-Learning develops an effective transition between policies learned
using human demonstrations and interventions to reinforcement learning. The
theoretical foundation developed by this research opens new research paths to
human-agent teaming scenarios where autonomous agents are able to learn from
human teammates and adapt to mission performance metrics in real-time and in
real world scenarios.

positive:
This article surveys reinforcement learning approaches in social robotics.
Reinforcement learning is a framework for decision-making problems in which an
agent interacts through trial-and-error with its environment to discover an
optimal behavior. Since interaction is a key component in both reinforcement
learning and social robotics, it can be a well-suited approach for real-world
interactions with physically embodied social robots. The scope of the paper is
focused particularly on studies that include social physical robots and
real-world human-robot interactions with users. We present a thorough analysis
of reinforcement learning approaches in social robotics. In addition to a
survey, we categorize existent reinforcement learning approaches based on the
used method and the design of the reward mechanisms. Moreover, since
communication capability is a prominent feature of social robots, we discuss
and group the papers based on the communication medium used for reward
formulation. Considering the importance of designing the reward function, we
also provide a categorization of the papers based on the nature of the reward.
This categorization includes three major themes: interactive reinforcement
learning, intrinsically motivated methods, and task performance-driven methods.
The benefits and challenges of reinforcement learning in social robotics,
evaluation methods of the papers regarding whether or not they use subjective
and algorithmic measures, a discussion in the view of real-world reinforcement
learning challenges and proposed solutions, the points that remain to be
explored, including the approaches that have thus far received less attention
is also given in the paper. Thus, this paper aims to become a starting point
for researchers interested in using and applying reinforcement learning methods
in this particular research field.

positive:
Robots navigating in human environments should use language to ask for
assistance and be able to understand human responses. To study this challenge,
we introduce Cooperative Vision-and-Dialog Navigation, a dataset of over 2k
embodied, human-human dialogs situated in simulated, photorealistic home
environments. The Navigator asks questions to their partner, the Oracle, who
has privileged access to the best next steps the Navigator should take
according to a shortest path planner. To train agents that search an
environment for a goal location, we define the Navigation from Dialog History
task. An agent, given a target object and a dialog history between humans
cooperating to find that object, must infer navigation actions towards the goal
in unexplored environments. We establish an initial, multi-modal
sequence-to-sequence model and demonstrate that looking farther back in the
dialog history improves performance. Sourcecode and a live interface demo can
be found at https://cvdn.dev/

positive:
Goal-oriented conversational agents are becoming prevalent in our daily
lives. For these systems to engage users and achieve their goals, they need to
exhibit appropriate social behavior as well as provide informative replies that
guide users through tasks. The first component of the research in this paper
applies statistical modeling techniques to understand conversations between
users and human agents for customer service. Analyses show that social language
used by human agents is associated with greater users' responsiveness and task
completion. The second component of the research is the construction of a
conversational agent model capable of injecting social language into an agent's
responses while still preserving content. The model uses a sequence-to-sequence
deep learning architecture, extended with a social language understanding
element. Evaluation in terms of content preservation and social language level
using both human judgment and automatic linguistic measures shows that the
model can generate responses that enable agents to address users' issues in a
more socially appropriate way.

positive:
Augmented-reality (AR) glasses that will have access to onboard sensors and
an ability to display relevant information to the user present an opportunity
to provide user assistance in quotidian tasks. Many such tasks can be
characterized as object-rearrangement tasks. We introduce a novel framework for
computing and displaying AR assistance that consists of (1) associating an
optimal action sequence with the policy of an embodied agent and (2) presenting
this sequence to the user as suggestions in the AR system's heads-up display.
The embodied agent comprises a "hybrid" between the AR system and the user,
with the AR system's observation space (i.e., sensors) and the user's action
space (i.e., task-execution actions); its policy is learned by minimizing the
task-completion time. In this initial study, we assume that the AR system's
observations include the environment's map and localization of the objects and
the user. These choices allow us to formalize the problem of computing AR
assistance for any object-rearrangement task as a planning problem,
specifically as a capacitated vehicle-routing problem. Further, we introduce a
novel AR simulator that can enable web-based evaluation of AR-like assistance
and associated at-scale data collection via the Habitat simulator for embodied
artificial intelligence. Finally, we perform a study that evaluates user
response to the proposed form of AR assistance on a specific quotidian
object-rearrangement task, house cleaning, using our proposed AR simulator on
mechanical turk. In particular, we study the effect of the proposed AR
assistance on users' task performance and sense of agency over a range of task
difficulties. Our results indicate that providing users with such assistance
improves their overall performance and while users report a negative impact to
their agency, they may still prefer the proposed assistance to having no
assistance at all.

positive:
Human collaborators coordinate effectively their actions through both verbal
and non-verbal communication. We believe that the the same should hold for
human-robot teams. We propose a formalism that enables a robot to decide
optimally between doing a task and issuing an utterance. We focus on two types
of utterances: verbal commands, where the robot expresses how it wants its
human teammate to behave, and state-conveying actions, where the robot explains
why it is behaving this way. Human subject experiments show that enabling the
robot to issue verbal commands is the most effective form of communicating
objectives, while retaining user trust in the robot. Communicating why
information should be done judiciously, since many participants questioned the
truthfulness of the robot statements.

positive:
Modern approaches to text to speech require the entire input character
sequence to be processed before any audio is synthesised. This latency limits
the suitability of such models for time-sensitive tasks like simultaneous
interpretation. Interleaving the action of reading a character with that of
synthesising audio reduces this latency. However, the order of this sequence of
interleaved actions varies across sentences, which raises the question of how
the actions should be chosen. We propose a reinforcement learning based
framework to train an agent to make this decision. We compare our performance
against that of deterministic, rule-based systems. Our results demonstrate that
our agent successfully balances the trade-off between the latency of audio
generation and the quality of synthesised audio. More broadly, we show that
neural sequence-to-sequence models can be adapted to run in an incremental
manner.

positive:
We address the problem of bootstrapping language acquisition for an
artificial system similarly to what is observed in experiments with human
infants. Our method works by associating meanings to words in manipulation
tasks, as a robot interacts with objects and listens to verbal descriptions of
the interactions. The model is based on an affordance network, i.e., a mapping
between robot actions, robot perceptions, and the perceived effects of these
actions upon objects. We extend the affordance model to incorporate spoken
words, which allows us to ground the verbal symbols to the execution of actions
and the perception of the environment. The model takes verbal descriptions of a
task as the input and uses temporal co-occurrence to create links between
speech utterances and the involved objects, actions, and effects. We show that
the robot is able form useful word-to-meaning associations, even without
considering grammatical structure in the learning process and in the presence
of recognition errors. These word-to-meaning associations are embedded in the
robot's own understanding of its actions. Thus, they can be directly used to
instruct the robot to perform tasks and also allow to incorporate context in
the speech recognition task. We believe that the encouraging results with our
approach may afford robots with a capacity to acquire language descriptors in
their operation's environment as well as to shed some light as to how this
challenging process develops with human infants.

positive:
Machine Learning models become increasingly proficient in complex tasks.
However, even for experts in the field, it can be difficult to understand what
the model learned. This hampers trust and acceptance, and it obstructs the
possibility to correct the model. There is therefore a need for transparency of
machine learning models. The development of transparent classification models
has received much attention, but there are few developments for achieving
transparent Reinforcement Learning (RL) models. In this study we propose a
method that enables a RL agent to explain its behavior in terms of the expected
consequences of state transitions and outcomes. First, we define a translation
of states and actions to a description that is easier to understand for human
users. Second, we developed a procedure that enables the agent to obtain the
consequences of a single action, as well as its entire policy. The method
calculates contrasts between the consequences of a policy derived from a user
query, and of the learned policy of the agent. Third, a format for generating
explanations was constructed. A pilot survey study was conducted to explore
preferences of users for different explanation properties. Results indicate
that human users tend to favor explanations about policy rather than about
single actions.

positive:
One of the long-term goals of artificial intelligence is to build an agent
that can communicate intelligently with human in natural language. Most
existing work on natural language learning relies heavily on training over a
pre-collected dataset with annotated labels, leading to an agent that
essentially captures the statistics of the fixed external training data. As the
training data is essentially a static snapshot representation of the knowledge
from the annotator, the agent trained this way is limited in adaptiveness and
generalization of its behavior. Moreover, this is very different from the
language learning process of humans, where language is acquired during
communication by taking speaking action and learning from the consequences of
speaking action in an interactive manner. This paper presents an interactive
setting for grounded natural language learning, where an agent learns natural
language by interacting with a teacher and learning from feedback, thus
learning and improving language skills while taking part in the conversation.
To achieve this goal, we propose a model which incorporates both imitation and
reinforcement by leveraging jointly sentence and reward feedbacks from the
teacher. Experiments are conducted to validate the effectiveness of the
proposed approach.

positive:
The combination of deep neural network models and reinforcement learning
algorithms can make it possible to learn policies for robotic behaviors that
directly read in raw sensory inputs, such as camera images, effectively
subsuming both estimation and control into one model. However, real-world
applications of reinforcement learning must specify the goal of the task by
means of a manually programmed reward function, which in practice requires
either designing the very same perception pipeline that end-to-end
reinforcement learning promises to avoid, or else instrumenting the environment
with additional sensors to determine if the task has been performed
successfully. In this paper, we propose an approach for removing the need for
manual engineering of reward specifications by enabling a robot to learn from a
modest number of examples of successful outcomes, followed by actively
solicited queries, where the robot shows the user a state and asks for a label
to determine whether that state represents successful completion of the task.
While requesting labels for every single state would amount to asking the user
to manually provide the reward signal, our method requires labels for only a
tiny fraction of the states seen during training, making it an efficient and
practical approach for learning skills without manually engineered rewards. We
evaluate our method on real-world robotic manipulation tasks where the
observations consist of images viewed by the robot's camera. In our
experiments, our method effectively learns to arrange objects, place books, and
drape cloth, directly from images and without any manually specified reward
functions, and with only 1-4 hours of interaction with the real world.

negative:
Reinforcement learning (RL) can be used to create a tactical decision-making
agent for autonomous driving. However, previous approaches only output
decisions and do not provide information about the agent's confidence in the
recommended actions. This paper investigates how a Bayesian RL technique, based
on an ensemble of neural networks with additional randomized prior functions
(RPF), can be used to estimate the uncertainty of decisions in autonomous
driving. A method for classifying whether or not an action should be considered
safe is also introduced. The performance of the ensemble RPF method is
evaluated by training an agent on a highway driving scenario. It is shown that
the trained agent can estimate the uncertainty of its decisions and indicate an
unacceptable level when the agent faces a situation that is far from the
training distribution. Furthermore, within the training distribution, the
ensemble RPF agent outperforms a standard Deep Q-Network agent. In this study,
the estimated uncertainty is used to choose safe actions in unknown situations.
However, the uncertainty information could also be used to identify situations
that should be added to the training process.

negative:
Obtaining policies that can generalise to new environments in reinforcement
learning is challenging. In this work, we demonstrate that language
understanding via a reading policy learner is a promising vehicle for
generalisation to new environments. We propose a grounded policy learning
problem, Read to Fight Monsters (RTFM), in which the agent must jointly reason
over a language goal, relevant dynamics described in a document, and
environment observations. We procedurally generate environment dynamics and
corresponding language descriptions of the dynamics, such that agents must read
to understand new environment dynamics instead of memorising any particular
information. In addition, we propose txt2$\pi$, a model that captures three-way
interactions between the goal, document, and observations. On RTFM, txt2$\pi$
generalises to new environments with dynamics not seen during training via
reading. Furthermore, our model outperforms baselines such as FiLM and
language-conditioned CNNs on RTFM. Through curriculum learning, txt2$\pi$
produces policies that excel on complex RTFM tasks requiring several reasoning
and coreference steps.

negative:
We study the problem of designing AI agents that can robustly cooperate with
people in human-machine partnerships. Our work is inspired by real-life
scenarios in which an AI agent, e.g., a virtual assistant, has to cooperate
with new users after its deployment. We model this problem via a parametric MDP
framework where the parameters correspond to a user's type and characterize her
behavior. In the test phase, the AI agent has to interact with a user of
unknown type. Our approach to designing a robust AI agent relies on observing
the user's actions to make inferences about the user's type and adapting its
policy to facilitate efficient cooperation. We show that without being
adaptive, an AI agent can end up performing arbitrarily bad in the test phase.
We develop two algorithms for computing policies that automatically adapt to
the user in the test phase. We demonstrate the effectiveness of our approach in
solving a two-agent collaborative task.

negative:
We tackle a task where an agent learns to navigate in a 2D maze-like
environment called XWORLD. In each session, the agent perceives a sequence of
raw-pixel frames, a natural language command issued by a teacher, and a set of
rewards. The agent learns the teacher's language from scratch in a grounded and
compositional manner, such that after training it is able to correctly execute
zero-shot commands: 1) the combination of words in the command never appeared
before, and/or 2) the command contains new object concepts that are learned
from another task but never learned from navigation. Our deep framework for the
agent is trained end to end: it learns simultaneously the visual
representations of the environment, the syntax and semantics of the language,
and the action module that outputs actions. The zero-shot learning capability
of our framework results from its compositionality and modularity with
parameter tying. We visualize the intermediate outputs of the framework,
demonstrating that the agent truly understands how to solve the problem. We
believe that our results provide some preliminary insights on how to train an
agent with similar abilities in a 3D environment.

negative:
Assistive robotic arms enable users with physical disabilities to perform
everyday tasks without relying on a caregiver. Unfortunately, the very
dexterity that makes these arms useful also makes them challenging to
teleoperate: the robot has more degrees-of-freedom than the human can directly
coordinate with a handheld joystick. Our insight is that we can make assistive
robots easier for humans to control by leveraging latent actions. Latent
actions provide a low-dimensional embedding of high-dimensional robot behavior:
for example, one latent dimension might guide the assistive arm along a pouring
motion. In this paper, we design a teleoperation algorithm for assistive robots
that learns latent actions from task demonstrations. We formulate the
controllability, consistency, and scaling properties that user-friendly latent
actions should have, and evaluate how different low-dimensional embeddings
capture these properties. Finally, we conduct two user studies on a robotic arm
to compare our latent action approach to both state-of-the-art shared autonomy
baselines and a teleoperation strategy currently used by assistive arms.
Participants completed assistive eating and cooking tasks more efficiently when
leveraging our latent actions, and also subjectively reported that latent
actions made the task easier to perform. The video accompanying this paper can
be found at: https://youtu.be/wjnhrzugBj4.

negative:
Building intelligent agents that can communicate with and learn from humans
in natural language is of great value. Supervised language learning is limited
by the ability of capturing mainly the statistics of training data, and is
hardly adaptive to new scenarios or flexible for acquiring new knowledge
without inefficient retraining or catastrophic forgetting. We highlight the
perspective that conversational interaction serves as a natural interface both
for language learning and for novel knowledge acquisition and propose a joint
imitation and reinforcement approach for grounded language learning through an
interactive conversational game. The agent trained with this approach is able
to actively acquire information by asking questions about novel objects and use
the just-learned knowledge in subsequent conversations in a one-shot fashion.
Results compared with other methods verified the effectiveness of the proposed
approach.

negative:
In the vision and language navigation task, the agent may encounter ambiguous
situations that are hard to interpret by just relying on visual information and
natural language instructions. We propose an interactive learning framework to
endow the agent with the ability to ask for users' help in such situations. As
part of this framework, we investigate multiple learning approaches for the
agent with different levels of complexity. The simplest model-confusion-based
method lets the agent ask questions based on its confusion, relying on the
predefined confidence threshold of a next action prediction model. To build on
this confusion-based method, the agent is expected to demonstrate more
sophisticated reasoning such that it discovers the timing and locations to
interact with a human. We achieve this goal using reinforcement learning (RL)
with a proposed reward shaping term, which enables the agent to ask questions
only when necessary. The success rate can be boosted by at least 15% with only
one question asked on average during the navigation. Furthermore, we show that
the RL agent is capable of adjusting dynamically to noisy human responses.
Finally, we design a continual learning strategy, which can be viewed as a data
augmentation method, for the agent to improve further utilizing its interaction
history with a human. We demonstrate the proposed strategy is substantially
more realistic and data-efficient compared to previously proposed
pre-exploration techniques.

negative:
Modern robotics applications that involve human-robot interaction require
robots to be able to communicate with humans seamlessly and effectively.
Natural language provides a flexible and efficient medium through which robots
can exchange information with their human partners. Significant advancements
have been made in developing robots capable of interpreting free-form
instructions, but less attention has been devoted to endowing robots with the
ability to generate natural language. We propose a navigational guide model
that enables robots to generate natural language instructions that allow humans
to navigate a priori unknown environments. We first decide which information to
share with the user according to their preferences, using a policy trained from
human demonstrations via inverse reinforcement learning. We then "translate"
this information into a natural language instruction using a neural
sequence-to-sequence model that learns to generate free-form instructions from
natural language corpora. We evaluate our method on a benchmark route
instruction dataset and achieve a BLEU score of 72.18% when compared to
human-generated reference instructions. We additionally conduct navigation
experiments with human participants that demonstrate that our method generates
instructions that people follow as accurately and easily as those produced by
humans.

negative:
As environments involving both robots and humans become increasingly common,
so does the need to account for people during planning. To plan effectively,
robots must be able to respond to and sometimes influence what humans do. This
requires a human model which predicts future human actions. A simple model may
assume the human will continue what they did previously; a more complex one
might predict that the human will act optimally, disregarding the robot;
whereas an even more complex one might capture the robot's ability to influence
the human. These models make different trade-offs between computational time
and performance of the resulting robot plan. Using only one model of the human
either wastes computational resources or is unable to handle critical
situations. In this work, we give the robot access to a suite of human models
and enable it to assess the performance-computation trade-off online. By
estimating how an alternate model could improve human prediction and how that
may translate to performance gain, the robot can dynamically switch human
models whenever the additional computation is justified. Our experiments in a
driving simulator showcase how the robot can achieve performance comparable to
always using the best human model, but with greatly reduced computation.

negative:
When deploying autonomous agents in the real world, we need effective ways of
communicating objectives to them. Traditional skill learning has revolved
around reinforcement and imitation learning, each with rigid constraints on the
format of information exchanged between the human and the agent. While scalar
rewards carry little information, demonstrations require significant effort to
provide and may carry more information than is necessary. Furthermore, rewards
and demonstrations are often defined and collected before training begins, when
the human is most uncertain about what information would help the agent. In
contrast, when humans communicate objectives with each other, they make use of
a large vocabulary of informative behaviors, including non-verbal
communication, and often communicate throughout learning, responding to
observed behavior. In this way, humans communicate intent with minimal effort.
In this paper, we propose such interactive learning as an alternative to reward
or demonstration-driven learning. To accomplish this, we introduce a
multi-agent training framework that enables an agent to learn from another
agent who knows the current task. Through a series of experiments, we
demonstrate the emergence of a variety of interactive learning behaviors,
including information-sharing, information-seeking, and question-answering.
Most importantly, we find that our approach produces an agent that is capable
of learning interactively from a human user, without a set of explicit
demonstrations or a reward function, and achieving significantly better
performance cooperatively with a human than a human performing the task alone.

negative:
We present a hierarchical planning and control framework that enables an
agent to perform various tasks and adapt to a new task flexibly. Rather than
learning an individual policy for each particular task, the proposed framework,
DISH, distills a hierarchical policy from a set of tasks by representation and
reinforcement learning. The framework is based on the idea of latent variable
models that represent high-dimensional observations using low-dimensional
latent variables. The resulting policy consists of two levels of hierarchy: (i)
a planning module that reasons a sequence of latent intentions that would lead
to an optimistic future and (ii) a feedback control policy, shared across the
tasks, that executes the inferred intention. Because the planning is performed
in low-dimensional latent space, the learned policy can immediately be used to
solve or adapt to new tasks without additional training. We demonstrate the
proposed framework can learn compact representations (3- and 1-dimensional
latent states and commands for a humanoid with 197- and 36-dimensional state
features and actions) while solving a small number of imitation tasks, and the
resulting policy is directly applicable to other types of tasks, i.e.,
navigation in cluttered environments. Video: https://youtu.be/HQsQysUWOhg

negative:
This paper presents a research platform that supports spoken dialogue
interaction with multiple robots. The demonstration showcases our crafted
MultiBot testing scenario in which users can verbally issue search, navigate,
and follow instructions to two robotic teammates: a simulated ground robot and
an aerial robot. This flexible language and robotic platform takes advantage of
existing tools for speech recognition and dialogue management that are
compatible with new domains, and implements an inter-agent communication
protocol (tactical behavior specification), where verbal instructions are
encoded for tasks assigned to the appropriate robot.

negative:
We demonstrate a reinforcement learning agent which uses a compositional
recurrent neural network that takes as input an LTL formula and determines
satisfying actions. The input LTL formulas have never been seen before, yet the
network performs zero-shot generalization to satisfy them. This is a novel form
of multi-task learning for RL agents where agents learn from one diverse set of
tasks and generalize to a new set of diverse tasks. The formulation of the
network enables this capacity to generalize. We demonstrate this ability in two
domains. In a symbolic domain, the agent finds a sequence of letters that is
accepted. In a Minecraft-like environment, the agent finds a sequence of
actions that conform to the formula. While prior work could learn to execute
one formula reliably given examples of that formula, we demonstrate how to
encode all formulas reliably. This could form the basis of new multitask agents
that discover sub-tasks and execute them without any additional training, as
well as the agents which follow more complex linguistic commands. The
structures required for this generalization are specific to LTL formulas, which
opens up an interesting theoretical question: what structures are required in
neural networks for zero-shot generalization to different logics?

negative:
We present a novel approach to shared control of human-machine systems. Our
method assumes no a priori knowledge of the system dynamics. Instead, we learn
both the dynamics and information about the user's interaction from observation
through the use of the Koopman operator. Using the learned model, we define an
optimization problem to compute the optimal policy for a given task, and
compare the user input to the optimal input. We demonstrate the efficacy of our
approach with a user study. We also analyze the individual nature of the
learned models by comparing the effectiveness of our approach when the
demonstration data comes from a user's own interactions, from the interactions
of a group of users and from a domain expert. Positive results include
statistically significant improvements on task metrics when comparing a
user-only control paradigm with our shared control paradigm. Surprising results
include findings that suggest that individualizing the model based on a user's
own data does not effect the ability to learn a useful dynamic system. We
explore this tension as it relates to developing human-in-the-loop systems
further in the discussion.

negative:
Inferring intent from observed behavior has been studied extensively within
the frameworks of Bayesian inverse planning and inverse reinforcement learning.
These methods infer a goal or reward function that best explains the actions of
the observed agent, typically a human demonstrator. Another agent can use this
inferred intent to predict, imitate, or assist the human user. However, a
central assumption in inverse reinforcement learning is that the demonstrator
is close to optimal. While models of suboptimal behavior exist, they typically
assume that suboptimal actions are the result of some type of random noise or a
known cognitive bias, like temporal inconsistency. In this paper, we take an
alternative approach, and model suboptimal behavior as the result of internal
model misspecification: the reason that user actions might deviate from
near-optimal actions is that the user has an incorrect set of beliefs about the
rules -- the dynamics -- governing how actions affect the environment. Our
insight is that while demonstrated actions may be suboptimal in the real world,
they may actually be near-optimal with respect to the user's internal model of
the dynamics. By estimating these internal beliefs from observed behavior, we
arrive at a new method for inferring intent. We demonstrate in simulation and
in a user study with 12 participants that this approach enables us to more
accurately model human intent, and can be used in a variety of applications,
including offering assistance in a shared autonomy framework and inferring
human preferences.

negative:
Designing reward functions is difficult: the designer has to specify what to
do (what it means to complete the task) as well as what not to do (side effects
that should be avoided while completing the task). To alleviate the burden on
the reward designer, we propose an algorithm to automatically generate an
auxiliary reward function that penalizes side effects. This auxiliary objective
rewards the ability to complete possible future tasks, which decreases if the
agent causes side effects during the current task. The future task reward can
also give the agent an incentive to interfere with events in the environment
that make future tasks less achievable, such as irreversible actions by other
agents. To avoid this interference incentive, we introduce a baseline policy
that represents a default course of action (such as doing nothing), and use it
to filter out future tasks that are not achievable by default. We formally
define interference incentives and show that the future task approach with a
baseline policy avoids these incentives in the deterministic case. Using
gridworld environments that test for side effects and interference, we show
that our method avoids interference and is more effective for avoiding side
effects than the common approach of penalizing irreversible actions.

negative:
While reinforcement learning has achieved considerable successes in recent
years, state-of-the-art models are often still limited by the size of state and
action spaces. Model-free reinforcement learning approaches use some form of
state representations and the latest work has explored embedding techniques for
actions, both with the aim of achieving better generalization and
applicability. However, these approaches consider only states or actions,
ignoring the interaction between them when generating embedded representations.
In this work, we establish the theoretical foundations for the validity of
training a reinforcement learning agent using embedded states and actions. We
then propose a new approach for jointly learning embeddings for states and
actions that combines aspects of model-free and model-based reinforcement
learning, which can be applied in both discrete and continuous domains.
Specifically, we use a model of the environment to obtain embeddings for states
and actions and present a generic architecture that leverages these to learn a
policy. In this way, the embedded representations obtained via our approach
enable better generalization over both states and actions by capturing
similarities in the embedding spaces. Evaluations of our approach on several
gaming, robotic control, and recommender systems show it significantly
outperforms state-of-the-art models in both discrete/continuous domains with
large state/action spaces, thus confirming its efficacy.

negative:
Maximizing the utility of human-robot teams in disaster response and search
and rescue (SAR) missions remains to be a challenging problem. This is due to
the dynamic, uncertain nature of the environment and the variability in
cognitive performance of the human operators. By having an autonomous agent
share control with the operator, we can achieve near-optimal performance by
augmenting the operator's input and compensate for the factors resulting in
degraded performance. What this solution does not consider though is the human
input latency and errors caused by potential hardware failures that can occur
during task completion when operating in disaster response and SAR scenarios.
In this paper, we propose the use of blended shared control (BSC) architecture
to address these issues and investigate the architecture's performance in
constrained, dynamic environments with a differential drive robot that has
input latency and erroneous odometry feedback. We conduct a validation study
(n=12) for our control architecture and then a user study (n=14) in 2 different
environments that are unknown to both the human operator and the autonomous
agent. The results demonstrate that the BSC architecture can prevent collisions
and enhance operator performance without the need of a complete transfer of
control between the human operator and autonomous agent.

negative:
One obstacle to applying reinforcement learning algorithms to real-world
problems is the lack of suitable reward functions. Designing such reward
functions is difficult in part because the user only has an implicit
understanding of the task objective. This gives rise to the agent alignment
problem: how do we create agents that behave in accordance with the user's
intentions? We outline a high-level research direction to solve the agent
alignment problem centered around reward modeling: learning a reward function
from interaction with the user and optimizing the learned reward function with
reinforcement learning. We discuss the key challenges we expect to face when
scaling reward modeling to complex and general domains, concrete approaches to
mitigate these challenges, and ways to establish trust in the resulting agents.

negative:
Learning predictive models from interaction with the world allows an agent,
such as a robot, to learn about how the world works, and then use this learned
model to plan coordinated sequences of actions to bring about desired outcomes.
However, learning a model that captures the dynamics of complex skills
represents a major challenge: if the agent needs a good model to perform these
skills, it might never be able to collect the experience on its own that is
required to learn these delicate and complex behaviors. Instead, we can imagine
augmenting the training set with observational data of other agents, such as
humans. Such data is likely more plentiful, but represents a different
embodiment. For example, videos of humans might show a robot how to use a tool,
but (i) are not annotated with suitable robot actions, and (ii) contain a
systematic distributional shift due to the embodiment differences between
humans and robots. We address the first challenge by formulating the
corresponding graphical model and treating the action as an observed variable
for the interaction data and an unobserved variable for the observation data,
and the second challenge by using a domain-dependent prior. In addition to
interaction data, our method is able to leverage videos of passive observations
in a driving dataset and a dataset of robotic manipulation videos. A robotic
planning agent equipped with our method can learn to use tools in a tabletop
robotic manipulation setting by observing humans without ever seeing a robotic
video of tool use.

negative:
To interact with humans in collaborative environments, machines need to be
able to predict (i.e., anticipate) future events, and execute actions in a
timely manner. However, the observation of the human limb movements may not be
sufficient to anticipate their actions unambiguously. In this work, we consider
two additional sources of information (i.e., context) over time, gaze, movement
and object information, and study how these additional contextual cues improve
the action anticipation performance. We address action anticipation as a
classification task, where the model takes the available information as the
input and predicts the most likely action. We propose to use the uncertainty
about each prediction as an online decision-making criterion for action
anticipation. Uncertainty is modeled as a stochastic process applied to a
time-based neural network architecture, which improves the conventional
class-likelihood (i.e., deterministic) criterion. The main contributions of
this paper are four-fold: (i) We propose a novel and effective decision-making
criterion that can be used to anticipate actions even in situations of high
ambiguity; (ii) we propose a deep architecture that outperforms previous
results in the action anticipation task when using the Acticipate collaborative
dataset; (iii) we show that contextual information is important to disambiguate
the interpretation of similar actions; and (iv) we also provide a formal
description of three existing performance metrics that can be easily used to
evaluate action anticipation models.Our results on the Acticipate dataset
showed the importance of contextual information and the uncertainty criterion
for action anticipation. We achieve an average accuracy of 98.75% in the
anticipation task using only an average of 25% of observations.

negative:
Contemporary coding education often presents students with the task of
developing programs that have user interaction and complex dynamic systems,
such as mouse based games. While pedagogically compelling, there are no
contemporary autonomous methods for providing feedback. Notably, interactive
programs are impossible to grade by traditional unit tests. In this paper we
formalize the challenge of providing feedback to interactive programs as a task
of classifying Markov Decision Processes (MDPs). Each student's program fully
specifies an MDP where the agent needs to operate and decide, under reasonable
generalization, if the dynamics and reward model of the input MDP should be
categorized as correct or broken. We demonstrate that by designing a
cooperative objective between an agent and an autoregressive model, we can use
the agent to sample differential trajectories from the input MDP that allows a
classifier to determine membership: Play to Grade. Our method enables an
automatic feedback system for interactive code assignments. We release a
dataset of 711,274 anonymized student submissions to a single assignment with
hand-coded bug labels to support future research.

negative:
Autonomous driving is a complex task, which has been tackled since the first
self-driving car ALVINN in 1989, with a supervised learning approach, or
behavioral cloning (BC). In BC, a neural network is trained with state-action
pairs that constitute the training set made by an expert, i.e., a human driver.
However, this type of imitation learning does not take into account the
temporal dependencies that might exist between actions taken in different
moments of a navigation trajectory. These type of tasks are better handled by
reinforcement learning (RL) algorithms, which need to define a reward function.
On the other hand, more recent approaches to imitation learning, such as
Generative Adversarial Imitation Learning (GAIL), can train policies without
explicitly requiring to define a reward function, allowing an agent to learn by
trial and error directly on a training set of expert trajectories. In this
work, we propose two variations of GAIL for autonomous navigation of a vehicle
in the realistic CARLA simulation environment for urban scenarios. Both of them
use the same network architecture, which process high dimensional image input
from three frontal cameras, and other nine continuous inputs representing the
velocity, the next point from the sparse trajectory and a high-level driving
command. We show that both of them are capable of imitating the expert
trajectory from start to end after training ends, but the GAIL loss function
that is augmented with BC outperforms the former in terms of convergence time
and training stability.

negative:
We analyze the language learned by an agent trained with reinforcement
learning as a component of the ActiveQA system [Buck et al., 2017]. In
ActiveQA, question answering is framed as a reinforcement learning task in
which an agent sits between the user and a black box question-answering system.
The agent learns to reformulate the user's questions to elicit the optimal
answers. It probes the system with many versions of a question that are
generated via a sequence-to-sequence question reformulation model, then
aggregates the returned evidence to find the best answer. This process is an
instance of \emph{machine-machine} communication. The question reformulation
model must adapt its language to increase the quality of the answers returned,
matching the language of the question answering system. We find that the agent
does not learn transformations that align with semantic intuitions but
discovers through learning classical information retrieval techniques such as
tf-idf re-weighting and stemming.

negative:
Despite widespread interests in reinforcement-learning for task-oriented
dialogue systems, several obstacles can frustrate research and development
progress. First, reinforcement learners typically require interaction with the
environment, so conventional dialogue corpora cannot be used directly. Second,
each task presents specific challenges, requiring separate corpus of
task-specific annotated data. Third, collecting and annotating human-machine or
human-human conversations for task-oriented dialogues requires extensive domain
knowledge. Because building an appropriate dataset can be both financially
costly and time-consuming, one popular approach is to build a user simulator
based upon a corpus of example dialogues. Then, one can train reinforcement
learning agents in an online fashion as they interact with the simulator.
Dialogue agents trained on these simulators can serve as an effective starting
point. Once agents master the simulator, they may be deployed in a real
environment to interact with humans, and continue to be trained online. To ease
empirical algorithmic comparisons in dialogues, this paper introduces a new,
publicly available simulation framework, where our simulator, designed for the
movie-booking domain, leverages both rules and collected data. The simulator
supports two tasks: movie ticket booking and movie seeking. Finally, we
demonstrate several agents and detail the procedure to add and test your own
agent in the proposed framework.

negative:
Deep learning often requires the manual collection and annotation of a
training set. On robotic platforms, can we partially automate this task by
training the robot to be curious, i.e., to seek out beneficial training
information in the environment? In this work, we address the problem of
curiosity as it relates to online, real-time, human-in-the-loop training of an
object detection algorithm onboard a drone, where motion is constrained to two
dimensions. We use a 3D simulation environment and deep reinforcement learning
to train a curiosity agent to, in turn, train the object detection model. This
agent could have one of two conflicting objectives: train as quickly as
possible, or train with minimal human input. We outline a reward function that
allows the curiosity agent to learn either of these objectives, while taking
into account some of the physical characteristics of the drone platform on
which it is meant to run. In addition, We show that we can weigh the importance
of achieving these objectives by adjusting a parameter in the reward function.

negative:
Multi-agent reinforcement learning involves multiple agents interacting with
each other and a shared environment to complete tasks. When rewards provided by
the environment are sparse, agents may not receive immediate feedback on the
quality of actions that they take, thereby affecting learning of policies. In
this paper, we propose a method called Shaping Advice in deep Multi-agent
reinforcement learning (SAM) to augment the reward signal from the environment
with an additional reward termed shaping advice. The shaping advice is given by
a difference of potential functions at consecutive time-steps. Each potential
function is a function of observations and actions of the agents. The shaping
advice needs to be specified only once at the start of training, and can be
easily provided by non-experts. We show through theoretical analyses and
experimental validation that shaping advice provided by SAM does not distract
agents from completing tasks specified by the environment reward.
Theoretically, we prove that convergence of policy gradients and value
functions when using SAM implies convergence of these quantities in the absence
of SAM. Experimentally, we evaluate SAM on three tasks in the multi-agent
Particle World environment that have sparse rewards. We observe that using SAM
results in agents learning policies to complete tasks faster, and obtain higher
rewards than: i) using sparse rewards alone; ii) a state-of-the-art reward
redistribution method.

negative:
Most model-free reinforcement learning methods leverage state representations
(embeddings) for generalization, but either ignore structure in the space of
actions or assume the structure is provided a priori. We show how a policy can
be decomposed into a component that acts in a low-dimensional space of action
representations and a component that transforms these representations into
actual actions. These representations improve generalization over large, finite
action sets by allowing the agent to infer the outcomes of actions similar to
actions already taken. We provide an algorithm to both learn and use action
representations and provide conditions for its convergence. The efficacy of the
proposed method is demonstrated on large-scale real-world problems.

negative:
The intuitive collaboration of humans and intelligent robots (embodied AI) in
the real-world is an essential objective for many desirable applications of
robotics. Whilst there is much research regarding explicit communication, we
focus on how humans and robots interact implicitly, on motor adaptation level.
We present a real-world setup of a human-robot collaborative maze game,
designed to be non-trivial and only solvable through collaboration, by limiting
the actions to rotations of two orthogonal axes, and assigning each axes to one
player. This results in neither the human nor the agent being able to solve the
game on their own. We use deep reinforcement learning for the control of the
robotic agent, and achieve results within 30 minutes of real-world play,
without any type of pre-training. We then use this setup to perform systematic
experiments on human/agent behaviour and adaptation when co-learning a policy
for the collaborative game. We present results on how co-policy learning occurs
over time between the human and the robotic agent resulting in each
participant's agent serving as a representation of how they would play the
game. This allows us to relate a person's success when playing with different
agents than their own, by comparing the policy of the agent with that of their
own agent.

negative:
A common vision from science fiction is that robots will one day inhabit our
physical spaces, sense the world as we do, assist our physical labours, and
communicate with us through natural language. Here we study how to design
artificial agents that can interact naturally with humans using the
simplification of a virtual environment. This setting nevertheless integrates a
number of the central challenges of artificial intelligence (AI) research:
complex visual perception and goal-directed physical control, grounded language
comprehension and production, and multi-agent social interaction. To build
agents that can robustly interact with humans, we would ideally train them
while they interact with humans. However, this is presently impractical.
Therefore, we approximate the role of the human with another learned agent, and
use ideas from inverse reinforcement learning to reduce the disparities between
human-human and agent-agent interactive behaviour. Rigorously evaluating our
agents poses a great challenge, so we develop a variety of behavioural tests,
including evaluation by humans who watch videos of agents or interact directly
with them. These evaluations convincingly demonstrate that interactive training
and auxiliary losses improve agent behaviour beyond what is achieved by
supervised learning of actions alone. Further, we demonstrate that agent
capabilities generalise beyond literal experiences in the dataset. Finally, we
train evaluation models whose ratings of agents agree well with human
judgement, thus permitting the evaluation of new agent models without
additional effort. Taken together, our results in this virtual environment
provide evidence that large-scale human behavioural imitation is a promising
tool to create intelligent, interactive agents, and the challenge of reliably
evaluating such agents is possible to surmount.

negative:
This paper develops a model that addresses sentence embedding, a hot topic in
current natural language processing research, using recurrent neural networks
with Long Short-Term Memory (LSTM) cells. Due to its ability to capture long
term memory, the LSTM-RNN accumulates increasingly richer information as it
goes through the sentence, and when it reaches the last word, the hidden layer
of the network provides a semantic representation of the whole sentence. In
this paper, the LSTM-RNN is trained in a weakly supervised manner on user
click-through data logged by a commercial web search engine. Visualization and
analysis are performed to understand how the embedding process works. The model
is found to automatically attenuate the unimportant words and detects the
salient keywords in the sentence. Furthermore, these detected keywords are
found to automatically activate different cells of the LSTM-RNN, where words
belonging to a similar topic activate the same cell. As a semantic
representation of the sentence, the embedding vector can be used in many
different applications. These automatic keyword detection and topic allocation
abilities enabled by the LSTM-RNN allow the network to perform document
retrieval, a difficult language processing task, where the similarity between
the query and documents can be measured by the distance between their
corresponding sentence embedding vectors computed by the LSTM-RNN. On a web
search task, the LSTM-RNN embedding is shown to significantly outperform
several existing state of the art methods. We emphasize that the proposed model
generates sentence embedding vectors that are specially useful for web document
retrieval tasks. A comparison with a well known general sentence embedding
method, the Paragraph Vector, is performed. The results show that the proposed
method in this paper significantly outperforms it for web document retrieval
task.

negative:
Writing is a complex non-linear process that begins with a mental model of
intent, and progresses through an outline of ideas, to words on paper (and
their subsequent refinement). Despite past research in understanding writing,
Web-scale consumer and enterprise collaborative digital writing environments
are yet to greatly benefit from intelligent systems that understand the stages
of document evolution, providing opportune assistance based on authors'
situated actions and context. In this paper, we present three studies that
explore temporal stages of document authoring. We first survey information
workers at a large technology company about their writing habits and
preferences, concluding that writers do in fact conceptually progress through
several distinct phases while authoring documents. We also explore,
qualitatively, how writing stages are linked to document lifespan. We
supplement these qualitative findings with an analysis of the longitudinal user
interaction logs of a popular digital writing platform over several million
documents. Finally, as a first step towards facilitating an intelligent digital
writing assistant, we conduct a preliminary investigation into the utility of
user interaction log data for predicting the temporal stage of a document. Our
results support the benefit of tools tailored to writing stages, identify
primary tasks associated with these stages, and show that it is possible to
predict stages from anonymous interaction logs. Together, these results argue
for the benefit and feasibility of more tailored digital writing assistance.

negative:
While deep reinforcement learning techniques have led to agents that are
successfully able to learn to perform a number of tasks that had been
previously unlearnable, these techniques are still susceptible to the
longstanding problem of reward sparsity. This is especially true for tasks such
as training an agent to play StarCraft II, a real-time strategy game where
reward is only given at the end of a game which is usually very long. While
this problem can be addressed through reward shaping, such approaches typically
require a human expert with specialized knowledge. Inspired by the vision of
enabling reward shaping through the more-accessible paradigm of
natural-language narration, we develop a technique that can provide the
benefits of reward shaping using natural language commands. Our
narration-guided RL agent projects sequences of natural-language commands into
the same high-dimensional representation space as corresponding goal states. We
show that we can get improved performance with our method compared to
traditional reward-shaping approaches. Additionally, we demonstrate the ability
of our method to generalize to unseen natural-language commands.

negative:
We seek to align agent behavior with a user's objectives in a reinforcement
learning setting with unknown dynamics, an unknown reward function, and unknown
unsafe states. The user knows the rewards and unsafe states, but querying the
user is expensive. To address this challenge, we propose an algorithm that
safely and interactively learns a model of the user's reward function. We start
with a generative model of initial states and a forward dynamics model trained
on off-policy data. Our method uses these models to synthesize hypothetical
behaviors, asks the user to label the behaviors with rewards, and trains a
neural network to predict the rewards. The key idea is to actively synthesize
the hypothetical behaviors from scratch by maximizing tractable proxies for the
value of information, without interacting with the environment. We call this
method reward query synthesis via trajectory optimization (ReQueST). We
evaluate ReQueST with simulated users on a state-based 2D navigation task and
the image-based Car Racing video game. The results show that ReQueST
significantly outperforms prior methods in learning reward models that transfer
to new environments with different initial state distributions. Moreover,
ReQueST safely trains the reward model to detect unsafe states, and corrects
reward hacking before deploying the agent.

negative:
We present an efficient document representation learning framework, Document
Vector through Corruption (Doc2VecC). Doc2VecC represents each document as a
simple average of word embeddings. It ensures a representation generated as
such captures the semantic meanings of the document during learning. A
corruption model is included, which introduces a data-dependent regularization
that favors informative or rare words while forcing the embeddings of common
and non-discriminative ones to be close to zero. Doc2VecC produces
significantly better word embeddings than Word2Vec. We compare Doc2VecC with
several state-of-the-art document representation learning algorithms. The
simple model architecture introduced by Doc2VecC matches or out-performs the
state-of-the-art in generating high-quality document representations for
sentiment analysis, document classification as well as semantic relatedness
tasks. The simplicity of the model enables training on billions of words per
hour on a single machine. At the same time, the model is very efficient in
generating representations of unseen documents at test time.

negative:
In this study, we investigate how a robot can generate novel and creative
actions from its own experience of learning basic actions. Inspired by a
machine learning approach to computational creativity, we propose a dynamic
neural network model that can learn and generate robot's actions. We conducted
a set of simulation experiments with a humanoid robot. The results showed that
the proposed model was able to learn the basic actions and also to generate
novel actions by modulating and combining those learned actions. The analysis
on the neural activities illustrated that the ability to generate creative
actions emerged from the model's nonlinear memory structure self-organized
during training. The results also showed that the different way of learning the
basic actions induced the self-organization of the memory structure with the
different characteristics, resulting in the generation of different levels of
creative actions. Our approach can be utilized in human-robot interaction in
which a user can interactively explore the robot's memory to control its
behavior and also discover other novel actions.

negative:
Document classification is a challenging task with important applications.
The deep learning approaches to the problem have gained much attention
recently. Despite the progress, the proposed models do not incorporate the
knowledge of the document structure in the architecture efficiently and not
take into account the contexting importance of words and sentences. In this
paper, we propose a new approach based on a combination of convolutional neural
networks, gated recurrent units, and attention mechanisms for document
classification tasks. The main contribution of this work is the use of
convolution layers to extract more meaningful, generalizable and abstract
features by the hierarchical representation. The proposed method in this paper
improves the results of the current attention-based approaches for document
classification.

negative:
This paper develops a stochastic programming framework for multi-agent
systems where task decomposition, assignment, and scheduling problems are
simultaneously optimized. The framework can be applied to heterogeneous mobile
robot teams with distributed sub-tasks. Examples include pandemic robotic
service coordination, explore and rescue, and delivery systems with
heterogeneous vehicles. Due to their inherent flexibility and robustness,
multi-agent systems are applied in a growing range of real-world problems that
involve heterogeneous tasks and uncertain information. Most previous works
assume one fixed way to decompose a task into roles that can later be assigned
to the agents. This assumption is not valid for a complex task where the roles
can vary and multiple decomposition structures exist. Meanwhile, it is unclear
how uncertainties in task requirements and agent capabilities can be
systematically quantified and optimized under a multi-agent system setting. A
representation for complex tasks is proposed: agent capabilities are
represented as a vector of random distributions, and task requirements are
verified by a generalizable binary function. The conditional value at risk
(CVaR) is chosen as a metric in the objective function to generate robust
plans. An efficient algorithm is described to solve the model, and the whole
framework is evaluated in two different practical test cases: capture-the-flag
and robotic service coordination during a pandemic (e.g., COVID-19). Results
demonstrate that the framework is generalizable, scalable up to 140 agents and
40 tasks for the example test cases, and provides low-cost plans that ensure a
high probability of success.

negative:
Reinforcement learning agents learn by encouraging behaviours which maximize
their total reward, usually provided by the environment. In many environments,
however, the reward is provided after a series of actions rather than each
single action, leading the agent to experience ambiguity in terms of whether
those actions are effective, an issue known as the credit assignment problem.
In this paper, we propose two strategies inspired by behavioural psychology to
enable the agent to intrinsically estimate more informative reward values for
actions with no reward. The first strategy, called self-punishment (SP),
discourages the agent from making mistakes that lead to undesirable terminal
states. The second strategy, called the rewards backfill (RB), backpropagates
the rewards between two rewarded actions. We prove that, under certain
assumptions and regardless of the reinforcement learning algorithm used, these
two strategies maintain the order of policies in the space of all possible
policies in terms of their total reward, and, by extension, maintain the
optimal policy. Hence, our proposed strategies integrate with any reinforcement
learning algorithm that learns a value or action-value function through
experience. We incorporated these two strategies into three popular deep
reinforcement learning approaches and evaluated the results on thirty Atari
games. After parameter tuning, our results indicate that the proposed
strategies improve the tested methods in over 65 percent of tested games by up
to over 25 times performance improvement.

negative:
Intelligent assistants that follow commands or answer simple questions, such
as Siri and Google search, are among the most economically important
applications of AI. Future conversational AI assistants promise even greater
capabilities and a better user experience through a deeper understanding of the
domain, the user, or the user's purposes. But what domain and what methods are
best suited to researching and realizing this promise? In this article we argue
for the domain of voice document editing and for the methods of model-based
reinforcement learning. The primary advantages of voice document editing are
that the domain is tightly scoped and that it provides something for the
conversation to be about (the document) that is delimited and fully accessible
to the intelligent assistant. The advantages of reinforcement learning in
general are that its methods are designed to learn from interaction without
explicit instruction and that it formalizes the purposes of the assistant.
Model-based reinforcement learning is needed in order to genuinely understand
the domain of discourse and thereby work efficiently with the user to achieve
their goals. Together, voice document editing and model-based reinforcement
learning comprise a promising research direction for achieving conversational
AI.

negative:
Conversational search is an approach to information retrieval (IR), where
users engage in a dialogue with an agent in order to satisfy their information
needs. Previous conceptual work described properties and actions a good agent
should exhibit. Unlike them, we present a novel conceptual model defined in
terms of conversational goals, which enables us to reason about current
research practices in conversational search. Based on the literature, we elicit
how existing tasks and test collections from the fields of IR, natural language
processing (NLP) and dialogue systems (DS) fit into this model. We describe a
set of characteristics that an ideal conversational search dataset should have.
Lastly, we introduce MANtIS (the code and dataset are available at
https://guzpenha.github.io/MANtIS/), a large-scale dataset containing
multi-domain and grounded information seeking dialogues that fulfill all of our
dataset desiderata. We provide baseline results for the conversation response
ranking and user intent prediction tasks.

negative:
Reinforcement learning is a powerful technique to train an agent to perform a
task. However, an agent that is trained using reinforcement learning is only
capable of achieving the single task that is specified via its reward function.
Such an approach does not scale well to settings in which an agent needs to
perform a diverse set of tasks, such as navigating to varying positions in a
room or moving objects to varying locations. Instead, we propose a method that
allows an agent to automatically discover the range of tasks that it is capable
of performing. We use a generator network to propose tasks for the agent to try
to achieve, specified as goal states. The generator network is optimized using
adversarial training to produce tasks that are always at the appropriate level
of difficulty for the agent. Our method thus automatically produces a
curriculum of tasks for the agent to learn. We show that, by using this
framework, an agent can efficiently and automatically learn to perform a wide
set of tasks without requiring any prior knowledge of its environment. Our
method can also learn to achieve tasks with sparse rewards, which traditionally
pose significant challenges.

negative:
In the Vision-and-Language Navigation (VLN) task an embodied agent navigates
a 3D environment, following natural language instructions. A challenge in this
task is how to handle 'off the path' scenarios where an agent veers from a
reference path. Prior work supervises the agent with actions based on the
shortest path from the agent's location to the goal, but such goal-oriented
supervision is often not in alignment with the instruction. Furthermore, the
evaluation metrics employed by prior work do not measure how much of a language
instruction the agent is able to follow. In this work, we propose a simple and
effective language-aligned supervision scheme, and a new metric that measures
the number of sub-instructions the agent has completed during navigation.

negative:
The role of conversational assistants has become more prevalent in helping
people increase their productivity. Document-centered assistance, for example
to help an individual quickly review a document, has seen less significant
progress, even though it has the potential to tremendously increase a user's
productivity. This type of document-centered assistance is the focus of this
paper. Our contributions are three-fold: (1) We first present a survey to
understand the space of document-centered assistance and the capabilities
people expect in this scenario. (2) We investigate the types of queries that
users will pose while seeking assistance with documents, and show that
document-centered questions form the majority of these queries. (3) We present
a set of initial machine learned models that show that (a) we can accurately
detect document-centered questions, and (b) we can build reasonably accurate
models for answering such questions. These positive results are encouraging,
and suggest that even greater results may be attained with continued study of
this interesting and novel problem space. Our findings have implications for
the design of intelligent systems to support task completion via natural
interactions with documents.

negative:
With the increase of information, document classification as one of the
methods of text mining, plays vital role in many management and organizing
information. Document classification is the process of assigning a document to
one or more predefined category labels. Document classification includes
different parts such as text processing, term selection, term weighting and
final classification. The accuracy of document classification is very
important. Thus improvement in each part of classification should lead to
better results and higher precision. Term weighting has a great impact on the
accuracy of the classification. Most of the existing weighting methods exploit
the statistical information of terms in documents and do not consider semantic
relations between words. In this paper, an automated document classification
system is presented that uses a novel term weighting method based on semantic
relations between terms. To evaluate the proposed method, three standard
Persian corpuses are used. Experiment results show 2 to 4 percent improvement
in classification accuracy compared with the best previous designed system for
Persian documents.

negative:
How to leverage cross-document interactions to improve ranking performance is
an important topic in information retrieval (IR) research. However, this topic
has not been well-studied in the learning-to-rank setting and most of the
existing work still treats each document independently while scoring. The
recent development of deep learning shows strength in modeling complex
relationships across sequences and sets. It thus motivates us to study how to
leverage cross-document interactions for learning-to-rank in the deep learning
framework. In this paper, we formally define the permutation-equivariance
requirement for a scoring function that captures cross-document interactions.
We then propose a self-attention based document interaction network and show
that it satisfies the permutation-equivariant requirement, and can generate
scores for document sets of varying sizes. Our proposed methods can
automatically learn to capture document interactions without any auxiliary
information, and can scale across large document sets. We conduct experiments
on three ranking datasets: the benchmark Web30k, a Gmail search, and a Google
Drive Quick Access dataset. Experimental results show that our proposed methods
are both more effective and efficient than baselines.

negative:
Document network embedding aims at learning representations for a structured
text corpus i.e. when documents are linked to each other. Recent algorithms
extend network embedding approaches by incorporating the text content
associated with the nodes in their formulations. In most cases, it is hard to
interpret the learned representations. Moreover, little importance is given to
the generalization to new documents that are not observed within the network.
In this paper, we propose an interpretable and inductive document network
embedding method. We introduce a novel mechanism, the Topic-Word Attention
(TWA), that generates document representations based on the interplay between
word and topic representations. We train these word and topic vectors through
our general model, Inductive Document Network Embedding (IDNE), by leveraging
the connections in the document network. Quantitative evaluations show that our
approach achieves state-of-the-art performance on various networks and we
qualitatively show that our model produces meaningful and interpretable
representations of the words, topics and documents.

negative:
An accurate abstractive summary of a document should contain all its salient
information and should be logically entailed by the input document. We improve
these important aspects of abstractive summarization via multi-task learning
with the auxiliary tasks of question generation and entailment generation,
where the former teaches the summarization model how to look for salient
questioning-worthy details, and the latter teaches the model how to rewrite a
summary which is a directed-logical subset of the input document. We also
propose novel multi-task architectures with high-level (semantic)
layer-specific sharing across multiple encoder and decoder layers of the three
tasks, as well as soft-sharing mechanisms (and show performance ablations and
analysis examples of each contribution). Overall, we achieve statistically
significant improvements over the state-of-the-art on both the CNN/DailyMail
and Gigaword datasets, as well as on the DUC-2002 transfer setup. We also
present several quantitative and qualitative analysis studies of our model's
learned saliency and entailment skills.

negative:
We propose a Label Propagation based algorithm for weakly supervised text
classification. We construct a graph where each document is represented by a
node and edge weights represent similarities among the documents. Additionally,
we discover underlying topics using Latent Dirichlet Allocation (LDA) and
enrich the document graph by including the topics in the form of additional
nodes. The edge weights between a topic and a text document represent level of
"affinity" between them. Our approach does not require document level
labelling, instead it expects manual labels only for topic nodes. This
significantly minimizes the level of supervision needed as only a few topics
are observed to be enough for achieving sufficiently high accuracy. The Label
Propagation Algorithm is employed on this enriched graph to propagate labels
among the nodes. Our approach combines the advantages of Label Propagation
(through document-document similarities) and Topic Modelling (for minimal but
smart supervision). We demonstrate the effectiveness of our approach on various
datasets and compare with state-of-the-art weakly supervised text
classification approaches.

negative:
One of the major prerequisites for any deep learning approach is the
availability of large-scale training data. When dealing with scanned document
images in real world scenarios, the principal information of its content is
stored in the layout itself. In this work, we have proposed an automated deep
generative model using Graph Neural Networks (GNNs) to generate synthetic data
with highly variable and plausible document layouts that can be used to train
document interpretation systems, in this case, specially in digital mailroom
applications. It is also the first graph-based approach for document layout
generation task experimented on administrative document images, in this case,
invoices.

negative:
This paper develops an inverse reinforcement learning algorithm aimed at
recovering a reward function from the observed actions of an agent. We
introduce a strategy to flexibly handle different types of actions with two
approximations of the Bellman Optimality Equation, and a Bellman Gradient
Iteration method to compute the gradient of the Q-value with respect to the
reward function. These methods allow us to build a differentiable relation
between the Q-value and the reward function and learn an approximately optimal
reward function with gradient methods. We test the proposed method in two
simulated environments by evaluating the accuracy of different approximations
and comparing the proposed method with existing solutions. The results show
that even with a linear reward function, the proposed method has a comparable
accuracy with the state-of-the-art method adopting a non-linear reward
function, and the proposed method is more flexible because it is defined on
observed actions instead of trajectories.

negative:
In many robotic applications, an autonomous agent must act within and explore
a partially observed environment that is unobserved by its human teammate. We
consider such a setting in which the agent can, while acting, transmit
declarative information to the human that helps them understand aspects of this
unseen environment. In this work, we address the algorithmic question of how
the agent should plan out what actions to take and what information to
transmit. Naturally, one would expect the human to have preferences, which we
model information-theoretically by scoring transmitted information based on the
change it induces in weighted entropy of the human's belief state. We formulate
this setting as a belief MDP and give a tractable algorithm for solving it
approximately. Then, we give an algorithm that allows the agent to learn the
human's preferences online, through exploration. We validate our approach
experimentally in simulated discrete and continuous partially observed
search-and-recover domains. Visit http://tinyurl.com/chitnis-corl-18 for a
supplementary video.

negative:
Feature based local attribution methods are amongst the most prevalent in
explainable artificial intelligence (XAI) literature. Going beyond standard
correlation, recently, methods have been proposed that highlight what should be
minimally sufficient to justify the classification of an input (viz. pertinent
positives). While minimal sufficiency is an attractive property, the resulting
explanations are often too sparse for a human to understand and evaluate the
local behavior of the model, thus making it difficult to judge its overall
quality. To overcome these limitations, we propose a novel method called
Path-Sufficient Explanations Method (PSEM) that outputs a sequence of
sufficient explanations for a given input of strictly decreasing size (or
value) -- from original input to a minimally sufficient explanation -- which
can be thought to trace the local boundary of the model in a smooth manner,
thus providing better intuition about the local model behavior for the specific
input. We validate these claims, both qualitatively and quantitatively, with
experiments that show the benefit of PSEM across all three modalities (image,
tabular and text). A user study depicts the strength of the method in
communicating the local behavior, where (many) users are able to correctly
determine the prediction made by a model.

negative:
When estimating the relevancy between a query and a document, ranking models
largely neglect the mutual information among documents. A common wisdom is that
if two documents are similar in terms of the same query, they are more likely
to have similar relevance score. To mitigate this problem, in this paper, we
propose a multi-agent reinforced ranking model, named MarlRank. In particular,
by considering each document as an agent, we formulate the ranking process as a
multi-agent Markov Decision Process (MDP), where the mutual interactions among
documents are incorporated in the ranking process. To compute the ranking list,
each document predicts its relevance to a query considering not only its own
query-document features but also its similar documents features and actions. By
defining reward as a function of NDCG, we can optimize our model directly on
the ranking performance measure. Our experimental results on two LETOR
benchmark datasets show that our model has significant performance gains over
the state-of-art baselines. We also find that the NDCG shows an overall
increasing trend along with the step of interactions, which demonstrates that
the mutual information among documents helps improve the ranking performance.

negative:
Recent approaches in literature have exploited the multi-modal information in
documents (text, layout, image) to serve specific downstream document tasks.
However, they are limited by their - (i) inability to learn cross-modal
representations across text, layout and image dimensions for documents and (ii)
inability to process multi-page documents. Pre-training techniques have been
shown in Natural Language Processing (NLP) domain to learn generic textual
representations from large unlabelled datasets, applicable to various
downstream NLP tasks. In this paper, we propose a multi-task learning-based
framework that utilizes a combination of self-supervised and supervised
pre-training tasks to learn a generic document representation applicable to
various downstream document tasks. Specifically, we introduce Document Topic
Modelling and Document Shuffle Prediction as novel pre-training tasks to learn
rich image representations along with the text and layout representations for
documents. We utilize the Longformer network architecture as the backbone to
encode the multi-modal information from multi-page documents in an end-to-end
fashion. We showcase the applicability of our pre-training framework on a
variety of different real-world document tasks such as document classification,
document information extraction, and document retrieval. We evaluate our
framework on different standard document datasets and conduct exhaustive
experiments to compare performance against various ablations of our framework
and state-of-the-art baselines.

negative:
A major challenge of multiagent reinforcement learning (MARL) is the curse of
multiagents, where the size of the joint action space scales exponentially with
the number of agents. This remains to be a bottleneck for designing efficient
MARL algorithms even in a basic scenario with finitely many states and actions.
This paper resolves this challenge for the model of episodic Markov games. We
design a new class of fully decentralized algorithms -- V-learning, which
provably learns Nash equilibria (in the two-player zero-sum setting),
correlated equilibria and coarse correlated equilibria (in the multiplayer
general-sum setting) in a number of samples that only scales with
$\max_{i\in[m]} A_i$, where $A_i$ is the number of actions for the $i^{\rm th}$
player. This is in sharp contrast to the size of the joint action space which
is $\prod_{i=1}^m A_i$. V-learning (in its basic form) is a new class of
single-agent RL algorithms that convert any adversarial bandit algorithm with
suitable regret guarantees into a RL algorithm. Similar to the classical
Q-learning algorithm, it performs incremental updates to the value functions.
Different from Q-learning, it only maintains the estimates of V-values instead
of Q-values. This key difference allows V-learning to achieve the claimed
guarantees in the MARL setting by simply letting all agents run V-learning
independently.

negative:
A recent trend in IR has been the usage of neural networks to learn retrieval
models for text based adhoc search. While various approaches and architectures
have yielded significantly better performance than traditional retrieval models
such as BM25, it is still difficult to understand exactly why a document is
relevant to a query. In the ML community several approaches for explaining
decisions made by deep neural networks have been proposed -- including DeepSHAP
which modifies the DeepLift algorithm to estimate the relative importance
(shapley values) of input features for a given decision by comparing the
activations in the network for a given image against the activations caused by
a reference input. In image classification, the reference input tends to be a
plain black image. While DeepSHAP has been well studied for image
classification tasks, it remains to be seen how we can adapt it to explain the
output of Neural Retrieval Models (NRMs). In particular, what is a good "black"
image in the context of IR? In this paper we explored various reference input
document construction techniques. Additionally, we compared the explanations
generated by DeepSHAP to LIME (a model agnostic approach) and found that the
explanations differ considerably. Our study raises concerns regarding the
robustness and accuracy of explanations produced for NRMs. With this paper we
aim to shed light on interesting problems surrounding interpretability in NRMs
and highlight areas of future work.

negative:
BERT based ranking models have achieved superior performance on various
information retrieval tasks. However, the large number of parameters and
complex self-attention operation come at a significant latency overhead. To
remedy this, recent works propose late-interaction architectures, which allow
pre-computation of intermediate document representations, thus reducing the
runtime latency. Nonetheless, having solved the immediate latency issue, these
methods now introduce storage costs and network fetching latency, which limits
their adoption in real-life production systems.
  In this work, we propose the Succinct Document Representation (SDR) scheme
that computes highly compressed intermediate document representations,
mitigating the storage/network issue. Our approach first reduces the dimension
of token representations by encoding them using a novel autoencoder
architecture that uses the document's textual content in both the encoding and
decoding phases. After this token encoding step, we further reduce the size of
entire document representations using a modern quantization technique.
  Extensive evaluations on passage re-reranking on the MSMARCO dataset show
that compared to existing approaches using compressed document representations,
our method is highly efficient, achieving 4x-11.6x better compression rates for
the same ranking quality.

negative:
Shared control in teleoperation leverages both human and robot's strengths
and has demonstrated great advantages of reducing the difficulties in
teleoperating a robot and increasing the task performance. One fundamental
question in shared control is how to effectively allocate the control power to
the human and robot. Researchers have been subjectively defining the arbitrate
policies following conflicting principles, which resulted in great
inconsistency in the policies. We attribute this inconsistency to the
inconsiderateness of the multi-resource uncertainty in the human-robot system.
To fill the gap, we developed a multi-source uncertainty model that was
applicable to various types of uncertainty in real world, and then a general
arbitration model was developed to comprehensively fuse the uncertainty and
regulate the arbitration weight assigned to the robotic agent. Beside
traditional macro performance metrics, we introduced objective and quantitative
metrics of robotic helpfulness and friendliness that evaluated the assistive
robot's cooperation at micro and macro levels. Results from simulations and
experiments showed the new arbitration model was more effective and friendly
over the existing policies and was robust to coping with multi-source
uncertainty. With this new arbitration model, we expect the increased adoption
of human-robot shared control in practical and complex teleoperation tasks.

negative:
Human interaction involves very sophisticated non-verbal communication skills
like understanding the goals and actions of others and coordinating our own
actions accordingly. Neuroscience refers to this mechanism as motor resonance,
in the sense that the perception of another person's actions and sensory
experiences activates the observer's brain as if (s)he would be performing the
same actions and having the same experiences. We analyze and model non-verbal
cues (arm movements) exchanged between two humans that interact and execute
handover actions. The contributions of this paper are the following: (i)
computational models, using recorded motion data, describing the motor
behaviour of each actor in action-in-interaction situations, (ii) a
computational model that captures the behaviour if the "giver" and "receiver"
during an object handover action, by coupling the arm motion of both actors,
and (iii) embedded these models in the iCub robot for both action execution and
recognition. Our results show that: (i) the robot can interpret the human arm
motion and recognize handover actions; and (ii) behave in a "human-like" manner
to receive the object of the recognized handover action.

negative:
Human recognition of the actions of other humans is very efficient and is
based on patterns of movements. Our theoretical starting point is that the
dynamics of the joint movements is important to action categorization. On the
basis of this theory, we present a novel action recognition system that employs
a hierarchy of Self-Organizing Maps together with a custom supervised neural
network that learns to categorize actions. The system preprocesses the input
from a Kinect like 3D camera to exploit the information not only about joint
positions, but also their first and second order dynamics. We evaluate our
system in two experiments with publicly available data sets, and compare its
performance to the performance with less sophisticated preprocessing of the
input. The results show that including the dynamics of the actions improves the
performance. We also apply an attention mechanism that focuses on the parts of
the body that are the most involved in performing the actions.

negative:
A fundamental question in neuroscience is how the brain creates an internal
model of the world to guide actions using sequences of ambiguous sensory
information. This is naturally formulated as a reinforcement learning problem
under partial observations, where an agent must estimate relevant latent
variables in the world from its evidence, anticipate possible future states,
and choose actions that optimize total expected reward. This problem can be
solved by control theory, which allows us to find the optimal actions for a
given system dynamics and objective function. However, animals often appear to
behave suboptimally. Why? We hypothesize that animals have their own flawed
internal model of the world, and choose actions with the highest expected
subjective reward according to that flawed model. We describe this behavior as
rational but not optimal. The problem of Inverse Rational Control (IRC) aims to
identify which internal model would best explain an agent's actions. Our
contribution here generalizes past work on Inverse Rational Control which
solved this problem for discrete control in partially observable Markov
decision processes. Here we accommodate continuous nonlinear dynamics and
continuous actions, and impute sensory observations corrupted by unknown noise
that is private to the animal. We first build an optimal Bayesian agent that
learns an optimal policy generalized over the entire model space of dynamics
and subjective rewards using deep reinforcement learning. Crucially, this
allows us to compute a likelihood over models for experimentally observable
action trajectories acquired from a suboptimal agent. We then find the model
parameters that maximize the likelihood using gradient ascent.

negative:
Humans have the fascinating capacity of processing non-verbal visual cues to
understand and anticipate the actions of other humans. This "intention reading"
ability is underpinned by shared motor-repertoires and action-models, which we
use to interpret the intentions of others as if they were our own.
  We investigate how the different cues contribute to the legibility of human
actions during interpersonal interactions. Our first contribution is a publicly
available dataset with recordings of human body-motion and eye-gaze, acquired
in an experimental scenario with an actor interacting with three subjects. From
these data, we conducted a human study to analyse the importance of the
different non-verbal cues for action perception. As our second contribution, we
used the motion/gaze recordings to build a computational model describing the
interaction between two persons. As a third contribution, we embedded this
model in the controller of an iCub humanoid robot and conducted a second human
study, in the same scenario with the robot as an actor, to validate the model's
"intention reading" capability.
  Our results show that it is possible to model (non-verbal) signals exchanged
by humans during interaction, and how to incorporate such a mechanism in
robotic systems with the twin goal of : (i) being able to "read" human action
intentions, and (ii) acting in a way that is legible by humans.

negative:
Additive Manufacturing (AM) is a crucial component of the smart industry. In
this paper, we propose an automated quality grading system for the AM process
using a deep convolutional neural network (CNN) model. The CNN model is trained
offline using the images of the internal and surface defects in the
layer-by-layer deposition of materials and tested online by studying the
performance of detecting and classifying the failure in AM process at different
extruder speeds and temperatures. The model demonstrates the accuracy of 94%
and specificity of 96%, as well as above 75% in three classifier measures of
the Fscore, the sensitivity, and precision for classifying the quality of the
printing process in five grades in real-time. The proposed online model adds an
automated, consistent, and non-contact quality control signal to the AM process
that eliminates the manual inspection of parts after they are entirely built.
The quality monitoring signal can also be used by the machine to suggest
remedial actions by adjusting the parameters in real-time. The proposed quality
predictive model serves as a proof-of-concept for any type of AM machines to
produce reliable parts with fewer quality hiccups while limiting the waste of
both time and materials.

negative:
We propose a novel approach to learn goal-conditioned policies for locomotion
in a batch RL setting. The batch data is collected by a policy that is not
goal-conditioned. For the locomotion task, this translates to data collection
using a policy learnt by the agent for walking straight in one direction, and
using that data to learn a goal-conditioned policy that enables the agent to
walk in any direction. The data collection policy used should be invariant to
the direction the agent is facing i.e. regardless of its initial orientation,
the agent should take the same actions to walk forward. We exploit this
property to learn a goal-conditioned policy using two key ideas: (1) augmenting
data by generating trajectories with the same actions in different directions,
and (2) learning an encoder that enforces invariance between these rotated
trajectories with a Siamese framework. We show that our approach outperforms
existing RL algorithms on 3-D locomotion agents like Ant, Humanoid and
Minitaur.

negative:
This paper presents a new state-of-the-art for document image classification
and retrieval, using features learned by deep convolutional neural networks
(CNNs). In object and scene analysis, deep neural nets are capable of learning
a hierarchical chain of abstraction from pixel inputs to concise and
descriptive representations. The current work explores this capacity in the
realm of document analysis, and confirms that this representation strategy is
superior to a variety of popular hand-crafted alternatives. Experiments also
show that (i) features extracted from CNNs are robust to compression, (ii) CNNs
trained on non-document images transfer well to document analysis tasks, and
(iii) enforcing region-specific feature-learning is unnecessary given
sufficient training data. This work also makes available a new labelled subset
of the IIT-CDIP collection, containing 400,000 document images across 16
categories, useful for training new CNNs for document analysis.

negative:
User preference learning is generally a hard problem. Individual preferences
are typically unknown even to users themselves, while the space of choices is
infinite. Here we study user preference learning from information-theoretic
perspective. We model preference learning as a system with two interacting
sub-systems, one representing a user with his/her preferences and another one
representing an agent that has to learn these preferences. The user with
his/her behaviour is modeled by a parametric preference function. To
efficiently learn the preferences and reduce search space quickly, we propose
the agent that interacts with the user to collect the most informative data for
learning. The agent presents two proposals to the user for evaluation, and the
user rates them based on his/her preference function. We show that the optimum
agent strategy for data collection and preference learning is a result of
maximin optimization of the normalized weighted Kullback-Leibler (KL)
divergence between true and agent-assigned predictive user response
distributions. The resulting value of KL-divergence, which we also call
remaining system uncertainty (RSU), provides an efficient performance metric in
the absence of the ground truth. This metric characterises how well the agent
can predict user and, thus, the quality of the underlying learned user
(preference) model. Our proposed agent comprises sequential mechanisms for user
model inference and proposal generation. To infer the user model (preference
function), Bayesian approximate inference is used in the agent. The data
collection strategy is to generate proposals, responses to which help resolving
uncertainty associated with prediction of the user responses the most. The
efficiency of our approach is validated by numerical simulations.

negative:
Recent years have witnessed an increasing number of interpretation methods
being developed for improving transparency of NLP models. Meanwhile,
researchers also try to answer the question that whether the obtained
interpretation is faithful in explaining mechanisms behind model prediction?
Specifically, (Jain and Wallace, 2019) proposes that "attention is not
explanation" by comparing attention interpretation with gradient alternatives.
However, it raises a new question that can we safely pick one interpretation
method as the ground-truth? If not, on what basis can we compare different
interpretation methods? In this work, we propose that it is crucial to have a
concrete definition of interpretation before we could evaluate faithfulness of
an interpretation. The definition will affect both the algorithm to obtain
interpretation and, more importantly, the metric used in evaluation. Through
both theoretical and experimental analysis, we find that although
interpretation methods perform differently under a certain evaluation metric,
such a difference may not result from interpretation quality or faithfulness,
but rather the inherent bias of the evaluation metric.

negative:
In this paper, we propose an AI-FML robotic agent for student learning
behavior ontology construction which can be applied in English speaking and
listening domain. The AI-FML robotic agent with the ontology contains the
perception intelligence, computational intelligence, and cognition intelligence
for analyzing student learning behavior. In addition, there are three
intelligent agents, including a perception agent, a computational agent, and a
cognition agent in the AI-FML robotic agent. We deploy the perception agent and
the cognition agent on the robot Kebbi Air. Moreover, the computational agent
with the Deep Neural Network (DNN) model is performed in the cloud and can
communicate with the perception agent and cognition agent via the Internet. The
proposed AI-FML robotic agent is applied in Taiwan and tested in Japan. The
experimental results show that the agents can be utilized in the human and
machine co-learning model for the future education.

negative:
In this paper, we present a multi-lingual sentence encoder that can be used
in search engines as a query and document encoder. This embedding enables a
semantic similarity score between queries and documents that can be an
important feature in document ranking and relevancy. To train such a customized
sentence encoder, it is beneficial to leverage users search data in the form of
query-document clicked pairs however, we must avoid relying too much on search
click data as it is biased and does not cover many unseen cases. The search
data is heavily skewed towards short queries and for long queries is small and
often noisy. The goal is to design a universal multi-lingual encoder that works
for all cases and covers both short and long queries. We select a number of
public NLI datasets in different languages and translation data and together
with user search data we train a language model using a multi-task approach. A
challenge is that these datasets are not homogeneous in terms of content, size
and the balance ratio. While the public NLI datasets are usually two-sentence
based with the same portion of positive and negative pairs, the user search
data can contain multi-sentence documents and only positive pairs. We show how
multi-task training enables us to leverage all these datasets and exploit
knowledge sharing across these tasks.

negative:
Anomaly detection incurs certain sampling and sensing costs and therefore it
is of great importance to strike a balance between the detection accuracy and
these costs. In this work, we study anomaly detection by considering the
detection of threshold crossings in a stochastic time series without the
knowledge of its statistics. To reduce the sampling cost in this detection
process, we propose the use of hierarchical generative adversarial networks
(GANs) to perform nonuniform sampling. In order to improve the detection
accuracy and reduce the delay in detection, we introduce a buffer zone in the
operation of the proposed GAN-based detector. In the experiments, we analyze
the performance of the proposed hierarchical GAN detector considering the
metrics of detection delay, miss rates, average cost of error, and sampling
ratio. We identify the tradeoffs in the performance as the buffer zone sizes
and the number of GAN levels in the hierarchy vary. We also compare the
performance with that of a sampling policy that approximately minimizes the sum
of average costs of sampling and error given the parameters of the stochastic
process. We demonstrate that the proposed GAN-based detector can have
significant performance improvements in terms of detection delay and average
cost of error with a larger buffer zone but at the cost of increased sampling
rates.

negative:
In reinforcement learning, it is common to let an agent interact for a fixed
amount of time with its environment before resetting it and repeating the
process in a series of episodes. The task that the agent has to learn can
either be to maximize its performance over (i) that fixed period, or (ii) an
indefinite period where time limits are only used during training to diversify
experience. In this paper, we provide a formal account for how time limits
could effectively be handled in each of the two cases and explain why not doing
so can cause state aliasing and invalidation of experience replay, leading to
suboptimal policies and training instability. In case (i), we argue that the
terminations due to time limits are in fact part of the environment, and thus a
notion of the remaining time should be included as part of the agent's input to
avoid violation of the Markov property. In case (ii), the time limits are not
part of the environment and are only used to facilitate learning. We argue that
this insight should be incorporated by bootstrapping from the value of the
state at the end of each partial episode. For both cases, we illustrate
empirically the significance of our considerations in improving the performance
and stability of existing reinforcement learning algorithms, showing
state-of-the-art results on several control tasks.

negative:
Re-finding electronic documents from a personal computer is a frequent demand
to users. In a simple re-finding task, people can use many methods to retrieve
a document, such as navigating directly to the document's folder, searching
with a desktop search engine, or checking the Recent Files List. However, when
encountering a difficult re-finding task, people usually cannot remember the
attributes used by conventional re-finding methods, such as file path, file
name, keywords etc., the re-finding would fail. We propose a new method to
support difficult re-finding tasks. When a user is reading a document, we
collect all kinds of possible memory pieces of the user about the document,
such as number of pages, number of images, number of math formulas, cumulative
reading time, reading frequency, printing experiences etc. If the user wants to
re-find a document later, we use these collected attributes to filter out the
target document. To alleviate the user's cognitive burden, we use a question
and answer wizard interface and provide recommendations to the answers for the
user, the recommendations are generated by analyzing the collected attributes
of each document and the user's experiences about them.

negative:
We introduce a novel type of text representation that preserves the 2D layout
of a document. This is achieved by encoding each document page as a
two-dimensional grid of characters. Based on this representation, we present a
generic document understanding pipeline for structured documents. This pipeline
makes use of a fully convolutional encoder-decoder network that predicts a
segmentation mask and bounding boxes. We demonstrate its capabilities on an
information extraction task from invoices and show that it significantly
outperforms approaches based on sequential text or document images.

negative:
Inverse reinforcement learning (IRL) is the problem of finding a reward
function that generates a given optimal policy for a given Markov Decision
Process. This paper looks at an algorithmic-independent geometric analysis of
the IRL problem with finite states and actions. A L1-regularized Support Vector
Machine formulation of the IRL problem motivated by the geometric analysis is
then proposed with the basic objective of the inverse reinforcement problem in
mind: to find a reward function that generates a specified optimal policy. The
paper further analyzes the proposed formulation of inverse reinforcement
learning with $n$ states and $k$ actions, and shows a sample complexity of
$O(n^2 \log (nk))$ for recovering a reward function that generates a policy
that satisfies Bellman's optimality condition with respect to the true
transition probabilities.

negative:
Traditional Relational Topic Models provide a way to discover the hidden
topics from a document network. Many theoretical and practical tasks, such as
dimensional reduction, document clustering, link prediction, benefit from this
revealed knowledge. However, existing relational topic models are based on an
assumption that the number of hidden topics is known in advance, and this is
impractical in many real-world applications. Therefore, in order to relax this
assumption, we propose a nonparametric relational topic model in this paper.
Instead of using fixed-dimensional probability distributions in its generative
model, we use stochastic processes. Specifically, a gamma process is assigned
to each document, which represents the topic interest of this document.
Although this method provides an elegant solution, it brings additional
challenges when mathematically modeling the inherent network structure of
typical document network, i.e., two spatially closer documents tend to have
more similar topics. Furthermore, we require that the topics are shared by all
the documents. In order to resolve these challenges, we use a subsampling
strategy to assign each document a different gamma process from the global
gamma process, and the subsampling probabilities of documents are assigned with
a Markov Random Field constraint that inherits the document network structure.
Through the designed posterior inference algorithm, we can discover the hidden
topics and its number simultaneously. Experimental results on both synthetic
and real-world network datasets demonstrate the capabilities of learning the
hidden topics and, more importantly, the number of topics.

negative:
Imitation learning is an effective alternative approach to learn a policy
when the reward function is sparse. In this paper, we consider a challenging
setting where an agent and an expert use different actions from each other. We
assume that the agent has access to a sparse reward function and state-only
expert observations. We propose a method which gradually balances between the
imitation learning cost and the reinforcement learning objective. In addition,
this method adapts the agent's policy based on either mimicking expert behavior
or maximizing sparse reward. We show, through navigation scenarios, that (i) an
agent is able to efficiently leverage sparse rewards to outperform standard
state-only imitation learning, (ii) it can learn a policy even when its actions
are different from the expert, and (iii) the performance of the agent is not
bounded by that of the expert, due to the optimized usage of sparse rewards.

negative:
Rapid increase of digitized document give birth to high demand of document
image retrieval. While conventional document image retrieval approaches depend
on complex OCR-based text recognition and text similarity detection, this paper
proposes a new content-based approach, in which more attention is paid to
features extraction and fusion. In the proposed approach, multiple features of
document images are extracted by different CNN models. After that, the
extracted CNN features are reduced and fused into weighted average feature.
Finally, the document images are ranked based on feature similarity to a
provided query image. Experimental procedure is performed on a group of
document images that transformed from academic papers, which contain both
English and Chinese document, the results show that the proposed approach has
good ability to retrieve document images with similar text content, and the
fusion of CNN features can effectively improve the retrieval accuracy.

negative:
Latent topic models have been successfully applied as an unsupervised topic
discovery technique in large document collections. With the proliferation of
hypertext document collection such as the Internet, there has also been great
interest in extending these approaches to hypertext [6, 9]. These approaches
typically model links in an analogous fashion to how they model words - the
document-link co-occurrence matrix is modeled in the same way that the
document-word co-occurrence matrix is modeled in standard topic models. In this
paper we present a probabilistic generative model for hypertext document
collections that explicitly models the generation of links. Specifically, links
from a word w to a document d depend directly on how frequent the topic of w is
in d, in addition to the in-degree of d. We show how to perform EM learning on
this model efficiently. By not modeling links as analogous to words, we end up
using far fewer free parameters and obtain better link prediction results.

negative:
Most of the textual information available to us are temporally variable. In a
world where information is dynamic, time-stamping them is a very important
task. Documents are a good source of information and are used for many tasks
like, sentiment analysis, classification of reviews etc. The knowledge of
creation date of documents facilitates several tasks like summarization, event
extraction, temporally focused information extraction etc. Unfortunately, for
most of the documents on the web, the time-stamp meta-data is either erroneous
or missing. Thus document dating is a challenging problem which requires
inference over the temporal structure of the document alongside the contextual
information of the document. Prior document dating systems have largely relied
on handcrafted features while ignoring such document-internal structures. In
this paper we propose NeuralDater, a Graph Convolutional Network (GCN) based
document dating approach which jointly exploits syntactic and temporal graph
structures of document in a principled way. We also pointed out some
limitations of NeuralDater and tried to utilize both context and temporal
information in documents in a more flexible and intuitive manner proposing AD3:
Attentive Deep Document Dater, an attention-based document dating system. To
the best of our knowledge these are the first application of deep learning
methods for the task. Through extensive experiments on real-world datasets, we
find that our models significantly outperforms state-of-the-art baselines by a
significant margin.

negative:
We present LDAExplore, a tool to visualize topic distributions in a given
document corpus that are generated using Topic Modeling methods. Latent
Dirichlet Allocation (LDA) is one of the basic methods that is predominantly
used to generate topics. One of the problems with methods like LDA is that
users who apply them may not understand the topics that are generated. Also,
users may find it difficult to search correlated topics and correlated
documents. LDAExplore, tries to alleviate these problems by visualizing topic
and word distributions generated from the document corpus and allowing the user
to interact with them. The system is designed for users, who have minimal
knowledge of LDA or Topic Modelling methods. To evaluate our design, we run a
pilot study which uses the abstracts of 322 Information Visualization papers,
where every abstract is considered a document. The topics generated are then
explored by users. The results show that users are able to find correlated
documents and group them based on topics that are similar.

negative:
Supernumerary Robotic Limbs (SRLs) exhibit inherently compliant behavior due
to the elasticity present at the intersection of human tissue and the robot.
This compliance, can prominently influence the operation of some SRLs,
depending on the application. In order to control the residual vibrations of
SRLs, we have used an input-shaping method which is a computationally
inexpensive approach. The effectiveness of this method in controlling the
residual vibrations of a SRL has been proven using robustness analysis. User
studies show that reducing the vibrations using input shaping directly
increases the user satisfaction and comfort by at least 9%. It is also observed
that 36% of the users preferred unshaped commands. We hypothesize that the
shaped commands put a higher cognitive load on the user compared to unshaped
commands. This shows that when dealing with human-robot interaction, user
satisfaction becomes an equally important parameter as traditional performance
criteria and should be taken into account while evaluating the success of any
vibration-control method.

negative:
Opinion summarization is the task of automatically creating summaries that
reflect subjective information expressed in multiple documents, such as product
reviews. While the majority of previous work has focused on the extractive
setting, i.e., selecting fragments from input reviews to produce a summary, we
let the model generate novel sentences and hence produce abstractive summaries.
Recent progress in summarization has seen the development of supervised models
which rely on large quantities of document-summary pairs. Since such training
data is expensive to acquire, we instead consider the unsupervised setting, in
other words, we do not use any summaries in training. We define a generative
model for a review collection which capitalizes on the intuition that when
generating a new review given a set of other reviews of a product, we should be
able to control the "amount of novelty" going into the new review or,
equivalently, vary the extent to which it deviates from the input. At test
time, when generating summaries, we force the novelty to be minimal, and
produce a text reflecting consensus opinions. We capture this intuition by
defining a hierarchical variational autoencoder model. Both individual reviews
and the products they correspond to are associated with stochastic latent
codes, and the review generator ("decoder") has direct access to the text of
input reviews through the pointer-generator mechanism. Experiments on Amazon
and Yelp datasets, show that setting at test time the review's latent code to
its mean, allows the model to produce fluent and coherent summaries reflecting
common opinions.

negative:
We consider the problem of minimizing regret in an $N$ agent heterogeneous
stochastic linear bandits framework, where the agents (users) are similar but
not all identical. We model user heterogeneity using two popularly used ideas
in practice; (i) A clustering framework where users are partitioned into groups
with users in the same group being identical to each other, but different
across groups, and (ii) a personalization framework where no two users are
necessarily identical, but a user's parameters are close to that of the
population average. In the clustered users' setup, we propose a novel
algorithm, based on successive refinement of cluster identities and regret
minimization. We show that, for any agent, the regret scales as
$\mathcal{O}(\sqrt{T/N})$, if the agent is in a `well separated' cluster, or
scales as $\mathcal{O}(T^{\frac{1}{2} + \varepsilon}/(N)^{\frac{1}{2}
-\varepsilon})$ if its cluster is not well separated, where $\varepsilon$ is
positive and arbitrarily close to $0$. Our algorithm is adaptive to the cluster
separation, and is parameter free -- it does not need to know the number of
clusters, separation and cluster size, yet the regret guarantee adapts to the
inherent complexity. In the personalization framework, we introduce a natural
algorithm where, the personal bandit instances are initialized with the
estimates of the global average model. We show that, an agent $i$ whose
parameter deviates from the population average by $\epsilon_i$, attains a
regret scaling of $\widetilde{O}(\epsilon_i\sqrt{T})$. This demonstrates that
if the user representations are close (small $\epsilon_i)$, the resulting
regret is low, and vice-versa. The results are empirically validated and we
observe superior performance of our adaptive algorithms over non-adaptive
baselines.

negative:
In this work, a region-based Deep Convolutional Neural Network framework is
proposed for document structure learning. The contribution of this work
involves efficient training of region based classifiers and effective
ensembling for document image classification. A primary level of `inter-domain'
transfer learning is used by exporting weights from a pre-trained VGG16
architecture on the ImageNet dataset to train a document classifier on whole
document images. Exploiting the nature of region based influence modelling, a
secondary level of `intra-domain' transfer learning is used for rapid training
of deep learning models for image segments. Finally, stacked generalization
based ensembling is utilized for combining the predictions of the base deep
neural network models. The proposed method achieves state-of-the-art accuracy
of 92.2% on the popular RVL-CDIP document image dataset, exceeding benchmarks
set by existing algorithms.

negative:
This paper describes the short-term competition on the Components
Segmentation Task of Document Photos that was prepared in the context of the
16th International Conference on Document Analysis and Recognition (ICDAR
2021). This competition aims to bring together researchers working in the field
of identification document image processing and provides them a suitable
benchmark to compare their techniques on the component segmentation task of
document images. Three challenge tasks were proposed entailing different
segmentation assignments to be performed on a provided dataset. The collected
data are from several types of Brazilian ID documents, whose personal
information was conveniently replaced. There were 16 participants whose results
obtained for some or all the three tasks show different rates for the adopted
metrics, like Dice Similarity Coefficient ranging from 0.06 to 0.99. Different
Deep Learning models were applied by the entrants with diverse strategies to
achieve the best results in each of the tasks. Obtained results show that the
currently applied methods for solving one of the proposed tasks (document
boundary detection) are already well established. However, for the other two
challenge tasks (text zone and handwritten sign detection) research and
development of more robust approaches are still required to achieve acceptable
results.

negative:
We show that Bayes' rule provides an effective mechanism for creating
document translation models that can be learned from only parallel sentences
and monolingual documents---a compelling benefit as parallel documents are not
always available. In our formulation, the posterior probability of a candidate
translation is the product of the unconditional (prior) probability of the
candidate output document and the "reverse translation probability" of
translating the candidate output back into the source language. Our proposed
model uses a powerful autoregressive language model as the prior on target
language documents, but it assumes that each sentence is translated
independently from the target to the source language. Crucially, at test time,
when a source document is observed, the document language model prior induces
dependencies between the translations of the source sentences in the posterior.
The model's independence assumption not only enables efficient use of available
data, but it additionally admits a practical left-to-right beam-search
algorithm for carrying out inference. Experiments show that our model benefits
from using cross-sentence context in the language model, and it outperforms
existing document translation approaches.

negative:
Document clustering is a text mining technique used to provide better
document search and browsing in digital libraries or online corpora. A lot of
research has been done on biomedical document clustering that is based on using
existing ontology. But, associations and co-occurrences of the medical concepts
are not well represented by using ontology. In this research, a vector
representation of concepts of diseases and similarity measurement between
concepts are proposed. They identify the closest concepts of diseases in the
context of a corpus. Each document is represented by using the vector space
model. A weight scheme is proposed to consider both local content and
associations between concepts. A Self-Organizing Map is used as document
clustering algorithm. The vector projection and visualization features of SOM
enable visualization and analysis of the clusters distributions and
relationships on the two dimensional space. The experimental results show that
the proposed document clustering framework generates meaningful clusters and
facilitate visualization of the clusters based on the concepts of diseases.

negative:
One major drawback of state of the art Neural Networks (NN)-based approaches
for document classification purposes is the large number of training samples
required to obtain an efficient classification. The minimum required number is
around one thousand annotated documents for each class. In many cases it is
very difficult, if not impossible, to gather this number of samples in real
industrial processes. In this paper, we analyse the efficiency of NN-based
document classification systems in a sub-optimal training case, based on the
situation of a company document stream. We evaluated three different
approaches, one based on image content and two on textual content. The
evaluation was divided into four parts: a reference case, to assess the
performance of the system in the lab; two cases that each simulate a specific
difficulty linked to document stream processing; and a realistic case that
combined all of these difficulties. The realistic case highlighted the fact
that there is a significant drop in the efficiency of NN-Based document
classification systems. Although they remain efficient for well represented
classes (with an over-fitting of the system for those classes), it is
impossible for them to handle appropriately less well represented classes.
NN-Based document classification systems need to be adapted to resolve these
two problems before they can be considered for use in a company document
stream.

negative:
A Bayesian view of data interpretation suggests that a visualization user
should update their existing beliefs about a parameter's value in accordance
with the amount of information about the parameter value captured by the new
observations. Extending recent work applying Bayesian models to understand and
evaluate belief updating from visualizations, we show how the predictions of
Bayesian inference can be used to guide more rational belief updating. We
design a Bayesian inference-assisted uncertainty analogy that numerically
relates uncertainty in observed data to the user's subjective uncertainty, and
a posterior visualization that prescribes how a user should update their
beliefs given their prior beliefs and the observed data. In a pre-registered
experiment on 4,800 people, we find that when a newly observed data sample is
relatively small (N=158), both techniques reliably improve people's Bayesian
updating on average compared to the current best practice of visualizing
uncertainty in the observed data. For large data samples (N=5208), where
people's updated beliefs tend to deviate more strongly from the prescriptions
of a Bayesian model, we find evidence that the effectiveness of the two forms
of Bayesian assistance may depend on people's proclivity toward trusting the
source of the data. We discuss how our results provide insight into individual
processes of belief updating and subjective uncertainty, and how understanding
these aspects of interpretation paves the way for more sophisticated
interactive visualizations for analysis and communication.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.

negative:
When we view visualizations, we not only have a visual representation of the
data, but also a verbal one. Recent work has shown that these visual
representations of data can be biased, such that the position of a line in a
chart will be consistently underestimated. But are the verbal representations
of position encodings also biased in the same manner, or is this a purely
visual bias that can be mitigated with verbal context? We explored the bias in
position reproductions for simple uniform lines for both visual and verbal
representations. We find that the direction of the bias changed depending on
the response modality, with visual reproductions showing a position
underestimation while verbal responses showed overestimation. This finding
indicates that, even for simple line charts, biases are still present for both
visual and verbal representations, although the directionality of this bias
depends on the modality.

negative:
We address the problem of solving mixed random linear equations. We have
unlabeled observations coming from multiple linear regressions, and each
observation corresponds to exactly one of the regression models. The goal is to
learn the linear regressors from the observations. Classically, Alternating
Minimization (AM) (which is a variant of Expectation Maximization (EM)) is used
to solve this problem. AM iteratively alternates between the estimation of
labels and solving the regression problems with the estimated labels.
Empirically, it is observed that, for a large variety of non-convex problems
including mixed linear regression, AM converges at a much faster rate compared
to gradient based algorithms. However, the existing theory suggests similar
rate of convergence for AM and gradient based methods, failing to capture this
empirical behavior. In this paper, we close this gap between theory and
practice for the special case of a mixture of $2$ linear regressions. We show
that, provided initialized properly, AM enjoys a \emph{super-linear} rate of
convergence in certain parameter regimes. To the best of our knowledge, this is
the first work that theoretically establishes such rate for AM. Hence, if we
want to recover the unknown regressors upto an error (in $\ell_2$ norm) of
$\epsilon$, AM only takes $\mathcal{O}(\log \log (1/\epsilon))$ iterations.
Furthermore, we compare AM with a gradient based heuristic algorithm
empirically and show that AM dominates in iteration complexity as well as
wall-clock time.