query: I am working on a project where I input a sentence, my system will return related images based on my input sentence. I call this task image retrieval from sentences. This system needs to work with different languages. My current plan is to first create a synthetic dataset that consists of tuples of sentences in different languages and an image. I am trying to use image captioning or image annotation pre-trained models to create this synthetic dataset. After we have created this synthetic dataset, I will build a model that can learn the representations of sentences with respect to the image. I believe my work is in the intersection of information retrieval, learning sentence representation and data augmentation. In the end, I want my work to be integrated with a search engine that can be deployed in the cloud.

positive:
The ability to describe images with natural language sentences is the
hallmark for image and language understanding. Such a system has wide ranging
applications such as annotating images and using natural sentences to search
for images.In this project we focus on the task of bidirectional image
retrieval: such asystem is capable of retrieving an image based on a sentence
(image search) andretrieve sentence based on an image query (image annotation).
We present asystem based on a global ranking objective function which uses a
combinationof convolutional neural networks (CNN) and multi layer perceptrons
(MLP).It takes a pair of image and sentence and processes them in different
channels,finally embedding it into a common multimodal vector space. These
embeddingsencode abstract semantic information about the two inputs and can be
comparedusing traditional information retrieval approaches. For each such pair,
the modelreturns a score which is interpretted as a similarity metric. If this
score is high,the image and sentence are likely to convey similar meaning, and
if the score is low then they are likely not to.
  The visual input is modeled via deep convolutional neural network. On
theother hand we explore three models for the textual module. The first one
isbag of words with an MLP. The second one uses n-grams (bigram, trigrams,and a
combination of trigram & skip-grams) with an MLP. The third is morespecialized
deep network specific for modeling variable length sequences (SSE).We report
comparable performance to recent work in the field, even though ouroverall
model is simpler. We also show that the training time choice of how wecan
generate our negative samples has a significant impact on performance, and can
be used to specialize the bi-directional system in one particular task.

positive:
In this paper we propose a model to learn multimodal multilingual
representations for matching images and sentences in different languages, with
the aim of advancing multilingual versions of image search and image
understanding. Our model learns a common representation for images and their
descriptions in two different languages (which need not be parallel) by
considering the image as a pivot between two languages. We introduce a new
pairwise ranking loss function which can handle both symmetric and asymmetric
similarity between the two modalities. We evaluate our models on
image-description ranking for German and English, and on semantic textual
similarity of image descriptions in English. In both cases we achieve
state-of-the-art performance.

positive:
Cross-modal retrieval is an important functionality in modern search engines,
as it increases the user experience by allowing queries and retrieved objects
to pertain to different modalities. In this paper, we focus on the
image-sentence retrieval task, where the objective is to efficiently find
relevant images for a given sentence (image-retrieval) or the relevant
sentences for a given image (sentence-retrieval). Computer vision literature
reports the best results on the image-sentence matching task using deep neural
networks equipped with attention and self-attention mechanisms. They evaluate
the matching performance on the retrieval task by performing sequential scans
of the whole dataset. This method does not scale well with an increasing amount
of images or captions. In this work, we explore different preprocessing
techniques to produce sparsified deep multi-modal features extracting them from
state-of-the-art deep-learning architectures for image-text matching. Our main
objective is to lay down the paths for efficient indexing of complex
multi-modal descriptions. We use the recently introduced TERN architecture as
an image-sentence features extractor. It is designed for producing fixed-size
1024-d vectors describing whole images and sentences, as well as
variable-length sets of 1024-d vectors describing the various building
components of the two modalities (image regions and sentence words
respectively). All these vectors are enforced by the TERN design to lie into
the same common space. Our experiments show interesting preliminary results on
the explored methods and suggest further experimentation in this important
research direction.

positive:
Text-to-Image translation has been an active area of research in the recent
past. The ability for a network to learn the meaning of a sentence and generate
an accurate image that depicts the sentence shows ability of the model to think
more like humans. Popular methods on text to image translation make use of
Generative Adversarial Networks (GANs) to generate high quality images based on
text input, but the generated images don't always reflect the meaning of the
sentence given to the model as input. We address this issue by using a
captioning network to caption on generated images and exploit the distance
between ground truth captions and generated captions to improve the network
further. We show extensive comparisons between our method and existing methods.

positive:
Recent work in computer vision has yielded impressive results in
automatically describing images with natural language. Most of these systems
generate captions in a sin- gle language, requiring multiple language-specific
models to build a multilingual captioning system. We propose a very simple
technique to build a single unified model across languages, using artificial
tokens to control the language, making the captioning system more compact. We
evaluate our approach on generating English and Japanese captions, and show
that a typical neural captioning architecture is capable of learning a single
model that can switch between two different languages.

positive:
Our objective is video retrieval based on natural language queries. In
addition, we consider the analogous problem of retrieving sentences or
generating descriptions given an input video. Recent work has addressed the
problem by embedding visual and textual inputs into a common space where
semantic similarities correlate to distances. We also adopt the embedding
approach, and make the following contributions: First, we utilize web image
search in sentence embedding process to disambiguate fine-grained visual
concepts. Second, we propose embedding models for sentence, image, and video
inputs whose parameters are learned simultaneously. Finally, we show how the
proposed model can be applied to description generation. Overall, we observe a
clear improvement over the state-of-the-art methods in the video and sentence
retrieval tasks. In description generation, the performance level is comparable
to the current state-of-the-art, although our embeddings were trained for the
retrieval tasks.

negative:
This paper introduces a large-scale multimodal and multilingual dataset that
aims to facilitate research on grounding words to images in their contextual
usage in language. The dataset consists of images selected to unambiguously
illustrate concepts expressed in sentences from movie subtitles. The dataset is
a valuable resource as (i) the images are aligned to text fragments rather than
whole sentences; (ii) multiple images are possible for a text fragment and a
sentence; (iii) the sentences are free-form and real-world like; (iv) the
parallel texts are multilingual. We set up a fill-in-the-blank game for humans
to evaluate the quality of the automatic image selection process of our
dataset. We show the utility of the dataset on two automatic tasks: (i)
fill-in-the-blank; (ii) lexical translation. Results of the human evaluation
and automatic models demonstrate that images can be a useful complement to the
textual context. The dataset will benefit research on visual grounding of words
especially in the context of free-form sentences, and can be obtained from
https://doi.org/10.5281/zenodo.5034604 under a Creative Commons licence.

negative:
Text contained in an image carries high-level semantics that can be exploited
to achieve richer image understanding. In particular, the mere presence of text
provides strong guiding content that should be employed to tackle a diversity
of computer vision tasks such as image retrieval, fine-grained classification,
and visual question answering. In this paper, we address the problem of
fine-grained classification and image retrieval by leveraging textual
information along with visual cues to comprehend the existing intrinsic
relation between the two modalities. The novelty of the proposed model consists
of the usage of a PHOC descriptor to construct a bag of textual words along
with a Fisher Vector Encoding that captures the morphology of text. This
approach provides a stronger multimodal representation for this task and as our
experiments demonstrate, it achieves state-of-the-art results on two different
tasks, fine-grained classification and image retrieval.

negative:
Image captioning has so far been explored mostly in English, as most
available datasets are in this language. However, the application of image
captioning should not be restricted by language. Only few studies have been
conducted for image captioning in a cross-lingual setting. Different from these
works that manually build a dataset for a target language, we aim to learn a
cross-lingual captioning model fully from machine-translated sentences. To
conquer the lack of fluency in the translated sentences, we propose in this
paper a fluency-guided learning framework. The framework comprises a module to
automatically estimate the fluency of the sentences and another module to
utilize the estimated fluency scores to effectively train an image captioning
model for the target language. As experiments on two bilingual
(English-Chinese) datasets show, our approach improves both fluency and
relevance of the generated captions in Chinese, but without using any manually
written sentences from the target language.

negative:
Image Captioning is a task that requires models to acquire a multi-modal
understanding of the world and to express this understanding in natural
language text. While the state-of-the-art for this task has rapidly improved in
terms of n-gram metrics, these models tend to output the same generic captions
for similar images. In this work, we address this limitation and train a model
that generates more diverse and specific captions through an unsupervised
training approach that incorporates a learning signal from an Image Retrieval
model. We summarize previous results and improve the state-of-the-art on
caption diversity and novelty. We make our source code publicly available
online.

negative:
Recently, generative adversarial networks have gained a lot of popularity for
image generation tasks. However, such models are associated with complex
learning mechanisms and demand very large relevant datasets. This work borrows
concepts from image and video captioning models to form an image generative
framework. The model is trained in a similar fashion as recurrent captioning
model and uses the learned weights for image generation. This is done in an
inverse direction, where the input is a caption and the output is an image. The
vector representation of the sentence and frames are extracted from an
encoder-decoder model which is initially trained on similar sentence and image
pairs. Our model conditions image generation on a natural language caption. We
leverage a sequence-to-sequence model to generate synthetic captions that have
the same meaning for having a robust image generation. One key advantage of our
method is that the traditional image captioning datasets can be used for
synthetic sentence paraphrases. Results indicate that images generated through
multiple captions are better at capturing the semantic meaning of the family of
captions.

negative:
A big part of achieving Artificial General Intelligence(AGI) is to build a
machine that can see and listen like humans. Much work has focused on designing
models for image classification, video classification, object detection, pose
estimation, speech recognition, etc., and has achieved significant progress in
recent years thanks to deep learning. However, understanding the world is not
enough. An AI agent also needs to know how to talk, especially how to
communicate with a human. While perception (vision, for example) is more common
across animal species, the use of complicated language is unique to humans and
is one of the most important aspects of intelligence.
  In this thesis, we focus on generating textual output given visual input. In
Chapter 3, we focus on generating the referring expression, a text description
for an object in the image so that a receiver can infer which object is being
described. We use a comprehension machine to directly guide the generated
referring expressions to be more discriminative. In Chapter 4, we introduce a
method that encourages discriminability in image caption generation. We show
that more discriminative captioning models generate more descriptive captions.
In Chapter 5, we study how training objectives and sampling methods affect the
models' ability to generate diverse captions. We find that a popular captioning
training strategy will be detrimental to the diversity of generated captions.
In Chapter 6, we propose a model that can control the length of generated
captions. By changing the desired length, one can influence the style and
descriptiveness of the captions. Finally, in Chapter 7, we rank/generate
informative image tags according to their information utility. The proposed
method better matches what humans think are the most important tags for the
images.

negative:
Automatically generating a natural language description of an image is a task
close to the heart of image understanding. In this paper, we present a
multi-model neural network method closely related to the human visual system
that automatically learns to describe the content of images. Our model consists
of two sub-models: an object detection and localization model, which extract
the information of objects and their spatial relationship in images
respectively; Besides, a deep recurrent neural network (RNN) based on long
short-term memory (LSTM) units with attention mechanism for sentences
generation. Each word of the description will be automatically aligned to
different objects of the input image when it is generated. This is similar to
the attention mechanism of the human visual system. Experimental results on the
COCO dataset showcase the merit of the proposed method, which outperforms
previous benchmark models.

negative:
Current image captioning systems perform at a merely descriptive level,
essentially enumerating the objects in the scene and their relations. Humans,
on the contrary, interpret images by integrating several sources of prior
knowledge of the world. In this work, we aim to take a step closer to producing
captions that offer a plausible interpretation of the scene, by integrating
such contextual information into the captioning pipeline. For this we focus on
the captioning of images used to illustrate news articles. We propose a novel
captioning method that is able to leverage contextual information provided by
the text of news articles associated with an image. Our model is able to
selectively draw information from the article guided by visual cues, and to
dynamically extend the output dictionary to out-of-vocabulary named entities
that appear in the context source. Furthermore we introduce `GoodNews', the
largest news image captioning dataset in the literature and demonstrate
state-of-the-art results.

negative:
Named entity recognition (NER) is frequently addressed as a sequence
classification task where each input consists of one sentence of text. It is
nevertheless clear that useful information for the task can often be found
outside of the scope of a single-sentence context. Recently proposed
self-attention models such as BERT can both efficiently capture long-distance
relationships in input as well as represent inputs consisting of several
sentences, creating new opportunitites for approaches that incorporate
cross-sentence information in natural language processing tasks. In this paper,
we present a systematic study exploring the use of cross-sentence information
for NER using BERT models in five languages. We find that adding context in the
form of additional sentences to BERT input systematically increases NER
performance on all of the tested languages and models. Including multiple
sentences in each input also allows us to study the predictions of the same
sentences in different contexts. We propose a straightforward method,
Contextual Majority Voting (CMV), to combine different predictions for
sentences and demonstrate this to further increase NER performance with BERT.
Our approach does not require any changes to the underlying BERT architecture,
rather relying on restructuring examples for training and prediction.
Evaluation on established datasets, including the CoNLL'02 and CoNLL'03 NER
benchmarks, demonstrates that our proposed approach can improve on the
state-of-the-art NER results on English, Dutch, and Finnish, achieves the best
reported BERT-based results on German, and is on par with performance reported
with other BERT-based approaches in Spanish. We release all methods implemented
in this work under open licenses.

negative:
State-of-the-art approaches for image captioning require supervised training
data consisting of captions with paired image data. These methods are typically
unable to use unsupervised data such as textual data with no corresponding
images, which is a much more abundant commodity. We here propose a novel way of
using such textual data by artificially generating missing visual information.
We evaluate this learning approach on a newly designed model that detects
visual concepts present in an image and feed them to a reviewer-decoder
architecture with an attention mechanism. Unlike previous approaches that
encode visual concepts using word embeddings, we instead suggest using regional
image features which capture more intrinsic information. The main benefit of
this architecture is that it synthesizes meaningful thought vectors that
capture salient image properties and then applies a soft attentive decoder to
decode the thought vectors and generate image captions. We evaluate our model
on both Microsoft COCO and Flickr30K datasets and demonstrate that this model
combined with our semi-supervised learning method can largely improve
performance and help the model to generate more accurate and diverse captions.

negative:
Prevalent models based on artificial neural network (ANN) for sentence
classification often classify sentences in isolation without considering the
context in which sentences appear. This hampers the traditional sentence
classification approaches to the problem of sequential sentence classification,
where structured prediction is needed for better overall classification
performance. In this work, we present a hierarchical sequential labeling
network to make use of the contextual information within surrounding sentences
to help classify the current sentence. Our model outperforms the
state-of-the-art results by 2%-3% on two benchmarking datasets for sequential
sentence classification in medical scientific abstracts.

negative:
Image and sentence matching has made great progress recently, but it remains
challenging due to the large visual-semantic discrepancy. This mainly arises
from that the representation of pixel-level image usually lacks of high-level
semantic information as in its matched sentence. In this work, we propose a
semantic-enhanced image and sentence matching model, which can improve the
image representation by learning semantic concepts and then organizing them in
a correct semantic order. Given an image, we first use a multi-regional
multi-label CNN to predict its semantic concepts, including objects,
properties, actions, etc. Then, considering that different orders of semantic
concepts lead to diverse semantic meanings, we use a context-gated sentence
generation scheme for semantic order learning. It simultaneously uses the image
global context containing concept relations as reference and the groundtruth
semantic order in the matched sentence as supervision. After obtaining the
improved image representation, we learn the sentence representation with a
conventional LSTM, and then jointly perform image and sentence matching and
sentence generation for model learning. Extensive experiments demonstrate the
effectiveness of our learned semantic concepts and order, by achieving the
state-of-the-art results on two public benchmark datasets.

negative:
Building upon recent Deep Neural Network architectures, current approaches
lying in the intersection of computer vision and natural language processing
have achieved unprecedented breakthroughs in tasks like automatic captioning or
image retrieval. Most of these learning methods, though, rely on large training
sets of images associated with human annotations that specifically describe the
visual content. In this paper we propose to go a step further and explore the
more complex cases where textual descriptions are loosely related to the
images. We focus on the particular domain of News articles in which the textual
content often expresses connotative and ambiguous relations that are only
suggested but not directly inferred from images. We introduce new deep learning
methods that address source detection, popularity prediction, article
illustration and geolocation of articles. An adaptive CNN architecture is
proposed, that shares most of the structure for all the tasks, and is suitable
for multitask and transfer learning. Deep Canonical Correlation Analysis is
deployed for article illustration, and a new loss function based on Great
Circle Distance is proposed for geolocation. Furthermore, we present
BreakingNews, a novel dataset with approximately 100K news articles including
images, text and captions, and enriched with heterogeneous meta-data (such as
GPS coordinates and popularity metrics). We show this dataset to be appropriate
to explore all aforementioned problems, for which we provide a baseline
performance using various Deep Learning architectures, and different
representations of the textual and visual features. We report very promising
results and bring to light several limitations of current state-of-the-art in
this kind of domain, which we hope will help spur progress in the field.

negative:
A widespread approach to processing spoken language is to first automatically
transcribe it into text. An alternative is to use an end-to-end approach:
recent works have proposed to learn semantic embeddings of spoken language from
images with spoken captions, without an intermediate transcription step. We
propose to use multitask learning to exploit existing transcribed speech within
the end-to-end setting. We describe a three-task architecture which combines
the objectives of matching spoken captions with corresponding images, speech
with text, and text with images. We show that the addition of the speech/text
task leads to substantial performance improvements on image retrieval when
compared to training the speech/image task in isolation. We conjecture that
this is due to a strong inductive bias transcribed speech provides to the
model, and offer supporting evidence for this.

negative:
In this paper, we propose a way of synthesizing realistic images directly
with natural language description, which has many useful applications, e.g.
intelligent image manipulation. We attempt to accomplish such synthesis: given
a source image and a target text description, our model synthesizes images to
meet two requirements: 1) being realistic while matching the target text
description; 2) maintaining other image features that are irrelevant to the
text description. The model should be able to disentangle the semantic
information from the two modalities (image and text), and generate new images
from the combined semantics. To achieve this, we proposed an end-to-end neural
architecture that leverages adversarial learning to automatically learn
implicit loss functions, which are optimized to fulfill the aforementioned two
requirements. We have evaluated our model by conducting experiments on
Caltech-200 bird dataset and Oxford-102 flower dataset, and have demonstrated
that our model is capable of synthesizing realistic images that match the given
descriptions, while still maintain other features of original images.

negative:
We introduce a variety of models, trained on a supervised image captioning
corpus to predict the image features for a given caption, to perform sentence
representation grounding. We train a grounded sentence encoder that achieves
good performance on COCO caption and image retrieval and subsequently show that
this encoder can successfully be transferred to various NLP tasks, with
improved performance over text-only models. Lastly, we analyze the contribution
of grounding, and show that word embeddings learned by this system outperform
non-grounded ones.

negative:
Content-based information retrieval is based on the information contained in
documents rather than using metadata such as keywords. Most information
retrieval methods are either based on text or image. In this paper, we
investigate the usefulness of multimodal features for cross-lingual news search
in various domains: politics, health, environment, sport, and finance. To this
end, we consider five feature types for image and text and compare the
performance of the retrieval system using different combinations. Experimental
results show that retrieval results can be improved when considering both
visual and textual information. In addition, it is observed that among textual
features entity overlap outperforms word embeddings, while geolocation
embeddings achieve better performance among visual features in the retrieval
task.

negative:
In this paper, we study the task of image retrieval, where the input query is
specified in the form of an image plus some text that describes desired
modifications to the input image. For example, we may present an image of the
Eiffel tower, and ask the system to find images which are visually similar but
are modified in small ways, such as being taken at nighttime instead of during
the day. To tackle this task, we learn a similarity metric between a target
image and a source image plus source text, an embedding and composing function
such that target image feature is close to the source image plus text
composition feature. We propose a new way to combine image and text using such
function that is designed for the retrieval task. We show this outperforms
existing approaches on 3 different datasets, namely Fashion-200k, MIT-States
and a new synthetic dataset we create based on CLEVR. We also show that our
approach can be used to classify input queries, in addition to image retrieval.

negative:
In this paper we explore the bi-directional mapping between images and their
sentence-based descriptions. We propose learning this mapping using a recurrent
neural network. Unlike previous approaches that map both sentences and images
to a common embedding, we enable the generation of novel sentences given an
image. Using the same model, we can also reconstruct the visual features
associated with an image given its visual description. We use a novel recurrent
visual memory that automatically learns to remember long-term visual concepts
to aid in both sentence generation and visual feature reconstruction. We
evaluate our approach on several tasks. These include sentence generation,
sentence retrieval and image retrieval. State-of-the-art results are shown for
the task of generating novel image descriptions. When compared to human
generated captions, our automatically generated captions are preferred by
humans over $19.8\%$ of the time. Results are better than or comparable to
state-of-the-art results on the image and sentence retrieval tasks for methods
using similar visual features.

negative:
Generating a novel textual description of an image is an interesting problem
that connects computer vision and natural language processing. In this paper,
we present a simple model that is able to generate descriptive sentences given
a sample image. This model has a strong focus on the syntax of the
descriptions. We train a purely bilinear model that learns a metric between an
image representation (generated from a previously trained Convolutional Neural
Network) and phrases that are used to described them. The system is then able
to infer phrases from a given image sample. Based on caption syntax statistics,
we propose a simple language model that can produce relevant descriptions for a
given test image using the phrases inferred. Our approach, which is
considerably simpler than state-of-the-art models, achieves comparable results
in two popular datasets for the task: Flickr30k and the recently proposed
Microsoft COCO.

negative:
In this paper, we propose multimodal convolutional neural networks (m-CNNs)
for matching image and sentence. Our m-CNN provides an end-to-end framework
with convolutional architectures to exploit image representation, word
composition, and the matching relations between the two modalities. More
specifically, it consists of one image CNN encoding the image content, and one
matching CNN learning the joint representation of image and sentence. The
matching CNN composes words to different semantic fragments and learns the
inter-modal relations between image and the composed fragments at different
levels, thus fully exploit the matching relations between image and sentence.
Experimental results on benchmark databases of bidirectional image and sentence
retrieval demonstrate that the proposed m-CNNs can effectively capture the
information necessary for image and sentence matching. Specifically, our
proposed m-CNNs for bidirectional image and sentence retrieval on Flickr30K and
Microsoft COCO databases achieve the state-of-the-art performances.

negative:
Hypernymy, textual entailment, and image captioning can be seen as special
cases of a single visual-semantic hierarchy over words, sentences, and images.
In this paper we advocate for explicitly modeling the partial order structure
of this hierarchy. Towards this goal, we introduce a general method for
learning ordered representations, and show how it can be applied to a variety
of tasks involving images and language. We show that the resulting
representations improve performance over current approaches for hypernym
prediction and image-caption retrieval.

negative:
Image captioning is a research hotspot where encoder-decoder models combining
convolutional neural network (CNN) and long short-term memory (LSTM) achieve
promising results. Despite significant progress, these models generate
sentences differently from human cognitive styles. Existing models often
generate a complete sentence from the first word to the end, without
considering the influence of the following words on the whole sentence
generation. In this paper, we explore the utilization of a human-like cognitive
style, i.e., building overall cognition for the image to be described and the
sentence to be constructed, for enhancing computer image understanding. This
paper first proposes a Mutual-aid network structure with Bidirectional LSTMs
(MaBi-LSTMs) for acquiring overall contextual information. In the training
process, the forward and backward LSTMs encode the succeeding and preceding
words into their respective hidden states by simultaneously constructing the
whole sentence in a complementary manner. In the captioning process, the LSTM
implicitly utilizes the subsequent semantic information contained in its hidden
states. In fact, MaBi-LSTMs can generate two sentences in forward and backward
directions. To bridge the gap between cross-domain models and generate a
sentence with higher quality, we further develop a cross-modal attention
mechanism to retouch the two sentences by fusing their salient parts as well as
the salient areas of the image. Experimental results on the Microsoft COCO
dataset show that the proposed model improves the performance of
encoder-decoder models and achieves state-of-the-art results.

negative:
We present an approach to improve statistical machine translation of image
descriptions by multimodal pivots defined in visual space. The key idea is to
perform image retrieval over a database of images that are captioned in the
target language, and use the captions of the most similar images for
crosslingual reranking of translation outputs. Our approach does not depend on
the availability of large amounts of in-domain parallel data, but only relies
on available large datasets of monolingually captioned images, and on
state-of-the-art convolutional neural networks to compute image similarities.
Our experimental evaluation shows improvements of 1 BLEU point over strong
baselines.

negative:
The traditional image captioning task uses generic reference captions to
provide textual information about images. Different user populations, however,
will care about different visual aspects of images. In this paper, we propose a
new task, Captioning with a Purpose (CapWAP). Our goal is to develop systems
that can be tailored to be useful for the information needs of an intended
population, rather than merely provide generic information about an image. In
this task, we use question-answer (QA) pairs---a natural expression of
information need---from users, instead of reference captions, for both training
and post-inference evaluation. We show that it is possible to use reinforcement
learning to directly optimize for the intended information need, by rewarding
outputs that allow a question answering model to provide correct answers to
sampled user questions. We convert several visual question answering datasets
into CapWAP datasets, and demonstrate that under a variety of scenarios our
purposeful captioning system learns to anticipate and fulfill specific
information needs better than its generic counterparts, as measured by QA
performance on user questions from unseen images, when using the caption alone
as context.

negative:
A conversational agent (chatbot) is a piece of software that is able to
communicate with humans using natural language. Modeling conversation is an
important task in natural language processing and artificial intelligence.
While chatbots can be used for various tasks, in general they have to
understand users' utterances and provide responses that are relevant to the
problem at hand.
  In my work, I conduct an in-depth survey of recent literature, examining over
70 publications related to chatbots published in the last 3 years. Then, I
proceed to make the argument that the very nature of the general conversation
domain demands approaches that are different from current state-of-of-the-art
architectures. Based on several examples from the literature I show why current
chatbot models fail to take into account enough priors when generating
responses and how this affects the quality of the conversation. In the case of
chatbots, these priors can be outside sources of information that the
conversation is conditioned on like the persona or mood of the conversers. In
addition to presenting the reasons behind this problem, I propose several ideas
on how it could be remedied.
  The next section focuses on adapting the very recent Transformer model to the
chatbot domain, which is currently state-of-the-art in neural machine
translation. I first present experiments with the vanilla model, using
conversations extracted from the Cornell Movie-Dialog Corpus. Secondly, I
augment the model with some of my ideas regarding the issues of encoder-decoder
architectures. More specifically, I feed additional features into the model
like mood or persona together with the raw conversation data. Finally, I
conduct a detailed analysis of how the vanilla model performs on conversational
data by comparing it to previous chatbot models and how the additional features
affect the quality of the generated responses.

negative:
Localizing moments in a longer video via natural language queries is a new,
challenging task at the intersection of language and video understanding.
Though moment localization with natural language is similar to other language
and vision tasks like natural language object retrieval in images, moment
localization offers an interesting opportunity to model temporal dependencies
and reasoning in text. We propose a new model that explicitly reasons about
different temporal segments in a video, and shows that temporal context is
important for localizing phrases which include temporal language. To benchmark
whether our model, and other recent video localization models, can effectively
reason about temporal language, we collect the novel TEMPOral reasoning in
video and language (TEMPO) dataset. Our dataset consists of two parts: a
dataset with real videos and template sentences (TEMPO - Template Language)
which allows for controlled studies on temporal language, and a human language
dataset which consists of temporal sentences annotated by humans (TEMPO - Human
Language).

negative:
Image captioning models are becoming increasingly successful at describing
the content of images in restricted domains. However, if these models are to
function in the wild - for example, as assistants for people with impaired
vision - a much larger number and variety of visual concepts must be
understood. To address this problem, we teach image captioning models new
visual concepts from labeled images and object detection datasets. Since image
labels and object classes can be interpreted as partial captions, we formulate
this problem as learning from partially-specified sequence data. We then
propose a novel algorithm for training sequence models, such as recurrent
neural networks, on partially-specified sequences which we represent using
finite state automata. In the context of image captioning, our method lifts the
restriction that previously required image captioning models to be trained on
paired image-sentence corpora only, or otherwise required specialized model
architectures to take advantage of alternative data modalities. Applying our
approach to an existing neural captioning model, we achieve state of the art
results on the novel object captioning task using the COCO dataset. We further
show that we can train a captioning model to describe new visual concepts from
the Open Images dataset while maintaining competitive COCO evaluation scores.

negative:
In this work we propose a simple and efficient framework for learning
sentence representations from unlabelled data. Drawing inspiration from the
distributional hypothesis and recent work on learning sentence representations,
we reformulate the problem of predicting the context in which a sentence
appears as a classification problem. Given a sentence and its context, a
classifier distinguishes context sentences from other contrastive sentences
based on their vector representations. This allows us to efficiently learn
different types of encoding functions, and we show that the model learns
high-quality sentence representations. We demonstrate that our sentence
representations outperform state-of-the-art unsupervised and supervised
representation learning methods on several downstream NLP tasks that involve
understanding sentence semantics while achieving an order of magnitude speedup
in training time.

negative:
Learning distributed sentence representations remains an interesting problem
in the field of Natural Language Processing (NLP). We want to learn a model
that approximates the conditional latent space over the representations of a
logical antecedent of the given statement. In our paper, we propose an approach
to generating sentences, conditioned on an input sentence and a logical
inference label. We do this by modeling the different possibilities for the
output sentence as a distribution over the latent representation, which we
train using an adversarial objective. We evaluate the model using two
state-of-the-art models for the Recognizing Textual Entailment (RTE) task, and
measure the BLEU scores against the actual sentences as a probe for the
diversity of sentences produced by our model. The experiment results show that,
given our framework, we have clear ways to improve the quality and diversity of
generated sentences.

negative:
Automatically creating the description of an image using any natural
languages sentence like English is a very challenging task. It requires
expertise of both image processing as well as natural language processing. This
paper discuss about different available models for image captioning task. We
have also discussed about how the advancement in the task of object recognition
and machine translation has greatly improved the performance of image
captioning model in recent years. In addition to that we have discussed how
this model can be implemented. In the end, we have also evaluated the
performance of model using standard evaluation matrices.

negative:
In this paper, we propose a new approach for retrieval of video segments
using natural language queries. Unlike most previous approaches such as
concept-based methods or rule-based structured models, the proposed method uses
image captioning model to construct sentential queries for visual information.
In detail, our approach exploits multiple captions generated by visual features
in each image with `Densecap'. Then, the similarities between captions of
adjacent images are calculated, which is used to track semantically similar
captions over multiple frames. Besides introducing this novel idea of 'tracking
by captioning', the proposed method is one of the first approaches that uses a
language generation model learned by neural networks to construct semantic
query describing the relations and properties of visual information. To
evaluate the effectiveness of our approach, we have created a new evaluation
dataset, which contains about 348 segments of scenes in 20 movie-trailers.
Through quantitative and qualitative evaluation, we show that our method is
effective for retrieval of video segments using natural language queries.

negative:
Distributed representation plays an important role in deep learning based
natural language processing. However, the representation of a sentence often
varies in different tasks, which is usually learned from scratch and suffers
from the limited amounts of training data. In this paper, we claim that a good
sentence representation should be invariant and can benefit the various
subsequent tasks. To achieve this purpose, we propose a new scheme of
information sharing for multi-task learning. More specifically, all tasks share
the same sentence representation and each task can select the task-specific
information from the shared sentence representation with attention mechanism.
The query vector of each task's attention could be either static parameters or
generated dynamically. We conduct extensive experiments on 16 different text
classification tasks, which demonstrate the benefits of our architecture.

negative:
Visual-semantic embedding aims to find a shared latent space where related
visual and textual instances are close to each other. Most current methods
learn injective embedding functions that map an instance to a single point in
the shared space. Unfortunately, injective embedding cannot effectively handle
polysemous instances with multiple possible meanings; at best, it would find an
average representation of different meanings. This hinders its use in
real-world scenarios where individual instances and their cross-modal
associations are often ambiguous. In this work, we introduce Polysemous
Instance Embedding Networks (PIE-Nets) that compute multiple and diverse
representations of an instance by combining global context with locally-guided
features via multi-head self-attention and residual learning. To learn
visual-semantic embedding, we tie-up two PIE-Nets and optimize them jointly in
the multiple instance learning framework. Most existing work on cross-modal
retrieval focuses on image-text data. Here, we also tackle a more challenging
case of video-text retrieval. To facilitate further research in video-text
retrieval, we release a new dataset of 50K video-sentence pairs collected from
social media, dubbed MRW (my reaction when). We demonstrate our approach on
both image-text and video-text retrieval scenarios using MS-COCO, TGIF, and our
new MRW dataset.

negative:
Knowledge-intensive NLP tasks can benefit from linking natural language text
with facts from a Knowledge Graph (KG). Although facts themselves are
language-agnostic, the fact labels (i.e., language-specific representation of
the fact) in the KG are often present only in a few languages. This makes it
challenging to link KG facts to sentences in languages other than the limited
set of languages. To address this problem, we introduce the task of
Multilingual Fact Linking (MFL) where the goal is to link fact expressed in a
sentence to corresponding fact in the KG, even when the fact label in the KG is
not available in the language of the sentence. To facilitate research in this
area, we present a new evaluation dataset, IndicLink. This dataset contains
11,293 linked WikiData facts and 6,429 sentences spanning English and six
Indian languages. We propose a Retrieval+Generation model, ReFCoG, that can
scale to millions of KG facts by combining Dual Encoder based retrieval with a
Seq2Seq based generation model which is constrained to output only valid KG
facts. ReFCoG outperforms standard Retrieval+Re-ranking models by 10.7 pts in
Precision@1. In spite of this gain, the model achieves an overall score of
52.1, showing ample scope for improvement in the task.ReFCoG code and IndicLink
data are available at https://github.com/SaiKeshav/mfl

negative:
A creative image-and-text generative AI system mimics humans' extraordinary
abilities to provide users with diverse and comprehensive caption suggestions,
as well as rich image creations. In this work, we demonstrate such an AI
creation system to produce both diverse captions and rich images. When users
imagine an image and associate it with multiple captions, our system paints a
rich image to reflect all captions faithfully. Likewise, when users upload an
image, our system depicts it with multiple diverse captions. We propose a
unified multi-modal framework to achieve this goal. Specifically, our framework
jointly models image-and-text representations with a Transformer network, which
supports rich image creation by accepting multiple captions as input. We
consider the relations among input captions to encourage diversity in training
and adopt a non-autoregressive decoding strategy to enable real-time inference.
Based on these, our system supports both diverse captions and rich images
generations. Our code is available online.

negative:
Discovering the logical sequence of events is one of the cornerstones in
Natural Language Understanding. One approach to learn the sequence of events is
to study the order of sentences in a coherent text. Sentence ordering can be
applied in various tasks such as retrieval-based Question Answering, document
summarization, storytelling, text generation, and dialogue systems.
Furthermore, we can learn to model text coherence by learning how to order a
set of shuffled sentences. Previous research has relied on RNN, LSTM, and
BiLSTM architecture for learning text language models. However, these networks
have performed poorly due to the lack of attention mechanisms. We propose an
algorithm for sentence ordering in a corpus of short stories. Our proposed
method uses a language model based on Universal Transformers (UT) that captures
sentences' dependencies by employing an attention mechanism. Our method
improves the previous state-of-the-art in terms of Perfect Match Ratio (PMR)
score in the ROCStories dataset, a corpus of nearly 100K short human-made
stories. The proposed model includes three components: Sentence Encoder,
Language Model, and Sentence Arrangement with Brute Force Search. The first
component generates sentence embeddings using SBERT-WK pre-trained model
fine-tuned on the ROCStories data. Then a Universal Transformer network
generates a sentence-level language model. For decoding, the network generates
a candidate sentence as the following sentence of the current sentence. We use
cosine similarity as a scoring function to assign scores to the candidate
embedding and the embeddings of other sentences in the shuffled set. Then a
Brute Force Search is employed to maximize the sum of similarities between
pairs of consecutive sentences.

negative:
Generating a description of an image is called image captioning. Image
captioning requires to recognize the important objects, their attributes and
their relationships in an image. It also needs to generate syntactically and
semantically correct sentences. Deep learning-based techniques are capable of
handling the complexities and challenges of image captioning. In this survey
paper, we aim to present a comprehensive review of existing deep learning-based
image captioning techniques. We discuss the foundation of the techniques to
analyze their performances, strengths and limitations. We also discuss the
datasets and the evaluation metrics popularly used in deep learning based
automatic image captioning.

negative:
Deep neural networks have achieved great successes on the image captioning
task. However, most of the existing models depend heavily on paired
image-sentence datasets, which are very expensive to acquire. In this paper, we
make the first attempt to train an image captioning model in an unsupervised
manner. Instead of relying on manually labeled image-sentence pairs, our
proposed model merely requires an image set, a sentence corpus, and an existing
visual concept detector. The sentence corpus is used to teach the captioning
model how to generate plausible sentences. Meanwhile, the knowledge in the
visual concept detector is distilled into the captioning model to guide the
model to recognize the visual concepts in an image. In order to further
encourage the generated captions to be semantically consistent with the image,
the image and caption are projected into a common latent space so that they can
reconstruct each other. Given that the existing sentence corpora are mainly
designed for linguistic research and are thus with little reference to image
contents, we crawl a large-scale image description corpus of two million
natural sentences to facilitate the unsupervised image captioning scenario.
Experimental results show that our proposed model is able to produce quite
promising results without any caption annotations.

negative:
Image captioning aims at automatically generating descriptions of an image in
natural language. This is a challenging problem in the field of artificial
intelligence that has recently received significant attention in the computer
vision and natural language processing. Among the existing approaches, visual
retrieval based methods have been proven to be highly effective. These
approaches search for similar images, then build a caption for the query image
based on the captions of the retrieved images. In this study, we present a
method for visual retrieval based image captioning, in which we use a multi
criteria decision making algorithm to effectively combine several criteria with
proportional impact weights to retrieve the most relevant caption for the query
image. The main idea of the proposed approach is to design a mechanism to
retrieve more semantically relevant captions with the query image and then
selecting the most appropriate caption by imitation of the human act based on a
weighted multi-criteria decision making algorithm. Experiments conducted on MS
COCO benchmark dataset have shown that proposed method provides much more
effective results in compare to the state-of-the-art models by using criteria
with proportional impact weights .

negative:
This paper discusses and demonstrates the outcomes from our experimentation
on Image Captioning. Image captioning is a much more involved task than image
recognition or classification, because of the additional challenge of
recognizing the interdependence between the objects/concepts in the image and
the creation of a succinct sentential narration. Experiments on several labeled
datasets show the accuracy of the model and the fluency of the language it
learns solely from image descriptions. As a toy application, we apply image
captioning to create video captions, and we advance a few hypotheses on the
challenges we encountered.

negative:
We present an easy and efficient method to extend existing sentence embedding
models to new languages. This allows to create multilingual versions from
previously monolingual models. The training is based on the idea that a
translated sentence should be mapped to the same location in the vector space
as the original sentence. We use the original (monolingual) model to generate
sentence embeddings for the source language and then train a new system on
translated sentences to mimic the original model. Compared to other methods for
training multilingual sentence embeddings, this approach has several
advantages: It is easy to extend existing models with relatively few samples to
new languages, it is easier to ensure desired properties for the vector space,
and the hardware requirements for training is lower. We demonstrate the
effectiveness of our approach for 50+ languages from various language families.
Code to extend sentence embeddings models to more than 400 languages is
publicly available.

negative:
Sentiment analysis is a research topic focused on analysing data to extract
information related to the sentiment that it causes. Applications of sentiment
analysis are wide, ranging from recommendation systems, and marketing to
customer satisfaction. Recent approaches evaluate textual content using Machine
Learning techniques that are trained over large corpora. However, as social
media grown, other data types emerged in large quantities, such as images.
Sentiment analysis in images has shown to be a valuable complement to textual
data since it enables the inference of the underlying message polarity by
creating context and connections. Multimodal sentiment analysis approaches
intend to leverage information of both textual and image content to perform an
evaluation. Despite recent advances, current solutions still flounder in
combining both image and textual information to classify social media data,
mainly due to subjectivity, inter-class homogeneity and fusion data
differences. In this paper, we propose a method that combines both textual and
image individual sentiment analysis into a final fused classification based on
AutoML, that performs a random search to find the best model. Our method
achieved state-of-the-art performance in the B-T4SA dataset, with 95.19%
accuracy.

negative:
In this paper, we propose a multimodal search engine that combines visual and
textual cues to retrieve items from a multimedia database aesthetically similar
to the query. The goal of our engine is to enable intuitive retrieval of
fashion merchandise such as clothes or furniture. Existing search engines treat
textual input only as an additional source of information about the query image
and do not correspond to the real-life scenario where the user looks for 'the
same shirt but of denim'. Our novel method, dubbed DeepStyle, mitigates those
shortcomings by using a joint neural network architecture to model contextual
dependencies between features of different modalities. We prove the robustness
of this approach on two different challenging datasets of fashion items and
furniture where our DeepStyle engine outperforms baseline methods by 18-21% on
the tested datasets. Our search engine is commercially deployed and available
through a Web-based application.

negative:
The task of image-text matching aims to map representations from different
modalities into a common joint visual-textual embedding. However, the most
widely used datasets for this task, MSCOCO and Flickr30K, are actually image
captioning datasets that offer a very limited set of relationships between
images and sentences in their ground-truth annotations. This limited ground
truth information forces us to use evaluation metrics based on binary
relevance: given a sentence query we consider only one image as relevant.
However, many other relevant images or captions may be present in the dataset.
In this work, we propose two metrics that evaluate the degree of semantic
relevance of retrieved items, independently of their annotated binary
relevance. Additionally, we incorporate a novel strategy that uses an image
captioning metric, CIDEr, to define a Semantic Adaptive Margin (SAM) to be
optimized in a standard triplet loss. By incorporating our formulation to
existing models, a \emph{large} improvement is obtained in scenarios where
available training data is limited. We also demonstrate that the performance on
the annotated image-caption pairs is maintained while improving on other
non-annotated relevant items when employing the full training set. Code with
our metrics and adaptive margin formulation will be made public.

negative:
In this technical report I present my method for automatic synthetic dataset
generation for object detection and demonstrate it on the video game League of
Legends. This report furthermore serves as a handbook on how to automatically
generate datasets and as an introduction on the dataset generation part of the
LeagueAI framework. The LeagueAI framework is a software framework that
provides detailed information about the game League of Legends based on the
same input a human player would have, namely vision. The framework allows
researchers and enthusiasts to develop their own intelligent agents or to
extract detailed information about the state of the game. A big problem of
machine vision applications usually is the laborious work of gathering large
amounts of hand labeled data. Thus, a crucial part of the vision pipeline of
the LeagueAI framework, the dataset generation, is presented in this report.
The method involves extracting image raw data from the game's 3D models and
combining them with the game background to create game-like synthetic images
and to generate the corresponding labels automatically. In an experiment I
compared a model trained on synthetic data to a model trained on hand labeled
data and a model trained on a combined dataset. The model trained on the
synthetic data showed higher detection precision on more classes and more
reliable tracking performance of the player character. The model trained on the
combined dataset did not perform better because of the different formats of the
older hand labeled dataset and the synthetic data.

negative:
Feature visualizations such as synthetic maximally activating images are a
widely used explanation method to better understand the information processing
of convolutional neural networks (CNNs). At the same time, there are concerns
that these visualizations might not accurately represent CNNs' inner workings.
Here, we measure how much extremely activating images help humans to predict
CNN activations. Using a well-controlled psychophysical paradigm, we compare
the informativeness of synthetic images by Olah et al. (2017) with a simple
baseline visualization, namely exemplary natural images that also strongly
activate a specific feature map. Given either synthetic or natural reference
images, human participants choose which of two query images leads to strong
positive activation. The experiments are designed to maximize participants'
performance, and are the first to probe intermediate instead of final layer
representations. We find that synthetic images indeed provide helpful
information about feature map activations ($82\pm4\%$ accuracy; chance would be
$50\%$). However, natural images - originally intended as a baseline -
outperform synthetic images by a wide margin ($92\pm2\%$). Additionally,
participants are faster and more confident for natural images, whereas
subjective impressions about the interpretability of the feature visualizations
are mixed. The higher informativeness of natural images holds across most
layers, for both expert and lay participants as well as for hand- and
randomly-picked feature visualizations. Even if only a single reference image
is given, synthetic images provide less information than natural images
($65\pm5\%$ vs. $73\pm4\%$). In summary, synthetic images from a popular
feature visualization method are significantly less informative for assessing
CNN activations than natural images. We argue that visualization methods should
improve over this baseline.

negative:
In this paper, we use the framework of neural machine translation to learn
joint sentence representations across six very different languages. Our aim is
that a representation which is independent of the language, is likely to
capture the underlying semantics. We define a new cross-lingual similarity
measure, compare up to 1.4M sentence representations and study the
characteristics of close sentences. We provide experimental evidence that
sentences that are close in embedding space are indeed semantically highly
related, but often have quite different structure and syntax. These relations
also hold when comparing sentences in different languages.

negative:
Recent literature shows that large-scale language modeling provides excellent
reusable sentence representations with both recurrent and self-attentive
architectures. However, there has been less clarity on the commonalities and
differences in the representational properties induced by the two
architectures. It also has been shown that visual information serves as one of
the means for grounding sentence representations. In this paper, we present a
meta-study assessing the representational quality of models where the training
signal is obtained from different modalities, in particular, language modeling,
image features prediction, and both textual and multimodal machine translation.
We evaluate textual and visual features of sentence representations obtained
using predominant approaches on image retrieval and semantic textual
similarity. Our experiments reveal that on moderate-sized datasets, a sentence
counterpart in a target language or visual modality provides much stronger
training signal for sentence representation than language modeling.
Importantly, we observe that while the Transformer models achieve superior
machine translation quality, representations from the recurrent neural network
based models perform significantly better over tasks focused on semantic
relevance.

negative:
Textual network embeddings aim to learn a low-dimensional representation for
every node in the network so that both the structural and textual information
from the networks can be well preserved in the representations. Traditionally,
the structural and textual embeddings were learned by models that rarely take
the mutual influences between them into account. In this paper, a deep neural
architecture is proposed to effectively fuse the two kinds of informations into
one representation. The novelties of the proposed architecture are manifested
in the aspects of a newly defined objective function, the complementary
information fusion method for structural and textual features, and the mutual
gate mechanism for textual feature extraction. Experimental results show that
the proposed model outperforms the comparing methods on all three datasets.

negative:
In this paper, we describe our end-to-end content-based image retrieval
system built upon Elasticsearch, a well-known and popular textual search
engine. As far as we know, this is the first time such a system has been
implemented in eCommerce, and our efforts have turned out to be highly
worthwhile. We end up with a novel and exciting visual search solution that is
extremely easy to be deployed, distributed, scaled and monitored in a
cost-friendly manner. Moreover, our platform is intrinsically flexible in
supporting multimodal searches, where visual and textual information can be
jointly leveraged in retrieval.
  The core idea is to encode image feature vectors into a collection of string
tokens in a way such that closer vectors will share more string tokens in
common. By doing that, we can utilize Elasticsearch to efficiently retrieve
similar images based on similarities within encoded sting tokens. As part of
the development, we propose a novel vector to string encoding method, which is
shown to substantially outperform the previous ones in terms of both precision
and latency.
  First-hand experiences in implementing this Elasticsearch-based platform are
extensively addressed, which should be valuable to practitioners also
interested in building visual search engine on top of Elasticsearch.

negative:
For general modeling methods applied to diverse languages, a natural question
is: how well should we expect our models to work on languages with differing
typological profiles? In this work, we develop an evaluation framework for fair
cross-linguistic comparison of language models, using translated text so that
all models are asked to predict approximately the same information. We then
conduct a study on 21 languages, demonstrating that in some languages, the
textual expression of the information is harder to predict with both $n$-gram
and LSTM language models. We show complex inflectional morphology to be a cause
of performance differences among languages.

negative:
In recent years, automatic generation of image descriptions (captions), that
is, image captioning, has attracted a great deal of attention. In this paper,
we particularly consider generating Japanese captions for images. Since most
available caption datasets have been constructed for English language, there
are few datasets for Japanese. To tackle this problem, we construct a
large-scale Japanese image caption dataset based on images from MS-COCO, which
is called STAIR Captions. STAIR Captions consists of 820,310 Japanese captions
for 164,062 images. In the experiment, we show that a neural network trained
using STAIR Captions can generate more natural and better Japanese captions,
compared to those generated using English-Japanese machine translation after
generating English captions.

negative:
We present COSTRA 1.0, a dataset of complex sentence transformations. The
dataset is intended for the study of sentence-level embeddings beyond simple
word alternations or standard paraphrasing. This first version of the dataset
is limited to sentences in Czech but the construction method is universal and
we plan to use it also for other languages. The dataset consist of 4,262 unique
sentences with average length of 10 words, illustrating 15 types of
modifications such as simplification, generalization, or formal and informal
language variation. The hope is that with this dataset, we should be able to
test semantic properties of sentence embeddings and perhaps even to find some
topologically interesting 'skeleton' in the sentence embedding space. A
preliminary analysis using LASER, multi-purpose multi-lingual sentence
embeddings suggests that the LASER space does not exhibit the desired
properties.

negative:
The standard recurrent neural network language model (RNNLM) generates
sentences one word at a time and does not work from an explicit global sentence
representation. In this work, we introduce and study an RNN-based variational
autoencoder generative model that incorporates distributed latent
representations of entire sentences. This factorization allows it to explicitly
model holistic properties of sentences such as style, topic, and high-level
syntactic features. Samples from the prior over these sentence representations
remarkably produce diverse and well-formed sentences through simple
deterministic decoding. By examining paths through this latent space, we are
able to generate coherent novel sentences that interpolate between known
sentences. We present techniques for solving the difficult learning problem
presented by this model, demonstrate its effectiveness in imputing missing
words, explore many interesting properties of the model's latent sentence
space, and present negative results on the use of the model in language
modeling.

negative:
Generative models have made significant progress in the tasks of modeling
complex data distributions such as natural images. The introduction of
Generative Adversarial Networks (GANs) and auto-encoders lead to the
possibility of training on big data sets in an unsupervised manner. However,
for many generative models it is not possible to specify what kind of image
should be generated and it is not possible to translate existing images into
new images of similar domains. Furthermore, models that can perform
image-to-image translation often need distinct models for each domain, making
it hard to scale these systems to multiple domain image-to-image translation.
We introduce a model that can do both, controllable image generation and
image-to-image translation between multiple domains. We split our image
representation into two parts encoding unstructured and structured information
respectively. The latter is designed in a disentangled manner, so that
different parts encode different image characteristics. We train an encoder to
encode images into these representations and use a small amount of labeled data
to specify what kind of information should be encoded in the disentangled part.
A generator is trained to generate images from these representations using the
characteristics provided by the disentangled part of the representation.
Through this we can control what kind of images the generator generates,
translate images between different domains, and even learn unknown
data-generating factors while only using one single model.

negative:
Few-Shot Learning is the challenge of training a model with only a small
amount of data. Many solutions to this problem use meta-learning algorithms,
i.e. algorithms that learn to learn. By sampling few-shot tasks from a larger
dataset, we can teach these algorithms to solve new, unseen tasks. This
document reports my work on meta-learning algorithms for Few-Shot Computer
Vision. This work was done during my internship at Sicara, a French company
building image recognition solutions for businesses. It contains: 1. an
extensive review of the state-of-the-art in few-shot computer vision; 2. a
benchmark of meta-learning algorithms for few-shot image classification; 3. the
introduction to a novel meta-learning algorithm for few-shot object detection,
which is still in development.

negative:
There is a lot of research interest in encoding variable length sentences
into fixed length vectors, in a way that preserves the sentence meanings. Two
common methods include representations based on averaging word vectors, and
representations based on the hidden states of recurrent neural networks such as
LSTMs. The sentence vectors are used as features for subsequent machine
learning tasks or for pre-training in the context of deep learning. However,
not much is known about the properties that are encoded in these sentence
representations and about the language information they capture. We propose a
framework that facilitates better understanding of the encoded representations.
We define prediction tasks around isolated aspects of sentence structure
(namely sentence length, word content, and word order), and score
representations by the ability to train a classifier to solve each prediction
task when using the representation as input. We demonstrate the potential
contribution of the approach by analyzing different sentence representation
mechanisms. The analysis sheds light on the relative strengths of different
sentence embedding methods with respect to these low level prediction tasks,
and on the effect of the encoded vector's dimensionality on the resulting
representations.

negative:
In order to operate autonomously, a robot should explore the environment and
build a model of each of the surrounding objects. A common approach is to
carefully scan the whole workspace. This is time-consuming. It is also often
impossible to reach all the viewpoints required to acquire full knowledge about
the environment. Humans can perform shape completion of occluded objects by
relying on past experience. Therefore, we propose a method that generates
images of an object from various viewpoints using a single input RGB image. A
deep neural network is trained to imagine the object appearance from many
viewpoints. We present the whole pipeline, which takes a single RGB image as
input and returns a sequence of RGB and depth images of the object. The method
utilizes a CNN-based object detector to extract the object from the natural
scene. Then, the proposed network generates a set of RGB and depth images. We
show the results both on a synthetic dataset and on real images.

negative:
We propose an approach to build a neural machine translation system with no
supervised resources (i.e., no parallel corpora) using multimodal embedded
representation over texts and images. Based on the assumption that text
documents are often likely to be described with other multimedia information
(e.g., images) somewhat related to the content, we try to indirectly estimate
the relevance between two languages. Using multimedia as the "pivot", we
project all modalities into one common hidden space where samples belonging to
similar semantic concepts should come close to each other, whatever the
observed space of each sample is. This modality-agnostic representation is the
key to bridging the gap between different modalities. Putting a decoder on top
of it, our network can flexibly draw the outputs from any input modality.
Notably, in the testing phase, we need only source language texts as the input
for translation. In experiments, we tested our method on two benchmarks to show
that it can achieve reasonable translation performance. We compared and
investigated several possible implementations and found that an end-to-end
model that simultaneously optimized both rank loss in multimodal encoders and
cross-entropy loss in decoders performed the best.

negative:
Pair-based metric learning has been widely adopted to learn sentence
embedding in many NLP tasks such as semantic text similarity due to its
efficiency in computation. Most existing works employed a sequence encoder
model and utilized limited sentence pairs with a pair-based loss to learn
discriminating sentence representation. However, it is known that the sentence
representation can be biased when the sampled sentence pairs deviate from the
true distribution of all sentence pairs. In this paper, our theoretical
analysis shows that existing works severely suffered from a good pair sampling
and instance weighting strategy. Instead of one time pair selection and
learning on equal weighted pairs, we propose a unified locality weighting and
learning framework to learn task-specific sentence embedding. Our model,
SentPWNet, exploits the neighboring spatial distribution of each sentence as
locality weight to indicate the informative level of sentence pair. Such weight
is updated along with pair-loss optimization in each round, ensuring the model
keep learning the most informative sentence pairs. Extensive experiments on
four public available datasets and a self-collected place search benchmark with
1.4 million places clearly demonstrate that our model consistently outperforms
existing sentence embedding methods with comparable efficiency.

negative:
While most machine translation systems to date are trained on large parallel
corpora, humans learn language in a different way: by being grounded in an
environment and interacting with other humans. In this work, we propose a
communication game where two agents, native speakers of their own respective
languages, jointly learn to solve a visual referential task. We find that the
ability to understand and translate a foreign language emerges as a means to
achieve shared goals. The emergent translation is interactive and multimodal,
and crucially does not require parallel corpora, but only monolingual,
independent text and corresponding images. Our proposed translation model
achieves this by grounding the source and target languages into a shared visual
modality, and outperforms several baselines on both word-level and
sentence-level translation tasks. Furthermore, we show that agents in a
multilingual community learn to translate better and faster than in a bilingual
communication setting.

negative:
Image captioning systems have recently improved dramatically, but they still
tend to produce captions that are insensitive to the communicative goals that
captions should meet. To address this, we propose Issue-Sensitive Image
Captioning (ISIC). In ISIC, a captioning system is given a target image and an
issue, which is a set of images partitioned in a way that specifies what
information is relevant. The goal of the captioner is to produce a caption that
resolves this issue. To model this task, we use an extension of the Rational
Speech Acts model of pragmatic language use. Our extension is built on top of
state-of-the-art pretrained neural image captioners and explicitly reasons
about issues in our sense. We establish experimentally that these models
generate captions that are both highly descriptive and issue-sensitive, and we
show how ISIC can complement and enrich the related task of Visual Question
Answering.

negative:
Automatically generating natural language descriptions from an image is a
challenging problem in artificial intelligence that requires a good
understanding of the visual and textual signals and the correlations between
them. The state-of-the-art methods in image captioning struggles to approach
human level performance, especially when data is limited. In this paper, we
propose to improve the performance of the state-of-the-art image captioning
models by incorporating two sources of prior knowledge: (i) a conditional
latent topic attention, that uses a set of latent variables (topics) as an
anchor to generate highly probable words and, (ii) a regularization technique
that exploits the inductive biases in syntactic and semantic structure of
captions and improves the generalization of image captioning models. Our
experiments validate that our method produces more human interpretable captions
and also leads to significant improvements on the MSCOCO dataset in both the
full and low data regimes.

negative:
Image captioning models have been able to generate grammatically correct and
human understandable sentences. However most of the captions convey limited
information as the model used is trained on datasets that do not caption all
possible objects existing in everyday life. Due to this lack of prior
information most of the captions are biased to only a few objects present in
the scene, hence limiting their usage in daily life. In this paper, we attempt
to show the biased nature of the currently existing image captioning models and
present a new image captioning dataset, Egoshots, consisting of 978 real life
images with no captions. We further exploit the state of the art pre-trained
image captioning and object recognition networks to annotate our images and
show the limitations of existing works. Furthermore, in order to evaluate the
quality of the generated captions, we propose a new image captioning metric,
object based Semantic Fidelity (SF). Existing image captioning metrics can
evaluate a caption only in the presence of their corresponding annotations;
however, SF allows evaluating captions generated for images without
annotations, making it highly useful for real life generated captions.

negative:
Every person speaks or writes their own flavor of their native language,
influenced by a number of factors: the content they tend to talk about, their
gender, their social status, or their geographical origin.
  When attempting to perform Machine Translation (MT), these variations have a
significant effect on how the system should perform translation, but this is
not captured well by standard one-size-fits-all models.
  In this paper, we propose a simple and parameter-efficient adaptation
technique that only requires adapting the bias of the output softmax to each
particular user of the MT system, either directly or through a factored
approximation.
  Experiments on TED talks in three languages demonstrate improvements in
translation accuracy, and better reflection of speaker traits in the target
text.

negative:
Dialogue systems attempt to facilitate conversations between humans and
computers, for purposes as diverse as small talk to booking a vacation. We are
here inspired by the performance of the recurrent neural network-based model
Sequicity, which when conducting a dialogue uses a sequence-to-sequence
architecture to first produce a textual representation of what is going on in
the dialogue, and in a further step use this along with database findings to
produce a reply to the user. We here propose a dialogue system based on the
Transformer architecture instead of Sequicity's RNN-based architecture, that
works similarly in an end-to-end, sequence-to-sequence fashion.

negative:
In this paper, we propose a multi-modal search engine for interior design
that combines visual and textual queries. The goal of our engine is to retrieve
interior objects, e.g. furniture or wall clocks, that share visual and
aesthetic similarities with the query. Our search engine allows the user to
take a photo of a room and retrieve with a high recall a list of items
identical or visually similar to those present in the photo. Additionally, it
allows to return other items that aesthetically and stylistically fit well
together. To achieve this goal, our system blends the results obtained using
textual and visual modalities. Thanks to this blending strategy, we increase
the average style similarity score of the retrieved items by 11%. Our work is
implemented as a Web-based application and it is planned to be opened to the
public.

negative:
Automatic image cropping techniques are commonly used to enhance the
aesthetic quality of an image; they do it by detecting the most beautiful or
the most salient parts of the image and removing the unwanted content to have a
smaller image that is more visually pleasing. In this thesis, I introduce an
additional dimension to the problem of cropping, semantics. I argue that image
cropping can also enhance the image's relevancy for a given entity by using the
semantic information contained in the image. I call this problem, Semantic
Image Cropping. To support my argument, I provide a new dataset containing 100
images with at least two different entities per image and four ground truth
croppings collected using Amazon Mechanical Turk. I use this dataset to show
that state-of-the-art cropping algorithms that only take into account
aesthetics do not perform well in the problem of semantic image cropping.
Additionally, I provide a new deep learning system that takes not just
aesthetics but also semantics into account to generate image croppings, and I
evaluate its performance using my new semantic cropping dataset, showing that
using the semantic information of an image can help to produce better
croppings.

negative:
Sentence embedding methods offer a powerful approach for working with short
textual constructs or sequences of words. By representing sentences as dense
numerical vectors, many natural language processing (NLP) applications have
improved their performance. However, relatively little is understood about the
latent structure of sentence embeddings. Specifically, research has not
addressed whether the length and structure of sentences impact the sentence
embedding space and topology. This paper reports research on a set of
comprehensive clustering and network analyses targeting sentence and
sub-sentence embedding spaces. Results show that one method generates the most
clusterable embeddings. In general, the embeddings of span sub-sentences have
better clustering properties than the original sentences. The results have
implications for future sentence embedding models and applications.

negative:
Natural language understanding involves reading between the lines with
implicit background knowledge. Current systems either rely on pre-trained
language models as the sole implicit source of world knowledge, or resort to
external knowledge bases (KBs) to incorporate additional relevant knowledge. We
propose an unsupervised framework based on self-talk as a novel alternative to
multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning
(Bruner, 1961), our approach inquires language models with a number of
information seeking questions such as "$\textit{what is the definition of
...}$" to discover additional background knowledge. Empirical results
demonstrate that the self-talk procedure substantially improves the performance
of zero-shot language model baselines on four out of six commonsense
benchmarks, and competes with models that obtain knowledge from external KBs.
While our approach improves performance on several benchmarks, the self-talk
induced knowledge even when leading to correct answers is not always seen as
useful by human judges, raising interesting questions about the inner-workings
of pre-trained language models for commonsense reasoning.

negative:
Data augmentation is an effective performance enhancement in neural machine
translation (NMT) by generating additional bilingual data. In this paper, we
propose a novel data augmentation enhancement strategy for neural machine
translation. Different from existing data augmentation methods which simply
choose words with the same probability across different sentences for
modification, we set sentence-specific probability for word selection by
considering their roles in sentence. We use dependency parse tree of input
sentence as an effective clue to determine selecting probability for every
words in each sentence. Our proposed method is evaluated on WMT14
English-to-German dataset and IWSLT14 German-to-English dataset. The result of
extensive experiments show our proposed syntax-aware data augmentation method
may effectively boost existing sentence-independent methods for significant
translation performance improvement.

negative:
Artificial agents today can answer factual questions. But they fall short on
questions that require common sense reasoning. Perhaps this is because most
existing common sense databases rely on text to learn and represent knowledge.
But much of common sense knowledge is unwritten - partly because it tends not
to be interesting enough to talk about, and partly because some common sense is
unnatural to articulate in text. While unwritten, it is not unseen. In this
paper we leverage semantic common sense knowledge learned from images - i.e.
visual common sense - in two textual tasks: fill-in-the-blank and visual
paraphrasing. We propose to "imagine" the scene behind the text, and leverage
visual cues from the "imagined" scenes in addition to textual cues while
answering these questions. We imagine the scenes as a visual abstraction. Our
approach outperforms a strong text-only baseline on these tasks. Our proposed
tasks can serve as benchmarks to quantitatively evaluate progress in solving
tasks that go "beyond recognition". Our code and datasets are publicly
available.

negative:
Everyone knows that thousand of words are represented by a single image. As a
result image search has become a very popular mechanism for the Web searchers.
Image search means, the search results are produced by the search engine should
be a set of images along with their Web page Unified Resource Locator. Now Web
searcher can perform two types of image search, they are Text to Image and
Image to Image search. In Text to Image search, search query should be a text.
Based on the input text data system will generate a set of images along with
their Web page URL as an output. On the other hand, in Image to Image search,
search query should be an image and based on this image system will generate a
set of images along with their Web page URL as an output. According to the
current scenarios, Text to Image search mechanism always not returns perfect
result. It matches the text data and then displays the corresponding images as
an output, which is not always perfect. To resolve this problem, Web
researchers have introduced the Image to Image search mechanism. In this paper,
we have also proposed an alternate approach of Image to Image search mechanism
using Histogram.

negative:
When writing a summary, humans tend to choose content from one or two
sentences and merge them into a single summary sentence. However, the
mechanisms behind the selection of one or multiple source sentences remain
poorly understood. Sentence fusion assumes multi-sentence input; yet sentence
selection methods only work with single sentences and not combinations of them.
There is thus a crucial gap between sentence selection and fusion to support
summarizing by both compressing single sentences and fusing pairs. This paper
attempts to bridge the gap by ranking sentence singletons and pairs together in
a unified space. Our proposed framework attempts to model human methodology by
selecting either a single sentence or a pair of sentences, then compressing or
fusing the sentence(s) to produce a summary sentence. We conduct extensive
experiments on both single- and multi-document summarization datasets and
report findings on sentence selection and abstraction.

negative:
Sentence Split and Rephrase aims to break down a complex sentence into
several simple sentences with its meaning preserved. Previous studies tend to
address the issue by seq2seq learning from parallel sentence pairs, which takes
a complex sentence as input and sequentially generates a series of simple
sentences. However, the conventional seq2seq learning has two limitations for
this task: (1) it does not take into account the facts stated in the long
sentence; As a result, the generated simple sentences may miss or inaccurately
state the facts in the original sentence. (2) The order variance of the simple
sentences to be generated may confuse the seq2seq model during training because
the simple sentences derived from the long source sentence could be in any
order.
  To overcome the challenges, we first propose the Fact-aware Sentence
Encoding, which enables the model to learn facts from the long sentence and
thus improves the precision of sentence split; then we introduce Permutation
Invariant Training to alleviate the effects of order variance in seq2seq
learning for this task. Experiments on the WebSplit-v1.0 benchmark dataset show
that our approaches can largely improve the performance over the previous
seq2seq learning approaches. Moreover, an extrinsic evaluation on oie-benchmark
verifies the effectiveness of our approaches by an observation that splitting
long sentences with our state-of-the-art model as preprocessing is helpful for
improving OpenIE performance.

negative:
Prior work on training generative Visual Dialog models with reinforcement
learning(Das et al.) has explored a Qbot-Abot image-guessing game and shown
that this 'self-talk' approach can lead to improved performance at the
downstream dialog-conditioned image-guessing task. However, this improvement
saturates and starts degrading after a few rounds of interaction, and does not
lead to a better Visual Dialog model. We find that this is due in part to
repeated interactions between Qbot and Abot during self-talk, which are not
informative with respect to the image. To improve this, we devise a simple
auxiliary objective that incentivizes Qbot to ask diverse questions, thus
reducing repetitions and in turn enabling Abot to explore a larger state space
during RL ie. be exposed to more visual concepts to talk about, and varied
questions to answer. We evaluate our approach via a host of automatic metrics
and human studies, and demonstrate that it leads to better dialog, ie. dialog
that is more diverse (ie. less repetitive), consistent (ie. has fewer
conflicting exchanges), fluent (ie. more human-like),and detailed, while still
being comparably image-relevant as prior work and ablations.

negative:
Recently, the progress of learning-by-synthesis has proposed a training model
for synthetic images, which can effectively reduce the cost of human and
material resources. However, due to the different distribution of synthetic
images compared to real images, the desired performance cannot still be
achieved. Real images consist of multiple forms of light orientation, while
synthetic images consist of a uniform light orientation. These features are
considered to be characteristic of outdoor and indoor scenes, respectively. To
solve this problem, the previous method learned a model to improve the realism
of the synthetic image. Different from the previous methods, this paper takes
the first step to purify real images. Through the style transfer task, the
distribution of outdoor real images is converted into indoor synthetic images,
thereby reducing the influence of light. Therefore, this paper proposes a
real-time style transfer network that preserves image content information (eg,
gaze direction, pupil center position) of an input image (real image) while
inferring style information (eg, image color structure, semantic features) of
style image (synthetic image). In addition, the network accelerates the
convergence speed of the model and adapts to multi-scale images. Experiments
were performed using mixed studies (qualitative and quantitative) methods to
demonstrate the possibility of purifying real images in complex directions.
Qualitatively, it compares the proposed method with the available methods in a
series of indoor and outdoor scenarios of the LPW dataset. In quantitative
terms, it evaluates the purified image by training a gaze estimation model on
the cross data set. The results show a significant improvement over the
baseline method compared to the raw real image.

negative:
Unsupervised image captioning is a challenging task that aims at generating
captions without the supervision of image-sentence pairs, but only with images
and sentences drawn from different sources and object labels detected from the
images. In previous work, pseudo-captions, i.e., sentences that contain the
detected object labels, were assigned to a given image. The focus of the
previous work was on the alignment of input images and pseudo-captions at the
sentence level. However, pseudo-captions contain many words that are irrelevant
to a given image. In this work, we investigate the effect of removing
mismatched words from image-sentence alignment to determine how they make this
task difficult. We propose a simple gating mechanism that is trained to align
image features with only the most reliable words in pseudo-captions: the
detected object labels. The experimental results show that our proposed method
outperforms the previous methods without introducing complex sentence-level
learning objectives. Combined with the sentence-level alignment method of
previous work, our method further improves its performance. These results
confirm the importance of careful alignment in word-level details.

negative:
My overarching research goal is to provide robots with perceptional abilities
that allow interactions with humans in a human-like manner. To develop these
perceptional abilities, I believe that it is useful to study the principles of
the human visual system. I use these principles to develop new computer vision
algorithms and validate their effectiveness in intelligent robotic systems. I
am enthusiastic about this approach as it offers the dual benefit of uncovering
principles inherent in the human visual system, as well as applying these
principles to its artificial counterpart. Fig. 1 contains a depiction of my
research.

negative:
Images of natural systems may represent patterns of network-like structure,
which could reveal important information about the topological properties of
the underlying subject. However, the image itself does not automatically
provide a formal definition of a network in terms of sets of nodes and edges.
Instead, this information should be suitably extracted from the raw image data.
Motivated by this, we present a principled model to extract network topologies
from images that is scalable and efficient. We map this goal into solving a
routing optimization problem where the solution is a network that minimizes an
energy function which can be interpreted in terms of an operational and
infrastructural cost. Our method relies on recent results from optimal
transport theory and is a principled alternative to standard image-processing
techniques that are based on heuristics. We test our model on real images of
the retinal vascular system, slime mold and river networks and compare with
routines combining image-processing techniques. Results are tested in terms of
a similarity measure related to the amount of information preserved in the
extraction. We find that our model finds networks from retina vascular network
images that are more similar to hand-labeled ones, while also giving high
performance in extracting networks from images of rivers and slime mold for
which there is no ground truth available. While there is no unique method that
fits all the images the best, our approach performs consistently across
datasets, its algorithmic implementation is efficient and can be fully
automatized to be run on several datasets with little supervision.

negative:
Neural machine translation (NMT) systems are usually trained on a large
amount of bilingual sentence pairs and translate one sentence at a time,
ignoring inter-sentence information. This may make the translation of a
sentence ambiguous or even inconsistent with the translations of neighboring
sentences. In order to handle this issue, we propose an inter-sentence gate
model that uses the same encoder to encode two adjacent sentences and controls
the amount of information flowing from the preceding sentence to the
translation of the current sentence with an inter-sentence gate. In this way,
our proposed model can capture the connection between sentences and fuse
recency from neighboring sentences into neural machine translation. On several
NIST Chinese-English translation tasks, our experiments demonstrate that the
proposed inter-sentence gate model achieves substantial improvements over the
baseline.

negative:
A desideratum of high-quality translation systems is that they preserve
meaning, in the sense that two sentences with different meanings should not
translate to one and the same sentence in another language. However,
state-of-the-art systems often fail in this regard, particularly in cases where
the source and target languages partition the "meaning space" in different
ways. For instance, "I cut my finger." and "I cut my finger off." describe
different states of the world but are translated to French (by both Fairseq and
Google Translate) as "Je me suis coupe le doigt.", which is ambiguous as to
whether the finger is detached. More generally, translation systems are
typically many-to-one (non-injective) functions from source to target language,
which in many cases results in important distinctions in meaning being lost in
translation. Building on Bayesian models of informative utterance production,
we present a method to define a less ambiguous translation system in terms of
an underlying pre-trained neural sequence-to-sequence model. This method
increases injectivity, resulting in greater preservation of meaning as measured
by improvement in cycle-consistency, without impeding translation quality
(measured by BLEU score).

negative:
Language recognition has been significantly advanced in recent years by means
of modern machine learning methods such as deep learning and benchmarks with
rich annotations. However, research is still limited in low-resource formal
languages. This consists of a significant gap in describing the colloquial
language especially for low-resourced ones such as Persian. In order to target
this gap for low resource languages, we propose a "Large Scale Colloquial
Persian Dataset" (LSCP). LSCP is hierarchically organized in a semantic
taxonomy that focuses on multi-task informal Persian language understanding as
a comprehensive problem. This encompasses the recognition of multiple semantic
aspects in the human-level sentences, which naturally captures from the
real-world sentences. We believe that further investigations and processing, as
well as the application of novel algorithms and methods, can strengthen
enriching computerized understanding and processing of low resource languages.
The proposed corpus consists of 120M sentences resulted from 27M tweets
annotated with parsing tree, part-of-speech tags, sentiment polarity and
translation in five different languages.

negative:
Classifying the human emotion through facial expressions is a big topic in
both the Computer Vision and Deep learning fields. Human emotion can be
classified as one of the basic emotion types like being angry, happy or
dimensional emotion with valence and arousal values. There are a lot of related
challenges in this topic, one of the most famous challenges is called the
'Affect-in-the-wild Challenge'(Aff-Wild Challenge). It is the first challenge
on the estimation of valence and arousal in-the-wild. This project is an
extension of the Aff-wild Challenge. Aff-wild database was created using images
with a mean resolution of 607*359, I and Dimitrios sought to find out the
performance of the model that is trained on a database that contains4K
resolution in-the-wild images. Since there is no existing database to satisfy
the requirement, I built this database from scratch with help from Dimitrios
and trained neural network models with different hyperparameters on this
database. I used network models likeVGG16, AlexNet, ResNet and also some
pre-trained models like Ima-geNet VGG. I compared the results of the different
network models alongside the results from the Aff-wild database to exploit the
optimal model for my database.

negative:
A relation tuple consists of two entities and the relation between them, and
often such tuples are found in unstructured text. There may be multiple
relation tuples present in a text and they may share one or both entities among
them. Extracting such relation tuples from a sentence is a difficult task and
sharing of entities or overlapping entities among the tuples makes it more
challenging. Most prior work adopted a pipeline approach where entities were
identified first followed by finding the relations among them, thus missing the
interaction among the relation tuples in a sentence. In this paper, we propose
two approaches to use encoder-decoder architecture for jointly extracting
entities and relations. In the first approach, we propose a representation
scheme for relation tuples which enables the decoder to generate one word at a
time like machine translation models and still finds all the tuples present in
a sentence with full entity names of different length and with overlapping
entities. Next, we propose a pointer network-based decoding approach where an
entire tuple is generated at every time step. Experiments on the publicly
available New York Times corpus show that our proposed approaches outperform
previous work and achieve significantly higher F1 scores.

negative:
The number of user reviews of tourist attractions, restaurants, mobile apps,
etc. is increasing for all languages; yet, research is lacking on how reviews
in multiple languages should be aggregated and displayed. Speakers of different
languages may have consistently different experiences, e.g., different
information available in different languages at tourist attractions or
different user experiences with software due to
internationalization/localization choices. This paper assesses the similarity
in the ratings given by speakers of different languages to London tourist
attractions on TripAdvisor. The correlations between different languages are
generally high, but some language pairs are more correlated than others. The
results question the common practice of computing average ratings from reviews
in many languages.

negative:
Emotion stimulus extraction is a fine-grained subtask of emotion analysis
that focuses on identifying the description of the cause behind an emotion
expression from a text passage (e.g., in the sentence "I am happy that I passed
my exam" the phrase "passed my exam" corresponds to the stimulus.). Previous
work mainly focused on Mandarin and English, with no resources or models for
German. We fill this research gap by developing a corpus of 2006 German news
headlines annotated with emotions and 811 instances with annotations of
stimulus phrases. Given that such corpus creation efforts are time-consuming
and expensive, we additionally work on an approach for projecting the existing
English GoodNewsEveryone (GNE) corpus to a machine-translated German version.
We compare the performance of a conditional random field (CRF) model (trained
monolingually on German and cross-lingually via projection) with a multilingual
XLM-RoBERTa (XLM-R) model. Our results show that training with the German
corpus achieves higher F1 scores than projection. Experiments with XLM-R
outperform their respective CRF counterparts.

negative:
Multilingual neural machine translation (NMT), which translates multiple
languages using a single model, is of great practical importance due to its
advantages in simplifying the training process, reducing online maintenance
costs, and enhancing low-resource and zero-shot translation. Given there are
thousands of languages in the world and some of them are very different, it is
extremely burdensome to handle them all in a single model or use a separate
model for each language pair. Therefore, given a fixed resource budget, e.g.,
the number of models, how to determine which languages should be supported by
one model is critical to multilingual NMT, which, unfortunately, has been
ignored by previous work. In this work, we develop a framework that clusters
languages into different groups and trains one multilingual model for each
cluster. We study two methods for language clustering: (1) using prior
knowledge, where we cluster languages according to language family, and (2)
using language embedding, in which we represent each language by an embedding
vector and cluster them in the embedding space. In particular, we obtain the
embedding vectors of all the languages by training a universal neural machine
translation model. Our experiments on 23 languages show that the first
clustering method is simple and easy to understand but leading to suboptimal
translation accuracy, while the second method sufficiently captures the
relationship among languages well and improves the translation accuracy for
almost all the languages over baseline methods

negative:
Recent advances in natural language processing (NLP) have the ability to
transform how classroom learning takes place. Combined with the increasing
integration of technology in today's classrooms, NLP systems leveraging
question answering and dialog processing techniques can serve as private tutors
or participants in classroom discussions to increase student engagement and
learning. To progress towards this goal, we use the classroom discourse
framework of academically productive talk (APT) to learn strategies that make
for the best learning experience. In this paper, we introduce a new task,
called future talk move prediction (FTMP): it consists of predicting the next
talk move -- an utterance strategy from APT -- given a conversation history
with its corresponding talk moves. We further introduce a neural network model
for this task, which outperforms multiple baselines by a large margin. Finally,
we compare our model's performance on FTMP to human performance and show
several similarities between the two.

negative:
In recent years online shopping has gained momentum and became an important
venue for customers wishing to save time and simplify their shopping process. A
key advantage of shopping online is the ability to read what other customers
are saying about products of interest. In this work, we aim to maintain this
advantage in situations where extreme brevity is needed, for example, when
shopping by voice. We suggest a novel task of extracting a single
representative helpful sentence from a set of reviews for a given product. The
selected sentence should meet two conditions: first, it should be helpful for a
purchase decision and second, the opinion it expresses should be supported by
multiple reviewers. This task is closely related to the task of Multi Document
Summarization in the product reviews domain but differs in its objective and
its level of conciseness. We collect a dataset in English of sentence
helpfulness scores via crowd-sourcing and demonstrate its reliability despite
the inherent subjectivity involved. Next, we describe a complete model that
extracts representative helpful sentences with positive and negative sentiment
towards the product and demonstrate that it outperforms several baselines.

negative:
Recently, there is a rising interest in perceiving image aesthetics. The
existing works deal with image aesthetics as a classification or regression
problem. To extend the cognition from rating to reasoning, a deeper
understanding of aesthetics should be based on revealing why a high- or
low-aesthetic score should be assigned to an image. From such a point of view,
we propose a model referred to as Neural Aesthetic Image Reviewer, which can
not only give an aesthetic score for an image, but also generate a textual
description explaining why the image leads to a plausible rating score.
Specifically, we propose two multi-task architectures based on shared
aesthetically semantic layers and task-specific embedding layers at a high
level for performance improvement on different tasks. To facilitate researches
on this problem, we collect the AVA-Reviews dataset, which contains 52,118
images and 312,708 comments in total. Through multi-task learning, the proposed
models can rate aesthetic images as well as produce comments in an end-to-end
manner. It is confirmed that the proposed models outperform the baselines
according to the performance evaluation on the AVA-Reviews dataset. Moreover,
we demonstrate experimentally that our model can generate textual reviews
related to aesthetics, which are consistent with human perception.

negative:
Our world is open-ended, non-stationary, and constantly evolving; thus what
we talk about and how we talk about it change over time. This inherent dynamic
nature of language contrasts with the current static language modelling
paradigm, which trains and evaluates models on utterances from overlapping time
periods. Despite impressive recent progress, we demonstrate that Transformer-XL
language models perform worse in the realistic setup of predicting future
utterances from beyond their training period, and that model performance
becomes increasingly worse with time. We find that, while increasing model size
alone -- a key driver behind recent progress -- does not solve this problem,
having models that continually update their knowledge with new information can
indeed mitigate this performance degradation over time. Hence, given the
compilation of ever-larger language modelling datasets, combined with the
growing list of language-model-based NLP applications that require up-to-date
factual knowledge about the world, we argue that now is the right time to
rethink the static way in which we currently train and evaluate our language
models, and develop adaptive language models that can remain up-to-date with
respect to our ever-changing and non-stationary world. We publicly release our
dynamic, streaming language modelling benchmarks for WMT and arXiv to
facilitate language model evaluation that takes temporal dynamics into account.

negative:
Training a deep network to perform semantic segmentation requires large
amounts of labeled data. To alleviate the manual effort of annotating real
images, researchers have investigated the use of synthetic data, which can be
labeled automatically. Unfortunately, a network trained on synthetic data
performs relatively poorly on real images. While this can be addressed by
domain adaptation, existing methods all require having access to real images
during training. In this paper, we introduce a drastically different way to
handle synthetic images that does not require seeing any real images at
training time. Our approach builds on the observation that foreground and
background classes are not affected in the same manner by the domain shift, and
thus should be treated differently. In particular, the former should be handled
in a detection-based manner to better account for the fact that, while their
texture in synthetic images is not photo-realistic, their shape looks natural.
Our experiments evidence the effectiveness of our approach on Cityscapes and
CamVid with models trained on synthetic data only.

negative:
Classroom discussions in English Language Arts have a positive effect on
students' reading, writing and reasoning skills. Although prior work has
largely focused on teacher talk and student-teacher interactions, we focus on
three theoretically-motivated aspects of high-quality student talk:
argumentation, specificity, and knowledge domain. We introduce an annotation
scheme, then show that the scheme can be used to produce reliable annotations
and that the annotations are predictive of discussion quality. We also
highlight opportunities provided by our scheme for education and natural
language processing research.

negative:
In this paper, we pose the question: do people talk about women and men in
different ways? We introduce two datasets and a novel integration of approaches
for automatically inferring gender associations from language, discovering
coherent word clusters, and labeling the clusters for the semantic concepts
they represent. The datasets allow us to compare how people write about women
and men in two different settings - one set draws from celebrity news and the
other from student reviews of computer science professors. We demonstrate that
there are large-scale differences in the ways that people talk about women and
men and that these differences vary across domains. Human evaluations show that
our methods significantly outperform strong baselines.

negative:
Convolututional Neural Networks have achieved state of the art in image
classification, object detection and other image related tasks. In this paper I
present another use of CNNs i.e. if given a set of images and then giving a
single test image the network identifies that the test image is part of which
image from the images given before. This is a task somehow similar to measuring
image similarity and can be done using a simple CNN. Doing this task manually
by looping can be quite a time consuming problem and won't be a generalizable
solution. The task is quite similar to doing object detection but for that lots
training data should be given or in the case of sliding window it takes lot of
time and my algorithm can work with much fewer examples, is totally
unsupervised and works much efficiently. Also, I explain that how unsupervised
algorithm like K-Means or supervised algorithm like K-NN are not good enough to
perform this task. The basic idea is that image encodings are collected for
each image from a CNN, when a test image comes it is replaced by a part of
original image, the encoding is generated using the same network, the frobenius
norm is calculated and if it comes under a tolerance level then the test image
is said to be the part of the original image.

negative:
Human-performed annotation of sentences in legal documents is an important
prerequisite to many machine learning based systems supporting legal tasks.
Typically, the annotation is done sequentially, sentence by sentence, which is
often time consuming and, hence, expensive. In this paper, we introduce a
proof-of-concept system for annotating sentences "laterally." The approach is
based on the observation that sentences that are similar in meaning often have
the same label in terms of a particular type system. We use this observation in
allowing annotators to quickly view and annotate sentences that are
semantically similar to a given sentence, across an entire corpus of documents.
Here, we present the interface of the system and empirically evaluate the
approach. The experiments show that lateral annotation has the potential to
make the annotation process quicker and more consistent.

negative:
This paper attempts to connects the evolution of computer languages with the
evolution of life, where the later has been dictated by \emph{theory of
evolution of species}, and tries to give supportive evidence that the new
languages are more robust than the previous, carry-over the mixed features of
older languages, such that strong features gets added into them and weak
features of older languages gets removed. In addition, an analysis of most
prominent programming languages is presented, emphasizing on how the features
of existing languages have influenced the development of new programming
languages. At the end, it suggests a set of experimental languages, which may
rule the world of programming languages in the time of new multi-core
architectures.
  Index terms- Programming languages' evolution, classifications of languages,
future languages, scripting-languages.

negative:
To date, most state-of-the-art sequence modeling architectures use attention
to build generative models for language based tasks. Some of these models use
all the available sequence tokens to generate an attention distribution which
results in time complexity of $O(n^2)$. Alternatively, they utilize depthwise
convolutions with softmax normalized kernels of size $k$ acting as a
limited-window self-attention, resulting in time complexity of $O(k{\cdot}n)$.
In this paper, we introduce Time-aware Large Kernel (TaLK) Convolutions, a
novel adaptive convolution operation that learns to predict the size of a
summation kernel instead of using a fixed-sized kernel matrix. This method
yields a time complexity of $O(n)$, effectively making the sequence encoding
process linear to the number of tokens. We evaluate the proposed method on
large-scale standard machine translation, abstractive summarization and
language modeling datasets and show that TaLK Convolutions constitute an
efficient improvement over other attention/convolution based approaches.

negative:
We propose a novel approach to synthesizing images that are effective for
training object detectors. Starting from a small set of real images, our
algorithm estimates the rendering parameters required to synthesize similar
images given a coarse 3D model of the target object. These parameters can then
be reused to generate an unlimited number of training images of the object of
interest in arbitrary 3D poses, which can then be used to increase
classification performances.
  A key insight of our approach is that the synthetically generated images
should be similar to real images, not in terms of image quality, but rather in
terms of features used during the detector training. We show in the context of
drone, plane, and car detection that using such synthetically generated images
yields significantly better performances than simply perturbing real images or
even synthesizing images in such way that they look very realistic, as is often
done when only limited amounts of training data are available.

negative:
Automatic image captioning has recently approached human-level performance
due to the latest advances in computer vision and natural language
understanding. However, most of the current models can only generate plain
factual descriptions about the content of a given image. However, for human
beings, image caption writing is quite flexible and diverse, where additional
language dimensions, such as emotion, humor and language styles, are often
incorporated to produce diverse, emotional, or appealing captions. In
particular, we are interested in generating sentiment-conveying image
descriptions, which has received little attention. The main challenge is how to
effectively inject sentiments into the generated captions without altering the
semantic matching between the visual content and the generated descriptions. In
this work, we propose two different models, which employ different schemes for
injecting sentiments into image captions. Compared with the few existing
approaches, the proposed models are much simpler and yet more effective. The
experimental results show that our model outperform the state-of-the-art models
in generating sentimental (i.e., sentiment-bearing) image captions. In
addition, we can also easily manipulate the model by assigning different
sentiments to the testing image to generate captions with the corresponding
sentiments.

negative:
Recently we see a rising number of methods in the field of eXplainable
Artificial Intelligence. To our surprise, their development is driven by model
developers rather than a study of needs for human end users. The analysis of
needs, if done, takes the form of an A/B test rather than a study of open
questions. To answer the question "What would a human operator like to ask the
ML model?" we propose a conversational system explaining decisions of the
predictive model. In this experiment, we developed a chatbot called dr_ant to
talk about machine learning model trained to predict survival odds on Titanic.
People can talk with dr_ant about different aspects of the model to understand
the rationale behind its predictions. Having collected a corpus of 1000+
dialogues, we analyse the most common types of questions that users would like
to ask. To our knowledge, it is the first study which uses a conversational
system to collect the needs of human operators from the interactive and
iterative dialogue explorations of a predictive model.

negative:
In this dissertation, I present my work towards exploring temporal
information for better video understanding. Specifically, I have worked on two
problems: action recognition and semantic segmentation. For action recognition,
I have proposed a framework, termed hidden two-stream networks, to learn an
optimal motion representation that does not require the computation of optical
flow. My framework alleviates several challenges faced in video classification,
such as learning motion representations, real-time inference, multi-framerate
handling, generalizability to unseen actions, etc. For semantic segmentation, I
have introduced a general framework that uses video prediction models to
synthesize new training samples. By scaling up the training dataset, my trained
models are more accurate and robust than previous models even without
modifications to the network architectures or objective functions. I believe
videos have much more potential to be mined, and temporal information is one of
the most important cues for machines to perceive the visual world better.

negative:
There is a growing interest in investigating what neural NLP models learn
about language. A prominent open question is the question of whether or not it
is necessary to model hierarchical structure. We present a linguistic
investigation of a neural parser adding insights to this question. We look at
transitivity and agreement information of auxiliary verb constructions (AVCs)
in comparison to finite main verbs (FMVs). This comparison is motivated by
theoretical work in dependency grammar and in particular the work of Tesni\`ere
(1959) where AVCs and FMVs are both instances of a nucleus, the basic unit of
syntax. An AVC is a dissociated nucleus, it consists of at least two words, and
an FMV is its non-dissociated counterpart, consisting of exactly one word. We
suggest that the representation of AVCs and FMVs should capture similar
information. We use diagnostic classifiers to probe agreement and transitivity
information in vectors learned by a transition-based neural parser in four
typologically different languages. We find that the parser learns different
information about AVCs and FMVs if only sequential models (BiLSTMs) are used in
the architecture but similar information when a recursive layer is used. We
find explanations for why this is the case by looking closely at how
information is learned in the network and looking at what happens with
different dependency representations of AVCs. We conclude that there may be
benefits to using a recursive layer in dependency parsing and that we have not
yet found the best way to integrate it in our parsers.

negative:
Current open-domain conversational models can easily be made to talk in
inadequate ways. Online learning from conversational feedback given by the
conversation partner is a promising avenue for a model to improve and adapt, so
as to generate fewer of these safety failures. However, current
state-of-the-art models tend to react to feedback with defensive or oblivious
responses. This makes for an unpleasant experience and may discourage
conversation partners from giving feedback in the future. This work proposes
SaFeRDialogues, a task and dataset of graceful responses to conversational
feedback about safety failures. We collect a dataset of 10k dialogues
demonstrating safety failures, feedback signaling them, and a response
acknowledging the feedback. We show how fine-tuning on this dataset results in
conversations that human raters deem considerably more likely to lead to a
civil conversation, without sacrificing engagingness or general conversational
ability.

negative:
We argue that logical semantics might have faltered due to its failure in
distinguishing between two fundamentally very different types of concepts:
ontological concepts, that should be types in a strongly-typed ontology, and
logical concepts, that are predicates corresponding to properties of and
relations between objects of various ontological types. We will then show that
accounting for these differences amounts to the integration of lexical and
compositional semantics in one coherent framework, and to an embedding in our
logical semantics of a strongly-typed ontology that reflects our commonsense
view of the world and the way we talk about it in ordinary language. We will
show that in such a framework a number of challenges in natural language
semantics can be adequately and systematically treated.

negative:
In this paper, we are concerned with the detection of progressive dynamic
saliency from video sequences. More precisely, we are interested in saliency
related to motion and likely to appear progressively over time. It can be
relevant to trigger alarms, to dedicate additional processing or to detect
specific events. Trajectories represent the best way to support progressive
dynamic saliency detection. Accordingly, we will talk about trajectory
saliency. A trajectory will be qualified as salient if it deviates from normal
trajectories that share a common motion pattern related to a given context.
First, we need a compact while discriminative representation of trajectories.
We adopt a (nearly) unsupervised learning-based approach. The latent code
estimated by a recurrent auto-encoder provides the desired representation. In
addition, we enforce consistency for normal (similar) trajectories through the
auto-encoder loss function. The distance of the trajectory code to a prototype
code accounting for normality is the means to detect salient trajectories. We
validate our trajectory saliency detection method on synthetic and real
trajectory datasets, and highlight the contributions of its different
components. We show that our method outperforms existing methods on several
scenarios drawn from the publicly available dataset of pedestrian trajectories
acquired in a railway station (Alahi 2014).

negative:
Our research topic is Human segmentation with static camera. This topic can
be divided into three sub-tasks, which are object detection, instance
identification and segmentation. These sub-tasks are three closely related
subjects. The development of each subject has great impact on the other two
fields. In this literature review, we will first introduce the background of
human segmentation and then talk about issues related to the above three fields
as well as how they interact with each other.

negative:
Virtual Personal Assistants like Siri have great potential but such
developments hit the fundamental problem of how to make computational devices
that understand human speech. Natural language understanding is one of the more
disappointing failures of AI research and it seems there is something we
computer scientists don't get about the nature of language. Of course
philosophers and linguists think quite differently about language and this
paper describes how we have taken ideas from other disciplines and implemented
them. The background to the work is to take seriously the notion of language as
action and look at what people actually do with language using the techniques
of Conversation Analysis. The observation has been that human communication is
(behind the scenes) about the management of social relations as well as the
(foregrounded) passing of information. To claim this is one thing but to
implement it requires a mechanism. The mechanism described here is based on the
notion of language being intentional - we think intentionally, talk about them
and recognise them in others - and cooperative in that we are compelled to help
out. The way we are compelled points to a solution to the ever present problem
of keeping the human on topic. The approach has led to a recent success in
which we significantly improve user satisfaction independent of task
completion. Talk Markup Language (TalkML) is a draft alternative to VoiceXML
that, we propose, greatly simplifies the scripting of interaction by providing
default behaviours for no input and not recognised speech events.

negative:
How will my face look when I get older? Or, for a more challenging question:
How will my brain look when I get older? To answer this question one must
devise (and learn from data) a multivariate auto-regressive function which
given an image and a desired target age generates an output image. While
collecting data for faces may be easier, collecting longitudinal brain data is
not trivial. We propose a deep learning-based method that learns to simulate
subject-specific brain ageing trajectories without relying on longitudinal
data. Our method synthesises images conditioned on two factors: age (a
continuous variable), and status of Alzheimer's Disease (AD, an ordinal
variable). With an adversarial formulation we learn the joint distribution of
brain appearance, age and AD status, and define reconstruction losses to
address the challenging problem of preserving subject identity. We compare with
several benchmarks using two widely used datasets. We evaluate the quality and
realism of synthesised images using ground-truth longitudinal data and a
pre-trained age predictor. We show that, despite the use of cross-sectional
data, our model learns patterns of gray matter atrophy in the middle temporal
gyrus in patients with AD. To demonstrate generalisation ability, we train on
one dataset and evaluate predictions on the other. In conclusion, our model
shows an ability to separate age, disease influence and anatomy using only 2D
cross-sectional data that should be useful in large studies into
neurodegenerative disease, that aim to combine several data sources. To
facilitate such future studies by the community at large our code is made
available at https://github.com/xiat0616/BrainAgeing.

negative:
Synthesizing images of a person in novel poses from a single image is a
highly ambiguous task. Most existing approaches require paired training images;
i.e. images of the same person with the same clothing in different poses.
However, obtaining sufficiently large datasets with paired data is challenging
and costly. Previous methods that forego paired supervision lack realism. We
propose a self-supervised framework named SPICE (Self-supervised Person Image
CrEation) that closes the image quality gap with supervised methods. The key
insight enabling self-supervision is to exploit 3D information about the human
body in several ways. First, the 3D body shape must remain unchanged when
reposing. Second, representing body pose in 3D enables reasoning about self
occlusions. Third, 3D body parts that are visible before and after reposing,
should have similar appearance features. Once trained, SPICE takes an image of
a person and generates a new image of that person in a new target pose. SPICE
achieves state-of-the-art performance on the DeepFashion dataset, improving the
FID score from 29.9 to 7.8 compared with previous unsupervised methods, and
with performance similar to the state-of-the-art supervised method (6.4). SPICE
also generates temporally coherent videos given an input image and a sequence
of poses, despite being trained on static images only.

negative:
Basic body-part names (BBPNs) were defined as body-part names in Swadesh
basic 200 words. Non-Mayan cognates of Mayan (MY) BBPNs were extensively
searched for, by comparing with non-MY vocabulary, including ca.1300 basic
words of 82 AN languages listed by Tryon (1985), etc. Thus found cognates (CGs)
in non-MY are listed in Table 1, as classified by language groups to which most
similar cognates (MSCs) of MY BBPNs belong. CGs of MY are classified to 23
mutually unrelated CG-items, of which 17.5 CG-items have their MSCs in
Austronesian (AN), giving its closest similarity score (CSS), CSS(AN) = 17.5,
which consists of 10.33 MSCs in Formosan, 1.83 MSCs in Western
Malayo-Polynesian (W.MP), 0.33 in Central MP, 0.0 in SHWNG, and 5.0 in Oceanic
[i.e., CSS(FORM)= 10.33, CSS(W.MP) = 1.88, ..., CSS(OC)= 5.0]. These CSSs for
language (sub)groups are also listed in the underline portion of every section
of (Section1 - Section 6) in Table 1. Chi-squar test (degree of freedom = 1)
using [Eq 1] and [Eqs.2] revealed that MSCs of MY BBPNs are distributed in
Formosan in significantly higher frequency (P < 0.001) than in other subgroups
of AN, as well as than in non-AN languages. MY is thus concluded to have been
derived from Formosan of AN. Eskimo shows some BBPN similarities to FORM and
MY.