query: I am currently engaged in a project that involves the generation of realistic text using large-scale, pre-trained language models. However, I am encountering difficulties in controlling the results of the generation process. Existing methods, such as prompting, do not offer the level of control I require.  I am in search of an innovative approach that can enhance the controllability of text generation while simultaneously preserving the quality of the text produced.  More specifically, I am considering the projection of certain constraints and control conditions into the latent space. By injecting these embeddings into the language model's inputs, I anticipate that it could regulate the output of the language model.  I am particularly interested in applying this method to tasks such as constrained text generation, where explicit constraints are present. My goal is for my proposed method to surpass the performance of other fine-tuned or prompting-based methods.

positive:
Large pre-trained language models have repeatedly shown their ability to
produce fluent text. Yet even when starting from a prompt, generation can
continue in many plausible directions. Current decoding methods with the goal
of controlling generation, e.g., to ensure specific words are included, either
require additional models or fine-tuning, or work poorly when the task at hand
is semantically unconstrained, e.g., story generation. In this work, we present
a plug-and-play decoding method for controlled language generation that is so
simple and intuitive, it can be described in a single sentence: given a topic
or keyword, we add a shift to the probability distribution over our vocabulary
towards semantically similar words. We show how annealing this distribution can
be used to impose hard constraints on language generation, something no other
plug-and-play method is currently able to do with SOTA language generators.
Despite the simplicity of this approach, we see it works incredibly well in
practice: decoding from GPT-2 leads to diverse and fluent sentences while
guaranteeing the appearance of given guide words. We perform two user studies,
revealing that (1) our method outperforms competing methods in human
evaluations; and (2) forcing the guide words to appear in the generated text
has no impact on the fluency of the generated text.

positive:
Large transformer-based language models (LMs) trained on huge text corpora
have shown unparalleled generation capabilities. However, controlling
attributes of the generated language (e.g. switching topic or sentiment) is
difficult without modifying the model architecture or fine-tuning on
attribute-specific data and entailing the significant cost of retraining. We
propose a simple alternative: the Plug and Play Language Model (PPLM) for
controllable language generation, which combines a pretrained LM with one or
more simple attribute classifiers that guide text generation without any
further training of the LM. In the canonical scenario we present, the attribute
models are simple classifiers consisting of a user-specified bag of words or a
single learned layer with 100,000 times fewer parameters than the LM. Sampling
entails a forward and backward pass in which gradients from the attribute model
push the LM's hidden activations and thus guide the generation. Model samples
demonstrate control over a range of topics and sentiment styles, and extensive
automated and human annotated evaluations show attribute alignment and fluency.
PPLMs are flexible in that any combination of differentiable attribute models
may be used to steer text generation, which will allow for diverse and creative
applications beyond the examples given in this paper.

positive:
Conditional text generation has been a challenging task that is yet to see
human-level performance from state-of-the-art models. In this work, we
specifically focus on the Commongen benchmark, wherein the aim is to generate a
plausible sentence for a given set of input concepts. Despite advances in other
tasks, large pre-trained language models that are fine-tuned on this dataset
often produce sentences that are syntactically correct but qualitatively
deviate from a human understanding of common sense. Furthermore, generated
sequences are unable to fulfill such lexical requirements as matching
part-of-speech and full concept coverage. In this paper, we explore how
commonsense knowledge graphs can enhance model performance, with respect to
commonsense reasoning and lexically-constrained decoding. We propose strategies
for enhancing the semantic correctness of the generated text, which we
accomplish through: extracting commonsense relations from Conceptnet, injecting
these relations into the Unified Language Model (UniLM) through attention
mechanisms, and enforcing the aforementioned lexical requirements through
output constraints. By performing several ablations, we find that commonsense
injection enables the generation of sentences that are more aligned with human
understanding, while remaining compliant with lexical requirements.

positive:
Large pre-trained language models are capable of generating realistic text.
However, controlling these models so that the generated text satisfies lexical
constraints, i.e., contains specific words, is a challenging problem. Given
that state-of-the-art language models are too large to be trained from scratch
in a manageable time, it is desirable to control these models without
re-training them. Methods capable of doing this are called plug-and-play.
Recent plug-and-play methods have been successful in constraining small
bidirectional language models as well as forward models in tasks with a
restricted search space, e.g., machine translation. However, controlling large
transformer-based models to meet lexical constraints without re-training them
remains a challenge. In this work, we propose Directed Beam Search (DBS), a
plug-and-play method for lexically constrained language generation. Our method
can be applied to any language model, is easy to implement and can be used for
general language generation. In our experiments we use DBS to control GPT-2. We
demonstrate its performance on keyword-to-phrase generation and we obtain
comparable results as a state-of-the-art non-plug-and-play model for lexically
constrained story generation.

positive:
Conditional text generation often requires lexical constraints, i.e., which
words should or shouldn't be included in the output text. While the dominant
recipe for conditional text generation has been large-scale pretrained language
models that are finetuned on the task-specific training data, such models do
not learn to follow the underlying constraints reliably, even when supervised
with large amounts of task-specific examples.
  We propose NeuroLogic Decoding, a simple yet effective algorithm that enables
neural language models -- supervised or not -- to generate fluent text while
satisfying complex lexical constraints. Our approach is powerful yet efficient.
It handles any set of lexical constraints that is expressible under predicate
logic, while its asymptotic runtime is equivalent to conventional beam search.
  Empirical results on four benchmarks show that NeuroLogic Decoding
outperforms previous approaches, including algorithms that handle a subset of
our constraints. Moreover, we find that unsupervised models with NeuroLogic
Decoding often outperform supervised models with conventional decoding, even
when the latter is based on considerably larger networks. Our results suggest
the limit of large-scale neural networks for fine-grained controllable
generation and the promise of inference-time algorithms.

positive:
Large language models benefit from training with a large amount of unlabeled
text, which gives them increasingly fluent and diverse generation capabilities.
However, using these models for text generation that takes into account target
attributes, such as sentiment polarity or specific topics, remains a challenge.
We propose a simple and flexible method for controlling text generation by
aligning disentangled attribute representations. In contrast to recent efforts
on training a discriminator to perturb the token level distribution for an
attribute, we use the same data to learn an alignment function to guide the
pre-trained, non-controlled language model to generate texts with the target
attribute without changing the original language model parameters. We evaluate
our method on sentiment- and topic-controlled generation, and show large
performance gains over previous methods while retaining fluency and diversity.

positive:
Large-scale language models show promising text generation capabilities, but
users cannot easily control particular aspects of the generated text. We
release CTRL, a 1.63 billion-parameter conditional transformer language model,
trained to condition on control codes that govern style, content, and
task-specific behavior. Control codes were derived from structure that
naturally co-occurs with raw text, preserving the advantages of unsupervised
learning while providing more explicit control over text generation. These
codes also allow CTRL to predict which parts of the training data are most
likely given a sequence. This provides a potential method for analyzing large
amounts of data via model-based source attribution. We have released multiple
full-sized, pretrained versions of CTRL at https://github.com/salesforce/ctrl.

positive:
Large language models (LM) based on Transformers allow to generate plausible
long texts. In this paper, we explore how this generation can be further
controlled at decoding time to satisfy certain constraints (e.g. being
non-toxic, conveying certain emotions, using a specific writing style, etc.)
without fine-tuning the LM. Precisely, we formalize constrained generation as a
tree exploration process guided by a discriminator that indicates how well the
associated sequence respects the constraint. This approach, in addition to
being easier and cheaper to train than fine-tuning the LM, allows to apply the
constraint more finely and dynamically. We propose several original methods to
search this generation tree, notably the Monte Carlo Tree Search (MCTS) which
provides theoretical guarantees on the search efficiency, but also simpler
methods based on re-ranking a pool of diverse sequences using the discriminator
scores. These methods are evaluated, with automatic and human-based metrics, on
two types of constraints and languages: review polarity and emotion control in
French and English. We show that discriminator-guided MCTS decoding achieves
state-of-the-art results without having to tune the language model, in both
tasks and languages. We also demonstrate that other proposed decoding methods
based on re-ranking can be really effective when diversity among the generated
propositions is encouraged.

positive:
Nowadays, there exist powerful language models such as OpenAI's GPT-2 that
can generate readable text and can be fine-tuned to generate text for a
specific domain. Considering GPT-2, it cannot directly generate synthetic news
with respect to a given topic and the output of the language model cannot be
explicitly controlled. In this paper, we study the novel problem of
topic-preserving synthetic news generation. We propose a novel deep
reinforcement learning-based method to control the output of GPT-2 with respect
to a given news topic. When generating text using GPT-2, by default, the most
probable word is selected from the vocabulary. Instead of selecting the best
word each time from GPT-2's output, an RL agent tries to select words that
optimize the matching of a given topic. In addition, using a fake news detector
as an adversary, we investigate generating realistic news using our proposed
method. In this paper, we consider realistic news as news that cannot be easily
detected by a fake news classifier. Experimental results demonstrate the
effectiveness of the proposed framework on generating topic-preserving news
content than state-of-the-art baselines.

positive:
Sequence-to-Sequence (S2S) neural text generation models, especially the
pre-trained ones (e.g., BART and T5), have exhibited compelling performance on
various natural language generation tasks. However, the black-box nature of
these models limits their application in tasks where specific rules (e.g.,
controllable constraints, prior knowledge) need to be executed. Previous works
either design specific model structure (e.g., Copy Mechanism corresponding to
the rule "the generated output should include certain words in the source
input") or implement specialized inference algorithm (e.g., Constrained Beam
Search) to execute particular rules through the text generation. These methods
require careful design case-by-case and are difficult to support multiple rules
concurrently. In this paper, we propose a novel module named Neural
Rule-Execution Tracking Machine that can be equipped into various
transformer-based generators to leverage multiple rules simultaneously to guide
the neural generation model for superior generation performance in a unified
and scalable way. Extensive experimental results on several benchmarks verify
the effectiveness of our proposed model in both controllable and general text
generation.

positive:
Existing pre-trained large language models have shown unparalleled generative
capabilities. However, they are not controllable. In this paper, we propose
MEGATRON-CNTRL, a novel framework that uses large-scale language models and
adds control to text generation by incorporating an external knowledge base.
Our framework consists of a keyword predictor, a knowledge retriever, a
contextual knowledge ranker, and a conditional text generator. As we do not
have access to ground-truth supervision for the knowledge ranker, we make use
of weak supervision from sentence embedding. The empirical results show that
our model generates more fluent, consistent, and coherent stories with less
repetition and higher diversity compared to prior work on the ROC story
dataset. We showcase the controllability of our model by replacing the keywords
used to generate stories and re-running the generation process. Human
evaluation results show that 77.5% of these stories are successfully controlled
by the new keywords. Furthermore, by scaling our model from 124 million to 8.3
billion parameters we demonstrate that larger models improve both the quality
of generation (from 74.5% to 93.0% for consistency) and controllability (from
77.5% to 91.5%).

positive:
We propose a Distributional Approach for addressing Controlled Text
Generation from pre-trained Language Models (LMs). This approach permits to
specify, in a single formal framework, both "pointwise" and "distributional"
constraints over the target LM -- to our knowledge, the first model with such
generality -- while minimizing KL divergence from the initial LM distribution.
The optimal target distribution is then uniquely determined as an explicit EBM
(Energy-Based Model) representation. From that optimal representation we then
train a target controlled Autoregressive LM through an adaptive distributional
variant of Policy Gradient. We conduct a first set of experiments over
pointwise constraints showing the advantages of our approach over a set of
baselines, in terms of obtaining a controlled LM balancing constraint
satisfaction with divergence from the initial LM. We then perform experiments
over distributional constraints, a unique feature of our approach,
demonstrating its potential as a remedy to the problem of Bias in Language
Models. Through an ablation study, we show the effectiveness of our adaptive
technique for obtaining faster convergence. (Code available at
https://github.com/naver/gdc)

positive:
Large-scale pretrained language models have led to dramatic improvements in
text generation. Impressive performance can be achieved by finetuning only on a
small number of instances (few-shot setting). Nonetheless, almost all previous
work simply applies random sampling to select the few-shot training instances.
Little to no attention has been paid to the selection strategies and how they
would affect model performance. In this work, we present a study on training
instance selection in few-shot neural text generation. The selection decision
is made based only on the unlabeled data so as to identify the most worthwhile
data points that should be annotated under some budget of labeling cost. Based
on the intuition that the few-shot training instances should be diverse and
representative of the entire data distribution, we propose a simple selection
strategy with K-means clustering. We show that even with the naive
clustering-based approach, the generation models consistently outperform random
sampling on three text generation tasks: data-to-text generation, document
summarization and question generation. We hope that this work will call for
more attention on this largely unexplored area.

positive:
We investigate large-scale latent variable models (LVMs) for neural story
generation -- an under-explored application for open-domain long text -- with
objectives in two threads: generation effectiveness and controllability. LVMs,
especially the variational autoencoder (VAE), have achieved both effective and
controllable generation through exploiting flexible distributional latent
representations. Recently, Transformers and its variants have achieved
remarkable effectiveness without explicit latent representation learning, thus
lack satisfying controllability in generation. In this paper, we advocate to
revive latent variable modeling, essentially the power of representation
learning, in the era of Transformers to enhance controllability without hurting
state-of-the-art generation effectiveness. Specifically, we integrate latent
representation vectors with a Transformer-based pre-trained architecture to
build conditional variational autoencoder (CVAE). Model components such as
encoder, decoder and the variational posterior are all built on top of
pre-trained language models -- GPT2 specifically in this paper. Experiments
demonstrate state-of-the-art conditional generation ability of our model, as
well as its excellent representation learning capability and controllability.

positive:
Large-scale transformer-based language models (LMs) demonstrate impressive
capabilities in open text generation. However, controlling the generated text's
properties such as the topic, style, and sentiment is challenging and often
requires significant changes to the model architecture or retraining and
fine-tuning the model on new supervised data. This paper presents a novel
approach for Topical Language Generation (TLG) by combining a pre-trained LM
with topic modeling information. We cast the problem using Bayesian probability
formulation with topic probabilities as a prior, LM probabilities as the
likelihood, and topical language generation probability as the posterior. In
learning the model, we derive the topic probability distribution from the
user-provided document's natural structure. Furthermore, we extend our model by
introducing new parameters and functions to influence the quantity of the
topical features presented in the generated text. This feature would allow us
to easily control the topical properties of the generated text. Our
experimental results demonstrate that our model outperforms the
state-of-the-art results on coherency, diversity, and fluency while being
faster in decoding.

positive:
Conditional story generation and contextual text continuation have become
increasingly popular topics in NLP community. Existing models are often prone
to output paragraphs of texts that gradually diverge from the given prompt.
Although the generated text may have a reasonable perplexity and diversity, it
could easily be identified by human as gibberish. The goal of our project is to
improve the coherence and consistency across sentences in a language-generation
model. We aim to solve this issue by first training a sentence pair coherence
classifier with GPT-2 pretrained model, and then co-train the GPT-2 language
model with this new coherence objective using a method analogous to the
REINFORCE algorithm. This fine-tuned language model is able to generate lengthy
paragraph conditioned on a given topic without diverging too much. The
simplicity of this model allows it to be applicable to a variety of underlying
language model architecture since it only modifies the final layer of the
pre-trained model.

positive:
When trained effectively, the Variational Autoencoder (VAE) can be both a
powerful generative model and an effective representation learning framework
for natural language. In this paper, we propose the first large-scale language
VAE model, Optimus. A universal latent embedding space for sentences is first
pre-trained on large text corpus, and then fine-tuned for various language
generation and understanding tasks. Compared with GPT-2, Optimus enables guided
language generation from an abstract level using the latent vectors. Compared
with BERT, Optimus can generalize better on low-resource language understanding
tasks due to the smooth latent space structure. Extensive experimental results
on a wide range of language tasks demonstrate the effectiveness of Optimus. It
achieves new state-of-the-art on VAE language modeling benchmarks. We hope that
our first pre-trained big VAE language model itself and results can help the
NLP community renew the interests of deep generative models in the era of
large-scale pre-training, and make these principled methods more practical.

positive:
When directly using existing text generation datasets for controllable
generation, we are facing the problem of not having the domain knowledge and
thus the aspects that could be controlled are limited. A typical example is
when using CNN/Daily Mail dataset for controllable text summarization, there is
no guided information on the emphasis of summary sentences. A more useful text
generator should leverage both the input text and the control signal to guide
the generation, which can only be built with a deep understanding of the domain
knowledge. Motivated by this vision, our paper introduces a new text generation
dataset, named MReD. Our new dataset consists of 7,089 meta-reviews and all its
45k meta-review sentences are manually annotated with one of the 9 carefully
defined categories, including abstract, strength, decision, etc. We present
experimental results on start-of-the-art summarization models, and propose
methods for structure-controlled generation with both extractive and
abstractive models using our annotated data. By exploring various settings and
analyzing the model behavior with respect to the control signal, we demonstrate
the challenges of our proposed task and the values of our dataset MReD.
Meanwhile, MReD also allows us to have a better understanding of the
meta-review domain.

positive:
There has been considerable progress made towards conversational models that
generate coherent and fluent responses; however, this often involves training
large language models on large dialogue datasets, such as Reddit. These large
conversational models provide little control over the generated responses, and
this control is further limited in the absence of annotated conversational
datasets for attribute specific generation that can be used for fine-tuning the
model. In this paper, we first propose and evaluate plug-and-play methods for
controllable response generation, which does not require dialogue specific
datasets and does not rely on fine-tuning a large model. While effective, the
decoding procedure induces considerable computational overhead, rendering the
conversational model unsuitable for interactive usage. To overcome this, we
introduce an approach that does not require further computation at decoding
time, while also does not require any fine-tuning of a large language model. We
demonstrate, through extensive automatic and human evaluation, a high degree of
control over the generated conversational responses with regard to multiple
desired attributes, while being fluent.

positive:
Large-scale language models (LMs) pretrained on massive corpora of text, such
as GPT-2, are powerful open-domain text generators. However, as our systematic
examination reveals, it is still challenging for such models to generate
coherent long passages of text (e.g., 1000 tokens), especially when the models
are fine-tuned to the target domain on a small corpus. Previous
planning-then-generation methods also fall short of producing such long text in
various domains. To overcome the limitations, we propose a simple but effective
method of generating text in a progressive manner, inspired by generating
images from low to high resolution. Our method first produces domain-specific
content keywords and then progressively refines them into complete passages in
multiple stages. The simple design allows our approach to take advantage of
pretrained LMs at each stage and effectively adapt to any target domain given
only a small set of examples. We conduct a comprehensive empirical study with a
broad set of evaluation metrics, and show that our approach significantly
improves upon the fine-tuned large LMs and various planning-then-generation
methods in terms of quality and sample efficiency. Human evaluation also
validates that our model generations are more coherent.

positive:
Neural text generation models are often autoregressive language models or
seq2seq models. These models generate text by sampling words sequentially, with
each word conditioned on the previous word, and are state-of-the-art for
several machine translation and summarization benchmarks. These benchmarks are
often defined by validation perplexity even though this is not a direct measure
of the quality of the generated text. Additionally, these models are typically
trained via maxi- mum likelihood and teacher forcing. These methods are
well-suited to optimizing perplexity but can result in poor sample quality
since generating text requires conditioning on sequences of words that may have
never been observed at training time. We propose to improve sample quality
using Generative Adversarial Networks (GANs), which explicitly train the
generator to produce high quality samples and have shown a lot of success in
image generation. GANs were originally designed to output differentiable
values, so discrete language generation is challenging for them. We claim that
validation perplexity alone is not indicative of the quality of text generated
by a model. We introduce an actor-critic conditional GAN that fills in missing
text conditioned on the surrounding context. We show qualitatively and
quantitatively, evidence that this produces more realistic conditional and
unconditional text samples compared to a maximum likelihood trained model.

positive:
Transfer learning through large pre-trained models has changed the landscape
of current applications in natural language processing (NLP). Recently Optimus,
a variational autoencoder (VAE) which combines two pre-trained models, BERT and
GPT-2, has been released, and its combination with generative adversial
networks (GANs) has been shown to produce novel, yet very human-looking text.
The Optimus and GANs combination avoids the troublesome application of GANs to
the discrete domain of text, and prevents the exposure bias of standard maximum
likelihood methods. We combine the training of GANs in the latent space, with
the finetuning of the decoder of Optimus for single word generation. This
approach lets us model both the high-level features of the sentences, and the
low-level word-by-word generation. We finetune using reinforcement learning
(RL) by exploiting the structure of GPT-2 and by adding entropy-based
intrinsically motivated rewards to balance between quality and diversity. We
benchmark the results of the VAE-GAN model, and show the improvements brought
by our RL finetuning on three widely used datasets for text generation, with
results that greatly surpass the current state-of-the-art for the quality of
the generated texts.

positive:
This work introduces Focused-Variation Network (FVN), a novel model to
control language generation. The main problems in previous controlled language
generation models range from the difficulty of generating text according to the
given attributes, to the lack of diversity of the generated texts. FVN
addresses these issues by learning disjoint discrete latent spaces for each
attribute inside codebooks, which allows for both controllability and
diversity, while at the same time generating fluent text. We evaluate FVN on
two text generation datasets with annotated content and style, and show
state-of-the-art performance as assessed by automatic and human evaluations.

positive:
Large-scale pretrained language models have shown thrilling generation
capabilities, especially when they generate consistent long text in thousands
of words with ease. However, users of these models can only control the prefix
of sentences or certain global aspects of generated text. It is challenging to
simultaneously achieve fine-grained controllability and preserve the
state-of-the-art unconditional text generation capability. In this paper, we
first propose a new task named "Outline to Story" (O2S) as a test bed for
fine-grained controllable generation of long text, which generates a
multi-paragraph story from cascaded events, i.e. a sequence of outline events
that guide subsequent paragraph generation. We then create dedicate datasets
for future benchmarks, built by state-of-the-art keyword extraction techniques.
Finally, we propose an extremely simple yet strong baseline method for the O2S
task, which fine tunes pre-trained language models on augmented sequences of
outline-story pairs with simple language modeling objective. Our method does
not introduce any new parameters or perform any architecture modification,
except several special tokens as delimiters to build augmented sequences.
Extensive experiments on various datasets demonstrate state-of-the-art
conditional story generation performance with our model, achieving better
fine-grained controllability and user flexibility. Our paper is among the first
ones by our knowledge to propose a model and to create datasets for the task of
"outline to story". Our work also instantiates research interest of
fine-grained controllable generation of open-domain long text, where
controlling inputs are represented by short text.

positive:
Recent neural approaches to data-to-text generation have mostly focused on
improving content fidelity while lacking explicit control over writing styles
(e.g., word choices, sentence structures). More traditional systems use
templates to determine the realization of text. Yet manual or automatic
construction of high-quality templates is difficult, and a template acting as
hard constraints could harm content fidelity when it does not match the record
perfectly. We study a new way of stylistic control by using existing sentences
as soft templates. That is, the model learns to imitate the writing style of
any given exemplar sentence, with automatic adaptions to faithfully describe
the content record. The problem is challenging due to the lack of parallel
data. We develop a neural approach that includes a hybrid attention-copy
mechanism, learns with weak supervisions, and is enhanced with a new content
coverage constraint. We conduct experiments in restaurants and sports domains.
Results show our approach achieves stronger performance than a range of
comparison methods. Our approach balances well between content fidelity and
style control given exemplars that match the records to varying degrees.

positive:
Advances in language modeling architectures and the availability of large
text corpora have driven progress in automatic text generation. While this
results in models capable of generating coherent texts, it also prompts models
to internalize social biases present in the training corpus. This paper aims to
quantify and reduce a particular type of bias exhibited by language models:
bias in the sentiment of generated text. Given a conditioning context (e.g., a
writing prompt) and a language model, we analyze if (and how) the sentiment of
the generated text is affected by changes in values of sensitive attributes
(e.g., country names, occupations, genders) in the conditioning context using a
form of counterfactual evaluation. We quantify sentiment bias by adopting
individual and group fairness metrics from the fair machine learning
literature, and demonstrate that large-scale models trained on two different
corpora (news articles, and Wikipedia) exhibit considerable levels of bias. We
then propose embedding and sentiment prediction-derived regularization on the
language model's latent representations. The regularizations improve fairness
metrics while retaining comparable levels of perplexity and semantic
similarity.

positive:
Text generation has become one of the most important yet challenging tasks in
natural language processing (NLP). The resurgence of deep learning has greatly
advanced this field by neural generation models, especially the paradigm of
pretrained language models (PLMs). In this paper, we present an overview of the
major advances achieved in the topic of PLMs for text generation. As the
preliminaries, we present the general task definition and briefly describe the
mainstream architectures of PLMs for text generation. As the core content, we
discuss how to adapt existing PLMs to model different input data and satisfy
special properties in the generated text. We further summarize several
important fine-tuning strategies for text generation. Finally, we present
several future directions and conclude this paper. Our survey aims to provide
text generation researchers a synthesis and pointer to related research.

positive:
Large-scale language models such as GPT-3 are excellent few-shot learners,
allowing them to be controlled via natural text prompts. Recent studies report
that prompt-based direct classification eliminates the need for fine-tuning but
lacks data and inference scalability. This paper proposes a novel data
augmentation technique that leverages large-scale language models to generate
realistic text samples from a mixture of real samples. We also propose
utilizing soft-labels predicted by the language models, effectively distilling
knowledge from the large-scale language models and creating textual
perturbations simultaneously. We perform data augmentation experiments on
diverse classification tasks and show that our method hugely outperforms
existing text augmentation methods. Ablation studies and a qualitative analysis
provide more insights into our approach.

positive:
Controllable text generation is an appealing but challenging task, which
allows users to specify particular attributes of the generated outputs. In this
paper, we propose a controllable dialogue generation model to steer response
generation under multi-attribute constraints. Specifically, we define and
categorize the commonly used control attributes into global and local ones,
which possess different granularities of effects on response generation. Then,
we significantly extend the conventional seq2seq framework by introducing a
novel two-stage decoder, which first uses a multi-grained style specification
layer to impose the stylistic constraints and determine word-level control
states of responses based on the attributes, and then employs a response
generation layer to generate final responses maintaining both semantic
relevancy to the contexts and fidelity to the attributes. Furthermore, we train
our model with an attribute consistency reward to promote response control with
explicit supervision signals. Extensive experiments and in-depth analyses on
two datasets indicate that our model can significantly outperform competitive
baselines in terms of response quality, content diversity and controllability.

positive:
Many tasks, including language generation, benefit from learning the
structure of the output space, particularly when the space of output labels is
large and the data is sparse. State-of-the-art neural language models
indirectly capture the output space structure in their classifier weights since
they lack parameter sharing across output labels. Learning shared output label
mappings helps, but existing methods have limited expressivity and are prone to
overfitting. In this paper, we investigate the usefulness of more powerful
shared mappings for output labels, and propose a deep residual output mapping
with dropout between layers to better capture the structure of the output space
and avoid overfitting. Evaluations on three language generation tasks show that
our output label mapping can match or improve state-of-the-art recurrent and
self-attention architectures, and suggest that the classifier does not
necessarily need to be high-rank to better model natural language if it is
better at capturing the structure of the output space.

positive:
Fine-tuning pre-trained generative language models to down-stream language
generation tasks has shown promising results. However, this comes with the cost
of having a single, large model for each task, which is not ideal in
low-memory/power scenarios (e.g., mobile). In this paper, we propose an
effective way to fine-tune multiple down-stream generation tasks simultaneously
using a single, large pre-trained model. The experiments on five diverse
language generation tasks show that by just using an additional 2-3% parameters
for each task, our model can maintain or even improve the performance of
fine-tuning the whole model.

positive:
We present a novel approach to data-to-text generation based on iterative
text editing. Our approach maximizes the completeness and semantic accuracy of
the output text while leveraging the abilities of recent pre-trained models for
text editing (LaserTagger) and language modeling (GPT-2) to improve the text
fluency. To this end, we first transform data items to text using trivial
templates, and then we iteratively improve the resulting text by a neural model
trained for the sentence fusion task. The output of the model is filtered by a
simple heuristic and reranked with an off-the-shelf pre-trained language model.
We evaluate our approach on two major data-to-text datasets (WebNLG, Cleaned
E2E) and analyze its caveats and benefits. Furthermore, we show that our
formulation of data-to-text generation opens up the possibility for zero-shot
domain adaptation using a general-domain dataset for sentence fusion.

positive:
Data-to-Text Generation (DTG) is a subfield of Natural Language Generation
aiming at transcribing structured data in natural language descriptions. The
field has been recently boosted by the use of neural-based generators which
exhibit on one side great syntactic skills without the need of hand-crafted
pipelines; on the other side, the quality of the generated text reflects the
quality of the training data, which in realistic settings only offer
imperfectly aligned structure-text pairs. Consequently, state-of-art neural
models include misleading statements - usually called hallucinations - in their
outputs. The control of this phenomenon is today a major challenge for DTG, and
is the problem addressed in the paper.
  Previous work deal with this issue at the instance level: using an alignment
score for each table-reference pair. In contrast, we propose a finer-grained
approach, arguing that hallucinations should rather be treated at the word
level. Specifically, we propose a Multi-Branch Decoder which is able to
leverage word-level labels to learn the relevant parts of each training
instance. These labels are obtained following a simple and efficient scoring
procedure based on co-occurrence analysis and dependency parsing. Extensive
evaluations, via automated metrics and human judgment on the standard WikiBio
benchmark, show the accuracy of our alignment labels and the effectiveness of
the proposed Multi-Branch Decoder. Our model is able to reduce and control
hallucinations, while keeping fluency and coherence in generated texts. Further
experiments on a degraded version of ToTTo show that our model could be
successfully used on very noisy settings.

positive:
In this paper, we present our submission for the English to Czech Text
Translation Task of IWSLT 2019. Our system aims to study how pre-trained
language models, used as input embeddings, can improve a specialized machine
translation system trained on few data. Therefore, we implemented a
Transformer-based encoder-decoder neural system which is able to use the output
of a pre-trained language model as input embeddings, and we compared its
performance under three configurations: 1) without any pre-trained language
model (constrained), 2) using a language model trained on the monolingual parts
of the allowed English-Czech data (constrained), and 3) using a language model
trained on a large quantity of external monolingual data (unconstrained). We
used BERT as external pre-trained language model (configuration 3), and BERT
architecture for training our own language model (configuration 2). Regarding
the training data, we trained our MT system on a small quantity of parallel
text: one set only consists of the provided MuST-C corpus, and the other set
consists of the MuST-C corpus and the News Commentary corpus from WMT. We
observed that using the external pre-trained BERT improves the scores of our
system by +0.8 to +1.5 of BLEU on our development set, and +0.97 to +1.94 of
BLEU on the test set. However, using our own language model trained only on the
allowed parallel data seems to improve the machine translation performances
only when the system is trained on the smallest dataset.

positive:
Given a sentence (e.g., "I like mangoes") and a constraint (e.g., sentiment
flip), the goal of controlled text generation is to produce a sentence that
adapts the input sentence to meet the requirements of the constraint (e.g., "I
hate mangoes"). Going beyond such simple constraints, recent works have started
exploring the incorporation of complex syntactic-guidance as constraints in the
task of controlled paraphrase generation. In these methods, syntactic-guidance
is sourced from a separate exemplar sentence. However, these prior works have
only utilized limited syntactic information available in the parse tree of the
exemplar sentence. We address this limitation in the paper and propose Syntax
Guided Controlled Paraphraser (SGCP), an end-to-end framework for syntactic
paraphrase generation. We find that SGCP can generate syntax conforming
sentences while not compromising on relevance. We perform extensive automated
and human evaluations over multiple real-world English language datasets to
demonstrate the efficacy of SGCP over state-of-the-art baselines. To drive
future research, we have made SGCP's source code available

positive:
Despite considerable advancements with deep neural language models, the
enigma of neural text degeneration persists when these models are tested as
text generators. The counter-intuitive empirical observation is that even
though the use of likelihood as training objective leads to high quality models
for a broad range of language understanding tasks, using likelihood as a
decoding objective leads to text that is bland and strangely repetitive.
  In this paper, we reveal surprising distributional differences between human
text and machine text. In addition, we find that decoding strategies alone can
dramatically effect the quality of machine text, even when generated from
exactly the same neural language model. Our findings motivate Nucleus Sampling,
a simple but effective method to draw the best out of neural generation. By
sampling text from the dynamic nucleus of the probability distribution, which
allows for diversity while effectively truncating the less reliable tail of the
distribution, the resulting text better demonstrates the quality of human text,
yielding enhanced diversity without sacrificing fluency and coherence.

positive:
Concept-to-text Natural Language Generation is the task of expressing an
input meaning representation in natural language. Previous approaches in this
task have been able to generalise to rare or unseen instances by relying on a
delexicalisation of the input. However, this often requires that the input
appears verbatim in the output text. This poses challenges in multilingual
settings, where the task expands to generate the output text in multiple
languages given the same input. In this paper, we explore the application of
multilingual models in concept-to-text and propose Language Agnostic
Delexicalisation, a novel delexicalisation method that uses multilingual
pretrained embeddings, and employs a character-level post-editing model to
inflect words in their correct form during relexicalisation. Our experiments
across five datasets and five languages show that multilingual models
outperform monolingual models in concept-to-text and that our framework
outperforms previous approaches, especially for low resource languages.

positive:
Large, pre-trained transformer-based language models such as BERT have
drastically changed the Natural Language Processing (NLP) field. We present a
survey of recent work that uses these large language models to solve NLP tasks
via pre-training then fine-tuning, prompting, or text generation approaches. We
also present approaches that use pre-trained language models to generate data
for training augmentation or other purposes. We conclude with discussions on
limitations and suggested directions for future research.

positive:
Graph-to-text generation has benefited from pre-trained language models
(PLMs) in achieving better performance than structured graph encoders. However,
they fail to fully utilize the structure information of the input graph. In
this paper, we aim to further improve the performance of the pre-trained
language model by proposing a structured graph-to-text model with a two-step
fine-tuning mechanism which first fine-tunes the model on Wikipedia before
adapting to the graph-to-text generation. In addition to using the traditional
token and position embeddings to encode the knowledge graph (KG), we propose a
novel tree-level embedding method to capture the inter-dependency structures of
the input graph. This new approach has significantly improved the performance
of all text generation metrics for the English WebNLG 2017 dataset.

positive:
Training generative models that can generate high-quality text with
sufficient diversity is an important open problem for Natural Language
Generation (NLG) community. Recently, generative adversarial models have been
applied extensively on text generation tasks, where the adversarially trained
generators alleviate the exposure bias experienced by conventional maximum
likelihood approaches and result in promising generation quality. However, due
to the notorious defect of mode collapse for adversarial training, the
adversarially trained generators face a quality-diversity trade-off, i.e., the
generator models tend to sacrifice generation diversity severely for increasing
generation quality. In this paper, we propose a novel approach which aims to
improve the performance of adversarial text generation via efficiently
decelerating mode collapse of the adversarial training. To this end, we
introduce a cooperative training paradigm, where a language model is
cooperatively trained with the generator and we utilize the language model to
efficiently shape the data distribution of the generator against mode collapse.
Moreover, instead of engaging the cooperative update for the generator in a
principled way, we formulate a meta learning mechanism, where the cooperative
update to the generator serves as a high level meta task, with an intuition of
ensuring the parameters of the generator after the adversarial update would
stay resistant against mode collapse. In the experiment, we demonstrate our
proposed approach can efficiently slow down the pace of mode collapse for the
adversarial text generators. Overall, our proposed method is able to outperform
the baseline approaches with significant margins in terms of both generation
quality and diversity in the testified domains.

positive:
Many neural network models nowadays have achieved promising performances in
Chit-chat settings. The majority of them rely on an encoder for understanding
the post and a decoder for generating the response. Without given assigned
semantics, the models lack the fine-grained control over responses as the
semantic mapping between posts and responses is hidden on the fly within the
end-to-end manners. Some previous works utilize sampled latent words as a
controllable semantic form to drive the generated response around the work, but
few works attempt to use more complex semantic patterns to guide the
generation. In this paper, we propose to use more detailed semantic forms,
including latent responses and part-of-speech sequences sampled from the
corresponding distributions, as the controllable semantics to guide the
generation. Our results show that the richer semantics are not only able to
provide informative and diverse responses, but also increase the overall
performance of response quality, including fluency and coherence.

negative:
This paper investigates whether the power of the models pre-trained on text
data, such as BERT, can be transferred to general token sequence classification
applications. To verify pre-trained models' transferability, we test the
pre-trained models on text classification tasks with meanings of tokens
mismatches, and real-world non-text token sequence classification data,
including amino acid, DNA, and music. We find that even on non-text data, the
models pre-trained on text converge faster, perform better than the randomly
initialized models, and only slightly worse than the models using task-specific
knowledge. We also find that the representations of the text and non-text
pre-trained models share non-trivial similarities.

negative:
Representing a text as a graph for obtaining automatic text summarization has
been investigated for over ten years. With the development of attention or
Transformer on natural language processing (NLP), it is possible to make a
connection between the graph and attention structure for a text. In this paper,
an attention matrix between the sentences of the whole text is adopted as a
weighted adjacent matrix of a fully connected graph of the text, which can be
produced through the pre-training language model. The GCN is further applied to
the text graph model for classifying each node and finding out the salient
sentences from the text. It is demonstrated by the experimental results on two
typical datasets that our proposed model can achieve a competitive result in
comparison with sate-of-the-art models.

negative:
This paper presents a systematic survey on recent development of neural text
generation models. Specifically, we start from recurrent neural network
language models with the traditional maximum likelihood estimation training
scheme and point out its shortcoming for text generation. We thus introduce the
recently proposed methods for text generation based on reinforcement learning,
re-parametrization tricks and generative adversarial nets (GAN) techniques. We
compare different properties of these models and the corresponding techniques
to handle their common problems such as gradient vanishing and generation
diversity. Finally, we conduct a benchmarking experiment with different types
of neural text generation models on two well-known datasets and discuss the
empirical results along with the aforementioned model properties.

negative:
The dominant language modeling paradigm handles text as a sequence of
discrete tokens. While that approach can capture the latent structure of the
text, it is inherently constrained to sequential dynamics for text generation.
We propose a new paradigm for introducing a syntactic inductive bias into
neural text generation, where the dependency parse tree is used to drive the
Transformer model to generate sentences iteratively.
  Our experiments show that this paradigm is effective at text generation, with
quality between LSTMs and Transformers, and comparable diversity, requiring
less than half their decoding steps, and its generation process allows direct
control over the syntactic constructions of the generated text, enabling the
induction of stylistic variations.

negative:
Recent trends in natural language processing using pretraining have shifted
focus towards pretraining and fine-tuning approaches for text generation. Often
the focus has been on task-agnostic approaches that generalize the language
modeling objective. We propose question generation as a pretraining method,
which better aligns with the text generation objectives. Our text generation
models pretrained with this method are better at understanding the essence of
the input and are better language models for the target task. When evaluated on
two text generation tasks, abstractive summarization and answer-focused
question generation, our models result in state-of-the-art performances in
terms of automatic metrics. Human evaluators also found our summaries and
generated questions to be more natural, concise and informative.

negative:
Controlled text perturbation is useful for evaluating and improving model
generalizability. However, current techniques rely on training a model for
every target perturbation, which is expensive and hard to generalize. We
present Tailor, a semantically-controlled text generation system. Tailor builds
on a pretrained seq2seq model and produces textual outputs conditioned on
control codes derived from semantic representations. We craft a set of
operations to modify the control codes, which in turn steer generation towards
targeted attributes. These operations can be further composed into higher-level
ones, allowing for flexible perturbation strategies. We demonstrate the
effectiveness of these perturbations in multiple applications. First, we use
Tailor to automatically create high-quality contrast sets for four distinct
natural language processing (NLP) tasks. These contrast sets contain fewer
spurious artifacts and are complementary to manually annotated ones in their
lexical diversity. Second, we show that Tailor perturbations can improve model
generalization through data augmentation. Perturbing just 2% of training data
leads to a 5.8-point gain on an NLI challenge set measuring reliance on
syntactic heuristics.

negative:
Lexically constrained sentence generation allows the incorporation of prior
knowledge such as lexical constraints into the output. This technique has been
applied to machine translation, and dialog response generation. Previous work
usually used Markov Chain Monte Carlo (MCMC) sampling to generate lexically
constrained sentences, but they randomly determined the position to be edited
and the action to be taken, resulting in many invalid refinements. To overcome
this challenge, we used a classifier to instruct the MCMC-based models where
and how to refine the candidate sentences. First, we developed two methods to
create synthetic data on which the pre-trained model is fine-tuned to obtain a
reliable classifier. Next, we proposed a two-step approach, "Predict and
Revise", for constrained sentence generation. During the predict step, we
leveraged the classifier to compute the learned prior for the candidate
sentence. During the revise step, we resorted to MCMC sampling to revise the
candidate sentence by conducting a sampled action at a sampled position drawn
from the learned prior. We compared our proposed models with many strong
baselines on two tasks, generating sentences with lexical constraints and text
infilling. Experimental results have demonstrated that our proposed model
performs much better than the previous work in terms of sentence fluency and
diversity. Our code and pre-trained models are available at
https://github.com/NLPCode/MCMCXLNet.

negative:
In this work, we tackle the problem of structured text generation,
specifically academic paper generation in $\LaTeX{}$, inspired by the
surprisingly good results of basic character-level language models. Our
motivation is using more recent and advanced methods of language modeling on a
more complex dataset of $\LaTeX{}$ source files to generate realistic academic
papers. Our first contribution is preparing a dataset with $\LaTeX{}$ source
files on recent open-source computer vision papers. Our second contribution is
experimenting with recent methods of language modeling and text generation such
as Transformer and Transformer-XL to generate consistent $\LaTeX{}$ code. We
report cross-entropy and bits-per-character (BPC) results of the trained
models, and we also discuss interesting points on some examples of the
generated $\LaTeX{}$ code.

negative:
Conventional methods for the image-text generation tasks mainly tackle the
naturally bidirectional generation tasks separately, focusing on designing
task-specific frameworks to improve the quality and fidelity of the generated
samples. Recently, Vision-Language Pre-training models have greatly improved
the performance of the image-to-text generation tasks, but large-scale
pre-training models for text-to-image synthesis task are still under-developed.
In this paper, we propose ERNIE-ViLG, a unified generative pre-training
framework for bidirectional image-text generation with transformer model. Based
on the image quantization models, we formulate both image generation and text
generation as autoregressive generative tasks conditioned on the text/image
input. The bidirectional image-text generative modeling eases the semantic
alignments across vision and language. For the text-to-image generation
process, we further propose an end-to-end training method to jointly learn the
visual sequence generator and the image reconstructor. To explore the landscape
of large-scale pre-training for bidirectional text-image generation, we train a
10-billion parameter ERNIE-ViLG model on a large-scale dataset of 145 million
(Chinese) image-text pairs which achieves state-of-the-art performance for both
text-to-image and image-to-text tasks, obtaining an FID of 7.9 on MS-COCO for
text-to-image synthesis and best results on COCO-CN and AIC-ICC for image
captioning.

negative:
The purpose of unconditional text generation is to train a model with real
sentences, then generate novel sentences of the same quality and diversity as
the training data. However, when different metrics are used for comparing the
methods of unconditional text generation, contradictory conclusions are drawn.
The difficulty is that both the diversity and quality of the sample should be
considered simultaneously when the models are evaluated. To solve this problem,
a novel metric of distributional discrepancy (DD) is designed to evaluate
generators based on the discrepancy between the generated and real training
sentences. However, it cannot compute the DD directly because the distribution
of real sentences is unavailable. Thus, we propose a method for estimating the
DD by training a neural-network-based text classifier. For comparison, three
existing metrics, bi-lingual evaluation understudy (BLEU) versus self-BLEU,
language model score versus reverse language model score, and Fr\'{e}chet
embedding distance, along with the proposed DD, are used to evaluate two
popular generative models of long short-term memory and generative pretrained
transformer 2 on both syntactic and real data. Experimental results show that
DD is significantly better than the three existing metrics for ranking these
generative models.

negative:
It has been shown for English that discrete and soft prompting perform
strongly in few-shot learning with pretrained language models (PLMs). In this
paper, we show that discrete and soft prompting perform better than finetuning
in multilingual cases: Crosslingual transfer and in-language training of
multilingual natural language inference. For example, with 48 English training
examples, finetuning obtains 33.74% accuracy in crosslingual transfer, barely
surpassing the majority baseline (33.33%). In contrast, discrete and soft
prompting outperform finetuning, achieving 36.43% and 38.79%. We also
demonstrate good performance of prompting with training data in multiple
languages other than English.

negative:
When fine-tuning pretrained models for classification, researchers either use
a generic model head or a task-specific prompt for prediction. Proponents of
prompting have argued that prompts provide a method for injecting task-specific
guidance, which is beneficial in low-data regimes. We aim to quantify this
benefit through rigorous testing of prompts in a fair setting: comparing
prompted and head-based fine-tuning in equal conditions across many tasks and
data sizes. By controlling for many sources of advantage, we find that
prompting does indeed provide a benefit, and that this benefit can be
quantified per task. Results show that prompting is often worth 100s of data
points on average across classification tasks.

negative:
Lexically constrained text generation aims to control the generated text by
incorporating some pre-specified keywords into the output. Previous work
injects lexical constraints into the output by controlling the decoding process
or refining the candidate output iteratively, which tends to generate generic
or ungrammatical sentences, and has high computational complexity. To address
these challenges, we propose Constrained BART (CBART) for lexically constrained
text generation. CBART leverages the pre-trained model BART and transfers part
of the generation burden from the decoder to the encoder by decomposing this
task into two sub-tasks, thereby improving the sentence quality. Concretely, we
extend BART by adding a token-level classifier over the encoder, aiming at
instructing the decoder where to replace and insert. Guided by the encoder, the
decoder refines multiple tokens of the input in one step by inserting tokens
before specific positions and re-predicting tokens with low confidence. To
further reduce the inference latency, the decoder predicts all tokens in
parallel. Experiment results on One-Billion-Word and Yelp show that CBART can
generate plausible text with high quality and diversity while significantly
accelerating inference.

negative:
Recent advances in neural network-based generative modeling have reignited
the hopes in having computer systems capable of seamlessly conversing with
humans and able to understand natural language. Neural architectures have been
employed to generate text excerpts to various degrees of success, in a
multitude of contexts and tasks that fulfil various user needs. Notably, high
capacity deep learning models trained on large scale datasets demonstrate
unparalleled abilities to learn patterns in the data even in the lack of
explicit supervision signals, opening up a plethora of new possibilities
regarding producing realistic and coherent texts. While the field of natural
language generation is evolving rapidly, there are still many open challenges
to address. In this survey we formally define and categorize the problem of
natural language generation. We review particular application tasks that are
instantiations of these general formulations, in which generating natural
language is of practical importance. Next we include a comprehensive outline of
methods and neural architectures employed for generating diverse texts.
Nevertheless, there is no standard way to assess the quality of text produced
by these generative models, which constitutes a serious bottleneck towards the
progress of the field. To this end, we also review current approaches to
evaluating natural language generation systems. We hope this survey will
provide an informative overview of formulations, methods, and assessments of
neural natural language generation.

negative:
Machine learning approaches applied to NLP are often evaluated by summarizing
their performance in a single number, for example accuracy. Since most test
sets are constructed as an i.i.d. sample from the overall data, this approach
overly simplifies the complexity of language and encourages overfitting to the
head of the data distribution. As such, rare language phenomena or text about
underrepresented groups are not equally included in the evaluation. To
encourage more in-depth model analyses, researchers have proposed the use of
multiple test sets, also called challenge sets, that assess specific
capabilities of a model. In this paper, we develop a framework based on this
idea which is able to generate controlled perturbations and identify subsets in
text-to-scalar, text-to-text, or data-to-text settings. By applying this
framework to the GEM generation benchmark, we propose an evaluation suite made
of 80 challenge sets, demonstrate the kinds of analyses that it enables and
shed light onto the limits of current generation models.

negative:
Many important NLP problems can be posed as dual-sequence or
sequence-to-sequence modeling tasks. Recent advances in building end-to-end
neural architectures have been highly successful in solving such tasks. In this
work we propose a new architecture for dual-sequence modeling that is based on
associative memory. We derive AM-RNNs, a recurrent associative memory (AM)
which augments generic recurrent neural networks (RNN). This architecture is
extended to the Dual AM-RNN which operates on two AMs at once. Our models
achieve very competitive results on textual entailment. A qualitative analysis
demonstrates that long range dependencies between source and target-sequence
can be bridged effectively using Dual AM-RNNs. However, an initial experiment
on auto-encoding reveals that these benefits are not exploited by the system
when learning to solve sequence-to-sequence tasks which indicates that
additional supervision or regularization is needed.

negative:
We investigate the difficulty levels of questions in reading comprehension
datasets such as SQuAD, and propose a new question generation setting, named
Difficulty-controllable Question Generation (DQG). Taking as input a sentence
in the reading comprehension paragraph and some of its text fragments (i.e.,
answers) that we want to ask questions about, a DQG method needs to generate
questions each of which has a given text fragment as its answer, and meanwhile
the generation is under the control of specified difficulty labels---the output
questions should satisfy the specified difficulty as much as possible. To solve
this task, we propose an end-to-end framework to generate questions of
designated difficulty levels by exploring a few important intuitions. For
evaluation, we prepared the first dataset of reading comprehension questions
with difficulty labels. The results show that the question generated by our
framework not only have better quality under the metrics like BLEU, but also
comply with the specified difficulty labels.

negative:
In this paper, we propose a novel controllable text-to-image generative
adversarial network (ControlGAN), which can effectively synthesise high-quality
images and also control parts of the image generation according to natural
language descriptions. To achieve this, we introduce a word-level spatial and
channel-wise attention-driven generator that can disentangle different visual
attributes, and allow the model to focus on generating and manipulating
subregions corresponding to the most relevant words. Also, a word-level
discriminator is proposed to provide fine-grained supervisory feedback by
correlating words with image regions, facilitating training an effective
generator which is able to manipulate specific visual attributes without
affecting the generation of other content. Furthermore, perceptual loss is
adopted to reduce the randomness involved in the image generation, and to
encourage the generator to manipulate specific attributes required in the
modified text. Extensive experiments on benchmark datasets demonstrate that our
method outperforms existing state of the art, and is able to effectively
manipulate synthetic images using natural language descriptions. Code is
available at https://github.com/mrlibw/ControlGAN.

negative:
Current multilingual vision-language models either require a large number of
additional parameters for each supported language, or suffer performance
degradation as languages are added. In this paper, we propose a Scalable
Multilingual Aligned Language Representation (SMALR) that supports many
languages with few model parameters without sacrificing downstream task
performance. SMALR learns a fixed size language-agnostic representation for
most words in a multilingual vocabulary, keeping language-specific features for
just a few. We use a masked cross-language modeling loss to align features with
context from other languages. Additionally, we propose a cross-lingual
consistency module that ensures predictions made for a query and its machine
translation are comparable. The effectiveness of SMALR is demonstrated with ten
diverse languages, over twice the number supported in vision-language tasks to
date. We evaluate on multilingual image-sentence retrieval and outperform prior
work by 3-4% with less than 1/5th the training parameters compared to other
word embedding methods.

negative:
The softmax operator is one of the most important functions in machine
learning models. When applying neural networks to multi-category
classification, the correlations among different categories are often ignored.
For example, in text generation, a language model makes a choice of each new
word based only on the former selection of its context. In this scenario, the
link statistics information of concurrent words based on a corpus (an analogy
of the natural way of expression) is also valuable in choosing the next word,
which can help to improve the sentence's fluency and smoothness. To fully
explore such important information, we propose a graph softmax function for
text generation. It is expected that the final classification result would be
dominated by both the language model and graphical text relationships among
words. We use a graph total variation term to regularize softmax so as to
incorporate the concurrent relationship into the language model. The total
variation of the generated words should be small locally. We apply the proposed
graph softmax to GPT2 for the text generation task. Experimental results
demonstrate that the proposed graph softmax achieves better BLEU and perplexity
than softmax. Human testers can also easily distinguish the text generated by
the graph softmax or softmax.

negative:
We improve the informativeness of models for conditional text generation
using techniques from computational pragmatics. These techniques formulate
language production as a game between speakers and listeners, in which a
speaker should generate output text that a listener can use to correctly
identify the original input that the text describes. While such approaches are
widely used in cognitive science and grounded language learning, they have
received less attention for more standard language generation tasks. We
consider two pragmatic modeling methods for text generation: one where
pragmatics is imposed by information preservation, and another where pragmatics
is imposed by explicit modeling of distractors. We find that these methods
improve the performance of strong existing systems for abstractive
summarization and generation from structured meaning representations.

negative:
Deep generative models are commonly used for generating images and text.
Interpretability of these models is one important pursuit, other than the
generation quality. Variational auto-encoder (VAE) with Gaussian distribution
as prior has been successfully applied in text generation, but it is hard to
interpret the meaning of the latent variable. To enhance the controllability
and interpretability, one can replace the Gaussian prior with a mixture of
Gaussian distributions (GM-VAE), whose mixture components could be related to
hidden semantic aspects of data. In this paper, we generalize the practice and
introduce DEM-VAE, a class of models for text generation using VAEs with a
mixture distribution of exponential family. Unfortunately, a standard
variational training algorithm fails due to the mode-collapse problem. We
theoretically identify the root cause of the problem and propose an effective
algorithm to train DEM-VAE. Our method penalizes the training with an extra
dispersion term to induce a well-structured latent space. Experimental results
show that our approach does obtain a meaningful space, and it outperforms
strong baselines in text generation benchmarks. The code is available at
https://github.com/wenxianxian/demvae.

negative:
The goal of text generation is to make machines express in human language. It
is one of the most important yet challenging tasks in natural language
processing (NLP). Since 2014, various neural encoder-decoder models pioneered
by Seq2Seq have been proposed to achieve the goal by learning to map input text
to output text. However, the input text alone often provides limited knowledge
to generate the desired output, so the performance of text generation is still
far from satisfaction in many real-world scenarios. To address this issue,
researchers have considered incorporating various forms of knowledge beyond the
input text into the generation models. This research direction is known as
knowledge-enhanced text generation. In this survey, we present a comprehensive
review of the research on knowledge enhanced text generation over the past five
years. The main content includes two parts: (i) general methods and
architectures for integrating knowledge into text generation; (ii) specific
techniques and applications according to different forms of knowledge data.
This survey can have broad audiences, researchers and practitioners, in
academia and industry.

negative:
Abstractive text summarization aims to shorten long text documents into a
human readable form that contains the most important facts from the original
document. However, the level of actual abstraction as measured by novel phrases
that do not appear in the source document remains low in existing approaches.
We propose two techniques to improve the level of abstraction of generated
summaries. First, we decompose the decoder into a contextual network that
retrieves relevant parts of the source document, and a pretrained language
model that incorporates prior knowledge about language generation. Second, we
propose a novelty metric that is optimized directly through policy learning to
encourage the generation of novel phrases. Our model achieves results
comparable to state-of-the-art models, as determined by ROUGE scores and human
evaluations, while achieving a significantly higher level of abstraction as
measured by n-gram overlap with the source document.

negative:
The advent of large pre-trained language models has made it possible to make
high-quality predictions on how to add or change a sentence in a document.
However, the high branching factor inherent to text generation impedes the
ability of even the strongest language models to offer useful editing
suggestions at a more global or document level. We introduce a new task,
document sketching, which involves generating entire draft documents for the
writer to review and revise. These drafts are built from sets of documents that
overlap in form - sharing large segments of potentially reusable text - while
diverging in content. To support this task, we introduce a Wikipedia-based
dataset of analogous documents and investigate the application of weakly
supervised methods, including use of a transformer-based mixture of experts,
together with reinforcement learning. We report experiments using automated and
human evaluation methods and discuss relative merits of these models.

negative:
Recent deep generative models are able to provide photo-realistic images as
well as visual or textual content embeddings useful to address various tasks of
computer vision and natural language processing. Their usefulness is
nevertheless often limited by the lack of control over the generative process
or the poor understanding of the learned representation. To overcome these
major issues, very recent work has shown the interest of studying the semantics
of the latent space of generative models. In this paper, we propose to advance
on the interpretability of the latent space of generative models by introducing
a new method to find meaningful directions in the latent space of any
generative model along which we can move to control precisely specific
properties of the generated image like the position or scale of the object in
the image. Our method does not require human annotations and is particularly
well suited for the search of directions encoding simple transformations of the
generated image, such as translation, zoom or color variations. We demonstrate
the effectiveness of our method qualitatively and quantitatively, both for GANs
and variational auto-encoders.

negative:
The adoption of natural language generation (NLG) models can leave
individuals vulnerable to the generation of harmful information memorized by
the models, such as conspiracy theories. While previous studies examine
conspiracy theories in the context of social media, they have not evaluated
their presence in the new space of generative language models. In this work, we
investigate the capability of language models to generate conspiracy theory
text. Specifically, we aim to answer: can we test pretrained generative
language models for the memorization and elicitation of conspiracy theories
without access to the model's training data? We highlight the difficulties of
this task and discuss it in the context of memorization, generalization, and
hallucination. Utilizing a new dataset consisting of conspiracy theory topics
and machine-generated conspiracy theories helps us discover that many
conspiracy theories are deeply rooted in the pretrained language models. Our
experiments demonstrate a relationship between model parameters such as size
and temperature and their propensity to generate conspiracy theory text. These
results indicate the need for a more thorough review of NLG applications before
release and an in-depth discussion of the drawbacks of memorization in
generative language models.

negative:
Some consider large-scale language models that can generate long and coherent
pieces of text as dangerous, since they may be used in misinformation
campaigns. Here we formulate large-scale language model output detection as a
hypothesis testing problem to classify text as genuine or generated. We show
that error exponents for particular language models are bounded in terms of
their perplexity, a standard measure of language generation performance. Under
the assumption that human language is stationary and ergodic, the formulation
is extended from considering specific language models to considering maximum
likelihood language models, among the class of k-order Markov approximations;
error probabilities are characterized. Some discussion of incorporating
semantic side information is also given.

negative:
Recent advances in natural language generation have introduced powerful
language models with high-quality output text. However, this raises concerns
about the potential misuse of such models for malicious purposes. In this
paper, we study natural language watermarking as a defense to help better mark
and trace the provenance of text. We introduce the Adversarial Watermarking
Transformer (AWT) with a jointly trained encoder-decoder and adversarial
training that, given an input text and a binary message, generates an output
text that is unobtrusively encoded with the given message. We further study
different training and inference strategies to achieve minimal changes to the
semantics and correctness of the input text.
  AWT is the first end-to-end model to hide data in text by automatically
learning -- without ground truth -- word substitutions along with their
locations in order to encode the message. We empirically show that our model is
effective in largely preserving text utility and decoding the watermark while
hiding its presence against adversaries. Additionally, we demonstrate that our
method is robust against a range of attacks.

negative:
Sentiment analysis is a task that may suffer from a lack of data in certain
cases, as the datasets are often generated and annotated by humans. In cases
where data is inadequate for training discriminative models, generate models
may aid training via data augmentation. Generative Adversarial Networks (GANs)
are one such model that has advanced the state of the art in several tasks,
including as image and text generation. In this paper, I train GAN models on
low resource datasets, then use them for the purpose of data augmentation
towards improving sentiment classifier generalization. Given the constraints of
limited data, I explore various techniques to train the GAN models. I also
present an analysis of the quality of generated GAN data as more training data
for the GAN is made available. In this analysis, the generated data is
evaluated as a test set (against a model trained on real data points) as well
as a training set to train classification models. Finally, I also conduct a
visual analysis by projecting the generated and the real data into a
two-dimensional space using the t-Distributed Stochastic Neighbor Embedding
(t-SNE) method.

negative:
The recent explosion of machine learning (ML) and artificial intelligence
(AI) shows great potential in the breakthrough of metal additive manufacturing
(AM) process modeling. However, the success of conventional machine learning
tools in data science is primarily attributed to the unprecedented large amount
of labeled data-sets (big data), which can be either obtained by experiments or
first-principle simulations. Unfortunately, these labeled data-sets are
expensive to obtain in AM due to the high expense of the AM experiments and
prohibitive computational cost of high-fidelity simulations.
  We propose a physics-informed neural network (PINN) framework that fuses both
data and first physical principles, including conservation laws of momentum,
mass, and energy, into the neural network to inform the learning processes. To
the best knowledge of the authors, this is the first application of PINN to
three dimensional AM processes modeling. Besides, we propose a hard-type
approach for Dirichlet boundary conditions (BCs) based on a Heaviside function,
which can not only enforce the BCs but also accelerate the learning process.
The PINN framework is applied to two representative metal manufacturing
problems, including the 2018 NIST AM-Benchmark test series. We carefully assess
the performance of the PINN model by comparing the predictions with available
experimental data and high-fidelity simulation results. The investigations show
that the PINN, owed to the additional physical knowledge, can accurately
predict the temperature and melt pool dynamics during metal AM processes with
only a moderate amount of labeled data-sets. The foray of PINN to metal AM
shows the great potential of physics-informed deep learning for broader
applications to advanced manufacturing.

negative:
A lot of work has been done to build text-based language models for
performing different NLP tasks, but not much research has been done in the case
of audio-based language models. This paper proposes a Convolutional Autoencoder
based neural architecture to model syntactically and semantically adequate
contextualized representations of varying length spoken words. The use of such
representations can not only lead to great advances in the audio-based NLP
tasks but can also curtail the loss of information like tone, expression,
accent, etc while converting speech to text to perform these tasks. The
performance of the proposed model is validated by (1) examining the generated
vector space, and (2) evaluating its performance on three benchmark datasets
for measuring word similarities, against existing widely used text-based
language models that are trained on the transcriptions. The proposed model was
able to demonstrate its robustness when compared to the other two
language-based models.

negative:
We investigate neural techniques for end-to-end computational argumentation
mining (AM). We frame AM both as a token-based dependency parsing and as a
token-based sequence tagging problem, including a multi-task learning setup.
Contrary to models that operate on the argument component level, we find that
framing AM as dependency parsing leads to subpar performance results. In
contrast, less complex (local) tagging models based on BiLSTMs perform robustly
across classification scenarios, being able to catch long-range dependencies
inherent to the AM problem. Moreover, we find that jointly learning 'natural'
subtasks, in a multi-task learning setup, improves performance.

negative:
The performance of text classification methods has improved greatly over the
last decade for text instances of less than 512 tokens. This limit has been
adopted by most state-of-the-research transformer models due to the high
computational cost of analyzing longer text instances. To mitigate this problem
and to improve classification for longer texts, researchers have sought to
resolve the underlying causes of the computational cost and have proposed
optimizations for the attention mechanism, which is the key element of every
transformer model. In our study, we are not pursuing the ultimate goal of long
text classification, i.e., the ability to analyze entire text instances at one
time while preserving high performance at a reasonable computational cost.
Instead, we propose a text truncation method called Text Guide, in which the
original text length is reduced to a predefined limit in a manner that improves
performance over naive and semi-naive approaches while preserving low
computational costs. Text Guide benefits from the concept of feature
importance, a notion from the explainable artificial intelligence domain. We
demonstrate that Text Guide can be used to improve the performance of recent
language models specifically designed for long text classification, such as
Longformer. Moreover, we discovered that parameter optimization is the key to
Text Guide performance and must be conducted before the method is deployed.
Future experiments may reveal additional benefits provided by this new method.

negative:
Natural language processing (NLP) techniques have become mainstream in the
recent decade. Most of these advances are attributed to the processing of a
single language. More recently, with the extensive growth of social media
platforms focus has shifted to code-mixed text. The code-mixed text comprises
text written in more than one language. People naturally tend to combine local
language with global languages like English. To process such texts, current NLP
techniques are not sufficient. As a first step, the text is processed to
identify the language of the words in the text. In this work, we focus on
language identification in code-mixed sentences for Hindi-English mixed text.
The task of language identification is formulated as a token classification
task. In the supervised setting, each word in the sentence has an associated
language label. We evaluate different deep learning models and input
representation combinations for this task. Mainly, character, sub-word, and
word embeddings are considered in combination with CNN and LSTM based models.
We show that sub-word representation along with the LSTM model gives the best
results. In general sub-word representations perform significantly better than
other input representations. We report the best accuracy of 94.52% using a
single layer LSTM model on the standard SAIL ICON 2017 test set.

negative:
Vagueness and uncertainty management is counted among one of the challenges
that remain unresolved in systems that generate texts from non-linguistic data,
known as data-to-text systems. In the last decade, work in fuzzy linguistic
summarization and description of data has raised the interest of using fuzzy
sets to model and manage the imprecision of human language in data-to-text
systems. However, despite some research in this direction, there has not been
an actual clear discussion and justification on how fuzzy sets can contribute
to data-to-text for modeling vagueness and uncertainty in words and
expressions. This paper intends to bridge this gap by answering the following
questions: What does vagueness mean in fuzzy sets theory? What does vagueness
mean in data-to-text contexts? In what ways can fuzzy sets theory contribute to
improve data-to-text systems? What are the challenges that researchers from
both disciplines need to address for a successful integration of fuzzy sets
into data-to-text systems? In what cases should the use of fuzzy sets be
avoided in D2T? For this, we review and discuss the state of the art of
vagueness modeling in natural language generation and data-to-text, describe
potential and actual usages of fuzzy sets in data-to-text contexts, and provide
some additional insights about the engineering of data-to-text systems that
make use of fuzzy set-based techniques.

negative:
We investigate whether and where multi-task learning (MTL) can improve
performance on NLP problems related to argumentation mining (AM), in particular
argument component identification. Our results show that MTL performs
particularly well (and better than single-task learning) when little training
data is available for the main task, a common scenario in AM. Our findings
challenge previous assumptions that conceptualizations across AM datasets are
divergent and that MTL is difficult for semantic or higher-level tasks.

negative:
Traditionally, text generation models take in a sequence of text as input,
and iteratively generate the next most probable word using pre-trained
parameters. In this work, we propose the architecture to use images instead of
text as the input of the text generation model, called StoryGen. In the
architecture, we design a Relational Text Data Generator algorithm that relates
different features from multiple images. The output samples from the model
demonstrate the ability to generate meaningful paragraphs of text containing
the extracted features from the input images. This is an undergraduate project
report. Completed Dec. 2019 at the Cooper Union.

negative:
A deep neural network (DNN)-based speech enhancement (SE) aiming to maximize
the performance of an automatic speech recognition (ASR) system is proposed in
this paper. In order to optimize the DNN-based SE model in terms of the
character error rate (CER), which is one of the metric to evaluate the ASR
system and generally non-differentiable, our method uses two DNNs: one for
speech processing and one for mimicking the output CERs derived through an
acoustic model (AM). Then both of DNNs are alternately optimized in the
training phase. Even if the AM is a black-box, e.g., like one provided by a
third-party, the proposed method enables the DNN-based SE model to be optimized
in terms of the CER since the DNN mimicking the AM is differentiable.
Consequently, it becomes feasible to build CER-centric SE model that has no
negative effect, e.g., additional calculation cost and changing network
architecture, on the inference phase since our method is merely a training
scheme for the existing DNN-based methods. Experimental results show that our
method improved CER by 8.8% relative derived through a black-box AM although
certain noise levels are kept.

negative:
Most work in machine reading focuses on question answering problems where the
answer is directly expressed in the text to read. However, many real-world
question answering problems require the reading of text not because it contains
the literal answer, but because it contains a recipe to derive an answer
together with the reader's background knowledge. One example is the task of
interpreting regulations to answer "Can I...?" or "Do I have to...?" questions
such as "I am working in Canada. Do I have to carry on paying UK National
Insurance?" after reading a UK government website about this topic. This task
requires both the interpretation of rules and the application of background
knowledge. It is further complicated due to the fact that, in practice, most
questions are underspecified, and a human assistant will regularly have to ask
clarification questions such as "How long have you been working abroad?" when
the answer cannot be directly derived from the question and text. In this
paper, we formalise this task and develop a crowd-sourcing strategy to collect
32k task instances based on real-world rules and crowd-generated questions and
scenarios. We analyse the challenges of this task and assess its difficulty by
evaluating the performance of rule-based and machine-learning baselines. We
observe promising results when no background knowledge is necessary, and
substantial room for improvement whenever background knowledge is needed.

negative:
We present an approach to analyze $C^1(\mathbb{R}^m)$ functions that
addresses limitations present in the Active Subspaces (AS) method of
Constantine et al.(2015; 2014). Under appropriate hypotheses, our Active
Manifolds (AM) method identifies a 1-D curve in the domain (the active
manifold) on which nearly all values of the unknown function are attained, and
which can be exploited for approximation or analysis, especially when $m$ is
large (high-dimensional input space). We provide theorems justifying our AM
technique and an algorithm permitting functional approximation and sensitivity
analysis. Using accessible, low-dimensional functions as initial examples, we
show AM reduces approximation error by an order of magnitude compared to AS, at
the expense of more computation. Following this, we revisit the sensitivity
analysis by Glaws et al. (2017), who apply AS to analyze a magnetohydrodynamic
power generator model, and compare the performance of AM on the same data. Our
analysis provides detailed information not captured by AS, exhibiting the
influence of each parameter individually along an active manifold. Overall, AM
represents a novel technique for analyzing functional models with benefits
including: reducing $m$-dimensional analysis to a 1-D analogue, permitting more
accurate regression than AS (at more computational expense), enabling more
informative sensitivity analysis, and granting accessible visualizations(2-D
plots) of parameter sensitivity along the AM.

negative:
This article describes an unsupervised language model adaptation approach
that can be used to enhance the performance of language identification methods.
The approach is applied to a current version of the HeLI language
identification method, which is now called HeLI 2.0. We describe the HeLI 2.0
method in detail. The resulting system is evaluated using the datasets from the
German dialect identification and Indo-Aryan language identification shared
tasks of the VarDial workshops 2017 and 2018. The new approach with language
identification provides considerably higher F1-scores than the previous HeLI
method or the other systems which participated in the shared tasks. The results
indicate that unsupervised language model adaptation should be considered as an
option in all language identification tasks, especially in those where
encountering out-of-domain data is likely.

negative:
In the era of big data, a large number of text data generated by the Internet
has given birth to a variety of text representation methods. In natural
language processing (NLP), text representation transforms text into vectors
that can be processed by computer without losing the original semantic
information. However, these methods are difficult to effectively extract the
semantic features among words and distinguish polysemy in language. Therefore,
a text feature representation model based on convolutional neural network (CNN)
and variational autoencoder (VAE) is proposed to extract the text features and
apply the obtained text feature representation on the text classification
tasks. CNN is used to extract the features of text vector to get the semantics
among words and VAE is introduced to make the text feature space more
consistent with Gaussian distribution. In addition, the output of the improved
word2vec model is employed as the input of the proposed model to distinguish
different meanings of the same word in different contexts. The experimental
results show that the proposed model outperforms in k-nearest neighbor (KNN),
random forest (RF) and support vector machine (SVM) classification algorithms.

negative:
The focus of my PhD thesis is on exploring parallel approaches to efficiently
solve problems modeled by constraints and presenting a new proposal. Current
solvers are very advanced; they are carefully designed to effectively manage
the high-level problems' description and include refined strategies to avoid
useless work. Despite this, finding a solution can take an unacceptable amount
of time. Parallelization can mitigate this problem when the instance of the
problem modeled is large, as it happens in real world problems. It is done by
propagating constraints in parallel and concurrently exploring different parts
of the search space. I am developing on a constraint solver that exploits the
many cores available on Graphics Processing Units (GPU) to speed up the search.

negative:
The ability to accurately quantify the performance an additively manufactured
(AM) product is important for a widespread industry adoption of AM as the
design is required to: (1) satisfy geometrical constraints, (2) satisfy
structural constraints dictated by its intended function, and (3) be cost
effective compared to traditional manufacturing methods. Optimization
techniques offer design aids in creating cost-effective structures that meet
the prescribed structural objectives. The fundamental problem in existing
approaches lies in the difficulty to quantify the structural performance as
each unique design leads to a new set of analyses to determine the structural
robustness and such analyses can be very costly due to the complexity of in-use
forces experienced by the structure. This work develops computationally
tractable methods tailored to maximize the structural performance of AM
products. A geometry preserving build orientation optimization method as well
as data-driven shape optimization approaches to structural design are
presented. Proposed methods greatly enhance the value of AM technology by
taking advantage of the design space enabled by it for a broad class of
problems involving complex in-use loads.

negative:
Hybrid manufacturing (HM) technologies combine additive and subtractive
manufacturing (AM/SM) capabilities, leveraging AM's strengths in fabricating
complex geometries and SM's precision and quality to produce finished parts. We
present a systematic approach to automated computer-aided process planning
(CAPP) for HM that can identify non-trivial, qualitatively distinct, and
cost-optimal combinations of AM/SM modalities. A multimodal HM process plan is
represented by a finite Boolean expression of AM and SM manufacturing
primitives, such that the expression evaluates to an 'as-manufactured'
artifact. We show that primitives that respect spatial constraints such as
accessibility and collision avoidance may be constructed by solving inverse
configuration space problems on the 'as-designed' artifact and manufacturing
instruments. The primitives generate a finite Boolean algebra (FBA) that
enumerates the entire search space for planning. The FBA's canonical
intersection terms (i.e., 'atoms') provide the complete domain decomposition to
reframe manufacturability analysis and process planning into purely symbolic
reasoning, once a subcollection of atoms is found to be interchangeable with
the design target. The approach subsumes unimodal (all-AM or all-SM) process
planning as special cases. We demonstrate the practical potency of our
framework and its computational efficiency when applied to process planning of
complex 3D parts with dramatically different AM and SM instruments.

negative:
One of the major challenges in training text-to-image generation models is
the need of a large number of high-quality image-text pairs. While image
samples are often easily accessible, the associated text descriptions typically
require careful human captioning, which is particularly time- and
cost-consuming. In this paper, we propose the first work to train text-to-image
generation models without any text data. Our method leverages the well-aligned
multi-modal semantic space of the powerful pre-trained CLIP model: the
requirement of text-conditioning is seamlessly alleviated via generating text
features from image features. Extensive experiments are conducted to illustrate
the effectiveness of the proposed method. We obtain state-of-the-art results in
the standard text-to-image generation tasks. Importantly, the proposed
language-free model outperforms most existing models trained with full
image-text pairs. Furthermore, our method can be applied in fine-tuning
pre-trained models, which saves both training time and cost in training
text-to-image generation models. Our pre-trained model obtains competitive
results in zero-shot text-to-image generation on the MS-COCO dataset, yet with
around only 1% of the model size and training data size relative to the
recently proposed large DALL-E model.

negative:
Speech-driven visual speech synthesis involves mapping features extracted
from acoustic speech to the corresponding lip animation controls for a face
model. This mapping can take many forms, but a powerful approach is to use deep
neural networks (DNNs). However, a limitation is the lack of synchronized
audio, video, and depth data required to reliably train the DNNs, especially
for speaker-independent models. In this paper, we investigate adapting an
automatic speech recognition (ASR) acoustic model (AM) for the visual speech
synthesis problem. We train the AM on ten thousand hours of audio-only data.
The AM is then adapted to the visual speech synthesis domain using ninety hours
of synchronized audio-visual speech. Using a subjective assessment test, we
compared the performance of the AM-initialized DNN to one with a random
initialization. The results show that viewers significantly prefer animations
generated from the AM-initialized DNN than the ones generated using the
randomly initialized model. We conclude that visual speech synthesis can
significantly benefit from the powerful representation of speech in the ASR
acoustic models.

negative:
In realistic environments, speech is usually interfered by various noise and
reverberation, which dramatically degrades the performance of automatic speech
recognition (ASR) systems. To alleviate this issue, the commonest way is to use
a well-designed speech enhancement approach as the front-end of ASR. However,
more complex pipelines, more computations and even higher hardware costs
(microphone array) are additionally consumed for this kind of methods. In
addition, speech enhancement would result in speech distortions and mismatches
to training. In this paper, we propose an adversarial training method to
directly boost noise robustness of acoustic model. Specifically, a jointly
compositional scheme of generative adversarial net (GAN) and neural
network-based acoustic model (AM) is used in the training phase. GAN is used to
generate clean feature representations from noisy features by the guidance of a
discriminator that tries to distinguish between the true clean signals and
generated signals. The joint optimization of generator, discriminator and AM
concentrates the strengths of both GAN and AM for speech recognition.
Systematic experiments on CHiME-4 show that the proposed method significantly
improves the noise robustness of AM and achieves the average relative error
rate reduction of 23.38% and 11.54% on the development and test set,
respectively.

negative:
AM dependency parsing is a method for neural semantic graph parsing that
exploits the principle of compositionality. While AM dependency parsers have
been shown to be fast and accurate across several graphbanks, they require
explicit annotations of the compositional tree structures for training. In the
past, these were obtained using complex graphbank-specific heuristics written
by experts. Here we show how they can instead be trained directly on the graphs
with a neural latent-variable model, drastically reducing the amount and
complexity of manual heuristics. We demonstrate that our model picks up on
several linguistic phenomena on its own and achieves comparable accuracy to
supervised training, greatly facilitating the use of AM dependency parsing for
new sembanks.

negative:
Emotion stimulus extraction is a fine-grained subtask of emotion analysis
that focuses on identifying the description of the cause behind an emotion
expression from a text passage (e.g., in the sentence "I am happy that I passed
my exam" the phrase "passed my exam" corresponds to the stimulus.). Previous
work mainly focused on Mandarin and English, with no resources or models for
German. We fill this research gap by developing a corpus of 2006 German news
headlines annotated with emotions and 811 instances with annotations of
stimulus phrases. Given that such corpus creation efforts are time-consuming
and expensive, we additionally work on an approach for projecting the existing
English GoodNewsEveryone (GNE) corpus to a machine-translated German version.
We compare the performance of a conditional random field (CRF) model (trained
monolingually on German and cross-lingually via projection) with a multilingual
XLM-RoBERTa (XLM-R) model. Our results show that training with the German
corpus achieves higher F1 scores than projection. Experiments with XLM-R
outperform their respective CRF counterparts.

negative:
Argumentation mining (AM) requires the identification of complex discourse
structures and has lately been applied with success monolingually. In this
work, we show that the existing resources are, however, not adequate for
assessing cross-lingual AM, due to their heterogeneity or lack of complexity.
We therefore create suitable parallel corpora by (human and machine)
translating a popular AM dataset consisting of persuasive student essays into
German, French, Spanish, and Chinese. We then compare (i) annotation projection
and (ii) bilingual word embeddings based direct transfer strategies for
cross-lingual AM, finding that the former performs considerably better and
almost eliminates the loss from cross-lingual transfer. Moreover, we find that
annotation projection works equally well when using either costly human or
cheap machine translations. Our code and data are available at
\url{http://github.com/UKPLab/coling2018-xling_argument_mining}.

negative:
In this study, listeners of varied Indian nativities are asked to listen and
recognize TIMIT utterances spoken by American speakers. We have three kinds of
responses from each listener while they recognize an utterance: 1. Sentence
difficulty ratings, 2. Speaker difficulty ratings, and 3. Transcription of the
utterance. From these transcriptions, word error rate (WER) is calculated and
used as a metric to evaluate the similarity between the recognized and the
original sentences.The sentences selected in this study are categorized into
three groups: Easy, Medium and Hard, based on the frequency ofoccurrence of the
words in them. We observe that the sentence, speaker difficulty ratings and the
WERs increase from easy to hard categories of sentences. We also compare the
human speech recognition performance with that using three automatic speech
recognition (ASR) under following three combinations of acoustic model (AM) and
language model(LM): ASR1) AM trained with recordings from speakers of Indian
origin and LM built on TIMIT text, ASR2) AM using recordings from native
American speakers and LM built ontext from LIBRI speech corpus, and ASR3) AM
using recordings from native American speakers and LM build on LIBRI speech and
TIMIT text. We observe that HSR performance is similar to that of ASR1 whereas
ASR3 achieves the best performance. Speaker nativity wise analysis shows that
utterances from speakers of some nativity are more difficult to recognize by
Indian listeners compared to few other nativities

negative:
A neuroscience method to understanding the brain is to find and study the
preferred stimuli that highly activate an individual cell or groups of cells.
Recent advances in machine learning enable a family of methods to synthesize
preferred stimuli that cause a neuron in an artificial or biological brain to
fire strongly. Those methods are known as Activation Maximization (AM) or
Feature Visualization via Optimization. In this chapter, we (1) review existing
AM techniques in the literature; (2) discuss a probabilistic interpretation for
AM; and (3) review the applications of AM in debugging and explaining networks.

negative:
The additive margin softmax (AM-Softmax) loss has delivered remarkable
performance in speaker verification. A supposed behavior of AM-Softmax is that
it can shrink within-class variation by putting emphasis on target logits,
which in turn improves margin between target and non-target classes. In this
paper, we conduct a careful analysis on the behavior of AM-Softmax loss, and
show that this loss does not implement real max-margin training. Based on this
observation, we present a Real AM-Softmax loss which involves a true margin
function in the softmax training. Experiments conducted on VoxCeleb1, SITW and
CNCeleb demonstrated that the corrected AM-Softmax loss consistently
outperforms the original one. The code has been released at
https://gitlab.com/csltstu/sunine.

negative:
The present document is an excerpt of an essay that I wrote as part of my
application material to graduate school in Computer Science (with a focus on
Artificial Intelligence), in 1986. I was not invited by any of the schools that
received it, so I became a theoretical physicist instead. The essay's full
title was "Some Topics in Philosophy and Computer Science". I am making this
text (unchanged from 1985, preserving the typesetting as much as possible)
available now in memory of Jerry Fodor, whose writings had influenced me
significantly at the time (even though I did not always agree).

negative:
In this paper, I will discuss the work I am currently doing as a Ph.D.
student at the University of Potsdam, under the tutoring of T. Schaub. I'm
currently looking into action description in ASP. More precisely, my goal is to
explore how to represent actions with durations in ASP, in different contexts.
Right now, I'm focused on Multi-Agent Path Finding (MAPF), looking at how to
represent speeds for different agents and contexts.
  Before tackling duration, I wanted to explore and compare different
representations of action taking in ASP. For this, I started comparing
different simple encodings tackling the MAPF problem. Even in simple code,
choices and assumptions have been made in their creations. The objective of my
work is to present the consequences of those design decisions in terms of
performance and knowledge representation. As far as I know, there is no current
research on this topic.
  Besides that, I'm also exploring different ways to represent duration and to
solve related problems. I planed to compare them the same way I described
before. I also want this to help me find innovative and effective ways to solve
problems with duration.

negative:
Additive Manufacturing (AM) is a crucial component of the smart industry. In
this paper, we propose an automated quality grading system for the AM process
using a deep convolutional neural network (CNN) model. The CNN model is trained
offline using the images of the internal and surface defects in the
layer-by-layer deposition of materials and tested online by studying the
performance of detecting and classifying the failure in AM process at different
extruder speeds and temperatures. The model demonstrates the accuracy of 94%
and specificity of 96%, as well as above 75% in three classifier measures of
the Fscore, the sensitivity, and precision for classifying the quality of the
printing process in five grades in real-time. The proposed online model adds an
automated, consistent, and non-contact quality control signal to the AM process
that eliminates the manual inspection of parts after they are entirely built.
The quality monitoring signal can also be used by the machine to suggest
remedial actions by adjusting the parameters in real-time. The proposed quality
predictive model serves as a proof-of-concept for any type of AM machines to
produce reliable parts with fewer quality hiccups while limiting the waste of
both time and materials.

negative:
Additive Manufacturing (AM) is a manufacturing paradigm that builds
three-dimensional objects from a computer-aided design model by successively
adding material layer by layer. AM has become very popular in the past decade
due to its utility for fast prototyping such as 3D printing as well as
manufacturing functional parts with complex geometries using processes such as
laser metal deposition that would be difficult to create using traditional
machining. As the process for creating an intricate part for an expensive metal
such as Titanium is prohibitive with respect to cost, computational models are
used to simulate the behavior of AM processes before the experimental run.
However, as the simulations are computationally costly and time-consuming for
predicting multiscale multi-physics phenomena in AM, physics-informed
data-driven machine-learning systems for predicting the behavior of AM
processes are immensely beneficial. Such models accelerate not only multiscale
simulation tools but also empower real-time control systems using in-situ data.
In this paper, we design and develop essential components of a scientific
framework for developing a data-driven model-based real-time control system.
Finite element methods are employed for solving time-dependent heat equations
and developing the database. The proposed framework uses extremely randomized
trees - an ensemble of bagged decision trees as the regression algorithm
iteratively using temperatures of prior voxels and laser information as inputs
to predict temperatures of subsequent voxels. The models achieve mean absolute
percentage errors below 1% for predicting temperature profiles for AM
processes. The code is made available for the research community at
https://github.com/paularindam/ml-iter-additive.

negative:
We address the problem of solving mixed random linear equations. We have
unlabeled observations coming from multiple linear regressions, and each
observation corresponds to exactly one of the regression models. The goal is to
learn the linear regressors from the observations. Classically, Alternating
Minimization (AM) (which is a variant of Expectation Maximization (EM)) is used
to solve this problem. AM iteratively alternates between the estimation of
labels and solving the regression problems with the estimated labels.
Empirically, it is observed that, for a large variety of non-convex problems
including mixed linear regression, AM converges at a much faster rate compared
to gradient based algorithms. However, the existing theory suggests similar
rate of convergence for AM and gradient based methods, failing to capture this
empirical behavior. In this paper, we close this gap between theory and
practice for the special case of a mixture of $2$ linear regressions. We show
that, provided initialized properly, AM enjoys a \emph{super-linear} rate of
convergence in certain parameter regimes. To the best of our knowledge, this is
the first work that theoretically establishes such rate for AM. Hence, if we
want to recover the unknown regressors upto an error (in $\ell_2$ norm) of
$\epsilon$, AM only takes $\mathcal{O}(\log \log (1/\epsilon))$ iterations.
Furthermore, we compare AM with a gradient based heuristic algorithm
empirically and show that AM dominates in iteration complexity as well as
wall-clock time.