query: I am currently involved in a project that aims to improve the safety and efficiency of self-driving cars by predicting the paths of surrounding objects in urban driving situations. Current methods, such as multimodal regression and occupancy maps, have shown limitations in accurately predicting an object’s path. My plan is to treat trajectory prediction as a ranking task, sorting through the set of all potential paths. I need algorithms that can generate a diverse set of potential paths, accommodating a wide range of situations while concurrently discarding paths that are physically unfeasible. To enhance this method, I plan to generate these path sets based on the vehicle's current situation. For example, when the vehicle is on a highway, the model should not generate paths for objects on the roadside. I am also interested in integrating real-time traffic data and weather conditions into the model to increase its accuracy. Lastly, I aim to validate my novel approach using real-world autonomous vehicle data and compare its performance with existing trajectory prediction models.

positive:
Autonomous explorative robots frequently encounter scenarios where multiple
future trajectories can be pursued. Often these are cases with multiple paths
around an obstacle or trajectory options towards various frontiers. Humans in
such situations can inherently perceive and reason about the surrounding
environment to identify several possibilities of either manoeuvring around the
obstacles or moving towards various frontiers. In this work, we propose a 2
stage Convolutional Neural Network architecture which mimics such an ability to
map the perceived surroundings to multiple trajectories that a robot can choose
to traverse. The first stage is a Trajectory Proposal Network which suggests
diverse regions in the environment which can be occupied in the future. The
second stage is a Trajectory Sampling network which provides a finegrained
trajectory over the regions proposed by Trajectory Proposal Network. We
evaluate our framework in diverse and complicated real life settings. For the
outdoor case, we use the KITTI dataset and our own outdoor driving dataset. In
the indoor setting, we use an autonomous drone to navigate various scenarios
and also a ground robot which can explore the environment using the
trajectories proposed by our framework. Our experiments suggest that the
framework is able to develop a semantic understanding of the obstacles, open
regions and identify diverse trajectories that a robot can traverse. Our
comparisons portray the performance gain of the proposed architecture over a
diverse set of methods against which it is compared.

positive:
To safely and efficiently navigate in complex urban traffic, autonomous
vehicles must make responsible predictions in relation to surrounding
traffic-agents (vehicles, bicycles, pedestrians, etc.). A challenging and
critical task is to explore the movement patterns of different traffic-agents
and predict their future trajectories accurately to help the autonomous vehicle
make reasonable navigation decision. To solve this problem, we propose a long
short-term memory-based (LSTM-based) realtime traffic prediction algorithm,
TrafficPredict. Our approach uses an instance layer to learn instances'
movements and interactions and has a category layer to learn the similarities
of instances belonging to the same type to refine the prediction. In order to
evaluate its performance, we collected trajectory datasets in a large city
consisting of varying conditions and traffic densities. The dataset includes
many challenging scenarios where vehicles, bicycles, and pedestrians move among
one another. We evaluate the performance of TrafficPredict on our new dataset
and highlight its higher accuracy for trajectory prediction by comparing with
prior prediction methods.

positive:
To navigate safely in urban environments, an autonomous vehicle (ego vehicle)
must understand and anticipate its surroundings, in particular the behavior and
intents of other road users (neighbors). Most of the times, multiple decision
choices are acceptable for all road users (e.g., turn right or left, or
different ways of avoiding an obstacle), leading to a highly uncertain and
multi-modal decision space. We focus here on predicting multiple feasible
future trajectories for both ego vehicle and neighbors through a probabilistic
framework. We rely on a conditional imitation learning algorithm, conditioned
by a navigation command for the ego vehicle (e.g., "turn right"). Our model
processes ego vehicle front-facing camera images and bird-eye view grid,
computed from Lidar point clouds, with detections of past and present objects,
in order to generate multiple trajectories for both ego vehicle and its
neighbors. Our approach is computationally efficient and relies only on
on-board sensors. We evaluate our method offline on the publicly available
dataset nuScenes, achieving state-of-the-art performance, investigate the
impact of our architecture choices on online simulated experiments and show
preliminary insights for real vehicle control

positive:
Mimicking human ability to forecast future positions or interpret complex
interactions in urban scenarios, such as streets, shopping malls or squares, is
essential to develop socially compliant robots or self-driving cars. Autonomous
systems may gain advantage on anticipating human motion to avoid collisions or
to naturally behave alongside people. To foresee plausible trajectories, we
construct an LSTM (long short-term memory)-based model considering three
fundamental factors: people interactions, past observations in terms of
previously crossed areas and semantics of surrounding space. Our model
encompasses several pooling mechanisms to join the above elements defining
multiple tensors, namely social, navigation and semantic tensors. The network
is tested in unstructured environments where complex paths emerge according to
both internal (intentions) and external (other people, not accessible areas)
motivations. As demonstrated, modeling paths unaware of social interactions or
context information, is insufficient to correctly predict future positions.
Experimental results corroborate the effectiveness of the proposed framework in
comparison to LSTM-based models for human path prediction.

positive:
Predicting a vehicle's trajectory is an essential ability for autonomous
vehicles navigating through complex urban traffic scenes. Bird's-eye-view
roadmap information provides valuable information for making trajectory
predictions, and while state-of-the-art models extract this information via
image convolution, auxiliary loss functions can augment patterns inferred from
deep learning by further encoding common knowledge of social and legal driving
behaviors. Since human driving behavior is inherently multimodal, models which
allow for multimodal output tend to outperform single-prediction models on
standard metrics. We propose a loss function which enhances such models by
enforcing expected driving rules on all predicted modes. Our contribution to
trajectory prediction is twofold; we propose a new metric which addresses
failure cases of the off-road rate metric by penalizing trajectories that
oppose the ascribed heading (flow direction) of a driving lane, and we show
this metric to be differentiable and therefore suitable as an auxiliary loss
function. We then use this auxiliary loss to extend the the standard multiple
trajectory prediction (MTP) and MultiPath models, achieving improved results on
the nuScenes prediction benchmark by predicting trajectories which better
conform to the lane-following rules of the road.

positive:
Annually, a large number of injuries and deaths around the world are related
to motor vehicle accidents. This value has recently been reduced to some
extent, via the use of driver-assistance systems. Developing driver-assistance
systems (i.e., automated driving systems) can play a crucial role in reducing
this number. Estimating and predicting surrounding vehicles' movement is
essential for an automated vehicle and advanced safety systems. Moreover,
predicting the trajectory is influenced by numerous factors, such as drivers'
behavior during accidents, history of the vehicle's movement and the
surrounding vehicles, and their position on the traffic scene. The vehicle must
move over a safe path in traffic and react to other drivers' unpredictable
behaviors in the shortest time. Herein, to predict automated vehicles' path, a
model with low computational complexity is proposed, which is trained by images
taken from the road's aerial image. Our method is based on an encoder-decoder
model that utilizes a social tensor to model the effect of the surrounding
vehicles' movement on the target vehicle. The proposed model can predict the
vehicle's future path in any freeway only by viewing the images related to the
history of the target vehicle's movement and its neighbors. Deep learning was
used as a tool for extracting the features of these images. Using the HighD
database, an image dataset of the road's aerial image was created, and the
model's performance was evaluated on this new database. We achieved the RMSE of
1.91 for the next 5 seconds and found that the proposed method had less error
than the best path-prediction methods in previous studies.

positive:
Predicting the future trajectory of surrounding vehicles in a randomly
varying traffic level is one of the most challenging problems in developing an
autonomous vehicle. Since there is no pre-defined number of interacting
vehicles participate in, the prediction network has to be scalable with respect
to the vehicle number in order to guarantee the consistency in terms of both
accuracy and computational load. In this paper, the first fully scalable
trajectory prediction network, SCALE-Net, is proposed that can ensure both
higher prediction performance and consistent computational load regardless of
the number of surrounding vehicles. The SCALE-Net employs the Edge-enhance
Graph Convolutional Neural Network (EGCN) for the inter-vehicular interaction
embedding network. Since the proposed EGCN is inherently scalable with respect
to the graph node (an agent in this study), the model can be operated
independently from the total number of vehicles considered. We evaluated the
scalability of the SCALE-Net on the publically available NGSIM datasets by
comparing variations on computation time and prediction accuracy per single
driving scene with respect to the varying vehicle number. The experimental test
shows that both computation time and prediction performance of the SCALE-Net
consistently outperform those of previous models regardless of the level of
traffic complexities.

positive:
Urban environments manifest a high level of complexity, and therefore it is
of vital importance for safety systems embedded within autonomous vehicles
(AVs) to be able to accurately predict the short-term future motion of nearby
agents. This problem can be further understood as generating a sequence of
future coordinates for a given agent based on its past motion data e.g.
position, velocity, acceleration etc, and whilst current approaches demonstrate
plausible results they have a propensity to neglect a scene's physical
constrains. In this paper we propose the model based on a combination of the
CNN and LSTM encoder-decoder architecture that learns to extract a relevant
road features from semantic maps as well as general motion of agents and uses
this learned representation to predict their short-term future trajectories. We
train and validate the model on the publicly available dataset that provides
data from urban areas, allowing us to examine it in challenging and uncertain
scenarios. We show that our model is not only capable of anticipating future
motion whilst taking into consideration road boundaries, but can also
effectively and precisely predict trajectories for a longer time horizon than
initially trained for.

positive:
Understanding the intentions of drivers at intersections is a critical
component for autonomous vehicles. Urban intersections that do not have traffic
signals are a common epicentre of highly variable vehicle movement and
interactions. We present a method for predicting driver intent at urban
intersections through multi-modal trajectory prediction with uncertainty. Our
method is based on recurrent neural networks combined with a mixture density
network output layer. To consolidate the multi-modal nature of the output
probability distribution, we introduce a clustering algorithm that extracts the
set of possible paths that exist in the prediction output, and ranks them
according to likelihood. To verify the method's performance and
generalizability, we present a real-world dataset that consists of over 23,000
vehicles traversing five different intersections, collected using a vehicle
mounted Lidar based tracking system. An array of metrics is used to demonstrate
the performance of the model against several baselines.

positive:
Autonomous vehicles should be able to predict the future states of its
environment and respond appropriately. Specifically, predicting the behavior of
surrounding human drivers is vital for such platforms to share the same road
with humans. Behavior of each of the surrounding vehicles is governed by the
motion of its neighbor vehicles. This paper focuses on predicting the behavior
of the surrounding vehicles of an autonomous vehicle on highways. We are
motivated by improving the prediction accuracy when a surrounding vehicle
performs lane change and highway merging maneuvers. We propose a novel pooling
strategy to capture the inter-dependencies between the neighbor vehicles.
Depending solely on Euclidean trajectory representation, the existing pooling
strategies do not model the context information of the maneuvers intended by a
surrounding vehicle. In contrast, our pooling mechanism employs polar
trajectory representation, vehicles orientation and radial velocity. This
results in an implicitly maneuver-aware pooling operation. We incorporated the
proposed pooling mechanism into a generative encoder-decoder model, and
evaluated our method on the public NGSIM dataset. The results of maneuver-based
trajectory predictions demonstrate the effectiveness of the proposed method
compared with the state-of-the-art approaches. Our "Pooling Toolbox" code is
available at https://github.com/m-hasan-n/pooling.

positive:
The trajectory prediction is a critical and challenging problem in the design
of an autonomous driving system. Many AI-oriented companies, such as Google
Waymo, Uber and DiDi, are investigating more accurate vehicle trajectory
prediction algorithms. However, the prediction performance is governed by lots
of entangled factors, such as the stochastic behaviors of surrounding vehicles,
historical information of self-trajectory, and relative positions of neighbors,
etc. In this paper, we propose a novel graph-based information sharing network
(GISNet) that allows the information sharing between the target vehicle and its
surrounding vehicles. Meanwhile, the model encodes the historical trajectory
information of all the vehicles in the scene. Experiments are carried out on
the public NGSIM US-101 and I-80 Dataset and the prediction performance is
measured by the Root Mean Square Error (RMSE). The quantitative and qualitative
experimental results show that our model significantly improves the trajectory
prediction accuracy, by up to 50.00%, compared to existing models.

positive:
Reliable prediction of surround vehicle motion is a critical requirement for
path planning for autonomous vehicles. In this paper we propose a unified
framework for surround vehicle maneuver classification and motion prediction
that exploits multiple cues, namely, the estimated motion of vehicles, an
understanding of typical motion patterns of freeway traffic and inter-vehicle
interaction. We report our results in terms of maneuver classification accuracy
and mean and median absolute error of predicted trajectories against the ground
truth for real traffic data collected using vehicle mounted sensors on
freeways. An ablative analysis is performed to analyze the relative importance
of each cue for trajectory prediction. Additionally, an analysis of execution
time for the components of the framework is presented. Finally, we present
multiple case studies analyzing the outputs of our model for complex traffic
scenarios

positive:
Predicting future trajectories of surrounding obstacles is a crucial task for
autonomous driving cars to achieve a high degree of road safety. There are
several challenges in trajectory prediction in real-world traffic scenarios,
including obeying traffic rules, dealing with social interactions, handling
traffic of multi-class movement, and predicting multi-modal trajectories with
probability. Inspired by people's natural habit of navigating traffic with
attention to their goals and surroundings, this paper presents a unique dynamic
graph attention network to solve all those challenges. The network is designed
to model the dynamic social interactions among agents and conform to traffic
rules with a semantic map. By extending the anchor-based method to multiple
types of agents, the proposed method can predict multi-modal trajectories with
probabilities for multi-class movements using a single model. We validate our
approach on the proprietary autonomous driving dataset for the logistic
delivery scenario and two publicly available datasets. The results show that
our method outperforms state-of-the-art techniques and demonstrates the
potential for trajectory prediction in real-world traffic.

positive:
The ability to predict multiple possible future positions of the ego-vehicle
given the surrounding context while also estimating their probabilities is key
to safe autonomous driving. Most of the current state-of-the-art Deep Learning
approaches are trained on trajectory data to achieve this task. However
trajectory data captured by sensor systems is highly imbalanced, since by far
most of the trajectories follow straight lines with an approximately constant
velocity. This poses a huge challenge for the task of predicting future
positions, which is inherently a regression problem. Current state-of-the-art
approaches alleviate this problem only by major preprocessing of the training
data, e.g. resampling, clustering into anchors etc. In this paper we propose an
approach which reformulates the prediction problem as a classification task,
allowing for powerful tools, e.g. focal loss, to combat the imbalance. To this
end we design a generative probabilistic model consisting of a deep neural
network with a Mixture of Gaussian head. A smart choice of the latent variable
allows for the reformulation of the log-likelihood function as a combination of
a classification problem and a much simplified regression problem. The output
of our model is an estimate of the probability density function of future
positions, hence allowing for prediction of multiple possible positions while
also estimating their probabilities. The proposed approach can easily
incorporate context information and does not require any preprocessing of the
data.

positive:
For autonomous agents to successfully operate in real world, the ability to
anticipate future motions of surrounding entities in the scene can greatly
enhance their safety levels since potentially dangerous situations could be
avoided in advance. While impressive results have been shown on predicting each
agent's behavior independently, we argue that it is not valid to consider road
entities individually since transitions of vehicle states are highly coupled.
Moreover, as the predicted horizon becomes longer, modeling prediction
uncertainties and multi-modal distributions over future sequences will turn
into a more challenging task. In this paper, we address this challenge by
presenting a multi-modal probabilistic prediction approach. The proposed method
is based on a generative model and is capable of jointly predicting sequential
motions of each pair of interacting agents. Most importantly, our model is
interpretable, which can explain the underneath logic as well as obtain more
reliability to use in real applications. A complicate real-world roundabout
scenario is utilized to implement and examine the proposed method.

positive:
Trajectory and intention prediction of traffic participants is an important
task in automated driving and crucial for safe interaction with the
environment. In this paper, we present a new approach to vehicle trajectory
prediction based on automatically generated maps containing statistical
information about the behavior of traffic participants in a given area. These
maps are generated based on trajectory observations using image processing and
map matching techniques and contain all typical vehicle movements and
probabilities in the considered area. Our prediction approach matches an
observed trajectory to a behavior contained in the map and uses this
information to generate a prediction. We evaluated our approach on a dataset
containing over 14000 trajectories and found that it produces significantly
more precise mid-term predictions compared to motion model-based prediction
approaches.

positive:
As autonomous vehicles (AVs) need to interact with other road users, it is of
importance to comprehensively understand the dynamic traffic environment,
especially the future possible trajectories of surrounding vehicles. This paper
presents an algorithm for long-horizon trajectory prediction of surrounding
vehicles using a dual long short term memory (LSTM) network, which is capable
of effectively improving prediction accuracy in strongly interactive driving
environments. In contrast to traditional approaches which require trajectory
matching and manual feature selection, this method can automatically learn
high-level spatial-temporal features of driver behaviors from naturalistic
driving data through sequence learning. By employing two blocks of LSTMs, the
proposed method feeds the sequential trajectory to the first LSTM for driver
intention recognition as an intermediate indicator, which is immediately
followed by a second LSTM for future trajectory prediction. Test results from
real-world highway driving data show that the proposed method can, in
comparison to state-of-art methods, output more accurate and reasonable
estimate of different future trajectories over 5s time horizon with root mean
square error (RMSE) for longitudinal and lateral prediction less than 5.77m and
0.49m, respectively.

positive:
In a given scenario, simultaneously and accurately predicting every possible
interaction of traffic participants is an important capability for autonomous
vehicles. The majority of current researches focused on the prediction of an
single entity without incorporating the environment information. Although some
approaches aimed to predict multiple vehicles, they either predicted each
vehicle independently with no considerations on possible interaction with
surrounding entities or generated discretized joint motions which cannot be
directly used in decision making and motion planning for autonomous vehicle. In
this paper, we present a probabilistic framework that is able to jointly
predict continuous motions for multiple interacting road participants under any
driving scenarios and is capable of forecasting the duration of each
interaction, which can enhance the prediction performance and efficiency. The
proposed traffic scene prediction framework contains two hierarchical modules:
the upper module and the lower module. The upper module forecasts the intention
of the predicted vehicle, while the lower module predicts motions for
interacting scene entities. An exemplar real-world scenario is used to
implement and examine the proposed framework.

positive:
The environmental perception of an autonomous vehicle is limited by its
physical sensor ranges and algorithmic performance, as well as by occlusions
that degrade its understanding of an ongoing traffic situation. This not only
poses a significant threat to safety and limits driving speeds, but it can also
lead to inconvenient maneuvers. Intelligent Infrastructure Systems can help to
alleviate these problems. An Intelligent Infrastructure System can fill in the
gaps in a vehicle's perception and extend its field of view by providing
additional detailed information about its surroundings, in the form of a
digital model of the current traffic situation, i.e. a digital twin. However,
detailed descriptions of such systems and working prototypes demonstrating
their feasibility are scarce. In this paper, we propose a hardware and software
architecture that enables such a reliable Intelligent Infrastructure System to
be built. We have implemented this system in the real world and demonstrate its
ability to create an accurate digital twin of an extended highway stretch, thus
enhancing an autonomous vehicle's perception beyond the limits of its on-board
sensors. Furthermore, we evaluate the accuracy and reliability of the digital
twin by using aerial images and earth observation methods for generating ground
truth data.

positive:
Accurate prediction of pedestrian and bicyclist paths is integral to the
development of reliable autonomous vehicles in dense urban environments. The
interactions between vehicle and pedestrian or bicyclist have a significant
impact on the trajectories of traffic participants e.g. stopping or turning to
avoid collisions. Although recent datasets and trajectory prediction approaches
have fostered the development of autonomous vehicles yet the amount of
vehicle-pedestrian (bicyclist) interactions modeled are sparse. In this work,
we propose Euro-PVI, a dataset of pedestrian and bicyclist trajectories. In
particular, our dataset caters more diverse and complex interactions in dense
urban scenarios compared to the existing datasets. To address the challenges in
predicting future trajectories with dense interactions, we develop a joint
inference model that learns an expressive multi-modal shared latent space
across agents in the urban scene. This enables our Joint-$\beta$-cVAE approach
to better model the distribution of future trajectories. We achieve state of
the art results on the nuScenes and Euro-PVI datasets demonstrating the
importance of capturing interactions between ego-vehicle and pedestrians
(bicyclists) for accurate predictions.

positive:
This paper addresses the problem of path prediction for multiple interacting
agents in a scene, which is a crucial step for many autonomous platforms such
as self-driving cars and social robots. We present \textit{SoPhie}; an
interpretable framework based on Generative Adversarial Network (GAN), which
leverages two sources of information, the path history of all the agents in a
scene, and the scene context information, using images of the scene. To predict
a future path for an agent, both physical and social information must be
leveraged. Previous work has not been successful to jointly model physical and
social interactions. Our approach blends a social attention mechanism with a
physical attention that helps the model to learn where to look in a large scene
and extract the most salient parts of the image relevant to the path. Whereas,
the social attention component aggregates information across the different
agent interactions and extracts the most important trajectory information from
the surrounding neighbors. SoPhie also takes advantage of GAN to generates more
realistic samples and to capture the uncertain nature of the future paths by
modeling its distribution. All these mechanisms enable our approach to predict
socially and physically plausible paths for the agents and to achieve
state-of-the-art performance on several different trajectory forecasting
benchmarks.

positive:
The ability to predict the future movements of other vehicles is a
subconscious and effortless skill for humans and key to safe autonomous
driving. Therefore, trajectory prediction for autonomous cars has gained a lot
of attention in recent years. It is, however, still a hard task to achieve
human-level performance. Interdependencies between vehicle behaviors and the
multimodal nature of future intentions in a dynamic and complex driving
environment render trajectory prediction a challenging problem. In this work,
we propose a new, data-driven approach for predicting the motion of vehicles in
a road environment. The model allows for inferring future intentions from the
past interaction among vehicles in highway driving scenarios. Using our
neighborhood-based data representation, the proposed system jointly exploits
correlations in the spatial and temporal domain using convolutional neural
networks. Our system considers multiple possible maneuver intentions and their
corresponding motion and predicts the trajectory for five seconds into the
future. We implemented our approach and evaluated it on two highway datasets
taken in different countries and are able to achieve a competitive prediction
performance.

positive:
Highway driving invariably combines high speeds with the need to interact
closely with other drivers. Prediction methods enable autonomous vehicles (AVs)
to anticipate drivers' future trajectories and plan accordingly. Kinematic
methods for prediction have traditionally ignored the presence of other
drivers, or made predictions only for a limited set of scenarios. Data-driven
approaches fill this gap by learning from large datasets to predict
trajectories in general scenarios. While they achieve high accuracy, they also
lose the interpretability and tools for model validation enjoyed by kinematic
methods. This letter proposes a novel kinematic model to describe car-following
and lane change behavior, and extends it to predict trajectories in general
scenarios. Experiments on highway datasets under varied sensing conditions
demonstrate that the proposed method outperforms state-of-the-art methods.

positive:
Accurately forecasting the future movements of surrounding vehicles is
essential for safe and efficient operations of autonomous driving cars. This
task is difficult because a vehicle's moving trajectory is greatly determined
by its driver's intention, which is often hard to estimate. By leveraging
attention mechanisms along with long short-term memory (LSTM) networks, this
work learns the relation between a driver's intention and the vehicle's
changing positions relative to road infrastructures, and uses it to guide the
prediction. Different from other state-of-the-art solutions, our work treats
the on-road lanes as non-Euclidean structures, unfolds the vehicle's moving
history to form a spatio-temporal graph, and uses methods from Graph Neural
Networks to solve the problem. Not only is our approach a pioneering attempt in
using non-Euclidean methods to process static environmental features around a
predicted object, our model also outperforms other state-of-the-art models in
several metrics. The practicability and interpretability analysis of the model
shows great potential for large-scale deployment in various autonomous driving
systems in addition to our own.

positive:
Path planning in dynamic environments is essential to high-risk applications
such as unmanned aerial vehicles, self-driving cars, and autonomous underwater
vehicles. In this paper, we generate collision-free trajectories for a robot
within any given environment with temporal and spatial uncertainties caused due
to randomly moving obstacles. We use two Poisson distributions to model the
movements of obstacles across the generated trajectory of a robot in both space
and time to determine the probability of collision with an obstacle. Measures
are taken to avoid an obstacle by intelligently manipulating the speed of the
robot at space-time intervals where a larger number of obstacles intersect the
trajectory of the robot. Our method potentially reduces the use of
computationally expensive collision detection libraries. Based on our
experiments, there has been a significant improvement over existing methods in
terms of safety, accuracy, execution time and computational cost. Our results
show a high level of accuracy between the predicted and actual number of
collisions with moving obstacles.

positive:
An efficient path planner for autonomous car-like vehicles should handle the
strong kinematic constraints, particularly in confined spaces commonly
encountered while maneuvering in city traffic, and should enable rapid
planning, as the city traffic scenarios are highly dynamic. State-of-the-art
planning algorithms handle such difficult cases at high computational cost,
often yielding non-deterministic results. However, feasible local paths can be
quickly generated leveraging the past planning experience gained in the same or
similar environment. While learning through supervised training is problematic
for real traffic scenarios, we introduce in this paper a novel neural
network-based method for path planning, which employs a gradient-based
self-supervised learning algorithm to predict feasible paths. This approach
strongly exploits the experience gained in the past and rapidly yields feasible
maneuver plans for car-like vehicles with limited steering-angle. The
effectiveness of such an approach has been confirmed by computational
experiments.

positive:
We address multi-modal trajectory forecasting of agents in unknown scenes by
formulating it as a planning problem. We present an approach consisting of
three models; a goal prediction model to identify potential goals of the agent,
an inverse reinforcement learning model to plan optimal paths to each goal, and
a trajectory generator to obtain future trajectories along the planned paths.
Analysis of predictions on the Stanford drone dataset, shows generalizability
of our approach to novel scenes.

positive:
In this paper, we present Goal-GAN, an interpretable and end-to-end trainable
model for human trajectory prediction. Inspired by human navigation, we model
the task of trajectory prediction as an intuitive two-stage process: (i) goal
estimation, which predicts the most likely target positions of the agent,
followed by a (ii) routing module which estimates a set of plausible
trajectories that route towards the estimated goal. We leverage information
about the past trajectory and visual context of the scene to estimate a
multi-modal probability distribution over the possible goal positions, which is
used to sample a potential goal during the inference. The routing is governed
by a recurrent neural network that reacts to physical constraints in the nearby
surroundings and generates feasible paths that route towards the sampled goal.
Our extensive experimental evaluation shows that our method establishes a new
state-of-the-art on several benchmarks while being able to generate a realistic
and diverse set of trajectories that conform to physical constraints.

positive:
It is critical to predict the motion of surrounding vehicles for self-driving
planning, especially in a socially compliant and flexible way. However, future
prediction is challenging due to the interaction and uncertainty in driving
behaviors. We propose planning-informed trajectory prediction (PiP) to tackle
the prediction problem in the multi-agent setting. Our approach is
differentiated from the traditional manner of prediction, which is only based
on historical information and decoupled with planning. By informing the
prediction process with the planning of ego vehicle, our method achieves the
state-of-the-art performance of multi-agent forecasting on highway datasets.
Moreover, our approach enables a novel pipeline which couples the prediction
and planning, by conditioning PiP on multiple candidate trajectories of the ego
vehicle, which is highly beneficial for autonomous driving in interactive
scenarios.

positive:
Vision-based navigation of autonomous vehicles primarily depends on the Deep
Neural Network (DNN) based systems in which the controller obtains input from
sensors/detectors, such as cameras and produces a vehicle control output, such
as a steering wheel angle to navigate the vehicle safely in a roadway traffic
environment. Typically, these DNN-based systems of the autonomous vehicle are
trained through supervised learning; however, recent studies show that a
trained DNN-based system can be compromised by perturbation or adversarial
inputs. Similarly, this perturbation can be introduced into the DNN-based
systems of autonomous vehicle by unexpected roadway hazards, such as debris and
roadblocks. In this study, we first introduce a roadway hazardous environment
(both intentional and unintentional roadway hazards) that can compromise the
DNN-based navigational system of an autonomous vehicle, and produces an
incorrect steering wheel angle, which can cause crashes resulting in fatality
and injury. Then, we develop a DNN-based autonomous vehicle driving system
using object detection and semantic segmentation to mitigate the adverse effect
of this type of hazardous environment, which helps the autonomous vehicle to
navigate safely around such hazards. We find that our developed DNN-based
autonomous vehicle driving system including hazardous object detection and
semantic segmentation improves the navigational ability of an autonomous
vehicle to avoid a potential hazard by 21% compared to the traditional
DNN-based autonomous vehicle driving system.

positive:
When driving, people make decisions based on current traffic as well as their
desired route. They have a mental map of known routes and are often able to
navigate without needing directions. Current self-driving models improve their
performances when using additional GPS information. Here we aim to push forward
self-driving research and perform route planning even in the absence of GPS.
Our system learns to predict in real-time vehicle's current location and future
trajectory, as a function of time, on a known map, given only the raw video
stream and the intended destination. The GPS signal is available only at
training time, with training data annotation being fully automatic. Different
from other published models, we predict the vehicle's trajectory for up to
seven seconds ahead, from which complete steering, speed and acceleration
information can be derived for the entire time span. Trajectories capture
navigational information on multiple levels, from instant steering commands
that depend on present traffic and obstacles ahead, to longer-term navigation
decisions, towards a specific destination. We collect our dataset with a
regular car and a smartphone that records video and GPS streams. The GPS data
is used to derive ground-truth supervision labels and create an analytical
representation of the traversed map. In tests, our system outperforms published
methods on visual localization and steering and gives accurate navigation
assistance between any two known locations.

negative:
Path prediction is a fundamental task for estimating how pedestrians or
vehicles are going to move in a scene. Because path prediction as a task of
computer vision uses video as input, various information used for prediction,
such as the environment surrounding the target and the internal state of the
target, need to be estimated from the video in addition to predicting paths.
Many prediction approaches that include understanding the environment and the
internal state have been proposed. In this survey, we systematically summarize
methods of path prediction that take video as input and and extract features
from the video. Moreover, we introduce datasets used to evaluate path
prediction methods quantitatively.

negative:
We present a new method for multi-modal, long-term vehicle trajectory
prediction. Our approach relies on using lane centerlines captured in rich maps
of the environment to generate a set of proposed goal paths for each vehicle.
Using these paths -- which are generated at run time and therefore dynamically
adapt to the scene -- as spatial anchors, we predict a set of goal-based
trajectories along with a categorical distribution over the goals. This
approach allows us to directly model the goal-directed behavior of traffic
actors, which unlocks the potential for more accurate long-term prediction. Our
experimental results on both a large-scale internal driving dataset and on the
public nuScenes dataset show that our model outperforms state-of-the-art
approaches for vehicle trajectory prediction over a 6-second horizon. We also
empirically demonstrate that our model is better able to generalize to road
scenes from a completely new city than existing methods.

negative:
Urban autonomous driving decision making is challenging due to complex road
geometry and multi-agent interactions. Current decision making methods are
mostly manually designing the driving policy, which might result in sub-optimal
solutions and is expensive to develop, generalize and maintain at scale. On the
other hand, with reinforcement learning (RL), a policy can be learned and
improved automatically without any manual designs. However, current RL methods
generally do not work well on complex urban scenarios. In this paper, we
propose a framework to enable model-free deep reinforcement learning in
challenging urban autonomous driving scenarios. We design a specific input
representation and use visual encoding to capture the low-dimensional latent
states. Several state-of-the-art model-free deep RL algorithms are implemented
into our framework, with several tricks to improve their performance. We
evaluate our method in a challenging roundabout task with dense surrounding
vehicles in a high-definition driving simulator. The result shows that our
method can solve the task well and is significantly better than the baseline.

negative:
Trajectory prediction is a fundamental and challenging task for numerous
applications, such as autonomous driving and intelligent robots. Currently,
most of existing work treat the pedestrian trajectory as a series of fixed
two-dimensional coordinates. However, in real scenarios, the trajectory often
exhibits randomness, and has its own probability distribution. Inspired by this
observed fact, also considering other movement characteristics of pedestrians,
we propose one simple and intuitive movement description, probability
trajectory, which maps the coordinate points of pedestrian trajectory into
two-dimensional Gaussian distribution in images. Based on this unique
description, we develop one novel trajectory prediction method, called social
probability. The method combines the new probability trajectory and powerful
convolution recurrent neural networks together. Both the input and output of
our method are probability trajectories, which provide the recurrent neural
network with sufficient spatial and random information of moving pedestrians.
And the social probability extracts spatio-temporal features directly on the
new movement description to generate robust and accurate predicted results. The
experiments on public benchmark datasets show the effectiveness of the proposed
method.

negative:
Progress towards advanced systems for assisted and autonomous driving is
leveraging recent advances in recognition and segmentation methods. Yet, we are
still facing challenges in bringing reliable driving to inner cities, as those
are composed of highly dynamic scenes observed from a moving platform at
considerable speeds. Anticipation becomes a key element in order to react
timely and prevent accidents. In this paper we argue that it is necessary to
predict at least 1 second and we thus propose a new model that jointly predicts
ego motion and people trajectories over such large time horizons. We pay
particular attention to modeling the uncertainty of our estimates arising from
the non-deterministic nature of natural traffic scenes. Our experimental
results show that it is indeed possible to predict people trajectories at the
desired time horizons and that our uncertainty estimates are informative of the
prediction error. We also show that both sequence modeling of trajectories as
well as our novel method of long term odometry prediction are essential for
best performance.

negative:
In this paper, we propose an efficient vehicle trajectory prediction
framework based on recurrent neural network. Basically, the characteristic of
the vehicle's trajectory is different from that of regular moving objects since
it is affected by various latent factors including road structure, traffic
rules, and driver's intention. Previous state of the art approaches use
sophisticated vehicle behavior model describing these factors and derive the
complex trajectory prediction algorithm, which requires a system designer to
conduct intensive model optimization for practical use. Our approach is
data-driven and simple to use in that it learns complex behavior of the
vehicles from the massive amount of trajectory data through deep neural network
model. The proposed trajectory prediction method employs the recurrent neural
network called long short-term memory (LSTM) to analyze the temporal behavior
and predict the future coordinate of the surrounding vehicles. The proposed
scheme feeds the sequence of vehicles' coordinates obtained from sensor
measurements to the LSTM and produces the probabilistic information on the
future location of the vehicles over occupancy grid map. The experiments
conducted using the data collected from highway driving show that the proposed
method can produce reasonably good estimate of future trajectory.

negative:
We present a weakly-supervised approach to segmenting proposed drivable paths
in images with the goal of autonomous driving in complex urban environments.
Using recorded routes from a data collection vehicle, our proposed method
generates vast quantities of labelled images containing proposed paths and
obstacles without requiring manual annotation, which we then use to train a
deep semantic segmentation network. With the trained network we can segment
proposed paths and obstacles at run-time using a vehicle equipped with only a
monocular camera without relying on explicit modelling of road or lane
markings. We evaluate our method on the large-scale KITTI and Oxford RobotCar
datasets and demonstrate reliable path proposal and obstacle segmentation in a
wide variety of environments under a range of lighting, weather and traffic
conditions. We illustrate how the method can generalise to multiple path
proposals at intersections and outline plans to incorporate the system into a
framework for autonomous urban driving.

negative:
Predicting future behavior of other traffic participants is an essential task
that needs to be solved by automated vehicles and human drivers alike to
achieve safe and situationaware driving. Modern approaches to vehicles
trajectory prediction typically rely on data-driven models like neural
networks, in particular LSTMs (Long Short-Term Memorys), achieving promising
results. However, the question of optimal composition of the underlying
training data has received less attention. In this paper, we expand on previous
work on vehicle trajectory prediction based on neural network models employing
distributed representations to encode automotive scenes in a semantic vector
substrate. We analyze the influence of variations in the training data on the
performance of our prediction models. Thereby, we show that the models
employing our semantic vector representation outperform the numerical model
when trained on an adequate data set and thereby, that the composition of
training data in vehicle trajectory prediction is crucial for successful
training. We conduct our analysis on challenging real-world driving data.

negative:
Motion Planning, as a fundamental technology of automatic navigation for the
autonomous vehicle, is still an open challenging issue in the real-life traffic
situation and is mostly applied by the model-based approaches. However, due to
the complexity of the traffic situations and the uncertainty of the edge cases,
it is hard to devise a general motion planning system for the autonomous
vehicle. In this paper, we proposed a motion planning model based on deep
learning (named as spatiotemporal LSTM network), which is able to generate a
real-time reflection based on spatiotemporal information extraction. To be
specific, the model based on spatiotemporal LSTM network has three main
structure. Firstly, the Convolutional Long-short Term Memory (Conv-LSTM) is
used to extract hidden features through sequential image data. Then, the 3D
Convolutional Neural Network(3D-CNN) is applied to extract the spatiotemporal
information from the multi-frame feature information. Finally, the fully
connected neural networks are used to construct a control model for autonomous
vehicle steering angle. The experiments demonstrated that the proposed method
can generate a robust and accurate visual motion planning results for the
autonomous vehicle.

negative:
Understanding and predicting the intention of pedestrians is essential to
enable autonomous vehicles and mobile robots to navigate crowds. This problem
becomes increasingly complex when we consider the uncertainty and multimodality
of pedestrian motion, as well as the implicit interactions between members of a
crowd, including any response to a vehicle. Our approach, Probabilistic Crowd
GAN, extends recent work in trajectory prediction, combining Recurrent Neural
Networks (RNNs) with Mixture Density Networks (MDNs) to output probabilistic
multimodal predictions, from which likely modal paths are found and used for
adversarial training. We also propose the use of Graph Vehicle-Pedestrian
Attention Network (GVAT), which models social interactions and allows input of
a shared vehicle feature, showing that inclusion of this module leads to
improved trajectory prediction both with and without the presence of a vehicle.
Through evaluation on various datasets, we demonstrate improvements on the
existing state of the art methods for trajectory prediction and illustrate how
the true multimodal and uncertain nature of crowd interactions can be directly
modelled.

negative:
Trajectory prediction is an important task, especially in autonomous driving.
The ability to forecast the position of other moving agents can yield to an
effective planning, ensuring safety for the autonomous vehicle as well for the
observed entities. In this work we propose a data driven approach based on
Markov Chains to generate synthetic trajectories, which are useful for training
a multiple future trajectory predictor. The advantages are twofold: on the one
hand synthetic samples can be used to augment existing datasets and train more
effective predictors; on the other hand, it allows to generate samples with
multiple ground truths, corresponding to diverse equally likely outcomes of the
observed trajectory. We define a trajectory prediction model and a loss that
explicitly address the multimodality of the problem and we show that combining
synthetic and real data leads to prediction improvements, obtaining state of
the art results.

negative:
The vision of automated driving is to increase both road safety and
efficiency, while offering passengers a convenient travel experience. This
requires that autonomous systems correctly estimate the current traffic scene
and its likely evolution. In highway scenarios early recognition of cut-in
maneuvers is essential for risk-aware maneuver planning. In this paper, a
statistical approach is proposed, which advantageously utilizes a set of
prototypical lane change trajectories to realize both early maneuver detection
and uncertainty-aware trajectory prediction for traffic participants.
Generation of prototype trajectories from real traffic data is accomplished by
Agglomerative Hierarchical Clustering. During clustering, the alignment of the
cluster prototypes to each other is optimized and the cohesion of the resulting
prototype is limited when two clusters merge. In the prediction stage, the
similarity of observed vehicle motion and typical lane change patterns in the
data base is evaluated to construct a set of significant features for maneuver
classification via Boosted Decision Trees. The future trajectory is predicted
combining typical lane change realizations in a mixture model. B-splines based
trajectory adaptations guarantee continuity during transition from actually
observed to predicted vehicle states. Quantitative evaluation results
demonstrate the proposed concept's improved performance for both maneuver and
trajectory prediction compared to a previously implemented reference approach.

negative:
As autonomous driving systems mature, motion forecasting has received
increasing attention as a critical requirement for planning. Of particular
importance are interactive situations such as merges, unprotected turns, etc.,
where predicting individual object motion is not sufficient. Joint predictions
of multiple objects are required for effective route planning. There has been a
critical need for high-quality motion data that is rich in both interactions
and annotation to develop motion planning models. In this work, we introduce
the most diverse interactive motion dataset to our knowledge, and provide
specific labels for interacting objects suitable for developing joint
prediction models. With over 100,000 scenes, each 20 seconds long at 10 Hz, our
new dataset contains more than 570 hours of unique data over 1750 km of
roadways. It was collected by mining for interesting interactions between
vehicles, pedestrians, and cyclists across six cities within the United States.
We use a high-accuracy 3D auto-labeling system to generate high quality 3D
bounding boxes for each road agent, and provide corresponding high definition
3D maps for each scene. Furthermore, we introduce a new set of metrics that
provides a comprehensive evaluation of both single agent and joint agent
interaction motion forecasting models. Finally, we provide strong baseline
models for individual-agent prediction and joint-prediction. We hope that this
new large-scale interactive motion dataset will provide new opportunities for
advancing motion forecasting models.

negative:
Autonomous driving consists of a multitude of interacting modules, where each
module must contend with errors from the others. Typically, the motion
prediction module depends upon a robust tracking system to capture each agent's
past movement. In this work, we systematically explore the importance of the
tracking module for the motion prediction task and ultimately conclude that the
overall motion prediction performance is highly sensitive to the tracking
module's imperfections. We explicitly compare models that use tracking
information to models that do not across multiple scenarios and conditions. We
find that the tracking information plays an essential role and improves motion
prediction performance in noise-free conditions. However, in the presence of
tracking noise, it can potentially affect the overall performance if not
studied thoroughly. We thus argue practitioners should be mindful of noise when
developing and testing motion/tracking modules, or that they should consider
tracking free alternatives.

negative:
Accurate pedestrian trajectory prediction is of great importance for
downstream tasks such as autonomous driving and mobile robot navigation. Fully
investigating the social interactions within the crowd is crucial for accurate
pedestrian trajectory prediction. However, most existing methods do not capture
group level interactions well, focusing only on pairwise interactions and
neglecting group-wise interactions. In this work, we propose a hierarchical
graph convolutional network, HGCN-GJS, for trajectory prediction which well
leverages group level interactions within the crowd. Furthermore, we introduce
a novel joint sampling scheme for modeling the joint distribution of multiple
pedestrians in the future trajectories. Based on the group information, this
scheme associates the trajectory of one person with the trajectory of other
people in the group, but maintains the independence of the trajectories of
outsiders. We demonstrate the performance of our network on several trajectory
prediction datasets, achieving state-of-the-art results on all datasets
considered.

negative:
This paper presents a method for future localization: to predict a set of
plausible trajectories of ego-motion given a depth image. We predict paths
avoiding obstacles, between objects, even paths turning around a corner into
space behind objects. As a byproduct of the predicted trajectories of
ego-motion, we discover in the image the empty space occluded by foreground
objects. We use no image based features such as semantic labeling/segmentation
or object detection/recognition for this algorithm. Inspired by proxemics, we
represent the space around a person using an EgoSpace map, akin to an
illustrated tourist map, that measures a likelihood of occlusion at the
egocentric coordinate system. A future trajectory of ego-motion is modeled by a
linear combination of compact trajectory bases allowing us to constrain the
predicted trajectory. We learn the relationship between the EgoSpace map and
trajectory from the EgoMotion dataset providing in-situ measurements of the
future trajectory. A cost function that takes into account partial occlusion
due to foreground objects is minimized to predict a trajectory. This cost
function generates a trajectory that passes through the occluded space, which
allows us to discover the empty space behind the foreground objects. We
quantitatively evaluate our method to show predictive validity and apply to
various real world scenes including walking, shopping, and social interactions.

negative:
To plan a safe and efficient route, an autonomous vehicle should anticipate
future motions of other agents around it. Motion prediction is an extremely
challenging task which recently gained significant attention of the research
community. In this work, we present a simple and yet strong baseline for
uncertainty aware motion prediction based purely on transformer neural
networks, which has shown its effectiveness in conditions of domain change.
While being easy-to-implement, the proposed approach achieves competitive
performance and ranks 1$^{st}$ on the 2021 Shifts Vehicle Motion Prediction
Competition.

negative:
Deep learning has enabled impressive progress in the accuracy of semantic
segmentation. Yet, the ability to estimate uncertainty and detect failure is
key for safety-critical applications like autonomous driving. Existing
uncertainty estimates have mostly been evaluated on simple tasks, and it is
unclear whether these methods generalize to more complex scenarios. We present
Fishyscapes, the first public benchmark for uncertainty estimation in a
real-world task of semantic segmentation for urban driving. It evaluates
pixel-wise uncertainty estimates towards the detection of anomalous objects in
front of the vehicle. We~adapt state-of-the-art methods to recent semantic
segmentation models and compare approaches based on softmax confidence,
Bayesian learning, and embedding density. Our results show that anomaly
detection is far from solved even for ordinary situations, while our benchmark
allows measuring advancements beyond the state-of-the-art.

negative:
Pedestrian trajectory prediction is a critical yet challenging task,
especially for crowded scenes. We suggest that introducing an attention
mechanism to infer the importance of different neighbors is critical for
accurate trajectory prediction in scenes with varying crowd size. In this work,
we propose a novel method, AVGCN, for trajectory prediction utilizing graph
convolutional networks (GCN) based on human attention (A denotes attention, V
denotes visual field constraints). First, we train an attention network that
estimates the importance of neighboring pedestrians, using gaze data collected
as subjects perform a bird's eye view crowd navigation task. Then, we
incorporate the learned attention weights modulated by constraints on the
pedestrian's visual field into a trajectory prediction network that uses a GCN
to aggregate information from neighbors efficiently. AVGCN also considers the
stochastic nature of pedestrian trajectories by taking advantage of variational
trajectory prediction. Our approach achieves state-of-the-art performance on
several trajectory prediction benchmarks, and the lowest average prediction
error over all considered benchmarks.

negative:
End-to-end approaches to autonomous driving have high sample complexity and
are difficult to scale to realistic urban driving. Simulation can help
end-to-end driving systems by providing a cheap, safe, and diverse training
environment. Yet training driving policies in simulation brings up the problem
of transferring such policies to the real world. We present an approach to
transferring driving policies from simulation to reality via modularity and
abstraction. Our approach is inspired by classic driving systems and aims to
combine the benefits of modular architectures and end-to-end deep learning
approaches. The key idea is to encapsulate the driving policy such that it is
not directly exposed to raw perceptual input or low-level vehicle dynamics. We
evaluate the presented approach in simulated urban environments and in the real
world. In particular, we transfer a driving policy trained in simulation to a
1/5-scale robotic truck that is deployed in a variety of conditions, with no
finetuning, on two continents. The supplementary video can be viewed at
https://youtu.be/BrMDJqI6H5U

negative:
Autonomous vehicles rely heavily upon their perception subsystems to see the
environment in which they operate. Unfortunately, the effect of variable
weather conditions presents a significant challenge to object detection
algorithms, and thus it is imperative to test the vehicle extensively in all
conditions which it may experience. However, development of robust autonomous
vehicle subsystems requires repeatable, controlled testing - while real weather
is unpredictable and cannot be scheduled. Real-world testing in adverse
conditions is an expensive and time-consuming task, often requiring access to
specialist facilities. Simulation is commonly relied upon as a substitute, with
increasingly visually realistic representations of the real-world being
developed. In the context of the complete autonomous vehicle control pipeline,
subsystems downstream of perception need to be tested with accurate recreations
of the perception system output, rather than focusing on subjective visual
realism of the input - whether in simulation or the real world. This study
develops the untapped potential of a lightweight weather augmentation method in
an autonomous racing vehicle - focusing not on visual accuracy, but rather the
effect upon perception subsystem performance in real time. With minimal
adjustment, the prototype developed in this study can replicate the effects of
water droplets on the camera lens, and fading light conditions. This approach
introduces a latency of less than 8 ms using compute hardware well suited to
being carried in the vehicle - rendering it ideal for real-time implementation
that can be run during experiments in simulation, and augmented reality testing
in the real world.

negative:
Vision-based prediction algorithms have a wide range of applications
including autonomous driving, surveillance, human-robot interaction, weather
prediction. The objective of this paper is to provide an overview of the field
in the past five years with a particular focus on deep learning approaches. For
this purpose, we categorize these algorithms into video prediction, action
prediction, trajectory prediction, body motion prediction, and other prediction
applications. For each category, we highlight the common architectures,
training methods and types of data used. In addition, we discuss the common
evaluation metrics and datasets used for vision-based prediction tasks. A
database of all the information presented in this survey including,
cross-referenced according to papers, datasets and metrics, can be found online
at https://github.com/aras62/vision-based-prediction.

negative:
Motion prediction of vehicles is critical but challenging due to the
uncertainties in complex environments and the limited visibility caused by
occlusions and limited sensor ranges. In this paper, we study a new task,
safety-aware motion prediction with unseen vehicles for autonomous driving.
Unlike the existing trajectory prediction task for seen vehicles, we aim at
predicting an occupancy map that indicates the earliest time when each location
can be occupied by either seen and unseen vehicles. The ability to predict
unseen vehicles is critical for safety in autonomous driving. To tackle this
challenging task, we propose a safety-aware deep learning model with three new
loss functions to predict the earliest occupancy map. Experiments on the
large-scale autonomous driving nuScenes dataset show that our proposed model
significantly outperforms the state-of-the-art baselines on the safety-aware
motion prediction task. To the best of our knowledge, our approach is the first
one that can predict the existence of unseen vehicles in most cases. Project
page at {\url{https://github.com/xrenaa/Safety-Aware-Motion-Prediction}}.

negative:
This paper is focused on the task of searching for a specific vehicle that
appeared in the surveillance networks. Existing methods usually assume the
vehicle images are well cropped from the surveillance videos, then use visual
attributes, like colors and types, or license plate numbers to match the target
vehicle in the image set. However, a complete vehicle search system should
consider the problems of vehicle detection, representation, indexing, storage,
matching, and so on. Besides, attribute-based search cannot accurately find the
same vehicle due to intra-instance changes in different cameras and the
extremely uncertain environment. Moreover, the license plates may be
misrecognized in surveillance scenes due to the low resolution and noise. In
this paper, a Progressive Vehicle Search System, named as PVSS, is designed to
solve the above problems. PVSS is constituted of three modules: the crawler,
the indexer, and the searcher. The vehicle crawler aims to detect and track
vehicles in surveillance videos and transfer the captured vehicle images,
metadata and contextual information to the server or cloud. Then multi-grained
attributes, such as the visual features and license plate fingerprints, are
extracted and indexed by the vehicle indexer. At last, a query triplet with an
input vehicle image, the time range, and the spatial scope is taken as the
input by the vehicle searcher. The target vehicle will be searched in the
database by a progressive process. Extensive experiments on the public dataset
from a real surveillance network validate the effectiveness of the PVSS.

negative:
Deciphering human behaviors to predict their future paths/trajectories and
what they would do from videos is important in many applications. Motivated by
this idea, this paper studies predicting a pedestrian's future path jointly
with future activities. We propose an end-to-end, multi-task learning system
utilizing rich visual features about human behavioral information and
interaction with their surroundings. To facilitate the training, the network is
learned with an auxiliary task of predicting future location in which the
activity will happen. Experimental results demonstrate our state-of-the-art
performance over two public benchmarks on future trajectory prediction.
Moreover, our method is able to produce meaningful future activity prediction
in addition to the path. The result provides the first empirical evidence that
joint modeling of paths and activities benefits future path prediction.

negative:
Most of researchers use the vehicle re-identification based on
classification. This always requires an update with the new vehicle models in
the market. In this paper, two types of vehicle re-identification will be
presented. First, the standard method, which needs an image from the search
vehicle. VRIC and VehicleID data set are suitable for training this module. It
will be explained in detail how to improve the performance of this method using
a trained network, which is designed for the classification. The second method
takes as input a representative image of the search vehicle with similar
make/model, released year and colour. It is very useful when an image from the
search vehicle is not available. It produces as output a shape and a colour
features. This could be used by the matching across a database to re-identify
vehicles, which look similar to the search vehicle. To get a robust module for
the re-identification, a fine-grained classification has been trained, which
its class consists of four elements: the make of a vehicle refers to the
vehicle's manufacturer, e.g. Mercedes-Benz, the model of a vehicle refers to
type of model within that manufacturer's portfolio, e.g. C Class, the year
refers to the iteration of the model, which may receive progressive alterations
and upgrades by its manufacturer and the perspective of the vehicle. Thus, all
four elements describe the vehicle at increasing degree of specificity. The aim
of the vehicle shape classification is to classify the combination of these
four elements. The colour classification has been separately trained. The
results of vehicle re-identification will be shown. Using a developed tool, the
re-identification of vehicles on video images and on controlled data set will
be demonstrated. This work was partially funded under the grant.

negative:
Varying weather conditions, including rainfall and snowfall, are generally
regarded as a challenge for computer vision algorithms. One proposed solution
to the challenges induced by rain and snowfall is to artificially remove the
rain from images or video using rain removal algorithms. It is the promise of
these algorithms that the rain-removed image frames will improve the
performance of subsequent segmentation and tracking algorithms. However, rain
removal algorithms are typically evaluated on their ability to remove synthetic
rain on a small subset of images. Currently, their behavior is unknown on
real-world videos when integrated with a typical computer vision pipeline. In
this paper, we review the existing rain removal algorithms and propose a new
dataset that consists of 22 traffic surveillance sequences under a broad
variety of weather conditions that all include either rain or snowfall. We
propose a new evaluation protocol that evaluates the rain removal algorithms on
their ability to improve the performance of subsequent segmentation, instance
segmentation, and feature tracking algorithms under rain and snow. If
successful, the de-rained frames of a rain removal algorithm should improve
segmentation performance and increase the number of accurately tracked
features. The results show that a recent single-frame-based rain removal
algorithm increases the segmentation performance by 19.7% on our proposed
dataset, but it eventually decreases the feature tracking performance and
showed mixed results with recent instance segmentation methods. However, the
best video-based rain removal algorithm improves the feature tracking accuracy
by 7.72%.

negative:
We introduce and show preliminary results of a fast randomized method that
finds a set of K paths lying in distinct homotopy classes. We frame the path
planning task as a graph search problem, where the navigation graph is based on
a Voronoi diagram. The search is biased by a cost function derived from the
social force model that is used to generate and select the paths. We compare
our method to Yen's algorithm, and empirically show that our approach is faster
to find a subset of homotopy classes. Furthermore our approach computes a set
of more diverse paths with respect to the baseline while obtaining a negligible
loss in path quality.

negative:
Despite significant progress on current state-of-the-art image generation
models, synthesis of document images containing multiple and complex object
layouts is a challenging task. This paper presents a novel approach, called
DocSynth, to automatically synthesize document images based on a given layout.
In this work, given a spatial layout (bounding boxes with object categories) as
a reference by the user, our proposed DocSynth model learns to generate a set
of realistic document images consistent with the defined layout. Also, this
framework has been adapted to this work as a superior baseline model for
creating synthetic document image datasets for augmenting real data during
training for document layout analysis tasks. Different sets of learning
objectives have been also used to improve the model performance.
Quantitatively, we also compare the generated results of our model with real
data using standard evaluation metrics. The results highlight that our model
can successfully generate realistic and diverse document images with multiple
objects. We also present a comprehensive qualitative analysis summary of the
different scopes of synthetic image generation tasks. Lastly, to our knowledge
this is the first work of its kind.

negative:
Autonomous driving system development is critically dependent on the ability
to replay complex and diverse traffic scenarios in simulation. In such
scenarios, the ability to accurately simulate the vehicle sensors such as
cameras, lidar or radar is essential. However, current sensor simulators
leverage gaming engines such as Unreal or Unity, requiring manual creation of
environments, objects and material properties. Such approaches have limited
scalability and fail to produce realistic approximations of camera, lidar, and
radar data without significant additional work.
  In this paper, we present a simple yet effective approach to generate
realistic scenario sensor data, based only on a limited amount of lidar and
camera data collected by an autonomous vehicle. Our approach uses
texture-mapped surfels to efficiently reconstruct the scene from an initial
vehicle pass or set of passes, preserving rich information about object 3D
geometry and appearance, as well as the scene conditions. We then leverage a
SurfelGAN network to reconstruct realistic camera images for novel positions
and orientations of the self-driving vehicle and moving objects in the scene.
We demonstrate our approach on the Waymo Open Dataset and show that it can
synthesize realistic camera data for simulated scenarios. We also create a
novel dataset that contains cases in which two self-driving vehicles observe
the same scene at the same time. We use this dataset to provide additional
evaluation and demonstrate the usefulness of our SurfelGAN model.

negative:
Occlusion is a long-standing problem that causes many modern tracking methods
to be erroneous. In this paper, we address the occlusion problem by exploiting
the current and future possible locations of the target object from its past
trajectory. To achieve this, we introduce a learning-based tracking method that
takes into account background motion modeling and trajectory prediction. Our
trajectory prediction module predicts the target object's locations in the
current and future frames based on the object's past trajectory. Since, in the
input video, the target object's trajectory is not only affected by the object
motion but also the camera motion, our background motion module estimates the
camera motion. So that the object's trajectory can be made independent from it.
To dynamically switch between the appearance-based tracker and the trajectory
prediction, we employ a network that can assess how good a tracking prediction
is, and we use the assessment scores to choose between the appearance-based
tracker's prediction and the trajectory-based prediction. Comprehensive
evaluations show that the proposed method sets a new state-of-the-art
performance on commonly used tracking benchmarks.

negative:
Driver attention prediction is currently becoming the focus in safe driving
research community, such as the DR(eye)VE project and newly emerged Berkeley
DeepDrive Attention (BDD-A) database in critical situations. In safe driving,
an essential task is to predict the incoming accidents as early as possible.
BDD-A was aware of this problem and collected the driver attention in
laboratory because of the rarity of such scenes. Nevertheless, BDD-A focuses
the critical situations which do not encounter actual accidents, and just faces
the driver attention prediction task, without a close step for accident
prediction. In contrast to this, we explore the view of drivers' eyes for
capturing multiple kinds of accidents, and construct a more diverse and larger
video benchmark than ever before with the driver attention and the driving
accident annotation simultaneously (named as DADA-2000), which has 2000 video
clips owning about 658,476 frames on 54 kinds of accidents. These clips are
crowd-sourced and captured in various occasions (highway, urban, rural, and
tunnel), weather (sunny, rainy and snowy) and light conditions (daytime and
nighttime). For the driver attention representation, we collect the maps of
fixations, saccade scan path and focusing time. The accidents are annotated by
their categories, the accident window in clips and spatial locations of the
crash-objects. Based on the analysis, we obtain a quantitative and positive
answer for the question in this paper.

negative:
This document is a document that has written procedures and methods for
collecting objects and unstructured dynamic data on the road for the
development of object recognition technology for self-driving cars, and
outlines the methods of collecting data, annotation data, object classifier
criteria, and data processing methods. On-road object and unstructured dynamic
data were collected in various environments, such as weather, time and traffic
conditions, and additional reception calls for police and safety personnel were
collected. Finally, 100,000 images of various objects existing on pedestrians
and roads, 200,000 images of police and traffic safety personnel, 5,000 images
of police and traffic safety personnel, and data sets consisting of 5,000 image
data were collected and built.

negative:
In this work we propose a novel interpretation of residual networks showing
that they can be seen as a collection of many paths of differing length.
Moreover, residual networks seem to enable very deep networks by leveraging
only the short paths during training. To support this observation, we rewrite
residual networks as an explicit collection of paths. Unlike traditional
models, paths through residual networks vary in length. Further, a lesion study
reveals that these paths show ensemble-like behavior in the sense that they do
not strongly depend on each other. Finally, and most surprising, most paths are
shorter than one might expect, and only the short paths are needed during
training, as longer paths do not contribute any gradient. For example, most of
the gradient in a residual network with 110 layers comes from paths that are
only 10-34 layers deep. Our results reveal one of the key characteristics that
seem to enable the training of very deep networks: Residual networks avoid the
vanishing gradient problem by introducing short paths which can carry gradient
throughout the extent of very deep networks.

negative:
The inference of minimum spanning arborescences within a set of objects is a
general problem which translates into numerous application-specific
unsupervised learning tasks. We introduce a unified and generic structure
called edit arborescence that relies on edit paths between data in a
collection, as well as the Min Edit Arborescence Problem, which asks for an
edit arborescence that minimizes the sum of costs of its inner edit paths.
Through the use of suitable cost functions, this generic framework allows to
model a variety of problems. In particular, we show that by introducing
encoding size preserving edit costs, it can be used as an efficient method for
compressing collections of labeled graphs. Experiments on various graph
datasets, with comparisons to standard compression tools, show the potential of
our method.

negative:
We introduce a novel type of text representation that preserves the 2D layout
of a document. This is achieved by encoding each document page as a
two-dimensional grid of characters. Based on this representation, we present a
generic document understanding pipeline for structured documents. This pipeline
makes use of a fully convolutional encoder-decoder network that predicts a
segmentation mask and bounding boxes. We demonstrate its capabilities on an
information extraction task from invoices and show that it significantly
outperforms approaches based on sequential text or document images.

negative:
In recent years, attention models have been extensively used for person and
vehicle re-identification. Most re-identification methods are designed to focus
attention on key-point locations. However, depending on the orientation, the
contribution of each key-point varies. In this paper, we present a novel
dual-path adaptive attention model for vehicle re-identification (AAVER). The
global appearance path captures macroscopic vehicle features while the
orientation conditioned part appearance path learns to capture localized
discriminative features by focusing attention on the most informative
key-points. Through extensive experimentation, we show that the proposed AAVER
method is able to accurately re-identify vehicles in unconstrained scenarios,
yielding state of the art results on the challenging dataset VeRi-776. As a
byproduct, the proposed system is also able to accurately predict vehicle
key-points and shows an improvement of more than 7% over state of the art. The
code for key-point estimation model is available at
https://github.com/Pirazh/Vehicle_Key_Point_Orientation_Estimation.

negative:
The digital camera captured document images may often be warped and distorted
due to different camera angles or document surfaces. A robust technique is
needed to solve this kind of distortion. The research on dewarping of the
document suffers due to the limited availability of benchmark public dataset.
In recent times, deep learning based approaches are used to solve the problems
accurately. To train most of the deep neural networks a large number of
document images is required and generating such a large volume of document
images manually is difficult. In this paper, we propose a technique to generate
a synthetic warped image from a flat-bedded scanned document image. It is done
by calculating warping factors for each pixel position using two warping
position parameters (WPP) and eight warping control parameters (WCP). These
parameters can be specified as needed depending upon the desired warping. The
results are compared with similar real captured images both qualitative and
quantitative way.

negative:
Methods to quantify the complexity of trajectory datasets are still a missing
piece in benchmarking human trajectory prediction models. In order to gain a
better understanding of the complexity of trajectory prediction tasks and
following the intuition, that more complex datasets contain more information,
an approach for quantifying the amount of information contained in a dataset
from a prototype-based dataset representation is proposed. The dataset
representation is obtained by first employing a non-trivial spatial sequence
alignment, which enables a subsequent learning vector quantization (LVQ) stage.
A large-scale complexity analysis is conducted on several human trajectory
prediction benchmarking datasets, followed by a brief discussion on indications
for human trajectory prediction and benchmarking.

negative:
The demand for on-device document recognition systems increases in
conjunction with the emergence of more strict privacy and security
requirements. In such systems, there is no data transfer from the end device to
a third-party information processing servers. The response time is vital to the
user experience of on-device document recognition. Combined with the
unavailability of discrete GPUs, powerful CPUs, or a large RAM capacity on
consumer-grade end devices such as smartphones, the time limitations put
significant constraints on the computational complexity of the applied
algorithms for on-device execution.
  In this work, we consider document location in an image without prior
knowledge of the document content or its internal structure. In accordance with
the published works, at least 5 systems offer solutions for on-device document
location. All these systems use a location method which can be considered
Hough-based. The precision of such systems seems to be lower than that of the
state-of-the-art solutions which were not designed to account for the limited
computational resources.
  We propose an advanced Hough-based method. In contrast with other approaches,
it accounts for the geometric invariants of the central projection model and
combines both edge and color features for document boundary detection. The
proposed method allowed for the second best result for SmartDoc dataset in
terms of precision, surpassed by U-net like neural network. When evaluated on a
more challenging MIDV-500 dataset, the proposed algorithm guaranteed the best
precision compared to published methods. Our method retained the applicability
to on-device computations.

negative:
This work addresses the problem of discovering, in an unsupervised manner,
interpretable paths in the latent space of pretrained GANs, so as to provide an
intuitive and easy way of controlling the underlying generative factors. In
doing so, it addresses some of the limitations of the state-of-the-art works,
namely, a) that they discover directions that are independent of the latent
code, i.e., paths that are linear, and b) that their evaluation relies either
on visual inspection or on laborious human labeling. More specifically, we
propose to learn non-linear warpings on the latent space, each one parametrized
by a set of RBF-based latent space warping functions, and where each warping
gives rise to a family of non-linear paths via the gradient of the function.
Building on the work of Voynov and Babenko, that discovers linear paths, we
optimize the trainable parameters of the set of RBFs, so as that images that
are generated by codes along different paths, are easily distinguishable by a
discriminator network. This leads to easily distinguishable image
transformations, such as pose and facial expressions in facial images. We show
that linear paths can be derived as a special case of our method, and show
experimentally that non-linear paths in the latent space lead to steeper, more
disentangled and interpretable changes in the image space than in state-of-the
art methods, both qualitatively and quantitatively. We make the code and the
pretrained models publicly available at:
https://github.com/chi0tzp/WarpedGANSpace.

negative:
Weather Recognition plays an important role in our daily lives and many
computer vision applications. However, recognizing the weather conditions from
a single image remains challenging and has not been studied thoroughly.
Generally, most previous works treat weather recognition as a single-label
classification task, namely, determining whether an image belongs to a specific
weather class or not. This treatment is not always appropriate, since more than
one weather conditions may appear simultaneously in a single image. To address
this problem, we make the first attempt to view weather recognition as a
multi-label classification task, i.e., assigning an image more than one labels
according to the displayed weather conditions. Specifically, a CNN-RNN based
multi-label classification approach is proposed in this paper. The
convolutional neural network (CNN) is extended with a channel-wise attention
model to extract the most correlated visual features. The Recurrent Neural
Network (RNN) further processes the features and excavates the dependencies
among weather classes. Finally, the weather labels are predicted step by step.
Besides, we construct two datasets for the weather recognition task and explore
the relationships among different weather conditions. Experimental results
demonstrate the superiority and effectiveness of the proposed approach. The new
constructed datasets will be available at
https://github.com/wzgwzg/Multi-Label-Weather-Recognition.

negative:
In urban theory, urban form is related to social and economic status. This
paper explores to uncover zip-code level income through urban form by analyzing
figure-ground map, a simple, prevailing and precise representation of urban
form in the field of urban study. Deep learning in computer vision enables such
representation maps to be studied at a large scale. We propose to train a DCNN
model to identify and uncover the internal bridge between social class and
urban form. Further, using hand-crafted informative visual features related
with urban form properties (building size, building density, etc.), we apply a
random forest classifier to interpret how morphological properties are related
with social class.

negative:
Training a supernet matters for one-shot neural architecture search (NAS)
methods since it serves as a basic performance estimator for different
architectures (paths). Current methods mainly hold the assumption that a
supernet should give a reasonable ranking over all paths. They thus treat all
paths equally, and spare much effort to train paths. However, it is harsh for a
single supernet to evaluate accurately on such a huge-scale search space (e.g.,
$7^{21}$). In this paper, instead of covering all paths, we ease the burden of
supernet by encouraging it to focus more on evaluation of those
potentially-good ones, which are identified using a surrogate portion of
validation data. Concretely, during training, we propose a multi-path sampling
strategy with rejection, and greedily filter the weak paths. The training
efficiency is thus boosted since the training space has been greedily shrunk
from all paths to those potentially-good ones. Moreover, we further adopt an
exploration and exploitation policy by introducing an empirical candidate path
pool. Our proposed method GreedyNAS is easy-to-follow, and experimental results
on ImageNet dataset indicate that it can achieve better Top-1 accuracy under
same search space and FLOPs or latency level, but with only $\sim$60\% of
supernet training cost. By searching on a larger space, our GreedyNAS can also
obtain new state-of-the-art architectures.

negative:
Nowadays autonomous technologies are a very heavily explored area and
particularly computer vision as the main component of vehicle perception. The
quality of the whole vision system based on neural networks relies on the
dataset it was trained on. It is extremely difficult to find traffic sign
datasets from most of the counties of the world. Meaning autonomous vehicle
from the USA will not be able to drive though Lithuania recognizing all road
signs on the way. In this paper, we propose a solution on how to update model
using a small dataset from the country vehicle will be used in. It is important
to mention that is not panacea, rather small upgrade which can boost autonomous
car development in countries with limited data access. We achieved about 10
percent quality raise and expect even better results during future experiments.

negative:
One of the major prerequisites for any deep learning approach is the
availability of large-scale training data. When dealing with scanned document
images in real world scenarios, the principal information of its content is
stored in the layout itself. In this work, we have proposed an automated deep
generative model using Graph Neural Networks (GNNs) to generate synthetic data
with highly variable and plausible document layouts that can be used to train
document interpretation systems, in this case, specially in digital mailroom
applications. It is also the first graph-based approach for document layout
generation task experimented on administrative document images, in this case,
invoices.

negative:
Since the low quality of document images will greatly undermine the chances
of success in automatic text recognition and analysis, it is necessary to
assess the quality of document images uploaded in online business process, so
as to reject those images of low quality. In this paper, we attempt to achieve
document image quality assessment and our contributions are twofold. Firstly,
since document image quality assessment is more interested in text, we propose
a text line based framework to estimate document image quality, which is
composed of three stages: text line detection, text line quality prediction,
and overall quality assessment. Text line detection aims to find potential text
lines with a detector. In the text line quality prediction stage, the quality
score is computed for each text line with a CNN-based prediction model. The
overall quality of document images is finally assessed with the ensemble of all
text line quality. Secondly, to train the prediction model, a large-scale
dataset, comprising 52,094 text line images, is synthesized with diverse
attributes. For each text line image, a quality label is computed with a
piece-wise function. To demonstrate the effectiveness of the proposed
framework, comprehensive experiments are evaluated on two popular document
image quality assessment benchmarks. Our framework significantly outperforms
the state-of-the-art methods by large margins on the large and complicated
dataset.

negative:
Document Layout Analysis is a fundamental step in Handwritten Text Processing
systems, from the extraction of the text lines to the type of zone it belongs
to. We present a system based on artificial neural networks which is able to
determine not only the baselines of text lines present in the document, but
also performs geometric and logic layout analysis of the document. Experiments
in three different datasets demonstrate the potential of the method and show
competitive results with respect to state-of-the-art methods.

negative:
Training a good supernet in one-shot NAS methods is difficult since the
search space is usually considerably huge (e.g., $13^{21}$). In order to
enhance the supernet's evaluation ability, one greedy strategy is to sample
good paths, and let the supernet lean towards the good ones and ease its
evaluation burden as a result. However, in practice the search can be still
quite inefficient since the identification of good paths is not accurate enough
and sampled paths still scatter around the whole search space. In this paper,
we leverage an explicit path filter to capture the characteristics of paths and
directly filter those weak ones, so that the search can be thus implemented on
the shrunk space more greedily and efficiently. Concretely, based on the fact
that good paths are much less than the weak ones in the space, we argue that
the label of "weak paths" will be more confident and reliable than that of
"good paths" in multi-path sampling. In this way, we thus cast the training of
path filter in the positive and unlabeled (PU) learning paradigm, and also
encourage a \textit{path embedding} as better path/operation representation to
enhance the identification capacity of the learned filter. By dint of this
embedding, we can further shrink the search space by aggregating similar
operations with similar embeddings, and the search can be more efficient and
accurate. Extensive experiments validate the effectiveness of the proposed
method GreedyNASv2. For example, our obtained GreedyNASv2-L achieves $81.1\%$
Top-1 accuracy on ImageNet dataset, significantly outperforming the ResNet-50
strong baselines.

negative:
Automatic document content processing is affected by artifacts caused by the
shape of the paper, non-uniform and diverse color of lighting conditions.
Fully-supervised methods on real data are impossible due to the large amount of
data needed. Hence, the current state of the art deep learning models are
trained on fully or partially synthetic images. However, document shadow or
shading removal results still suffer because: (a) prior methods rely on
uniformity of local color statistics, which limit their application on
real-scenarios with complex document shapes and textures and; (b) synthetic or
hybrid datasets with non-realistic, simulated lighting conditions are used to
train the models. In this paper we tackle these problems with our two main
contributions. First, a physically constrained learning-based method that
directly estimates document reflectance based on intrinsic image formation
which generalizes to challenging illumination conditions. Second, a new dataset
that clearly improves previous synthetic ones, by adding a large range of
realistic shading and diverse multi-illuminant conditions, uniquely customized
to deal with documents in-the-wild. The proposed architecture works in a
self-supervised manner where only the synthetic texture is used as a weak
training signal (obviating the need for very costly ground truth with
disentangled versions of shading and reflectance). The proposed approach leads
to a significant generalization of document reflectance estimation in real
scenes with challenging illumination. We extensively evaluate on the real
benchmark datasets available for intrinsic image decomposition and document
shadow removal tasks. Our reflectance estimation scheme, when used as a
pre-processing step of an OCR pipeline, shows a 26% improvement of character
error rate (CER), thus, proving the practical applicability.

negative:
In this paper, a new task is proposed, namely, weather translation, which
refers to transferring weather conditions of the image from one category to
another. It is important for photographic style transfer. Although lots of
approaches have been proposed in traditional image translation tasks, few of
them can handle the multi-category weather translation task, since weather
conditions have rich categories and highly complex semantic structures. To
address this problem, we develop a multi-domain weather translation approach
based on generative adversarial networks (GAN), denoted as Weather GAN, which
can achieve the transferring of weather conditions among sunny, cloudy, foggy,
rainy and snowy. Specifically, the weather conditions in the image are
determined by various weather-cues, such as cloud, blue sky, wet ground, etc.
Therefore, it is essential for weather translation to focus the main attention
on weather-cues. To this end, the generator of Weather GAN is composed of an
initial translation module, an attention module and a weather-cue segmentation
module. The initial translation module performs global translation during
generation procedure. The weather-cue segmentation module identifies the
structure and exact distribution of weather-cues. The attention module learns
to focus on the interesting areas of the image while keeping other areas
unaltered. The final generated result is synthesized by these three parts. This
approach suppresses the distortion and deformation caused by weather
translation. our approach outperforms the state-of-the-arts has been shown by a
large number of experiments and evaluations.

negative:
This paper mainly focuses on environment perception in snowy situations which
forms the backbone of the autonomous driving technology. For the purpose,
semantic segmentation is employed to classify the objects while the vehicle is
driven autonomously. We train the Fully Convolutional Networks (FCN) on our own
dataset and present the experimental results. Finally, the outcomes are
analyzed to give a conclusion. It can be concluded that the database still
needs to be optimized and a favorable algorithm should be proposed to get
better results.

negative:
Graphical elements: particularly tables and figures contain a visual summary
of the most valuable information contained in a document. Therefore,
localization of such graphical objects in the document images is the initial
step to understand the content of such graphical objects or document images. In
this paper, we present a novel end-to-end trainable deep learning based
framework to localize graphical objects in the document images called as
Graphical Object Detection (GOD). Our framework is data-driven and does not
require any heuristics or meta-data to locate graphical objects in the document
images. The GOD explores the concept of transfer learning and domain adaptation
to handle scarcity of labeled training images for graphical object detection
task in the document images. Performance analysis carried out on the various
public benchmark data sets: ICDAR-2013, ICDAR-POD2017,and UNLV shows that our
model yields promising results as compared to state-of-the-art techniques.

negative:
This study developed a traffic sign detection and recognition algorithm based
on the RetinaNet. Two main aspects were revised to improve the detection of
traffic signs: image cropping to address the issue of large image and small
traffic signs; and using more anchors with various scales to detect traffic
signs with different sizes and shapes. The proposed algorithm was trained and
tested in a series of autonomous driving front-view images in a virtual
simulation environment. Results show that the algorithm performed extremely
well under good illumination and weather conditions. Its drawbacks are that it
sometimes failed to detect object under bad weather conditions like snow and
failed to distinguish speed limits signs with different limit values.

negative:
Documents often exhibit various forms of degradation, which make it hard to
be read and substantially deteriorate the performance of an OCR system. In this
paper, we propose an effective end-to-end framework named Document Enhancement
Generative Adversarial Networks (DE-GAN) that uses the conditional GANs (cGANs)
to restore severely degraded document images. To the best of our knowledge,
this practice has not been studied within the context of generative adversarial
deep networks. We demonstrate that, in different tasks (document clean up,
binarization, deblurring and watermark removal), DE-GAN can produce an enhanced
version of the degraded document with a high quality. In addition, our approach
provides consistent improvements compared to state-of-the-art methods over the
widely used DIBCO 2013, DIBCO 2017 and H-DIBCO 2018 datasets, proving its
ability to restore a degraded document image to its ideal condition. The
obtained results on a wide variety of degradation reveal the flexibility of the
proposed model to be exploited in other document enhancement problems.

negative:
One major drawback of state of the art Neural Networks (NN)-based approaches
for document classification purposes is the large number of training samples
required to obtain an efficient classification. The minimum required number is
around one thousand annotated documents for each class. In many cases it is
very difficult, if not impossible, to gather this number of samples in real
industrial processes. In this paper, we analyse the efficiency of NN-based
document classification systems in a sub-optimal training case, based on the
situation of a company document stream. We evaluated three different
approaches, one based on image content and two on textual content. The
evaluation was divided into four parts: a reference case, to assess the
performance of the system in the lab; two cases that each simulate a specific
difficulty linked to document stream processing; and a realistic case that
combined all of these difficulties. The realistic case highlighted the fact
that there is a significant drop in the efficiency of NN-Based document
classification systems. Although they remain efficient for well represented
classes (with an over-fitting of the system for those classes), it is
impossible for them to handle appropriately less well represented classes.
NN-Based document classification systems need to be adapted to resolve these
two problems before they can be considered for use in a company document
stream.

negative:
Rapid increase of digitized document give birth to high demand of document
image retrieval. While conventional document image retrieval approaches depend
on complex OCR-based text recognition and text similarity detection, this paper
proposes a new content-based approach, in which more attention is paid to
features extraction and fusion. In the proposed approach, multiple features of
document images are extracted by different CNN models. After that, the
extracted CNN features are reduced and fused into weighted average feature.
Finally, the document images are ranked based on feature similarity to a
provided query image. Experimental procedure is performed on a group of
document images that transformed from academic papers, which contain both
English and Chinese document, the results show that the proposed approach has
good ability to retrieve document images with similar text content, and the
fusion of CNN features can effectively improve the retrieval accuracy.

negative:
This paper presents results of Document Visual Question Answering Challenge
organized as part of "Text and Documents in the Deep Learning Era" workshop, in
CVPR 2020. The challenge introduces a new problem - Visual Question Answering
on document images. The challenge comprised two tasks. The first task concerns
with asking questions on a single document image. On the other hand, the second
task is set as a retrieval task where the question is posed over a collection
of images. For the task 1 a new dataset is introduced comprising 50,000
questions-answer(s) pairs defined over 12,767 document images. For task 2
another dataset has been created comprising 20 questions over 14,362 document
images which share the same document template.

negative:
This paper presents a new state-of-the-art for document image classification
and retrieval, using features learned by deep convolutional neural networks
(CNNs). In object and scene analysis, deep neural nets are capable of learning
a hierarchical chain of abstraction from pixel inputs to concise and
descriptive representations. The current work explores this capacity in the
realm of document analysis, and confirms that this representation strategy is
superior to a variety of popular hand-crafted alternatives. Experiments also
show that (i) features extracted from CNNs are robust to compression, (ii) CNNs
trained on non-document images transfer well to document analysis tasks, and
(iii) enforcing region-specific feature-learning is unnecessary given
sufficient training data. This work also makes available a new labelled subset
of the IIT-CDIP collection, containing 400,000 document images across 16
categories, useful for training new CNNs for document analysis.

negative:
In this work, a region-based Deep Convolutional Neural Network framework is
proposed for document structure learning. The contribution of this work
involves efficient training of region based classifiers and effective
ensembling for document image classification. A primary level of `inter-domain'
transfer learning is used by exporting weights from a pre-trained VGG16
architecture on the ImageNet dataset to train a document classifier on whole
document images. Exploiting the nature of region based influence modelling, a
secondary level of `intra-domain' transfer learning is used for rapid training
of deep learning models for image segments. Finally, stacked generalization
based ensembling is utilized for combining the predictions of the base deep
neural network models. The proposed method achieves state-of-the-art accuracy
of 92.2% on the popular RVL-CDIP document image dataset, exceeding benchmarks
set by existing algorithms.

negative:
Weather conditions often disrupt the proper functioning of transportation
systems. Present systems either deploy an array of sensors or use an in-vehicle
camera to predict weather conditions. These solutions have resulted in
incremental cost and limited scope. To ensure smooth operation of all
transportation services in all-weather conditions, a reliable detection system
is necessary to classify weather in wild. The challenges involved in solving
this problem is that weather conditions are diverse in nature and there is an
absence of discriminate features among various weather conditions. The existing
works to solve this problem have been scene specific and have targeted
classification of two categories of weather. In this paper, we have created a
new open source dataset consisting of images depicting three classes of weather
i.e rain, snow and fog called RFS Dataset. A novel algorithm has also been
proposed which has used super pixel delimiting masks as a form of data
augmentation, leading to reasonable results with respect to ten Convolutional
Neural Network architectures.

negative:
In this paper, we focus on the problem of unsupervised image-sentence
matching. Existing research explores to utilize document-level structural
information to sample positive and negative instances for model training.
Although the approach achieves positive results, it introduces a sampling bias
and fails to distinguish instances with high semantic similarity. To alleviate
the bias, we propose a new sampling strategy to select additional
intra-document image-sentence pairs as positive or negative samples.
Furthermore, to recognize the complex pattern in intra-document samples, we
propose a Transformer based model to capture fine-grained features and
implicitly construct a graph for each document, where concepts in a document
are introduced to bridge the representation learning of images and sentences in
the context of a document. Experimental results show the effectiveness of our
approach to alleviate the bias and learn well-aligned multimodal
representations.

negative:
Document segmentation is a method of rending the document into distinct
regions. A document is an assortment of information and a standard mode of
conveying information to others. Pursuance of data from documents involves ton
of human effort, time intense and might severely prohibit the usage of data
systems. So, automatic information pursuance from the document has become a big
issue. It is been shown that document segmentation will facilitate to beat such
problems. This paper proposes a new approach to segment and classify the
document regions as text, image, drawings and table. Document image is divided
into blocks using Run length smearing rule and features are extracted from
every blocks. Discipulus tool has been used to construct the Genetic
programming based classifier model and located 97.5% classification accuracy.

negative:
Computation of document image quality metrics often depends upon the
availability of a ground truth image corresponding to the document. This limits
the applicability of quality metrics in applications such as hyperparameter
optimization of image processing algorithms that operate on-the-fly on unseen
documents. This work proposes the use of surrogate models to learn the behavior
of a given document quality metric on existing datasets where ground truth
images are available. The trained surrogate model can later be used to predict
the metric value on previously unseen document images without requiring access
to ground truth images. The surrogate model is empirically evaluated on the
Document Image Binarization Competition (DIBCO) and the Handwritten Document
Image Binarization Competition (H-DIBCO) datasets.

negative:
Identity of a vehicle is done through the vehicle license plate by traffic
police in general. Au- tomatic vehicle license plate recognition has several
applications in intelligent traffic management systems. The security situation
across the globe and particularly in India demands a need to equip the traffic
police with a system that enables them to get instant details of a vehicle. The
system should be easy to use, should be mobile, and work 24 x 7. In this paper,
we describe a mobile phone based, client-server architected, license plate
recognition system. While we use the state of the art image processing and
pattern recognition algorithms tuned for Indian conditions to automatically
recognize non-uniform license plates, the main contribution is in creating an
end to end usable solution. The client application runs on a mobile device and
a server application, with access to vehicle information database, is hosted
centrally. The solution enables capture of license plate image captured by the
phone camera and passes to the server; on the server the license plate number
is recognized; the data associated with the number plate is then sent back to
the mobile device, instantaneously. We describe the end to end system
architecture in detail. A working prototype of the proposed system has been
implemented in the lab environment.

negative:
Identity documents recognition is an important sub-field of document
analysis, which deals with tasks of robust document detection, type
identification, text fields recognition, as well as identity fraud prevention
and document authenticity validation given photos, scans, or video frames of an
identity document capture. Significant amount of research has been published on
this topic in recent years, however a chief difficulty for such research is
scarcity of datasets, due to the subject matter being protected by security
requirements. A few datasets of identity documents which are available lack
diversity of document types, capturing conditions, or variability of document
field values. In addition, the published datasets were typically designed only
for a subset of document recognition problems, not for a complex identity
document analysis. In this paper, we present a dataset MIDV-2020 which consists
of 1000 video clips, 2000 scanned images, and 1000 photos of 1000 unique mock
identity documents, each with unique text field values and unique artificially
generated faces, with rich annotation. For the presented benchmark dataset
baselines are provided for such tasks as document location and identification,
text fields recognition, and face detection. With 72409 annotated images in
total, to the date of publication the proposed dataset is the largest publicly
available identity documents dataset with variable artificially generated data,
and we believe that it will prove invaluable for advancement of the field of
document analysis and recognition. The dataset is available for download at
ftp://smartengines.com/midv-2020 and http://l3i-share.univ-lr.fr .

negative:
This paper describes the short-term competition on the Components
Segmentation Task of Document Photos that was prepared in the context of the
16th International Conference on Document Analysis and Recognition (ICDAR
2021). This competition aims to bring together researchers working in the field
of identification document image processing and provides them a suitable
benchmark to compare their techniques on the component segmentation task of
document images. Three challenge tasks were proposed entailing different
segmentation assignments to be performed on a provided dataset. The collected
data are from several types of Brazilian ID documents, whose personal
information was conveniently replaced. There were 16 participants whose results
obtained for some or all the three tasks show different rates for the adopted
metrics, like Dice Similarity Coefficient ranging from 0.06 to 0.99. Different
Deep Learning models were applied by the entrants with diverse strategies to
achieve the best results in each of the tasks. Obtained results show that the
currently applied methods for solving one of the proposed tasks (document
boundary detection) are already well established. However, for the other two
challenge tasks (text zone and handwritten sign detection) research and
development of more robust approaches are still required to achieve acceptable
results.

negative:
This report explores the latest advances in the field of digital document
recognition. With the focus on printed document imagery, we discuss the major
developments in optical character recognition (OCR) and document image
enhancement/restoration in application to Latin and non-Latin scripts. In
addition, we review and discuss the available technologies for hand-written
document recognition. In this report, we also provide some company-accumulated
benchmark results on available OCR engines.

negative:
With the advent of mobile and hand-held cameras, document images have found
their way into almost every domain. Dewarping of these images for the removal
of perspective distortions and folds is essential so that they can be
understood by document recognition algorithms. For this, we propose an
end-to-end CNN architecture that can produce distortion free document images
from warped documents it takes as input. We train this model on warped document
images simulated synthetically to compensate for lack of enough natural data.
Our method is novel in the use of a bifurcated decoder with shared weights to
prevent intermingling of grid coordinates, in the use of residual networks in
the U-Net skip connections to allow flow of data from different receptive
fields in the model, and in the use of a gated network to help the model focus
on structure and line level detail of the document image. We evaluate our
method on the DocUNet dataset, a benchmark in this domain, and obtain results
comparable to state-of-the-art methods.