Object Tracking Using Deep Learning Full Potential Of#

J., Puzicha, J.: Shape matching and object recognition using shape contexts.Prologue:This is a three part series which will elaborate on Object Detection in images using Convolutional Neural Networks (CNN). Specifically, various applications of Kernelized Correlation Function (KCF) and Deep Learning have led to numerous implementations of single object trackers using publicly available libraries.Letessier, J., Brard, F.: Visual tracking of bare fingers for interactive. Recent interest in computer vision has led to a great advance in the development of visual trackers. Real-time Single Object Tracking using Deep Learning.

So I will discuss those too.Accurate Multi-Object Tracking Using Deep Learning. By the time I started writing this post, I came across two papers which deal with the context in a much different way than what we devised. Third part will be about one of our more sophisticated methods in performing detection in a domain like fashion where context around the regions play an important role. Second part will give an overview on some of the fancier methodologies that have been published recently.

It assumes that there is single object of interest in the image and it covers a significant portion of image. In the first part weвЂ™ll learn how to extend last weekвЂ™s tutorial to apply real-time object detection using deep learning and OpenCV to work with video streams and video files.An image classification problem is predicting the label of an image among the predefined labels. TodayвЂ™s blog post is broken into two parts. Real-time object detection with deep learning and OpenCV. Implementing algorithms for simultaneous tracking of multiple objects is essential for applications involving. Computer vision has progressed much over the past decade.

They construct a representation in a hierarchial manner with increasing order of abstraction from lower to higher levels of neural network. Deep learning (Convolutional networks) based methods have become the state of the art in object detection in image. These methods encode a very low level characteristics of the objects and therefore are not able to distinguish well among the different labels. Exact location and size of object in image is desiredTraditional methods of detection involved using a block-wise orientation histogram(SIFT or HOG) feature which could not achieve high accuracy in standard datasets such as PASCAL VOC. The object can be lying anywhere in the image and can be of any size(scale) as can be seen in the figure 1.So object classification is no more helpful when:

The patch with high probability will not only the class of that region but also implicitly gives its location too in the image. This post will try to cover salient points in some of the most influential works in this direction.One could perform detection by carrying out a classification on different sub-windows or patches or regions extracted from the image. These have used more or less the same pipeline involving CNN(convolutional neural network)as feature extractor. Paper, a flurry of papers have been published which have either focused on improving run-time efficiency or accuracy.

These object regions are warped to a fixed sized(227X227) pixels and are fed to a classification convolutional network which gives the individual probability of the region belonging to background and classes. This does not use any fancy supervised algorithm and therefore is class-agnostic which can be used independent of the domain. R-CNN therefore uses a object proposal algorithm like selective search(SS) in its pipeline which gives out a number(~2000) of TENTATIVE object locations and extents on the basis of local cues like color rgb, hsv etc. One brute force method is to run classification on all the sub-windows formed by sliding different sized patches(to cover each and every location and scale) all through the image as in the figure 2.As you may have guessed it right, this will be a tedious process from computational time point of view as each sub-window would require passing it through CNN and calculating the feature for that region.

As in the classification training, SGD is used to train the network end to end. Therefore regions which have an IOU greater than 0.5 with the objects are marked with class of that object and those with overlap < 0.3 are marked as background. If random regions that do not have anything to do with the object classes are fed as background the network wont be able to distinguish between the object regions and regions which are partially containing the objects.

Since this training is done independent of classification training, conv5 layer and layers preceding it cannot be finetuned once their weights have been optimized for classification(because we will be using the same network for both to reduce computation time). For this a linear regression layer is optimized on top of Conv5 layer after fine-tuning the network for classification. This is required because SS need not necessarily produces regions which can encompass the objects perfectly.

After passing the image through conv layers, independent feature need to be calculated for each of the regions generated by SS. Since major chunk of time(~90%) is spent on the convolutional layers it reduces the computation time drastically. This saves a lot of time because same patch may belong to multiple regions and convolutions on them are not calculated multiple time as done in RCNN thereby enabling a shared computation of conv layers among the regions. SPP allows the whole image(not individual regions but the whole image) to be passed through the convolutional layer only once. RCNN had to pass all the ~2000 regions from SS independently through CNN and is therefore a very slow algorithm.

SPP solves this problem by replacing max pooling layer with spatial pooling layer. Normally a max pooling layer follows the final conv layer of CNN, but the feature vector produced by max pool layer depends upon the size of region(3X3 max pool with stride 1 on 5X5 maps would produce 3X3 output while on 7X7 would give 5X5 output) and therefore vector obtained cannot be fed into the following FC layers which require a fixed sized vector as input. The rectangular section of conv layer corresponding to a region can be calculated by projecting the region on conv layer by taking into account the downsampling happening in the intermediate layers(simply dividing the coordinates by 16 in case of VGG).

Object Tracking Using Deep Learning Full Potential Of

The second part follows on the same lines as RCNN wherein it fits a linear regression layer for localization on top of conv5 layers.Still SPP has certain downsides that it does not use the full potential of CNN because training is not end-to-end. The only difference in SPP net is that since windows of arbitrary size are used for pooling operation, back-propagation through SPP layer and finetuning the network end-to-end is not trivial and therefore SPP net finetunes just FC part of the network and gradients are not propagated across the SPP layer. The training windows for positive and negative samples are chosen in the same manner as RCNN. Since number of bins remain the same, a constant size vector is produced as demonstrated in the figure 4 below.

These two changes reduces the overall training time and increases the accuracy in comparision to SPP net because of end to end learning of CNN.In the pipeline of fast-rcnn, the slowest part is generating regions from selective search SS(~2s) or edgeboxes(~0.2s). This multitask objective is a salient feature of Fast-rcnn as it no longer requires training of the network independently for classification and localization. Second major change is a single network with two loss branches pertaining to soft-max classification and bounding box regression. It is a simple back-propagation calculation and is very similar to max-pooling gradient calculation with the exception that pooling regions overlap and therefore a cell can have gradients pumping in from multiple regions.

Since the task of RPN is to generate potential object like regions classification layer has only two outputs for background and foreground. This is then fed to the two sibling layers just as in previous detection network for the two tasks of classification and localization. So for RPN a small network with kernel size of 3X3 is run through the final conv feature map and a smaller 256 dimension feature is obtained at every spatial location. Since the conv layers are shared it does not add to the computation time and the only additional time involved is the two additional conv layers which have relatively small number of filters. This is done by using the convolutional layers from detection network(therefore no overhead) and introducing two convolutional layers on top of this (in parallel to FC layers of detection network) to generate regions at various spatial location.