In this thesis, we consider the problems of image classification and object localization. In particular, we focus on the problem of the weakly supervised object localization in complex scenes. In such setting, the time-consuming step of annotating the object locations in the training set is sidestepped and a localization model is learned by using image-wide labels. At the test time, however, bounding boxes have to be predicted for each learned object class as in the strongly supervised case. We have proposed a novel weakly supervised localization method based on Fisher vector image representation and model sparsity at the component level. The Fisher embedding allows weakly supervised training of localization models by employing image-wide labels. The obtained model is then applied in a sliding window manner to determine the patches responsible for the image label. Model sparsity reduces overfitting and enables fast evaluation of the patch score. In order to generate object predictions, we have proposed two novel methods which allow us to generate bounding polygons from the patches with a positive score. In order to improve the localization performance, we have proposed to use intra-component metric normalization in conjunction with the component-level sparsity. We have also introduced a first-order approximation of the normalized Fisher vector score to make our approach compatible with non-linear normalizations. The proposed approximation allows us to determine the patch-level contribution by a simple dot product. Finally, due to the fact that the Fisher vectors represent an image as an orderless collection of patch descriptors, we have proposed two novel representations of pairwise patch layout: spatial histograms and spatial Fisher vectors. We have performed an extensive evaluation of the proposed method for the tasks of traffic sign and pedestrian crossing localization. The experiments show that our method can be used for the purpose of the road-environment mapping. For the task of pedestrian crossings localization, we have introduced a novel dataset containing 2381 images of pedestrian crossings obtained by semi-automated matching of OpenStreetMap data (longitude, latitude) to GPS references of video frames. The experiments indicate that the proposed method is able to deliver reliable performance, despite weak supervision and different qualities of the involved object classes. Keywords: object localization, Fisher vectors, weakly supervised learning, sparse localization models, spatial layout representations, computer vision, machine learning.