MUTUAL IMAGE TRANSFORMATION ALGORITHMS FOR VISUAL INFORMATION PROCESSING AND RETRIEVAL

Kukharev Georgy A, Matveev Yuri Nikolaevich, Oleinik Andrei L

2017 , VOLUME 17, NUMBER 1 ( January–February )

ISSN 2226-1494 (print), ISSN 2500-0373 (online)

Publications

Editor-in-Chief

Nikiforov
Vladimir O.
D.Sc., Prof.

Partners

doi: 10.17586/2226-1494-2017-17-1-62-74

MUTUAL IMAGE TRANSFORMATION ALGORITHMS FOR VISUAL INFORMATION PROCESSING AND RETRIEVAL

G. A. Kukharev, Y. N. Matveev, A. L. Oleinik

Read the full article

Article in Russian

For citation: Kukharev G.A., Matveev Yu.N., Oleinik A.L. Mutual image transformation algorithms for visual information processing and retrieval. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2017, vol. 17, no. 1, pp. 62–74. doi: 10.17586/2226-1494-2017-17-1-62-74

Abstract

Subject of Research. The paper deals with methods and algorithms for mutual transformation of related pairs of images in order to enhance the capabilities of cross-modal multimedia retrieval (CMMR) technologies. We have thoroughly studied the problem of mutual transformation of face images of various kinds (e.g. photos and drawn pictures). This problem is widely represented in practice. Research is this area is based on existing datasets. The algorithms we have proposed in this paper can be applied to arbitrary pairs of related images due to the unified mathematical specification. Method. We have presented three image transformation algorithms. The first one is based on principal component analysis and Karhunen-Loève transform (1DPCA/1DKLT). Unlike the existing solution, it does not use the training set during the transformation process. The second algorithm assumes generation of an image population. The third algorithm performs the transformation based on two-dimensional principal component analysis and Karhunen-Loève transform (2DPCA/2DKLT). Main Results. The experiments on image transformation and population generation have revealed the main features of each algorithm. The first algorithm allows construction of an accurate and stable model of transition between two given sets of images. The second algorithm can be used to add new images to existing bases and the third algorithm is capable of performing the transformation outside the training dataset. Practical Relevance. Taking into account the qualities of the proposed algorithms, we have provided recommendations concerning their application. Possible scenarios include construction of a transition model for related pairs of images, mutual transformation of the images inside and outside the dataset as well as population generation in order to increase representativeness of existing datasets. Thus, the proposed algorithms can be used to improve reliability of face recognition performed on images of various kinds. Moreover, these techniques can be applied to address a wide variety of other CMMR problems.

Keywords: cross-modal multimedia retrieval, principal component analysis, face images, sketch, facial composite

Acknowledgements. This work was partially financially supported by the Government of the Russian Federation, Grant 074-U01

References

1. Kato T. Database architecture for content-based image retrieval. Proceedings of SPIE, 1992, vol. 1662, pp. 112–123.

2. Wang K., Yin Q., Wang W., Wu S., Wang L. A comprehensive survey on cross-modal retrieval. ArXiv Preprint, 2016, ArXiv160706215, 20 p.

3. Liao S., Yi D., Lei Z., Qin R., Li S.Z. Heterogeneous face recognition from local structures of normalized appearance. Lecture Notes in Computer Science, 2009, vol. 5558, pp. 209–218. doi: 10.1007/978-3-642-01793-3_22

4. Guo G. Heterogeneous face recognition: an emerging topic in biometrics. Intel Technology Journal, 2014, vol. 18, no. 4, pp. 80–97.

5. Kukharev G.A., Shchegoleva N.L., Kamenskaya E.I. Representation and comparison methods for semantically different images. Pattern Recognition and Image Analysis, 2014, vol. 24, no. 4, pp. 518–529. doi: 10.1134/S1054661814040105

6. Tang X., Wang X. Face sketch recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2004, vol. 14, no. 1, pp. 50–57. doi: 10.1109/TCSVT.2003.818353

7. Kukharev G., Oleinik A. Face photo-sketch transformation and population generation. Lecture Notes in Computer Science, 2016, vol. 9972, pp. 329–340. doi: 10.1007/978-3-319-46418-3_29

8. Kukharev G.A., Kamenskaya E.I., Matveev Y.N., Shchegoleva N.L. Metody Obrabotki i Raspoznavaniya Izobrazhenii Lits v Zadachakh Biometrii [Methods for Face Image Processing and Recognition in Biometric Applications] Ed. M.V. Khitrov. St. Petersburg, Politekhnika Publ., 2013, 388 p.

9. CUHK Face Sketch Database. Available at: http://mmlab.ie.cuhk.edu.hk/facesketch.html (accessed 11.08.2014).

10. CUHK Face Sketch FERET Database. Available at: http://mmlab.ie.cuhk.edu.hk/cufsf/ (accessed 11.08.2014).

11. Chen H., Liu Z., Rose C., Xu Y., Shum H.Y., Salesin D. Example-based composite sketching of human portraits. Proc. 3^rd Int. Symposium on Non-Photorealistic Animation and Rendering. Annecy, France, 2004, pp. 95–102.

12. Ivanova A. Dog instinctively imitates humans. Dogs, like owners. Available at: http://zdravkom.ru/nauka_i_obrazovanie/sobaki_kopiruyt_hozyaev (accessed 20.11.2016).

13. Shchegoleva N.L., Kukharev G.A. Automatic Sketch Generation Method and System Therefor. Patent RU2541132, 2015.

14. Zhang Y., Ellyson S., Zone A., Gangam P., Sullins J., McCullough C., Canavan S., Yin L. Recognizing face sketches by a large number of human subjects: a perception-based study for facial distinctiveness. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition and Workshops, FG 2011. Santa Barbara, USA, 2011, pp. 707–712. doi: 10.1109/FG.2011.5771335

15. Kukharev G.A., Matveev Y.N., Shchegoleva N.L. New solutions for face photo retrieval based on sketches. Pattern Recognition and Image Analysis, 2016, vol. 26, no. 1, pp. 165–175. doi: 10.1134/S1054661816010144

16. Kukharev G.A., Matveev Yu.N., Forczmanski P. People retrieval by means of composite pictures – methods, systems and practical decisions. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2015, vol.15, no. 4, pp. 640–653. doi: 10.17586/2226-1494-2015-15-4-640-653

17. Tsapatsoulis N., Alexopoulos V., Kollias S. A vector based approximation of KLT and its application to face recognition. Proc. 9^th European Signal Processing Conference EUSIPCO-98. Island of Rhodes, Greece, 1998, vol. 1581, pp. 1–4.

18. Shchegoleva N.L., Kukharev G.A. Application of two-dimensional principal component analysis for recognition of face images. Pattern Recognition and Image Analysis, 2010, vol. 20, no. 4, pp. 513–527. doi: 10.1134/S1054661810040127

19. Kukharev G., Forczmanski P. Data dimensionality reduction for face recognition. Machine Graphics and Vision, 2004, vol. 13, no. 1-2, pp. 99–121.

Kukharev G., Tujaka A., Forczmanski P. Face recognition using two-dimensional CCA and PLS. International Journal of Biometrics, 2011, vol. 3, no. 4, pp. 300–321. doi: 10.1504/IJBM.2011.042814

PICTURES

Fig. 1. The training sample and the results of transformation. Sequences 1 and 2 show training data sets. Sequence 3 is the reconstruction result based on the images of the sequence 2. Sequence 4 presents images of sequence 2 in low resolution with added noise. Sequence 5 is the transformation result of the images out of sequence 4

Fig. 2. Transformation of the test sketches. Sequence 1 – the initial sketches, sequence 2 is the transformation result for the sketches from sequence 1 into the photo, sequence 3 – original photos

Fig. 3. Sketches of one and the same person, made by various artists [14]

Fig. 4. Scheme of the proposed algorithm of transformation and populations generation

a) b)

Fig. 5. Populations of photos generated from the sketch (a), and three-dimensional visualization of photograph projections on the three first principal components (b). Each photograph of a population corresponds to one point on the graph

Fig. 6. Populations of photos, generated on the basis of sketches from the training (a) and test (b) samples

Fig. 7. Transformation of sketches into pictures by two-dimensional algorithm. Sequence 1 – the initial sketches, sequence 2 – transformation result. Transformation was carried out for sketches from the training (a) and test (b) samples and also for the synthesized sketches [13] (c)

Fig. 8. Transformation of sketches from AR base into the photos by two-dimensional algorithm. Sequence 1 – the initial sketches, sequence 2 – the result of transformation

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License