Tampere University of Technology

TUTCRIS Research Portal

A New R-D Optimization Criterion for Fast Mode Decision Algorithms in Video Coding and Transrating

Research output: Contribution to journalArticleScientificpeer-review


Original languageEnglish
Pages (from-to)696-710
JournalIEEE Transactions on Circuits and Systems for Video Technology
Issue number4
Publication statusPublished - 2015
Publication typeA1 Journal article-refereed


Mode decision has a significant effect on the quality and complexity of video coding. It is even more challenging when generating multiple bitstreams with different bitrates in, for example, Dynamic Adaptive Streaming for HTTP (DASH) or transrating systems. Full search and simplified fast search mode decision methods either suffer from a high computational complexity or have a negative impact on quality. Furthermore, mode selection in conventional approaches strongly depends on the quantization parameter. Hence, modes that have been selected for high bitrate compression may not be suitable for low bitrate when transrating a bitstream. In this paper, we propose a Rate-Distortion (R-D) optimized criterion for fast mode decision algorithms. The proposed cost function when adopted in different fast mode decision algorithms, not only improves the R-D performance by up to 6.6% in terms of Bjøntegaard delta rate, but also reduces the execution time of the encoder by up to 6.8%. We also show that modes selected by the proposed criterion are less sensitive to changes in bitrate or quantization parameter. As a result, the same modes in an encoded bitstream may be used even after transrating using re-quantization, resulting in a significant R-D performance improvement of up to 33.3%.


  • Rate distortion optimization, mode decision, Video coding, video transrating

Publication forum classification

Field of science, Statistics Finland