No-Reference Metrics For No-Reference Metrics For Video Streaming ApplicationsVideo Streaming Applications

International Packet Video Workshop (PV 2004) International Packet Video Workshop (PV 2004)

Presented by : BhavanaPresented by : Bhavana

CPSC 538CPSC 538

February 21, 2004February 21, 2004

Video Quality AssessmentVideo Quality Assessment

What’s Quality ?What’s Quality ?

- Implies Comparison => reference- Implies Comparison => reference

Three Techniques :Three Techniques :

- Full-Reference eg. MSE, PSNR- Full-Reference eg. MSE, PSNR

- Reduced Reference- Reduced Reference

- No Reference- No Reference

What is a No-Reference metric ?What is a No-Reference metric ?

Estimating end-user’s QoE of a Estimating end-user’s QoE of a multimedia stream without using an multimedia stream without using an original stream as a reference.original stream as a reference.

In other words :In other words :

“ “ Quantify quality via blind distortion Quantify quality via blind distortion measurement”measurement”

PurposePurpose

To evaluate two types of distortions in To evaluate two types of distortions in streaming of compressed video over streaming of compressed video over packet-switched networkspacket-switched networks

- Compression related : block-edge - Compression related : block-edge impairment impairment

- Transmission related : packet-loss - Transmission related : packet-loss impairment impairment

Where Can It Be Used ?Where Can It Be Used ?

For real time monitoring .For real time monitoring .

Reference unavailable or expensive to sendReference unavailable or expensive to send

Feedback to Streaming Server .Feedback to Streaming Server .

Evaluation of Compression AlgorithmsEvaluation of Compression Algorithms

What are Block-Based Codecs ?What are Block-Based Codecs ?

Process several pixels of video together Process several pixels of video together in blocksin blocks At high compression rates, strong At high compression rates, strong discontinuities called block edges come discontinuities called block edges come up.up.What’s blockiness ? What’s blockiness ?

“ “Distortion of image characterized by Distortion of image characterized by appearance of underlying block encoding appearance of underlying block encoding structurestructure

Block – based DistortionBlock – based Distortion

Idea Idea : A block-edge gradient can be : A block-edge gradient can be masked by a region of high spatial activity masked by a region of high spatial activity around it .around it .

Measure two things :Measure two things :

- spatial activity around block edges : - spatial activity around block edges : σσ

- block-edge gradient : - block-edge gradient : ΔΔ

Calculation of NR Blockiness MetricCalculation of NR Blockiness Metric

For each 8 x 8 Block BFor each 8 x 8 Block Bij ij , ,

For each edge IFor each edge Ik k of Bof Bijij ,,

divide edge into 3 segments adivide edge into 3 segments aklkl

For each segment of aFor each segment of akl kl

calculate calculate σσklkl

calculate calculate ΔΔklkl

I4

I1

E2E4

E3

I2

E1

I3

Bij

0 1 2 3 4 5 6 7

a1

a2

a3

An 8 x 8 block and its edges

Three segments akl of a block edge

NR Blockiness Metric contd’NR Blockiness Metric contd’

CCB B = No. of Blocks for which at least one edge = No. of Blocks for which at least one edge

satisfies :satisfies :

σσkl kl < < εε where where εε = 0.1 = 0.1

ΔΔkl kl > > ττ where where ττ = 2.0 = 2.0

εε = min. spatial activity required to mask gradient = min. spatial activity required to mask gradient

ττ = max. gradient which is imperceivable.= max. gradient which is imperceivable.

ββFF = = CCB B / Total no. of blocks in the frame/ Total no. of blocks in the frame

Simulation Setup For NR Simulation Setup For NR Blockiness MetricBlockiness Metric

Aim : to measure how well the NR Aim : to measure how well the NR Blockiness metric conveys QoEBlockiness metric conveys QoE

Codec : MPEG -4 , GOP = 30 framesCodec : MPEG -4 , GOP = 30 frames

Bit Rate => compression levelBit Rate => compression level

NR Packet Loss MetricNR Packet Loss Metric

Error Concealment : Replace Error Concealment : Replace damaged/lost macroblock with damaged/lost macroblock with corresponding macroblock from previous corresponding macroblock from previous frame.frame.

Idea Idea : Use length of artifact to estimate : Use length of artifact to estimate amount of distortion caused by packet lossamount of distortion caused by packet loss

Calculation of NR Packet Loss Calculation of NR Packet Loss MetricMetric

For a m x n frameFor a m x n frame

For each 16 x 16 macroblockFor each 16 x 16 macroblock

Calculate :Calculate :

ÊÊjj = strength vector across macroblock = strength vector across macroblock

edgeedge

ÊÊ΄́jj = strength vector within macroblock = strength vector within macroblock

near the edgenear the edge

Macroblock 1

Macroblock 2

Figure : Calculating Strength vector across and within a macroblock

Convert strength vectors into binary Convert strength vectors into binary vectorsvectors

EEjj(k) = 1 if (k) = 1 if ÊÊj j > > ττ

= 0 otherwise= 0 otherwise

EE΄́jj(k) = 1 if (k) = 1 if ÊÊ΄́j j > > ττ

= 0 otherwise= 0 otherwise

where where ττ = 15 = 15

If the sum of differences between the two binary If the sum of differences between the two binary edge vectors is substantial , then there is edge vectors is substantial , then there is distortiondistortion

Packet loss metric for jPacket loss metric for j thth macroblock macroblock

HHj j = = ∑ | ∑ | EEjj(k) - E(k) - E΄́jj(k) | if (k) | if ∑ | ∑ | EEjj(k) - E(k) - E΄́jj(k) | > (k) | > ζζ

= 0 otherwise= 0 otherwise

where where ζζ = 10% of frame width (n) = 10% of frame width (n)

Packet loss metric for whole frame Packet loss metric for whole frame

F = ∑ HF = ∑ Hjj22

Simulation Setup for NR Packet Simulation Setup for NR Packet Loss MetricLoss Metric

Bit Rate = 1.5 MbpsBit Rate = 1.5 Mbps

Frame Rate = 30 fpsFrame Rate = 30 fps

Frame Size = 352 x 240Frame Size = 352 x 240

Used NTT DoCoMo packet loss Used NTT DoCoMo packet loss generating software .generating software .

Limitations Of NR-metricsLimitations Of NR-metrics

Blockiness metric might fail in the Blockiness metric might fail in the presence of strong de-blocking filters presence of strong de-blocking filters which might otherwise introduce blurwhich might otherwise introduce blur

Metric predictions lose meaning in Metric predictions lose meaning in presence of other distortions like blur, presence of other distortions like blur, noise etc.noise etc.

Future DirectionsFuture Directions

VQEG standardization effortsVQEG standardization efforts HVS based approachesHVS based approaches Statistical models for natural scenesStatistical models for natural scenes NR QA schemes for NR QA schemes for

- Non-block based compression schemes - Non-block based compression schemes such Wavelet-based such Wavelet-based

-Targeting full range of artifacts -Targeting full range of artifacts

ReferencesReferences

No Reference Image and Video Quality No Reference Image and Video Quality AssessmentAssessment http://http://live.ece.utexas.edu/research/quality/nrqa.htmlive.ece.utexas.edu/research/quality/nrqa.htm Objective video Quality Assessment Objective video Quality Assessment http://www.cns.nyu.edu/~zwang/files/papers/QAhttp://www.cns.nyu.edu/~zwang/files/papers/QA_hvd_bookchapter.pdf_hvd_bookchapter.pdfPerceptual Video Quality and Blockiness Metrics Perceptual Video Quality and Blockiness Metrics for Multimedia Streaming Applicationsfor Multimedia Streaming Applicationswww.stefan.winkler.net/Publications/www.stefan.winkler.net/Publications/wpmc2001.pdf wpmc2001.pdf