where the wild bots are opsny june 2016

Where the Wild Bots Are

June 2016Augustine Fou, PhD.acfou [at] mktsci.com 212. 203 .7239

mailto:[email protected]

mailto:[email protected]

June 2016 / marketing.scienceconsulting group, inc.

linkedin.com/in/augustinefou

Brief Overview

Ad fraud and ad blocking lower the effectiveness of digital media and messes up measurement.

• Ad Fraud – quick review‐ fraud bot activity (fake traffic, fake clicks) wastes ad dollars

and messes up measurement

• Ad Blocking – new, original data (AdMonsters study)‐ bots don’t use ad blocking; ad blocking must be measured

together with bots and viewability

• Actions – looking ahead

Ad Fraud Background



Fraud continues upward as digital ad spend goes up

Digital ad fraud

Digital ad spendSource: IAB 2015 FY Report

$ billions

E

High / Low Estimates



Ad fraud is a “double-whammy” for advertisers

Messed Up AnalyticsWasted Ad Dollars

Ad shown to bots are wasted

Fake traffic, impressions, clicks are all recorded by

analytics



Ad fraud is a “QUAD-whammy” for good publishers

2. “Bottom line” profitability squeezed

1. “Top line” ad revenue stolen

4. Reputations ruined by bad guys covering tracks

3. Ad blockers further reduce ad revenue



Fraud siphons 1/2 of dollars out of ad ecosystem

Advertisers “ad spend” in digital

is $60B in FY2015

Publishers are left with only 1/2 of the dollars

Bad Guyssiphon 1/2 of ad spend OUT of the ecosystem

Ad dollars are being siphoned OUT of the ecosystem into the pockets of the bad guys

1/2

1/2

Usersuse ad blocking and

need to protect privacy



Bad guys follow the money – CPM, CPC fraud

Impressions(CPM/CPV)

Clicks(CPC)

Search32%

91% digital spend

Display12%

Video7%

Mobile40%

Leads(CPL)

Sales(CPA)

Lead Gen$2.0B

Other$5.0B

• classifieds• sponsorship• rich media

(86% in FY2014)Source: IAB 2015 FY Report

(83% in FY2013)



It is SO extremely profitable, bad guys won’t stop doing it

Source: https://hbr.org/2015/10/why-fraudulent-ad-networks-continue-to-thrive

“the profit margin is 99% … [especially with pay-for-use cloud services ]…”

Source: Digital Citizens Alliance Study, Feb 2014

“highly lucrative, and profitable… with margins from 80% to as high as 94%…”

https://hbr.org/2015/10/why-fraudulent-ad-networks-continue-to-thrive

https://hbr.org/2015/10/why-fraudulent-ad-networks-continue-to-thrive

https://media.gractions.com/314A5A5A9ABBBBC5E3BD824CF47C46EF4B9D3A76/4af7db7f-03e7-49cb-aeb8-ad0671a4e1c7.pdf



Two main types of fraud and how each is generated

Impression (CPM) Fraud

(includes mobile display, video ads)

1. Put up fake websites and load tons of ads on the pages

Search Click (CPC) Fraud

(includes mobile search ads)

2. Use bots to repeatedly load pages to generate fake ad impressions (hide the true origins to avoid detection)

1. Put up fake websites and participate in search networks

2. Use bots to type keywords to cause search ads to load and then to click on the ad to generate the CPC revenue



Bots are the cause of all automated ad fraud

Headless BrowsersSeleniumPhantomJSZombie.jsSlimerJS

Mobile Simulators35 listed

Bots are made from malware compromised PCs or headless browsers (no screen) in datacenters.

http://www.mobilexweb.com/emulators



Any device with chip/connectivity can be used as a bot

Traffic cameras turned into botnet (Engadget, Oct 2015) mobile devices

webcams

connected traffic lights

connected cars

thermostat

connected fridge

Security cams used as 400 Gbps DDoS botnet (Engadget, Jun 2016)



What I heard (at Publishers Forum)

“Ad fraud doesn’t affect us”

“I wasn’t really aware of bots and fraud”

“Our SSP has an anti-fraud vendor”

“we checked, we have very low bots”

Bots and Bad Guys



Websites – spectrum from bad to good

Ad Fraud Sites

Click Fraud Sites

100% bot

mostly human

longtail mid-tail mainstream

Sites w/ Sourced Traffic

Piracy Sites

“cash-out sites” “sites w/ questionable practices”

Premium Publishers

“good guys”



Bots – from easy-to-detect to advanced bots

10,000bots observed

in the wild

user-agents.org

bad guys’ bots3%

Dstillery, Oct 9, 2014_“findings from two independent third parties,

Integral Ad Science and White Ops”

3.7%Rocket Fuel, Sep 22, 2014

“Forensiq results confirmed that ... only 3.72% of impressions categorized as high risk.”

2 - 3%comScore, Sep 26, 2014

“most campaigns have far less; more in the 2% to 3% range.”

bot list-matching

“not on any list”disguised as normal browsers –

Internet Explorer; constantly adapting to avoid detection

http://finance.yahoo.com/news/dstillery-leads-industry-1-inventory-173635834.html

http://finance.yahoo.com/news/dstillery-leads-industry-1-inventory-173635834.html

http://rocketfuel.com/blog/forensiq-confirms-the-quality-of-rocket-fuel-programs

http://rocketfuel.com/blog/forensiq-confirms-the-quality-of-rocket-fuel-programs

http://adexchanger.com/platforms/fraud-day-with-comscore-an-ad-impression-is-a-terrible-thing-to-waste/

http://adexchanger.com/platforms/fraud-day-with-comscore-an-ad-impression-is-a-terrible-thing-to-waste/



Premium publishers have lots of humans



Programmatic impressions look much different



Humans (blue) on ad networks vs good publishers

Ad Networks

Publishers



End of month traffic and impressions fulfillment

Traffic surge

Impressions surge

volume bars (green)

Stacked percentBlue (human)Red (bots)

red vs blue trendlines



Real traffic surges, human visits due to news

Traffic surgesvolume bars (green)

Stacked percentBlue (human)Red (bots)

red v blue trendlines



Fraud Activities Mess Up Measurement



http://www.olay.com/skin-care-products/OlayPro-X?utm_source=msn&utm_medium=cpc&utm_campaign=Olay_Search_Desktop

Bad guys easily hide fraud by passing fake parameters

Click thru URL passes fake source “utm_source=msn”

buy eye cream online(expensive CPC keyword)

1. Fake site that carries search ads

Olay.com ad in #1 position

2. search ad served, fake click

Destination pagefake source declared

3. Click through to destination page

http://www.olay.com/skin-care-products/OlayPro-X?utm_source=msn&utm_medium=cpc&utm_campaign=Olay_Search_Desktop_Category+Interest+Product.Phrase&utm_term=eye%20cream&utm_content=TZsrSzFz_eye%20cream_p_2990456911









Bad guys fake KPIs, trick measurement systemsBad guys have higher CTR Bad guys have higher viewability

AD

Bad guys stack ads above the fold to fake 100% viewability

Good guys have to array ads on the page – e.g. lower average viewability.



Bad guys’ bots can fake most quantity metrics

click on links

load webpages tune bounce rate

tune pages/visit



Recognizing human vs bot traffic patternsBot traffic is “programmed” so the amount of traffic is the same (red line, flat across)

Human visit websites during waking hours, using search



Google analytics view of traffic from fraud source

Despite cutting off the traffic from the fraud site, there was no change to the number of pledges and downloads, during the same period of time.

102,231 sessions

0 sessions

goal event – no change

“ … because bots don’t make donations!”



AppNexus example – cleaned up 92% of impressions

Increased CPM prices by 800%

Decreased impression volume by 92%

Source: http://adexchanger.com/ad-exchange-news/6-months-after-fraud-cleanup-appnexus-shares-effect-on-its-exchange/

260 billion

20 billion

> $1.60

< 20 cents

“good for them; good for advertisers who buy from them”



Bad guys’ bots earn more money, more efficientlyHigher bots in retargetingBots collect cookies to look attractive

Source: DataXu/DoubleVerify Webinar, April 2015 Source: White Ops / ANA 2014 Bot Baseline



Fraud operations are massively scalableCash out sites are massively scalableAuto create fraud sites with algos

131 ads on pageX

100 iframes=

13,100 ads /page

Stacked redirects (e.g. dozens)Known blackhat technique to hide real referrer and replace with faked referrer.

Example how-to:http://www.blackhatworld.com/blackhat-seo/cloaking-content-generators/36830-cloaking-redirect-referer.html

Thousands of requests per page

The Connection to Ad Blocking



Humans block ads; fraud bots don’tHigh human samples High bot samples

17% blocked

42% blocked

1% blocked

3% blocked



Humans use ad block; ads served to non-blocking bots

Total Internet Users – 285 millionNon-Human Traffic

adblocking humans

Total Human Users – 120 million

Adblock Users (humans) – 50 million

U.S. Only

Source: eMarketer 2016 estimate

Source: Distil Networks 2015

170 million 50 million

70 million

non-adblocking humans

Source: PageFair / Adobe 2015

“subtracting adblocking humans, your programmatic ads are served to a population that is disproportionally (71%) non-human.”



Blocking, bots, viewability must be measured together

bots(White Ops)

viewability(Moat)

adblocking(PageFair)

“fraud sites with lots of bots also have very high viewability”

“sites with lots of bots have abnormally low adblocking” (bots don’t block ads)

“sites that cheat have abnormally high viewability and low ad blocking”



Change perspective to focus on positive/reliable

human

visible loaded



AdMonsters Publishers Study



Desperately seeking high “LVH” ad inventory

human

visible loaded



Publishers participating in study - examples



More great publishers who participated in study

• 5+2 pattern visible; lower traffic overnight too• humans (blue) much higher than bots (red)



Publisher site with great content and humans

|A| ad loaded 64%

|B| visible 86%

|C| human 89%

57%



By contrast, impressions served on ad networks

|A| ad loaded 23%

|B| visible 19%

|C| human 39%

4%



Examples of widely varying LVH measurements

|A| ad loaded 81%

|B| visible 92%

|C| human 91%

77%

|A| ad loaded 58%

|B| visible 66%

|C| human 71% 27%

Publisher (High LVH)

Publisher (Low LVH)

|A| ad loaded 55%

|B| visible 60%

|C| human 44%

|A| ad loaded 35%

|B| visible 48%

|C| human 38%

Ad Network (High)

Ad Network (Low)

6%

12%



Current industry level view – can be more accurate

Source: Terence Kawaja @tkawaja – Digital Media Summit, May 2016



Programmatic traffic – bots, ad blocking, viewability

Non-Human Traffic (NHT) HUMAN VISITORS

List

-mat

ch b

ot d

etec

tion

Ad b

lock

ed b

y hu

man

use

r

simpl

e bo

ts, c

raw

lers

advanced bots (mouse, scroll, click)

humans tricked• invisible ads• domain spoofing• site bundling• ad injection• pixel stuffing• cookie cloning• clickjacking• sourced traffic• arbitrage• click bait• ad carousel

ad loaded, visible,

human (LVH)

ads served

advertiser VALUEadvertiser WASTE

“cash-out sites” “sites w/ questionable practices” “good guys”



Premium publishers – LVH (loaded, visible, human)(NHT) HUMAN VISITORS

List

-mat

ch b

ot d

etec

tion

Ad b

lock

ed b

y hu

man

use

r

simpl

e bo

ts, c

raw

lers

adva

nced

bot

s

ads served

advertiser VALUE

ad loaded, visible,

human (LVH)



AdMonsters Publishers Study – Class of May 2016

AdMonsters Publishers Study• 30 days, directly measured• 30 publishers/sites• 1 billion pageviews• ocean of blue



Take Action Now



Challenge all assumptions• mobile ad blocking is lower – perhaps, but it is also possible that

it is due to more incomplete measurement in mobile

• desktop ad blocking is low – but this may be due to more bots visiting (bots don’t use ad block)

• programmatic ads have higher CTR – this may be due to bots creating fake clicks to trick you into sending them more money

• fraud is in the lowest cost inventory – no, in fact there is much more fraud in the highest CPM ads like video ads

• ads are not served if ad block is on – some ad blockers now call the ad to be served, then suppress it from displaying

• viewability vendor takes care of it – viewability is supposed to mean no IVT and no ad blocking; it doesn’t actually, ask about it.



Top Advertiser Concerns



About the Author



Dr. Augustine Fou – Recognized Expert on Ad Fraud

2013

2014

2015SPEAKING ENGAGEMENTS / PANELS4A’s Webinar on Ad Fraud AdCouncil Webinar on Ad Fraud TelX Marketplace LiveARF Audience Measurement / ReThinkIAB Webinar on Ad Fraud / Botnets AdMonsters Publishers Forum / OPS



Harvard Business Review – October 2015

Excerpt:

Hunting the Bots

Fou, a prodigy who earned a Ph.D. from MIT at 23, belongs to the generation that witnessed the rise of digital marketers, having crafted his trade at American Express, one of the most successful American consumer brands, and at Omnicom, one of the largest global advertising agencies. Eventually stepping away from corporate life, Fou started his own practice, focusing on digital marketing fraud investigation.

Fou’s experiment proved that fake traffic is unproductive traffic. The fake visitors inflated the traffic statistics but contributed nothing to conversions, which stayed steady even after the traffic plummeted (bottom chart). Fake traffic is generated by “bad-guy bots.” A bot is computer code that runs automated tasks.

where the wild bots are opsny june 2016

Data & Analytics