Internet Economicsכלכלת האינטרנט

Class 9 – predictions and scoring rules

1

Today

2

• Prediction markets and scoring rules.

• Some guidelines for the 2nd semester

Wisdom of the crowds

3

• The Internet is used to aggregate information from the general population.

• Creation of “information markets”.

• Main use: calculating probabilities of one-time events– Winners in elections.– Outcomes of sports events.– Political changes.– Financial events (future exchange rates, future stock values)

• In large companies: elicit information from employees.– Probability that our new smartphone will gain more than 10% of the

market.

Information markets

intrade

Contracts

5

• We need to sign a contract with the market participants.

• Contract should be based on verifiable data.– Sometimes it is tricky….

– Israel will strike Iran by December 31st 2011.– Robert De-Niro will win the 2011 academy

awards for best actor in main role.

6

Winner-takes-all contract

7

• A bond is issued:– If event A occurs, it pays $100.– Otherwise, pays 0.

• For example: Saddam securityPaid out $100 if Saddam is no longer in power by June 2003.– Traded on website tradesports.com– In the hour after President Bush's war "declaration", the Saddam-gone-by-

March bet shot up nearly 10 percentage points (odds went from about 60 to 69 percent)

– Price on Jan 1, 2003: $55– Price on March 1, 2003: $70

Trading Contracts

8

• How are contracts traded?– In most prediction markets, the mechanism used

is the continuous double auction.– This is the mechanism used on the NYSE.

Winner-takes-all Contracts

9

• US 2008 elections – price history (from Iowa Electronic Market)

Scoring rules

10

• We sometime aim to more delicate contracts

• For example, think about the following simple scenario:

we do not want to aggregate the wisdom of the crowds, but the wisdom of one individual.

Example: Weatherman

11

• Tomorrow can be either sunny or rainy.• We would like to know the probability of a sunny day.• A weather forecaster knows, but he can lie.• Can we incentivize him to tell us the truth?

75% for a sunny day tomorrow

Contracts

12

• We need to sign a contract with the weatherman.

• Contract with the weatherman can depend on:– The probability declared by the forecaster.– The realization of the outcome (sunny/rainy)

• Contract cannot depend on the true probability– It is unobservable, even in retrospect.

Proper scoring rules

13

• We would like that given the contract, a utility-maximizing forecaster will report the true probability.– Via a bonus system.

• Contracts where the forecaster strictly prefers to report the truth are called proper scoring rules.– It is easy to construct scoring rules that are not proper.

Model

14

• Consider a binary outcome: z=0 or z=1– Example: z=1 means rain, z=0 means sun.

• p is the true probability of z=1.– Pr(z=1)=p, Pr(z=0)=1-p

• q is the report of the forecaster.

• The contract:– After realizing the outcome z, the forecaster pays

R(z,q).

• Are there proper scoring rules in this model?

First try: pay your prediction

15

• Consider the following contract:– R(z,q) = q

• Pay the probability that you gave to the realized outcome.

– i.e., when z=0 forecaster is paid R(0,q) = 1-q when z=1 forecaster is paid R(1,q) = q

• Expected surplus for forecaster:– p*q + (1-p)*(1-q) = (1-p)+(2p-1)q

• Forecaster will optimize q.– Report 1 if p>1/2, report 0 otherwise.

• Not a proper scoring rule.

Example: quadratic scoring rules

16

• Consider the following contract:– R(z,q) = 1 - (z-q)2 – i.e., when z=0 forecaster is paid R(0,q) = 1-q2

when z=1 forecaster is paid R(1,q) = 1-(1-q)2 =q(2-q)

• Expected surplus for forecaster:– p*R(1,q) + (1-p)*R(0,q) = pq(2-q)+(1-p)(1-q2)

• Forecaster will optimize q.– FOC:

• p(2-2q)+(1-p)(-2q) = 2p-2pq-2q+2pq = 0 q=p

• A proper scoring rule (“quadratic scoring rule”)

Example: logarithmic scoring rule

17

• Consider the following contract:– R(z,q) = zlogq + (1-z)log(1-q)– i.e., when z=0 forecaster is paid R(0,q) = log(1-q)

when z=1 forecaster is paid R(1,q) = logq

• Expected surplus for forecaster:– p*R(1,q)+(1-p)*R(0,q) = p* log(q) + (1-p)* log(1-q)

• FOC:– p(1/q)-(1-p)(1/(1-q))=0 q=p

• A proper scoring rule (“logarithmic scoring rule”)

Finding proper scoring rule

18

A general result:• n possible events (1,…,n)• Probability estimate r=r1,…,rn

• Scoring rule for each event si(ri) (i=1,…,n)• Expected score function:

Theorem: (McCarthy ‘56, Hendrickson and Buehlera'71 )

Every homogeneous and convex expected score function defines a proper scoring rule.

• Example: quadratic scoring rule H(q)=q2(2-q)+(1-q)(1-q2)

n

iiiin rsrrrH

11 )(),...,(

Summary

19

• Often one would like to estimate probabilities of future events– Aggregate information from the crowds

• Used in Information/prediction markets

• Strong ties with financial securities trade– Options, etc.

• Some beautiful economic theory behind it.

Today

20

• Prediction markets and scoring rules.

• Some guidelines for the 2nd semester

Grade• 30% presentation• 58% seminar paper• 12% participation

– You will get 12 points if you attend at least 80% of the classes in both the 1st and the 2nd semester.

– If you miss the 80% in one of the semesters, you get 0 points.– You will not get a grade in the course if you attend less than 50% of

the classes in at least one of the semesters.

• 12% problem set (“magen”)

Course duties: presentation

22

• In pairs.– Let the better speaker talk, but two speakers possible.

• Practice– Remember: presentations in real-life are much faster

than the practice talks.

• 25 minute talk. (~3 in each meeting)– Schedule will be posted in the beginning of the semester break.– You should be ready to present in the first week of the semester.

• Make it interesting to the class!– Speak to class, not to instructor.

• Language: Slides in English (preferred, not mandatory), Speak Hebrew.

Course duties: presentation

23

• Goal: present the paper to the class.

– Present what is written in the paper, no need for criticism or your interpretation.

– Give the background, and make the main results clear(do not present all results).

– If suitable, spend at least 5-10 minutes on technical details (proof idea, analysis methods etc.)

– Survey related work very briefly, only if needed. (sometimes it is good to read some of these…)

– Don’t follow the structure of the paper exactly.

– Explain in your own words.

Course duties: presentation

24

• Presentation grading will be based on:– Clarity.– Understanding.– Identifying and understanding the main results.– Quality of presentation.– Conveying complicated ideas in a simple way.– Presenting your own understanding (not just cut

and paste).

“Make everything as simple as possible, but not simpler. ”

Albert Einstein

Some presentation tips.

25

• Use large fonts – in general, rarely use fonts below 24.

• Don’t have dense slides – split them.• A figure is worth a thousand words:

If x is an individually rational revenue maximizing auction then there exists a Pure Nash equilibrium in the game.

• Of course: Be politically correct– No offensive language, images, examples etc.

X is individually rational revenue

maximizing auction

Pure Nash equilibrium exists

Some presentation tips.

26

• Use colors.• Use images.• Use animation

– But with care, do not overuse itplease

• Make it interesting, but remember that this is an academic duty. Content is the main thing. – Most effort should go on understanding the paper you

read.

Some presentation tips.

27

• Basic presentation structure:– Motivation (usually, real life problem, informal) +

Background– Informal results (if possible)– Model– Results– Technical details– Summary

• Of course, you should fit the structure to your paper.

Course duties: Problem set

28

• Optional (Magen 12%)

• Strict deadline: February 1st

• By your own. (not in pairs.)

• Available tomorrow on the website.

Course duties: Seminar paper

29

• Should be on the same topic as your presentation– With the same partner.

• Up to 10 pages, font 12, 1.5 spacing. (strict!)

• Submission time: May 30th.

Course duties: Seminar paper

30

In the paper: Part 1: Summarize the paper.

Background, main results, main technical details .– Some proofs (outline if needed), methods of data

analysis. (Tables with data should not be part of the 10 pages you submit,

can be in an appendix.)– Go to depth regarding the main result/s.

Part 2: A new research direction, – Phrase exact questions and challenges, one (small) step

toward to solution.– Idea may be theoretical, empirical or other. You need to

define the framework of the research (model, goals, questions, difficulties, data resources, suggested theorems)

Course duties: Seminar paper

31

Originality:

• Cite all materials that you use.

• Statements that are not from the paper you read should be given with a bibliographic reference.

• Especially, let me know if you intend writing any project for another course on a similar subject.

• Do not copy text, even from the paper you read.– Explain in your own language. Give new examples. Give

the intuition behind the proofs.

Timeline

32

Winter vacation starts

Winter vacation ends

Summervacation starts

All seminar topics are chosenand approved(January 1st) Students

presentations start(February 16thth)

All seminar papers are submitted(May 30th)

Problem set due date(Feb. 1st)

Presentation outline + hard scheduling constraints sent to Avi.(January 12th)

presentations

Presentation schedule posted(January 20th)

33

Good luck.

Don’t hesitate to come to me for questions or advise.