micro 2012 program chair’s remarks - carnegie …omutlu/pub/onur_program-chair...micro 2012...
TRANSCRIPT
Micro 2012 Program Chair’s Remarks
Onur Mutlu PC Chair
December 3, 2012 Vancouver, BC, Canada
1
Purpose of This Session n Provide insight and transparency into the Micro-45 paper
submission and selection process
n Provide statistics and some informal analyses
n Describe some new things we tried
n Get your feedback – involve the community
n Enable better future Micro conferences q Hopefully, enable the institution of best practices
2
Basic Process and Statistics n 228 submissions; 40 papers accepted (17.5% Acc. Rate)
n Reviewed by q The 44-member Program Committee q The 106-member External Review Committee (fully involved) q At least 98 more external/aiding reviewers
n Review process q 1325 total reviews, 815 from PC members; 5.8 reviews/paper q Rebuttal period to allow for author response q Extensive online discussions (160 papers) q 1.5-day PC meeting on August 25-26, 2012
n All PC members present the entire first day
n After decision q All papers shepherded
3
More Information on the Process … n Now – Program Chair’s Remarks
q Statistics on submitted and accepted papers q Paper selection process q Metrics used to rank papers for discussion q Best paper selection process q New things we tried this year q What worked and what did not? What can we do better?
n Message from the Program Chair (in your proceedings) q http://www.microsymposia.org/micro45/message-from-the-
program-chair.pdf
n Feedback very much welcome q I would appreciate all kinds of honest feedback
4
Our Goals in New Things Tried n Improve quality, transparency; institute best practices
1. Quality of the reviews and paper selection process 2. Transparency of the selection process 3. Strong involvement of external reviewers and community 4. Authors’ ability to respond to initial reviewer evaluations
and potential questions that may come up after rebuttal 5. Quality of the final program and final versions 6. Quality of interactions during conference (especially in the
presence of parallel sessions)
5
The Life of a Micro Submission/Paper n Submission process n Review process n Rebuttal process n Post rebuttal online discussion and re-scoring n PC meeting n Post-PC meeting n During conference n After conference – ad infinitum
6
Paper Submission Process n Double-blind submission and review process n Conflicts marked by authors
n Submission format the same as the final format; no extra page charges q Main goal: eliminate fairness issues… Mitigates the concern
“Does this paper fit in the final format?” q Could be done better in the future
n We did: 12-page 10-pt submissions, 12-page 9-pt final format n Better: 11-page 9pt submissions, 12-page 9pt final format.
7
0 10 20 30 40 50 60 70 80 90 100
I/O (input/output) Quality of service
Security Storage
Mobile systems Dynamic program opGmizaGon
Programmer producGvity; debugging Real system evaluaGon and analysis
System soMware (OS, virtualizaGon) support InstrucGon level parallelism
Data center, cloud, cluster systems Reconfigurable systems
Embedded systems Workload characterizaGon
Thread-‐level parallelism Hardware/soMware interface
Circuit-‐architecture interacGon and codesign Compilers
Graphics and GPUs Special-‐purpose systems, accelerators
Hardware/soMware codesign or cooperaGon Parallel programming; parallelizaGon
Data parallelism; SIMD Modeling, measurement, and simulaGon methods
System architecture Emerging technologies
Reliability, fault tolerance, variability Interconnects and communicaGon mechanisms
Core/processor microarchitecture Energy and power-‐efficiency
Memory systems MulG-‐core systems, mulGprocessors
Number of Papers
Accepted Submi^ed
8
Origin of Papers
0
20
40
60
80
100
120
140
160
180
Academia only Industry only Both
Num
ber o
f Papers
Submi^ed
Accepted
9
Main Country of Papers
0 20 40 60 80 100 120 140 160 180
Brazil Chile
Germany India Italy Japan
PalesGne Belgium
Iran Singapore
Taiwan Israel UK
Cyprus Sweden
Switzerland France Korea
Canada Spain China USA
Number of Papers
Accepted
Submi^ed
10
Number of Authors Per Paper
11
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10
Num
ber o
f Pap
ers
Submi^ed Accepted
Optional Information with Submission n Authors able to provide an appendix
q Peripheral material that can aid reviewers, such as full proofs of theorems, details of the experiments conducted, or more experimental results
q Main goal: Satisfy reviewer curiosity; handle out-of-scope issues
n Authors able to say “this paper was submitted to a past conference”
n Authors able to provide past reviews and responses to them (if the paper was submitted to a previous venue and rejected) q Main goal: proactive addressing of potential concerns that can
come up post-rebuttal… Anticipate and address “I had reviewed this paper previously and the authors did…”
12
Papers with an Appendix
0
50
100
150
200
250
Submi^ed Accepted
Num
ber o
f Papers
Total Papers with an Appendix
13
Papers with Past Reviews
0
50
100
150
200
250
Submi^ed Accepted
Num
ber o
f Papers
Total Papers with Past Reviews
14
Papers with “Submi^ed to Past Conference” Checked
0
50
100
150
200
250
Submi^ed Accepted
Num
ber o
f Papers
Total Papers with “Submi^ed to Past Conference” Checked
15
PC-‐Authored papers
0
50
100
150
200
250
Submi^ed Accepted
Num
ber o
f Papers
Total PC-‐Authored Papers
16
Review Process n Reviewers
q The 44-member Program Committee q The 106-member External Review Committee (fully involved) q At least 98 more external/aiding reviewers
q Review form explicitly asked for three specific rebuttal questions
q External reviewers fully involved in the process, up until the PC meeting
17
Number of Reviews n 1325 total reviews
n 815 from PC members n 510 from external reviewers n All reviewers assigned by me
n 5.8 reviews/paper n 216 papers received >= 5 reviews n 159 papers received >= 6 reviews n 44 papers received >= 7 reviews
18
Reviews Categorized by ExperGse Score
0 100 200 300 400 500 600 700
Li^le familiarity with the subject area
Some familiarity with the subject area
Knowledgeable, but not an expert in the subject area
Expert in the subject area of the paper
Number of Reviews
Accepted Submi^ed
19
Reviews Categorized by Novelty Score
0 100 200 300 400 500 600
Published before
Done before (not necessarily published)
Very incremental improvement
Incremental improvement
New contribuGon
Surprisingly new contribuGon
Number of Reviews
Accepted Submi^ed
20
Can We Publish Your Review?
0
100
200
300
400
500
600
700
800
Yes No
Num
ber o
f Reviews
Submi^ed Accepted
21
Rebuttal Process n Authors’ chance to respond to reviewer concerns after
initial evaluation by reviewers
n Authors were able to see all initial evaluation scores and all comments of reviewers
n All reviewers were required to read and update scores and reviews based on rebuttal n A majority of reviewers updated their reviews and scores, and
provided post-rebuttal comments
q At least 160 papers discussed extensively online post rebuttal q External reviewers fully involved in the discussion
22
Overall Merit Score Categories
23
Pre- and Post-Rebuttal OM Scores n Pre-rebuttal score is the reviewer’s score at the time the
rebuttal is exposed to authors
n Post-rebuttal score is the overall merit score at the time the entire review process is over. Affected by: q Authors’ rebuttal q Online discussion among reviewers q Reading of other reviews q Discussion during the PC meeting q i.e., everything that happens after rebuttal
24
DistribuGon of Review Scores Received by Submi^ed Papers
0 50
100 150 200 250 300 350 400 450 500
Very poor Poor Average Good Very good Excellent
Num
ber o
f Reviews
Pre-‐rebu^al scores Post-‐rebu^al scores
25
DistribuGon of Review Scores Received by Accepted Papers
0
20
40
60
80
100
120
140
Very poor Poor Average Good Very good Excellent
Num
ber o
f Reviews
Pre-‐rebu^al scores Post-‐rebu^al scores
26
Magnitude of Change in Average OM Score AMer Rebu^al vs. Acceptance
0
20
40
60
80
100
120
Lowered by more than 1
Lowered by 0.5 to
1
Lowered by up to
0.5
No change
Raised by up to 0.5
Raised by 0.5 to 1
Num
ber o
f Papers
Rejected
Accepted
27
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 1 – 1.5
28
0
1
2
3
4
Lowered by 1 No Change Raised by 1
Nu
mb
er o
f P
aper
s
Rejected
Accepted
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 1.5 – 2
29
0
0.5
1
1.5
2
2.5
3
3.5
Nu
mb
er o
f P
aper
s
Rejected
Accepted
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 2 – 2.5
30
0
2
4
6
8
10
12
14
16
Nu
mb
er o
f P
aper
s
Rejected
Accepted
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 2.5 – 3
31
0
5
10
15
20
25
30
35
40 N
um
ber
of
Pap
ers
Rejected
Accepted
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 3 – 3.5
32
0
5
10
15
20
25
30
35
Nu
mb
er o
f P
aper
s Rejected
Accepted
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 3.5 – 4
33
0
5
10
15
20
25
30
Nu
mb
er o
f P
aper
s Rejected
Accepted
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 4 – 4.5
34
0
1
2
3
4
5
6
7
8
Lowered by 1 No Change Raised by 1
Nu
mb
er o
f P
aper
s
Rejected
Accepted
Score Changes for Papers w/ Avg. Pre-Rebuttal OM 4.5 – 5
35
0
1
2
Lowered by 1 No Change Raised by 1
Nu
mb
er o
f P
aper
s
Rejected
Accepted
0
10
20
30
40
50
60
Num
ber o
f Pap
ers
Average Overall Merit
Rejected
Accepted
Paper Overall Score DistribuGon, Pre-‐ and Post-‐Rebu^al
Pre-‐Rebu^al
Post-‐Rebu^al
36
Paper Avg. “Novelty” Score DistribuGon
0
10
20
30
40
50
60
70
80 Num
ber o
f Pap
ers
Rejected
Accepted
37
Paper Avg. “ExperGse” Score DistribuGon
0
20
40
60
80
100
120
Num
ber o
f Pap
ers
Rejected
Accepted
38
Paper Avg. “Importance” Score DistribuGon
0
10
20
30
40
50
60
70
80
90
Num
ber o
f Pap
ers
Rejected Accepted
39
Paper Avg. “WriGng Quality” Score DistribuGon
0
10
20
30
40
50
60
70
80
Num
ber o
f Pap
ers
Rejected
Accepted
40
Paper Avg. “Soundness of Ideas” Score DistribuGon
0 10 20 30 40 50 60 70 80 90
100
Num
ber o
f Pap
ers
Rejected Accepted
41
Paper Avg. “Soundness of EvaluaGon” Score DistribuGon
0
10
20
30
40
50
60
70
80
90
Num
ber o
f Pap
ers
Rejected Accepted
42
Program Committee Meeting n 1.5-day meeting
n Main goal: avoid hasty decisions. Improve decision quality. … n Extra day enables mulling over decisions. More time allows for more
and higher-quality discussions. No “lost” PC members.
q August 25-26, Hilton O’Hare airport n All PC members present the entire first day n 42/44 PC members present the second day
q 82 papers discussed in (somewhat of a) rank order n Not average overall merit score rank
q Goal: Consensus across the entire PC for accept/reject decision n Entire PC voted when consensus was not clear
43
Metrics for Paper Discussion Order (I) n Determined after many different ratings are considered
q Average Overall Merit (OM) q Average OM weighted with expertise (2 ways) q Average OM weighted with reviewer generosity q Average OM weighted with reviewer expertise and generosity
(2 ways)
q Total 6 OM metrics per paper q 6 different versions of each OM metric
n [pre-rebuttal, post-rebuttal] x [all reviewers, reviewers with expertise > 1, reviewers with expertise > 2]
q 36 metrics and ranks for each paper n A kitchen-sink metric averages all these n A correlation analysis would be interesting
44
Metrics for Paper Discussion Order (II) n No single best metric n Plus, many things metrics cannot capture n We took all the rankings with a large grain of salt
n Final discussion order determined by examining all metrics and each paper individually q 9 groups of papers formed; 7 of which were discussed
n Bottom line: q Top 72 papers in terms of post-rebuttal rank (with non-
experts excluded) ended up on the discussion list q +7 more with at least one A score but lower averages q +3 more papers brought up by reviewers
45
New Metrics to Rank Papers
46
Problem with Average Overall Merit
All scores are given the same weight • Varying experGse
– Different reviewers have different experGse
• Varying “generosity” – Some reviewers more harsh/generous than others
47
Expert Mean Score
Taking experGse of each reviewer into account • Idea: Give more weight to scores from experts • Weight each score by experGse
48
∑
∑=
reviewsExpertise
reviewsScore * Expertise
Paper a ofMean Expert
“score” refers to the overall merit
Generosity Mean Score Take generosity of each reviewer into account • Idea: A review with a low score compared to other reviews for the paper is considered “less generous”
49
⎟⎟⎠
⎞⎜⎜⎝
⎛=
paper for the scoreMean reviewer by the ScoreMean Reviewer a of Generosity
papers) all(
reviewer theof Generosityscore Review Review a of score Generosity =
reviews ofNumber
review theof score Generosity Paper a ofMean Generosity reviews) (all∑
=
Expert Generosity
• Take experGse and generosity into account • Idea: An expert review with low score is likely not harsh compared to a non-‐expert review with high score
• Two flavors of expert-‐generosity – Pre-‐expert generosity – Post-‐expert generosity
50
Pre-‐expert Generosity Mean
• Account for generosity before experGse • Similar to expert mean • Use generosity score instead of overall merit
51
∑
∑=
reviewsExpertise
reviewsScore Generosity * Expertise
Mean Generosityexpert -Pre
∑
∑=
reviewsExpertise
reviewsScore * Expertise
Paper a ofMean Expert
Post-‐expert Generosity Mean
• Account for experGse before generosity
52
⎟⎟⎠
⎞⎜⎜⎝
⎛=
scoremean Expert reviewer by the ScoreMean Reviewer a of generosity-Expert
papers) all(
reviewer theof generosity-Expertscore Review Review a of score generosity-Expert =
reviews ofNumber
review theof score generosity-Expert Mean Generosityexpert -Post reviews) (all∑
=
Case Study (for Generosity)
Reviewer 1 (Generous)
Reviewer 2 (Neutral)
Reviewer 3 (Harsh)
Mean Average Score
Mean Generosity
Score
Paper 1 (Good) 6 3 4.5 4.57
Paper 2 (Average) 5 3 4 3.38
Paper 3 (Bad) 2 1 1.5 1.71
Generosity 1.29 1.04 0.67
Accounting for generosity increases the gap between the good paper that received a harsh review AND the average paper that received a generous review.
53
Post PC-Meeting n Summary statements for discussed and rejected papers
q Written by myself or an expert PC member q Goal: provide insight/visibility for authors into the discussion
of the paper in the PC meeting… Provide major reasons for rejection.
q Can be done more methodically in the future: assign a scribe for each paper
n All papers shepherded q Goal: improve quality of final program q Did achieve its purpose in many cases
54
During Conference n Goal:
q Improve exposure of papers in the presence of parallel sessions
q Enable better technical interactions q Enable better “Best *” decisions
n Lightning session
n Poster session
55
Best Paper Award Process n 8 papers chosen as candidates based on
q Rankings (based on 36 different metrics) q PC member vetting q Reviewer vetting
n 7-person Best Paper Award Committee q Not conflicted with any of the 8 candidate papers q Can select another paper as the winner q Can select 0-N papers q The same committee will select the Best Lightning Session
Presentation and Best Poster Award winners n Sessions to aid the selection process
56
Feedback n Any type of feedback on any part of the process is very
much appreciated
n Goal: Living document that continually improves the process and institutes best practices
n Methods of providing feedback q Online feedback form
n http://www.surveymonkey.com/s/Z8J6FXT
q In person q Via email q Snail mail! q …
57
Micro-45 Survey Link on Website www.microsymposia.org/micro45
58
Thanks n Literally hundreds of people
n PC, ERC, Steering Committee n All reviewers n All submitting authors n All presenters n All attendees n Many others (see my note)
n Strong and thriving community effort
59
Thanks n These slides were prepared in part by
n Vivek Seshadri, Carnegie Mellon n Chris Fallin, Carnegie Mellon n Justin Meza, Carnegie Mellon
60
Thank You.
Onur Mutlu PC Chair
December 3, 2012 Vancouver, BC, Canada
61
Vancouver Aquarium Tonight – Buses leave from 6:45pm to 7:45pm, return from 10pm to midnight. Extra Tickets Available
62
63
Micro 2012 Program Chair’s Remarks
Onur Mutlu PC Chair
December 3, 2012 Vancouver, BC, Canada
64
Additional Data
65
Reviews Categorized by Magnitude of Score Change
0
200
400
600
800
1000
1200
Num
ber o
f Reviews
Submi^ed Accepted
66
Reviews Categorized by Magnitude of Score Change (Detail)
0 20 40 60 80
100 120 140 160 180 200
Num
ber o
f Reviews
Submi^ed Accepted 1003 215
67
Papers Categorized by Post-‐Rebu^al Avg. Score Change DirecGon
0
20
40
60
80
100
120
140
Moved up Moved down No change
Num
ber o
f Papers
68
Papers Categorized by Post-‐Rebu^al Avg. Score Change DirecGon
0
20
40
60
80
100
120
140
Papers Whose Score Moved Up
Papers Whose Score Moved Down
Papers Whose Score Did Not Change
Num
ber o
f Papers
Accepted Rejected
2.87 4.37 3.97
3.94
2.72
1.97
(number on top indicates average pre-‐rebu^al score across all papers in the respecGve group) 69