graduate school of information science and technology, osaka university, japan norihisa komoda,...

of 23/23
Graduate School of Information Science and Technology, Osaka University, Japan Norihisa Komoda, Shingo Tamura, Yoshitomo Ikkai, Koichi Higuchi 20th Workshop on Methodologies and Tools for Complex System Modeling and Integrated Policy Assessment (CSM2006) 2006.8.28-30: Laxenburg, Austria A support tool for composing questionnaires in social survey data archive SRDQ

Post on 27-Dec-2015

213 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • Slide 1
  • Graduate School of Information Science and Technology, Osaka University, Japan Norihisa Komoda, Shingo Tamura, Yoshitomo Ikkai, Koichi Higuchi 20th Workshop on Methodologies and Tools for Complex System Modeling and Integrated Policy Assessment (CSM2006) 2006.8.28-30: Laxenburg, Austria A support tool for composing questionnaires in social survey data archive SRDQ
  • Slide 2
  • 1.Social Survey Data Archive The social survey data archive is an archive that collects, storages and disseminates lots of social survey data such as Social Network Survey. Each survey data contains various types of items such as question items, dataset (answers of respondents), sample design, and papers/repots about the survey. Objectives Maintaining the quality of social surveys When composing questionnaires for new surveys, it is imperative to review question items and dataset of existing surveys for maintaining the quality. Effective use of existing data It reduces the need to conduct repetitious surveys for similar purposes. Thus large amounts of survey costs can be eliminated. Education The archive makes it possible to develop social survey methodology lessons using high quality survey data.
  • Slide 3
  • 1.Social Survey Data Archive: SRDQ SRDQ: the Social Research Database on Questionnaires One of the most advanced social survey data archive in Japan. (http://srdq.hus.osaka-u.ac.jp/en)http://srdq.hus.osaka-u.ac.jp/en developed by Graduate School of Human Science, Osaka University in 2003.
  • Slide 4
  • 1.Social Survey Data Archive: SRDQ Hierarchical textual data Searching system (string search) Dataset analysis system (crosstab, etc.) Subjects, sample designs, papers & reports of each survey are also stored Simple string search of question items or surveys Infor- mation Information Society Survey Question item 1 Qeestion item 2 119 surveys 17,232 items Subjects: Information, Social Psychology, etc. Question Items, Dataset, Papers & Reports. Social Network Survey Subjects: Information, Family, etc. Question Items, Dataset, Papers & Reports. SRDQ: the Social Research Database on Questionnaires Specifications Class
  • Slide 5
  • 1.Social Survey Data Archive: SRDQ SRDQ allows the direct analysis of dataset over the web pages Example: Crosstab analysis 3020 10050 A B male female Alternatives of column item alternatives of row item 30 male answered A. Crosstab analysis In this window, the row and column items of the crosstab are selected.
  • Slide 6
  • 1.Social Survey Data Archive: SRDQ An analyst selects the variables he/she wants to use, then push >>. Finally, the analyst pushes crosstabs for starting analysis. How to execute Crosstab analysis by using SRDQ
  • Slide 7
  • 1.Social Survey Data Archive: SRDQ The Result: 47% of male use PC, while only 29% of female do. Row: Gender Column: PC
  • Slide 8
  • Select exiting surveys or question items to compare with new ones 2. Purpose of the Study To make SRDQ more useful, we planed to add a new function to help researchers in composing new questionnaires. Procedures to compose a new questionnaire: Summarize existing question items Decide the purpose and the design Create new question items Decide the order of question items 3 man-month 0.5 man-month 0.75 man-month 0.25 man-month Survey containing 200 - 300 questions Intermittent discussion by research group members (approx. 10 researchers), Continue 3 12 months. Search for related surveys A tool to support this process has been developed. Summarize existing question items
  • Slide 9
  • Select exiting surveys or question items to compare with new ones 2. Purpose of the Study Procedures to compose a new questionnaire: Summarize existing question items Decide the purpose and the design Create new question items Decide the order of question items A tool to support this process has been developed. In this process, Summary of Question Items is used.
  • Slide 10
  • 2. Summary of Question Items Do you use the following items? E-Mail Question Items ISS 2001 ISS 2002 JGSS 2003 q3 q22a q45q22b Trends of the question items and differences between the surveys become clear. q1b q1f q1a Do you use the following items? Fax Do you use the following items? Home Page q1. Do you use the following items? a. E-Mail b. Fax f Home Page q3.Do you use e-mail on your cell phone or PC 1. yes 2. no q45.Do you use Home Page on your cell phone or PC 1. yes 2. no Information Society Survey 2001Information Society Survey 2002 Summary of Question Items is a synopsis of similar question items included in particular surveys. surveys Break down the question items to the minimum units (red underlined). And summarize the similar items/units. SRDQ Searched with keywords and name of surveys
  • Slide 11
  • It takes approx. 1 week to process only 3 or 4 surveys manually. Goal The automatic creation of the summary that is sufficiently accurate to meet the demands of social survey specialists. And, the provision of the editing interface to correct the errors and to produce a final, completed summary in less time. Surveys Summary of Question Items Evaluation of accuracy: E = W * Non-Detection items + Miss Detection items ( W > 1 ) Number of rows includes detection errors should be under 10% Question Items ISS 2001 ISS 2002 JGSS 2003 q3 q22a q45q22b q1b q1f q1a Do you use the following items? E-Mail Do you use the following items? Home Page Do you use the following items? Fax 2.Support System for the Summarization
  • Slide 12
  • 3.Overview of the System Input Surveys + Keywords (ex. Survey A, B, C, D + mail ) Output 1. How often do you use e-mail for each of the purposes listed below? 1.1 business communication a. every day b. 3 or more days a week 1.2 Personal communication with friends 1.3 Personal communication with family 19 question items about mail ABC D q2 q3 q15q23 ABCD For similarity judgments of question items, Jaccard Coefficient is used. Search Target (several surveys) a. every day b. 3 or more days a week a. every day b. 3 or more days in a week
  • Slide 13
  • Original Method Jaccard Coefficient: J = a / (a+b+c) a: number of common words between 2 question items b, c: number of words which appear in only 1 question items Q.A1 Q.A2 Q.C1 Q.C2 Q.A1 Q.B1 Q.A2 Q.C1 Q.C2 Q.A1Q.B1 : Maximum similarity Calculate similarity for all combination of question items in target surveys The pair which has maximum similarity value will be judged as similar. (Repeat this step while similarity values are higher than the threshold) Q.B1 Q.B2 Similarity Judgment 3. Similarity Judgments by Jaccard Coefficient
  • Slide 14
  • 3.Difficulty of Similarity Judgments How often do you use e-mail for personal communication with friends? How often do you use e-mail for personal communication with family? 2. Almost all words are same except one core word, but the intended purposes of the questions are different. 3. Different expression, but asking the same thing. Do you perform following actions in your everyday life? Reuse bathwater for laundering to conserve water. Do you try to do things in this list? Saving resources such as water. 1. Partial match in juxtaposed words How often do you do the things on this list? Practice flower arranging, tea ceremony, or calligraphy Do you practice cooking, sewing, or calligraphy? Survey A Non- Detection Non Detection Miss Detec- tion 1. Treat juxtaposed words as a group 2. Apply a penalty if core words dont match 3.Apply neighborhood bonus for word matches Survey B Survey A Survey B Survey A Survey B
  • Slide 15
  • 3.Similarity Judgments (1/2) 1. Treat juxtaposed words as a group Juxtaposed words can be viewed as a group If one or more words matches in juxtaposed words, treat those words as a group and ignore unmatched words when calculating similarity If one or more words matches in juxtaposed words, treat those words as a group and ignore unmatched words when calculating similarity 2. Apply penalty if core words dont match For pairs of similar question items within one survey, if only a few words differs, that words are recognized as core words. Q1-a. How often do you use e-mail for each of the purposes? communication with family Q1-a. How often do you use e-mail for each of the purposes? business communication core words Q1-a. How often do you use e-mail for each of the purposes? communication with friends Dont Match Survey B Survey A Under specific conditions, values of existing Jaccard coefficient are adjusted. New similarity measure which uses structural characteristics of surveys Penalty If a pair within one survey has similarity value higher than 0.6, un-matched words are recognized as core words. Detect core words before calculating similarity, and decrease similarity value if core words dont match. Detect core words before calculating similarity, and decrease similarity value if core words dont match.
  • Slide 16
  • 3.Similarity Judgments (2/2) There is significance to the order of the question items. Question items having the same meaning tend to be arranged in the same order. Increase similarity values if highly similar pairs are found in the neighborhood Question items in the same hierarchical positions 3.Apply neighborhood bonus for word matches Q7. Do you perform following actions in your daily life? 1.Turn off lights not in use 2. Reuse bathwater for laundering to conserve water. Q2. Do you try to do things in this list? 1. Always turn off lights not in use. a. yes b. no 2. Saving resources such as water. a. yes b. no Survey A Survey B High similarity value 1. Do you use e-mail on your pc? 2. How often do you use e-mail? 1. Do you use e-mail on your cell phone or pc? 2. How many times do you send/receive e-mails? 2-1. To get info about everyday life 2-a. Gathering info for daily life High similarity value Bonus Survey C Survey D
  • Slide 17
  • 4.Evaluation of Similarity Judgments (1/2) 36 question items about environmental protection (from 3 surveys) T = 0.5 Threshold value of similarity judgments Detection Errors Non- Detection Miss Detection ERows Rows contain errors Jaccard87122305 Proposed2022222 Non-detection: a pair was judged as not similar while it should be judged as similar Miss detection: a pair was judged as similar while it should be judged as not similar Penalty: 0.5, Neighborhood bonus: 0.3 T = 0.6 Evaluation: E = W (3 = number of surveys) * Non-Detection + Miss Detection Detection Errors Non- Detection Miss Detection ERows Rows contain errors Jaccard65116285 Proposed2022222 Compare correct result manually prepared with result using proposed measure and result using Jaccard coefficient only Non-detections are more problematic than miss detections
  • Slide 18
  • 4.Evaluation of Similarity Judgments (2/2) 113 question items about Leisure (from 10 surveys) T = 0.5 Threshold value of similarity judgments Detection Errors Non- Detection Miss Detection ERows Rows contain errors Jaccard363513517017 Proposed19154154507 Non-detection: a pair was judged as not similar while it should be judged as similar Miss detection: a pair was judged as similar while it should be judged as not similar T = 0.6 Evaluation: E = W (10 = number of surveys) * Non-Detection + Miss Detection Detection Errors Non- Detection Miss Detection ERows Rows contain errors Jaccard353323326715 Proposed101919433 Non-detection & miss detection are reduced, and thus E is improved Number of rows containing detection errors is under 10% The efficiency of the proposed method has been confirmed. Penalty: 0.5, Neighborhood bonus: 0.3
  • Slide 19
  • 5.Editing Interface Possible non-detection: does not exceed the threshold but the value is close to the threshold Select item to move Specify the destination Click to open an editing window Possible miss detection : exceeds the threshold but the value is close to the threshold value 0.5 0.6 0.4 0.5 The prototype tool has been developed. The editing interface is build as CGI script.(Perl). scrolling total 10 surveys
  • Slide 20
  • 5. Editing Interface Possible miss detection Possible non-detection Moved an item to a new row (the last row) to correct a detection error. scrolling total 10 surveys Survey E scrolling moved
  • Slide 21
  • 5.Evaluation Test of the Editing Interface Manual Proposed System Time taken to create a correct summary 3 hours 20 minutes view & check the question items 15 min. move the items to correct errors 5 min. 3 rows contain detection errors 10 question items are moved Possible miss detection: 6 items Possible non-detection: 22 items (All detection errors are displayed as these possible error) Possible miss detection Possible non-detection Evaluation test: compare the time taken to create the summary by hand with the time using the proposed system / interface. Material: 113 question items about Leisure ( from 10 surveys ) Contains 1 non-detection and 9 miss detection ( T = 0.5 ).
  • Slide 22
  • 6. Conclusions Using structural characteristics of social survey questionnaires, we have developed a support tool for generation of the summary of question items. The proposed method is capable of automatically creating the summary that is sufficiently accuracy to meet the demands of specialists. With the man-machine interface system, final and completed summaries can be generated in less time than manual means.
  • Slide 23
  • Thank you for your kind attention. Really? Great ! Doubtful ! Check detail. 20th Workshop on Methodologies and Tools for Complex System Modeling and Integrated Policy Assessment (CSM2006) 2006.8.28-30: Laxenburg, Austria
  • Slide 24
  • x.Details on Detection Errors How many times do you send/receive e-mails for following purposes? Personal communication with individuals whom you see frequently How often do you use e-mail? To keep in touch with people whom you often meet Cause: Phrased very differently Survey A Non-Detection: a pair was judged as not similar while it should be judged as similar Prepare a thesaurus Decease threshold value Increase neighborhood bonus value Non-d. Miss d. Jaccard84 Proposed86 Test: 113 question items about Leisure (from 10 surveys) Errors 2 15 11 8 Bonus applied to word matches regardless of context / usage. Smart application of the bonus: Increase the bonus value considering how near to the highly similar pair, and how similar the The bonus increased to 0.6 Survey B
  • Slide 25
  • x. Deletion of Unnecessary Part (2) Response categories Similarity value decrease if dont delete response categories Ex. Do you use the following items? Please tell me all the items you use 1 E-Mail 2 PC 8 Fax Do you use E-mail on your pc? Survey A Survey B (1) Phrases irrelevant to the subject of questions How often do you use e-mail for each of the purposes listed below? How often do you use e-mail for each of the purposes listed below? Please tell me Every day , At least once a week,,,, While some response categories are irrelevant to the subject of questions, others are necessary parts of the questions. 1. Yes 2. No Delete response categories irrelevant to the subject. And treat each of the necessary response categories as question items (break down the question items). Necessary response categories Irrelevant to the subject of questions Delete irrelevant phrases. Survey A Delete Survey B Difficulty
  • Slide 26
  • x. Work Flow of Similarity Judgments Input Surveys + Keywords (ex. Survey A, B, C, D + mail ) Output ABCD word extraction (morphological parse) Search Target (several surveys) Selected Question Items similarity judgments deletion of some phrases N of common words are checked possible miss detection (incorrectly judged as similar) possible non-detection (incorrectly judged as not similar) Response Categories: agree, strongly agree, etc. Questions: Using the five-point scale on the card, please, Delete unnecessary part to judge similarity specific phrases which are common to many question items 1. How often do you use e-mail for each of the purposes listed below? 1.1 business communication a. every day b. 3 or more days a week 1.2 Personal communication with friends 1.3 Personal communication with family 19 question items about mail ABC D q2 q3 q15q23 a. every day b. 3 or more days a week a. every day b. 3 or more days a week q16 q7 When clicked, actual questions & response categories of each survey are displayed