graduate school of information science and technology, osaka university, japan norihisa komoda,...

Graduate School of Information Science and Technology, Osaka University, Japan

Norihisa Komoda, Shingo Tamura, Yoshitomo Ikkai, Koichi Higuchi

20th Workshop on Methodologies and Tools for Complex System Modeling and Integrated Policy Assessment (CSM2006) 2006.8.28-30: Laxenburg, Austria

A support tool for composing questionnaires in social survey

data archive SRDQ

1.Social Survey Data Archive

The social survey data archive is an archive that collects, storages and disseminates lots of social survey data such as “Social Network Survey”.Each survey data contains various types of items such as question items, dataset (answers of respondents), sample design, and papers/repots about the survey.

Objectives

Maintaining the quality of social surveysWhen composing questionnaires for new surveys,

it is imperative to review question items and dataset of existing surveys for maintaining the quality.

Effective use of existing dataIt reduces the need to conduct repetitious surveys for similar purposes.

Thus large amounts of survey costs can be eliminated.

EducationThe archive makes it possible to develop social survey methodology lessons using high quality survey data.

1.Social Survey Data Archive: “SRDQ”“SRDQ”: 　 the Social Research Database on Questionnaires

One of the most advanced social survey data archive in Japan. (http://srdq.hus.osaka-u.ac.jp/en)

developed by Graduate School of Human Science, Osaka University in 2003.

http://srdq.hus.osaka-u.ac.jp/en

1.Social Survey Data Archive: “SRDQ”

•Hierarchical textual data

•Searching system (string search)

•Dataset analysis system (crosstab, etc.)

•Subjects, sample designs, papers & reports of each survey are also stored

Simple string search of question items or surveys

Infor-mation

“Information Society Survey”

・・・

Question item 1

Qeestion item 2

・・・

119 surveys 17,232 items

Subjects: Information, Social Psychology, etc.

Question Items, Dataset, Papers & Reports.

“Social Network Survey”Subjects: Information, Family, etc.

Question Items,Dataset, Papers & Reports.

“SRDQ”: 　 the Social Research Database on Questionnaires

Specifications

Class

・・・

1.Social Survey Data Archive: “SRDQ”SRDQ allows the direct analysis of dataset over the web pages

Example: Crosstab analysis

30 20

100 50

A B

male

female

Alternatives of column itemalternatives

of row item

30 male answered “A”.

Crosstab analysis

In this window, the row and column items of the crosstab are selected.


An analyst selects the variables he/she wants to use, then push “>>”.Finally, the analyst pushes “crosstabs” for starting analysis.

How to execute Crosstab analysis by using SRDQ


The Result:

47% of male use PC, while only 29% of female do.

Row:Gender

Column:PC

Select exiting surveys or question items to compare with new ones

2. Purpose of the StudyTo make SRDQ more useful, we planed to add a new function to help researchers in composing new questionnaires.

Procedures to compose a new questionnaire:

Summarize existing question items

Decide the purpose and the design

Create new question items

Decide the order of question items

3 man-month

0.5 man-month

0.75 man-month

0.75 man-month

0.25 man-monthSurvey containing 200 - 300 questions

Intermittent discussion by research group members (approx. 10 researchers),Continue 3 – 12 months.

Search for related surveys

A tool to support this process has been developed.


Select exiting surveys or question items to compare with new ones

2. Purpose of the Study

Procedures to compose a new questionnaire:


Decide the purpose and the design

Create new question items

Decide the order of question items

A tool to support this process has been developed.

In this process, “Summary of Question Items” is used.

2. Summary of Question Items

Do you use the following items? E-Mail

Question Items ISS2001

ISS2002

JGSS2003

q3 q22a

q45

q22b

・・Trends of the question items and differences between the surveys become clear.

q1b

q1f

q1a

Do you use the following items? Fax

Do you use the following items?Home Page

q1. Do you use the following items?　　　　 a. E-Mail　　　　 b. Fax

f ． Home Page・・

q3.Do you use e-mail on your cell phone or PC

1. yes 　 2. no

q45.Do you use Home Page on your cell phone or PC

1. yes 　 2. no

・・

Information Society Survey 2001 Information Society Survey 2002

“Summary of Question Items” is a synopsis of similar question items included in particular surveys.

surveys

Break down the question items to the minimum units (red underlined).And summarize the similar items/units.

SRDQ

Searched with keywords and name of surveys

It takes approx. 1 week to process only 3 or 4 surveys manually.

Goal The automatic creation of the summary that is sufficiently accurate to meet the demands of social survey specialists. And, the provision of the editing interface to correct the errors and to produce a final, completed summary in less time.

SurveysSummary of Question Items

Evaluation of accuracy:

　 E = W * Non-Detection items + Miss Detection items ( W > 1 )

　 Number of rows includes detection errors should be under 10%

Question Items ISS2001

ISS2002

JGSS2003

q3 q22a

q45

q22b

q1b

q1f

q1aDo you use the following items? E-Mail

Do you use the following items?Home Page

Do you use the following items? Fax

2.Support System for the Summarization

3.Overview of the System

Input Surveys + Keywords

(ex. Survey A, B, C, D + “mail” )

Output1. How often do you use e-mail for each of the purposes listed below?

1.1 business communication a. every day b. 3 or more days a week

1.2 Personal communication with friends

1.3 Personal communication with family

19 question items about “mail”

A B C D

q2 q2 q2

q3 q3

q15 q23

A B C D

For similarity judgments of question items, “Jaccard Coefficient” is used.

Search Target(several surveys)

a. every day b. 3 or more days a week

a. every dayb. 3 or more days in a week

Original Method

Jaccard Coefficient: 　 J = a / (a+b+c)a: number of common words between 2 question itemsb, c: number of words which appear in only 1 question items

Q.A1

Q.A2

Q.C1

Q.C2・・

Q.A1 ， Q.B1

Q.A2

Q.C1

Q.C2

Q.A1 Q.B1: Maximum similarity

•Calculate similarity for all combination of question items in target surveys•The pair which has maximum similarity value will be judged as “similar”. (Repeat this step while similarity values are higher than the threshold)

・・

Q.B1

Q.B2

・・・・

Q.B2

・・・・・・

Similarity Judgment

3. Similarity Judgments by Jaccard Coefficient

3.Difficulty of Similarity Judgments

•How often do you use e-mail for personal communication with friends?

•How often do you use e-mail for personal communication with family?

2. Almost all words are same except one core word, but the intended purposes of the questions are different.

3. Different expression, but asking the same thing.•Do you perform following actions in your everyday life?

Reuse bathwater for laundering to conserve water.

•Do you try to do things in this list? Saving resources such as water.

1. Partial match in juxtaposed words•How often do you do the things on this list? Practice flower arranging, tea ceremony, or calligraphy

•Do you practice cooking, sewing, or calligraphy?

Survey A

Non-Detection

Non Detection

MissDetec-tion

1. Treat juxtaposed words as a group

2. Apply a penalty if core words don’t match

3.Apply “neighborhood bonus” for word matches

Survey B

Survey A

Survey B

Survey A

Survey B

3.Similarity Judgments (1/2)

1. Treat juxtaposed words as a group

Juxtaposed words can be viewed as a group

If one or more words matches in juxtaposed words, treat those words as a group and ignore unmatched words when calculating similarityIf one or more words matches in juxtaposed words, treat those words as a group and ignore unmatched words when calculating similarity

2. Apply penalty if core words don’t match

For pairs of similar question items within one survey,if only a few words differs, that words are recognized as core words.

Q1-a. How often do you use e-mail for each

of the purposes? communication with family

Q1-a. How often do you use e-mail for eachof the purposes? business communication

core words

Q1-a. How often do you use e-mail for eachof the purposes? communication with friends

Don’t Match・・

・・

Survey BSurvey A

Under specific conditions, values of existing Jaccardcoefficient are adjusted.

New similarity measure which uses structural characteristics of surveys

Penalty

If a pair within one survey has similarity value higher than 0.6, un-matched words are recognized as core words.

Detect core words before calculating similarity,and decrease similarity value if core words don’t match.Detect core words before calculating similarity,and decrease similarity value if core words don’t match.

3.Similarity Judgments (2/2)

There is significance to the order of the question items. Question items having the same meaning tend to be arranged in the same order.

Increase similarity values if highly similar pairs are found in the neighborhoodIncrease similarity values if highly similar pairs are found in the neighborhood

Question items in the same hierarchical positions

3.Apply “neighborhood bonus” for word matches

Q7. Do you perform following actions in your daily life?

1.Turn off lights not in use

2. Reuse bathwater for laundering to conserve

water.

・・

Q2. Do you try to do things in this list?

　　 1. Always turn off lights not in use.

　　　　　 a. yes 　　 b. no

　　 2. Saving resources such as water.

　　　　　　　　　　　 a. yes 　　　 b. no

Survey A Survey B

・・

High similarity value

1. Do you use e-mail on your pc?

2. How often do you use e-mail?

1. Do you use e-mail on your cell phone or pc?

2. How many times do you send/receive e-mails?

2-1. To get info about everyday life

・・

　　 2-a. Gathering info for daily life

High similarity value

・・

BonusSurvey C Survey D

4.Evaluation of Similarity Judgments (1/2)

36 question items about environmental protection (from 3 surveys)

T = 0.5

Threshold value ofsimilarity judgments

DetectionErrors

Non-

Detection

MissDetection

E Rows Rows contain errors

Jaccard 8 7 1 22 30 5

Proposed

2 0 2 2 22 2

•Non-detection: a pair was judged as not similar while it should be judged as similar•Miss detection: a pair was judged as similar while it should be judged as not similar

Penalty: 0.5, “Neighborhood bonus”: 0.3

T = 0.6

Evaluation: E = W (3 = number of surveys) * Non-Detection + Miss Detection

DetectionErrors

Non-

Detection

MissDetection


Jaccard 6 5 1 16 28 5

Proposed

2 0 2 2 22 2

•Compare correct result manually prepared with result using proposed measure and result using Jaccard coefficient only

•Non-detections are more problematic than miss detections

4.Evaluation of Similarity Judgments (2/2) 113 question items about Leisure (from 10 surveys)

T = 0.5

Threshold value ofsimilarity judgments

DetectionErrors

Non-

Detection

MissDetection


Jaccard 36 35 1 351 70 17

Proposed 19 15 4 154 50 7

•Non-detection: a pair was judged as not similar while it should be judged as similar•Miss detection: a pair was judged as similar while it should be judged as not similar

T = 0.6

Evaluation: E = W (10 = number of surveys) * Non-Detection + Miss Detection

DetectionErrors

Non-

Detection

MissDetection


Jaccard 35 33 2 332 67 15

Proposed 10 1 9 19 43 3

•Non-detection & miss detection are reduced, and thus E is improved•Number of rows containing detection errors is under 10%

The efficiency of the proposed method has been confirmed.

Penalty: 0.5, “Neighborhood bonus”: 0.3

5.Editing Interface

Possible non-detection: does not exceed the threshold but the value is close to the threshold

Select item to moveSpecify the destination

Click to open an editing window

Possible miss detection: exceeds the thresholdbut the value is close to the threshold value

0.5 ～ 0.6

0.4 ～ 0.5

The prototype tool has been developed.

The editing interface is build as CGI script.(Perl).

scrolling

scrolling

total 10 surveys

5. Editing Interface

Possible miss detection

Possible non-detection

Moved an item to a new row (the last row) to correct a detection error.

scrolling total 10 surveysSurvey E

scrolling

moved

5.Evaluation Test of the Editing Interface

　　　　　　　　　　　　 Manual 　 Proposed System

Time taken to create a correct summary 　 3 hours 　　　 20 minutes

view & check the question items 15 min.

move the items to correct errors 5 min.

3 rows contain detection errors10 question items are moved

Possible miss detection: 6 itemsPossible non-detection: 22 items(All detection errors are displayed as these “possible error”)

Possible miss detection

Possible non-detection

Evaluation test: compare the time taken to create the summary by hand with the time using the proposed system / interface.

Material: 113 question items about Leisure ( from 10 surveys ) Contains 1 non-detection and 9 miss detection ( T = 0.5 ).

6. Conclusions

Using structural characteristics of social survey questionnaires, we have developed a support tool for generation of the “summary of question items”.

The proposed method is capable of automatically creating the summary that is sufficiently accuracy to meet the demands of specialists.

With the man-machine interface system, final and completed summaries can be generated in less time than manual means.

Thank you for your kind attention.

Really?

Great !

Doubtful !

Check detail.

20th Workshop on Methodologies and Tools for Complex System Modeling and Integrated Policy Assessment (CSM2006) 2006.8.28-30: Laxenburg, Austria

graduate school of information science and technology, osaka university, japan norihisa komoda,...

Documents