nov1 webinar intro_slides v
TRANSCRIPT
Digital Scholar
Webinar
November 1st, 2017
Hosted by the Southern California Clinical and Translational Science Institute (SC CTSI)
University of Southern California (USC) and Children’s Hospital Los Angeles (CHLA)
Katja Reuter, PhD,
Director of the Digital
Scholar Program
About Today’s Session
TurkPrime
Web link: https://www.turkprime.com
Today’s Learning Objectives
Understand the methodological and other advantages of using Mechanical
Turk
Understand how TurkPrime’s Toolkit significantly expands Mechanical
Turk’s functionality
Understand how PrimePanels helps to overcome the limitations of
Mechanical Turk’s population and sampling
Describe the tradeoffs of using MTurk, TurkPrime, and PrimePanels
Leib Litman, PhD
Today’s Speaker
Topic: Using the research platform TurkPrime to
crowdsource data for the health sciences
Speaker: Leib Litman, PhD,
Associate Professor of Psychology, Lander College
Director of Research, TurkPrime
Questions: Please use the Q&A
Feature
1. Click on the tab here to
access Q&A
2. Ask and post question here
1
2
Online Data Collection
o Research in numerous scientific fields relies on online sources of
human participants (Bohannon, 2016).
o Mechanical Turk (MTurk) is currently the most commonly used
participant recruitment platform.
o While MTurk is the original and most popular online participant
recruitment platform it has several limitations.
o In this talk we will describe:
– MTurk’s API-based add-on tools available to supplement MTurk
functionality
– Alternative platforms which provide access to larger and more
diverse populations
– Strengths and weaknesses of each
Outline of today’s talk
o Describe the advantages of Mechanical Turk
– Sample study: Examining the relationship between the
internalization of media ideals of beauty and disordered eating
– A brief look at the data: Replication and data quality
o Describe the TurkPrime Toolkit
– How to create a TurkPrime account and link it to Mechanical
Turk
– Describe several Toolkit features
o Describe PrimePanels
– Strength of PrimePanels
– Strengths and weaknesses of PrimePanels and MTurk samples
o Not a detail tutorial
Mechanical Turk
Technical challenges Sampling challenges
Mechanical Turk
Exclusion from previous/simultaneous
studies, longitudinal, communication,
sampling, data quality, anonymity.
Technical challenges
TurkPrime Toolkit
Solution
Mechanical Turk
Exclusion from previous/simultaneous
studies, longitudinal, communication,
sampling, data quality, anonymity.
Technical challenges
Mechanical Turk Sample study
Suppose a researcher wants to conduct a study examining the relationship
between Thin-ideal internalization and disordered eating.
Research question: Is there a relationship between media ideal
internalization and disordered eating, controlling for covariates such as
depression?
This study addresses issues relevant to various disciplines including clinical,
health and social psychology, sociology, psychiatry, and public health.
Methods are representative of many HITs that are posted on Mechanical Turk
by social and behavioral scientists.
Why conduct this study on Mechanical
Turk?
o Mechanical Turk is a convenient and easily accessible source of participants
and can be used to selectively recruit specific populations such as women.
o Once launched, data collection is expected to complete within a few hours.
o The cost is expected to be less than $200.
o The stimuli used in the study can be easily developed online using survey
platforms, such as Qualtrics, and then seamlessly linked to Mechanical Turk.
o Once data are collected a fully labeled csv or SPSS file will be available for
download immediately, saving time over traditional pencil and paper
approaches which involve time consuming manual data entry.
Methodological advantages.
o MTurk samples are more representative compared to typical
samples collected from the undergraduate subject pool.
o The sample is expected to be more diverse in terms of age,
race, education, occupation and geographical location.
o Ability to follow up with the sample longitudinally with relatively
low dropout over time.
o Easily conduct follow-up studies to examine how the relationship
between thin- ideal internalization and disordered eating may
interact with factors such as age, education, SES, religious
background and many others.
Mechanical Turk is more than a platform for conducting a single
study, but rather an environment on which a phenomenon of
scientific interest can be systematically explored across a series of
studies using a wide range of research methodologies.
Basic design and methodology
The study uses survey instruments to
measure multiple constructs
Thin ideal internalization will be
measured with the SATAQ-3
Disordered eating will be measured with the
EAT-26.
Depression will be used as a covariate, and measured using
the BDI.
We will also ask various demographic
and personal information questions.
SATAQ – Social attitudes towards
appearance questionnaire
on
Thompson, J. K., van den Berg, P., Roehrig, M., Guarda, A. S., & Heinberg, L. J. (2004). The sociocultural attitudes towards appearance scale‐3 (SATAQ‐3): Development and validation. International Journal of Eating Disorders, 35(3), 293-304.
EAT– Eating Attitudes Test
Garner, D. M., Olmsted, M. P., Bohr, Y., & Garfinkel, P. E. (1982). The eating attitudes test: psychometric features and clinical correlates. Psychological medicine, 12(4), 871-878.
BDI– Beck depression Inventory
MTurk does not provide sophisticated stimulus
development software. Most studies are designed
on Qualtrics, Survey Monkey Survey Gizmo etc.
o Final block structure in Qualtrics. Creating studies with Qualtrics is not
covered in this webinar, but a Power Point Qualtrics tutorial is available
upon request.
Once a link has been generated, it can be sent to study participants.
Clicking on this link or pasting it into the address bar will take a
participant to the survey. This link should be copied and pasted on
MTurk to distribute to MTurk participants.
Creating an MTurk account
Click on sign in as a Requester
Create Pageo On the left side there are a series of options for different study types. Most of these are
not relevant for psychological research. The Survey Link option is for a study that
allows a Requester to paste a link to a study hosted on a third party platform, such as
Qualtrics.
1.Click Survey Link 2. Click Create Project
Enter Properties tabo Creating a study requires sequentially filling fields for three tabs.
In the Enter Properties tab under Create, fill out basic information about the
study’s specifications.
Enter Properties tabo The drop-down menu lists system (MTurk) qualifications, including
gender.
Launching the studyo After a few confirmation screens you will see the option to publish the
study.
Monitoring the study o You are able to see the study’s progress, and average completion time.
Brief overview of resultso The full dataset was collected within 3 hours.
o The Cronbach’s alpha reliabilities for the EAT, SATAQ, and BDI were
.75, .89 and .95 respectively and are all well within acceptable norms.
o Disordered eating, thin-ideal internalization, and depressive symptoms
were strongly correlated with each other (see Table 1).
o This short project demonstrates that it is possible to conduct a study
within a matter of hours.
Methodologically complex studies are difficult to conduct using
MTurk’s GUI. Here are a few examples of follow-up studies we
may want to conduct.
o Conduct follow-up study with 200 more female workers. – Methodological requirements – Exclude those who already participated. Sample across
the whole day/week. Pay workers automatically when they provide the correct secret
code, prevent duplicate IP addresses, verify that workers are from the US.
o Longitudinal follow-up with the same participants– Methodological requirements – Send email invitations. Communication. Provide bonuses
for longitudinal participation. Automatically add worker IDs to the data file in order to
match participants across data sets.
o Diary study to monitor media exposure. – Methodological requirements – Longitudinal follow up. Communication. Set daily studies
to start automatically at the same time each day.
o Video interviews to hear participant’s experiences with disordered eating
and eating disorders. – Methodological requirements – Set up Audio/Video HIT to interact with workers. Record
each interview.
Mechanical Turk’s Application Programming
Language (API) offers significantly greater
functionality, compared to the GUI.
o Mechanical Turk was designed to provide basic functionality through its
graphical user interface (GUI) but to provide significantly more
functionality through its API.
o Third party platforms that utilize MTurk’s API provide researchers with
powerful tools that can be used together with Mechanical Turk.
o There are now multiple platforms that provide various add-on tools that
facilitate the research process: RTurk, MmmTurkey, Longii, TurkGate
(Goldin & Darlow, 2013), PsiTurk (Gureckis, et al., 2016), TurkServer
(Mao et al., 2014).
o In this webinar we describe TurkPrime (Litman, Robinson and
Abberbrock, 2017).
Getting started with TurkPrime’s Toolkit.
Step 1
Create a TurkPrime account
Step 2
Link to your MTurk Requester account
Step 3
Design Survey on Qualtrics, SurveyMonkey, etc
Step 4
Launch HITs through TurkPrime
Step 5
HIT Launches on Mechanical Turk
Step 6
Workers Complete HIT
Step 7
Monitor Progress on TurkPrime’sDashboard
Creating a Turkprime account• To set up a TurkPrime account go to: https://www.turkprime.com
• If you don’t have an account you will be asked to sign up and to create a
username and a password
• Once an account is created a video tutorial will be sent to show you how to link
your MTurk account to your Turkprime account. The process takes five minutes.
Creating a study
To start creating a study click on Design Study Basic and Panel Studies
Study Design Interface
o The Design page has 9 sequential tabs.
o The first tab has several options for targeted recruitment.
o Can select between video and survey HITs.
o What follows is not a systematic tutorial, but rather a few slides
highlighting some of TurkPrime’s features
Initial setup and examples of targeting
criteria to chose from
Examples of targeting criteria:
Personality profile
Video, focus group, and screen
capture HITs
Requester
Worker 1
Worker 1
Worker 2Add more workers
Switch between
audio, video and text
Video, focus group, and screen
capture HITs
Many steps during setup are similar to
MTurko The Title, Description, Custom Instructions and Keywords fields are the
same as on MTurk
Many steps during setup are similar to
MTurko These fields are also the same as on MTurk
Examples of features not available on
MTurk
o Delayed launch – schedule a HIT’s time of launch. Useful for diary
studies, researchers from outside of the US, and night owls.
Select past studies from a dropdown menu to exclude
workers, or include workers for longitudinal studies.
o No need to set qualifications
When studies are added to a group,
workers are excluded from HITs that are
running at the same time.
o Useful for labs where multiple projects are being run at the same
time.
• For longitudinal studies, once the study is launched, Workers
receive invitation emails within minutes
• Emails sent from MTurk
• Worker Identities remain hidden
To match workers’ IDs across multiple waves, TurkPrime
automatically sends their IDs to the data file – SPSS for
example.
Microbatch spreads HITs across time for
enhanced sample representativeness.
Worker Groupso Requesters can create groups such as Republicans and White
Males. These groups are created from the workers who have
previously completed studies with a Requester.
o Worker groups can also be used to create universal exclude lists.
Groups can be selected from a
dropdown menu during study setup
Additional features
Ongoing studies are monitored on the
Dashboard
Extended Dashboard view offers more control
over HITS and advanced indicators
Mechanical Turk
Small population, skewed population,
Difficult to reach rare groups, difficult to
selectively recruit
Sampling challenges
Mechanical Turk
PrimePanels
Solution
Small population, skewed population,
Difficult to reach rare groups, difficult to
selectively recruit
Sampling challenges
Limitations of the Mechanical Turk
worker pool
o Small population
o According to MTurk there are 500,000 participants in
it’s Worker pool. The number of active workers is
closer to 25,000 on any given month.
o Market Research platforms from which PrimePanels
draws samples have access to tens of millions of
participants.
Naivete: MTurk workers are significantly
more likely to have been exposed to
common manipulations.
Litman et al., (in prep)
Representativeness and access: Close to 70% of
MTurk workers are below 40. Very limited access to
60+ age group.
Litman et al., (in prep)
Representativeness and access: MTurk workers are
much less religious compared to the general
population, and compared to PrimePanels
Litman et al., (in prep)
Representativeness and access: MTurk workers are
much more liberal compared to the general population,
and to PrimePanels.
Litman et al., (in prep)
Representativeness and access: MTurk workers are
much more likely to be college educated compared to
the general population, and to PrimePanels.
Litman et al., (in prep)
Advantages of market research panels
o Large pool of participants > 20,000,00
o More representative of the US population
o Worldwide
o Ability to selectively target very specific populations
o Little exposure to psychological manipulations
Examples of selective targeting studies
run on PrimePanels that could not be run
on MTurk
Examples of studies run on
PrimePanels that could not be run on
MTurk
Creating a PrimePanels study
To start creating a study click on Design Study PrimePanels
Initial setup – similar to MTurk
Multiple targeting variables to
choose from
Census template automatically matches the
sample to the US Census along multiple
variables
Numerous variables can be added and
percentages can be adjusted within each
bin
Tradeoffs between MTurk and
PrimePanelso PrimePanels studies should be under 20 minutes
o MTurk data quality for open ended responses is much better
o MTurk is better for tasks requiring sustained and effortful attention
over long time periods (e.g. website testing, aptitude tests)
o PrimePanels data is more representative
o PrimePanels has a larger, global pool of participants who have not
been exposed to common protocols
o PrimePanels has more sophisticated targeted recruitment
capabilities
Thank you!
For inquiries about Mechanical Turk, TurkPrime or Prime Panels please contact
Leib Litman, PhD
TurkPrime Director of Research
Or
Jon Robinson, PhD
Chief Technology Officer
Q u e s t i o n s
Program director: Katja Reuter, PhD
Email: [email protected]
Twitter: @dmsci
Next Digital Scholar Webinar
I n f o r m a t i o n a b o u t
t h e p r o g r a m
http://sc-ctsi.org/digital-scholar/
Dec 6, 2017 | 12-1PM PST
Topic: Accelerating systematic review studies using the online
tool Covidence
Speaker: Anneliese Arno, community manager at Covidence
Register at: sc-ctsi.org/digital-scholar/register