2017년 콘텐츠 서비스 디자인/ux 트렌드와 전망: 파괴적 ux혁신 방향성 고찰로...

2017년 콘텐츠서비스 디자인/UX

트렌드와 전망: 파괴적 UX혁신 방향성 고찰로

새로운 프레임 추구

2017Billy(최병호)/[email protected]

중앙대학교 교수

홍익대학교 영상대학원(HCI개론 강의)/연세대학교 공학대학원(서비스디자인경영 강의)/

성균관대학교 일반대학원 휴먼ICT융합학과(교수)/HEDcentric UX미래융합전략연구소(연구소장)

InnoUX(대표이사)

Research Data: http://www.slideshare.net/BillyChoi/Blog: http://blog.naver.com/soularchitec

Twitter/Facebook: ILOVEHCI

© 2017 Billy All rights reserved.2017년 콘텐츠 서비스 디자인/UX 트렌드와 전망: 파괴적 UX혁신 방향성 고찰로 새로운 프레임 추구

Table of Contents

1. 의류 쇼핑몰에서 ‘의류＇전시를 포기한 사례 분석 및 통찰

2. Here-and-now & answer engine으로서의 콘텐츠 서비스

3. 인공지능이 탑재된 챗봇(Chatbot)의 콘텐츠 서비스에서 추구해야 할 UX전략

4. 인간의 목소리(Voice) 자체가 콘텐츠 서비스가 되는 시대에서의 UX전략

5. 인간의 후각을 자연스럽고 조용히(natural & silent) 자극하는 콘텐츠 서비스

6. 인간의 감성 상태에 따라 자동으로 변하는 공간 내 콘텐츠 서비스

7. Second hand와 공진하는 콘텐츠 서비스

8. VR과 결합한 콘텐츠 서비스

1

의류 쇼핑몰에서‘의류＇전시를포기한 사례 분석및 통찰


Source: https://www.stitchfix.com

쇼핑몰에서 콘텐츠를 제거하는 UX전략 채택?

Stitch Fix is the first fashion retailer to blend expert styling, proprietary technology and unique product to deliver a shopping experience that is truly personalized for you. Simply fill out the Stitch Fix Style Profile and our personal stylists will handpick a selection of five clothing items and accessories unique to your taste, budget and lifestyle. You can buy what you like and return the rest (shipping is free both ways).

Here-and-now & answer engine 으로서의 콘텐츠서비스


Source: https://socratic.org/

지금 즉시(here-and-now) 문제해결을 원하는(answer engine) 시대!


지금 즉시(here-and-now) 문제해결을 원하는(answer engine) 시대!(cont.)

Source: https://socratic.org/

• A digital tutor in your pocket• Socratic uses artificial intelligence and data from millions of student questions to

understand where a student is stuck

인공지능이탑재된챗봇(Chatbot)의콘텐츠서비스에서추구해야 할UX전략

Reference: http://www.slideshare.net/EnjoyDigitAll/are-chatbots-customer-relation-future-infographic-made-by-enjoydigitall?qid=99d3c408-6f18-40a2-a373-4c5504939949&v=&b=&from_search=24

Reference: http://www.slideshare.net/Warply/chatbot-workshop-how-to-build-onedigitized16

Reference: http://www.slideshare.net/TeeWeeAng/ai-assistant-and-chatbot-trends-for-enterprises?qid=99d3c408-6f18-40a2-a373-4c5504939949&v=&b=&from_search=1

Reference: https://twitter.com/MikeQuindazzi/status/786189180573483008

Reference: https://uxdesign.cc/impact-of-chatbots-personal-assistants-on-uxd-2072ab749b5#.qvfuhicfx

Context

Scenarios

Culture

BM with UX Model

Social Trust

Contextual Design

Scenario Design

UX Modeling

Change Habits

Design the Trust Model

Natural UI

Zero effort (and connected) UI

Craft a Personality

References: • http://www.slideshare.net/Warply/chatbot-workshop-how-to-build-onedigitized16• https://uxdesign.cc/impact-of-chatbots-personal-assistants-on-uxd-2072ab749b5#.qvfuhicfx

Our job as an Experience Designer would be to predict the rate of change then anticipate the behaviour of user and how it might changewith the introduction of new revolutions (machines, voice interaction systems, conversational bots, BCI etc.) And, therefore, create groundbreaking products.

“Use” Users For UX

“Buy” Customers For CX or ServiceDesign For Customer eXperience or Service experience

Design For User eXperience or (Interactive) product experience


“Buy” Customers For CX or Service

“Talk” Player For PX(Playful eXperience)(Saying/Typing and Respond)

Design For Customer eXperience or Service experience


Design For Player’s Playful eXperience(embodied (interactive) product experience and service experience)


“Buy” Customers For CX or Service

“Talk” Player For PX(Playful eXperience)(Saying/Typing and Respond)

“He has a magic lamp with a genie inside, who grants wishes.”


Design For Customer eXperience or Service experience


“Talk” Player(Saying/Typing and Respond)

“He has a magic lamp with a genie inside, who grants wishes.”

For PX(Playful eXperience)

• Creating a magic lamp with a genie(Craft a Personality); Creating a genie-like UX

• Personalized god in a box; Era of IPA(Intelligent Personal assistant)

• Conversation-as-a-Platform(CaaP); Software-as-a-Service (SaaS); Platform-for-Everything


Designing Conversations: Conversational interfaces, Bot Interactions, Chatbot as personalities


Avoid nagging users with unhelpful or irrelevant messages

• Version 1.0* Every time a user typed “Dear . . . ,” Clippy would dutifully

propose, “I see you are writing a letter. Would you like some help?”—no matter how many times the user had rejected this offer in the past.

Clippy would give unhelpful answers to questions, and when the user rephrased the question, Clippy would give the same unhelpful answers again.Image source: http://bit.ly/1PDos4G

• Source: Clifford Nass & Corina Yen, 2010• Reference:: https://www.punchkick.com/blog/2016/10/05/how-bots-fit-into-your-apps-ux


Scapegoating(Create a scapegoat; 죄를 뒤집어씌우는 전략) (1/2)

• Strategy* Social science literature to find simple tactics that unpopular

people use to make friends.

Without any fundamental change in the software, the right social strategy rescued Clippy from the list of Most Hated Software of All Time; creating a scapegoat bonded Clippy and the user against a common enemy.

Image source: http://bit.ly/1PDos4G

* Source: Clifford Nass & Corina Yen, 2010


Scapegoating(Create a scapegoat; 죄를 뒤집어씌우는 전략) (2/2)

• Version 2.0* After Clippy made a suggestion or answered a question, he

would ask, “Was that helpful?” and then present buttons for “yes” and “no.” If the user clicked “no,” Clippy would say, “That gets me really angry! Let’s tell Microsoft how bad their help system is.”

He would then pop up an e-mail to be sent to “Manager, Microsoft Support,” with the subject, “Your help system needs work!” After giving the user a couple of minutes to type a complaint, Clippy would say, “C’mon! You can be tougher than that. Let ’em have it!”

Image source: http://bit.ly/1PDos4G



착각과 통찰

• The More the Better(대화 요청 빈도와 시기, 대화의 양, 대화의 속도 등) & Less is More(사용자의 컨텍스트 추적 기반의 알고리즘적 접근, 일정 수준의 정해진 행동패턴을반영한 자동 처리) for conversation?!

• The wisdom from the social science!!; 사회심리학적 지식의 응용 필요

21


Watch your message pacing

• 아내의 속도에 맞게 대화를!Our A.I. can type a thousand words per minute, but that’s not what people want from a

chat interface.

• 아내의 호흡(사용자의 특성)에 맞게 탄력적으로(학습을 통해) 대화를!We intentionally pace how fast a user receives our messages to make the experience feel

more natural; Pace messages at human reading speed. If your bot blurts out too much text instantaneously, this can be jarring for users to keep up with.

We’ve tested a lot of different levels of speed and found that adding a .02 second delayhelps with engagement.

We hope to create a feature that will analyze the way a user interacts with our system and adjust the pacing for each individual, and we already have enough data to appropriately adjust the pace by age.

22

References: • http://venturebeat.com/2016/07/27/5-tips-for-writing-chatbot-scripts/• https://chatbotsmagazine.com/top-6-conversational-skills-to-teach-your-chatbots-ec4eb019a23d#.xt1h59xhu


Limit the length and number of messages

• Engagement drops with every line of text over three lines, which we call the “glanceable tipping point.”

They need to be invested in the answer they are about to receive. If they ask for advice from a local doctor who accepts their insurance, they will take the time to read a long message because the information matters to them. We’ve also noticed that users don’t like receiving too many messages in a row without a break.

• We’ve now added a user input option after 4-5 messages to break up the text and give the user a few seconds to catch their breath. Even a simple response like “OK” or “cool” works to pace the influx of texts.

• Keep messages short, sweet, and readable!

23

Reference: http://venturebeat.com/2016/07/27/5-tips-for-writing-chatbot-scripts/

References: • http://www.slideshare.net/Warply/chatbot-workshop-how-to-build-onedigitized16• https://chatbotsmagazine.com/top-6-conversational-skills-to-teach-your-chatbots-ec4eb019a23d#.xt1h59xhu

Add personality to your bot using appropriate avatars, names, biographical background, and other humanizing assets


Computers Are Social Actors

• Humans expect computers to act as though they were people and get annoyed when technology fails to respond in socially appropriate ways.(컴퓨터가 사람처럼행동하길 기대하고, 기술이 인간적인 방식으로 반응하지 않는 것에 불만을 가짐)

Show that people treat computers as if they were real people(컴퓨터와 미디어를사람처럼 대하는 사람들의 성향 관련 연구) Nass, C., and Brave, S. B. (2005). Wired for speech: How voice activates and

enhances the human computer relationship. Cambridge, MA: MIT Press. Reeves, B., and Nass, C. (1996). The media equation: How people treat

computers, television, and new media like real people and places. New York: Cambridge University Press.



Similarity-attraction(유사성-매력 전략)(1/4)

• Personality traits(성격 특성); 5살 무렵에 형성• Four Personalities

26

비판형 외향형

내향형 수용형

냉정형 다정형

순응형

지배형

협력




• This is a reproduction of one of the most famous of the Tiffany stained-glass pieces—the colors are absolutely sensational! This first-class, handmade copper-foiled stained-glass shade is over six and one-half inches in diameter and over five inches tall. I am sure that this gorgeous lamp will accent any environment and bring a classic touch of the past to a stylish present. It is guaranteed to be in excellent condition! I very highly recommend it.

• This is a reproduction of a Tiffany stained-glass piece. The colors are quite rich. The handmade copper-foiled stained-glass shade is about six and one-half inches in diameter and five inches tall.

27




• You should definitely select option A instead of option B. There are at least six reasons why this is the right option. I am 90 percent confident of this assessment.

• Perhaps you should select option A instead of option B? It seems like there are reasons why this might be the right choice. I am 40 percent confident of this assessment.

28




• Similarity-attraction affects people to such a degree that they feel positive toward not only similar people but also anything associated with those similar people. For example, in the experiment, not only did participants like the sellers who were similar to themselves, they also felt more positive about the items associated with the similar sellers.(유사성-매력 효과는 긍정적 감정 뿐만 아니라 유대감 유발. 심지어성격이 비슷한 판매자가 경매에 올린 제품까지 선호)

• 외향성 음성과 내향성 음성 동일 적용; 음량, 음역, 음성 속도; 성격과 음성의 일관성중요하게 판단함

• When the introduction to a computer-based “Entertainment Guide” matched users’ personalities, users found the recommended music to be significantly better, even though the recommendations themselves were identical.(동일 음악을 추천하여도서비스 도입부가 자신의 성격에 부합하면 선호 발생)

29


Reference: Dormehl, Luke (2014-04-03). The Formula: How Algorithms Solve all our Problems … and Create More. Ebury Publishing.

Quantified Self movementSelf-knowledge through numbers

(숫자를 통한 자기 이해)

Based upon speech patterns, the particular words they used, and even details as seemingly trivial as whether they said “um” or “err” – and then utilise these insights to put them through to the agent best suited for dealing with their emotional needs?

(Chicago’s Mattersight Corporation does exactly that. Based on custom algorithms, Mattersight calls its business “predictive behavioral routing”.)



Based upon speech patterns, the particular words they used, and even details as seemingly trivial as whether they said “um” or“err” – and then utilise these insights to put them through to the agent best suited for dealing with their emotional needs?(Chicago’s Mattersight Corporation does exactly that. Based on custom algorithms, Mattersight calls its business “predictive behavioral routing”.)

The man behind Mattersight’s behavioural models is a clinical psychologist named Dr Taibi Kahler. Kahler is the creator of a type of psychological behavioural profiling called Process Communication.

What Kahler noticed was that certain predictable signs precede particular incidents of distress, and that these distress signs are linked to specific speech patterns. These, in turn, led to him developing profiles on the six different personality types he saw recurring.




Based upon speech patterns, the particular words they used, and even details as seemingly trivial as whether they said “um” or“err” – and then utilise these insights to put them through to the agent best suited for dealing with their emotional needs?(Chicago’s Mattersight Corporation does exactly that. Based on custom algorithms, Mattersight calls its business “predictive behavioral routing”.)

The man behind Mattersight’s behavioural models is a clinical psychologist named Dr Taibi Kahler. Kahler is the creator of a type of psychological behavioural profiling called Process Communication. What Kahler noticed was that certain predictable signs precede particular incidents of distress, and that these distress signs are linked to specific speech patterns. These, in turn, led to him developing profiles on the six different personality types he saw recurring.

A person patched through to an individual with a similar personality type to their own will have an average conversation length of five minutes, with a 92 percent problem-resolution rate. A caller paired up to a conflicting personality type, on the other hand, will see their call length double to ten minutes – while the problem-resolution rate tumbles to 47 percent.


Personality type Personality traits How common?

“Thinkers”Thinkers view the world through data. Their primary way of dealing with situations is based upon logical analysis of a situation. They have the potential to become humourless and controlling.

1 in 4 people

“Rebels”Rebels interact with the world based on reactions. They either love things orhate them. Many innovators come from this group. Under pressure they can be negative and blameful.

1 in 5 people

“Persisters”Persisters filter everything through their opinions. Everything is measured upagainst their world view. This describes the majority of politicians.

1 in 10 people

“Harmonisers”Harmonisers deal with everything in terms of emotions and relationships. Tight situations make this group overreactive.

3 in 10 people

“Promoters”Promoters view everything through action. These are the salesmen of the world, always looking to close a deal. They can be irrational and impulsive.

1 in 20 people

“Imaginers”Imaginers deal in unfocused thought and reflection. These people operate in vivid internal worlds and are likely to spot patterns where others cannot.

1 in 10 people

Dr Taibi Kahler’s the six different personality types



착각과 통찰

• Computers Are Social Actors & Similarity-attraction 그냥 입력 데이터 분석해서 적정하게 검색 결과를 대화형으로 보여주면 되는 거

아닌가? (X) Social Actor로 정의하자! 즉 성격을 부과하자! 특히 초기에 고정(세팅)된 성격

제공(또는 선택)이 아니라 사용자의 특성을 점차 학습하여 사용자와 유사한 성격유형으로 적응하는 방향을 추구해야 함. 이것이 최근 부각되는 N=1의 사용자 경험전략 방향임 이것이 가능하기 위해서는 사용자의 성격 특성을 측정하고 적응할 수 있는

알고리즘 개발이 필요함 지속적인 구매 등의 지속적인 행동 변화를 도모하는데 큰 기여를 할 수 있음

34


Empathize With Emotions: Humor

• I’ve even heard that comedy writers are becoming the next hot UX hires in the hopes of making copy more engaging.

• We often use humor to lighten up a situation and smooth over our mistakes. We might consider using this tactic when the bot encounters a limitation or needs to give

an error message.

• To succeed with humor, consider what type of personality your users would find engaging. Make sure to fit the tone to the task at hand, as well as the brand your bot is representing. For example, if you’re building a banking bot, avoid flippantly humorous error messages

when a user’s balance can’t be retrieved. This is a very important place to check in with our User Personas to get a feel for their

preferences and their intents.

35

References: • https://willowtreeapps.com/blog/part-2-shortcuts-to-chatbot-emotional-intelligence/• https://chatbotsmagazine.com/top-6-conversational-skills-to-teach-your-chatbots-ec4eb019a23d#.xt1h59xhu• https://replika.ai/


Empathize With Emotions: Use emojis

• We did learn to only use emojis in positive affirmation responses and to introduce them later in the onboarding process.

• We use templated responses for many interactions that the user inputs, and one thing we learned is that people don’t like being “forced” to send emojis back to a bot. Our responses tend to be fairly generic because people need to connect to the things we are allowing them to say.

36

References: • http://venturebeat.com/2016/07/27/5-tips-for-writing-chatbot-scripts/


Natural Flow and Cadence: First Impression

• First impressions really do make or break an interaction, so we should consider our greeting carefully.

It’s often helpful to explain what our bot can do in the first interaction,

But it can be jarring for the user to get a huge chunk of text before exchanging a greeting.

Like it or not, your first-time users are essentially talking with a stranger, which in the range of enjoyable life activities ranks right up there with surprise dental emergencies.

They may grow to love your bot at some point, but in the first engagement, your bot is essentially the guy on the subway who just started asking them questions.

37

References: • https://willowtreeapps.com/blog/part-2-shortcuts-to-chatbot-emotional-intelligence/• http://venturebeat.com/2016/10/05/chatbot-design-how-to-build-a-beautiful-ui-when-all-you-have-is-text/


Natural Flow and Cadence: Don’t Be A Broken Record

You: What is your name?Mitsuku: My name is Mitsuku.

You: What is your name?Mitsuku: You just asked me that. It’s still Mitsuku. It hasn’t changed in the last 10

secs.

• Nobody likes being told the same thing over and over again, So why do chatbots keep doing it? Bots should detect when they’re about to repeat a

previously given answer and switch strategies.

As an egregious example, I ran into a bug with the 1–800-Flowers chatbot on Facebook Messenger. This chatbot presents categories and subcategories of bouquets you can choose from, but if you go into a subcategory card menu, the previous menus become deactivated and there’s no obvious way to go back to them. Pretty silly, eh?

38

References: • https://chatbotsmagazine.com/top-6-conversational-skills-to-teach-your-chatbots-ec4eb019a23d#.xt1h59xhu• http://designrfix.com/design/key-principles-chatbot-design


Writing scripts

• Our audience comes from a huge cross-section of society. We sell our product directly to employers of a highly educated workforce as well as to people who didn’t finish high school. We’ve found that, regardless of the audience, making your scripts as simple as possible results in the greatest level of engagement.

• You can use the Flesch-Kincaid Grade Level feature in Microsoft Word or an app like Hemingwayapp.com to figure out the present grade level of your scripts. I don’t think there is a hard and fast rule of what grade level you want to write to, but in general, the lower the better.

39

References: • http://venturebeat.com/2016/07/27/5-tips-for-writing-chatbot-scripts/• http://beta.hemingwayapp.com/


New Command Line Interface(CLI)? Prompt chatbot

• A Prompt bot in Slack

Typing "@uber from Penn Station to LaGuardia Airport" would order a car to pick you up to drop you at the airport.

Typing "@nest 73" and Prompt will turn up your heat if you have a Nest thermostat.

Typing "@flightstats AA21" would get you the status for American Airlines flight 21

Typing "@hue" will turn on and off your Philips Hue bulbs

Typing "@showtimes" will tell you what movies are playing near you, and so on.

• Just like the command line, though, Prompt's biggest barrier to entry is syntax.That's especially frustrating on smartphones, where on-screen keyboards and autocorrect can wreak havoc with Prompt's super-specific vocabulary.

40

References: • https://www.fastcodesign.com/3057596/prompt-wants-to-be-the-one-chatbot-to-rule-them-all• http://www.message.io/

인간의목소리(Voice) 자체가 콘텐츠서비스가 되는시대에서의UX전략

Source: 드라마 <보이스>

Sources: • Gursimarjot Singh Walia, Gurjot Kaur Walia: Level of Asthma: A Numerical Approach based on Voice Profiling (2016)• Pragnesh Parmar, Udhayabanu R. “Voice Fingerprinting: A Very Important Tool against Crime”. J Indian Acad Forensic Med. Jan- March 2012, Vol. 34, No.1, ISSN: 0971-0973.

• The voice of each person is differentbecause the anatomy of vocal cavity, oral cavity, nasal cavity, and vocal cords is specific to the individual.

• People in different countries, in fact, people in different parts of the same country, speak with different accents. There are some people who run their words together, and there are others who talk with pauses between their words.

• If a person is having some kind of illness, such as cough, cold, fever etc., or feeling some kind of emotion, such as happiness, sadness, stress, anxiety etc., then their voice would be different from what they sound when they are normal.

비강(鼻腔)

구강(口腔)

성대(聲帶)

Source: Rita Singh, Joseph Keshet, Eduard Hovy: Profiling Hoax Callers (2016)

“지금 뉴스에 나오는보이스피싱 사건은 지금인터뷰하는 손자의자작극이에요.

변조로 목소리를 다르게 하려고애를 썼지만 손자의 호흡, 말투, 억양이 일치해요.

사람 목소리는 지문과 같아요.

손자가 자작극을 했을 확률이99.9%에요.”


In police and Forensic Scientists, sometimes voice is the only clue available in identifying the criminal.

Source: Gursimarjot Singh Walia, Gurjot Kaur Walia: Level of Asthma: A Numerical Approach based on Voice Profiling (2016)

“대표적인 ‘오원춘사건‘.

피해자가 112에 죽기직전에 전화했는데, 결국 시간이지연되면서 다음날아침 시신으로 발견.

심리분석이 가능한‘보이스프로파일러＇가 그전화를 받았다면현장에 돌입할 수있었을 것이다.”

Source: 드라마 <보이스> 홍보영상 https://www.facebook.com/CJTVING/videos/1385055181535656/

“범죄 현장의 ‘소리＇를통해 많은 정보를수집하는 것은 굉장히좋은 수사의 방향이 될수 있다.”

Source: 드라마 <보이스> 홍보영상 https://www.facebook.com/CJTVING/videos/1385055181535656/

© 2017 Billy All rights reserved.Future of Voice UX: Everywhere and 합류(合流)(Voice 관련 연구 탐색 및 Voice 서비스 통찰 중심으로)

FVC(forensic voice comparison)

• In forensic voice comparison(FVC), Speech recordings from an unknown voice, usually of an offender, are compared with recordings from a

known voice, usually the suspect.

• In the first type, The expert considers their aim to be to say how likely it is, given the evidence, that the suspect said the

incriminating speech.

• In the second type of FVC, The expert’s aim is seen as restricted to estimating the strength of the speech evidence with a

Likelihood Ratio(LR)? In other words, to estimate how much more likely the difference between the suspect and offender

speech samples is, assuming the offender sample has come from the suspect, rather than from another randomly chosen speaker in the relevant population.

• For some time now, The use of a LR has been theoretically recognised as the correct logical framework for the evaluation of

forensic evidence.

Source: Rose, Phil.: Where the Science Ends and the Law Begins: Theory and Reality in Likelihood Ratio-based Forensic Voice Comparison.(2012) References:• Association of Forensic Science Providers: Standards for the formulation of evaluative forensic science expert opinion. Science & Justice 49, 161-164 (2009)• Gonzalez-Rodriguez J., Rose P., Ramos, D., Torre, D. & Ortega-Garcia, J.: Emulating DNA: Rigorous Quantification of Evidential Weight in Transparent and Testable Forensic Speaker Recognition.

IEEE Trans. on Audio Speech and Language Processing 15(7), 2104-2115 (2007)• Balding, D.J.: Weight of Evidence for Forensic DNA Profiles. Wiley, Chichester (2005)• Aitken, C.G.G., Taroni, F.: Statistics and the Evaluation of Evidence for Forensic Scientists. Wiley, Chichester (2004)

50


LR(Likelihood Ratio) approach

• A crucial desideratum in forensic comparison science? The accuracy (and more recently the precision) of a LR-based FVC(forensic voice

comparison) system are also straightforwardly tested.

• Apart from its correctness, The LR approach has several other important properties. It allows, for example, the combination of evidence of different types, nicely

demonstrated in the testing of both automatic and acoustic-phonetic features in hybrid FVC systems.The LR-based testing of other forensic evidence types is now following:

fingerprints, handwriting and SMS texting.

Source: Rose, Phil.: Where the Science Ends and the Law Begins: Theory and Reality in Likelihood Ratio-based Forensic Voice Comparison.(2012) References:• Morrison, G.S.: Measuring the Validity and Reliability of forensic likelihood-ratio systems. Science & Justice (51) 3, 91-98 (2011)• Morrison: G.S. Forensic voice comparison and the paradigm shift. Science & Justice 49,298-308 (2009)• Gonzalez-Rodriguez J., Drygajlo, A., Ramos-Castro, D., Garcia-Gomar, M., Ortega-Garcia, J.: Robust estimation, interpretation and assessment of likelihood ratios in forensic speaker recognition.

Computer Speech and Language 20, 331-355 (2006)

51

Source: Rose, Phil.: Where the Science Ends and the Law Begins: Theory and Reality in Likelihood Ratio-based Forensic Voice Comparison.(2012)

• Figure 1 shows the F0 realising the [H.L.LH] intonational pitch of the offender, aligned with its wideband spectrogram.

• F0 on not can be seen to drop from about 200 Hz to 175 Hz; whence it drops further on the nucleus of too to about 125 Hz.

• The F0 shows a small ca. 15 Hz increase from its minimum value of 125 Hz in the /b/ hold, and rises on the nucleus of bad with a slightly convex contour from about 145 Hz to peak at about 185 Hz.

Source: Rose, Phil.: Where the Science Ends and the Law Begins: Theory and Reality in Likelihood Ratio-based Forensic Voice Comparison.(2012)

• Figure 2 compares the offender F0 with the F0 of the suspect’s 15 not too bad utterances.

• The similarity is considerable, with the offender’s F0 time-course lying completely within, and in some places almost exactly in the middle of, the suspect’s distribution.

• Note too the suspect’s use of both H and L on not.

• Table shows the parameters that can be extracted using voice analysis, and the information that can be extracted from those voice parameters.

Sources: • Gursimarjot Singh Walia, Gurjot Kaur Walia: Level of Asthma: A Numerical Approach based on Voice Profiling (2016)• Khushboo Batra, Swati Bhasin, Amandeep Singh. “ Acoustic Analysis of voice samples to differentiate Healthy and Asthmatic persons”. International Journal of Engineering and Computer Science,

ISSN 2319-7242, Volume 4, Issue 7, July 2015, Page No. 13161-13164.• Pragnesh Parmar, Udhayabanu R. “Voice Fingerprinting: A Very Important Tool against Crime”. J Indian Acad Forensic Med. Jan- March 2012, Vol. 34, No.1, ISSN: 0971-0973.

※ Extraction of voice parametersThe above parameters were extracted using the MDVP (Model 5105, KayPENTAX) tool of CSL (Model 4500, KayPENTAX) system.

Fo: Average Fundamental FrequencyJitt: Jitter (%)Shim: Shimmer (%)vFo: Coefficient of fundamental frequency variationDUV: Degree of VoicelessDSH: Degree of Sub-HarmonicsSPI: Soft Phonation IndexDVB: Degree of Voice BreaksNHR: Noise-to-Harmonic RatioPPQ: Pitch Period Perturbation Quotient (%)RAP: Relative Average Perturbation (%)To: Average Pitch Period


Voice 서비스 통찰

1. 공간 구분의 필요성 폐쇄 공간 & 오픈 공간: 지속적으로 상주하는 특정인의 유무 기준으로 구분

2. 폐쇄 공간 내 서비스 예측 및 서비스 디자인 범죄 예방 및 범죄 골든타임 지원 서비스

① 집이나 어린이집 등에서 가까운 관계의 상대에서 가해지는 일시적 또는지속적으로 자행되는 범죄 관련 지원 서비스

② 외부인이 침입 시도 또는 침입하여 발생하는 범죄 관련 지원 서비스

패턴과 Context 기반의 서비스 시나리오 디자인 수요자와 공급자 모두 대상(일시 방문자 및 외부 침입자 포함) 자동 대처와 그 외의 대처 상황 고려 필요 다양한 생태계와 유기적인 공조 서비스 시나리오 디자인 필요 보안 정책 및 사생활 침해 등 윤리 가이드라인 수립 필요

55


Voice 서비스 통찰(cont.)

3. 폐쇄 공간 내 공급자 제공 기술 예측 범죄 예방 지원 기술

① 지속적으로 상주하는 특정인의 보이스 프로파일링 및 보이스 인증 기술② 일시 방문자의 보이스 프로파일링③ 폐쇄 공간 내 노이즈 프로파일링 및 이상 노이즈 중점 분석 기술④ 행동패턴 누적 및 분석 기술(지속적인 모니터링이 가능한 관제 관련 기술 포함)

범죄 골든타임 지원 기술 영상인식 등 다른 기술과의 연계 지원 기술

56

Sources: • Khushboo Batra, Swati Bhasin, Amandeep Singh: Acoustic Analysis of voice samples to differentiate Healthy and Asthmatic persons(2015)• Rachna, Dinesh Singh, Vikas: FEATURE EXTRACTION FROM ASTHMA PATIENT’S VOICE USING MEL-FREQUENCY CEPSTRAL COEFFICIENTS(2014)• Saloni, R. K. Sharma, and A. K. Gupta. "Disease detection using voice analysis: a review." International Journal of Medical Engineering and Informatics 6.3(2014): 189-209.• Sonu, R. K. Sharma “Disease detection using analysis of voice”, TECHNIA – International Journal of Computing Science and Communication Technologies, VOL.4 NO. 2, January 2012.

• Speech is produced by vocal folds. It involves the interaction of various body parts*. It can hurt the sound quality of the voice.

• Asthma is a lung disease that affects airflow to and fro from lungs. A whistling sound comes when asthmatic patient breathes.

* This includes various components like abdominal, ribcage, lungs, pharynx, oral cavity and nose and each performs its own function in speech production.

Sources: • Gursimarjot Singh Walia, Gurjot Kaur Walia: Level of Asthma: A Numerical Approach based on Voice Profiling (2016)• Khushboo Batra, Swati Bhasin, Amandeep Singh. “ Acoustic Analysis of voice samples to differentiate Healthy and Asthmatic persons”. International Journal of Engineering and Computer Science,

ISSN 2319-7242, Volume 4, Issue 7, July 2015, Page No. 13161-13164.

• There are several voice pathologic disorders related with nasal, neural, respiratory and larynx diseases. (코, 신경, 호흡, 후두관련 질병)

• As a result, analysis and diagnosis of vocal disorders has become an important medical procedure.


Asthma patient’s voice & Voice Recognition

59

Source: Rachna, Dinesh Singh, Vikas: FEATURE EXTRACTION FROM ASTHMA PATIENT’S VOICE USING MEL-FREQUENCY CEPSTRAL COEFFICIENTS(2014)

In the above graphs an analysis of mean of five vowels (a,e,i,o,u) for males and females are presented for various voice parameters like JITTER and SHIMMER of different asthma and healthy persons are compared.


Asthma patient’s voice & Voice Recognition(cont.)

• Asthma has no cure, just it can be controlled. Major risk factors are bedding dust, carpet, furniture dust, also family history or allergy.

• It can be controlled during asthma stages by doing long term meditation daily, regular check up by doctor in case of serious patients, taking some drugs through inhalers when asthma attack came etc.

• Further these extracted coefficients will be analyzed for finding similarities between patients and normal persons.

60

Source: Rachna, Dinesh Singh, Vikas: FEATURE EXTRACTION FROM ASTHMA PATIENT’S VOICE USING MEL-FREQUENCY CEPSTRAL COEFFICIENTS(2014)


Voice Analysis of Parkinson Disease & Voice Recognition

61

Sources: • Saloni, R. K. Sharma, Anil K. Gupta: Voice Analysis for Telediagnosis of Parkinson Disease Using Artificial Neural Networks and Support Vector Machines (2015)• Max A. Little, P. E. Macsharry, E. J. Hunter, J.Sielman, L. O. Raming, “Suitability Of Dysophonia Measurements for Telemonitoring of Parkinson’s Disease, IEEE Transaction on biomedical engg,Vol.

56,2009, pp 1015-1022.


Acoustic Analysis of voice samples to differentiate Healthy

• A human voice is very closely related to the human health conditions, both physical and mental. Changes in voice quality and pitch occur frequently in hormonal imbalances or deficiencies.

• Through acoustic analysis, factors that affect the production mechanism of human voice leads to the non-invasive diagnosis of diseases.

• The person’s voice suffering from any disease different from the healthy person in some extent. As the various diseases like Parkinson, dysphonia, cardio-vascular, dystharia & respiratory tract infection lay their impression on voice of a person.

• With the help of speech we will extract various information about the speaker, gender, language, emotions health.

62

Sources: • Gursimarjot Singh Walia, Gurjot Kaur Walia: Level of Asthma: A Numerical Approach based on Voice Profiling (2016)• Khushboo Batra, Swati Bhasin, Amandeep Singh: Acoustic Analysis of voice samples to differentiate Healthy and Asthmatic persons(2015)• Cyril R. Pernet et al. “The human voice areas: Spatial organization and inter-individual variability in temporal and extratemporal cortices”. Neuroimage 119(2015) 164-174.• Saloni, R. K. Sharma, and A. K. Gupta. "Disease detection using voice analysis: a review." International Journal of Medical Engineering and Informatics 6.3 (2014):189-209.• Teixeira, João Paulo, and Paula Odete Fernandes. "Jitter, Shimmer and HNR Classification within Gender, Tones and Vowels in Healthy Voices." Procedia Technology 16 (2014): 1228-1237.• Pati, Debadatta and SR Mahadeva Prasanna. "Speaker recognition from excitation source perspective." IETE Technical Review 27.2 (2010): 138-157.• Peng, Ce, et al. "Pathological voice classification based on a single Vowel's acoustic features." Computer and Information Technology, 2007. CIT 2007. 7th IEEE International Conference on. IEEE,

2007.• Friedrich S. Brodnitz. “Hormones and the Human Voice”, Bulletin of the New York Academy of Medicine 03/1971; 47(2):183-91.


Voice 서비스 통찰

1. 공간 구분의 필요성 폐쇄 공간 & 오픈 공간: 지속적으로 상주하는 특정인의 유무 기준으로 구분

2. 폐쇄 공간 내 공급자 제공 기술 예측 질병 예방 지원 기술

① 지속적으로 상주하는 특정인의 질병 프로파일링 및 보이스 인증 기술② 외부인의 보이스 프로파일링으로 질병 예측 및 감염 여부 분석 기술③ 폐쇄 공간 내/외 환경 분석 기술④ 지속적인 모니터링이 가능한 관제 관련 기술

골든타임 지원 기술 공기 측정, 스마트 드러그 서비스, 영상인식 등 다른 기술과의 연계 지원 기술

63


Voice 서비스 통찰(cont.)

3. 폐쇄 공간 내 서비스 예측 및 서비스 디자인 질병 예방 및 골든타임 지원 서비스

① 집이나 어린이집 등에서 질병 (감염) 유발 환경 모니터링② 경증, 중증 환자의 경우, 지속적인 건강 상태 모니터링

패턴과 Context 기반의 서비스 시나리오 디자인 수요자와 공급자 모두 대상(외부인 포함) 자동 대처와 그 외의 대처 상황 고려 필요 다양한 생태계와 유기적인 공조 서비스 시나리오 디자인 필요 보안 정책 및 사생활 침해 등 윤리 가이드라인 수립 필요

64

Source: Dave Winsborough and Tomas Chamorro-Premuzic: Talent Identification in the Digital World: New Talent Signals and the Future of HR Assessment(2016)


New Talent Signals and the Future of HR Assessment

• Through the addition of innovations, such as text analytics and algorithmic reading of voice-generated emotions, a wider universe of talent signals can be sampled.

• In the case of voice mining, candidates’ speech patterns are compared with an “attractive” exemplar, derived from the voice patterns of high performing employees. Undesirable candidate voices are eliminated from the context, and those who fit move to the next round.

• More recent developments include video-mediated scenario-based questions, images, video, and work samples and automated reading of micro-emotions during the interview.

• For example, Hirevue.com, a leading provider of digital interview technologies, employs coding challenges to screen software engineers for their software writing ability. Likewise, Uber uses similar tools to test and evaluate potential drivers exclusively via their smartphones.

66

Sources: • Tomas Chamorro-Premuzic, Dave Winsborough, Ryne A. Sherman, Robert Hogan: New Talent Signals: Shiny New Objects or a Brave New World?(2016)• Dave Winsborough and Tomas Chamorro-Premuzic: Talent Identification in the Digital World: New Talent Signals and the Future of HR Assessment(2016)

Source: Daniel Chen, Yosh Halberstam, Alan Yu: Covering: Mutable Characteristics and Perceptions of Voice in the U.S. Supreme Court (2016)


The emotion detection based on speech signal analyses features

1. Pitch

2. Energy (computing Teager Energy Operator – TEO)

3. Energy fluctuation

4. Average level crossing rate (ALCR)

5. Extrema based signal track length (ESTL)

6. Liner prediction cepstrum coefficients (LPCC)

7. Mel frequency cestrum coefficients (MFCC)

8. Formants

9. Consonant vowel transition

68

Source: Batalla, J.M., Mastorakis, G., Mavromoustakis, C.X., Pallis, E.: Beyond the Internet of Things: Everything Interconnected(2017)

The OCC modelSource: Ortony, A., Clore, G.L., & Collins A. (1990). The Cognitive Structure of Emotions Cambridge Univ. Press


Appendix. Emotional attributions significantly associated with acoustic parameters

70

Source: Klaus R. Scherer: Expression of Emotion in Voice and Music(1995)


Appendix. Microexpressions

• Based on Ekman’s research on emotions (Ekman, 1993), the security sector has developed microexpression detection and analysis technology to enhance the accuracy of interrogation techniques for identifying deception (Ryan, Cohn, & Lucey, 2009). Ekman, P. (1993). Facial expression and emotion. The American Psychologist, 48(4), 384–392. doi:10.1037/0003-066X.48.4.384 Ryan, A., Cohn, J., & Lucey, S. (2009). Automated facial expression recognition system. 43rd Annual 2009 International

Carnahan Conference on Security Technologies, 172–177. doi:10.1109/CCST.2009.5335546

• The recent creation of large databases of microexpressions (Yan, Wang, Liu, Wu, & Fu, 2014) is likely to facilitate the standardization and validation of these methods.Yan, W. J., Wang, S. J., Liu, Y. J., Wu, Q., & Fu, X. (2014). For micro-expression recognition: Database and suggestions.

Neurocomputing, 136, 82–87. doi:10.1016/j.neucom.2014.01.029

• Beyond using automated emotion reading, new research aims to correlate facial features and habitual expression with personality (Kosinski, 2016). Kosinski, M. (2016, January). Mining big data to understand the link between facial features and personality. Paper

presented at the 17th Annual Convention of the Society of Personality and Social Psychology, San Diego, CA.

71

Sources: • Tomas Chamorro-Premuzic, Dave Winsborough, Ryne A. Sherman, Robert Hogan: New Talent Signals: Shiny New Objects or a Brave New World?(2016)• Dave Winsborough and Tomas Chamorro-Premuzic: Talent Identification in the Digital World: New Talent Signals and the Future of HR Assessment(2016)

인간의 후각을자연스럽고조용히(natural & silent) 자극하는콘텐츠 서비스

Sources: • https://ksr-video.imgix.net/projects/2773429/video-742380-h264_high.mp4• https://www.kickstarter.com/projects/1947880314/kunk-smart-diffuser?ref=nav_search


SMART DIFFUSE

• Without the use of allergenic particles or solvents existent in candles and perfumes.

• No wheezing and no sneezing.

• Precisely the 124 colors of the spectrum.

• Turn on or off through voice commands or with a simple shake motion of your smart phone.

• We only utilize 100% organic oils based on natural floral water. No artificial methods are used in manufacturing, allowing us to provide a truly natural scent. We are starting with 5 basic fragrances and will continue to research and develop additional scents to meet the future needs of our customers.

76

• Source: https://www.kickstarter.com/projects/1947880314/kunk-smart-diffuser?ref=nav_search

인간의 감성상태에 따라자동으로 변하는공간 내 콘텐츠서비스

Image source: http://blog.naver.com/PostView.nhn?blogId=rabu_rabu&logNo=220770319639

Image source: http://www.popphotos.net/media/BOui2GAgsD_

Image source: http://www.playtime.co.kr/brand/brand007.php


Sensing and Managing Vehicle Behavior Based on Occupant Awareness

• Disney patent would alter rides immediately based on passenger emotions

• Disney’s patents seeks to read rider’s emotions or predetermined interests to alter rides and make them more enjoyable.

• For example, the patent states that via a camera on the vehicle or a wearable ID device — say Disney's MagicBands — a ride system could read rider facial expressions such as being excited or bored, and then alter the course of the attraction to increase/decrease speed, spin more or less often, change the tone of display scenery and/or more to improve the ride for guests.

79

Sources: • http://www.bizjournals.com/orlando/news/2017/01/30/disney-patent-would-alter-rides-immediately-based.html• http://www.bizjournals.com/orlando/news/2017/01/30/disney-patent-would-alter-rides-immediately-based.html#i1

Second hand와공진하는 콘텐츠서비스


Source: https://www.kickstarter.com/projects/1225090958/raybaby-worlds-only-non-contact-health-and-sleep-m

아바타(Second hand)의 서비스가 우리의 삶에 들어오는 시대!


Source: https://www.kickstarter.com/projects/1225090958/raybaby-worlds-only-non-contact-health-and-sleep-m

아바타(Second hand)의 서비스가 우리의 삶에 들어오는 시대!(cont.)

• Welcome to Raybaby, the world's first and only non-contact sleep and breathing tracker, created for moms and dads to bring in that much needed sanity in their lives.

• Supported by Johnson & Johnson, and HAX as part of the Joint Consumer Health Device Program, Raybaby safely monitors vitals, breathing rates and sleeping habits all from a 'Smart Journal' app while even making recommendations to improve the baby's sleep...and yours! Raybaby is suitable for infants and toddlers (0-3 years of age)


Source: https://robohon.com/global/index.php



Source: https://robohon.com/global/index.php


• It’s small. It’s a robot! Talk to it, make a call, check email. Take photos, do other things too. Watch it do a little dance!

• Little by little it learns. Take it with you on the go. The more attached you become to RoBoHoN.


Source: http://myfriendcayla.co.uk/cayla


VR과 결합한콘텐츠 서비스


Source: http://www.cyclevr.com/

낯선 곳을 탐험하고 싶은 동기를 촉발하는 VR 시스템과 결합한 콘텐츠 서비스


Source: http://www.asphaltgreen.org/ues/programs/fitness/ag6

참조: 공간과 결합한 시스템에서의 게임처럼 즐기는 콘텐츠 서비스


Source: http://www.asphaltgreen.org/ues/programs/fitness/ag6

참조: 공간과 결합한 시스템에서의 게임처럼 즐기는 콘텐츠 서비스(cont.)

• Push your limits in AG6™, a high-intensity, circuit-based program and fitness experience like no other. AG6™ focuses on movement with purpose to uncover your athletic sixth sense and strengthen all major muscle groups, burning up to 1,000 calories in 45 minutes.

• Experience Europe’s hottest fitness sensation only at Asphalt Green! PRAMA technology features pressure sensitive walls and floors that come alive with integrated LED light and sound to create an immersive fitness experience.

경청해주셔서고맙습니다!