WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo...

2/21/16

1

01 WORLD IA DAY 2016 PRESENTATION TITLE HERE

HEADER OPTION SUB HEAD OR SHORT DESCRIPTION!

Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.

Some kind of illustration or image?

Affective UX"Challenges in UX design involving affective computing

Alfredo Sánchez

Universidad de las Américas Puebla

©2006-2014,AlfredoSánchez

h8p://www.udlap.m

x

Architecturing happiness Howdoweknowusersarehappy?

CaninteracIvesystemsdetecthappiness(orthelackthereof)?

CaninteracIvesystemsintenIonallyproducehappinesss?

Cantechnologyatleastnotgetintheway?


Agenda

•  TermsofaffecIon• AffecIvecompuIng

•  DetecIon•  Synthesis•  MediaIon

• AffectinUX• Recentdevelopment• Openissues

Terms of affection


Emotion

AreacIonexperiencedusuallytowardaspecificobjectandtypicallyaccompaniedbyphysiologicalandbehavioralchangesinthebody


2/21/16

2

Mood

• Aconsciousstateofmind(typicallylasInglongerthananemoIon)

joy depression


Sentiment

• AnaYtudeoropinion–astatethatresultsfroma8ributesassignedtoobjects,whicharegroupedintocategories


The affect spectrum

emoIon

mood

senIment

may trigger

may prompt

Duration ( - ) ( + ) ©2006-2014,AlfredoSánchez

Affective computing


(Computersytems)relatedto,arisingfrom,ordeliberatelyinfluencingaffecIvephenomena- MITAffec)veCompu)ngGroup

AffecIveCompuIngRosalindPicard1995

Affective computing (for engineers) J

Thestudyanddevelopmentofsystemsanddevicesthatcanrecognize,interpret,process,simulateandmediatehumanaffects.


Why affective computing?


2/21/16

3



Ichoosenottofeel

h8p://anim

ala.mx/no

Icias/20

15/02/27

/adios-m

r-spock/

Why Affective Computing

•  “Reasonis,andoughttobe,onlytheslaveofthepassions”

DavidHume,1711-1776• WithoutpassionswewouldlackallmoIvaIon,allimpulseordrivetoact,oreventoreason



• Alove/haterelaIonshipwiththecomputerandITsingeneral


h8ps://w

ww.you

tube

.com

/watch?v=ZcZBT

B-dkqQ

Why affective computing? (for engineers) J

•  StudiesbymanyresearchersshowavitalroleofemoIoninmanyprocesses:

•  PercepIon•  Decisionmaking•  CreaIvity•  Comprehension•  Memory•  SocialinteracIon


h8p://www.bintegra.com/w

p-conten

t/up

loads/2012/05/Fotolia_socila_u

sers_S.jp

g

Some application areas •  EducaIon–intelligenttutoring•  Health–monitoringpaIents•  Safety–monitoringdrivers/pilots•  Security–detecInglies•  E-commerce–aYtudestowardsproductsorservices

•  Videogames–producingengagingenvironments,enrichingUX

•  Userstudies–incorporaIngemoIonintoevaluaIon


Why affective computing (for our community)

EmoIonisthewaywecommunicateourexperience


2/21/16

4

Main aspects of affective computing

perception

synthesis

mediation


(ofemoIons,moodsandsenIments)

Affect detection


Emotion detection: From facial cues


h8p://npx.auIsmspot.com/facial_expression.jpg

Emotion detection: From postural cues


h8p://instrid

ebod

ywork.com/w

p-conten

t/up

loads/2013/05/po

stures-of-e

moI

on1.jpg

Emotion detection: From voice cues


h8p://pandodaily.files.wordpress.com/2013/07/whisper-in-ear.jpg?w=584&h=329

h8p://gigaom

2.files.wordp

ress.com

/2013/07/emoI

on-spe

ech.jpg?w=300&h=

200&

crop

=1

Emotion detection: From usage patterns


h8p://web

.deepn

etsecurity.com/produ

cts2/typesen

se.asp

Keystrokedynamics NavigaIonpa8erns Eyetracking

2/21/16

5

Emotion detection: From brain activity


Emo)

vEPOCse

nsor

Hond

aBC

Isystem

Emotion detection: From sensors of autonomic activity •  Skinconductanceincreasesevenwithali8lesurpriseoranxiety

• Bloodvolume/pressureincreaseswithangerorstress

• Respiratoryrhythmvarieswithfear,sadness,happiness,anxiety,fear

• Muscletensionincreaseswithsurpriseandanxiety


Affec)vasensor

NeXusse

nsor

Emotion detection: From sensors of autonomic activity


“Knowyourinnerself”W/Me

“Hackhappiness”Feel(bySenIo)

Some general techniques to support affective computing

FordetecIon,synthesisandmediaIon


Facial Action Coding System (FACS)


h8p://www.th

eroadtosiliconvalley.com/tag/Im-roth/

FACS •  FacialAcIonCodingSystem(FACS),verypopular• DefinesAcIonUnits(AUs),contracIonorrelaxaIonofoneormorefacemuscles

…


2/21/16

6

FACS (for “universal” emotions)

h8p://www.paulekm

an.com

/facs/


Posture, motion and gesture analysis


Posture analysis

h8p://pe

ople.ict.u

sc.edu

/~gratch/CSCI534/ACII-H

andb

ook-Ge

stureSyn.pdf


Posture and motion analysis

•  E.g.armmovementsandemoIon:


h8p://pe

ople.ict.u

sc.edu


andb

ook-Ge

stureSyn.pdf

Gesture analysis


h8p://pe

ople.ict.u

sc.edu


andb

ook-Ge

stureSyn.pdf

MPEG-4

• MPEG-4(ISO144996)defines•  86FaceAnimaIonParameters(FAPs)•  196BodyAnimaIonParameters(BAPs)


2/21/16

7

Some existing tools

•  FaceReader,SHORE,eMoIon,RealEyes


Some challenges

• Prototypicalvs.non-prototypicalemoIons• Purevs.combinedemoIons• MulIplesubjects• Partlycoveredfaces/bodies•  Trackingfaceorbodymovement•  LighIngcondiIons

•  IntegraIngtechniques•  SupporIngapplicaIondevelopment


Emotion in UX evaluation


Evaluating user experience with novel technologies – impact of emotion

©2006-2014,AlfredoSánchez[Magallanes, Molina, Méndez & Sánchez 2012]

Gestures + Contextual Menus Heuristic evaluation

• Problemsobservedinactualuse


•  IssuesidenIfiedbyexpertsID Issue Occurrences

P1 Movement of objects using two fingers. 13

P2 Size of the keyboard. 8 P3 Finding help for the user. 5 P4 Finding the toolbar associated

with each object. 4

P5 Navigating using the fan menu. 3 P6 Navigating between nodes. 3 P7 Identifying the icon for linking to

new pages 2

P8 Incomplete or inaccessible menus. 1

ID Task Issue

1 Insert two images, change their position and size. P1

2 Enter a text next to the images as a caption. P2

3 Obtain the toolbar for an image. P3, P4

4 Pop a menu and discover how it works. P5

5 Build a hierarchy of linked workspaces. P6, P7

6 Get the menu in different positions. Cause the menu to go off the screen.

P8

7 Insert two images, change their position and size. P1

8 Enter a text next to the images as a caption. P2

2/21/16

8

Additional feedback on emotions, based on Product Emotion Measurement Tool (PrEmo)

Disgust Indignation Contempt Dissatisfaction Disappointment Unpleasantsurprise

boredom

Desire Inspiration Admiration Satisfaction Fascination Pleasantsurprise

Amusement

[Desmet 2002]

ParIcipantswereaskedtoselectananimatedrepresenta:onincorrespondencewiththeiremoIonalreacIons


[IncollaboraIonwithUniversidaddelCauca,Colombia]

Positive emotions in spite of problems!

©2006-2016,AlfredoSánchez!

! !PosiIveemoIonsassociatedwithnovelty?

NegaIveemoIonsassociatedwithknowninterfaceelements?[Magallanes, Molina, Méndez & Sánchez 2012]

Implications

•  TechniquesforevaluaIonofusabilityanduserexperienceareneededthattakeaffectintoaccount

•  FrameworkdevelopedatUniCauca,Colombia[Méndez,Collazos,Granollers,Sánchez2014;Méndez2015]


EmoVal

• AframeworkforemoIonalevaluaIondesignedtoenrichexisIngUXevaluaIonmethods

•  Threemajorcomponents•  EmoVal-C:Behavioralevalua:on(facialexpressions,gaze,posture,movements)

•  EmoVal-S:Self-reportofemo:onswhileusingasystem•  EmoVal-F:Physiologicalsensing,mainlybrainacIvityduringsystemusage

•  SupporIngwebsite(h8p://artemisa.unicauca.edu.co/~ymendal/EmoVal)

©2006-2014,AlfredoSánchez[Méndez2015]

EmoVal: Detailed support / facial expressions

©2006-2014,AlfredoSánchez[Méndez2015]

EmoVal: Detailed support / posture and movement

©2006-2014,AlfredoSánchez [Méndez2015]

2/21/16

9

Building capabilities for affect detection


Current scenario: Extending an app to include more techniques is not straightforward

Facial&cue&analysis&Monitoring&pilot&alertness&

Lie&detec4on&

Usability&tes4ng&

Online&tutoring&

Posture&analysis&

Voice&analysis&

Keystroke&dynamics&

Eye&tracking&

Naviga4on&pa@erns&

ApplicaIondomain Technique Technologies


Goal: Integration platform

Usability)tes+ng)

Eye)tracking)

Vikara)Integra+on)Services)

Vikara)Monitoring)Services)

Lie)detec+on)

Online)tutoring)

Alertness)monitor)

Naviga+on)pa@erns)

Keystroke)dynamics)

Voice)analysis)

Posture)analysis)

Facial)cue)analysis)

Visualiza+on)interfaces)

©2006-2014,AlfredoSánchez[Sánchezetal.2015]

Architecture for implementation

Database

Controller EmotionsController

View

Data Model

Home - Index - UploadUserContent

HomeController

SessionsController

Vikara Services Interface

Data entities

cshtml Session - Index - List

EmotionModel

UserContentController

FileController

Emotion

UserContent Session

MS SQL

Resources Google Chart Tools

WebGrease Modernizr

Android Emotion Interpreter

Windows Emotion Interpeter

Server

Visualization interfaces

Other Emotion Interpreters

Observer / administrator

End

user

s


Simple data model and messaging protocol

!©2006-2014,AlfredoSánchez

[Sánchezetal.2015]

Prototypical apps: Kinect-based emotion detection


14sorpresa

[Sánchezetal.2015]

2/21/16

10

Prototypical apps: Kinect-based emotion detection


Prototype evaluation

!


Prototypical apps: Android-based emotion self reporter


![Sánchezetal.2015]

Prototype: Monitoring services


UX for BCI

h8p://www.gadgeIfy.com/w

p-conten

t/up

loads/2014/09/15/epo

c.pn

g

Experimenting with EPOC+

2/21/16

11

What about sentiment?


Sentiment analysis

• BydefiniIon,TheuseofNLPandcomputaIonallinguisIcstoidenIfyaLtudes

inspeakersorwriterswithrespecttospecificsubjects


h8p://datameer2.datam

eer.c

om/blog/wp-conten

t/up

loads/2012/10/22nd

.png


Applications of sentiment analysis

DetecIng/studying:• ResponsestomarkeIng/poliIcalcampaigns• UsersaIsfacIon• Biasinnews• Appropriatenessofcontentsfordifferentaudiences


Some techniques

• MostlyheurisIcmethods•  Socialmediaanalysis

• Keyword-based•  SemanIcapproaches• Annotateddatasets•  SupportVectorMachines(SVMs)forclassificaIon


ExpressionArchitecture

AflexiblearchitectureenablesExpressiontolistenfromanydatasourceandhaveanindependentcomponent-basedfront-end.

Expression–SenImentAnalysisPlazormatUDLAPArchitecture

[Cervantes,GuIérrez,GuIérrez,CasIllo,Sánchez,Wan2015]

2/21/16

12

ThefrontendofExpressionisbasedonsen:mentscardsthatrepresenttopicsofinterest.

SenImentcards

h8ps://github.com/Innova4D/expression

ASen:mentCardgraphicallysummarizesopiniontrendsinsocialmedia

SenImentCard

Some challenges

•  LanguagesubtleIes–Irony,sarcasm,abbreviaIons,misspellings,idioms

• RepresentaIveness–DissaIsfieduserstweetmoreacIvely• ContextdependentsemanIcs•  Spam• ClassificaIonbeyondposiIveornegaIve


Affect mediation


Mediation: Conveying affect through ICTs

• Videoconferencing

•  Immersiveenvironments

• AsynchronouscommunicaIon•  Email,forums,blogs…


Some issues and challenges •  AudioandvideoqualityoverInternet

•  QualityofmulImediaexperience(h8p://i-guide.org/)

•  Metrics,assessmentmethods

•  Gazeawareness

•  3Dmodeling•  Issuessharedwithaffectsynthesis


2/21/16

13

Emotion over low data rates: Sending codes instead of full video

IncollaboraIonwithUniversityofAgder,Norway

h8p://w

ww.pau

lekman.c

om/fac

s/

1.  Sensing2.  Encoding3.  TransmiYng4.  ReconstrucIng

[Lamei,Li,Sánchez2014]

Conveying mood


Instant messaging •  EmoIconsworkforemoIons,instantreacIonsassociatedtospecificstatements

• Moodislongerlas:ng,overarching•  IdenIfyingmoodcanbeachallenge,evenforthepersoninthataffecIvestate


Russell’s Circumplex Model can help identify mood

•  Two-dimensional,valence-arousalmodel

• ConsistentspaIallocaIoninself-reportexperimentsacrossindividualsandacrosscultures/languages


[Russell1983]

Extending Russell’s model - Intensity


Extended, discrete model


UsertesIngshowedaccuracyinaffectidenIficaIonButusingthemodeldirectlywouldresultinaclu8eredinterface

2/21/16

14

Gradual revelation

annoyed

happy

sad

calm

Representative affectives states per quadrant (according to user testing)

sleepy

aroused

upset satisfied


Emoticon design


Prototype (“Russkman”)

[Sánchezetal.2006]©2006-2014,AlfredoSánchez

Prototype


Prototype


Evaluation

• Chatsessionsamong2-3subjects• HypotheIcalsituaIons(upcomingexams,goodgrades,recentparty,lostwallet…)

•  Subjectswereabletospecifytheirmoodrapidly•  SubjectswereabletoidenIfymoodofotherusers(highcorrelaIon)• PersistentemoIconshelpedinkeepingawarenessofmoods


2/21/16

15

Affect synthesis: Modeling and displaying

emotion and mood

Synthesizing emotions

•  IfanemoIon(ormoodorsenIment)isrecognizedintheuser,thecomputer(oranadapIveenvironment)could(should?)respondaccordingly

• ApproachestoemoIonsynthesis•  ReasoningaboutemoIons—givenasetofcondiIons,producespecificreacIons

•  Characterswithpersonality(temperament,senIment)


Emotion synthesis Characterswithanimatedfacialexpressions(facialcuesforsensingalsoapplyhere)

[CSLU]


Responding to emotion

AffecIveCompanionTextfromautomaIcspeechrecognizerandEmoVoice(fortheuser)Naturallanguageunderstanding [Cavazzaetal.2010]


FACS applied to emotion synthesis

FACSGe

n[Roe

schetal.2010]


Affective animated avatars

• DesignusingpopulartoolsandareducedsetofFACS’AUs

• Availableath8p://chopis.com

•  TesInginMéxicoandChina


2/21/16

16

Some challenges

• Prototypicalvs.non-prototypicalemoIons• Purevs.combinedemoIons•  IntegraIngfacial,postureandvoiceexpressions• OfferingserviceplazormsforappconstrucIon


Perspectives of affective UX

• AffectiskeyinUXdesignandevaluaIon

• OpportuniIesforresearchanddevelopmenttoimprovedetecIon,synthesisandmediaIon

• WorkneededinmoodandsenImentanalysis

• OpportuniIesforinter-disciplinarycollaboraIon


Affective computing becoming mainstream?


“Pepper”,So}Bank’saffect-awarerobot,1,000unitssoldout,December2015,foraboutUSD1,800…InvolvesfacialrecogniIon,videoandaudioprocessing,sensors,machinelearning,…

But wait…

•  Forward-lookingstatementsarebytheirnaturesubjecttovariousrisksanduncertain:es,including,withoutlimitaIon,adeclineingeneraleconomiccondiIons,generalmarketcondiIons,technologicaldevelopments,changesincustomerdemandforproductsandservices,increasedcompeIIon,risksassociatedwithinternaIonaloperaIons,andotherimportantfactors,eachofwhichmaycauseactualresultsandfuturedevelopmentstodiffermateriallyfromthoseexpressedorimpliedinanyforward-lookingstatement.[…]Useoforrelianceontheinforma:oninthispresenta:onisatyourownrisk.

h8p://webcast.so}bankmobile.co.jp/en/conference/20140605/index.html


01 WORLD IA DAY 2016 PRESENTATION TITLE HERE

HEADER OPTION SUB HEAD OR SHORT DESCRIPTION!

Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.

Some kind of illustration or image?

Affective UX"Challenges in UX design involving affective computing

Alfredo Sánchez

Universidad de las Américas Puebla http://ict.udlap.mx

References •  Cervantes,O.,GuIérrez,F.,GuIérrez,E.,CasIlloE.,Sánchez,J.A.,Wan,W.2015.Expression:UnderstandingaffecIvecontentinsocialmedia.7thLaInAmericanConferenceonHuman-computerInteracIon(CLIHC2015),Cordoba,ArgenIna,Nov.18-23.AvailablefromtheACMDigitalLibrary(h8p://dl.acm.org/citaIon.cfm?doid=2824893.2824903).

•  Magallanes,Y.,Molina,A.,Méndez,Y.,Sánchez,J.A.2012.CombininggesturesandgraphicalelementsforcollaboraIonusingmulItouchsurfaces.Proceedingsofthe22ndInternaIonalConferenceonElectronics,CommunicaIonsandComputers(CONIELECOMP2012,Feb.).IEEEComputerSocietyPress,173-178.

•  Magallanes,Y.,Molina,A.,Méndez,Y.,Sánchez,J.A.2012.TowardsanemoIonalvalidaIonofheurisIcapproachesforusabilityevaluaIon.ActaUniversitaria22,3,119-125.(h8p://www.actauniversitaria.ugto.mx/index.php/acta/arIcle/view/351).

•  Méndez,Y.A.,Collazos,C.A.,Sánchez,J.A.,MerchánParedes,L.2012.CriteriosparalaevaluacióndesuperficiesmulItácIlesysusaplicaciones.Lámpsakos7(Jan.),64-68.(disponibleenh8p://www.funlam.edu.co/revistas/index.php/lampsakos/arIcle/view/847).

•  Méndez,Y.A.2015.Marcodereferenciaparalacapturasemi-invasivayanálisisdelasemocionesenelprocesodeevaluaciónconusuariosdesistemasinteracIvos.Tesisdoctoral.UniversidaddelCauca,Colombia.Disponibleenh8p://www.unicauca.edu.co/doctoradoce/publicaciones/Monografia_Mendez.pdf


2/21/16

17

References •  Sánchez,J.A.,Hernández,N.P.,Penagos,J.C.,Ostróvskaya,Y.2006.ConveyingmoodandemoIonininstantmessagingbyusingatwo-dimensionalmodelforaffecIvestates.ProceedingsoftheSymposiumonHumanFactorsinComputerSystems(IHC2006,Natal,Brazil),66-72.AvailablefromtheACMDigitalLibrary(h8p://dl.acm.org/citaIon.cfm?id=1298033&CFID=582956970&CFTOKEN=61490837).

•  Méndez,Y.A.,Collazos,C.A.,Granollers,T.,Sánchez,J.A.2014.EvaluaInginteracIvesystemsfromanemoIonalperspecIve.RevistaCien�ficaGuillermodeOckham12(1),43-49.(disponibleenh8p://www.redalyc.org/pdf/1053/105332478005.pdf).

•  Sánchez,J.A.,Cortés,X.,Starostenko,O.,Cervantes,O.,Wan,W.2015.AnExtensiblePlazormforSeamlessIntegraIonandManagementofApplicaIonsforEmoIonSensingandInterpretaIon.JournalofAmbientIntelligenceandSmartEnvironments7(1),5-19.(availablefromh8p://iospress.metapress.com/content/y1g64422t78107v7).

•  Starostenko,O.,Cortés,X.,Sánchez,J.A.2015.UnobtrusiveemoIonsensingandinterpretaIoninsmartenvironments.JournalofAmbientIntelligenceandSmartEnvironments7,59-83.(avalilablefromh8p://iospress.metapress.com/content/w2681p2827l2u22p/).


WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo...

Education

Transcript of WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo...