›› 2020, Vol. 26 ›› Issue (第1): 134-144.DOI: 10.13196/j.cims.2020.01.014

Previous Articles     Next Articles

Research on image affection tagging based on multi-modality information fusion

  

  • Online:2020-01-31 Published:2020-01-31
  • Supported by:
    Project supported by the Humanities and Social Sciences Research Foundation of MOE,China(No.17YJC870018),the National Natural Science Foundation,China(No.61702454),the Fundamental Research Funds for the Provincial Universities of Zhejiang Province,China(No.GB201901006),and the Zhejiang Provincial Natural Science Foundation,China(No.LY20F020028).

基于多模态信息融合的图像情感标注方法

唐智川,刘肖健,杨红春,卢纯福+   

  1. 浙江工业大学工业设计研究院
  • 基金资助:
    教育部人文社会科学研究资助项目(17YJC870018);国家自然科学基金资助项目(61702454);浙江省属高校基本科研业务费专项资金资助项目(GB201901006);浙江省自然科学基金资助项目(LY20F020028)。

Abstract: With the increase of image resource in the web,affection,as one of important semantics of image,is the essential basic to retrieve and select images.Therefore,the image affection tagging has been widely concerned.In this paper,we propose an image affection tagging method based on multi-modality information fusion(EEG and image content).Firstly,spectral features of EEG and color and texture features of image are extracted.Then,based on two kinds of features and two kinds of fusion strategy(feature level and decision level),support vector machine(SVM)classification model is built for the image affection tagging and classification.The IAPS data set is used to evaluate the effectiveness of the proposed method.The results demonstrate that the image affection tagging method based on multi-modality information fusion has a better classification performance than the method only using EEG feature or image feature.Besides,our proposed method contribute to narrowing the semantic gap between low-level visual features and high-level emotion semantic.

Key words: image affection tagging, multi-modality information fusion, electroencephalogram, image content, international affective picture system

摘要: 随着互联网中图像资源的不断增长,情感作为图像的一个重要语义,是人们检索和选择图像的重要依据,因此对于图像进行情感标注显得至关重要。结合脑电信号(EEG)和图像内容,提出了一种基于多模态信息融合的图像情感标注方法。首先,提取EEG频域特征及图像特征(颜色及纹理);其次,结合两者特征信息,基于两种融合策略(特征层和决策层),构建支持向量机分类模型,进行图像情感识别与标注。为了评估方法的有效性,使用国际情绪图片系统公共数据集进行了实验验证。结果表明,提出的多模态信息融合图像情感标注方法优于单独使用EEG或图像内容的标注方法。此外,该成果有助于缩小低层视觉特征和高层情感语义之间的语义鸿沟。

关键词: 图像情感标注, 多模态信息融合, 脑电信号, 图像内容, 国际情绪图片系统

CLC Number: