Privacy protection in transformer-based neural network

Jiaqi Lang, Linjing Li, Weiyun Chen, Daniel Zeng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the great success of neural networks, it is important to improve the information security of application systems based on them. This paper investigates a scenario where an attacker eavesdrops the intermediate representation computed by the encoder layers and tries to recover the private information of the input text. We propose a new metric to evaluate the encoder's ability to protect privacy and evaluate the Transformer-based encoder, which is the first privacy research conducted on Transformer-based neural networks. We also propose an adversarial training method to enhance the privacy of Transformer-based neural networks.

Original languageEnglish (US)
Title of host publication2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019
EditorsXiaolong Zheng, Ahmed Abbasi, Michael Chau, Alan Wang, Lina Zhou
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages182-184
Number of pages3
ISBN (Electronic)9781728125046
DOIs
StatePublished - Jul 2019
Externally publishedYes
Event17th IEEE International Conference on Intelligence and Security Informatics, ISI 2019 - Shenzhen, China
Duration: Jul 1 2019Jul 3 2019

Publication series

Name2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019

Conference

Conference17th IEEE International Conference on Intelligence and Security Informatics, ISI 2019
CountryChina
CityShenzhen
Period7/1/197/3/19

Fingerprint

Neural networks
Security of data
Privacy
Information security
Scenarios
Private information
Training methods

Keywords

  • Neural network
  • Privacy protection
  • Representation learning
  • Transformer

ASJC Scopus subject areas

  • Safety, Risk, Reliability and Quality
  • Computer Vision and Pattern Recognition
  • Information Systems and Management
  • Information Systems

Cite this

Lang, J., Li, L., Chen, W., & Zeng, D. (2019). Privacy protection in transformer-based neural network. In X. Zheng, A. Abbasi, M. Chau, A. Wang, & L. Zhou (Eds.), 2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019 (pp. 182-184). [8823346] (2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISI.2019.8823346

Privacy protection in transformer-based neural network. / Lang, Jiaqi; Li, Linjing; Chen, Weiyun; Zeng, Daniel.

2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019. ed. / Xiaolong Zheng; Ahmed Abbasi; Michael Chau; Alan Wang; Lina Zhou. Institute of Electrical and Electronics Engineers Inc., 2019. p. 182-184 8823346 (2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lang, J, Li, L, Chen, W & Zeng, D 2019, Privacy protection in transformer-based neural network. in X Zheng, A Abbasi, M Chau, A Wang & L Zhou (eds), 2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019., 8823346, 2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019, Institute of Electrical and Electronics Engineers Inc., pp. 182-184, 17th IEEE International Conference on Intelligence and Security Informatics, ISI 2019, Shenzhen, China, 7/1/19. https://doi.org/10.1109/ISI.2019.8823346
Lang J, Li L, Chen W, Zeng D. Privacy protection in transformer-based neural network. In Zheng X, Abbasi A, Chau M, Wang A, Zhou L, editors, 2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 182-184. 8823346. (2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019). https://doi.org/10.1109/ISI.2019.8823346
Lang, Jiaqi ; Li, Linjing ; Chen, Weiyun ; Zeng, Daniel. / Privacy protection in transformer-based neural network. 2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019. editor / Xiaolong Zheng ; Ahmed Abbasi ; Michael Chau ; Alan Wang ; Lina Zhou. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 182-184 (2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019).
@inproceedings{7af36bcdd3dd4443a203e29b60dc18fc,
title = "Privacy protection in transformer-based neural network",
abstract = "With the great success of neural networks, it is important to improve the information security of application systems based on them. This paper investigates a scenario where an attacker eavesdrops the intermediate representation computed by the encoder layers and tries to recover the private information of the input text. We propose a new metric to evaluate the encoder's ability to protect privacy and evaluate the Transformer-based encoder, which is the first privacy research conducted on Transformer-based neural networks. We also propose an adversarial training method to enhance the privacy of Transformer-based neural networks.",
keywords = "Neural network, Privacy protection, Representation learning, Transformer",
author = "Jiaqi Lang and Linjing Li and Weiyun Chen and Daniel Zeng",
year = "2019",
month = "7",
doi = "10.1109/ISI.2019.8823346",
language = "English (US)",
series = "2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "182--184",
editor = "Xiaolong Zheng and Ahmed Abbasi and Michael Chau and Alan Wang and Lina Zhou",
booktitle = "2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019",

}

TY - GEN

T1 - Privacy protection in transformer-based neural network

AU - Lang, Jiaqi

AU - Li, Linjing

AU - Chen, Weiyun

AU - Zeng, Daniel

PY - 2019/7

Y1 - 2019/7

N2 - With the great success of neural networks, it is important to improve the information security of application systems based on them. This paper investigates a scenario where an attacker eavesdrops the intermediate representation computed by the encoder layers and tries to recover the private information of the input text. We propose a new metric to evaluate the encoder's ability to protect privacy and evaluate the Transformer-based encoder, which is the first privacy research conducted on Transformer-based neural networks. We also propose an adversarial training method to enhance the privacy of Transformer-based neural networks.

AB - With the great success of neural networks, it is important to improve the information security of application systems based on them. This paper investigates a scenario where an attacker eavesdrops the intermediate representation computed by the encoder layers and tries to recover the private information of the input text. We propose a new metric to evaluate the encoder's ability to protect privacy and evaluate the Transformer-based encoder, which is the first privacy research conducted on Transformer-based neural networks. We also propose an adversarial training method to enhance the privacy of Transformer-based neural networks.

KW - Neural network

KW - Privacy protection

KW - Representation learning

KW - Transformer

UR - http://www.scopus.com/inward/record.url?scp=85072949978&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072949978&partnerID=8YFLogxK

U2 - 10.1109/ISI.2019.8823346

DO - 10.1109/ISI.2019.8823346

M3 - Conference contribution

AN - SCOPUS:85072949978

T3 - 2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019

SP - 182

EP - 184

BT - 2019 IEEE International Conference on Intelligence and Security Informatics, ISI 2019

A2 - Zheng, Xiaolong

A2 - Abbasi, Ahmed

A2 - Chau, Michael

A2 - Wang, Alan

A2 - Zhou, Lina

PB - Institute of Electrical and Electronics Engineers Inc.

ER -