Colorless green ideas do sleep furiously

Gradient acceptability and the nature of the grammar

Jon Sprouse, Beracah Yankama, Sagar Indurkhya, Sandiway Fong, Robert C. Berwick

Research output: Contribution to journalArticle

Abstract

In their recent paper, Lau, Clark, and Lappin explore the idea that the probability of the occurrence of word strings can form the basis of an adequate theory of grammar (Lau, Jey H., Alexander Clark & 15 Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A prob- abilistic view of linguistic knowledge. Cognitive Science 41(5):1201-1241). To make their case, they present the results of correlating the output of several probabilistic models trained solely on naturally occurring sentences with the gradient acceptability judgments that humans report for ungrammatical sentences derived from roundtrip machine translation errors. In this paper, we first explore the logic of the Lau et al. argument, both in terms of the choice of evaluation metric (gradient acceptability), and in the choice of test data set (machine translation errors on random sentences from a corpus). We then present our own series of studies intended to allow for a better comparison between LCL's models and existing grammatical theories. We evaluate two of LCL's probabilistic models (trigrams and recurrent neural network) against three data sets (taken from journal articles, a textbook, and Chomsky's famous colorless-green-ideas sentence), using three evaluation metrics (LCL's gradience metric, a categorical version of the metric, and the experimental-logic metric used in the syntax literature). Our results suggest there are very real, measurable cost-benefit tradeoffs inherent in LCL's models across the three evaluation metrics. The gain in explanation of gradience (between 13% and 31% of gradience) is offset by losses in the other two metrics: a 43%-49% loss in coverage based on a categorical metric of explaining acceptability, and a loss of 12%-35% in explaining experimentally-defined phenomena. This suggests that anyone wishing to pursue LCL's models as competitors with existing syntactic theories must either be satisfied with this tradeoff, or modify the models to capture the phenomena that are not currently captured.

Original languageEnglish (US)
JournalLinguistic Review
DOIs
StateAccepted/In press - May 29 2018

Fingerprint

sleep
grammar
evaluation
neural network
syntax
textbook
Grammar
Sleep
Acceptability
coverage
linguistics
costs
Gradience
Evaluation

Keywords

  • acceptability
  • gradience
  • grammaticality
  • n-grams
  • probability
  • recurrent neural networks

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language

Cite this

Colorless green ideas do sleep furiously : Gradient acceptability and the nature of the grammar. / Sprouse, Jon; Yankama, Beracah; Indurkhya, Sagar; Fong, Sandiway; Berwick, Robert C.

In: Linguistic Review, 29.05.2018.

Research output: Contribution to journalArticle

Sprouse, Jon ; Yankama, Beracah ; Indurkhya, Sagar ; Fong, Sandiway ; Berwick, Robert C. / Colorless green ideas do sleep furiously : Gradient acceptability and the nature of the grammar. In: Linguistic Review. 2018.
@article{74db217c80854fc9b936d295c0198e04,
title = "Colorless green ideas do sleep furiously: Gradient acceptability and the nature of the grammar",
abstract = "In their recent paper, Lau, Clark, and Lappin explore the idea that the probability of the occurrence of word strings can form the basis of an adequate theory of grammar (Lau, Jey H., Alexander Clark & 15 Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A prob- abilistic view of linguistic knowledge. Cognitive Science 41(5):1201-1241). To make their case, they present the results of correlating the output of several probabilistic models trained solely on naturally occurring sentences with the gradient acceptability judgments that humans report for ungrammatical sentences derived from roundtrip machine translation errors. In this paper, we first explore the logic of the Lau et al. argument, both in terms of the choice of evaluation metric (gradient acceptability), and in the choice of test data set (machine translation errors on random sentences from a corpus). We then present our own series of studies intended to allow for a better comparison between LCL's models and existing grammatical theories. We evaluate two of LCL's probabilistic models (trigrams and recurrent neural network) against three data sets (taken from journal articles, a textbook, and Chomsky's famous colorless-green-ideas sentence), using three evaluation metrics (LCL's gradience metric, a categorical version of the metric, and the experimental-logic metric used in the syntax literature). Our results suggest there are very real, measurable cost-benefit tradeoffs inherent in LCL's models across the three evaluation metrics. The gain in explanation of gradience (between 13{\%} and 31{\%} of gradience) is offset by losses in the other two metrics: a 43{\%}-49{\%} loss in coverage based on a categorical metric of explaining acceptability, and a loss of 12{\%}-35{\%} in explaining experimentally-defined phenomena. This suggests that anyone wishing to pursue LCL's models as competitors with existing syntactic theories must either be satisfied with this tradeoff, or modify the models to capture the phenomena that are not currently captured.",
keywords = "acceptability, gradience, grammaticality, n-grams, probability, recurrent neural networks",
author = "Jon Sprouse and Beracah Yankama and Sagar Indurkhya and Sandiway Fong and Berwick, {Robert C.}",
year = "2018",
month = "5",
day = "29",
doi = "10.1515/tlr-2018-0005",
language = "English (US)",
journal = "Linguistic Review",
issn = "0167-6318",
publisher = "Walter de Gruyter GmbH & Co. KG",

}

TY - JOUR

T1 - Colorless green ideas do sleep furiously

T2 - Gradient acceptability and the nature of the grammar

AU - Sprouse, Jon

AU - Yankama, Beracah

AU - Indurkhya, Sagar

AU - Fong, Sandiway

AU - Berwick, Robert C.

PY - 2018/5/29

Y1 - 2018/5/29

N2 - In their recent paper, Lau, Clark, and Lappin explore the idea that the probability of the occurrence of word strings can form the basis of an adequate theory of grammar (Lau, Jey H., Alexander Clark & 15 Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A prob- abilistic view of linguistic knowledge. Cognitive Science 41(5):1201-1241). To make their case, they present the results of correlating the output of several probabilistic models trained solely on naturally occurring sentences with the gradient acceptability judgments that humans report for ungrammatical sentences derived from roundtrip machine translation errors. In this paper, we first explore the logic of the Lau et al. argument, both in terms of the choice of evaluation metric (gradient acceptability), and in the choice of test data set (machine translation errors on random sentences from a corpus). We then present our own series of studies intended to allow for a better comparison between LCL's models and existing grammatical theories. We evaluate two of LCL's probabilistic models (trigrams and recurrent neural network) against three data sets (taken from journal articles, a textbook, and Chomsky's famous colorless-green-ideas sentence), using three evaluation metrics (LCL's gradience metric, a categorical version of the metric, and the experimental-logic metric used in the syntax literature). Our results suggest there are very real, measurable cost-benefit tradeoffs inherent in LCL's models across the three evaluation metrics. The gain in explanation of gradience (between 13% and 31% of gradience) is offset by losses in the other two metrics: a 43%-49% loss in coverage based on a categorical metric of explaining acceptability, and a loss of 12%-35% in explaining experimentally-defined phenomena. This suggests that anyone wishing to pursue LCL's models as competitors with existing syntactic theories must either be satisfied with this tradeoff, or modify the models to capture the phenomena that are not currently captured.

AB - In their recent paper, Lau, Clark, and Lappin explore the idea that the probability of the occurrence of word strings can form the basis of an adequate theory of grammar (Lau, Jey H., Alexander Clark & 15 Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A prob- abilistic view of linguistic knowledge. Cognitive Science 41(5):1201-1241). To make their case, they present the results of correlating the output of several probabilistic models trained solely on naturally occurring sentences with the gradient acceptability judgments that humans report for ungrammatical sentences derived from roundtrip machine translation errors. In this paper, we first explore the logic of the Lau et al. argument, both in terms of the choice of evaluation metric (gradient acceptability), and in the choice of test data set (machine translation errors on random sentences from a corpus). We then present our own series of studies intended to allow for a better comparison between LCL's models and existing grammatical theories. We evaluate two of LCL's probabilistic models (trigrams and recurrent neural network) against three data sets (taken from journal articles, a textbook, and Chomsky's famous colorless-green-ideas sentence), using three evaluation metrics (LCL's gradience metric, a categorical version of the metric, and the experimental-logic metric used in the syntax literature). Our results suggest there are very real, measurable cost-benefit tradeoffs inherent in LCL's models across the three evaluation metrics. The gain in explanation of gradience (between 13% and 31% of gradience) is offset by losses in the other two metrics: a 43%-49% loss in coverage based on a categorical metric of explaining acceptability, and a loss of 12%-35% in explaining experimentally-defined phenomena. This suggests that anyone wishing to pursue LCL's models as competitors with existing syntactic theories must either be satisfied with this tradeoff, or modify the models to capture the phenomena that are not currently captured.

KW - acceptability

KW - gradience

KW - grammaticality

KW - n-grams

KW - probability

KW - recurrent neural networks

UR - http://www.scopus.com/inward/record.url?scp=85048144900&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048144900&partnerID=8YFLogxK

U2 - 10.1515/tlr-2018-0005

DO - 10.1515/tlr-2018-0005

M3 - Article

JO - Linguistic Review

JF - Linguistic Review

SN - 0167-6318

ER -