1.2: Intuition

Intuited data would not be the only alternative to corpus data, but it is the one proposed and used by critics of the latter, so let us look more closely at this practice. Given the importance of grammaticality judgments, one might expect them to have been studied extensively to determine exactly what it is that people are doing when they are making such judgments. Surprisingly, this is not the case, and the few studies that do exist are hardly ever acknowledged as potentially problematic by those linguists that routinely rely on them, let alone discussed with respect to their place in scientific methodology.

One of the few explicit discussions is found in Jackendoff (1994). Jackendoff introduces the practice of intuiting grammaticality judgments as follows:

[A]mong the kinds of experiments that can be done on language, one kind is very simple, reliable, and cheap: simply present native speakers of a language with a sentence or phrase, and ask them to judge whether or not it is grammatical in their language or whether it can have some particular meaning. [...] The idea is that although we can’t observe the mental grammar of English itself, we can observe the judgments of grammaticality and meaning that are produced by using it (Jackendoff 1994: 47, emphasis mine).

This statement is representative of the general assumptions underlying the practice of grammaticality judgments in generative linguistics (and many other frameworks) in two ways: first, in that it presents individual grammaticality judgments as a kind of scientific experiment on a par with more sophisticated experiments, and second, in that it presents grammaticality judgments as a direct reflection of a speaker’s mental representation of the language in question. Jackendoff briefly touches upon a crucial problem of the first assumption:

Ideally, we might want to check these experiments out by asking large numbers of people under controlled circumstances, and so forth. But in fact the method is so reliable that, for a very good first approximation, linguists tend to trust their own judgments and those of their colleagues (Jackendoff 1994: 48).

It is certainly true that linguists trust their own judgments, but that does not mean, of course, that this trust is justified. There is little evidence that individual grammaticality judgments are reliable: in the linguistic literature, grammaticality judgments of the same sentences by different authors often differ considerably and the few studies that have investigated the reliability of grammaticality judgments have consistently shown that such judgments display too much variation within and across speakers to use them as linguistic data (cf., e.g., Schütze (1996) (reissued under a Creative-Commons license by Language Science Press in 2016), esp. Ch. 3 on factors influencing grammaticality judgments, and Cowart (1997)).

The attraction of grammaticality judgments lies not so much in their reliability, then, but in the ease with which they can be collected, and Jackendoff is very explicit about this when he says that

other kinds of experiments can be used to explore properties of the mental grammar [...] Their disadvantage is their relative inefficiency: it takes a great deal of time to set up the experiment. By contrast, when the experiment consists of making judgments of grammaticality, there is nothing simpler than devising and judging some more sentences (Jackendoff 1994: 49).

However, the fact that something can be done quickly and effortlessly does not make it a good scientific method. If one is serious about using grammaticality judgments – and there are research questions that are not easily addressed without them –, then these judgments must be made as reliable as possible; among other things, this involves the two aspects mentioned by Jackendoff in passing: first, asking large numbers of speakers (or at least more than one) and, second, controlling the circumstances under which they are asked (cf. Schütze 1996 and Cowart 1997 for detailed suggestions as to how this is to be done and Bender 2005 for an interesting alternative; cf. also Section 4.2.3 in Chapter 4). In order to distinguish such empirically collected introspective data from data intuited by the researcher, I will refer to the former as elicitation data and continue to reserve for the latter the term intuition or intuited “data”.

In sum, there are serious problems with the reliability of linguistic intuition in general, a point I will briefly return to in Section 1.3. In the case of isolated judgments by the researchers themselves, these problems are compounded by two additional ones: first, the researchers are language experts, whose judgments will hardly be representative of the average native speaker – as Ronald Langacker has quipped (in an example sentence meant to illustrate syntactic complexity): “Linguists are no different from any other people who spend nineteen hours a day pondering the complexity of grammar [...]” (Langacker 1973: 109). Second, they will usually know what it is that they want to prove, and this will distort their judgments. Thus, expert judgments should be used with extreme caution (cf. Labov 1996) if at all (Schütze 1996), instead of serving as the default methodology in linguistics.

Let us return to the second assumption in the passage quoted above – that grammaticality judgments are transparently related to the mental grammar of the speaker producing them. In particular, let us discuss whether intuited “data” fare better than corpus data in terms of the three major points of criticism discussed in the preceding section:

1. Are intuited “data” a more direct reflection of linguistic knowledge (competence) than corpus data;
2. are intuited “data” more complete than corpus data; and
3. do intuited “data” contain information about the semantics, pragmatics, etc. of these forms.

1.2.1 Intuition as performance

The most fundamental point of criticism leveled against corpus data concerns the claim that since corpora are samples of language use (“performance”), they are useless in the study of linguistic knowledge (“competence”). I argued in Section 1.1.1 above that this claim makes sense only in the context of rather implausible assumptions concerning linguistic knowledge and linguistic usage, but even if we accept these assumptions, the question remains whether intuited judgments are different from corpus data in this respect.

It seems obvious that both inventing sentences and judging their grammaticality are kinds of behavior and, as such, performance in the generative linguistics sense. In fact, Chomsky himself admits this:

[W]hen we study competence – the speaker-hearer’s knowledge of his language – we may make use of his reports and his behavior as evidence, but we must be careful not to confuse “evidence” with the abstract constructs that we develop on the basis of evidence and try to justify in terms of evidence. [...] Since performance – in particular, judgments about sentences – obviously involves many factors apart from competence, one cannot accept as an absolute principle that the speaker’s judgments will give an accurate account of his knowledge. (Chomsky 1972: 187, emphasis mine).

There is little to add to this statement, other than to emphasize that if it is possible to construct a model of linguistic competence on the basis of intuited judgments that involve factors other than competence, it should also be possible to do so on the basis of corpus data that involve factors other than competence, and the competence/performance argument against corpus data collapses.

1.2.2 The incompleteness of intuition

Next, let us turn to the issue of incompleteness. As discussed in Section 1.1.2, corpus data are necessarily incomplete, both in a quantitative sense (since every corpus is finite in size) and in a qualitative sense (since even the most carefully constructed corpus is skewed with respect to the language varieties it contains). This incompleteness is not an argument against using corpora as such, but it might be an argument in favor of intuited judgments if there was reason to believe that they are more complete.

To my knowledge, this issue has never been empirically addressed, and it would be difficult to do so, since there is no complete data set against which intuited judgments could be compared. However, it seems implausible to assume that such judgments are more complete than corpus data. First, just like a corpus, the linguistic experience of a speaker is finite and any mental generalizations based on this experience will be partial in the same way that generalizations based on corpus data must be partial (although it must be admitted that the linguistic experience a native speaker gathers over a lifetime exceeds even a large corpus like the BNC in terms of quantity). Second, just like a corpus, a speaker’s linguistic experience is limited to certain language varieties: most English speakers have never been to confession or planned an illegal activity, for example, which means they will lack knowledge of certain linguistic structures typical of these situations.

To exemplify this point, consider that many speakers of English are unaware of the fact that there is a use of the verb bring that has the valency pattern (or subcategorization frame) [bring NP_LIQUID [_PP to the boil]] (in British English) or [bring NP_LIQUID [_PP to a boil]] (in American English). This use is essentially limited to a single genre, – recipes: of the 145 matches in the BNC, 142 occur in recipes and the remaining three in narrative descriptions of someone following a recipe. Thus, a native speaker of English who never reads cookbooks or cooking-related journals and websites and never watches cooking shows on television can go through their whole life without encountering the verb bring used in this way. When describing the grammatical behavior of the verb bring based on their intuition, this use would not occur to them, and if they were asked to judge the grammaticality of a sentence like Half-fill a large pan with water and bring to the boil [BNC A7D], they would judge it ungrammatical. Thus, this valency pattern would be absent from their description in the same way that transitive croak ‘die’ or [it doesn’t matter the N] would be absent from a grammatical description based on the BNC (where, as we saw in Section 1.1.2, these patterns do not occur).

If this example seems too speculative, consider Culicover’s analysis of the phrase no matter (Culicover 1999: 106f.). Culicover is an excellent linguist by any standard, but he bases his intricate argument concerning the unpredictable nature of the phrase no matter on the claim that the construction [it doesn’t matter the N] is ungrammatical. If he had consulted the BNC, he might be excused for coming to this wrong conclusion, but he reaches it without consulting a corpus at all, based solely on his native-speaker intuition.⁴

1.2.3 Intuitions about form and meaning

Finally, let us turn to the question whether intuited “data” contain information about meaning. At first glance, the answer to this question would appear to be an obvious “yes”: if I make up a sentence, of course I know what that sentence means. However, a closer look shows that matters are more complex and the answer is less obvious. Constructing a sentence and interpreting a sentence are two separate activities. As a consequence, I do not actually know what my constructed sentence means, but only what I think it means. While I may rightly consider myself the final authority on the intended meaning of a sentence that I myself have produced, my interpretation ceases to be privileged in this way once the issue is no longer my intention, but the interpretation that my constructed sentence would conventionally receive in a particular speech community. In other words, the interpretation of a constructed sentence is subjective in the same way that the interpretation of a sentence found in a corpus is subjective. In fact, interpreting other people’s utterances, as we must do in corpus linguistic research, may actually lead to more intersubjectively stable results, as interpreting other people’s utterances is a more natural activity than interpreting our own: the former is what we routinely engage in in communicative situations, the latter, while not exactly unnatural, is a rather exceptional activity.

On the other hand, it is very difficult not to interpret a sentence, but that is exactly what I would have to do in intuiting grammaticality judgments – judging a sentence to be grammatical or ungrammatical is supposed to be a judgment purely about form, dependent on meaning only insofar as that meaning is relevant to the grammatical structure. Consider the examples in (3):

(3) a. When she’d first moved in she hadn’t cared about anything, certainly not her surroundings – they had been the least of her problems – and if the villagers hadn’t so kindly donated her furnishings she’d probably still be existing in empty rooms. (BNC H9V)

b. [_VP donated [_NP her] [_NP furnishings] ]

c. [_VP donated [_NP [DET her] [_N furnishings] ] ]

d. Please have a look at our wish-list and see if you can donate us a plant we need. (headway-cambs.org.uk)

The grammaticality of the clause [T]he villagers [...] donated her furnishings in (3a) can be judged for its grammaticality only after disambiguating between the meanings associated with the structures in (3b) and (3c).

The structure in (3b) is a ditransitive, which is widely agreed to be impossible with donate (but see Stefanowitsch 2007a), so the sentence would be judged ungrammatical under this reading by the vast majority of English speakers. The structure in (3c), in contrast, is a simple transitive, which is one of the two most frequent valency patterns for donate, so the sentence would be judged grammatical by all English speakers. The same would obviously be true if the sentence was constructed rather than taken from a corpus.

But the semantic considerations that increase or decrease our willingness to judge an utterance as grammatical are frequently more subtle than the difference between the readings in (3b) and (3c).

Consider the example in (3d), which contains a clear example of donate with the supposedly ungrammatical ditransitive valency pattern. Since this is an authentic example, we cannot simply declare it ungrammatical; instead, we must look for properties that distinguish this example from more typical uses of donate and try to arrive an an explanation for such exceptional, but possible uses. In Stefanowitsch (2007a), looking at a number of such exceptional uses, I suggest that they may be made possible by the highly untypical sense in which the verb donate is used here. In (3d) and other ditransitive uses, donate refers to a direct transfer of something relatively valueless from one individual to another in a situation of personal contact. This is very different from the typical use, where a sum of money is transferred from an individual to an organization without personal contact. If this were an intuited example, I might judge it grammatical (at least marginally so) for similar reasons, while another researcher, unaware of my subtle reconceptualization, would judge it ungrammatical, leading to no insights whatsoever into the semantics of the verb donate or the valency patterns it occurs in.

⁴ Culicover is a speaker of American English, so if he were writing his book today, he might check the 450-Million-word Corpus of Contemporary American English (COCA), first released in 2008, instead of the BNC. If he did, he would find a dozen or more instances of the construction, depending which version he were to use – for example It doesn’t matter the number of zeros they attach to it, from a 1997 transcript of ABC Nightline –, so he would not have to rely on his incomplete native-speaker intuition.