10.2: Tests of Evidence

Will Lion – proper browsing – CC BY-NC-ND 2.0

One of the best “checklists” a communicator can integrate into their reflexive evaluation practice is the “tests of evidence.” (Jensen) Evaluation of the “evidence” is part of the deliberation process in many professional areas. Courts of law and legal professionals (law enforcement officers, lawyers, judges) are concerned about the admissibility of evidence, the appropriateness of evidence to the making of a case, and other judicial standards. Rhetoricians and scholars of debate concentrate on the types of data that communicators can consider as evidence, sources of evidence, and the classification of evidence for the purposes of effective persuasion or debate.

For mass communicators, the use of standards in considering the information / evidence you discover during your information seeking tasks takes on a slightly different hue. Evidence can be in the form of written materials (articles, books, letters, court records, websites), oral communications (interview testimony, focus group sessions), or personal observation, knowledge or experience.

You can apply traditional tests of evidence to all the information that you gather through the information strategy process. When skillfully applied, these tests of evidence can alert you to potential problems or inconsistencies in information, to the gaps or “holes” in your information search, and to areas that require further investigation.

The tests of evidence – also known as standards of evaluation – as they apply to mass communication information gathering are:

Clarity

During an interview with a person who witnessed a holdup, you discover that the witness can be clear on just two points about the robbery and the perpetrator. You would have to evaluate the testimony of the witness based upon their clarity and you may decide that the evidence is not trustworthy and can’t be used. Similarly, as an advertising account executive you might decide that the advertising campaign goals of the client are unclear, requiring further information gathering before you can effectively plan the campaign.

Verifiability

Miracle Cure – public domain

A famous author may write an autobiography that includes a number of impressive claims about his or her education, achievements, and financial success. You may be unable to verify that the author actually attended the schools because enrollment and alumni records may not include the author’s name. You would have to do further research to determine the truth of the author’s claims. On the strategic communications side, an advertising client may wish to make claims about the superiority of the company’s product but may not be able to provide the substantiating documents to verify the claims. If this is the case, then you can’t create an ad which makes that kind of claim.

Accuracy

You are a reporter attending an event as an observer and count about 75 people in the audience. Two hours after you get back to the newsroom room, the event sponsor sends out a news release claiming that about 250 people were there. You would reject that news release on the basis of accuracy.

Recency

A public relations firm working on the problem of airport noise for an airport commission needs the most recent figures on the number of takeoffs and landings since expansion at the airport. Information about how recent flight activity compares with pre-expansion activity is also useful. Or, an ad campaign for a new brand of pizza might require the most recent information about consumer preferences and media habits of pizza eaters. You would reject a marketing study that was two years old on the basis of the recency test.

Relevance

An advertiser evaluating market research about consumer characteristics must decide whether the research is relevant to the product or service being advertised. A particular market research study about people who buy chicken may or may not be something that is relevant to an advertiser for a turkey company. Or, a reporter writing about the nation’s fiscal policy may reject the notion of interviewing an expert on monetary policy based on the relevance test.

Reputation

People, institutions, and print and digital records have reputations. A reporter might try to arrange an interview with a government official who has a reputation for being candid and forthright on the issue of tax reform. The same reporter might avoid another official because of his or her reputation for stonewalling the press. Or, a public relations practitioner may rely on a scholarly study of the effects of factory emissions on air quality rather than on a study done internally by the factory’s own staff because the PR professional knows that the public will hold the independent study in higher regard.

Sufficiency

You’re in charge of the ad campaign for a new chain of teen athletic clothing stores. You get a marketing research report from a group that monitors teen spending. However, the category for teen spending in the area of clothing does not break down the specific clothing types—the report is not “sufficient” or complete in its analysis of teen spending for athletic clothing. This report, then, would be rejected on the basis of the sufficiency test because the data available are not adequate for you to make decisions on the direction of the campaign.

Internal Consistency

You might decide not to use a particular document because, for example, the figures about average household income used on page 15 of the study are inconsistent with those on page 38, indicating a serious internal consistency problem. This should make you question the accuracy of all the data if there is so little attention to detail. Or, the testimony of an interviewee varies widely over the course of a three-hour interview, suggesting that the interviewee is being inconsistent in his story.

External Consistency

Two reports on the implications of a change in property taxes come to different conclusions about the projected tax on a $300,000 house, alerting you that there is an external consistency problem with the evidence. Further information is therefore needed to clarify the inconsistency. Or, an advertising firm conducts its own research about the market for a new brand of jeans, and the conclusions differ from those of an industry-wide research project about the same product, suggesting an external consistency problem. Using reports of breaking news from multiple sources (eyewitness Twitter posts, Associated Press wire service filings, etc.) can be a challenge in evaluating the consistency of information as an event is unfolding.

Comparative Quality

Decisions about what constitutes “good” information or the “best” information are necessary at some point. Through a comparison between two studies of new parents, an ad researcher preparing an ad campaign for a new line of strollers might decide that the study that specifically addresses new parents’ leisure activities is the most useful compared to a report on the impact of parenthood on their careers. Or, a reporter might decide that among all the interviews she conducted with “experts” on the subject of rising health care costs, two stand out as being the most informative and containing the freshest perspectives on the topic.

Contextuality

All information exists within a particular context. An interviewee’s comments at one point in an interview cannot be taken out of context of the rest of the interview. You also cannot take isolated figures from a report about the incidence of police loafing on the job and report them out of the context of the rest of the report. The complete context may indicate that the problem is not widespread. Both the context and the time period within which information exists must be recognized and respected. If the context or time period cannot be determined, the information is suspect.

Calculator – CC0

Statistical Validity

It is not necessary for you to become a statistician in order to test the validity of statistical claims. Simple questions about the method of gathering the statistics, the method of analyzing the data, and the standards for drawing the conclusions from the data are within your ability.

Ambiguity

Words and phrases have amazing flexibility, suggesting one meaning to some and another, very different, meaning to others. Think about the different connotations of the words “terrorist” and “freedom fighter,” the words “tax increase” and “revenue enhancement,” or the words “layoffs” and “reorganization.” The words carry entirely different meanings and these meanings change even more when the word or words are placed in the larger context of a sentence, a paragraph, or a fully-developed document, speech or publication. You need to be able to recognize and flag the key words or phrases that can convey the main point in any document, interview transcript, Web page or other source of information. Pay attention to the role these key words play in the overall argument or conclusion the information source wants to make.

There are many examples of how the tests of evidence are applied in mass media message-making:

• Fact-checking articles that appear in a magazine

• Evaluating information about audiences and consumers for advertising and marketing decisions

• Verifying information for news accounts

In the context of this course, knowing how to apply the tests of evidence as you find and evaluate information is one of the key skills that must be developed.

Tests of evidence also must be considered when thinking about the audience for your message. Is the material understandable, believable, useful to the point being made, from a familiar source? If not, you may need to collect more evidence than you’d normally need to gather and present if you are to overcome the audiences’ natural skepticism.

For example, if a fast food restaurant is going to make a claim in an advertising message that its food is healthful, you need to understand that the audience will demand a higher level of evidence than would normally be the case because it is counter to common sense that such food is healthy.