bookmark_borderRepost: Extraordinary Claims Require Extraordinary Evidence (ECREE), Part 2: Is ECREE False? A Reply to William Lane Craig

(This article was originally published on this blog on June 21, 2012. I am reposting because William Lane Craig recently tweeted a link to a video in which he objects to ECREE.)
In my last post, I offered a Bayesian interpretation of the principle, “extraordinary claims require extraordinary evidence” (ECREE). William Lane Craig, however, disagrees with ECREE. In a response to philosopher Stephen Law, Craig wrote this.

This sounds so commonsensical, doesn’t it? But in fact it is demonstrably false. Probability theorists studying what sort of evidence it would take to establish a highly improbable event came to realize that if you just weigh the improbability of the event against the reliability of the testimony, we’d have to be sceptical of many commonly accepted claims. Rather what’s crucial is the probability that we should have the evidence we do if the extraordinary event had not occurred.3 This can easily offset any improbability of the event itself. In the case of the resurrection of Jesus, for example, this means that we must also ask, “What is the probability of the facts of the empty tomb, the post-mortem appearances, and the origin of the disciples’ belief in Jesus’ resurrection, if the resurrection had not occurred?” It is highly, highly, highly, improbable that we should have that evidence if the resurrection had not occurred.
————
[3] See the very nice account by S. L. Zabell, “The Probabilistic Analysis of Testimony,” Journal of Statistical Planning and Inference 20 (1988): 327-54.

I agree with Craig that it would be incorrect to “just weigh the improbability of the event against the reliability of the testimony.” I also agree with Craig that “the probability that we should have the evidence we do if the extraordinary event had not occurred … can easily offset any improbability of the event itself.” I disagree with Craig, however, regarding his interpretation that ECREE requires that we ignore that probability. This can be seen using Bayes’s Theorem (BT).
Let B represent our background information; E represent our evidence to be explained; H be an explanatory hypothesis, and ~H be the falsity of H. Here is one form of BT:
BT
As I argued in my last post, an “extraordinary claim” is an explanatory hypothesis which is extremely improbable, conditional upon background information alone, i.e., Pr(H | B) <<<  0.5. And “extraordinary evidence” can be interpreted as the requirement that a hypothesis’s explanatory power is proportionally high enough to offset its prior improbability (the “extraordinary claim”). Here I offer an even more precise definition.
It follows from BT that H will have a high epistemic probability on the evidence B and E:
BT2just in case it has a greater overall balance of prior probability and explanatory power than its denial:
BT3Thus, we can somewhat abstractly define “extraordinary evidence” as evidence that makes the following inequality true:
BT4With that inequality in mind, let’s return to Craig’s objection to ECREE. Here again is the relevant portion of his objection:

Probability theorists studying what sort of evidence it would take to establish a highly improbable event came to realize that if you just weigh the improbability of the event against the reliability of the testimony, we’d have to be sceptical of many commonly accepted claims. Rather what’s crucial is the probability that we should have the evidence we do if the extraordinary event had not occurred.

It seems, then, that Craig’s objection to ECREE is based upon an interpretation of ECREE which requires that we only consider the “extraordinary claim,” i.e., Pr(H | B). If that interpretation is correct, then I will join Craig in rejecting ECREE. But is it correct?
In mathematical notation, “the probability that we should have the evidence we do if the extraordinary event had not occurred” is Pr(E | B & ~H). But now consider again the inequality used to define extraordinary evidence.
BT4
The expression, Pr(E | B & ~H), is literally right there, in the numerator on the right-hand side. It appears, then, that Craig’s objection is based upon a misinterpretation of ECREE. For the same reason, Craig’s reason that ECREE would cause us “to be sceptical of many commonly accepted claims” is therefore misplaced.
I could be wrong, but I suspect there are two factors which contributed to this misinterpretation. First, many skeptics have used ECREE in connection with (or as support for) Hume’s argument against miracles. While I’m inclined to agree with John Earman that Hume’s argument is highly overrated–i.e., it may be the case that BT does not provide Hume with the support many skeptics think it provides–this is not of obvious relevance to ECREE. ECREE, like BT, is not dependent on Hume.
The other factor which may have contributed to the misinterpretation is the definition of “extraordinary claim;” Craig may disagree with the criteria skeptics have used to determine whether a claim is extraordinary. I think it is helpful to use probabilistic notation to clarify the issue. Again, I proposed that an “extraordinary claim” is an explanatory hypothesis which is extremely improbable, conditional upon background information alone, i.e., Pr(H | B) <<<  0.5. Let’s assume, for the sake of argument, that definition is wrong. Instead, define an “extraordinary claim” as any explanatory hypothesis H which has a prior probability below some number x, i..e., Pr(H | B) < x, where x can be any real number between 0 and 1. Here’s the point. X can be any real number between 0 and 1. It doesn’t matter which value one chooses, since BT can accommodate all probability values. In terms of calculating the final probability of H, Pr(H | E & B), we use the same formula–BT–regardless of whether H is an extraordinary claim. From a mathematical perspective, it makes no difference whatsoever whether we label a claim “extraordinary” or “ordinary.” We can use BT to assess the epistemic probabilities of both types of claims.

bookmark_borderHow to Think about Historical Evidence about Anything, Part 1: The Credibility of Testimony

Note: So far as I know, no one working in New Testament scholarship, apologetics, counter-apologetics, or ancient history is applying the concepts in this blog post. As will soon become obvious, most of the ideas in this blog post are not mine, but if other people find these techniques useful, I would appreciate being given credit for the idea to apply them outside of their source discipline.

1. Schumian Framework for Decomposition of the Credibility of Testimony

Suppose witness W testifies E* that event E occurred. We wish to determine whether E actually happened. Clearly, E* and E are not identical; we could have E* when E did not happen. E* would entail E only if W were perfectly credible; otherwise, E* is evidence of E to the degree that we consider W credible.[1] Following evidence scholar David Schum,[2] I shall analyze the credibility of testimony in terms of three major attributes: veracity, objectivity, and observational sensitivity.  These attributes can be assessed by asking three questions respectively:
(a) Does W believe what he testified? This question focuses on W’s veracity or truthfulness.  If it is doubtful that W even believed what he testified, then W is not truthful and E* does not probabilistically favor E.  On the other hand, even if W believes what he testified, it does not follow that E is true. W could be entirely honest in reporting E, but E might not have happened. This would be the case if W were unobjective or inaccurate.[3]
(b) Did W’s senses give evidence of what he believed? If not, we would say that W is not objective. This would be the case if, for example, W so strongly wished E to occur that he believed E regardless of what his senses told him.[4]
(c) Was the sensory evidence accurate? Even if W believed what he testified and W’s senses gave evidence of what he believes, it could still be the case that what W believes is false. In other words, sensory evidence is not conclusive unless we believe that W is perfectly observationally sensitive. Both W’s physical condition and the general circumstances of observation at the time of observation might have caused the sensory evidence to be inaccurate.[5]
Since there are three credibility attributes and each attribute has two possible values, it follows that there are 23=8 combinations of potential values.
1: W is honest, objective, and observationally accurate.
2: W is honest, objective, and not observationally accurate.
3: W is honest, not objective, and observationally accurate.
4: W is honest, not objective, and not observationally accurate.
5: W is not honest, objective, and observationally accurate.
6: W is not honest, objective, and not observationally accurate.
7: W is not honest, not objective, and observationally accurate.
8: W is not honest, not objective, and not observationally accurate.[6]
The decomposition of the credibility attributes of W’s testimony is illustrated in Figure 1.[7] From W’s testimony (E*) we first draw an inference about his veracity. If we conclude that W did believe what he testified (Eb), we next consider his objectivity. Again, if we conclude that W is objective (Es), we then assess his observational sensitivity. If we then conclude that his sensory evidence was accurate, we then infer that E did occur.  Each inference based upon a credibility attribute is an example of an inferential link or stage of reasoning in a chain of reasoning.
schumian-figure1

Figure 1 Stages of reasoning involving attributes of W’s credibility

As Schum correctly observes, each inferential link in a chain of reasoning must ultimately be justified on the basis of an inductive argument.[8] Typically, such justification appeals to a statistical generalization. For example, the inference from E* to Eb might appeal to the following statistical generalization concerning W’s veracity: “If an ancient author writes that an event occurred, then at the time the document was written the author probably believed that E occurred.” In support of W’s objectivity, one might argue: “If at the time of his alleged observation a person believed that an event occurred, then this person’s senses often give evidence that the event occurred.” In support of W’s observational sensitivity, one could claim: “If a person’s senses give evidence that an event occurred, then usually the event occurred.”[9]
Clearly, not just any statistical generalization can justify an inferential link in a chain of reasoning regarding the credibility a witness’s testimony. Those who have studied the formal structures of inductive arguments know that statistical generalizations are just one premise in the type of inductive argument known as the statistical syllogism.

Z percent of F are G.

x is F.

[Z% probable] Therefore, x is G.

But the Rule of Total Evidence requires that we must take into account the total relevant and available evidence when selecting F (the reference class). This makes it difficult, however, to apply statistical generalizations to unique or singular events. And W has offered a specific account (E*) about a singular event.[10] Any reference class that satisfies the Rule of Total Evidence will be so specific that there will be no frequency data available to justify the generalization.
Drawing upon the work of philosopher L.J. Cohen,[11] Schum describes a method for testing statistical generalizations about singular events that is roughly based upon eliminative induction.[12] In his words:

What we can do is to put this generalization to a variety of different relevant evidential tests, each one designed to invalidate this generalization as far as W and his present testimony are concerned. … The more of these tests that W passes, the more we are entitled to infer that this … generalization holds in the present instance of W and his testimony E*. … A generalization is supported to the extent that this generalization survives our best attempts to show that it is invalid in the particular instance of concern.[13]

The results obtained from this testing constitute ancillary evidence (i.e., “evidence about evidence”) regarding the strength or weakness of the statistical generalization’s relevance to the link in question. If ancillary evidence is not provided in support of a statistical generalization, then the generalization is unsupported and may or may not be applicable to the inferential link.[14] In other words, the generalization that would justify the inferential link was “never put to the test.”[15] Therefore, any probabilistic inference made upon the basis of an unsupported generalization is weak.
Schum deduces, “a chain of reasoning cannot be any stronger than its weakest link.”[16]  It is an immediate consequence of this Schumian framework for witness credibility that the overall degree of credibility for a given piece of testimony (E*) is only as great as the weakest credibility attribute, regardless of how strong the other attributes may be. Accordingly, in order for E* to be credible, there must not be reasonable doubts about W’s veracity, objectivity, or observational sensitivity.

2. Nth Hand Evidence

For any piece of testimony (E*), a witness (W) may have obtained his or her information in one of three ways. First, the witness may have made a direct observation of the event she reports. Second, the witness may have received this information from another source, which I shall call nth hand evidence. I use the term “nth hand” to accommodate situations in which there are multiple sources in the chain. Third, the witness may have inferred the information, based upon information about other events.[17]
Let a source be a person or device who/that allegedly testified to the occurrence or nonoccurrence of some event of interest. A source is a primary source if it allegedly recorded the occurrence or nonoccurrence of that event. An immediate source is the source who/that informed you about this event. If an immediate source is not also a primary source, then the immediate source is either secondhand or nth hand evidence. Evidence is secondhand if there are no intermediate sources between the primary source and the immediate source. If there are intermediate sources, then the immediate source is nth hand evidence.[18]
Figure 2 depicts the chain of reasoning involved in a relatively simple appeal to secondhand evidence. In this example, we have a report (E*2,1) from an immediate source (S2) that a primary source (S1) reported (E*1) the occurrence of event E, which probabilistically favors hypothesis H.[19]
schumian-figure2

Figure 2 Secondhand Evidence

Thus, in order to determine the force of evidence E*2,1 upon hypothesis H in Figure 2, we must assess the credibility of two sources—immediate source S2 and primary source S1. If these two sources are people, we must consider the veracity, objectivity, and observational sensitivity of each witness. Figured 3 illustrates the stages of reasoning, when the credibility of each witness’s testimony is decomposed into those three credibility attributes.

{E, ¬E} S1’s Observational Sensitivity
{Es,1, ¬Es,1} S1’s Objectivity
{Eb,1, ¬Eb,1} S1’s Veracity
{E*1, ¬E*1} S2’s Observational Sensitivity
{Es,2, ¬Es,2} S2’s Objectivity
{Eb,2, ¬Eb,2} S2’s Veracity
E*2,1

Figure 3 Decomposed Credibility Attributes of Secondhand Evidence

It is an immediate consequence of this decomposition and the results of section 1 that the overall degree of credibility for a given piece of secondhand testimony (E*2,1) is only as great as the weakest credibility attribute of either witness (S1 or S2).
Moreover, due to the (alleged) dependence of our immediate source’s (S2’s)  testimony upon the primary source’s (S1’s) testimony, there is an interesting feature of the negations in the partitions for each of the credibility-related attributes. For example, the negation ¬Eb,2 means “Source S2 does not believe that event E happened (as allegedly reported by S1).”[20] Due to S2’s purported dependence upon S1, however, ¬Eb,2 can have two interpretations. First, ¬Eb,2 could mean that S2 believes E did not occur. Second, it could also mean that S2 has no belief about event E, which could be the case if S2 invented a story about S1 telling him that event E occurred.  In order to account for this possibility, then, ¬Es,2 includes the possibility that S2’s senses gave no evidence about what S1 said, ¬E*1 includes the possibility that S1 said nothing to S2; ¬Eb,1 includes the possibility that S1 has no belief about event E; and ¬Es,1 could mean that S1 made no sensory observations of event E. As Schum writes, “If we have to rely entirely upon S2, we have to consider the possibility that S1 said nothing at all about event E to S2 and perhaps never even made a relevant observation.”[21]
We are now in a position to fully appreciate the full significance of the fact that the overall degree of credibility for a given piece of secondhand testimony (E*2,1) is only as great as the weakest credibility attribute of either witness (S1 or S2). The strength of secondhand evidence, unlike that of firsthand evidence, may depend upon a witness whose credibility is unknown. (Indeed, in cases of ancient secondhand evidence, it is not uncommon for hearsay testimony to involve a putative primary witness whose very existence is otherwise unknown!) As Schum writes, “Absent evidence about the veracity, objectivity, and observational sensitivity of all sources in a chain of hearsay, we could hardly form any settled judgment of the inferential force of this species of evidence.”[22]  Any inference based upon such hearsay could not be regarded as a strong inductive argument, viz., such hearsay would not make E more probable than not. In plain English, such hearsay is worthless as evidence for E.
This same observation also applies to information for which we cannot identify a primary or intermediate source, which Schum labels rumor or gossip.[23] If information has come to us through a chain of sources and we cannot identify the primary source, then we do not know where or how they obtained this information.[24] In other words, we have information with an unknown primary source and therefore unknown credibility. Therefore, any inductive or probabilistic inferences based upon rumor or gossip must be regarded as weak.

3. Nth Hand Evidence and Observed vs. Inferred Sources

Although Wigmorean methods are appropriate for analyzing historical evidence, most discussions of these methods focus on how to apply them in legal contexts.  Thus, for example, when describing secondhand evidence, Schum provides the example of testimony (E*2,1) from an immediate source (S2) that a primary source (S1) reported (E*1) the occurrence of event E, which probabilistically favors hypothesis H.[25] While such an example is surely representative of hearsay testimony in modern courtrooms, it is not representative of nth hand evidence in the writings of ancient historians. As contemporary historian Michael Grant writes:

We nowadays like our historiography to be supported by documents. This did not function in the ancient world, for two reasons. First, the documents and archives, whether public or private, were hopelessly inadequate and without meaning, even if relatively numerous (and in some cases of early date). Second, the Greek and Roman historians did not care very much about these documents and rarely quoted or even paraphrased them.[26]

Nevertheless, there are other sources of evidence for the existence of a primary source besides an explicit reference from the immediate source; there are ways in which we may sometimes confidently detect the existence of a prior source, even if our immediate source fails to mention it. For example, it may be possible to infer the existence of a prior source based upon such factors as the proximity of the immediate source to the event(s) described or textual clues (grammar, vocabulary, etc.).
Since I have been unable to locate any discussion of an inferred prior source in the writings of the new evidence scholars, I shall attempt to advance the discussion by formulating the concept and its implications for the credibility of evidence.
Let us distinguish implicit and explicit forms of nth hand evidence. An immediate source is explicit nth hand evidence if it makes explicit reference to a prior source. For example, suppose we have a report (E*2,1) from an immediate source (S2) that a primary source (S1) reported (E*1) the occurrence of event E. S1 is an observed source because it was explicitly mentioned by an immediate source (S2).
Not all primary sources are explicitly mentioned by the immediate sources that use them, however. As we’ve seen, ancient writers often failed to satisfy modern expectations concerning the identification of the sources of their information. Let us define, therefore, implicit nth hand evidence as an immediate source that is based upon, but does not explicitly mention, a prior source. Thus, we may have a report (E*2,1) from an immediate source (S2) that is based upon, but does not explicitly mention, a primary source (S1) that reported (E*1), the occurrence of event E. S1 is an inferred source because it was not explicitly referenced by an (extant) immediate source.
By the very nature of the case, the credibility of an inferred source may often be unknown. This may be because the information is gossip (and hence the identity of the primary source is unknown) or, even if the identity of the source is inferred, nothing else is known about the identity of the source and hence the credibility of the inferred source is also unknown. If the credibility of the inferred source is unknown, then any inductive inferences based upon the inferred source must be regarded as weak.
Notes
[1] Joseph B. Kadane and David A. Schum, A Probabilistic Analysis of the Sacco and Vanzetti Evidence, (New York: Wiley, 1996), 46, 53.
[2] David A. Schum, The Evidential Foundations of Probabilistic Reasoning (New York: Wiley, 1992), 105.
[3] Schum 1992, 102.
[4] Schum 1992, 102.
[5] Schum 1992, 103-104.
[6] Schum 1992, 229.
[7] Cf. Kadane and Schum 1996, 56.
[8] Schum states that inferential links must always be justified on the basis of a statistical syllogism (in his words, “inductive generalizations”). While Schum is undoubtedly correct that links in a chain of reasoning are in practice usually justified by generalizations, especially in law, I see no reason to believe that inferential links must always be justified in this way. It seems at least possible that an inferential link in a chain of reasoning could also be justified by an explanatory argument.  See Kadane and Schum 1996, pp. 45-46, 51.
[9] Cf. Kadane and Schum 1996, 51.
[10] Schum 1992, 251.
[11] Especially his The Probable and the Provable (Clarendon: Oxford, 1977).
[12] Schum 1992, 243-251.
[13] Schum 1992, 249-251.
[14] Kadane and Schum 1996, 87, 152.
[15] Kadane and Schum 1996, 152.
[16] Schum 1992, 302.
[17] Schum 1992, 94-95.
[18] Schum 1992, 344.
[19] Schum 1992, 346-347.
[20] Schum 1992, 348.
[21] Schum 1992, 349.
[22] Schum 1992, 350.
[23] Schum 1996, 113.
[24] Cf. Kadane and Schum 1996, 113.
[25] Schum 1992, 346-347.
[26] Michael Grant, Greek and Roman Historians: Information and Misinformation (New York: Routledge, 1995), 34.

bookmark_borderThe Argument from Silence, Part 9: Mormonism’s Missing Golden Plates

I began this series with a Bayesian interpretation of arguments from silence and then proceeded to use that interpretation to evaluate various arguments from silence about Jesus and God. In this post, I want to assess an argument from silence against a central claim of Mormonism, namely, that the Book of Mormon is the English translation of golden plates which church founder Joseph Smith received from the angel Moroni.
If we abbreviate “the golden plates are nowhere to be found” as S, let B represent our background knowledge, and let M represent the claim, “The Book of Mormon is the English translation of golden plates which church founder Joseph Smith received from the angel Moroni,” then the missing links argument can be summarized using the argument schema I outlined in Part 1, as follows.

(1′) S is known to be true, i.e., Pr(S) is close to 1.
(2′) Relative to background knowledge B, the intrinsic  probability of M is not very high, i.e., Pr(M | B) is not very much greater than 1/2.
(3′) ~M gives us more reason to expect S than M, i.e., Pr(S | ~M) > Pr(S | M).
(4′) Other evidence held equal, M is probably false, i.e., Pr(M | B & S) < 1/2.

In plain English, this becomes:

(1”) The golden plates are nowhere to be found.
(2”) Prior to examining the evidence, the intrinsic probability of M is not very much greater than 1/2.
(3”) We have much more reason to expect that the golden plates are nowhere to be found on the assumption that Mis false than on the assumption that M is true.
(4”) Therefore, other evidence held equal, M is probably false.

The typical Mormon explanation for the absence of the golden plates is that Smith returned the golden plates to Moroni once he finished translating them. How convenient!
Let’s assume, but only for the sake of argument, that Moroni really did reveal the plates to Smith, just as Mormons claim. While it is no doubt possible that Moroni took the golden plates back once Smith was done with them, it’s also possible (and no less antecedently likely) that Moroni never asked for them back. In fact, even on the assumption that this special revelation took place, we would still have good reason to expect that the plates would still be available for inspection today. Why? Well, considering the radical differences between orthodox Christianity and Mormonism, not to mention the numerous moral failings of Joseph Smith which undermine his credibility as a witness, we would expect that both God and Moroni would desire that better, objective evidence of Smith’s claims be retained.
For anyone who isn’t already a Mormon believer, it’s hard to avoid the conclusion that “Smith returned the golden plates to Moroni” is just an ad hoc, “just so” story designed to explain away the clear evidence that Smith was a fraud.