I am doing some summer reading and just came across a nice literary example of one of the key methodological approaches in science: hypothesis testing.

What do we do when we perform a hypothesis test? We form a theory and call it our null hypothesis.  We then look at data and ask ourselves how probable it is that we would see this data (or something like it) if the null hypothesis were true. This probability is called the p-value. If this probability is very low, we then abandon our null hypothesis in favor of its opposite.

Detective stories are generally a good potential source for examples of this approach, as detectives constantly entertain theories or hypotheses that have to be revised or rejected as new evidence is found. The present example is special in that the author really gives us all the steps of such a test in a specific setting, including the calculation of the p-value, that is the probability of seeing such data as was observed under the assumption that the null hypothesis is true.

The book I have just read is A A Milne’s (author of the Winnie the Pooh stories and poems) “The Red House Mystery” written in 1922. Before I give you my annotated quote from this book, I probably need to roughly explain the situation (without giving too much away in case you would like to read the book and solve the murder mystery yourself).  There are two brothers, Mark and Robert. Mark lives in England somewhere and Robert has just returned to Mark’s house (the red house) after being “sent out of the country in disgrace” and spending the last 15 years in faraway Australia. Robert, almost immediately after his arrival at the red house is found murdered there. His brother Mark, who was overheard having an argument with Robert just before the murder, is missing since then.  The amateur detectives trying to solve the case are Antony, assuming a Sherlock Holmes role, and Bill as Antony’s John Watson. The issue they are debating is a letter that Mark received from Robert. Specifically they are trying to establish when Mark received Robert’s letter. Mark had claimed to have received it on Tuesday. This is our null hypothesis.

Now to the book. We begin with Antony saying:

“Yes. Mark hoped to marry Miss Norbury. Now, if Robert really was a blot upon the family honour, Mark would want to do one of two things. Either keep it from the Norburys altogether, or else, if it had to come out, tell them himself before the news came to them indirectly. Well, he told them. But the funny thing is that he told them the day before Robert’s letter came. Robert came, and was killed, the day before yesterday—Tuesday. Mark told Mrs. Norbury about him on Monday. What do you make of that?”

Here we have the data. Mark told Mrs. Norbury about Robert on Monday.

“Coincidence,” said Bill, after careful thought. “He’d always meant to tell her; his suit was prospering, and just before it was finally settled, he told her. That happened to be Monday. On Tuesday he got Robert’s letter, and felt jolly glad that he’d told her in time.”

This is Bill’s calculation of the p-value. He says it is a coincidence, which is a reasonably low probability event, but one that could happen from time to time. The p-value does not seem low enough for Bill to give up on the null hypothesis.

“Well, it might be that, but it’s rather a curious coincidence. And here is something which makes it very curious indeed. It only occurred to me in the bath this morning. Inspiring place, a bathroom. Well, it’s this—he told her on Monday morning, on his way to Middleston in the car.”

We are now looking at the data more carefully.

“Well?”

“Well.”

“Sorry, Tony; I’m dense this morning.”

“In the car, Bill. And how near can the car get to Jallands?”

“About six hundred yards.”

“Yes. And on his way to Middleston, on some business or other, Mark stops the car, walks six hundred yards down the hill to Jallands, says, ‘Oh, by the way, Mrs. Norbury, I don’t think I ever told you that I have a shady brother called Robert,’ walks six hundred yards up the hill again, gets into the car, and goes off to Middleston. Is that likely?”

And here we have the key data: Mark went out of his way to tell Miss Norbury about his brother on Monday. And then we have Antony reconsidering the p-value calculation, that is the calculation of the likelihood of Mark’s behavior under the assumption that on Monday Mark had not yet received Robert’s letter.

Bill frowned heavily.

“Yes, but I don’t see what you’re getting at. Likely or not likely, we know he did do it.”

We now have Bill asking about the implication of a low p-value.

“Of course he did. All I mean is that he must have had some strong reason for telling Mrs. Norbury at once. And the reason I suggest is that he knew on that morning—Monday morning, not Tuesday—that Robert was coming to see him, and had to be in first with the news.

Antony explains the alternative hypothesis, which must be true if the null is not true: that Mark had received the letter already on Monday (or before). He also explains that the data is more likely (easier to explain) under this alternative hypothesis.

“But—but—”

“And that would explain the other point—his instantaneous decision at breakfast to tell you all about his brother. It wasn’t instantaneous. He knew on Monday that Robert was coming, and decided then that you would all have to know.”

They are now looking at more data in the light of the two hypotheses.

“Then how do you explain the letter?”

“Well, let’s have a look at it.”

Antony took the letter from his pocket and spread it out on the grass between them.

“Mark, your loving brother is coming to see you to-morrow, all the way from Australia. I give you warning, so that you will be able to conceal your surprise but not I hope your pleasure. Expect him at three or thereabouts.”

“No date mentioned, you see,” said Antony. “Just to-morrow.”

“But he got this on Tuesday.”

“Did he?”

“Well, he read it out to us on Tuesday.”

“Oh, yes! he read it out to you.”

Bill read the letter again, and then turned it over and looked at the back of it. The back of it had nothing to say to him.

“What about the postmark?” he asked.

“We haven’t got the envelope, unfortunately.”

“And you think that he got this letter on Monday.”

“I’m inclined to think so, Bill. Anyhow, I think—I feel almost certain—that he knew on Monday that his brother was coming.”

Antony cannot find any evidence against the alternative hypothesis and thus finally comes to reject the null hypothesis in favor of the alternative. He is only “almost certain” which indicates that his subjectively calculated p-value is close to but not equal to zero.