You are not answering his question, which is about the situation where the null hypothesis is true (what Taleb would call a "true" p-value of 0.5; for some reason he decided to define as the "true" p-value the average of the distribution - but it's worth noticing that there is not such a thing as a "true" p-value).
The only thing that matters to answer his question is the sampling distribution of the p-value when the null hypothesis is true, which is uniform in [0 1].
> Could someone clarify what is meant here?
If the null hypothesis is true every p-value is equally likely, by construction. However, getting a extreme value (under a small threshold) is less likely that getting not-so-extreme value (under a not-so-small threshold). The probability of getting a p-value below 0.05 is 5%, the probability of getting a p-value below 0.25 is 25%, the probability of getting a p-value below 1 is 100%.
If you show me a drug that cured 2 out of 10 ebola patients, when it's known that 10% of patients recover without treatment, I won't be impressed (high p-value). If you show me a drug that cured 9 out of 10 patients you're onto something (low p-value).
> At what point would this author say something is not consistent with the null hypothesis?
The author just doesn't like hypothesis testing. His view is that the null hypothesis is always false and everything is consistent with it.
> The probability of getting a p-value below 0.05 is 5%, the probability of getting a p-value below 0.25 is 25%, the probability of getting a p-value below 1 is 100%.
But getting a p value between 0.82 and 0.87 is also just 5% probability, so could also be seen as a probably-won't-happen-by-chance event. Sure it's clear most of the time which 5% is the significant one,but not always, for example paradoxically the same result can be significant or not, depending on what your intended stopping criterion was, even if you happened to have to stop at the same time using either criterion. This is because the two variants would lead to declaring a different 5% portion of the null's possible outcomes as the significant part.
The only thing that matters to answer his question is the sampling distribution of the p-value when the null hypothesis is true, which is uniform in [0 1].
> Could someone clarify what is meant here?
If the null hypothesis is true every p-value is equally likely, by construction. However, getting a extreme value (under a small threshold) is less likely that getting not-so-extreme value (under a not-so-small threshold). The probability of getting a p-value below 0.05 is 5%, the probability of getting a p-value below 0.25 is 25%, the probability of getting a p-value below 1 is 100%.
If you show me a drug that cured 2 out of 10 ebola patients, when it's known that 10% of patients recover without treatment, I won't be impressed (high p-value). If you show me a drug that cured 9 out of 10 patients you're onto something (low p-value).
> At what point would this author say something is not consistent with the null hypothesis?
The author just doesn't like hypothesis testing. His view is that the null hypothesis is always false and everything is consistent with it.