What does p-value really mean ?

[Asuka]

I know that as long as p-value < alpha (significance level) and each sample has its p-value, Ho can be rejected. But what does p-value really mean ? Moreover, why as long as p-value < alpha (significance level), Ho can be rejected ?

 

[busprof]

The p-value is the probability of observing the given level of the test statistic or greater by chance given that the null is true.
Remember - the test statistic is a random variable. So, using a standard normal distribution as an example, even if the population mean is truly zero, you’ll occasionally see an observation that’s greater 1.98 just by random chance. But how often? About 2.39% of the time. Likewise for less than -1.98, since the distribution is symmetric.
So, if you saw a test statistic of 1.98 and you had a two-tailed test against a null of 0, you’d have a p-value of 4.78%. In words, this means that it’s possible that you’d see this by chance. But you’d only see it by chance 4.78% of the time. If you had an alpha of (for example) 10% you’d reject the null since you’re “further away” than the shosen level.
Looked at another way, assume that alpha of 10%. This is equivalent to asking “how far away from the null does the test stat have to be so that I’d only see this stat or larger 10% of the time?” - The answer is 1.67 - since you’re further away than that (1.98>1.67), the p-value is smaller than 10%, and you can reject the null at the 10% level. In other words, if you’re far enough away from the null that you’d only see that value of the test stat 4.78% of the time, you’re also far enough away that you’d see it less than 10% of the time.
Uisng a similar logic, you can also reject the nuyll at a 5% level of significance. But you can’t reject at the 1% level, since being “far enough way” to have a p-vale of 4.78% is not far enough away to hit the 1% level.

 

[S2000magician]

A p-value is just an alpha (α): a level of significance.
The p-value is simply the level of significance that puts your test statistic exactly at the cut-off point between rejecting the null hypothesis and failing to reject the null hypothesis.
So, you compare the level of significance that puts your observation on the brink (its p-value) to the level of significance you’ve chosen for your test (your α): if p > α, you fail to reject the null hypothesis; if p < α, you reject the null hypothesis.

 

[Asuka]

busprof wrote:
So, if you saw a test statistic of 1.98 and you had a two-tailed test against a null of 0, you’d have a p-value of 4.78%. In words, this means that it’s possible that you’d see this by chance. But you’d only see it by chance 4.78% of the time. If you had an alpha of (for example) 10% you’d reject the null since you’re “further away” than the shosen level.
Looked at another way, assume that alpha of 10%. This is equivalent to asking “how far away from the null does the test stat have to be so that I’d only see this stat or larger 10% of the time?” - The answer is 1.67 - since you’re further away than that (1.98>1.67), the p-value is smaller than 10%, and you can reject the null at the 10% level. In other words, if you’re far enough away from the null that you’d only see that value of the test stat 4.78% of the time, you’re also far enough away that you’d see it less than 10% of the time.

As far as know, apha = 10% means that the maximun chance or probability of making a Type I error is allowed.
Using your example, does it mean that while the chance or probability of making Type I error is only 4.78% (from the sample’s test statitic), the evidence of having maked a Type I error is actually found and, therefore, Ho shall be rejected ?
Conversely, if alpha = 1%, the evidence of having maked a Type error is actually not found and, therefore, Ho shall not be rejected ?
Am I correct ? Please correct me if I am wrong ! Thanks !

 

[S2000magician]

Asuka wrote:

busprof wrote:So, if you saw a test statistic of 1.98 and you had a two-tailed test against a null of 0, you’d have a p-value of 4.78%. In words, this means that it’s possible that you’d see this by chance. But you’d only see it by chance 4.78% of the time. If you had an alpha of (for example) 10% you’d reject the null since you’re “further away” than the shosen level.
Looked at another way, assume that alpha of 10%. This is equivalent to asking “how far away from the null does the test stat have to be so that I’d only see this stat or larger 10% of the time?” - The answer is 1.67 - since you’re further away than that (1.98>1.67), the p-value is smaller than 10%, and you can reject the null at the 10% level. In other words, if you’re far enough away from the null that you’d only see that value of the test stat 4.78% of the time, you’re also far enough away that you’d see it less than 10% of the time.

As far as know, apha = 10% means that the maximun chance or probability of making a Type I error is allowed.
Using your example, does it mean that while the chance or probability of making Type I error is only 4.78% (from the sample’s test statitic), the evidence of having maked a Type I error is actually found and, therefore, Ho shall be rejected ?
Conversely, if alpha = 1%, the evidence of having maked a Type error is actually not found and, therefore, Ho shall not be rejected ?
Am I correct ? Please correct me if I am wrong ! Thanks !

Alpha is the probability of making a Type I error; if you choose α = 10%, then you are accepting that you will make a Type I error 10% of the time (i.e., there is a 10% chance of making a Type I error).
I believe that it’s counterproductive to interpret the p-value as a probability of making a Type I error; you’ve already established the probability with your selection of α. It’s better to think of it as the level of significance that puts your statistic on the border between rejection and failure to reject, and compare the p-value to your chosen α to determine whether you reject the null or fail to reject the null based on the level of significance you’ve chosen.