Question for the stats folks: consider someone who ABX tests and gets 5 of 7. This is 77.3% confidence (22.7% chance to do that well by guessing). He does this in 3 separate/independent tests, on different days. We could aggregate this as 3*5 = 15 of 3*7 = 21 and compute 15 of 21 is 96.1% confidence (3.9% chance to do that well by guessing).
But we could tackle this differently: There are 3 different tests, each independent and having 23% chance to pass by guessing. If you pass all three, the probability you're guessing should be .23 * .23 * .23 = 1.2%. This would be 98.8% confident.
These numbers are different so they can't both be right. Which is correct?
In my non-expert opinion,
@Blumlein 88 was going in the right direction in thinking of it in terms of distributions.
96.1% is the only correct answer, and it's to do with the fact that this is a binomial distribution. Once each trial has been run, the number of ways of distributing correct responses changes. Since the p-value is dependent on the distribution, it keeps changing after each trial. You therefore can't take the p-value (or its inverse) after x trials and multiply it by the p-value after x+y trials.
I'm not a statistician so I can't explain it more elegantly (or formally) than that, but the clearest way I can think of to say it is that, although each trial is independent, the probability that the subject was guessing after a given number of trials is not.
At some level, this also makes intuitive sense. If Subject A does ten tests in a row getting 7/7 in each test, the chances they are guessing when they get 7/7 in their eleventh test are surely lower than for Subject B who got an average of 3.5/7 in their first ten tests and then 7/7 in their eleventh test.
Anyway, I have a statistician relative I can talk to, so I'm gonna ask him and try to come back with a more rigorous answer...