Let's get a simple scenario done. Person changes power cord and "hears" an improvement. Such a person is not going to be able to get a switchbox for this, and probably wouldn't accept one anyway. So let's cover this scenario for now.
Intro:
Our ears receives audio signals but it is our brain that interprets that. Unfortunately they brain uses more than sonic input to decide what it is perceiving. It routinely uses our experiences in the past and other sensory input to draw conclusions. This is why high-end restaurants serve food with pretty presentation and use the saying, "you eat with your eyes." It is important that when assessing audio differences, that we don't "hear with our eyes."
0. This FQA is for devices that don't have variable output such as cables, digital sources, various tweaks (e.g. cable lifters, outlets, etc.).
1. Since we are interested in just the sound waves that hit your ears, it is critical that any testing removes this knowledge. Note that not wanting an outcome does not fix this problem. You listen differently when testing from moment to moment and even if the sound has not changed, you can perceive a difference.
2. Run the test as you have been but this time have someone else switch the test item with another. You decide how long you want to listen. If you need a full day, then have the switching happen once a day. If you think the difference is very immediate and large, then have the switching happen quickly in sequence as you ask for it.
3. It is extremely easy to get lucky and guess correctly which items is which once, twice or maybe even five times in a row. Try this with a coin flip and you will see the reality of this. As such, the test needs to be run enough times so that we can rule out chance. So repeat the experiment at 10 times and aim to get 9 instances right. Anything less unfortunately means that the difference is not likely to be there even though your intuition says otherwise. Remember, if the difference is large, then you should be able to get 10 out of 10 right.
4. Have the person doing the switching keep a log and randomly switch or not switch the samples. You can use a random number generator like:
https://www.random.org/. Put in 0 for minimum and 1 for maximum. Hit generate and it will give you a random 0 or 1. Assign 0 to one sample and 1 to the other. Have the person doing the switching (proctor) do this 10 time and generate the sequence for the test. Once there, have the proctor write down your answers without any kind of communication/feedback to you.
5. It goes without saying that you have to hide the nature of change. Simple solutions like turning your back to the device, or using a cover may be effective.
6. Note that you can run the test with no stress as no one is looking at you and the outcome is just between you and the equipment.
@CMOT, please focus on getting this done as this is the most pressing need I have in dealing with myriads of people coming to us with this type of scenario.