I've still been experimenting. I've realised a few things;
First regarding workflow, how I was testing overlapped IR's in Audacity can be achieved a similar way in REW. You do this by using Filtered IR settings close to the target frequency, "Zero phase filtering" which moves the direct wave to time=0, along with the dBFS graph setting. You can overlap filter IR's on the Overlay screen, to see improvements. This screen is also handy in other ways because;
1) it shows the Schroeder Intergral line which is an indicator of the "sound/echo left to hear" in a way (until -60dB reached). The steeper this line the better, because that indicates reduced decay times. Any areas touching the line are what is "holding up the line", so if those areas are reduced, the line should improve. Also any areas that stick above the line are really bad, because they are really overpowering considering their dB:time relationship!
2) it shows the potential "range" for a new direct wave to cancel a late reflection. More on this later.
Next thing I noticed is that the 1st room length mode (which will be 30-50Hz in most room) is such a long frequency that it seems to blend nicely in to a big hump. Shorter frequencies like the 2nd and 3rd room modes often have "secondary humps" cause by reflections on at least their 2nd pass of the LP. If you outptut a cancel tone, those 2nd humps will occur again too. I also found that because my speakers are rolling off higher than the room mode, the potential gains are reduced due to being closer to the noise floor already (and I haven't tested my sub yet). But looking at OCA's measurements where his speakers are near full power at the room mode, better results can be achieved. I did manage to make an IR that flattened the "back of the peak", but it didn't do so much after that.
Edit: Had to recreate the mdat (also attached it). The cancellation impulse is not bandpassed to make trial and error faster, so these results should only be looked at for 30-40Hz. The improvement is obvious in the time domain (on the spectrogram, with wavelet rendering) because the normal result was "red" until ~150ms, whereas with VBA it stops at ~80ms. I also made a version with EQ only, and whilst it fixes matches up to ~80ms, it doesn't have the benefits longer in time, AND the surrounding frequencies (30-33, 35-40) are not improved.
Last thing I looked at properly was how quick the back wall reflections arrive. In my 4m long room, it only takes the back wall reflection 12.8ms more than the direct wave. The reflection of the back wave off the front wall is then only 9ms later. It seems they blend in to the "primary hump" that occurs just after the direct wave.
So here's what I've been doing in REW: I take a screenshot of the IR screen and put that in Paint, with the cursor over the first peak after t=0 (the first peak of the direct wave). Then I change the graph color and move the cursor to a peak where I want the cancellation wave to start, and take a screenshot in another session of Paint. Then I copy a chunk of the 1st screenshot and overlay it transparent with the second. Here's the effect;
You can see if I made an custom IR with that delay, the cancellation direct wave would cancel the original "2nd hump", but then the cancellation wave would have it's own 2nd hump. This could be good or bad, and it seems from a few trials that it's often bad because there's a high chance of phase shift over that long of a time period (look at the dramatic phase shift at 100ms for example). If it were good (destructive), then it would improve the decay, but if it's bad (constructive), then it will just make more noise at the time, in fact making the decay worse. Then I wonder if I need to "cascade" cancellation tones, to cancel the OG cancels 2nd hump? When would it end?!
Anyway, to finish that workflow off I take note of the cursors. The direct wave is 4.2ms, and the wave I want it to cancel is 33.3ms, thus the delay I want is 29.1ms (but if the cancel tone is bandpassed, it may have require a bit more delay, and I haven't worked out how to account for this yet beyond adjusting by a millisecond or 2 and seeing what happens). Also the cursors show the relative SPL that can be used to reduce the gain of the cancel tone, but maybe the SPL of the 3rd peak is a better metric in this example. Lastly you have to switch to the % graph and check the amplitude of the peaks at the 2 times. If they are both the same, then the cancel tone needs inverting.
Last thing for this post is that I tested if the "simulations" (REW altered measurements) actually apply to new measurements, and they do! I tested my Right speaker only with it's current EQ, with EQ and VBA that I made the other day, and compared that to EQ*VBA simulation. The simulation of VBA matches the measurement with VBA very well. In this case though, the VBA has filled the decay slope in, making it worse lol. Possibly I got the inversion wrong, or possibly there is an "issue" because the VBA was developed against a stereo sweep, and this is it applied to a mono sweep?
Here are a few example frequencies showing how well the VBA measurement vs simulation tracks (not to judge the performance);
it doesn't sound like you understand how that view is generated in fourier mode and how there is a trade off between temporal resolution & frequency resolution, I suggest reviewing that and then it should be clear. You can also look at it in wavelet mode and switch between the freq resolutions to get an idea of what is going on.
OK, I haven't looked in to that yet, but it's good to know that the spectrograms method can have temporal "affects", which is related to what I was noticing.
It is still weird seeing large phase swings on the graph though, like in the following screenshot of the measurements I was just posting about;
But it does seem that the phase jumps are 360° out (so still in phase), so I guess it isn't a problem, as long as elsewhere it hasn't adjusted t=0 SPL peak.