@mansr,
Well, after some time of digesting the known ESS white paper and the ES9038q2m datasheet I would say it's all in there, or at least most of it.
What was a new discovery not explicitly touched in the papers is that the common-mode signal on the I/Vs is quite large and distorted. And it is rf-polluted (as is the differential signal) and the whole concept relies on a stable and deep cancelling. This I could clearly see with the D10B, looking at the single ended signals individually as well as at the differential signal in comparision.
When the indivual I/V's are not up to it and have problems coping with the fast transitions and/or additional glitches and then create asymmetrical errors from demodulation, slew-limiting or whatever, the cancelling falls apart. Additionally we may have less than optimum general operating points here, spoiling the baseline some more.
Same applies for the subtractor itself as some glitch content may just jump over the I/V beyond its bandwidth and create problems in the subtractor as well.
At least that's my pet theory for the momement as to what is happening on the KTB and why the cross-capacitor helped.