...dig out some measurements from the past couple of years.
The problem you are talking about seems to me, not that I am any kind of an expert, to be the vertical movement of the entire floor... plus harmonics of that fundamental frequency. Maybe there are other things going on. I have a 1925 house, so I know of what you speak.
Everything I say will be magnified if you have a flat ceiling, as I do.
First, keep in mind the floor has a lot of mass, different materials, and the mass of the furniture and people on top. It will take a lot of energy to get all that mass moving. But rhythmic sound, harmonics, once it gets going it is going to get GOING! So at some point more volume will just lead to more problems, since the only thing that can change the level of sound produced by the floor is the sound going into the floor.
Also, "floor modes" at the low end will be related to, but not identical to, dimensionally associated room modes. Suckers like to hide. If you see a steep null go flat then shoot up a peak at higher and higher volumes.... floor.
The long axis of your room is likely the problem, so consider a slightly wider more toed in set up... if you think that might work for you given the rest of the space.
Finally, think about this part of the process as literally playing your floor to produce sound in a more controlled way... before going wild. My overall strategy is to try to delay energy organizing in certain ways.
Here's some measurement options, from very direct to more inferential, then finally sweeps.
Play your floor like a kickdrum, once, record RTA peak. This is from a different set up (different floor weighting/tuning) of my floor, and the yellow line is the kickdrum test:
The subsonic stuff sure, but also 30.8. That's my floor fundamental. Though really 30 works much better as a single EQ, and two narrow ones work even better. I'll get to those later.
The red and blue are from B.R.A.T measurements (J. Ramone, et. al., 1976). Following the method, I input a point source signal of 10 seconds at ~4hz at the maximum level I was willing to do (red), and then found an input level that was levelish to 200 (blue.)
I folded up two towels and hit my floor with a baseball bat.
Blue: thirty is always there, but we can see the subsonic stuff, good for AV rumble for sure. While 30 is my entire floor, 50 is the engineered wood floor on the front 2/3rds of my room. I have two floor fundamentals. Everything else is floor resonances, and a bit of room. Close enough, since it is pretty room specific.
Red: Break up mode energy, that peak at 230ish is the engineered floor breaking up like an overdriven drumhead. But note how it "sucks" energy from the frequencies below it.
Floor modes shift energy. Solution for measuring that? Try white noise at various volume levels, look for patterns. At some smoothing levels (I use a 5 second peak RTA for these), you might see a peak go to a double peak and back and forth at different SPL. Likely a floor issue there.
Alternatively, this graph shows white noise, then I gently patted the floor at ~4hz. Notice how my floor with more energy fills things in nicely. So potentially...in actuality it means more floor produces less extreme problems, not more, at my reference SPL for EQing.
You can see some dips turn into peaks, and a double peak emerge at 30. That one's just for fun. Mostly.
That's 1/24 smoothing btw. The most practical way I have for finding double peaks is to watch for them develop as my 5 second RTA of white noise fills in the lows.
OK, so I recently did some measures of "what happens if I open THAT window that makes no sense to open?" And I start with a sweep.
Here's my current, 1/24. I do use a 512 sample and start my sweeps at 15 to get the floor moving :
I really don't care about that, but note the seeming drop off at 13k for later. Instead I am going to zoom in on impulse:
See 200ms? Say hello to my floor delay!
If I align the REW EQ suggested frequencies to be in phase with that delay I get a closer match between predicted and expected performance. This explains why REW said 30.8hz for both sweeps and white noise in that very first measurement I posted-yellow line, but 30hz worked way better.
Finally, using this sweep, the spectrograph shows some indication of overall floor level:
Up above 10k, what looks like late reflections at 200-400ms is my floor, mostly in the back 1/3 of the room, winding down. A new rug pad with rubber backing made it go away for a while, until the squeaky floor boards beat cavities into it. Then it gradually came back.
All of that is great for finding out what the floor does at a specific frequency, but what it misses is how energy ends up in a frequency that comes from other frequencies. So when tuning, here's what I do.
I use white noise, usually. Feel free to use a sweep. Develop EQ to target. Then....
I use a 15 second sweep, 200-20-200, from Bunker Audio (maybe bunker analog?) on Amazon. It is a wider sine wave than REW. This gives me sweep accuracy for the peaks, but white noise accuracy for the dips. I then use the a 20k-20-20k AV sweep for over 200, and to measure residual energy that goes from above 30 and 50 into 30 and 50. (White noise works well for under 30, btw).
1/12 smoothing here. I usually eq with 1/6.
That's where things end up, just a bit over level for the bass. For my sierra LXs 10 feet off the wall (6" two way speaker). Any bumps on the rolled off AV sweep below 60 show where I'll get a bit extra boost with music, more boost with more bass and with more volume. So 50, 40, and 30 will be higher than the sweep measure shows.
I hope that helps. I hope your issue turns out to be simple. And my simple TL;DR advice is...
Tune your EQ frequencies to be in phase with your floor delay, roll off your bass as steeply as you can stand, and use 2 narrow PEQs on your fundamental frequency... hard!
If things improve, you've found a path. If not, not much time wasted.
I'm sure there are better ways, but I really didn't find much helpful advice on this issue in my searches. So you get my trial and error results for methods.
(my apologies for any typos, I was running out of time for this.)