If you are going to use a speaker meant for vertical use on it's side then I'd be looking at the vertical contour plots because that's what you're going to get more or less + room gain and interactions. Check out
www.spinoramma.org where you can use the search to narrow down your size needs and see what's out there with verifiable data.
Given the limited size constraints of the speakers, even if you used the full volume of the shelf space allowed, I don't think there's going to be a whole lot of speakers able to drive >80dB at 15' away without getting into distortion. You've got a big room and big rooms need big speakers (or multiple sets) to excite them. Adding a sub in a room this big will be an interesting venture as well.
Have you considered coaxial speakers? At least it wouldn't matter so much putting them on their sides.
Best option: DIY them yourself with high sensitivity drivers and build them directly into your bookcase if that's the only placement option.