Exactly.
Having a 3.5mm jack also makes waterproofing/IPX(58) certification more difficult which is a near requirement on high-end phones these days. Removing it makes life easier for manufacturers. It's half the reason why most high-end phones don't have removable batteries these days.
That's certainly not to say its not possible. A few years back I had a Japanese phone which was waterproof/IPX certified and had both a removable battery & 3.5mm headphone port, but there were rubber seals all over & would overheat super easily. A few manufacturers have IPX-certified phones with 3.5mm headphone ports but they must have to coat the PCBs in a sealant which is a pain to inspect post-production.
Honestly, the mass-majority of people who own the high-end Samsungs, Apples, etc that don't have 3.5mm headphone jacks wireless headphones/earbuds or listen to music over their car bluetooth...but lots of these people are business men that get caught in the rain. Many, many mid-tier and low-end phones still have headphone jacks...because in the end its all about the targeted income range that each phone manufacturer sets for their products.