I figured it out - I had to enable DoP in settings as you said, but I can only do that before the app plays any songs, because even when a song is stopped, the options are greyed out. So I had to do it first thing on startup.
Where was the "Direct Mode" option? It still bypasses Core Audio, as it seems to be sending bitperfect to my DAC, otherwise the sampling rate wouldn't change song to song as necessary. It also hogs the audio output and shows Integer mode - see image below. How can you tell it doesn't have "Direct Mode" anymore, functionality wise?
View attachment 74024
It is normal behaviour of Audirvana that you can only change the sound settings if a song is not playing. You don't have to do it before the app plays any songs. You can simply stop a song while playing and than you can change the sound settings too.
Since I use Windows I can't answer your question about direct mode.