I use the hdmi output on the Apple TV to feed the digital signa to my tv, and from there, optical output into the dac/amp/speakers.
Any idea what might be going on?
I understand your signal path is ATV>HDMI cable>TV>Optical cable>DAC.
Firstly I am aware that HDMI is known for being a noisy connection due to its design.
It might be your TV which is in between causing the issue, either resampling/converting or reclocking the signal.
It could be that your TV is adding high amounts of jitter to the signal giving your DAC a tough time.
What model DAC do you have? Does your DAC have DPLL setting which you can adjust? This usually helps with the jitter problem when using with TV.
In my previous setup I used ATV3>Optical cable>DAC and never experienced any issues with audio quality.
ATV3 was the last model to have optical output and still receives security updates to this day. You may want to try finding one for cheap as an alternate solution.