Automated Conversion of FLAC to OPUS on Linux

[Berwhale]

Hi,

I have a reasonably large library of FLACs (~24k tracks, just over 1TB size). The FLACs reside on an SMB share on my Synology NAS. I'd like to keep a second copy of the files, transcoded with the Opus codec. The Opus copies will be used for mobile listening on my phone, tablet, portable media player and car. I'd also like the Opus library to be automatically updated whenever I add new FLACs to the main library (synchronisation of updates, renames and deletions would also be nice).

I have just begun bulk converting the library using a trial version of TuneFusion (from the dBpoweramp folks). The app is currently syncing the FLACs on my Synology NAS to Opus ones on a local SSD on my main Windows 11 PC - I looks like it's going to take around 3 hours using all 24 logical cores of my Ryzen 9 7900

Now, I could easily pay for TuneFusion and set it up to synchronise the FLAC share on my NAS with an Opus one. However, this places a dependency on my main PC and I have a lower power 'utility server' that is always on for this sort of thing.

The utility server is running Ubuntu Server 22.04.3 LTS on a virtual machine running on my little vmware ESXi host (a Dell Optiplex 7060 micro with an 8th gen i3 CPU). The server runs an assortment of containerized services including LMS.

Is anyone aware of any nice solutions to achieve the transcoding and synchronization above on Linux? (preferably one that is already containerized).

 

[Berwhale]

Ah, sometimes just writing what you want down helps you find it!

It looks like Tdarr will do the trick. You just need to supply the correct arguments for ffmpeg

 

[Berwhale]

OK, it took 3 hours and 10 minutes to convert my FLAC library (23,918 files) . However, 24 of the files had errors and could not be transcoded and raised the following errors:

17 with 'FLAC__STREAM_DECODER_SEARCH_FOR_FRAME_SYNC'
3 with 'md5 did not match decoded data'
2 with ''FLAC__STREAM_DECODER_READ_FRAME'
2 with 'FLAC__STREAM_DECODER_READ_METADATA'

I don't believe that I've ever had any issues playing these files and ones i've spot checked play fine in MusicBee.

So now I am now attempting to decode to WAV and re-encode to FLAC using FLAC Tools (via the Windows Frontend GUI) and then retag with MP3Tag...

 

[AnalogSteph]

3 with 'md5 did not match decoded data'

Could be bit rot. Do you happen to have an old backup to compare against?

 

[Berwhale]

Could be bit rot. Do you happen to have an old backup to compare against?

I have a second copy on another Synology NAS (Both the NAS are using BTRFS), I might have another copy on an off-line USB drive somewhere. I'll check the 2nd copy when I get to those files.

 

[Berwhale]

Bah! Tdarr does not embed artwork in the transcoded .opus files or support the extraction of the artwork to a file...

I hacked* one of the community plug-ins (Tdarr_Plugin_Greg_MP3_FFMPEG_CPU) to convert Flacs to Opus and created another version that extracts the cover art of each file to <filename>.jpg (Cover art is stored as a single frame video stream, it's not part of the metadata!). However, I can't find a way to create a single folder.jpg file using any of the plug-ins or the example code i've found.

Given the way that files are parsed by plug-ins recursively, one approach would be to create a plug-in that renames <filename>.jpg to folder.jpg, unless folder.jpg already exists, in which case <filename>.jpg should be deleted. This would rename the 1st .jpg and delete all the others (which would work for me as i'm fanatical about keeping cover art consistent in my Flacs).

At this point, the solution is starting to require a significant investment of my time and i'm beginning to wonder if the juice is worth the squeeze. I am considering doing the transcoding and library synchronization in MusicBee instead!

* I don't speak Javascript (although I can carry a simple conversation in Python, C++, Basic and a few other languages).

 

[Berwhale]

So I realized (duh!) that my PC can transcode FLACs to Opus or MP3 way quicker that it can transfer the files to my phone, tablet, media player, etc. So, although it galls me to not keep a single, size optimized, version of my media, it's actually easier to just transcode the library on the fly.

This approach has the added benefit of being able to tailor the sync to the target device. It turns out that my Shanling M0 does not support Opus and my car doesn't support Opus or like album art over 600px, there's also no point sending hi-res artwork to the M0 with its 240x240 screen. Using MusicBee, it is very easy to tailor the transcode codec and extracted album art per device (Opus for phone and tablet with original album art size, MP3 for M0 and car, with album art sized to 240px and 600px respectively).

So, rather than try to remove my main Windows PC from the workflow, i'm now making it the centre of attention. In fact, I might buy a another 2TB NVMe drive to store the 'master' copy of my FLAC library and sync it to my Synology NAS in the same way as all the other devices i've mentioned (obviously without the transcoding or art work manipulation).

 

[kchap]

I'm about to go through a similar exercise for FLAC to MP3. What made go for opus over MP3?

 

[kysa]

Is anyone aware of any nice solutions to achieve the transcoding and synchronization above on Linux? (preferably one that is already containerized).

I fuzzy find every *.flac in directory x and convert it into VBR opus via .sh i wrote and then rsync it across my NAS and 3 SD cards, but it's not as fancy as transcoding on the fly. I don't see much reason to transcode music every time when you can encode a music file once and store it on SD Card/stream it via mpd/Jellyfin/Plex with less CPU usage.

I'm about to go through a similar exercise for FLAC to MP3. What made go for opus over MP3?

I'm failing on 256 kbps MP3 ABX test and opus 128 VBR is 0/10 for me. Opus files also tend to be smaller than LAME MP3 based on my experience.

and created another version

Make sure to publish it on GitHub.

 

[Berwhale]

I'm about to go through a similar exercise for FLAC to MP3. What made go for opus over MP3?

Potentially smaller and better quality. However, I subsequently discovered its's not supported by all my devices. I could have fallen back to OGG for the M0, but that's been deprecated in favour of Opus.

I don't see much reason to transcode music every time when you can encode a music file once and store it on SD Card/stream it via mpd/Jellyfin/Plex with less CPU usage.

Device support is one reason, my Shanling M0 and car don't support Opus, so I would have to compromise by reverting to MP3 or keeping multiple lossy copies of my library. With respect to transcoding on the fly, it's really only done once per device. It's not like I have to transcode everything when I update my library, it's only the new stuff.

Make sure to publish it on GitHub.

My bad, I killed the Tdarr and Tdarr worker containers and their config once i'd decided not to go down this route. To be honest, it was a pretty trivial change to the Tdarr_Plugin_Greg_MP3_FFMPEG_CPU plug-in to change the output to Opus or to extract the first video stream as a .jpg.

 

[phofman]

IMO this could be an inspiration, if not directly a solution https://github.com/nschlia/ffmpegfs

 

[Berwhale]

IMO this could be an inspiration, if not directly a solution https://github.com/nschlia/ffmpegfs

Interesting solution, but this is for more for a streaming use case. Anything I would stream to at home can handle flac files and has sufficient bandwidth to transfer in real time.

I can see this being a good option for mobile use, but my Plex server already does this with the PlexAmp application - Plex will transcode to Opus on the fly and cache or download locally if requested. However, there are a couple of reasons why I don't use the feature:

1. My Plex server runs on a Synology NAS, although it is an Intel based unit with a Celeron CPU and plenty of RAM, I am averse to giving it any transcoding tasks (yes, I realise that transcoding audio is not really going to stress it, but in my mind 'transcoding + Plex' = Bad)

2. Plex has a feature to pre-stage lower bit rate versions of video files for playback on devices which lack the bandwidth or capability to display the original file, it's called Plex Optimized Versions. This avoids transcoding video streams on the fly (see point #1 for my view of this!). However, Plex has not implemented the same capability for audio files. If they had, I would have probably just let Plex pre-stage everything to Opus.

For those that are interested in ffmpegfs, here's an article showing what it looks like: https://www.linuxuprising.com/2020/03/ffmpegfs-is-fuse-based-filesystem-for.html

 

[phofman]

Interesting solution, but this is for more for a streaming use case.

FUSE is a user-space filesystem. It would let you mount a virtual filesystem where the files would be converted from the original formats to the new requested format, transcoded on the fly when opening the file. You can copy the converted files to a new location, or keep the original files and do the conversion on the fly every time when accessing the file.

 

[Ricardus]

Interesting that I use linux as my daily driver and I am just hearing about Opus now. I must be living under a rock.

 

[Berwhale]

FUSE is a user-space filesystem. It would let you mount a virtual filesystem where the files would be converted from the original formats to the new requested format, transcoded on the fly when opening the file. You can copy the converted files to a new location, or keep the original files and do the conversion on the fly every time when accessing the file.

Yes, I get that. I suppose I could set it up and use something like FreeFileSync on Windows to synchronize the FUSE file system with the target devices via USB. This is essentially the same as what i'm doing with MusicBee, but with the transcoding work moved to my NAS. The downside is that the transcoding would be done by a far less capable CPU, the upside would be that I don't have to transfer relatively large FLAC files over the network. With more capable clients (i.e. those running Android) i'd also be able to sync them directly with the NAS over my network.