A planer voice coil, e.g Magnepan, is a good example of a non-round voice coil. But as others have highlighted, a rectangular or other design leads to a lot of complications in design and construction and usually degraded performance. Many, many years ago I asked a B&W (I think) designer about a rectangular woofer they had tried and rejected. One thing I remember was that corner stress and flex led to much higher distortion and worse long-term reliability, with additional undesired modes forming as driver excursion and frequency increased.
A (one) drawback of planer magnetic designs is that, with a one-sided magnet structure, linear excursion is fairly limited so as output increases distortion rises quickly. Using magnets front and back helps, but rather than the voice coil being "immersed" in the magnetic field by wrapping around the magnet, you still have a rapidly-falling magnetic field with distance that makes large excursion impractical (IMO). Perhaps when we have superconductors and much stronger magnets...
I had an idea many years ago of creating a magnet in the form of a tube or rectangle, pierced by holes so more of a screen, then suspending a conductive membrane inside so you had a planer speaker enclosed by the magnet. The idea was two magnets in the form of channels with the membrane between: [ | ] so magnet - diaphragm - magnet. A little thought shows it is not significantly better than a conventional approach though should offer higher field strength. Should have patented it just for fun.