Great first print (pla?) Patrick Bateman in DIYUADIO recommends the use of some kind of plaster or putty to seal the area between the driver and the throat. I have seen his charts and the difference is impressive, several db at crucial points. That technique will cover less that a 5 mm radius difference. for more you need and adapter. You can try also with an acetone bath or some paint/fluid to smooth the surface if if bothers you. Throat is probably much more important that the ridges. I outsource my 3d prints, but it is a common well documented procedure.
The guy who makes my molds for the motion control rigs I build, 3d prints the brackets, then uses bondo for filing and polishes to finish the parts. Afterwards he applies some coating for finishing. Its a length process but the results are amazingly glass smooth ( we have to do that as we mold and cast them later), but...
....if I were you will not even bother at this time, make your adapter, try it, test it, and see how you like it. I think it will be great.
keep posting pics, really interested on your printing / fabing process.
pd: you maybe able to use a small scaling factor to scale down the size of the print to make it fit the driver. Measure the difference of the print and the whole opening and calculate the scale down factor.