Its not intuitively obvious, but a jig can make holes smaller than the baseplate. The "price" you pay for small circles is that the pivot point on the jig is underneath the router baseplate, which adds a small degree of difficulty to a jig design. The Milescraft jig mentioned above, which I also own/use (with a full size router), solves this by having a sort of quick-release secondary baseplate you attach to the router. The pivot screw can then fasten the jig to the work piece before the router "snaps" into place above the pivot screw. Simpler jigs may use a small nail or pin as the pivot, and so for small circles you must limit its extension above the work piece to be = or less than the jig thickness so it doesn't interfere with the router. In those cases, once the pin is place on the piece, you may need to semi-blindly try to "land" the jig in the right spot onto the pin, since the router may obscure any potential view.
Cutting the "outer" hole (ie, a 104mm hole 4mm deep in the case of the tweeter) will likely require multiple passes of decreasing diameter (if you're using a 1/4" cutter) to create an appropriate width rebate/rabbet "shelf". But eventually you'll need to cut the inner hole (77mm in case of tweeter) and this presents a challenge, since if you cut all the way through to the other side, the inner circle scrap (with the pivot point) will ultimately become detached from the panel and the router can then easily move off into places you don't want it to go. A couple choices to cope with this (or hybrid of the two): Cut this circle just deep enough to leave a mm or two material in place below the cut, or cut three portions of the circle (say 110 degree each) which would leave three "spokes" holding the material. At this point hand tools can be used to finish the penetration.
Even with a jig, getting the circle precisely the correct diameter and the precisely correct rabbet/rebate depth is somewhat challenging. Definitely refine your technique and verify dimensions on scrap material before cutting the real baffle panel. Its a necessary skill for decent DIY speakers, because there are meaningful negative consequences to the drivers not being properly flush with the baffle. A jigsaw may suffice for simple mounting in a crude subwoofer, but decent speaker baffles are not in the jigsaw domain.