As with many things, the answer depends and too much of anything is usually bad or wasteful…
For a closed box speaker, if the empty ideal box case makes it critically damped, more damping is likely to make it sound worse. Critically damped is what some audiophiles prefer to get optimal transient response, but most often, it does not sound great imo. So, this all starts with a specific driver and putting it in an enclosure of a chosen volume. Since you chose these before adding damping, this is the foundation for how a closed box speaker will sound. With too much damping material, you can get frictional losses and so would sound more “dead”.
Since ideal boxes do not exist, damping materials can be used to help control resonances too. Adding excess damping material might help reduce resonances in a marginal enclosure, but taking that approach will more likely get you to one of the aforementioned bad conditions.