Lately I have adopted a new practice when it comes to being asked to serve on a committee or do some other service task: I only agree to add a new service responsibility if I can also subtract one.
Of course it is hard to balance things exactly, so there are hazards in this seemingly simple approach. For example, the added committee might be a lot more work than the subtracted one. In some cases it can be difficult to know in advance what the time commitment and degree of satisfaction with a service responsibility will be. So you might actually be dropping a committee that didn't take much time and adding a committee that will consume your life. It's quite possible.
It can also be difficult to balance the administrative species (for lack of a better term) of committees. For example, it might be possible to subtract a department committee if asked to serve on a university committee, but the reverse would be more awkward unless the department committee is really important (e.g., a hiring committee, when there were such things).
And then there are professional service obligations that might be important to continue, even if you get loaded up with service work in your department/university.
But sometimes this system works. I was recently on a committee that wasn't a huge amount of time, but I really didn't like the committee dynamics and some of the things we had to deal with, so that was the one I dropped (with great glee) when asked to be on a different one at the same administrative level of the university.
You don't need an excuse to quit a committee that you find boring, useless, or otherwise abhorrent, but a handy one is that you are taking on another committee responsibility and will no longer have time for the Time-Sucking Boring Committee.
1 hour ago