This is not a problem! In fact, group photographs often take a very long time to set up, which simply eats into your day. We would recommend only having a limited number of photographs of close family or friends, say up to 6, so that you can get on with your day. We often capture shots of groups through the day anyway and we find our clients prefer these spur-of-the-moment pictures.
As for the “whole group” shot, this is also becoming more obsolete. What we have found is that they take 20 minutes to set up one shot, people don’t want to be in it, and that the bride and groom often then overlook the image anyway. We are happy to take it but we would ask you to make sure it is what you want before committing to it. We would also ask that any ushers you may have are on hand to help with the organisation of this.