Hi and welcome!
In short, no! Two boars around girls equals bloody fights; neutering takes only away the ability to make babies but it doesn't not change boar behaviour.
In the wild, a core group of sows will live with one dominant boar while the other bachelors are hanging losely around in the hope of being able to get the top spot themselves eventually. Keeping two boars together mimics a bachelor environment. As soon as you add even one girl, you get the core group beaviour, where boars will fight for the right to sire the next generation.
Please don't break what is not broken and leave your two groups as they are. If you wish, add a neutered boar to your girl group (it helps cutting down on some squabbling), but make sure that he will be accepted - not all adult sows will actually do so.