The Federation has defeated several forces in the past which appear much superior to the Rebel Alliance.
The Federation defeated The Dominion, a far larger and superior force.
The Federation has had many successful engagements against The Borg Collective, who have assimilated the technologies and cultures of thousands of worlds.
Many of the mentioned engagements against the Borg were either won through sheer luck or some form of outside interference (Time Travellers, Q, etc). The most well known engagements ended in severe amounts of casualties and lost ships (See: Battle of Wolf 359, Battle of Sector 001)
The Rebel Alliance is accustomed to fighting a superior force. That is their constant reality. The Federation rarely comes up against such opponents.