You get approximately 50k XP for successfully defending a fort, while you get approximately 160k XP for failing to defend a fort and then retaking it. You earn a lot more experience by retaking a fort than you will successfully defending it. The best and quickest way to gain lots of XP in Act 4 is by losing your keeps instead of defending them.
