The Fatimid caliphate contested Abbasid legitimacy itself, not just their rule.

Evidence

The Cairo caliphs, it should be remembered, were not merely rebellious rulers achieving some kind of local autonomy or independence, as happened in many places during the decline of Abbasid power. They were challenging not just the suzerainty but the very legitimacy of the Abbasid caliphs. For them, the Abbasids were usurpers, and their Islam was corrupted. According to Isma ili teaching, the Fatimids represented the authentic line of heirs of the Prophet, and their Isma ili doctrine was the true Islam. — Bernard Lewis, From Babel to Dragomans, 444