Your infamy doesn't affect anything NPC's are supposed to attack you more often but that is still not that used. You are also supposed to not supposed to be allowed in certain shops but no guild owner uses that. You infamy goes down over time.
