Jesus tells a parable at the end of Matthew 18 about a guy who is forgiven a fortune and then exacts the debt of someone who owes him comparably little. It's not a fun story.
They were both in debt. The debts were significant (though the one significantly more).
Jesus told stories to provoke people. Here's how I'm provoked...
The part that gets me is how easily I can hold a grudge, remember a person's flaws, be frustrated beyond what is appropriate, and on and on. I can make people wear their sin even though Jesus has taken it off and wore it for them.
Actually, I can't. I can't undo what Jesus has done.
That's the issue of unforgiveness. It's not a matter of the other person having to wear the clothes I put on them. It's a matter of me only being able to see them with that kind of lens. I may think I'm punishing them by not forgiving them, but the only person being spiritually affected by that is ME. Jesus has already taken their sin from them.
Is the debt that needs forgiveness real? Yes. Sometimes even significant.
I can just get a little prideful, trying to put back on them what Jesus has carried away. It's not just affront to them. It's an affront to Him.