if I take a coworker home, go out to lunch, etc w/o telling my SO, and they see that deviation in my routine, they could start doubting that trust

This means there are still significant insecurities in the relationship that can bubble up and become problems, and you know about these.

You do not trust your spouse to trust you and not misinterpret your intentions.

Paradoxally You can defeat some of this insecurity by being transparent and welcoming misinterpretation if you believe you both have full trust in each other.

As a high anxiety person myself, this works to defeat the anxiety which is often feared of the unknown. By proving that deviations to your routine are not something they should feel anxious about, then that anxiety can melt away.