Just because you cared about a person in the past doesn’t mean that you will always care about them, or you should care about them. If they were toxic to your being, then leaving them behind was the best choice you made. You left them in the past for a reason. Don’t let them give you a reason to go back there.
The worst thing in the world can happen, but the next day the sun will come up. And you will eat your toast. And you will drink your tea.