Well, I'm pretty sure everyone has this problem or will eventually encounter this problem. It happens to me & a lot of other people too. No one really knows how to solve this problem yet. You should consider sending STEAM Support a bug report sometime. For me, whatever server I like, I would write down the server IP on a piece of paper so I don't lose the server.