My wife and I did a lot of swinging in our mid-30s, and yeah, we eventually met another couple that we really hit it off with and eventually kind of 'settled down' with them. We've been together with them for about six years now.
I wouldn't really say we're poly since we don't all live together or anything, but we did all fall in love with one another, message each other daily, and get together (both together and separately) about once a week. My wife and I have a kid and we're not interested in 'coming out' to him, our family, or our co-workers. Because of that we only have a little bit of time that we can sneak away each week, and once we met them we pretty much spent any and all of it with them. They're childless though and have had way more free time to 'get out there' and meet others. This caused a lot of difficulties early on as we all began to transition from swinging to.. something else, because there were times it felt like 'unspoken rules' were being broken.
When you start swinging it's really important that you've been with your partner long enough to feel 'solid' together, and to set rules and boundaries so that both of you feel safe and secure. But when you begin to see someone who is already in another relationship as your partner as well, you haven't had time to build up that solid foundation with that person *and* you can't set rules and boundaries with them since their primary relationship has to come first. If any of the four of you aren't on the same page, then, welp, there's going to be drama, heartache, frustration, the works.
In our case, as I realized I was falling in love with the other woman, I tried to put some distance between us because I realized all of this might be getting out of hand. But by then she had fallen for me as well, and couldn't bear the thought of us not being together, so we persisted. I needed them to slow things down with others while we figured all of this out, but her husband didn't want to. They were a lot newer to swinging than we were and so it was still new and fun and exciting for them, she was willing to slow down but he wasn't, and so there were a lot of times I felt like my feelings didn't matter to them, which hurt. Those first couple of years were difficult, and a number of times I broke things off with them, but we always found our way back to one other.