Interesting. Almost all of my friends use and are comfortable with swinging vs polyamory for that distinction, but I've certainly met people who aren't so happy with swinging as a term. I'm happy to take up terms that cause the least confusion.

However, I don't think using polyfidelity in that sense helps, as I think the meaning I suggest is already widespread(*), and that the differences between those two relationship structures are large enough that interchanging them would cause quite a bit of confusion.

*at least, among those likely to discuss alternative relationship structures at all