I’d say that it depends on whether you expect them to be there when you return, and/or if your presence is expected in the meantime for whatever reason.
Just have some situational awareness. Normally it wouldn’t be an issue to leave a room to get a soda from the fridge and then return. However if you’re in the middle of a timed game or while moving heavy furniture, then it would be rather annoying for the rest of the people if you just suddenly disappear without explanation.
Salsa