This sounds like something that should be obvious, but it isn’t. If you just password protect your party you can still encounter other players in multi-party areas.
Before you start a quest, make sure the radio button inside the red box in the above picture is checked. This will not only prevent other parties from joining you in a multi-party area, it will also start a brand new multi-party area on the server. With this, you will never have to encounter another player in the field who isn’t invited.