Going from RCA to XLR is easy, as whether the screen is joined to pin 1 or pin 3 doesn't matter much, and more importantly won't do any damage to the RCA or XLR ends. Use whichever results in less hum and best signal quality. If the XLR input is transformer coupled, fully floating, then leaving pin 3 disconnected will give very weak sound, so making it obvious that it's wrong.
Going from XLR to RCA, one needs to know what type of balanced output is behind the XLR. If fully floating, transformer coupled, then pin 3 MUST be joined to pin 1. If centre-tapped transformer coupled, then pin 3 must be left unconnected, but the signal level is halved. If electronically balanced, i.e. without transformers, then if fully floating, pin 3 can be connected to pin 1 or left floating, whichever works better, but if the output is electronically balanced, but effectively centre-tapped, then connecting pin 3 to pin 1 will short out half the output, which can lead to higher distortion, and possible damage to the XLR output stage. Safest way to unbalance an unknown XLR output is to leave pin 3 unconnected. If it works fine, then all well and good. If the sound is weak, especially with high frequencies only, then connect pin 3 to pin 1.
Better equipment will state how to unbalance outputs and inputs, as getting it wrong will either be obvious, or could damage the XLR output.
S.
