What you are seeing in this case is voltage difference between the batteries.
If using an ohm meter, & that voltage difference is very small, it "tricks" the ohm meter to where it thinks its seeing a resistive circuit.
If you energise that relay, it will measure zero ohms (or very close to zero) as would be expected.
To prove the relay contact is 100% open circuit without any resistive 'leakage',
You will need to remove at least one of battery lead from it and measure resistance across its terminals as usual.
In all, it sounds like its working normally, just confused on measuring to verify.