> You are aware of course that you can't use any hashing function on its own to
> detect duplicates? - the best you can do is detect *probable* duplicates,

Actually, if you choose the right hash function you can detect duplicates.

If you create a UDF based on/using SHA256, the result would be unique (with a 2^256 certainty) -- there is no known collision of a SHA256 hash (https://en.wikipedia.org/wiki/Hash_function_security_summary).


Sean