Hash :
35fd3063
Author :
Date :
2018-11-20T10:02:48
Use xxHash for faster hash map lookups. This most strongly affects the Vulkan back-end which uses a hash map to cache VkPipelines. It also should speed up some parts of the D3D11 back-end. Bug: angleproject:2522 Change-Id: I10801654f043abce17339c30d7bf62bc8644e49c Reviewed-on: https://chromium-review.googlesource.com/c/1316889 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
xxHash is an Extremely fast Hash algorithm, running at RAM speed limits. It successfully completes the SMHasher test suite which evaluates collision, dispersion and randomness qualities of hash functions. Code is highly portable, and hashes are identical on all platforms (little / big endian).
| Branch | Status |
|---|---|
| master |
|
| dev |
|
The benchmark uses SMHasher speed test, compiled with Visual 2010 on a Windows Seven 32-bit box. The reference system uses a Core 2 Duo @3GHz
| Name | Speed | Quality | Author |
|---|---|---|---|
| xxHash | 5.4 GB/s | 10 | Y.C. |
| MurmurHash 3a | 2.7 GB/s | 10 | Austin Appleby |
| SBox | 1.4 GB/s | 9 | Bret Mulvey |
| Lookup3 | 1.2 GB/s | 9 | Bob Jenkins |
| CityHash64 | 1.05 GB/s | 10 | Pike & Alakuijala |
| FNV | 0.55 GB/s | 5 | Fowler, Noll, Vo |
| CRC32 | 0.43 GB/s | 9 | |
| MD5-32 | 0.33 GB/s | 10 | Ronald L.Rivest |
| SHA1-32 | 0.28 GB/s | 10 |
Q.Score is a measure of quality of the hash function. It depends on successfully passing SMHasher test set. 10 is a perfect score. Algorithms with a score < 5 are not listed on this table.
A more recent version, XXH64, has been created thanks to Mathias Westerdahl, which offers superior speed and dispersion for 64-bit systems. Note however that 32-bit applications will still run faster using the 32-bit version.
SMHasher speed test, compiled using GCC 4.8.2, on Linux Mint 64-bit. The reference system uses a Core i5-3340M @2.7GHz
| Version | Speed on 64-bit | Speed on 32-bit |
|---|---|---|
| XXH64 | 13.8 GB/s | 1.9 GB/s |
| XXH32 | 6.8 GB/s | 6.0 GB/s |
This project also includes a command line utility, named xxhsum, offering similar features as md5sum,
thanks to Takayuki Matsuoka contributions.
The library files xxhash.c and xxhash.h are BSD licensed.
The utility xxhsum is GPL licensed.
The following macros can be set at compilation time, they modify xxhash behavior. They are all disabled by default.
XXH_INLINE_ALL : Make all functions inline, with bodies directly included within xxhash.h. There is no need for an `xxhash.o` module in this case.
Inlining functions is generally beneficial for speed on small keys.
It's especially effective when key length is a compile time constant,
with observed performance improvement in the +200% range .
See [this article](https://fastcompression.blogspot.com/2018/03/xxhash-for-small-keys-impressive-power.html) for details.
XXH_ACCEPT_NULL_INPUT_POINTER : if set to 1, when input is a null-pointer, xxhash result is the same as a zero-length key
(instead of a dereference segfault).
XXH_FORCE_MEMORY_ACCESS : default method 0 uses a portable memcpy() notation. Method `1` uses a gcc-specific `packed` attribute, which can provide better performance for some targets.
Method `2` forces unaligned reads, which is not standard compliant, but might sometimes be the only way to extract better performance.
XXH_CPU_LITTLE_ENDIAN : by default, endianess is determined at compile time. It's possible to skip auto-detection and force format to little-endian, by setting this macro to 1.
Setting it to 0 forces big-endian.
XXH_FORCE_NATIVE_FORMAT : on big-endian systems : use native number representation. Breaks consistency with little-endian results.
XXH_PRIVATE_API : same impact as XXH_INLINE_ALL. Name underlines that symbols will not be published on library public interface.
XXH_NAMESPACE : prefix all symbols with the value of XXH_NAMESPACE. Useful to evade symbol naming collisions,
in case of multiple inclusions of xxHash source code.
Client applications can still use regular function name,
symbols are automatically translated through `xxhash.h`.
XXH_STATIC_LINKING_ONLY : gives access to state declaration for static allocation. Incompatible with dynamic linking, due to risks of ABI changes.
XXH_NO_LONG_LONG : removes support for XXH64, for targets without 64-bit support.
Calling xxhash 64-bit variant from a C program :
#include "xxhash.h"
unsigned long long calcul_hash(const void* buffer, size_t length)
{
unsigned long long const seed = 0; /* or any other value */
unsigned long long const hash = XXH64(buffer, length, seed);
return hash;
}
Using streaming variant is more involved, but makes it possible to provide data in multiple rounds :
#include "stdlib.h" /* abort() */
#include "xxhash.h"
unsigned long long calcul_hash_streaming(someCustomType handler)
{
XXH64_state_t* const state = XXH64_createState();
if (state==NULL) abort();
size_t const bufferSize = SOME_VALUE;
void* const buffer = malloc(bufferSize);
if (buffer==NULL) abort();
unsigned long long const seed = 0; /* or any other value */
XXH_errorcode const resetResult = XXH64_reset(state, seed);
if (resetResult == XXH_ERROR) abort();
(...)
while ( /* any condition */ ) {
size_t const length = get_more_data(buffer, bufferSize, handler); /* undescribed */
XXH_errorcode const addResult = XXH64_update(state, buffer, length);
if (addResult == XXH_ERROR) abort();
(...)
}
(...)
unsigned long long const hash = XXH64_digest(state);
free(buffer);
XXH64_freeState(state);
return hash;
}
Beyond the C reference version, xxHash is also available on many programming languages, thanks to great contributors. They are listed here.
- The “master” branch is considered stable, at all times.
- The “dev” branch is the one where all contributions must be merged
before being promoted to master.
- If you plan to propose a patch, please commit into the “dev” branch,
or its own feature branch.
Direct commit to "master" are not permitted.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167
xxHash - Extremely fast hash algorithm
======================================
xxHash is an Extremely fast Hash algorithm, running at RAM speed limits.
It successfully completes the [SMHasher](http://code.google.com/p/smhasher/wiki/SMHasher) test suite
which evaluates collision, dispersion and randomness qualities of hash functions.
Code is highly portable, and hashes are identical on all platforms (little / big endian).
|Branch |Status |
|------------|---------|
|master | [](https://travis-ci.org/Cyan4973/xxHash?branch=master) |
|dev | [](https://travis-ci.org/Cyan4973/xxHash?branch=dev) |
Benchmarks
-------------------------
The benchmark uses SMHasher speed test, compiled with Visual 2010 on a Windows Seven 32-bit box.
The reference system uses a Core 2 Duo @3GHz
| Name | Speed | Quality | Author |
|---------------|----------|:-------:|------------------|
| [xxHash] | 5.4 GB/s | 10 | Y.C. |
| MurmurHash 3a | 2.7 GB/s | 10 | Austin Appleby |
| SBox | 1.4 GB/s | 9 | Bret Mulvey |
| Lookup3 | 1.2 GB/s | 9 | Bob Jenkins |
| CityHash64 | 1.05 GB/s| 10 | Pike & Alakuijala|
| FNV | 0.55 GB/s| 5 | Fowler, Noll, Vo |
| CRC32 | 0.43 GB/s| 9 | |
| MD5-32 | 0.33 GB/s| 10 | Ronald L.Rivest |
| SHA1-32 | 0.28 GB/s| 10 | |
[xxHash]: http://www.xxhash.com
Q.Score is a measure of quality of the hash function.
It depends on successfully passing SMHasher test set.
10 is a perfect score.
Algorithms with a score < 5 are not listed on this table.
A more recent version, XXH64, has been created thanks to [Mathias Westerdahl](https://github.com/JCash),
which offers superior speed and dispersion for 64-bit systems.
Note however that 32-bit applications will still run faster using the 32-bit version.
SMHasher speed test, compiled using GCC 4.8.2, on Linux Mint 64-bit.
The reference system uses a Core i5-3340M @2.7GHz
| Version | Speed on 64-bit | Speed on 32-bit |
|------------|------------------|------------------|
| XXH64 | 13.8 GB/s | 1.9 GB/s |
| XXH32 | 6.8 GB/s | 6.0 GB/s |
This project also includes a command line utility, named `xxhsum`, offering similar features as `md5sum`,
thanks to [Takayuki Matsuoka](https://github.com/t-mat) contributions.
### License
The library files `xxhash.c` and `xxhash.h` are BSD licensed.
The utility `xxhsum` is GPL licensed.
### Build modifiers
The following macros can be set at compilation time,
they modify xxhash behavior. They are all disabled by default.
- `XXH_INLINE_ALL` : Make all functions `inline`, with bodies directly included within `xxhash.h`.
There is no need for an `xxhash.o` module in this case.
Inlining functions is generally beneficial for speed on small keys.
It's especially effective when key length is a compile time constant,
with observed performance improvement in the +200% range .
See [this article](https://fastcompression.blogspot.com/2018/03/xxhash-for-small-keys-impressive-power.html) for details.
- `XXH_ACCEPT_NULL_INPUT_POINTER` : if set to `1`, when input is a null-pointer,
xxhash result is the same as a zero-length key
(instead of a dereference segfault).
- `XXH_FORCE_MEMORY_ACCESS` : default method `0` uses a portable `memcpy()` notation.
Method `1` uses a gcc-specific `packed` attribute, which can provide better performance for some targets.
Method `2` forces unaligned reads, which is not standard compliant, but might sometimes be the only way to extract better performance.
- `XXH_CPU_LITTLE_ENDIAN` : by default, endianess is determined at compile time.
It's possible to skip auto-detection and force format to little-endian, by setting this macro to 1.
Setting it to 0 forces big-endian.
- `XXH_FORCE_NATIVE_FORMAT` : on big-endian systems : use native number representation.
Breaks consistency with little-endian results.
- `XXH_PRIVATE_API` : same impact as `XXH_INLINE_ALL`.
Name underlines that symbols will not be published on library public interface.
- `XXH_NAMESPACE` : prefix all symbols with the value of `XXH_NAMESPACE`.
Useful to evade symbol naming collisions,
in case of multiple inclusions of xxHash source code.
Client applications can still use regular function name,
symbols are automatically translated through `xxhash.h`.
- `XXH_STATIC_LINKING_ONLY` : gives access to state declaration for static allocation.
Incompatible with dynamic linking, due to risks of ABI changes.
- `XXH_NO_LONG_LONG` : removes support for XXH64,
for targets without 64-bit support.
### Example
Calling xxhash 64-bit variant from a C program :
```c
#include "xxhash.h"
unsigned long long calcul_hash(const void* buffer, size_t length)
{
unsigned long long const seed = 0; /* or any other value */
unsigned long long const hash = XXH64(buffer, length, seed);
return hash;
}
```
Using streaming variant is more involved, but makes it possible to provide data in multiple rounds :
```c
#include "stdlib.h" /* abort() */
#include "xxhash.h"
unsigned long long calcul_hash_streaming(someCustomType handler)
{
XXH64_state_t* const state = XXH64_createState();
if (state==NULL) abort();
size_t const bufferSize = SOME_VALUE;
void* const buffer = malloc(bufferSize);
if (buffer==NULL) abort();
unsigned long long const seed = 0; /* or any other value */
XXH_errorcode const resetResult = XXH64_reset(state, seed);
if (resetResult == XXH_ERROR) abort();
(...)
while ( /* any condition */ ) {
size_t const length = get_more_data(buffer, bufferSize, handler); /* undescribed */
XXH_errorcode const addResult = XXH64_update(state, buffer, length);
if (addResult == XXH_ERROR) abort();
(...)
}
(...)
unsigned long long const hash = XXH64_digest(state);
free(buffer);
XXH64_freeState(state);
return hash;
}
```
### Other programming languages
Beyond the C reference version,
xxHash is also available on many programming languages,
thanks to great contributors.
They are [listed here](http://www.xxhash.com/#other-languages).
### Branch Policy
> - The "master" branch is considered stable, at all times.
> - The "dev" branch is the one where all contributions must be merged
before being promoted to master.
> + If you plan to propose a patch, please commit into the "dev" branch,
or its own feature branch.
Direct commit to "master" are not permitted.