kmx git

Commit

Hash : 35e69fc7
Author :
Date : 2018-02-26T09:04:36

New feature: "Large Window Brotli" (#640)

* New feature: "Large Window Brotli"

By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.

Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).

Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.

Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.

Other C changes:
 - combined / moved `context.h` to `common`
 - moved transforms to `common`
 - unified some aspects of code formatting
 - added an abstraction for encoder (static) dictionary
 - moved default allocator/deallocator functions to `common`

brotli CLI:
 - window size is auto-adjusted if not specified explicitly

Java:
 - added "eager" decoding both to JNI wrapper and pure decoder
 - huge speed-up of `DictionaryData` initialization

* Add dictionaryless compressed dictionary

* Fix `sources.lst`

* Fix `sources.lst` and add a note that `libtool` is also required.

* Update setup.py

* Fix `EagerStreamTest`

* Fix BUILD file

* Add missing `libdivsufsort` dependency

* Fix "unused parameter" warning.

c/enc/histogram_inc.h

/* NOLINT(build/header_guard) */
/* Copyright 2013 Google Inc. All Rights Reserved.

   Distributed under MIT license.
   See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/

/* template parameters: Histogram, DATA_SIZE, DataType */

/* A simple container for histograms of data in blocks. */

typedef struct FN(Histogram) {
  uint32_t data_[DATA_SIZE];
  size_t total_count_;
  double bit_cost_;
} FN(Histogram);

static BROTLI_INLINE void FN(HistogramClear)(FN(Histogram)* self) {
  memset(self->data_, 0, sizeof(self->data_));
  self->total_count_ = 0;
  self->bit_cost_ = HUGE_VAL;
}

static BROTLI_INLINE void FN(ClearHistograms)(
    FN(Histogram)* array, size_t length) {
  size_t i;
  for (i = 0; i < length; ++i) FN(HistogramClear)(array + i);
}

static BROTLI_INLINE void FN(HistogramAdd)(FN(Histogram)* self, size_t val) {
  ++self->data_[val];
  ++self->total_count_;
}

static BROTLI_INLINE void FN(HistogramAddVector)(FN(Histogram)* self,
    const DataType* p, size_t n) {
  self->total_count_ += n;
  n += 1;
  while (--n) ++self->data_[*p++];
}

static BROTLI_INLINE void FN(HistogramAddHistogram)(FN(Histogram)* self,
    const FN(Histogram)* v) {
  size_t i;
  self->total_count_ += v->total_count_;
  for (i = 0; i < DATA_SIZE; ++i) {
    self->data_[i] += v->data_[i];
  }
}

static BROTLI_INLINE size_t FN(HistogramDataSize)(void) { return DATA_SIZE; }

kc3-lang/brotli/c/enc/histogram_inc.h

Commit

c/enc/histogram_inc.h