[feature] simpler cache size configuration (#2051)

* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>

1 files changed, 48 insertions, 0 deletions

diff --git a/vendor/github.com/DmitriyVTitov/size/README.md b/vendor/github.com/DmitriyVTitov/size/README.md
new file mode 100644
index 000000000..215a749cc
--- /dev/null
+++ b/vendor/github.com/DmitriyVTitov/size/README.md
@@ -0,0 +1,48 @@
+# size - calculates variable's memory consumption at runtime
+
+### Part of the [Transflow Project](http://transflow.ru/)
+
+Sometimes you may need a tool to measure the size of object in your Go program at runtime. This package makes an attempt to do so. Package based on `binary.Size()` from Go standard library.
+
+Features:
+- supports non-fixed size variables and struct fields: `struct`, `int`, `slice`, `string`, `map`;
+- supports complex types including structs with non-fixed size fields;
+- supports all basic types (numbers, bool);
+- supports `chan` and `interface`;
+- supports pointers;
+- implements infinite recursion detection (i.e. pointer inside struct field references to parent struct).
+
+### Usage example
+
+```
+package main
+
+import (
+	"fmt"
+
+	// Use latest tag.
+	"github.com/DmitriyVTitov/size"
+)
+
+func main() {
+	a := struct {
+		a int
+		b string
+		c bool
+		d int32
+		e []byte
+		f [3]int64
+	}{
+		a: 10,                    // 8 bytes
+		b: "Text",                // 16 (string itself) + 4 = 20 bytes
+		c: true,                  // 1 byte
+		d: 25,                    // 4 bytes
+		e: []byte{'c', 'd', 'e'}, // 24 (slice itself) + 3 = 27 bytes
+		f: [3]int64{1, 2, 3},     // 3 * 8 = 24 bytes
+	} // 84 + 3 (padding) = 87 bytes
+
+	fmt.Println(size.Of(a))
+}
+
+// Output: 87
+```