How to configure log collection buffers

Vector agent buffers and delivery to the platform log storage.

The shturval-log-collector agent on each node writes logs to its Vector disk buffer before sending them to the platform. Logs are also stored in the container log directory on the node disk and in VictoriaLogs.

Buffer size and the way logs are sent are set by default. You may need to change the defaults when:

• the receiverThe VictoriaLogs instance in the management cluster. After the buffer, the Vector agent sends logs to it for storage. is unavailable and the buffer is already full;
• after VictoriaLogs recovers, accumulated logs from several nodes arrive at once and overload VictoriaLogs;
• you need a buffer larger than 256 MiB per receiverOne of the five platform log receivers (platform sinks): auditd, journald, k8s_audit, k8s_events, or k8s_logs. Each has its own disk buffer.. Before that, limit /var/lib/vector on the node: give it a separate disk partition or a disk quota, so a larger buffer does not fill the whole node disk.

To change how long logs are kept, see How to set up log rotation. To turn log sending off, see How to manage sending logs in the platform. To configure a buffer for external storage, see How to set up log collection and redirection.

Buffer parameters #

By default, parameters are set for each buffer of all receivers (sinks): auditd, journald, k8s_audit, k8s_events, or k8s_logs:

The sum of the five default buffers is about 1.25 GiB per node. /var/lib/vector needs at least 1.25 GiB free. The buffer has no separate retention period. The agent deletes a buffer segment after it has fully sent it to the receiver. Writes to disk happen about every 500 ms. If writing the buffer to disk fails, the Vector process stops.

Increase buffer.max_size only when /var/lib/vector is already on a separate partition or quota. Otherwise a larger buffer fills the node disk sooner.

The full parameter list is in the Vector buffering model.

If the buffer is full #

By default the internal metrics source (spec.agent.internalMetrics on the Vector resource) is disabled. When the source is enabled, fullness can be seen from buffer metrics:

Without these metrics, buffer fullness can be judged from the node disk: the /var/lib/vector and /var/log/pods directories.

You can set when_full separately for each receiver. To change when_full on one buffer object:

1. Prepare a ShturvalServicePatch manifest for shturval-log-collector. The module must be in Auto mode. In Manual mode the SSP may not apply. The difference between the modes is on Collection and processing of logs (ClusterVectorPipeline).
2. Load the manifest in the cluster where the module runs, through Importing manifests: Administration → Cluster Actions.
3. Wait until it is applied. On the local log collection module page the service status changes to Patched, and the SSP appears on the Applied ShturvalServicePatch tab.

ShturvalServicePatch example with block mode #

The default mode is block. With block, the sink waits for space in the buffer and applies backpressure to the source. New events are not dropped. While the buffer is full, the kubernetes_logs read loop does not reach closing of files that are already deleted. kubelet deletes a rotated file, and the agent still holds it open and does not get to close it. Space on the node disk is not freed. If /var/log/pods is not limited, the node disk fills up: containers cannot write files, and new pods do not start. A full buffer on the other four receivers also stops reading of their own source, but /var/log/pods is filled by container log collection.

To keep events again after drop_newest, set block back. Example for k8s_logs. For auditd, journald, k8s_audit, and k8s_events set it the same way, inside their own platform_logs_sink.

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: shturval-log-collector-buffer-block
spec:
  shturvalServiceConfigName: shturval-log-collector
  customvalues:
    shturval:
      k8s_logs:
        platform_logs_sink:
          buffer:
            when_full: block

While a buffer in block mode is full, the 60-second pause does not start. The agent closes the deleted file only when there is space in the buffer or the Vector process exits.

ShturvalServicePatch example with drop_newest mode #

Enable drop_newest only for receivers where losing new events is acceptable. While the buffer is completely full, the agent drops the new event and keeps reading the source. For container logs, a deleted file is closed after the 60-second pause, and space on the node disk is freed. Logs that did not enter the buffer will not appear in platform storage.

ShturvalServicePatch example with drop_newest for all receivers.

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: shturval-log-collector-buffer-drop-newest
spec:
  shturvalServiceConfigName: shturval-log-collector
  customvalues:
    shturval:
      auditd:
        platform_logs_sink:
          buffer:
            when_full: drop_newest
      journald:
        platform_logs_sink:
          buffer:
            when_full: drop_newest
      k8s_audit:
        platform_logs_sink:
          buffer:
            when_full: drop_newest
      k8s_events:
        platform_logs_sink:
          buffer:
            when_full: drop_newest
      k8s_logs:
        platform_logs_sink:
          buffer:
            when_full: drop_newest

Discarded events show up in the Vector metric vector_buffer_discarded_events_total if the agent has internal metrics enabled.

Request concurrency #

By default, for all sinks the number of concurrent requests adapts to responses: request.concurrency: adaptive. When adaptation does not fit a particular sink, you can change it by applying an SSP for shturval-log-collector. The limit of concurrent HTTP requests to the sink starts from a lower value. When requests succeed without a growth in response latency, the limit increases. When the sink rejects requests because of load, for example HTTP 429, or when latency rises noticeably, the limit decreases, but it is not set below 1.

After the sink recovers, a preset maximum of parallel requests is not applied immediately. Accumulated events leave gradually. While sinks are idle, disk is not freed: the buffer fills up to max_size, and then the when_full mode policy applies.

A fixed number, for example request.concurrency: 10, turns adaptation off for this sink.

By default request.rate_limit_duration_secs and request.rate_limit_num are not set. The window and the number of requests in it set how often the sink sends HTTP requests, separately from the number of concurrent requests. According to the Vector HTTP sink documentation, this limit applies both with concurrency: adaptive and with a fixed number. The example from the documentation is a 1 second window and 10 requests, that is no more than 10 requests per second.

How to check the node disk, enable a project quota, and align kubelet rotation is described in Container logs fill the node disk.