Configuration

The default configuration file is:

config/kube-insight.example.yaml

The normalized processing model is documented in processing-model.md. In short, resourceProfiles.rules is the single resource classification point; filter chains, extractor sets, and retention policies are reusable named components selected by the profile.

Validate it with:

kube-insight config validate
kube-insight config validate --file config/kube-insight.example.yaml
kube-insight config validate --output json

Print the final effective config after embedded defaults, file overlay, environment variables, and CLI flags:

kube-insight --config kube-insight.yaml --log-level debug config show
kube-insight --config kube-insight.yaml config show --output json

config show expands built-in resource profile rules into resourceProfiles.rules and sets resourceProfiles.replaceDefaults: true in the printed output. This makes the output self-contained: humans and agents can see the exact classification order, and the output can be saved as a fully materialized config if a deployment wants to own every profile rule explicitly.

List the built-in processing catalog:

kube-insight config catalog
kube-insight config catalog --output json

This is the supported-discovery path for agents that need to generate or review processing.filters, processing.filterChains, processing.extractors, and processing.extractorSets without reading source code.

Runtime configuration is resolved in this order:

1. Embedded default config.
2. YAML configuration file overlay.
3. Environment variables.
4. Command-line flags.

Storage

storage.driver selects one active backend. The MVP central evidence backend is ClickHouse. The default artifact has no storage-backend suffix, stays small and pure Go, and uses SQLite for local single-file runs. A separate chDB-enabled artifact can use storage.driver: chdb when libchdb.so is installed, and it still supports the SQLite and ClickHouse drivers. Normal default builds fail at runtime with an explicit unavailable-adapter error if storage.driver: chdb is selected.

Use storage.driver: clickhouse with KUBE_INSIGHT_CLICKHOUSE_DSN for the current compose-backed dev watcher. Use storage.driver: sqlite for the default local single-file path and compatibility tests. Use storage.driver: chdb only with the chDB-enabled binary or image.

The Helm chart is optimized for continuous in-cluster collection and defaults to storage.driver: chdb. Release container images are chDB-capable by default. Select storage.driver: sqlite explicitly only for Helm smoke tests, short demos, or temporary runs; do not use SQLite for long-running retained history.

YAML overlay rules:

• Mapping/object fields are deep-merged.
• Scalar fields replace the default value.
• List fields replace the whole default list. They are not appended.
• Empty lists explicitly clear the default list.
• Empty config files load the embedded default config.
• Unknown YAML fields are rejected so typos fail during config validate.

For example, this keeps all default storage, processing, and server settings, but replaces the default resource exclude list with only pods:

collection:
  resources:
    exclude: [pods]

Environment variables use the KUBE_INSIGHT_ prefix plus the YAML path in upper snake case:

KUBE_INSIGHT_INSTANCE_ROLE=writer
KUBE_INSIGHT_LOGGING_LEVEL=info
KUBE_INSIGHT_LOGGING_FORMAT=text
KUBE_INSIGHT_STORAGE_SQLITE_PATH=./kubeinsight.db
KUBE_INSIGHT_COLLECTION_KUBECONFIG=$HOME/.kube/config
KUBE_INSIGHT_COLLECTION_CONTEXTS=staging,prod
KUBE_INSIGHT_COLLECTION_NAMESPACE=payments
KUBE_INSIGHT_COLLECTION_CONCURRENCY=8

Environment list overrides also replace the whole list. Lists of strings use comma-separated values. Structured maps such as processing.filters can be replaced with inline YAML:

KUBE_INSIGHT_PROCESSING_FILTERS='{managed_fields: {type: builtin, action: keep_modified, removePaths: [/metadata/managedFields]}}'

Design References

The config model intentionally follows the operational shape of Prometheus and Loki/Promtail-style tools:

• A YAML file is the primary runtime configuration surface.
• CLI flags are for selecting files, startup role, listen addresses, and targeted overrides.
• Defaults are explicit and inspectable through config validate and config show.
• Invalid config should fail before runtime behavior changes are applied.

Prometheus separates immutable command-line flags from reloadable YAML and only applies a reload when the new config is well formed. Loki documents the same YAML-first approach and has a -print-config-stderr workflow that prints the fully materialized config after built-in defaults, file values, and flags. Promtail is now EOL, so it is useful as historical reference only; current Grafana-family behavior should be checked against Loki/Alloy. The model kube-insight should keep moving toward is: make the effective config easy for humans and agents to inspect, but keep merging rules simple enough that list-heavy sections do not surprise users.

Equivalent flag overrides:

kube-insight --config kube-insight.yaml \
  --role writer \
  --kubeconfig "$HOME/.kube/config" \
  --context staging \
  --namespace payments \
  --log-level debug \
  --log-format json \
  --db ./kubeinsight.db \
  collect ingest --discover-resources

Logging

Logs are written to stderr so stdout can stay machine-readable for JSON command results. The CLI keeps the standard slog call surface, but uses charm.land/log/v2 underneath for more readable terminal output. Supported formats are text, json, and logfmt:

logging:
  level: info
  format: text

Equivalent overrides:

kube-insight --log-level debug --log-format json watch pods
kube-insight --log-format logfmt serve --watch --app

Default info logs are tuned for long-running service mode: lifecycle, batch ingest summaries, watch start/finish, and warnings stay visible; high volume per-object watch events, bookmarks, individual resource list details, and single-object ingest summaries move to debug. Recoverable watch stream disconnects are reported as reconnects instead of warnings; health status keeps them visible as retrying and counts them as unstable until the stream resumes.

Instance Roles

Kube-insight should support multiple application instances against the same backend. Exactly one production instance should normally own discovery/watch/ ingest writes, while other instances only serve API/Web/MCP reads.

instance:
  role: writer

collection:
  enabled: true

server:
  api:
    enabled: false
  web:
    enabled: false
mcp:
  enabled: false

Reader/API instance:

instance:
  role: api

collection:
  enabled: false

server:
  api:
    enabled: true

Supported roles:

• all: local single-process mode; may collect/watch and serve APIs.
• writer: discovery/watch/ingest only; API/Web/MCP listeners disabled.
• api: query/API/Web/MCP only; collection/watch disabled.

Agent chat configuration is server-side. The first Web UI milestone uses a server-owned agent runtime, so provider and model defaults live under server.chat:

server:
  chat:
    enabled: true
    provider: openai
    apiKeyEnv: OPENAI_API_KEY
    baseUrlEnv: OPENAI_BASE_URL
    model: gpt-5.2
    maxIterations: 32

The deprecated openaiApiKeyEnv key is still accepted for compatibility, but new configs should use apiKeyEnv so non-OpenAI-compatible providers can share the same shape later. baseUrlEnv is optional for OpenAI-compatible endpoints and points to the environment variable containing the API base URL; the URL value is not exposed by server info responses. maxIterations caps autonomous agent loop iterations. It defaults to 32; set it higher for long-running investigation sessions or lower in deterministic tests. It is still a guardrail, not the primary way to fix repeated low-signal tool calls.

Agent session retention is also server-side. The API server periodically removes completed retry branches that the chat projection can no longer reach and terminal-run artifact events that the final answer did not cite. Keep this enabled for Web UI development so retries and LLM-selected evidence do not leave unbounded transient rows behind:

server:
  agentRetention:
    enabled: true
    intervalSeconds: 600
    runOnStart: true

The same compaction can be run manually with POST /api/v1/agent/retention/compact; the periodic job uses the default non-dry-run options. In-progress runs are not artifact-compacted because citations may be emitted after artifacts.

Supported Running Modes

Kube-insight supports these operational shapes:

Mode	Command	Writes	Read surfaces	Intended use
One-shot ingest	kube-insight ingest --file/--dir ...	yes	no	Offline samples, CI, fixture import
Watcher only	kube-insight watch [RESOURCE_PATTERN ...]	yes	no	Dedicated collector process
Local agent app	kube-insight serve --app	no	HTTP API, HTTP MCP, Web UI	Built-in Web UI and agent read surfaces
API only	kube-insight serve --api or serve api	no	HTTP API	Read-only query service
Metrics only	kube-insight serve --metrics	no	Prometheus /metrics	Scrape storage, filter, and watch health metrics
MCP stdio	kube-insight serve mcp	no	stdio MCP	Local agent process launch
MCP HTTP	kube-insight serve --app	no	Streamable HTTP /mcp, legacy SSE /sse	Long-running service deployment with the app listener
Web UI	kube-insight serve --app	no	Embedded HTTP Web UI	Agent chat and dashboard surface
All-in-one local	kube-insight serve --watch --app --metrics	yes	HTTP API, HTTP MCP, Metrics, Web UI	Local PoC or small single-instance deployment
Split production	one --watch writer plus N --api/--mcp/--metrics/--webui readers	writer only	readers only	HA/scale-out with one writer owner

In production, prefer one writer instance and multiple read-only instances against the same backend. This avoids duplicate watch streams and duplicate history writes while still allowing API/MCP/WebUI scale-out.

Compact service examples:

kube-insight serve --app --db kubeinsight.db
kube-insight serve --watch --app --db kubeinsight.db
kube-insight serve --watch --app --metrics --db kubeinsight.db
kube-insight serve --watch pods events.events.k8s.io --api --db kubeinsight.db

watch and serve --watch start lightweight SQLite maintenance by default. This periodic task checkpoints/truncates WAL, runs pragma optimize, and runs incremental vacuum when possible. It does not run full VACUUM; use kube-insight db compact for offline compaction after large migrations, retention purges, or duplicate-version pruning.

When processing filters change, existing retained history can be reprocessed with the current effective configuration:

kube-insight db backfill --db kubeinsight.db                       # dry run
kube-insight db backfill --db kubeinsight.db --yes                 # apply
kube-insight db backfill --db kubeinsight.db --yes --batch-objects 200

Backfill only rewrites retained history (versions, blobs, facts, edges, changes, and observation version pointers). It does not rewrite latest_raw_index; that table reflects future latest observations from the watcher. Existing databases can only be backfilled from the JSON that was already retained.

Backfill applies work in object batches instead of one database-wide transaction. Each batch scans complete object histories, applies updates, commits, and releases SQLite’s write lock before the next batch. This lets a watcher keep making progress during large rule migrations, though individual write attempts can still wait briefly while a batch is committing. Tune --batch-objects downward for busier clusters or upward for faster offline maintenance.

Retention is optional and disabled by default. Enable it when a deployment wants bounded local storage instead of full historical proof:

storage:
  retention:
    enabled: true
    maxAgeSeconds: 2592000 # 30 days
    minVersionsPerObject: 2
    filterDecisionMaxAgeSeconds: 604800 # 7 days
    policies:
      standard:
        minVersionsPerObject: 2
      hot:
        maxAgeSeconds: 1209600 # 14 days
        minVersionsPerObject: 5
      large_status:
        maxAgeSeconds: 2592000 # 30 days
        minVersionsPerObject: 2
      topology:
        maxAgeSeconds: 2592000 # 30 days
        minVersionsPerObject: 2
      workload:
        maxAgeSeconds: 2592000 # 30 days
        minVersionsPerObject: 3
      gitops_state:
        maxAgeSeconds: 2592000 # 30 days
        minVersionsPerObject: 3
      control_plane:
        maxAgeSeconds: 7776000 # 90 days
        minVersionsPerObject: 3
      security:
        maxAgeSeconds: 7776000 # 90 days
        minVersionsPerObject: 3
      sensitive_metadata:
        maxAgeSeconds: 1209600 # 14 days
        minVersionsPerObject: 1
      certificate_lifecycle:
        maxAgeSeconds: 7776000 # 90 days
        minVersionsPerObject: 3
      events_short_window:
        maxAgeSeconds: 604800 # 7 days
        minVersionsPerObject: 1
      crd_long_window:
        maxAgeSeconds: 7776000 # 90 days
        minVersionsPerObject: 2
      derived_short_window:
        maxAgeSeconds: 604800 # 7 days
        minVersionsPerObject: 1

The built-in policy names are intentionally descriptive rather than storage-engine specific: hot for Pods, large_status for Nodes, topology for Services and network objects, workload for controllers, gitops_state for Flux and Argo CD resources, control_plane for admission/webhook resources, security for RBAC, sensitive_metadata for Secrets, certificate_lifecycle for cert-manager, and shorter windows for Events and derived reports.

Retention always keeps each object’s latest version and at least minVersionsPerObject retained versions. It deletes expired older versions, their version-scoped facts/edges/changes, unreferenced blobs, and expired filter decision audit rows. Large purges create free pages; run kube-insight db compact after a manual purge when immediate disk reclamation matters. Policies are selected by resourceProfiles.rules[].retentionPolicy; resource classification stays in one place.

Manual retention can be run explicitly:

kube-insight db retention --max-age 720h --min-versions-per-object 2 --yes
kube-insight db retention --profile event_rollup --max-age 168h --yes

Service listen flags:

kube-insight serve --app \
  --listen 0.0.0.0:8090

Release binaries embed the built Web UI assets. Open the UI at the configured app address, for example http://127.0.0.1:8090. With --app, one listener serves Web UI at /, API at /api/v1/*, MCP Streamable HTTP at /mcp, and legacy SSE at /sse; --metrics adds /metrics on the same listener. Use component flags such as --api, --mcp, --webui, or their hidden listener overrides only when intentionally splitting surfaces across ports.

Metrics are exposed in Prometheus text format at /metrics. The metrics server uses the official Prometheus Go client and currently exports storage row counts, storage byte counts, processing profile counts, filter decision results, redaction/removal counters, secret safety counters, ingestion offset lag, and resource stream health.

For production watchers, prefer the cloud/provider direct Kubernetes endpoint over UI/proxy paths such as Rancher when available. Proxy paths can be fine for human kubectl usage while still being less stable for many long-lived WATCH streams.

With no serve component flags, kube-insight serve --config <file> uses the enabled components in the configuration file.

Watch All Resources

For local PoC and broad cluster capture, use discovery-backed collection:

collection:
  enabled: true
  kubeconfig: ""
  useClientGo: true
  contexts: []
  allContexts: false
  namespace: ""
  watch:
    disableHttp2: false
    maxConcurrentStreams: 64
    minBackoffMillis: 500
    maxBackoffMillis: 30000
    streamStartStaggerMillis: 200
    queuedRelistIntervalSeconds: 300
    streamRotationSeconds: 900
  resources:
    all: true
    include: []
    exclude:
      # Prefer events.events.k8s.io as the canonical Event source; core v1
      # events usually duplicates the same evidence.
      - events
      - leases.coordination.k8s.io
      - "*policyreports.wgpolicyk8s.io"
      - "*ephemeralreports.reports.kyverno.io"

Watch tuning defaults are conservative for clusters with many aggregated APIs or CRDs:

• disableHttp2: false keeps Kubernetes’ normal HTTP/2 behavior. Set it to true only for a cluster or proxy proven to behave better with HTTP/1.1.
• maxConcurrentStreams limits active long-running WATCH streams. Initial LIST coverage still runs for every selected resource; lower-priority watch streams wait instead of overloading the API front door.
• streamStartStaggerMillis spreads initial watch stream creation so reconnects and bookmarks do not arrive as one burst.
• queuedRelistIntervalSeconds periodically refreshes resources that completed their initial LIST but are still waiting for a long-running WATCH slot. Set it to 0 to disable queued relists.
• streamRotationSeconds bounds each active WATCH slot before the worker releases it and re-enters the queue. This gives lower-priority or quiet resources periodic WATCH windows without permanently increasing maxConcurrentStreams; set it to 0 to disable rotation.
• minBackoffMillis and maxBackoffMillis control exponential reconnect backoff with jitter.

The default excludes keep the high-value evidence path small: canonical Events come from events.events.k8s.io, Leases are dropped, and derived policy/report resources are skipped unless explicitly included for policy-engine debugging. Skipped resources are hidden from db resources health by default and can be shown with --include-skipped.

Equivalent CLI paths:

kube-insight dev collect ingest --client-go --discover-resources --db kubeinsight.db
kube-insight watch
kube-insight watch pods services
kube-insight watch 'v1/*' 'apps/v1/*'
kube-insight watch '*.cert-manager.io' 'gateway.networking.k8s.io/*/*'

Every config field named resources uses the same resource-pattern forms:

• bare resource: pods
• qualified resource: deployments.apps
• GVR: apps/v1/deployments
• glob over any of those forms: *.reports.kyverno.io, apps/v1/*, cert-manager.io/*/*

Discovery writes api_resources, and later ingestion uses that table as the authoritative GVR, scope, and kind mapping.

Without --db, watch and storage management commands use ./kubeinsight.db. The watcher resolves a stable cluster ID from the Kubernetes cluster identity and stores the kubeconfig context as metadata, so context renames do not create a new cluster record.

Stored clusters can be inspected and removed with:

kube-insight db clusters
kube-insight db clusters delete <cluster-id> --yes

Resource Profiles

Resource profiles classify resources into processing lanes. They decide the retention policy, filter chain, extractor set, compaction strategy, priority, stream queue order, and whether a low-value resource is disabled by default.

Custom profile rules are matched before built-in defaults:

resourceProfiles:
  defaults:
    enabled: true
    retentionPolicy: standard
    filterChain: default
    extractorSet: generic
    compactionStrategy: full_json
    priority: normal
    maxEventBuffer: 256
  replaceDefaults: false
  rules:
    - name: custom_gateway
      resources: [gateways.gateway.networking.k8s.io]
      retentionPolicy: standard
      extractorSet: service_topology
      priority: high

Built-in profile rules include high-value defaults for Pods, Nodes, Events, EndpointSlices, Services, workload controllers, admission webhooks, RBAC, cert-manager, CRDs, GitOps controllers, kagent ecosystem resources, and common observability CRDs from Prometheus Operator, VictoriaMetrics Operator, and Grafana Agent/Alloy-style monitoring APIs. The GitOps rules cover Flux toolkit resources (source.toolkit.fluxcd.io, kustomize.toolkit.fluxcd.io, helm.toolkit.fluxcd.io, image.toolkit.fluxcd.io, and notification.toolkit.fluxcd.io) plus Argo CD Application, AppProject, and ApplicationSet resources under argoproj.io.

Rule matching supports:

• resources: bare resource (pods), qualified resource (events.events.k8s.io), GVR (events.k8s.io/v1/events), or glob (*.cert-manager.io, gateway.networking.k8s.io/*/*).
• groups: API groups such as apps or cert-manager.io.
• kinds: Kubernetes kinds such as Pod or Certificate.

When multiple matchers are set on one rule they are combined with AND. Values inside the same matcher list are OR. Set enabled: false to keep a resource discoverable in resource_processing_profiles while making watch scheduling treat it as disabled/low value. Set replaceDefaults: true only when the config fully defines every resource profile rule the deployment needs.

At ingest time, the selected resource profile chooses the active filterChain and extractorSet. filterChain: default runs the configured default filter chain; filterChain: none skips filters for that profile. extractorSet: none skips evidence extraction, and named sets run the configured extractor components when their guard also matches.

Processing

Processing contains reusable component libraries. It should not duplicate resource classification; use guard only as a safety boundary for components that must not run on the wrong resource.

processing:
  filterChains:
    default: [managed_fields, resource_version, metadata_generation, status_condition_set]
    secret_metadata_only: [managed_fields, resource_version, secret_metadata_only]
    none: []
  filters:
    secret_metadata_only:
      type: builtin
      enabled: true
      action: keep_modified
      guard:
        resources: [secrets]
      removePaths: [/data, /stringData]
      keepSecretKeys: true

Component guard blocks support resources, kinds, namespaces, and names. Resource and kind guards protect by type; namespace and name guards protect object-specific normalizers:

guard:
  resources: [configmaps]
  namespaces: [kube-system]
  names: [cluster-autoscaler-status]

The current runtime supports these built-in filters:

Filter	Required action	Notes
managed_fields	keep_modified	Removes /metadata/managedFields when removePaths is set.
resource_version	keep_modified	Removes /metadata/resourceVersion from retained document hashes.
metadata_generation	keep_modified	Removes derived /metadata/generation; spec/status content still carries the actual change.
status_condition_set	keep_modified	Removes condition timestamp fields and sorts conditions by type so order-only churn does not create retained versions.
status_condition_timestamps	keep_modified	Removes condition timestamp churn while preserving condition status/reason/message.
leader_election_configmap	keep_modified	Removes legacy leader-election annotation churn from ConfigMaps.
cluster_autoscaler_status	keep_modified	Normalizes kube-system/cluster-autoscaler-status probe timestamps and node group order.
gke_webhook_heartbeat	keep_modified	Normalizes kube-system/gke-common-webhook-heartbeat timestamp-keyed heartbeat entries into host/version evidence.
event_series	keep_modified	Removes repeated Event count/last-seen timestamp churn while preserving the event proof.
secret_metadata_only	keep_modified	Removes Secret payload values while preserving metadata and keys.
lease_skip	discard_resource	Drops Lease resources from retained history.
report_skip	discard_resource	Drops low-value derived report resources.

For built-ins, enabled, chain order, guard, action, and removePaths are applied on the write path before retained document hashing. action must match the built-in action; config validate fails fast when a built-in name or action is wrong. Use enabled: false to disable a built-in filter.

Supported first-version actions:

• keep
• keep_modified
• discard_change
• discard_resource

Future plugin path:

processing:
  filters:
    custom_noise_filter:
      type: goja
      enabled: true
      action: keep_modified
      script: ./plugins/filters/noise.js

The Goja plugin contract should receive an observation and return the same auditable decision shape as built-in filters.

Extractors can be enabled or disabled and can set a guard over resources/kinds. Their resource scopes use the same resource-pattern forms as collection and resource profiles. The selected resource profile’s extractorSet decides which configured extractor set runs for a resource.

The current runtime supports these built-in extractors:

• reference
• pod
• node
• event
• endpointslice
• service

processing:
  extractorSets:
    generic: [reference]
    pod: [reference, pod]
    service_topology: [reference, service]
    none: []

  extractors:
    pod:
      type: builtin
      enabled: true
      guard:
        resources: [pods]
    service:
      type: builtin
      enabled: true
      guard:
        resources: [services]

Future plugin path:

processing:
  extractors:
    custom_rollout_facts:
      type: goja
      enabled: true
      guard:
        resources: [deployments.apps]
      script: ./plugins/extractors/rollout.js

Extractor plugins should emit the same logical outputs as built-ins: facts, edges, changes, and optional summaries. Versions remain proof.

Storage

SQLite is the default local single-file backend in the default pure-Go artifact:

storage:
  driver: sqlite
  sqlite:
    path: kubeinsight.db

Use SQLite only for tests, short demos, CI fixtures, or temporary local runs. Do not use it for long-running retained-history deployments; use chDB or ClickHouse for continuous collection.

ClickHouse can be selected as the primary append-only evidence backend:

storage:
  driver: clickhouse
  clickhouse:
    initOnStart: true
    dsnEnv: KUBE_INSIGHT_CLICKHOUSE_DSN
    database: kube_insight

When storage.driver: clickhouse is selected, ingest, watch, and serve --watch write retained observations, versions, facts, edges, changes, and object aliases over the ClickHouse HTTP interface. The HTTP API and serve --metrics also read from ClickHouse for the MVP read surface: schema, read-only SQL, resource health, object history, evidence search, topology, and service investigation. initOnStart applies the idempotent MVP DDL before writes. Writes are client-batched with batchSize/flushIntervalMillis; pending ingestion offsets are coalesced per resource and event before flush because they model current watch state rather than proof history. When asyncInsert is true, HTTP inserts add ClickHouse async_insert=1 and wait_for_async_insert=1 so the server can merge small inserts without hiding write failures from the caller.

Cold tiering is opt-in. The default example leaves coldVolume empty and coldAfterSeconds: 0; set them only after the ClickHouse server has a matching storage policy, for example an S3-compatible cold volume.

chDB is the embedded local ClickHouse-compatible backend in the chDB-enabled artifact:

storage:
  driver: chdb
  chdb:
    path: kubeinsight.chdb
    database: kube_insight

Use config/kube-insight.chdb.example.yaml with bin/kube-insight-chdb, the chDB-enabled release artifact, or the chDB-enabled image. The chDB-enabled binary needs libchdb.so discoverable through the system dynamic linker, LD_LIBRARY_PATH, or CHDB_LIB_PATH. The default binary intentionally keeps the unavailable placeholder so normal installs do not carry the large dynamic chDB runtime. PostgreSQL and CockroachDB remain possible metadata/control-plane candidates, but they are not the MVP evidence-storage path.