Disk fill

Introduction

Disk fill is a Kubernetes pod-level chaos fault that applies disk stress by filling the pod's ephemeral storage on a node. This fault evicts the application pod if its capacity exceeds the pod's ephemeral storage limit.

Use cases

Disk fill:

• Tests the ephemeral storage limits and ensures that the parameters are sufficient.
• Determines the resilience of the application to unexpected storage exhaustions.
• Evaluates the application's resilience to disk stress or replica evictions.
• Simulates the filled data mount points.
• Verifies file system performance, and thin-provisioning support.
• Verifies space reclamation (UNMAP) capabilities on storage.

note

• Kubernetes > 1.16 is required to execute this fault.
• The application pods should be in the running before and after injecting chaos.
• Appropriate Ephemeral storage requests and limits should be set for the application before running the fault. An example specification is shown below:

    apiVersion: v1
    kind: Pod
    metadata:
      name: frontend
    spec:
      containers:
      - name: db
        image: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "password"
        resources:
          requests:
            ephemeral-storage: "2Gi"
          limits:
            ephemeral-storage: "4Gi"
      - name: wp
        image: wordpress
        resources:
          requests:
            ephemeral-storage: "2Gi"
          limits:
            ephemeral-storage: "4Gi"

Fault tunables

Mandatory tunables

Tunable	Description	Notes
FILL_PERCENTAGE	Percentage to fill the ephemeral storage limit. This limit is set in the target pod.	It can be set to more than 100 which force evicts the pod. For more information, go to disk fill percentage
EPHEMERAL_STORAGE_MEBIBYTES	Ephemeral storage required to be filled (in mebibytes). It is mutually exclusive with FILL_PERCENTAGE environment variable. If both are provided, FILL_PERCENTAGE takes precedence.	For more information, go to disk fill mebibytes

Optional tunables

Tunable	Description	Notes
TARGET_CONTAINER	Name of the container subject to disk fill.	If it is not provided, the first container in the target pod will be subject to chaos. For more information, go to kill specific container
CONTAINER_PATH	Storage location of containers.	Default: /var/lib/docker/containers. For more information, go to container path
TOTAL_CHAOS_DURATION	Duration for which to insert chaos (in seconds).	Default: 60 s. For more information, go to duration of the chaos
TARGET_PODS	Comma-separated list of application pod names subject to disk fill chaos.	If not provided, the fault selects the target pods randomly based on provided appLabels. For more information, go to target specific pods
DATA_BLOCK_SIZE	Data block size used to fill the disk (in KB).	Default: 256 KB. For more information, go to data block size
PODS_AFFECTED_PERC	Percentage of total pods to target. Provide numeric values.	Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage
RAMP_TIME	Period to wait before injecting chaos (in seconds).	For example, 30 s. For more information, go to ramp time
SEQUENCE	Sequence of chaos execution for multiple target pods.	Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution

Disk fill percentage

Percentage of ephemeral storage limit to be filled at resource.limits.ephemeral-storage within the target application. Tune it by using the FILL_PERCENTAGE environment variable.

The following YAML snippet illustrates the use of this environment variable:

## percentage of ephemeral storage limit specified at `resource.limits.ephemeral-storage` inside target application
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            ## percentage of ephemeral storage limit, which needs to be filled
            - name: FILL_PERCENTAGE
              value: "80" # in percentage
            - name: TOTAL_CHAOS_DURATION
              VALUE: "60"

Disk fill mebibytes

Ephemeral storage required to be filled in the target pod. Tune it by using the EPHEMERAL_STORAGE_MEBIBYTES environment variable.

EPHEMERAL_STORAGE_MEBIBYTES is mutually exclusive with the FILL_PERCENTAGE environment variable. If FILL_PERCENTAGE environment variable is set, the fault uses FILL_PERCENTAGE for the fill. Otherwise, the dault fills the ephemeral storage based on EPHEMERAL_STORAGE_MEBIBYTES environment variable.

The following YAML snippet illustrates the use of this environment variable:

# ephemeral storage which needs to fill in will application
# if ephemeral-storage limits is not specified inside target application
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            ## ephemeral storage size, which needs to be filled
            - name: EPHEMERAL_STORAGE_MEBIBYTES
              value: "256" #in MiBi
            - name: TOTAL_CHAOS_DURATION
              VALUE: "60"

Data block size

Size of the data block required to fill the ephemeral storage of the target pod. It is in terms of KB. The default value of DATA_BLOCK_SIZE is 256 KB. Tune it by using the DATA_BLOCK_SIZE environment variable.

The following YAML snippet illustrates the use of this environment variable:

# size of the data block used to fill the disk
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            ## size of data block used to fill the disk
            - name: DATA_BLOCK_SIZE
              value: "256" #in KB
            - name: TOTAL_CHAOS_DURATION
              VALUE: "60"

Container path

Storage location of the containers inside the host (node or VM). Tune it by using the CONTAINER_PATH environment variable.

The following YAML snippet illustrates the use of this environment variable:

# path inside node/vm where containers are present
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            # storage location of the containers
            - name: CONTAINER_PATH
              value: "/var/lib/docker/containers"
            - name: TOTAL_CHAOS_DURATION
              VALUE: "60"