Skip to main content

Azure instance memory hog

Azure instance memory hog disrupts the state of infrastructure resources.

  • It induces stress on the Azure Instance using the Azure Run command. The Azure Run command is executed using the in-built bash scripts within the fault.
  • It utilizes memory in excess on the Azure Instance using the bash script for a specific duration.

Azure Instance Memory Hog

Use cases

Azure instance memory hog:

  • Determines the resilience of an Azure instance when memory resources are unexpectedly utilized in excess.
  • Determines how Azure scales the memory to maintain the application when resources are consumed heavily.
  • Simulates the situation of memory leaks in the deployment of microservices.
  • Simulates a slowed application caused by lack of memory.
  • Simulates noisy neighbour problems due to hogging.
  • Verifies pod priority and QoS setting for eviction purposes.
  • Verifies application restarts on OOM (out of memory) kills.
note
  • Kubernetes >= 1.17 is required to execute this fault.
  • Azure Run Command agent is installed and running in the target Azure instance.
  • Azure instance should be in a healthy state.
  • Use Azure file-based authentication to connect to the instance using Azure GO SDK. To generate the auth file ,run az ad sp create-for-rbac --sdk-auth > azure.auth Azure CLI command.
  • Kubernetes secret should contain the auth file created in the previous step in the CHAOS_NAMESPACE. Below is a sample secret file:
apiVersion: v1
kind: Secret
metadata:
name: cloud-secret
type: Opaque
stringData:
azure.auth: |-
{
"clientId": "XXXXXXXXX",
"clientSecret": "XXXXXXXXX",
"subscriptionId": "XXXXXXXXX",
"tenantId": "XXXXXXXXX",
"activeDirectoryEndpointUrl": "XXXXXXXXX",
"resourceManagerEndpointUrl": "XXXXXXXXX",
"activeDirectoryGraphResourceId": "XXXXXXXXX",
"sqlManagementEndpointUrl": "XXXXXXXXX",
"galleryEndpointUrl": "XXXXXXXXX",
"managementEndpointUrl": "XXXXXXXXX"
}
  • If you change the secret key name from azure.auth to a new name, ensure that you update the AZURE_AUTH_LOCATION environment variable in the chaos experiment with the new name.

Fault tunables

Mandatory fields

Variables Description Notes
AZURE_INSTANCE_NAMES Names of the target Azure instances. Multiple values can be provided as a comma-separated string. For example, instance-1,instance-2. For more information, go to stop instance by name.
RESOURCE_GROUP The Azure Resource Group name where the instances will be created. All the instances must be from the same resource group. For more information, go to resource group field in the YAML file.

Optional fields

Variables Description Notes
TOTAL_CHAOS_DURATION Duration that you specify, through which chaos is injected into the target resource (in seconds). Defaults to 30s. For more information, go to duration of the chaos.
CHAOS_INTERVAL Time interval between two successive container kills (in seconds). Defaults to 60s. For more information, go to chaos interval.
AZURE_AUTH_LOCATION Name of the Azure secret credentials files. Defaults to azure.auth.
SCALE_SET Check if the instance is a part of Scale Set. Defaults to disable. Also supports enable. For more information, go to scale set instances.
MEMORY_CONSUMPTION Amount of memory to be consumed in the Azure instance (in megabytes). Defaults to 500 MB. For more information, go to memory consumption in megabytes.
MEMORY_PERCENTAGE Amount of memory to be consumed in the Azure instance (in percentage). Defaults to 0. For more information, go to memory consumption in percentage.
NUMBER_OF_WORKERS Number of workers used to run the stress process. Defaults to 1. For more information, go to multiple workers.
SEQUENCE Sequence of chaos execution for multiple target instances. Defaults to parallel. Also supports serial sequence. For more information, go to sequence of chaos execution.
RAMP_TIME Period to wait before and after injecting chaos (in seconds). For example, 30s. For more information, go to ramp time.

Memory consumption in megabytes

It specifies the memory utilised (in MB) on the Azure instance. Tune it by using the MEMORY_CONSUMPTION environment variable.

Use the following example to tune it:

# memory in mb to utilize
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: azure-instance-memory-hog
spec:
components:
env:
- name: MEMORY_CONSUMPTION
VALUE: '1024'
# name of the Azure instance
- name: AZURE_INSTANCE_NAMES
value: 'instance-1'
# resource group for the Azure instance
- name: RESOURCE_GROUP
value: 'rg-azure'

Memory consumption in percentage

It specifies the memory utilised (in percentage) on the Azure instance. Tune it by using the MEMORY_PERCENTAGE environment variable.

Use the following example to tune it:

# memory percentage to utilize
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: azure-instance-memory-hog
spec:
components:
env:
- name: MEMORY_PERCENTAGE
VALUE: '50'
# name of the Azure instance
- name: AZURE_INSTANCE_NAMES
value: 'instance-1'
# resource group for the Azure instance
- name: RESOURCE_GROUP
value: 'rg-azure'

Multiple Azure instances

It specifies comma-separated Azure instance names that are subject to chaos in a single run. Tune it by using the AZURE_INSTANCE_NAMES environment variable.

Use the following example to tune it:

# mutilple instance targets
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: azure-instance-memory-hog
spec:
components:
env:
# names of the Azure instance
- name: AZURE_INSTANCE_NAMES
value: 'instance-1,instance-2'
# resource group for the Azure instance
- name: RESOURCE_GROUP
value: 'rg-azure'

Multiple workers

It specifies the CPU threads that are run to spike the memory utilisation. As a consequence, this increases the memory consumption. Tune it by using the NUMBER_OF_WORKERS environment variable..

Use the following example to tune this:

# multiple workers to utilize resources
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: azure-instance-memory-hog
spec:
components:
env:
- name: NUMBER_OF_WORKERS
VALUE: '3'
# name of the Azure instance
- name: AZURE_INSTANCE_NAMES
value: 'instance-1'
# resource group for the Azure instance
- name: RESOURCE_GROUP
value: 'rg-azure'