Skip to main content

EC2 HTTP latency

EC2 HTTP latency disrupts the state of infrastructure resources. This fault induces HTTP chaos on an AWS EC2 instance using the Amazon SSM Run command, carried out using SSM Docs that is in-built in the fault.

  • It injects HTTP response latency to the service whose port is specified using TARGET_SERVICE_PORT environment variable by starting the proxy server and redirecting the traffic through the proxy server.
  • It introduces HTTP latency chaos on the EC2 instance using an SSM doc for a certain duration.

EC2 HTTP Latency

Use cases​

EC2 HTTP latency:

  • Delays the network connectivity from the VM to the target hosts.
  • Simulates latency to specific API services for (or from) a given microservice.
  • Simulates a slow response on specific third party (or dependent) components (or services).

Prerequisites​

  • Kubernetes >= 1.17
  • SSM agent is installed and running on the target EC2 instance.
  • You can pass the VM credentials as secrets or as a ChaosEngine environment variable.
  • The EC2 instance should be in a healthy state.
  • The Kubernetes secret should have the AWS Access Key ID and Secret Access Key credentials in the CHAOS_NAMESPACE. Below is the sample secret file: 
    apiVersion: v1
    kind: Secret
    metadata:
      name: cloud-secret
    type: Opaque
    stringData:
      cloud_config.yml: |-
        # Add the cloud AWS credentials respectively
        [default]
        aws_access_key_id = XXXXXXXXXXXXXXXXXXX
        aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tip

HCE recommends that you use the same secret name, that is, cloud-secret. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.

Below is an example AWS policy to execute the fault.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:SendCommand",
                "ssm:CancelCommand",
                "ssm:CreateDocument",
                "ssm:DeleteDocument",
                "ssm:GetCommandInvocation",          
                "ssm:UpdateInstanceInformation",
                "ssm:DescribeInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
note

Mandatory tunables​

Tunable Description Notes
EC2_INSTANCE_ID ID of the target EC2 instance For example, i-044d3cb4b03b8af1f. For more information, go to EC2 instance ID.
REGION The AWS region ID where the EC2 instance has been created. For example, us-east-1.
LATENCY Provide latency to be added to request in milliseconds. For example, 1000. For more information, go to latency.
TARGET_SERVICE_PORT Port of the service to target Default: port 80. For more information, go to target service port.

Optional tunables​

Tunable Description Notes
TOTAL_CHAOS_DURATION Duration that you specify, through which chaos is injected into the target resource (in seconds). Default: 30 s. For more information, go to duration of the chaos.
CHAOS_INTERVAL Time interval between two successive instance terminations (in seconds). Default: 30 s. For more information, go to chaos interval.
AWS_SHARED_CREDENTIALS_FILE Provide the path for AWS secret credentials. Default: /tmp/cloud_config.yml.
SEQUENCE It defines a sequence of chaos execution for multiple instances. Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.
RAMP_TIME Period to wait before and after injection of chaos (in seconds). For example, 30 s. For more information, go to ramp time.
INSTALL_DEPENDENCY Specify the dependencies to be installed to run the network chaos. If the dependency exists, it can be turned off. If the dependency already exists, you can turn it off. Default: True.
PROXY_PORT Port where the proxy listens to requests. Default: 20000. For more information, go to proxy port.
TOXICITY Percentage of HTTP requests affected. Default: 100. For more information, go to toxicity.
NETWORK_INTERFACE Network interface used for the proxy. Default: eth0. For more information, go to netowkr interface.

Target service port​

Port of the target service. Tune it by using the TARGET_SERVICE_PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Proxy port​

Port where the proxy server listens to the requests. Tune it by using the PROXY_PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

# provide the port for proxy server
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the port for proxy server
        - name: PROXY_PORT
          value: '8080'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Latency​

Delay added to the HTTP request. Tune it by using the LATENCY environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the latency value
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the latency value
        - name: LATENCY
          value: '2000'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Toxicity​

Percentage of the total number of HTTP requests that are affected. Tune it by using the TOXICITY environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the toxicity
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # toxicity is the probability of the request to be affected
        # provide the percentage value in the range of 0-100
        # 0 means no request will be affected and 100 means all request will be affected
        - name: TOXICITY
          value: "100"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Network interface​

Network interface used for the proxy. Tune it by using the NETWORK_INTERFACE environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the network interface for proxy
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the network interface for proxy
        - name: NETWORK_INTERFACE
          value: "eth0"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: '80'