Skip to main content

EC2 HTTP reset peer

EC2 HTTP reset peer injects HTTP reset on the service whose port is specified using the TARGET_SERVICE_PORT environment variable. This fault stops the outgoing HTTP requests by resetting the TCP connection for the requests.

EC2 HTTP Reset Peer

Use cases​

EC2 HTTP reset peer:

  • Verifies connection timeout by simulating premature connection loss (firewall issues or other issues) between microservices.
  • Simulates connection resets due to resource limitations on the server side like out of memory server (or process killed or overload on the server due to a high amount of traffic).
  • Determines the application's resilience to a lossy (or flaky) HTTP connection.

Prerequisites​

  • Kubernetes >= 1.17
  • The EC2 instance should be in a healthy state.
  • SSM agent is installed and running in the target EC2 instance.
  • You can pass the VM credentials as secrets or as an chaosengine environment variable.
  • The Kubernetes secret should have the AWS Access Key ID and Secret Access Key credentials in the CHAOS_NAMESPACE. Below is the sample secret file: 
    apiVersion: v1
    kind: Secret
    metadata:
      name: cloud-secret
    type: Opaque
    stringData:
      cloud_config.yml: |-
        # Add the cloud AWS credentials respectively
        [default]
        aws_access_key_id = XXXXXXXXXXXXXXXXXXX
        aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tip

HCE recommends that you use the same secret name, that is, cloud-secret. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.

Below is an example AWS policy to execute the fault.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:SendCommand",
                "ssm:CancelCommand",
                "ssm:CreateDocument",
                "ssm:DeleteDocument",
                "ssm:GetCommandInvocation",          
                "ssm:UpdateInstanceInformation",
                "ssm:DescribeInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
note

Mandatory tunables​

Tunable Description Notes
EC2_INSTANCE_ID ID of the target EC2 instance. For example, i-044d3cb4b03b8af1f. For more information, go to EC2 instance ID.
REGION The AWS region ID where the EC2 instance has been created. For example, us-east-1.
RESET_TIMEOUT Duration after which the connection is reset. Default: 0. For more information, go to reset timeout.
TARGET_SERVICE_PORT Port of the service to target. Default: port 80. For more information, go to target service port.

Optional tunables​

Tunable Description Notes
TOTAL_CHAOS_DURATION Duration that you specify, through which chaos is injected into the target resource (in seconds). Default: 30 s. For more information, go to duration of the chaos.
CHAOS_INTERVAL Time interval between two successive instance terminations (in seconds). Default: 30 s. For more information, go to chaos interval.
AWS_SHARED_CREDENTIALS_FILE Provide the path for AWS secret credentials. Default: /tmp/cloud_config.yml.
SEQUENCE It defines the sequence of chaos execution for multiple instances. Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.
RAMP_TIME Period to wait before and after injection of chaos (in seconds). For example, 30. For more information, go to ramp time.
INSTALL_DEPENDENCY Select to install dependencies used to run the network chaos. It can be either True or False. If the dependency already exists, you can turn it off. Defaults to True.
PROXY_PORT Port where the proxy will be listening to requests. Default: 20000. For more information, go to proxy port.
TOXICITY Percentage of HTTP requests to be affected. Default: 100. For more information, go to toxicity.
NETWORK_INTERFACE Network interface to be used for the proxy. Default: eth0. For more information, go to network interface.

Target service port​

Port of the target service. Tune it by using the TARGET_SERVICE_PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Proxy port​

Port where the proxy server listens for requests. Tune it by using the PROXY_PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

# provide the port for proxy server
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # provide the port for proxy server
        - name: PROXY_PORT
          value: '8080'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Reset timeout​

Duration after which the connection is reset, that is, the value added to the HTTP request. Tune it by using the RESET_TIMEOUT environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the reset timeout value
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # reset timeout specifies after how much duration to reset the connection
        - name: RESET_TIMEOUT #in ms
          value: '2000'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Toxicity​

Percentage of the total number of HTTP requests that are affected. Tune it by using the TOXICITY environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the toxicity
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # toxicity is the probability of the request that is affected
        # provide the percentage value in the range of 0-100
        # 0 means no request will be affected and 100 means all requests will be affected
        - name: TOXICITY
          value: "100"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Network interface​

Network interface used for the proxy. Tune it by using the NETWORK_INTERFACE environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the network interface for proxy
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # provide the network interface for proxy
        - name: NETWORK_INTERFACE
          value: "eth0"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: '80'