EC2 DNS chaos
EC2 DNS chaos causes DNS errors such as unavailability or malfunctioning of DNS servers on the specified EC2 instance for a specific duration.
Use cases
EC2 DNS chaos:
- Determines the performance of the application (or process) running on the EC2 instance(s).
- Simulates the unavailability (or distorted) network connectivity from the VM to the target hosts.
- Determines the impact of DNS chaos on the infrastructure and standalone tasks.
- Simulates unavailability of the DNS server (loss of access to any external domain from a given microservice, access to cloud provider dependencies, and access to specific third party services).
Prerequisites
- Kubernetes >= 1.17
- SSM agent is installed and running on the target EC2 instance.
- The EC2 instance should be in a healthy state.
- You can pass the VM credentials as secrets or as a
ChaosEngine
environment variable. - The Kubernetes secret should have the AWS Access Key ID and Secret Access Key credentials in the
CHAOS_NAMESPACE
. Below is the sample secret file:apiVersion: v1
kind: Secret
metadata:
name: cloud-secret
type: Opaque
stringData:
cloud_config.yml: |-
# Add the cloud AWS credentials respectively
[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
HCE recommends that you use the same secret name, that is, cloud-secret
. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE
environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.
Below is an example AWS policy to execute the fault.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:SendCommand",
"ssm:CancelCommand",
"ssm:CreateDocument",
"ssm:DeleteDocument",
"ssm:GetCommandInvocation",
"ssm:UpdateInstanceInformation",
"ssm:DescribeInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceStatus",
"ec2:DescribeInstances"
],
"Resource": [
"*"
]
}
]
}
- Go to the common tunables to tune the common tunables for all the faults.
- Go to AWS named profile for chaos to use a different profile for AWS faults and superset permission or policy to execute all AWS faults.
Fault tunables
Mandatory tunables
Tunable | Description | Notes |
---|---|---|
EC2_INSTANCE_ID | ID of the target EC2 instance. | For example, i-044d3cb4b03b8af1f . For more information, go to EC2 instance ID. |
REGION | AWS region ID where the EC2 instance has been created. | For example: us-east-1 . |
PORT | DNS port where chaos is injected. | Default: port 54. For more information, go to port. |
Optional tunables
Tunable | Description | Notes |
---|---|---|
MACHINE | Whether chaos is applied on a Windows VM or a Linux VM. | |
TOTAL_CHAOS_DURATION | Duration that you specify, through which chaos is injected into the target resource (in seconds). | Default: 30 s. For more information, go to duration of the chaos. |
CHAOS_INTERVAL | Time interval between two successive instance terminations (in seconds). | Default: 30 s. For more information, go to chaos interval. |
SEQUENCE | Sequence of chaos execution for multiple instance | Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution. |
RAMP_TIME | Period to wait before and after injection of chaos in sec | For example, 30 s. For more information, go to ramp time. |
INSTALL_DEPENDENCY | Select to install dependencies used to run the network chaos. It can be either True or False | If the dependency already exists, you can turn it off. Defaults to True. |
TARGET_HOSTNAMES | List of the target host names. If this is not provided, all the host names (or domains) will be targeted. | For example, '["litmuschaos","chaosnative.com"]' . For more information, go to target host names. |
MATCH_SCHEME | Determines whether the DNS query should exactly match the targets or can be a substring. | Default: exact. For more information, go to match scheme. |
UPSTREAM_SERVER | Custom upstream server to which the intercepted DNS requests are forwarded. | Default: Server mentioned in the resolv.conf file. For more information, go to upstream server. |
Run DNS chaos with port
DNS port to inject DNS chaos. Tune it by using the PORT
environment variable.
The following YAML snippet illustrates the use of this environment variable:
# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# target port
- name: PORT
value: '54'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'
Run DNS chaos with target host names
List of the target host names to inject DNS chaos. Tune it by using the TARGET_HOSTNAMES
environment variable.
The following YAML snippet illustrates the use of this environment variable:
# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# list of target host names
- name: TARGET_HOSTNAMES
value: '["litmuschaos","chaosnative.com"]'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'
Run DNS chaos with match scheme
Determine whether the DNS query exactly matches the target or is a substring. Tune it by using the MATCH_SCHEME
environment variable.
The following YAML snippet illustrates the use of this environment variable:
# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# match scheme type
- name: MATCH_SCHEME
value: 'exact'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'
Run DNS chaos with upstream server
Custom upstream server where the intercepted DNS requests are forwarded. It defaults to the server mentioned in the resolv.conf file. Tune it by using the UPSTREAM_SERVER
environment variable.
The following YAML snippet illustrates the use of this environment variable:
# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# name of the upstream server
- name: UPSTREAM_SERVER
value: '8.8.8.8'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'