Chaos Injection for AWS Lambda - chaos_lambda

Documentation Status Issues Maintenance Pypi Travis Coveralls Twitter

chaos_lambda is a small library injecting chaos into AWS Lambda. It offers simple python decorators to do delay, exception and statusCode injection and a Class to add delay to any 3rd party dependencies called from your function. This allows to conduct small chaos engineering experiments for your serverless application in the AWS Cloud.

  • Support for Latency injection using delay

  • Support for Exception injection using exception_msg

  • Support for HTTP Error status code injection using error_code

  • Using for SSM Parameter Store to control the experiment using isEnabled

  • Support for adding rate of failure using rate. (Default rate = 1)

  • Per Lambda function injection control using Environment variable (CHAOS_PARAM)

Install

pip install chaos-lambda

Example

# function.py

import os
from chaos_lambda import inject_delay, inject_exception, inject_statuscode

# this should be set as a Lambda environment variable
os.environ['CHAOS_PARAM'] = 'chaoslambda.config'

@inject_exception
def handler_with_exception(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_exception(exception_type=TypeError, exception_msg='foobar')
def handler_with_exception_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_exception(exception_type=ValueError)
def handler_with_exception_arg_2(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_statuscode
def handler_with_statuscode(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_statuscode(error_code=400)
def handler_with_statuscode_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_delay
def handler_with_delay(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_delay(delay=1000)
def handler_with_delay_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_delay(delay=0)
def handler_with_delay_zero(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

When excecuted, the Lambda function, e.g handler_with_exception('foo', 'bar'), will produce the following result:

exception_msg from config I really failed seriously with a rate of 1
corrupting now
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../chaos_lambda.py", line 199, in wrapper
    raise Exception(exception_msg)
Exception: I really failed seriously

Configuration

The configuration for the failure injection is stored in the AWS SSM Parameter Store and accessed at runtime by the get_config() function:

{
    "isEnabled": true,
    "delay": 400,
    "error_code": 404,
    "exception_msg": "I really failed seriously",
    "rate": 1
}

To store the above configuration into SSM using the AWS CLI do the following:

aws ssm put-parameter --region eu-north-1 --name chaoslambda.config --type String --overwrite --value "{ "delay": 400, "isEnabled": true, "error_code": 404, "exception_msg": "I really failed seriously", "rate": 1 }"

AWS Lambda will need to have IAM access to SSM.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeParameters"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters",
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:eu-north-1:12345678910:parameter/chaoslambda.config"
        }
    ]
}

Supported Decorators:

chaos_lambda currently supports the following decorators:

  • @inject_delay - add delay in the AWS Lambda execution

  • @inject_exception - Raise an exception during the AWS Lambda execution

  • @inject_statuscode - force AWS Lambda to return a specific HTTP error code

More information:

chaos_lambda.get_config(config_key)[source]

Retrieve the configuration from the SSM parameter store The config always returns a tuple (value, rate) value: requested configuration rate: the injection probability (default 1 –> 100%)

How to use:

>>> import os
>>> from chaos_lambda import get_config
>>> os.environ['CHAOS_PARAM'] = 'chaoslambda.config'
>>> get_config('delay')
(400, 1)
>>> get_config('exception_msg')
('I really failed seriously', 1)
>>> get_config('error_code')
(404, 1)
chaos_lambda.inject_delay(func=None, delay=None)[source]

Add delay to the lambda function - delay is returned from the SSM paramater using get_config('delay') which returns a tuple delay, rate.

Default use:

>>> @inject_delay
... def handler(event, context):
...    return {
...       'statusCode': 200,
...       'body': 'Hello from Lambda!'
...    }
>>> handler('foo', 'bar')
Injecting 400 of delay with a rate of 1
Added 402.20ms to handler
{'statusCode': 200, 'body': 'Hello from Lambda!'}

With argument:

>>> @inject_delay(delay=1000)
... def handler(event, context):
...    return {
...       'statusCode': 200,
...       'body': 'Hello from Lambda!'
...    }
>>> handler('foo', 'bar')
Injecting 1000 of delay with a rate of 1
Added 1002.20ms to handler
{'statusCode': 200, 'body': 'Hello from Lambda!'}
chaos_lambda.inject_exception(func=None, exception_type=None, exception_msg=None)[source]

Forces the lambda function to fail and raise an exception using get_config('exception_msg') which returns a tuple exception_msg, rate.

Default use (Error type is Exception):

>>> @inject_exception
... def handler(event, context):
...     return {
...        'statusCode': 200,
...        'body': 'Hello from Lambda!'
...     }
>>> handler('foo', 'bar')
Injecting exception_type <class "Exception"> with message I really failed seriously a rate of 1
corrupting now
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/.../chaos_lambda.py", line 316, in wrapper
        raise _exception_type(_exception_msg)
Exception: I really failed seriously

With Error type argument:

>>> @inject_exception(exception_type=ValueError)
... def lambda_handler_with_exception_arg_2(event, context):
...     return {
...         'statusCode': 200,
...         'body': 'Hello from Lambda!'
...     }
>>> lambda_handler_with_exception_arg_2('foo', 'bar')
Injecting exception_type <class 'ValueError'> with message I really failed seriously a rate of 1
corrupting now
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/.../chaos_lambda.py", line 316, in wrapper
        raise _exception_type(_exception_msg)
ValueError: I really failed seriously

With Error type and message argument:

>>> @inject_exception(exception_type=TypeError, exception_msg='foobar')
... def lambda_handler_with_exception_arg(event, context):
...     return {
...         'statusCode': 200,
...         'body': 'Hello from Lambda!'
...     }
>>> lambda_handler_with_exception_arg('foo', 'bar')
Injecting exception_type <class 'TypeError'> with message foobar a rate of 1
corrupting now
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/.../chaos_lambda.py", line 316, in wrapper
        raise _exception_type(_exception_msg)
TypeError: foobar
chaos_lambda.inject_statuscode(func=None, error_code=None)[source]

Forces the lambda function to return with a specific Status Code using get_config('error_code') which returns a tuple error_code, rate.

Default use:

>>> @inject_statuscode
... def handler(event, context):
...    return {
...       'statusCode': 200,
...       'body': 'Hello from Lambda!'
...    }
>>> handler('foo', 'bar')
Injecting Error 404 at a rate of 1
corrupting now
{'statusCode': 404, 'body': 'Hello from Lambda!'}

With argument:

>>> @inject_statuscode(error_code=400)
... def lambda_handler_with_statuscode_arg(event, context):
...     return {
...         'statusCode': 200,
...         'body': 'Hello from Lambda!'
...     }
>>> lambda_handler_with_statuscode_arg('foo', 'bar')
Injecting Error 400 at a rate of 1
corrupting now
{'statusCode': 400, 'body': 'Hello from Lambda!'}