Chaos Injection for AWS Lambda - chaos_lambda

Documentation Status Issues Maintenance Pypi Travis Coveralls Twitter

chaos_lambda is a small library injecting chaos into AWS Lambda. It offers simple python decorators to do delay, exception and statusCode injection and a Class to add delay to any 3rd party dependencies called from your function. This allows to conduct small chaos engineering experiments for your serverless application in the AWS Cloud.

  • Support for Latency injection using delay

  • Support for Exception injection using exception_msg

  • Support for HTTP Error status code injection using error_code

  • Using for SSM Parameter Store to control the experiment using isEnabled

  • Support for adding rate of failure using rate. (Default rate = 1)

  • Per Lambda function injection control using Environment variable (CHAOS_PARAM)

Install

pip install chaos-lambda

Example

# function.py

import os
from chaos_lambda import inject_delay, inject_exception, inject_statuscode

# this should be set as a Lambda environment variable
os.environ['CHAOS_PARAM'] = 'chaoslambda.config'

@inject_exception
def handler_with_exception(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_exception(exception_type=TypeError, exception_msg='foobar')
def handler_with_exception_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_exception(exception_type=ValueError)
def handler_with_exception_arg_2(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_statuscode
def handler_with_statuscode(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_statuscode(error_code=400)
def handler_with_statuscode_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_delay
def handler_with_delay(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_delay(delay=1000)
def handler_with_delay_arg(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }


@inject_delay(delay=0)
def handler_with_delay_zero(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

When excecuted, the Lambda function, e.g handler_with_exception('foo', 'bar'), will produce the following result:

exception_msg from config I really failed seriously with a rate of 1
corrupting now
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../chaos_lambda.py", line 199, in wrapper
    raise Exception(exception_msg)
Exception: I really failed seriously

Configuration

The configuration for the failure injection is stored in the AWS SSM Parameter Store and accessed at runtime by the get_config() function:

{
    "isEnabled": true,
    "delay": 400,
    "error_code": 404,
    "exception_msg": "I really failed seriously",
    "rate": 1
}

To store the above configuration into SSM using the AWS CLI do the following:

aws ssm put-parameter --region eu-north-1 --name chaoslambda.config --type String --overwrite --value "{ "delay": 400, "isEnabled": true, "error_code": 404, "exception_msg": "I really failed seriously", "rate": 1 }"

AWS Lambda will need to have IAM access to SSM.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeParameters"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters",
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:eu-north-1:12345678910:parameter/chaoslambda.config"
        }
    ]
}

Supported Decorators:

chaos_lambda currently supports the following decorators:

  • @inject_delay - add delay in the AWS Lambda execution

  • @inject_exception - Raise an exception during the AWS Lambda execution

  • @inject_statuscode - force AWS Lambda to return a specific HTTP error code

and the following class:

  • SessionWithDelay - enabled to sub-classing requests library and call dependencies with delay

More information:

class chaos_lambda.SessionWithDelay(delay=None)[source]

This is a class for injecting delay to 3rd party dependencies. Subclassing the requests library is useful if you want to conduct other chaos experiments within the library, like error injection or requests modification. This is a simple subclassing of the parent class requests.Session to add delay to the request method.

Usage:

>>> from chaos_lambda import SessionWithDelay
>>> def dummy():
...     session = SessionWithDelay(delay=300)
...     session.get('https://stackoverflow.com/')
...     pass
>>> dummy()
Added 300.00ms of delay to GET
request(method, url, *args, **kwargs)[source]

Constructs a Request, prepares it and sends it. Returns Response object.

Parameters
  • method – method for the new Request object.

  • url – URL for the new Request object.

  • params – (optional) Dictionary or bytes to be sent in the query string for the Request.

  • data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the Request.

  • json – (optional) json to send in the body of the Request.

  • headers – (optional) Dictionary of HTTP Headers to send with the Request.

  • cookies – (optional) Dict or CookieJar object to send with the Request.

  • files – (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.

  • auth – (optional) Auth tuple or callable to enable Basic/Digest/Custom HTTP Auth.

  • timeout (float or tuple) – (optional) How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.

  • allow_redirects (bool) – (optional) Set to True by default.

  • proxies – (optional) Dictionary mapping protocol or protocol and hostname to the URL of the proxy.

  • stream – (optional) whether to immediately download the response content. Defaults to False.

  • verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Defaults to True.

  • cert – (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert’, ‘key’) pair.

Return type

requests.Response

chaos_lambda.get_config(config_key)[source]

Retrieve the configuration from the SSM parameter store The config always returns a tuple (value, rate) value: requested configuration rate: the injection probability (default 1 –> 100%)

How to use:

>>> import os
>>> from chaos_lambda import get_config
>>> os.environ['CHAOS_PARAM'] = 'chaoslambda.config'
>>> get_config('delay')
(400, 1)
>>> get_config('exception_msg')
('I really failed seriously', 1)
>>> get_config('error_code')
(404, 1)
chaos_lambda.inject_delay(func=None, delay=None)[source]

Add delay to the lambda function - delay is returned from the SSM paramater using get_config('delay') which returns a tuple delay, rate.

Default use:

>>> @inject_delay
... def handler(event, context):
...    return {
...       'statusCode': 200,
...       'body': 'Hello from Lambda!'
...    }
>>> handler('foo', 'bar')
Injecting 400 of delay with a rate of 1
Added 402.20ms to handler
{'statusCode': 200, 'body': 'Hello from Lambda!'}

With argument:

>>> @inject_delay(delay=1000)
... def handler(event, context):
...    return {
...       'statusCode': 200,
...       'body': 'Hello from Lambda!'
...    }
>>> handler('foo', 'bar')
Injecting 1000 of delay with a rate of 1
Added 1002.20ms to handler
{'statusCode': 200, 'body': 'Hello from Lambda!'}
chaos_lambda.inject_exception(func=None, exception_type=None, exception_msg=None)[source]

Forces the lambda function to fail and raise an exception using get_config('exception_msg') which returns a tuple exception_msg, rate.

Default use (Error type is Exception):

>>> @inject_exception
... def handler(event, context):
...     return {
...        'statusCode': 200,
...        'body': 'Hello from Lambda!'
...     }
>>> handler('foo', 'bar')
Injecting exception_type <class "Exception"> with message I really failed seriously a rate of 1
corrupting now
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/.../chaos_lambda.py", line 316, in wrapper
        raise _exception_type(_exception_msg)
Exception: I really failed seriously

With Error type argument:

>>> @inject_exception(exception_type=ValueError)
... def lambda_handler_with_exception_arg_2(event, context):
...     return {
...         'statusCode': 200,
...         'body': 'Hello from Lambda!'
...     }
>>> lambda_handler_with_exception_arg_2('foo', 'bar')
Injecting exception_type <class 'ValueError'> with message I really failed seriously a rate of 1
corrupting now
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/.../chaos_lambda.py", line 316, in wrapper
        raise _exception_type(_exception_msg)
ValueError: I really failed seriously

With Error type and message argument:

>>> @inject_exception(exception_type=TypeError, exception_msg='foobar')
... def lambda_handler_with_exception_arg(event, context):
...     return {
...         'statusCode': 200,
...         'body': 'Hello from Lambda!'
...     }
>>> lambda_handler_with_exception_arg('foo', 'bar')
Injecting exception_type <class 'TypeError'> with message foobar a rate of 1
corrupting now
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/.../chaos_lambda.py", line 316, in wrapper
        raise _exception_type(_exception_msg)
TypeError: foobar
chaos_lambda.inject_statuscode(func=None, error_code=None)[source]

Forces the lambda function to return with a specific Status Code using get_config('error_code') which returns a tuple error_code, rate.

Default use:

>>> @inject_statuscode
... def handler(event, context):
...    return {
...       'statusCode': 200,
...       'body': 'Hello from Lambda!'
...    }
>>> handler('foo', 'bar')
Injecting Error 404 at a rate of 1
corrupting now
{'statusCode': 404, 'body': 'Hello from Lambda!'}

With argument:

>>> @inject_statuscode(error_code=400)
... def lambda_handler_with_statuscode_arg(event, context):
...     return {
...         'statusCode': 200,
...         'body': 'Hello from Lambda!'
...     }
>>> lambda_handler_with_statuscode_arg('foo', 'bar')
Injecting Error 400 at a rate of 1
corrupting now
{'statusCode': 400, 'body': 'Hello from Lambda!'}