Tests¶

We’ve created a JSON Editor that has full semantic and syntax check so that it’s as easy as possible to define a test case.

You have the possibility of:

saving your work locally to a JSON file for later use
loading your work from a JSON file
copying a test case that on your colleague’s made and reuse it’s logic

Permissions¶

Member role vs API call	List tests	Start test	Cancel test	Get test details	Set test sharing	Remove test
Reader	True	False	False	True	False	False
Editor	True	True	True	True	False	True
Admin	True	True	True	True	True	True
AdminWithBilling	True	True	True	True	True	True
Owner	True	True	True	True	True	True

Test properties¶

The properties that can set up for yout tests are:

run_time:
- Property type: MANDATORY
- Value type: Integer
- Explanation: How long should the test run for in seconds
- Restrictions: Minimum is 60 (seconds) and Maximum is 3600 (seconds), depending on your subscription type
num_clients:
- Property type: MANDATORY
- Value type: Integer
- Explanation: How many virtual users to simulate on each thread (container) per second
- Restrictions: Minimum is 1 and Maximum is 25
threads_per_region:
- Property type: MANDATORY
- Value type: Integer
- Explanation: The number of containers to spawn in paralel in each region
- Restrictions: Minimum is 1 and Maximum is 200, depending on your subscription type
test_region:
- Property type: MANDATORY
- Value type: Array of Strings
- Explanation: Array of one or more regions to run the test in
- Restrictions: Values must include one or more of the following

["ap-northeast-1", "ap-northeast-2", "ap-southeast-1", "ap-southeast-2", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "us-east-1", "us-east-2", "us-west-1", "us-west-2", "ca-central-1", "ap-south-1", "sa-east-1"]
These values emulate the following regions:
ap-northeast-1 = Tokyo
ap-northeast-2 = Seoul
ap-southeast-1 = Singapore
ap-southeast-2 = Sydney
eu-central-1 = Frankfurt
eu-west-1 = Ireland
eu-west-2 = London
eu-west-3 = Paris
us-east-1 = North Virginia
us-east-2 = Ohio
us-west-1 = North California
us-west-2 = Oregon
ca-central-1 = Canada (Central)
ap-south-1 = Mumbai
sa-east-1 = Sao Paulo

workflow:
- Property type: MANDATORY
- Value type: Array of Objects
- Explanation: Array of object-based steps to execute sequentially
- Object properties:
  5.1. path:
  
  Property type: MANDATORY
  
  Value type: String
  
  Explanation: Path to test against
  
  Restrictions: Must start with a “/”
  
  5.2. method:
  
  Property type: MANDATORY
  
  Value type: String
  
  Explanation: Method to test against
  
  Restrictions: Must be one of [“GET”, “POST”, “PATCH”, “HEAD”, “PUT”, “DELETE”, “OPTIONS”]
  
  5.3. data:
  
  Property type: OPTIONAL (default is a blank string)
  
  Value type: String
  
  Explanation: Data to include as parameters in the body of the request (aka payload)
  
  5.4. headers:
  
  Property type: OPTIONAL (default is a blank object)
  
  Value type: Key-Value object
  
  Explanation: Any key-value pair to include as headers in the request
  
  5.5. extract_cookies:
  
  Property type: OPTIONAL (default has value “no”)
  
  Value type: String
  
  Explanation: This triggers our platform to extract the cookies which are present in the response for that particular step, and then push them forward throughout your future workflow steps. This is useful usually to PHP based platform such as Laravel/Magento/Wordpress (eg after doing the /login with POST)
  
  5.6. extract:
  
  Property type: OPTIONAL (default is a blank object)
  
  Value type: List of key-value objects
  
  Explanation: A list of values to extract from either BODY or HEADERS to reuse in future steps of workflow
  
  5.6.1. parameter_name:
  
  Property type: REQUIRED
  
  Value type: String
  
  Explanation: The name under which to store this value
  
  5.6.2. location:
  
  Property type: REQUIRED
  
  Value type: String
  
  Explanation: The location from which to fetch this value (can be either “body” or “headers”)
  
  5.6.3. key:
  
  Property type: OPTIONAL
  
  Value type: String
  
  Explanation: If this property is specified, the platform will check for that specific “KEY” in either the body (if it’s a JSON) or the headers, depending on the value set up for “location”
  
  Note: You can either specify “key”, “element_with_regex” or “json_path_expression”, but not more than one!
  
  5.6.4. element_with_regex:
  
  Property type: OPTIONAL
  
  Value type: String
  
  Explanation: If this property is specified, the platform will check for that specific “KEY” in either the body (if it’s a JSON) or the headers, depending on the value set up for “location”
  
  Note: You can either specify “key”, “element_with_regex” or “json_path_expression”, but not more than one!
  
  5.6.5. json_path_expression:
  
  Property type: OPTIONAL
  
  Value type: String
  
  Explanation: If this property is specified, the platform will check for that specific “JSON PATH EXPRESSION” in ONLY THE BODY (which should be a JSON) and extract a random match for that expression
  
  Note: You can either specify “key”, “element_with_regex” or “json_path_expression”, but not more than one!
  
  5.6.6. default_value:
  
  Property type: OPTIONAL
  
  Value type: String
  
  Explanation: Default value in case key/expression does not return anything or errors out. Supports PLAIN TEXT as well as VAULT config
  
  Note: If not default_value is specified, and your extract operation does not return anything (or errors out), then a failure will be recorded in your test results!

Good to know!¶

Reusing responses in workflow steps¶

Yes, it is possible!

ALl you have to do is properly define the _extract_ field in the workflow step and afterwards you can reuse it in any future step, in either the payload or the headers through a bash-like notation -> ${parameter-name-here}.

Here’s a sample:

"workflow": [
    {
        "path": "/login_path",
        "method": "POST",
        "headers": {
            "Content-Type": "application/x-www-form-urlencoded"
        },
        "data": "user=user&password=pass",
        "extract": [
            {
                "parameter_name": "authtoken",
                "location": "body",
                "key": "access_token"
            }
        ]
    },
    {
        "path": "/my/profile",
        "method": "GET",
        "data": "",
        "headers": {
            "Authorization": "Bearer ${authtoken}"
        }
    }
]

The code above extracts the value of access_token from the respone body, stores it in the parameter named authtoken, and then reuses it in the next step inside the headers block by using a bash-like syntax for ${authtoken}.

Makes sense now ? :)

List tests¶

Web¶

In order to view the tests on our platform, all you have to do is browse to the Dashboard page.

API call¶

The API call is:

curl -s \
    -d '{"team_id":"my-team"}' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/list | jq . -r

{
  "Tests": [
    {
        "test_id": "my_test_id",
        "test_status": "FINISHED",
        "threads_per_region": 2,
        "created_date": "2020-03-26T09:08:15Z",
        "run_time": "30s",
        "member_email": "[email protected]",
        "sharing_type": "PRIVATE",
        "test_region": "ap-southeast-1,ap-southeast-2",
        "test_type": "MANUAL"
    },
    {
        "test_id": "my_test_id_2",
        "test_status": "FINISHED",
        "threads_per_region": 5,
        "created_date": "2020-03-29T10:12:34Z",
        "run_time": "60s",
        "member_email": "[email protected]",
        "sharing_type": "PRIVATE",
        "test_region": "us-east-1,us-east-2",
        "test_type": "MANUAL"
    }
}

You can also filter based on test_id to generate the summary for only that specific test_id:

curl -s \
    -d '{"team_id":"my-team", "test_id": "my_test_id"}' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/list | jq . -r

{
  "Tests": [
    {
        "test_id": "my_test_id",
        "test_status": "FINISHED",
        "threads_per_region": 2,
        "created_date": "2020-03-26T09:08:15Z",
        "run_time": "30s",
        "member_email": "[email protected]",
        "sharing_type": "PRIVATE",
        "test_region": "ap-southeast-1,ap-southeast-2",
        "test_type": "MANUAL"
    }
}

CLI¶

List all tests:

$ rungutan tests list
{
    "Tests": [
        {
            "test_id": "ad837abd-15d4-47ce-a344-01a3323bdd09",
            "test_name": "test name",
            "test_status": "CANCELLED",
            "threads_per_region": 1,
            "created_date": "2020-05-19T12:04:42Z",
            "run_time": 60,
            "member_email": "[email protected]",
            "sharing_type": "PRIVATE",
            "test_region": "us-east-1",
            "test_type": "MANUAL"
        },
        {
            "test_id": "15fa78b9-097b-401a-bedb-a70ddc4726ed",
            "test_name": "Sample worklfow test",
            "test_status": "FINISHED",
            "threads_per_region": 1,
            "created_date": "2020-05-19T11:56:19Z",
            "run_time": 60,
            "member_email": "[email protected]",
            "sharing_type": "PRIVATE",
            "test_region": "us-east-1",
            "test_type": "MANUAL"
        }
    ]
}

Filter based on test_id:

$ rungutan tests list --test_id ad837abd-15d4-47ce-a344-01a3323bdd09
{
    "Tests": [
        {
            "test_id": "ad837abd-15d4-47ce-a344-01a3323bdd09",
            "test_name": "test name",
            "test_status": "CANCELLED",
            "threads_per_region": 1,
            "created_date": "2020-05-19T12:04:42Z",
            "run_time": 60,
            "member_email": "[email protected]",
            "sharing_type": "PRIVATE",
            "test_region": "us-east-1",
            "test_type": "MANUAL"
        }
    ]
}

Start test¶

Web¶

In order to start a new test, all you have to do is browse to the Dashboard page and click on the Start test button.

API call¶

The API call is:

curl -s \
    -d '{
        "team_id": "my-team",
        "run_time": 60,
        "num_clients": 10,
        "threads_per_region": 1,
        "test_region": [
            "us-east-1"
        ],
        "workflow": [
            {
                "path": "https://example.com/",
                "method": "GET",
                "data": "",
                "headers": {}
            }
        ]}' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/add | jq . -r

{
    "test_id": "some_test_id",
    "message": "Successfully created new test"
}

CLI¶

Start the test in the background:

$ rungutan tests add --test_file test_file.json
{
    "test_id": "fff6580d-999b-4ce5-8e1b-b14eb66f42be",
    "test_name": "test from cli",
    "message": "Successfully created new test"
}

Or, start it and wait for it to finish as well:

$ rungutan tests add --test_file test_file.json --wait_to_finish
{
    "test_id": "61c8bc4b-e1b1-4fb5-bb24-d860bf79e460",
    "test_name": "test from cli",
    "message": "Successfully created new test"
}
Waiting for test to finish...
Test with ID 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460 has finished running with status FINISHED
{
    "Results": {
        "region": "overall",
        "failures": [],
        "public_ips": [
            "107.20.40.171"
        ],
        "requests": [
            {
                "Method": "GET",
                "Path": "/",
                "num_requests": 4,
                "min_response_time": 21,
                "median_response_time": 22,
                "avg_response_time": 56,
                "max_response_time": 149,
                "response_time_percentiles": {
                    "55": 31,
                    "65": 31,
                    "75": 150,
                    "85": 150,
                    "95": 150
                },
                "total_rps": 1,
                "total_rpm": 60
            }
        ]
    },
    "ResultsMetadata": {
        "team_id": "rungutan",
        "test_id": "61c8bc4b-e1b1-4fb5-bb24-d860bf79e460"
    }
}

Cancel test¶

Web¶

In order to cancel a running test, all you have to do is browse to the Dashboard page and click on the Cancel test button for the respective test.

You can only cancel a test if it’s not already finished running.

API call¶

The API call is:

curl -s \
    -d '{
        "test_id": "some_test_id",
        "team_id": "my-team"
        }' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/cancel | jq . -r

{
    "test_id": "some_test_id",
    "message": "Successfully cancelled test.",
}

CLI¶

It’s as simple as:

$ rungutan tests cancel --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460
{
    "message": "The test cannot be cancelled since it's currently in FINISHED status"
}

Get test details¶

Web¶

In order to get the details of a test, all you have to do is browse to the Dashboard page and click on the Copy test details button for the respective test.

API call¶

The API call is:

curl -s \
    -d '{
        "test_id": "some_test_id",
        "team_id": "my-team"
        }' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/get | jq . -r

{
    "TestData": {
        "run_time": 60,
        "num_clients": 10,
        "threads_per_region": 1,
        "workflow": [
            {
                "path": "https://example.com/",
                "method": "GET",
                "data": "",
                "headers": {}
            }
        ],
        "test_region": [
            "us-east-1"
        ]
      }
}

CLI¶

This is how you run the CLI:

$ rungutan tests get --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460
{
    "TestData": {
        "run_time": 5,
        "num_clients": 10,
        "threads_per_region": 1,
        "workflow": [
            {
                "path": "https://example.com/",
                "method": "GET",
                "data": "",
                "headers": {}
            }
        ],
        "test_region": [
            "us-east-1"
        ]
    }
}

And this is how you get the TestData and save it to a file for later use:

$ rungutan tests get --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460 | jq -r .TestData > file.json
$ cat file.json
{
  "run_time": 5,
  "num_clients": 10,
  "threads_per_region": 1,
  "workflow": [
    {
      "path": "https://example.com/",
      "method": "GET",
      "data": "",
      "headers": {}
    }
  ],
  "test_region": [
    "us-east-1"
  ]
}

Remove test¶

Web¶

In order to remove a test, all you have to do is browse to the Dashboard page, click on the “View Results” button for the test you want to remove and then hit the “Delete test” button.

API call¶

The API call is:

curl -s \
    -d '{
        "test_id": "some_id",
        "team_id": "my-team"
        }' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/remove | jq . -r

{
    "message": "Successfully deleted test"
}

CLI¶

Remove the template using the CLI:

$ rungutan tests remove --test_id fff6580d-999b-4ce5-8e1b-b14eb66f42be
{
    "message": "Successfully deleted test"
}

Tests¶

Permissions¶

Test properties¶

Good to know!¶

Reusing responses in workflow steps¶

List tests¶

Web¶

API call¶

CLI¶

Start test¶

Web¶

API call¶

CLI¶

Cancel test¶

Web¶

API call¶

CLI¶

Get test details¶

Web¶

API call¶

CLI¶

Set test sharing¶

Web¶

API call¶

CLI¶

Remove test¶

Web¶

API call¶

CLI¶