Tests

We’ve created a JSON Editor that has full semantic and syntax check so that it’s as easy as possible to define a test case.

You have the possibility of:

  • saving your work locally to a JSON file for later use

  • loading your work from a JSON file

  • copying a test case that on your colleague’s made and reuse it’s logic

Permissions

Member role vs API call

List tests

Start test

Cancel test

Get test details

Set test sharing

Remove test

Reader

True

False

False

True

False

False

Editor

True

True

True

True

False

True

Admin

True

True

True

True

True

True

AdminWithBilling

True

True

True

True

True

True

Owner

True

True

True

True

True

True

Test properties

The properties that can set up for yout tests are:

  1. run_time:

    • Property type: MANDATORY

    • Value type: Integer

    • Explanation: How long should the test run for in seconds

    • Restrictions: Minimum is 60 (seconds) and Maximum is 3600 (seconds), depending on your subscription type

  2. num_clients:

    • Property type: MANDATORY

    • Value type: Integer

    • Explanation: How many virtual users to simulate on each thread (container) per second

    • Restrictions: Minimum is 1 and Maximum is 25

  3. threads_per_region:

    • Property type: MANDATORY

    • Value type: Integer

    • Explanation: The number of containers to spawn in paralel in each region

    • Restrictions: Minimum is 1 and Maximum is 200, depending on your subscription type

  4. test_region:

    • Property type: MANDATORY

    • Value type: Array of Strings

    • Explanation: Array of one or more regions to run the test in

    • Restrictions: Values must include one or more of the following

["ap-northeast-1", "ap-northeast-2", "ap-southeast-1", "ap-southeast-2", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "us-east-1", "us-east-2", "us-west-1", "us-west-2", "ca-central-1", "ap-south-1", "sa-east-1"]
These values emulate the following regions:
ap-northeast-1 = Tokyo
ap-northeast-2 = Seoul
ap-southeast-1 = Singapore
ap-southeast-2 = Sydney
eu-central-1 = Frankfurt
eu-west-1 = Ireland
eu-west-2 = London
eu-west-3 = Paris
us-east-1 = North Virginia
us-east-2 = Ohio
us-west-1 = North California
us-west-2 = Oregon
ca-central-1 = Canada (Central)
ap-south-1 = Mumbai
sa-east-1 = Sao Paulo
  1. workflow:

    • Property type: MANDATORY

    • Value type: Array of Objects

    • Explanation: Array of object-based steps to execute sequentially

    • Object properties:

      5.1. path:

      • Property type: MANDATORY

      • Value type: String

      • Explanation: Path to test against

      • Restrictions: Must start with a “/”

      5.2. method:

      • Property type: MANDATORY

      • Value type: String

      • Explanation: Method to test against

      • Restrictions: Must be one of [“GET”, “POST”, “PATCH”, “HEAD”, “PUT”, “DELETE”, “OPTIONS”]

      5.3. data:

      • Property type: OPTIONAL (default is a blank string)

      • Value type: String

      • Explanation: Data to include as parameters in the body of the request (aka payload)

      5.4. headers:

      • Property type: OPTIONAL (default is a blank object)

      • Value type: Key-Value object

      • Explanation: Any key-value pair to include as headers in the request

      5.5. extract_cookies:

      • Property type: OPTIONAL (default has value “no”)

      • Value type: String

      • Explanation: This triggers our platform to extract the cookies which are present in the response for that particular step, and then push them forward throughout your future workflow steps. This is useful usually to PHP based platform such as Laravel/Magento/Wordpress (eg after doing the /login with POST)

      5.6. extract:

      • Property type: OPTIONAL (default is a blank object)

      • Value type: List of key-value objects

      • Explanation: A list of values to extract from either BODY or HEADERS to reuse in future steps of workflow

      5.6.1. parameter_name:
      • Property type: REQUIRED

      • Value type: String

      • Explanation: The name under which to store this value

      5.6.2. location:
      • Property type: REQUIRED

      • Value type: String

      • Explanation: The location from which to fetch this value (can be either “body” or “headers”)

      5.6.3. key:
      • Property type: OPTIONAL

      • Value type: String

      • Explanation: If this property is specified, the platform will check for that specific “KEY” in either the body (if it’s a JSON) or the headers, depending on the value set up for “location”

      • Note: You can either specify “key”, “element_with_regex” or “json_path_expression”, but not more than one!

      5.6.4. element_with_regex:
      • Property type: OPTIONAL

      • Value type: String

      • Explanation: If this property is specified, the platform will check for that specific “KEY” in either the body (if it’s a JSON) or the headers, depending on the value set up for “location”

      • Note: You can either specify “key”, “element_with_regex” or “json_path_expression”, but not more than one!

      5.6.5. json_path_expression:
      • Property type: OPTIONAL

      • Value type: String

      • Explanation: If this property is specified, the platform will check for that specific “JSON PATH EXPRESSION” in ONLY THE BODY (which should be a JSON) and extract a random match for that expression

      • Note: You can either specify “key”, “element_with_regex” or “json_path_expression”, but not more than one!

      5.6.6. default_value:
      • Property type: OPTIONAL

      • Value type: String

      • Explanation: Default value in case key/expression does not return anything or errors out. Supports PLAIN TEXT as well as VAULT config

      • Note: If not default_value is specified, and your extract operation does not return anything (or errors out), then a failure will be recorded in your test results!

Good to know!

Reusing responses in workflow steps

Yes, it is possible!

ALl you have to do is properly define the _extract_ field in the workflow step and afterwards you can reuse it in any future step, in either the payload or the headers through a bash-like notation -> ${parameter-name-here}.

Here’s a sample:

"workflow": [
    {
        "path": "/login_path",
        "method": "POST",
        "headers": {
            "Content-Type": "application/x-www-form-urlencoded"
        },
        "data": "user=user&password=pass",
        "extract": [
            {
                "parameter_name": "authtoken",
                "location": "body",
                "key": "access_token"
            }
        ]
    },
    {
        "path": "/my/profile",
        "method": "GET",
        "data": "",
        "headers": {
            "Authorization": "Bearer ${authtoken}"
        }
    }
]

The code above extracts the value of access_token from the respone body, stores it in the parameter named authtoken, and then reuses it in the next step inside the headers block by using a bash-like syntax for ${authtoken}.

Makes sense now ? :)

List tests

Web

In order to view the tests on our platform, all you have to do is browse to the Dashboard page.

API call

The API call is:

curl -s \
    -d '{"team_id":"my-team"}' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/list | jq . -r

{
  "Tests": [
    {
        "test_id": "my_test_id",
        "test_status": "FINISHED",
        "threads_per_region": 2,
        "created_date": "2020-03-26T09:08:15Z",
        "run_time": "30s",
        "member_email": "[email protected]",
        "sharing_type": "PRIVATE",
        "test_region": "ap-southeast-1,ap-southeast-2",
        "test_type": "MANUAL"
    },
    {
        "test_id": "my_test_id_2",
        "test_status": "FINISHED",
        "threads_per_region": 5,
        "created_date": "2020-03-29T10:12:34Z",
        "run_time": "60s",
        "member_email": "[email protected]",
        "sharing_type": "PRIVATE",
        "test_region": "us-east-1,us-east-2",
        "test_type": "MANUAL"
    }
}

You can also filter based on test_id to generate the summary for only that specific test_id:

curl -s \
    -d '{"team_id":"my-team", "test_id": "my_test_id"}' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/list | jq . -r

{
  "Tests": [
    {
        "test_id": "my_test_id",
        "test_status": "FINISHED",
        "threads_per_region": 2,
        "created_date": "2020-03-26T09:08:15Z",
        "run_time": "30s",
        "member_email": "[email protected]",
        "sharing_type": "PRIVATE",
        "test_region": "ap-southeast-1,ap-southeast-2",
        "test_type": "MANUAL"
    }
}

CLI

List all tests:

$ rungutan tests list
{
    "Tests": [
        {
            "test_id": "ad837abd-15d4-47ce-a344-01a3323bdd09",
            "test_name": "test name",
            "test_status": "CANCELLED",
            "threads_per_region": 1,
            "created_date": "2020-05-19T12:04:42Z",
            "run_time": 60,
            "member_email": "[email protected]",
            "sharing_type": "PRIVATE",
            "test_region": "us-east-1",
            "test_type": "MANUAL"
        },
        {
            "test_id": "15fa78b9-097b-401a-bedb-a70ddc4726ed",
            "test_name": "Sample worklfow test",
            "test_status": "FINISHED",
            "threads_per_region": 1,
            "created_date": "2020-05-19T11:56:19Z",
            "run_time": 60,
            "member_email": "[email protected]",
            "sharing_type": "PRIVATE",
            "test_region": "us-east-1",
            "test_type": "MANUAL"
        }
    ]
}

Filter based on test_id:

$ rungutan tests list --test_id ad837abd-15d4-47ce-a344-01a3323bdd09
{
    "Tests": [
        {
            "test_id": "ad837abd-15d4-47ce-a344-01a3323bdd09",
            "test_name": "test name",
            "test_status": "CANCELLED",
            "threads_per_region": 1,
            "created_date": "2020-05-19T12:04:42Z",
            "run_time": 60,
            "member_email": "[email protected]",
            "sharing_type": "PRIVATE",
            "test_region": "us-east-1",
            "test_type": "MANUAL"
        }
    ]
}

Start test

Web

In order to start a new test, all you have to do is browse to the Dashboard page and click on the Start test button.

API call

The API call is:

curl -s \
    -d '{
        "team_id": "my-team",
        "run_time": 60,
        "num_clients": 10,
        "threads_per_region": 1,
        "test_region": [
            "us-east-1"
        ],
        "workflow": [
            {
                "path": "https://example.com/",
                "method": "GET",
                "data": "",
                "headers": {}
            }
        ]}' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/add | jq . -r

{
    "test_id": "some_test_id",
    "message": "Successfully created new test"
}

CLI

Start the test in the background:

$ rungutan tests add --test_file test_file.json
{
    "test_id": "fff6580d-999b-4ce5-8e1b-b14eb66f42be",
    "test_name": "test from cli",
    "message": "Successfully created new test"
}

Or, start it and wait for it to finish as well:

$ rungutan tests add --test_file test_file.json --wait_to_finish
{
    "test_id": "61c8bc4b-e1b1-4fb5-bb24-d860bf79e460",
    "test_name": "test from cli",
    "message": "Successfully created new test"
}
Waiting for test to finish...
Test with ID 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460 has finished running with status FINISHED
{
    "Results": {
        "region": "overall",
        "failures": [],
        "public_ips": [
            "107.20.40.171"
        ],
        "requests": [
            {
                "Method": "GET",
                "Path": "/",
                "num_requests": 4,
                "min_response_time": 21,
                "median_response_time": 22,
                "avg_response_time": 56,
                "max_response_time": 149,
                "response_time_percentiles": {
                    "55": 31,
                    "65": 31,
                    "75": 150,
                    "85": 150,
                    "95": 150
                },
                "total_rps": 1,
                "total_rpm": 60
            }
        ]
    },
    "ResultsMetadata": {
        "team_id": "rungutan",
        "test_id": "61c8bc4b-e1b1-4fb5-bb24-d860bf79e460"
    }
}

Cancel test

Web

In order to cancel a running test, all you have to do is browse to the Dashboard page and click on the Cancel test button for the respective test.

You can only cancel a test if it’s not already finished running.

API call

The API call is:

curl -s \
    -d '{
        "test_id": "some_test_id",
        "team_id": "my-team"
        }' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/cancel | jq . -r

{
    "test_id": "some_test_id",
    "message": "Successfully cancelled test.",
}

CLI

It’s as simple as:

$ rungutan tests cancel --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460
{
    "message": "The test cannot be cancelled since it's currently in FINISHED status"
}

Get test details

Web

In order to get the details of a test, all you have to do is browse to the Dashboard page and click on the Copy test details button for the respective test.

API call

The API call is:

curl -s \
    -d '{
        "test_id": "some_test_id",
        "team_id": "my-team"
        }' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/get | jq . -r

{
    "TestData": {
        "run_time": 60,
        "num_clients": 10,
        "threads_per_region": 1,
        "workflow": [
            {
                "path": "https://example.com/",
                "method": "GET",
                "data": "",
                "headers": {}
            }
        ],
        "test_region": [
            "us-east-1"
        ]
      }
}

CLI

This is how you run the CLI:

$ rungutan tests get --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460
{
    "TestData": {
        "run_time": 5,
        "num_clients": 10,
        "threads_per_region": 1,
        "workflow": [
            {
                "path": "https://example.com/",
                "method": "GET",
                "data": "",
                "headers": {}
            }
        ],
        "test_region": [
            "us-east-1"
        ]
    }
}

And this is how you get the TestData and save it to a file for later use:

$ rungutan tests get --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460 | jq -r .TestData > file.json
$ cat file.json
{
  "run_time": 5,
  "num_clients": 10,
  "threads_per_region": 1,
  "workflow": [
    {
      "path": "https://example.com/",
      "method": "GET",
      "data": "",
      "headers": {}
    }
  ],
  "test_region": [
    "us-east-1"
  ]
}

Set test sharing

Web

All tests are by default PRIVATE.

However, should you want to brag with your results or share them to a person that is not a member of your team, we thought you should be able to do so.

Keep in mind though, that once the tests has been switched to PUBLIC view, any non-authenticated person can see it.

Don’t worry though, you can easily switch it back to PRIVATE mode any time you want.

In order to get the set the sharing type of a test, all you have to do is browse to the Dashboard page and click on the Enable public results button for the respective test.

API call

The API call is:

curl -s \
    -d '{
        "test_id": "some_test_id",
        "team_id": "my-team",
        "test_public": "PUBLIC"                 // or "PRIVATE", depending on your scenario
        }' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/set-sharing | jq . -r

{
    "message": "Successfully updated test sharing",
    "test_id": "some_test_id"
}

CLI

Setting it as public:

$ rungutan tests set-sharing --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460 --test_public public
{
    "message": "Successfully updated test sharing",
    "test_id": "61c8bc4b-e1b1-4fb5-bb24-d860bf79e460"
}

Or as private:

$ rungutan tests set-sharing --test_id 61c8bc4b-e1b1-4fb5-bb24-d860bf79e460 --test_public private
{
    "message": "Successfully updated test sharing",
    "test_id": "61c8bc4b-e1b1-4fb5-bb24-d860bf79e460"
}

Remove test

Web

In order to remove a test, all you have to do is browse to the Dashboard page, click on the “View Results” button for the test you want to remove and then hit the “Delete test” button.

API call

The API call is:

curl -s \
    -d '{
        "test_id": "some_id",
        "team_id": "my-team"
        }' \
    -H "X-Api-Key: my_api_key" \
    -X POST \
    https://app.rungutan.com/v1/api/tests/remove | jq . -r

{
    "message": "Successfully deleted test"
}

CLI

Remove the template using the CLI:

$ rungutan tests remove --test_id fff6580d-999b-4ce5-8e1b-b14eb66f42be
{
    "message": "Successfully deleted test"
}