API Reference

Client Library

touca/touca.hpp is the main entry-point to the Touca SDK for C++. In most cases, it is the only header-file that users should include in their regression test tool. It provides all the functions necessary to configure the client, declare testcases, capture test results, and submit them to the Touca server. This section documents the API exposed by this header file.

Configuring the Library

void touca::configure(const std::unordered_map<std::string, std::string> &opts)

Configures the touca client.

Must be called before declaring testcases and adding results to the client. Should be regarded as a potentially expensive operation. Takes configuration parameters as a list of key value pairs whose keys are from the list below.

  • api-key API Key issued by the Touca server. Instead of passing this parameter to configure, you can set environment variable TOUCA_API_KEY.

  • api-url URL of Touca server API. Can be provided either in long format like @/myteam/mysuite or in short format like . If slug of the team and suite to which the results belong are specified, separately passing team and suite configuration parameters will not be required.

  • version Version of the workflow under test. Required if api-url is not set or if api-url does not include the version.

  • suite Name of the testsuite to which testresults belong. Required if api-url is not set or if api-url is provided in short format.

  • team Name of the team to which this testsuite belongs. Required if api-url is not set or if api-url is provided in short format.

  • handshake Ensures Touca server is ready to accept incoming testresults as part of client configuration process. Handshake is performed only if api-key and api-url parameters are set. Defaults to true.

  • allow-empty-suite Declare configuration of the client as successful even if the suite has no registered test cases on the Touca Server. Defaults to true.

  • post-testcases Maximum number of testcases whose results may be included in a single http post request, when touca::post is called. Defaults to 10.

  • post-maxretries Maximum number of consecutive attempts the client library should make to re-submit testresults if initial http post request fails when touca::post is called. Defaults to 2.

  • concurrency-mode Can be one of all-threads and per-thread. Defaults to all-threads. Indicates whether testcase declaration is per thread or shared among all threads of the process. If testcase declaration is for all threads, when a thread calls declare_testcase all other threads also have their most recent testcase changed to the newly declared testcase and any future call to logging functions like add_result will affect the newly declared testcase.

The most common pattern for configuring the client is to set configuration parameters api-url and version as shown below, while providing TOUCA_API_KEY as an environment variable.

touca::configure({
    { "api-url": "https://api.touca.io/@/your-team/your-suite" },
    { "version": "4.2.0" }
});

It is possible to provide your API key as a value for api-key configuration parameter instead of setting an environment variable. We advise against doing so. API key should be considered as sensitive user information and should not be hard-coded.

It is also possible to provide api-url in short format and to separately specify team slug and suite slug as shown below.

touca::configure({
    { "api-key": "03dda763-62ea-436f-8395-f45296e56e4b" },
    { "api-url": "https://api.touca.io" },
    { "team": "your-team" },
    { "suite": "your-suite" },
    { "version": "4.2.0" }
});

When api-key and api-url are provided, the configuration process performs authentication to Touca server, preparing for submission of results when touca::post is called in the future. In case you do not intend to submit any result to the server, you can opt not to provide any of api-key and api-url parameters and to use the following pattern instead.

touca::configure({
    { "team": "some-team" },
    { "suite": "some-suite" },
    { "version": "4.2.0" }
});

Parameters

opts – a string-based map of configuration parameters

void touca::configure(const std::string &path)

Configures the touca client using a configuration file in specified filesystem path.

Convenience function that configures client based on a a given configuration file. Expects the configuration file to be in json format. The configuration file must include a top-level property touca whose value is an object describing configuration parameters of the client. Formal specification of the expected json file is shown below.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "https://touca.io/schemas/sdk/cpp/config-file.schema.json",
  "definitions": {
    "touca": {
      "properties": {
        "api-key": {
          "type": "string"
        },
        "api-url": {
          "type": "string"
        },
        "concurrency-mode": {
          "type": "string"
        },
        "post-testcases": {
          "maximum": 10,
          "minimum": 1,
          "multipleOf": 1.0,
          "type": "integer"
        },
        "post-maxretries": {
          "maximum": 100,
          "minimum": 0,
          "multipleOf": 1.0,
          "type": "integer"
        },
        "team": {
          "type": "string"
        },
        "suite": {
          "type": "string"
        },
        "version": {
          "type": "string"
        }
      },
      "type": "object"
    }
  },
  "additionalProperties": true,
  "properties": {
    "touca": {
      "$ref": "#/definitions/touca"
    }
  },
  "required": [
    "touca"
  ],
  "type": "object"
}

See

configure for more information about individual configuration parameters

Since

v1.1

Parameters

path – path to a configuration file with json format

bool touca::is_configured()

Checks if the client is configured to perform basic operations.

Client is considered configured if it can capture test results and store them locally on the filesystem. The following configuration parameters shall be provided, directly or indirectly, together in a single call, or separately in a sequence of calls, in order for the client to be considered as configured.

  • team

  • suite

  • version

The configuration parameters above may be provided indirectly, in part or in full, as components of the configuration parameter api-url.

See

configure for a list of permissible configuration parameters

Warning

In addition to the configuration parameters above, the parameters api-url and api-key shall be provided for the client to be able to submit captured test results to the server.

Returns

true if the client is properly configured

std::string touca::configuration_error()

Provides the most recent error, if any, encountered during client configuration.

Returns

short description of the most recent configuration error

void touca::add_logger(const std::shared_ptr<touca::logger> logger)

registers a custom logger that is notified when an event of potential interest takes place.

This function enables users to register their own logger derived from touca::logger and listen for log events generated by the client library. Log events include warnings and errors if client library is misused or fails to perform an instructed action. Adding a logger is not necessary to use the client library.

Parameters

logger – a custom logger derived from touca::logger that is notified of log events generated by the client library.

Declaring Testcases

void touca::declare_testcase(const std::string &name)

Declares name of the testcase to which all subsequent results will be submitted until a new testcase is declared.

if configuration parameter concurrency-mode is set to all-threads, when a thread calls declare_testcase all other threads also have their most recent testcase changed to the newly declared one. Otherwise, each thread will submit to its own testcase.

Parameters

name – name of the testcase to be declared

void touca::forget_testcase(const std::string &name)

Removes all logged information associated with a given testcase.

Removes from memory, all information that is logged for the previously-declared testcase, for all threads, regardless of the concurrency-mode configuration parameter. This function does not remove testcase results from the server, in case they are already submitted. It clears all information about that testcase from the client library such that switching back to an already-declared or already-submitted testcase would behave similar to when that testcase was first declared. Calling this function is useful in long-running regression test frameworks, after submission of testcase to the server, if memory consumed by the client library is a concern or if there is a risk that a future testcase with a similar name may be executed.

Parameters

name – name of the testcase to be removed from memory

Capturing Test Results

template<typename Char, typename Value>
void touca::add_result(Char &&key, const Value &value)

Logs a given value as a test result for the declared testcase and associates it with the specified key.

This function provides the primary interface for adding test results to the declared testcase.

Template Parameters
  • Char – type of string to be associated with the value stored as a result. Expected to be convertible to std::basic_string<char>.

  • Value – original type of value value to be stored as a result in association with given key key.

Parameters
  • key – name to be associated with the logged test result.

  • value – value to be logged as a test result

template<typename Char, typename Value>
void touca::add_assertion(Char &&key, const Value &value)

Logs a given value as an assertion for the declared testcase and associates it with the specified key.

Assertions are a special category of test results that are hardly ever expected to change for a given test case between different versions of the workflow. Assertions are treated differently by the Touca server: The server specially highlights assertions if they are different between two test versions and removes them from user focus if they remain unchanged. Therefore, assertions are particularly helpful for verifying assumptions about input data and their properties.

See

add_result

Template Parameters
  • Char – type of string to be associated with the value stored as an assertion. Expected to be convertible to std::basic_string<char>.

  • Value – original type of value value to be stored as an assertion in association with given key key.

Parameters
  • key – name to be associated with the logged test result.

  • value – value to be logged as an assertion

template<typename Char, typename Value>
void touca::add_array_element(Char &&key, const Value &value)

adds a given element to a list of results for the declared testcase which is associated with the specified key.

May be considered as a helper utility function. This method is particularly helpful to log a list of elements as they are found:

for (const auto number : numbers) {
    if (isPrime(number)) {
        touca::add_array_element("prime numbers", number);
        touca::add_hit_count("number of primes");
    }
}
This pattern can be considered as a syntactic sugar for for the following alternative:
std::vector<unsigned> primes;
for (const auto number : numbers) {
    if (isPrime(number)) {
        primes.emplace_back(number);
    }
}
if (!primes.empty()) {
    touca::add_result("prime numbers", primes);
    touca::add_result("number of primes", primes.size());
}

The elements added to the list are not required to be of the same type. The following code is acceptable:

touca::add_array_element("elements", 42);
touca::add_array_element("elements", "forty three");

See

add_result

Since

v1.1

Template Parameters
  • Char – type of string to be associated with the value stored as an element. Expected to be convertible to std::basic_string<char>.

  • Value – original type of value value to be stored as an element of an array associated with given key key.

Parameters
  • key – name to be associated with the logged test result.

  • value – element to be appended to the array

Throws

std::invalid_argument – if specified key is already associated with a test result whose type is not a derivative of touca::types::array.

void touca::add_hit_count(const std::string &key)

Increments value of key key every time it is executed. creates the key with initial value of one if it does not exist.

May be considered as a helper utility function. This method is particularly helpful to track variables whose values are determined in loops with indeterminate execution cycles:

for (const auto number : numbers) {
    if (isPrime(number)) {
        add_array_element("prime numbers", number);
        add_hit_count("number of primes");
    }
}
This pattern can be considered as a syntactic sugar for for the following alternative:
std::vector<unsigned> primes;
for (const auto number : numbers) {
    if (isPrime(number)) {
        primes.emplace_back(number);
    }
}
if (!primes.empty()) {
    touca::add_result("prime numbers", primes);
    touca::add_result("number of primes", primes.size());
}

Since

v1.1

Parameters

key – name to be associated with the logged test result.

Throws

std::invalid_argument – if specified key is already associated with a test result which was not an integer.

Capturing Metrics

void touca::add_metric(const std::string &key, const unsigned duration)

adds an already obtained performance measurements.

useful for logging a metric that is measured without using the client library.

Since

v1.2.0

Parameters
  • key – name to be associated with the performance metric

  • duration – duration in number of milliseconds

void touca::start_timer(const std::string &key)

starts performance measurement of a given metric.

records the time of invocation of this function, associates it with the given key and awaits a future call to stop_timer with the same key to log the duration as a performance metric.

Since

v1.1

Parameters

key – name to be associated with the performance metric

void touca::stop_timer(const std::string &key)

stops performance measurement of a given metric.

logs a performance metric whose value is the duration between this call and a previous call to start_timer with the same key.

Since

v1.1

Parameters

key – name to be associated with the performance metric

class scoped_timer

a simple class that helps clients log the duration between its instantiation and destruction as a performance metric.

TOUCA_SCOPED_TIMER

convenience macro for logging performance of a function as a performance metric.

Saving Test Results

void touca::save_binary(const std::string &path, const std::vector<std::string> &testcases = {}, const bool overwrite = true)

Stores testresults in binary format in a file of specified path.

Stores testresults assigned to given set of testcases in a file of specified path in binary format. We do not recommend as a general practice for regression test tools to locally store their testresults. This feature may be helpful for special cases such as when regression test tools have to be run in environments that have no access to the Touca server (e.g. running with no network access).

Parameters
  • path – path to file in which testresults should be stored

  • testcases – set of names of testcases whose results should be stored to disk. if given set is empty, all testcases will be stored in the specified file.

  • overwrite – determines whether to overwrite any file that exists in the specified path. Defaults to true.

void touca::save_json(const std::string &path, const std::vector<std::string> &testcases = {}, const bool overwrite = true)

Stores testresults in json format in a file of specified path.

Stores testresults assigned to given set of testcases in a file of specified path in json format.

Parameters
  • path – path to file in which testresults should be stored

  • testcases – set of names of testcases whose results should be stored to disk. if given set is empty, all testcases will be stored in the specified file.

  • overwrite – determines whether to overwrite any file that exists in the specified path. Defaults to true.

Submitting Test Results

bool touca::post()

Submits all testresults recorded so far to Touca server.

posts all testresults of all testcases declared by this client to Touca server in flatbuffers format. Uses the following configuration parameters that are provided during configuration time:

  • api-key: API Key for Authenticating to Touca server.

  • api-url: URL to Touca server API.

  • post-testcases: maximum number of testcases to include in every HTTP post request.

  • post-maxretries: maximum number of attempts to submit testresults in a bundle of testcases until the HTTP post request is successful.

It is possible to call touca::post() multiple times during runtime of the regression test tool. Test cases already submitted to the server whose test results have not changed, will not be resubmitted. It is also possible to add testresults to a testcase after it is submitted to the server. Any subsequent call to touca::post() will resubmit the modified testcase.

Throws

runtime_error – if configuration parameter api-url is not provided during configuration operation.

Returns

true if all testresults are successfully posted to the server.

Extending Touca Type System

template<typename T, typename Enable = void>
struct Conversion

Non-specialized template declaration of conversion logic for handling objects of custom types by the Touca SDK for C++.

Allows users developing regression tools to provide explicit full speciailization of this class that makes it convenient to pass objects of their non-trivial type directly to Touca API functions that accept testresults.

tparam T

type whose handling logic is to be implemented in operator()() member function.

The following example illustrates a spcialization of Conversion for a custom type Date.

struct Date
{
    const unsigned short _year;
    const unsigned short _month;
    const unsigned short _day;
};

template <>
struct touca::convert::Conversion<Date>
{
    std::shared_ptr<types::IType> operator()(const Date& value)
    {
        auto out = std::make_shared<types::Object>("Date");
        out->add("year", value._year);
        out->add("month", value._month);
        out->add("day", value._day);
        return out;
    }
};

You may also opt to leverage convenience macros provided by the Touca SDK for C++ to abstract away the template specialization syntax. To illustrate, the conversion logic shown above can also be declared as the following:

TOUCA_CONVERSION_LOGIC(Date)
{
    TOUCA_CONVERSION_FUNCTION(Date, value)
    {
        auto out = TOUCA_CUSTOM_TYPE("Date");
        out->add("year", value._year);
        out->add("month", value._month);
        out->add("day", value._day);
        return out;
    }
};

Once declared, this specialization allows user to directly pass objects of type Date to Touca server API that accepts test results.

Date date { 1961, 8, 4 };
touca::add_result("Obama's Birthday", date);

Noteworthy, that declaring a conversion logic for Date, enables objects of this type to be used as smaller pieces of even more complex types:

struct Person
{
    const std::string _name;
    const Date _birthday;
};

TOUCA_CONVERSION_LOGIC(Person)
{
    TOUCA_CONVERSION_FUNCTION(Person, value)
    {
        auto out = TOUCA_CUSTOM_TYPE("Person");
        out->add("name", val._name);
        out->add("birthday", val._birthday);
        return out;
    }
};

Person president { "obama", { 1961, 8, 4 } };
touca::add_result("44th", president);

Test Framework

touca/framework.hpp is the main entry-point to the Touca Test Framework for C++. In typical test tools, it is the main header file used in the regression test tool. The Test Framework performs Touca client configuration, testcase declaration, and saving and submitting the test results. As a result, users may not need to include touca/touca.hpp if capturing test results happens from within the code under test and outside the regression test tool.

Basic Types

using touca::framework::Testcase = std::string

An abstract representation of an input to the workflow under test that is expected to trigger a consistent behavior by that workflow.

On Touca, a framework::Testcase can be thought of as a unique identifier for the input to the workflow and not the input itself. This distinction gives users the freedom to choose what constitutes an appropriate input for their workflow. The input may be a file or a large dataset with the same name as the framework::Testcase. The mapping from the framework::Testcase name to the input can be done at the start of the workflow execution.

See

Workflow::execute

Since

v1.2.0

using touca::framework::Errors = std::vector<std::string>

A list of errors encountered during execution of the workflow under test with a given input.

If the list of errors is empty, the workflow execution is determined to have been successful. Otherwise, the framework marks the framework::Testcase as failed and prints the errors, in the error that they were received, to the standard output stream and the application log files.

Since

v1.2.0

using touca::framework::Options = std::unordered_map<std::string, std::string>

A simple container for the configuration parameters used by the application, the Touca Test Framework and the workflow under test.

For simplicity, the configuration parameters are always stored in string format. While these options are accessible from the Workflow instance, the user can choose whether to use them or not.

Since

v1.2.0

Main Function

int touca::framework::main(int argc, char *argv[], Workflow &workflow)

Takes over running your regression test application.

Designed to abstract away many of the common features expected of a regression test tool, such as parsing of command line arguments and configuration files, logging, error handling, managing test results on filesystem and submitting them to the Touca server.

In most typical regression test tools, this function is meant to be called from the application’s main function, using the pattern shown below.

int main(int argc, char* argv[])
{
    MyWorkflow workflow;
    return touca::framework::main(argc, argv, workflow);
}

Where MyWorkflow would be a custom class derived from Workflow and implemented by the test author.

framework::main calls various methods of the provided Workflow instance in the order shown below.

See

Workflow

Since

v1.2.0

Parameters
  • argc – number of arguments provided to your application’s main function.

  • argv – arguments provided to your applications main function.

  • workflow – instance of a class derived from Workflow that describes what the test tool should do for each testcase.

Returns

zero on success and non-zero on failure

Workflow Class

class touca::framework::Workflow

Formulates how a regression test tool using this Test Framework should operate.

An abstract class to be implemented by the test authors. Designed to provide maximum flexibility that allows engineers to tailor the regression test for their workflow under test and its requirements.

See

Suite to learn how to declare and define testcases to run against the workflow under test.

Since

v1.2.0

Subclassed by MyWorkflow

Public Functions

inline virtual std::string describe_options() const

Describes extra command line arguments supported by this test tool.

Shown to the users of the Regression Test Tool if they invoke the test application with argument --help.

Implementing this function is optional.

void add_options(const Options &options)

Merges configuration options options into the set of configuration options of the Workflow _options.

This function is used by the Touca Test Framework to make framework’s own configuration parameters accessible from Workflow::execute and Workflow::skip functions, just in case developers want to use them.

This function is already implemented.

See

Workflow::execute for a full list of configuration options populated by the Touca Test Framework.

Parameters

options – configuration options to be added to _options.

virtual bool parse_options(int argc, char *argv[])

Enables parsing of additional command line arguments.

Allows test authors to parse additional configuration options, specified as command line arguments, that they may want to use either in the implementation of other functions of this class or in the workflow under test.

Implementing this function is optional.

Parameters
  • argc – number of arguments provided to your application’s main function.

  • argv – arguments provided to your applications main function.

Returns

false if extra configuration options could not be correctly parsed which should prompt the application to abort without running the testcases.

inline virtual bool validate_options() const

Enables validation of extra configuration parameters.

Validates workflow-specific options that are either specified in the configuration file, or parsed via Workflow::parse_options.

Implementing this function is optional.

Returns

true if values for extra configuration options meet expectations and requirements of the test authors.

inline virtual std::shared_ptr<LogSubscriber> log_subscriber() const

Registers an instance of LogSubscriber to subscribe to the log events produced by the Touca Test Framework.

Useful for Test Authors who use their own loggers to capture events of the workflow under test. Implementing this function is optional. It allows relaying the log events produced by the framework to any custom logger.

While this function allows registration of a single derived instance of LogSubscriber, developers have control over how that derived class is implemented and may consume any log event by multiple external loggers with different filtering of levels of detail.

Implementing this function is optional.

See

LogSubscriber for an example of how to extract log events produced by the Test Framework.

inline virtual bool initialize()

Enables test authors to perform any one-time initialization, that should be performed prior to running the workflow under test.

Returns

false if workflow-specific initializations failed for any reason that should prompt the application to abort without running the testcases.

virtual std::shared_ptr<Suite> suite() const = 0

Determines the set of testcases to be run against the workflow under test.

Test authors may decide whether to implement their own custom suite by deriving from class Suite or to use one of the general purpose classes offered by the Test Framework.

Implementing this function is required.

See

touca/framework/suites.hpp to learn about the set of ready to use subclasses of Suite.

Returns

An instance of a Suite that contains the set of testcases to run the regression test with.

virtual bool skip(const Testcase &testcase) const

Checks if a given testcase should be skipped by the Test Framework.

Called by the Touca Test Framework for each testcase prior to its execution. This function provides a mechanism for Test Authors to add custom logic to determine whether a given testcase should be excluded from the test.

By default, the test framework will skip a testcase if there is already a binary or a json result file for that testcase in the directory referenced by the output-dir configuration parameter, provided that at least one of the configuration parameters save-as-binary or save-as-json are set to true.

Implementing this function is optional.

See

framework::main for general information about the call order of different functions within this class, by the Test Framework.

Parameters

testcase – unique identifier for the testcase that should be tested for exclusion.

Returns

true if testcase should be excluded

virtual Errors execute(const Testcase &testcase) const = 0

Runs the workflow under test for the input associated with a given testcase.

Intended to invoke the workflow under test with the given testcase. The framework captures any exception thrown by this function and will consider the testcase causing the exception as failed. The framework also captures any content printed to standard output or standard error streams and redirects them to separate files generated in the results directory of the testcase. The framework measures and reports the runtime of this function for each given testcase.

Implementing this function is required.

See

framework::main for general information about the call order of different functions within this class, by the Test Framework.

Parameters

testcase – unique identifier for the testcase that should be tested for exclusion.

Returns

true if workflow under test processed the given testcase as expected.

Suite Class

class touca::framework::Suite

Provides the set of testcases to be used in this regression test tool.

Serves as a container for the set of testcases to be given, one by one, to the workflow under test. Test authors are expected to create an instance of a class derived from Suite to be returned from Workflow::suite.

See

Workflow

Since

v1.2.0

Subclassed by touca::framework::FileSuite, touca::framework::RemoteSuite

Public Functions

inline virtual void initialize()

Populates the set of testcases stored in this object.

To be implemented by the test authors if they choose to create their own Suite. The implementation should use Suite::push to store testcases without the risk of duplication. It is okay if this function throws an exception. Touca Test Framework is supposed to handle any thrown exception.

Throws

std::runtime_error – if we failed to populate list of test cases.

inline std::vector<Testcase>::const_iterator begin() const

This function enables easy iteration, by the Touca Test Framework, over the set of testcases in this suite.

Returns

an iterator to the first testcase in this testsuite.

inline std::vector<Testcase>::const_iterator end() const

This testcase acts as a placeholder; attempting to access it results in undefined behavior.

Returns

an iterator to the testcase following the last testcase in this testsuite.

inline std::size_t size() const
Returns

the number of testcases in this suite.

Available Implementations

touca/framework/suites.hpp provides the following implementations of the abstract class Suite.

class FileSuite : public touca::framework::Suite

Provides entries in each line of a given file as the set of testcases to be used in this workflow.

Skips empty lines and any line starting with two consecutive # characters.

Since

v1.2.0

class RemoteSuite : public touca::framework::Suite

Provides testcases that were used in the baseline version of this workflow.

Queries the Touca server to obtain the list of testcases that are part of the current baseline version of the workflow under test.

Since

v1.2.0

Logging

enum touca::framework::LogLevel

Different levels of detail with which framework log events may be reported.

See

LogSubscriber

Since

v1.2.0

Values:

enumerator Debug

Captures order of framework function calls.

enumerator Info

Captures outcome of the workflow execution for each framework::Testcase.

enumerator Warning

Captures warnings about potentially incorrect behavior of the test application.

enumerator Error

Captures failed framework::Testcase instances and their error messages.

struct touca::framework::LogSubscriber

Allows extraction of log events produced by the Touca Test Framework in a controlled level of detail.

Registering an instance of LogSubscriber to subscribe to the log events of the framework is optional and can be done through overriding and implementing Workflow::log_subscriber.

See

Workflow::log_subscriber

Since

v1.2.0

Public Functions

virtual void log(const LogLevel level, const std::string &event) = 0

Function called by the Touca Test Framework every time it publishes a log event.

Since Workflow allows registering only one LogSubscriber, this function will be called for all events, regardless of their level of detail. We leave it to the test author to decide whether and how to handle log events with higher levels of detail.

Parameters
  • level – verbosity of the log event published by the framework

  • event – log message published by the framework