API Reference

Client Library

touca/touca.hpp is the main entry-point to the Touca SDK for C++. In most cases, it is the only header-file that users should include in their regression test tool. It provides all the functions necessary to configure the client, declare testcases, capture test results, and submit them to the Touca server. This section documents the API exposed by this header file.

Configuring the Library

void touca::configure(const std::unordered_map<std::string, std::string> &opts = {})

Configures the touca client.

Must be called before declaring testcases and adding results to the client. Should be regarded as a potentially expensive operation. Takes configuration parameters as a list of key value pairs whose keys are from the list below.

  • api-key API Key issued by the Touca server. Instead of passing this parameter to configure, you can set environment variable TOUCA_API_KEY.

  • api-url URL of Touca server API. Can be provided either in long format like @/myteam/mysuite or in short format like . If slug of the team and suite to which the results belong are specified, separately passing team and suite configuration parameters will not be required.

  • version Version of the workflow under test. Required if api-url is not set or if api-url does not include the version.

  • suite Name of the testsuite to which testresults belong. Required if api-url is not set or if api-url is provided in short format.

  • team Name of the team to which this testsuite belongs. Required if api-url is not set or if api-url is provided in short format.

  • offline Disables all communications with the Touca server. Defaults to false.

  • single-thread Restricts scope of the testcase to the declaring thread. Defaults to false. If testcase declaration is for all threads, when a thread calls declare_testcase all other threads also have their most recent testcase changed to the newly declared testcase and any future call to data capturing functions like check will affect the newly declared testcase.

The most common pattern for configuring the client is to set configuration parameters api-url and version as shown below, while providing TOUCA_API_KEY as an environment variable.

touca::configure({
    { "api-url": "https://api.touca.io/@/your-team/your-suite" },
    { "version": "4.2.0" }
});

It is possible to provide your API key as a value for api-key configuration parameter instead of setting an environment variable. We advise against doing so. API key should be considered as sensitive user information and should not be hard-coded.

It is also possible to provide api-url in short format and to separately specify team slug and suite slug as shown below.

touca::configure({
    { "api-key": "03dda763-62ea-436f-8395-f45296e56e4b" },
    { "api-url": "https://api.touca.io" },
    { "team": "your-team" },
    { "suite": "your-suite" },
    { "version": "4.2.0" }
});

When api-key and api-url are provided, the configuration process performs authentication to Touca server, preparing for submission of results when touca::post is called in the future. In case you do not intend to submit any result to the server, you can opt not to provide any of api-key and api-url parameters and to use the following pattern instead.

touca::configure({
    { "team": "some-team" },
    { "suite": "some-suite" },
    { "version": "4.2.0" }
});

Parameters

opts – a string-based map of configuration parameters

void touca::configure(const std::string &path)

Configures the touca client using a configuration file in specified filesystem path.

Convenience function that configures client based on a a given configuration file. Expects the configuration file to be in json format. The configuration file must include a top-level property touca whose value is an object describing configuration parameters of the client.

See also

configure for more information about individual configuration parameters

Since

v1.1

Parameters

path – path to a configuration file with json format

bool touca::is_configured()

Checks if the client is configured to perform basic operations.

Client is considered configured if it can capture test results and store them locally on the filesystem. The following configuration parameters shall be provided, directly or indirectly, together in a single call, or separately in a sequence of calls, in order for the client to be considered as configured.

  • team

  • suite

  • version

The configuration parameters above may be provided indirectly, in part or in full, as components of the configuration parameter api-url.

See also

configure for a list of permissible configuration parameters

Warning

In addition to the configuration parameters above, the parameters api-url and api-key shall be provided for the client to be able to submit captured test results to the server.

Returns

true if the client is properly configured

std::string touca::configuration_error()

Provides the most recent error, if any, encountered during client configuration.

Returns

short description of the most recent configuration error

void touca::add_logger(const std::shared_ptr<touca::logger> logger)

registers a custom logger that is notified when an event of potential interest takes place.

This function enables users to register their own logger derived from touca::logger and listen for log events generated by the client library. Log events include warnings and errors if client library is misused or fails to perform an instructed action. Adding a logger is not necessary to use the client library.

Parameters

logger – a custom logger derived from touca::logger that is notified of log events generated by the client library.

Declaring Testcases

void touca::declare_testcase(const std::string &name)

Declares name of the testcase to which all subsequent results will be submitted until a new testcase is declared.

Unless configuration options single-thread is set, when a thread calls declare_testcase all other threads also have their most recent testcase changed to the newly declared one.

Parameters

name – name of the testcase to be declared

void touca::forget_testcase(const std::string &name)

Removes all logged information associated with a given testcase.

Removes from memory, all information that is logged for the previously-declared testcase, for all threads, regardless of whether configuration option single-thread is set. This function does not remove testcase results from the server, in case they are already submitted. It clears all information about that testcase from the client library such that switching back to an already-declared or already-submitted testcase would behave similar to when that testcase was first declared. Calling this function is useful in long-running regression test frameworks, after submission of testcase to the server, if memory consumed by the client library is a concern or if there is a risk that a future testcase with a similar name may be executed.

Parameters

name – name of the testcase to be removed from memory

Capturing Test Results

template<typename Char, typename Value>
void touca::check(Char &&key, const Value &value)

Logs a given value as a test result for the declared testcase and associates it with the specified key.

This function provides the primary interface for adding test results to the declared testcase.

Template Parameters
  • Char – type of string to be associated with the value stored as a result. Expected to be convertible to std::basic_string<char>.

  • Value – original type of value value to be stored as a result in association with given key key.

Parameters
  • key – name to be associated with the logged test result.

  • value – value to be logged as a test result

template<typename Char, typename Value>
void touca::assume(Char &&key, const Value &value)

Logs a given value as an assumption for the declared testcase and associates it with the specified key.

Assertions are a special category of test results that are hardly ever expected to change for a given test case between different versions of the workflow. Assertions are treated differently by the Touca server: The server specially highlights assumptions if they are different between two test versions and removes them from user focus if they remain unchanged. Therefore, assumptions are particularly helpful for verifying assumptions about input data and their properties.

See also

check

Template Parameters
  • Char – type of string to be associated with the value stored as an assumption. Expected to be convertible to std::basic_string<char>.

  • Value – original type of value value to be stored as an assumption in association with given key key.

Parameters
  • key – name to be associated with the logged test result.

  • value – value to be logged as an assumption

template<typename Char, typename Value>
void touca::add_array_element(Char &&key, const Value &value)

adds a given element to a list of results for the declared testcase which is associated with the specified key.

Could be considered as a helper utility function. This method is particularly helpful to log a list of elements as they are found:

for (const auto number : numbers) {
    if (isPrime(number)) {
        touca::add_array_element("prime numbers", number);
        touca::add_hit_count("number of primes");
    }
}
This pattern can be considered as a syntactic sugar for the following alternative:
std::vector<unsigned> primes;
for (const auto number : numbers) {
    if (isPrime(number)) {
        primes.emplace_back(number);
    }
}
if (!primes.empty()) {
    touca::check("prime numbers", primes);
    touca::check("number of primes", primes.size());
}

The items added to the list are not required to be of the same type. The following code is acceptable:

touca::add_array_element("elements", 42);
touca::add_array_element("elements", "forty three");

See also

check

Since

v1.1

Template Parameters
  • Char – type of string to be associated with the value stored as an element. Expected to be convertible to std::basic_string<char>.

  • Value – original type of value value to be stored as an element of an array associated with given key key.

Parameters
  • key – name to be associated with the logged test result.

  • value – element to be appended to the array

Throws

std::invalid_argument – if specified key is already associated with a test result whose type is not a derivative of touca::array.

void touca::add_hit_count(const std::string &key)

Increments value of key key every time it is executed. creates the key with initial value of one if it does not exist.

May be considered as a helper utility function. This method is particularly helpful to track variables whose values are determined in loops with indeterminate execution cycles:

for (const auto number : numbers) {
    if (isPrime(number)) {
        add_array_element("prime numbers", number);
        add_hit_count("number of primes");
    }
}
This pattern can be considered as a syntactic sugar for the following alternative:
std::vector<unsigned> primes;
for (const auto number : numbers) {
    if (isPrime(number)) {
        primes.emplace_back(number);
    }
}
if (!primes.empty()) {
    touca::check("prime numbers", primes);
    touca::check("number of primes", primes.size());
}

Since

v1.1

Parameters

key – name to be associated with the logged test result.

Throws

std::invalid_argument – if specified key is already associated with a test result which was not an integer.

Capturing Metrics

void touca::add_metric(const std::string &key, const unsigned duration)

adds an already obtained performance measurements.

useful for logging a metric that is measured without using the client library.

Since

v1.2.0

Parameters
  • key – name to be associated with the performance metric

  • duration – duration in number of milliseconds

void touca::start_timer(const std::string &key)

starts performance measurement of a given metric.

records the time of invocation of this function, associates it with the given key and awaits a future call to stop_timer with the same key to log the duration as a performance metric.

Since

v1.1

Parameters

key – name to be associated with the performance metric

void touca::stop_timer(const std::string &key)

stops performance measurement of a given metric.

logs a performance metric whose value is the duration between this call and a previous call to start_timer with the same key.

Since

v1.1

Parameters

key – name to be associated with the performance metric

class scoped_timer

a simple class that helps clients log the duration between its instantiation and destruction as a performance metric.

TOUCA_SCOPED_TIMER

convenience macro for logging performance of a function as a performance metric.

Saving Test Results

void touca::save_binary(const std::string &path, const std::vector<std::string> &testcases = {}, const bool overwrite = true)

Stores testresults in binary format in a file of specified path.

Stores testresults assigned to given set of testcases in a file of specified path in binary format. We do not recommend as a general practice for regression test tools to locally store their testresults. This feature may be helpful for special cases such as when regression test tools have to be run in environments that have no access to the Touca server (e.g. running with no network access).

Parameters
  • path – path to file in which testresults should be stored

  • testcases – set of names of testcases whose results should be stored. if given set is empty, all test cases will be stored in the specified file.

  • overwrite – determines whether to overwrite any file that exists in the specified path. Defaults to true.

void touca::save_json(const std::string &path, const std::vector<std::string> &testcases = {}, const bool overwrite = true)

Stores testresults in json format in a file of specified path.

Stores testresults assigned to given set of testcases in a file of specified path in json format.

Parameters
  • path – path to file in which testresults should be stored

  • testcases – set of names of testcases whose results should be stored to disk. if given set is empty, all testcases will be stored in the specified file.

  • overwrite – determines whether to overwrite any file that exists in the specified path. Defaults to true.

Submitting Test Results

bool touca::post()

Submits all testresults recorded so far to Touca server.

posts all testresults of all testcases declared by this client to Touca server in flatbuffers format. Uses the following configuration parameters that are provided during configuration time:

  • api-key: API Key for Authenticating to Touca server.

  • api-url: URL to Touca server API.

It is possible to call touca::post() multiple times during runtime of the regression test tool. Test cases already submitted to the server whose test results have not changed, will not be resubmitted. It is also possible to add testresults to a testcase after it is submitted to the server. Any subsequent call to touca::post() will resubmit the modified testcase.

Throws

runtime_error – if configuration parameter api-url is not provided during configuration operation.

Returns

true if all testresults are successfully posted to the server.

Extending Touca Type System

template<typename T, typename>
struct serializer

Non-specialized template declaration of conversion logic for handling objects of custom types by the Touca SDK for C++.

Allows users developing regression tools to provide explicit full specialization of this class that makes it convenient to pass objects of their non-trivial type directly to Touca API functions that accept testresults.

The following example illustrates a specialization of serializer for a custom type Date.

struct Date {
  const unsigned short year;
  const unsigned short month;
  const unsigned short day;
};

template <>
struct touca::serializer<Date> {
  data_point serialize(const Date& value) {
    return object("Date")
      .add("year", value.year)
      .add("month", value.month)
      .add("day", value.day);
  }
};

Once declared, this specialization allows user to directly pass objects of type Date to Touca server API that accepts test results.

Date date { 1961, 8, 4 };
touca::check("birthday", date);

Noteworthy, that declaring a conversion logic for Date, enables objects of this type to be used as smaller pieces of even more complex types:

struct Person {
  const std::string name;
  const Date birthday;
};

template <>
struct touca::serializer<Person> {
  data_point serialize(const Person& value) {
    return object("Person")
      .add("name", val.name)
      .add("birthday", val.birthday);
  }
};

Person person { "alex", { 1961, 8, 4 } };
touca::check("person", person);

Template Parameters

T – type whose handling logic is to be implemented in serialize member function.

Test Framework

int touca::run(int argc, char *argv[])
void touca::workflow(const std::string &name, const std::function<void(const std::string&)> &workflow)
struct FrameworkOptions : public touca::ClientOptions