Conformance and compliance testing
Conformance testing is an element of conformity assessment, and also known as compliance testing, or type testing - is testing or other activities that determine whether a process, product, or service complies with the requirements of a specification, technical standard, contract, or regulation.
Web standards
WebAssembly
QUIC: A UDP-Based Multiplexed and Secure Transport
HTML (HyperText Markup Language)
WebSocket
WebCL
https://github.com/KhronosGroup/WebCL-conformance
WebRTC
https://webrtc.org/testing/conformance/
ECMAScript
ECMAScript Language Specification - https://tc39.es/ecma262/ test262
RFC 2616: Hypertext Transfer Protocol – HTTP/1.1
RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2)
There are two test suites: http2-test and h2spec.
RFC 4791: Calendaring Extensions to WebDAV (CalDAV)
https://github.com/apple/ccs-caldavtester
RFC 4918: HTTP Extensions for Web Distributed Authoring and Versioning (WebDAV)
http://www.webdav.org/neon/litmus/
RFC 1738: Uniform Resource Locators (URL)
Hypothesis + lark-grammars
RFC 2396: Uniform Resource Identifiers (URI): Generic Syntax
Hypothesis + lark-grammars
RFC 2397: The “data” URL scheme
Hypothesis + lark-grammars
RFC 5545: Internet Calendaring and Scheduling Core Object Specification (iCalendar)
Hypothesis + lark-grammars
Robots Exclusion Protocol
A file robots.txt
used by websites primarily to manage crawler traffic to
site. For a long time format has differences in descriptions provided by Google
and Yandex in their documentations for webmasters. For example Yandex
describes
an option “Crawl-delay” but there is no such option in documentation provided
by Google.
In 2019 Google had an attempt to formalize
the Robots Exclusion Protocol Specification and they published a
draft, that was expired in
Jan 2, 2020. However proposed specification has a grammar that can be used for
testing parsers and producers on robots.txt
files. For such purposes once can
use Hypothesis and lark-grammars or
robotstxt-spec-test.
Structured Query Language (SQL)
- sqllogictest
- sqlancer
- go-sqlancer
- sqltest
- Structured Query Language (SQL) Test Suite
- Hypothesis + lark-grammars
- elliotchance/sqltest - a comprehensive suite of SQL tests for testing the conformance of databases.
POSIX
- AT&T Research regex(3) regression tests
- https://github.com/pjd/pjdfstest
- https://github.com/managarm/mlibc/tree/master/tests
- Open Posix Tests
- libc benchmark
- https://git.musl-libc.org/cgit/libc-testsuite/
- Regex: https://github.com/kokke/tiny-regex-c/tree/master/tests
- IEEE 1003.3-1991 - IEEE Standard for Information Technology - Test Methods for Measuring Conformance to POSIX®
- POSIX regex
- regfuzz
- re_bench
- https://github.com/mpieuchot/libpthread-regress
- https://sortix.org/os-test/
- PCTS:151-2, POSIX Test Suite
- Linux Test Project: https://github.com/linux-test-project/ltp/tree/master/testcases/openposixtestsuite/conformance/interfaces
- malloc benchmark
- mimalloc tests
Video
- XServer XTS - X Test Suite
- OpenGL The SGI OpenGL conformance tests
- OpenGL Piglit
- OpenCL OpenCL conformance tests
- OpenVX OpenVX conformance tests
- OpenGL OpenGL conformance tests, https://www.khronos.org/webgl/wiki/Testing/Conformance
Network
TLS
- tls-attacker/TLS-Anvil - TLS-Anvil, a fully automated TLS testsuite for client and servers.
Samba
Kerberos
RDP (Remote Desktop Protocol)
memcached
A Stateless Transport Tunneling Protocol for Network Virtualization (STT)
RFC 4960: Stream Control Transmission Protocol
Signaling System 7 (SS7)
RFC 4666: Signaling System 7 (SS7) Message Transfer Part 3 (MTP3) - User Adaptation Layer (M3UA)
RFC 3331: Signaling System 7 (SS7) Message Transfer Part 2 (MTP2) - User Adaptation Layer
RFC 3686: Signalling Connection Control Part User Adaptation Layer (SUA)
RFC 4165: Signaling System 7 (SS7) Message Transfer Part 2 (MTP2) - User Peer-to-Peer Adaptation Layer (M2PA)
RFC 3261: SIP: Session Initiation Protocol
RFC 1035: Domain Names - Implementation And Specification
IPSec
- TTCN3 + titan.ProtocolModules.IPsec
TCP (Transmission Control Protocol)
- Some Testing Tools for TCP Implementors
- Network Semantics
- packetdrill testcases
- TTCN-3 and intel/net-test-suites, zephyrproject-rtos/net-tools, echo server
UDP
SMTP
SMTP smtpscript
MQTT
Data Formats
IEEE 754
There is a test suite for IEEE 754R compliance published by IBM. It contains test vectors arranged in files. Each file represents the test vectors for a model described in the document. Documentation, IBM Fpgen Floating-Point Test Generator.
There is a FPSpy, a tool that provide mechanisms for analyzing floating point arithmetic as it executes, making it possible to find and track problematic operations in unmodified x64 application binaries on Linux, including those using thread- and process-level parallelism. FPSpy can observe application behavior without any cooperation from the application or developer, and can potentially be deployed as part of a job launch process. Tool published as a part of research described in “Spying on the Floating Point Behavior of Existing, Unmodified Scientific Applications”. fpspy
Few C programs to test the floating-point arithmetic of your machine: Floating-Point Arithmetic Test Programs
Zoned Device ATA Command Set (ZAC)
https://zonedstorage.io/tests/zbc-tests/
RFC 822: Standard for the Format of ARPA Internet Text Messages
Hypothesis + lark-grammars
TAP (Test Anything Protocol)
TAP 13 TAP 14 Hypothesis + lark-grammars
UTF-8
UTF-8 decoder capability and stress test
YAML
YAML, Hypothesis + lark-grammars
RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format
https://json.org JSON Schema https://github.com/nst/JSONTestSuite JSON-Schema-Test-Suite
https://github.com/json-schema-org/JSON-Schema-Test-Suite
S3
OPDS
OPDS (Open Publication Distribution System) OPDS Test Catalog
Markdown
Markdown is a simple widely-used text markup format. It was intended to be as easy-to-read and easy-to-write as is feasible. Initially it had a limited syntax, that was later was extended by additional syntax (like footnotes, math support etc). There are two testsuites that allows to test parsers and generators of texts with Markdown markup: mdtest and MarkdownTest.
GEDCOM
Hardware
ACPI
RISC
There is a RISC-V compliance testsuite - https://github.com/riscv/riscv-compliance.
There is a MicroTesk. MicroTESK is a reconfigurable and extendable framework for generating test programs for functional verification of microprocessors. MicroTESK allows automatically constructing test program generators based on formal specifications of microprocessor architectures. MicroTESK supports a wide range of architectures including RISC, CISC, VLIW, and DSP. MicroTESK supports online test program generation.
USB (Universal Serial Bus)
- USB 2.0 USB20CV
- USB 2.0 USBHTT
- USB 3.2 USB32CV
- USB TTCN3 + titan.TestPorts.USB
UEFI