7bd45f8714
Reviewers: mferencevic, ipaljak Reviewed By: mferencevic Subscribers: pullbot Differential Revision: https://phabricator.memgraph.io/D2203 |
||
|---|---|---|
| .. | ||
| steps | ||
| tests | ||
| .gitignore | ||
| apollo_runs.yaml | ||
| continuous_integration | ||
| environment.py | ||
| qa.py | ||
| README.md | ||
| requirements.txt | ||
Memgraph quality assurance
Python script used to run quality assurance tests against Memgraph. To run the script execute:
source ve3/bin/activate
./qa.py --help
./qa.py memgraph_V1
The script requires one positional parameter that specifies which test suite
should be executed. All available test suites can be found in the tests/
directory.
openCypher TCK tests
The script uses Behave to run Cucumber tests.
Some gotchas exist when adding openCypher TCK tests to our QA engine:
-
In some tests example injection did not work. Behave stores the first row in Cucumber tables as headings and the example injection failed to work. To correct this behavior, one row was added to tables where injection was used.
-
Some tests don't have fully defined result ordering. Because the tests rely on result order, some tests fail. If you find a flaky test to ignore output ordering you should change the tag "the result should be" to "the result should be (ignoring element order for lists)".
-
Behave can't escape character '|' and it throws a parse error. The query was then changed and the result was returned with a different name.
Comparability.feature tests are failing because integers are compared to
strings what is not allowed in openCypher.
QA engine issues:
Comparing tables with ordering doesn't always work, example:
ORDER BY x DESC
| x | y | | x | y |
| 3 | 2 | | 3 | 1 |
| 3 | 1 | | 3 | 2 |
| 1 | 4 | | 1 | 4 |
Side effect aren't tracked or verified, example:
| +properties | 1 |
| -properties | 1 |
This is because Memgraph currently doesn't give out the list of side effects that happend on query execution.