e92036cfcc
Reviewers: teon.banek, buda, mculinovic Reviewed By: teon.banek Subscribers: pullbot Differential Revision: https://phabricator.memgraph.io/D1752
58 lines
1.7 KiB
Markdown
58 lines
1.7 KiB
Markdown
# Memgraph quality assurance
|
|
|
|
Python script used to run quality assurance tests against Memgraph.
|
|
To run the script execute:
|
|
|
|
```
|
|
source ve3/bin/activate
|
|
./qa.py --help
|
|
./qa.py memgraph_V1
|
|
```
|
|
|
|
The script requires one positional parameter that specifies which test suite
|
|
should be executed. All available test suites can be found in the `tests/`
|
|
directory.
|
|
|
|
## openCypher TCK tests
|
|
|
|
The script uses Behave to run Cucumber tests.
|
|
|
|
Some gotchas exist when adding openCypher TCK tests to our QA engine:
|
|
|
|
- In some tests example injection did not work. Behave stores the first row in
|
|
Cucumber tables as headings and the example injection failed to work. To
|
|
correct this behavior, one row was added to tables where injection was used.
|
|
|
|
- Some tests don't have fully defined result ordering. Because the tests rely
|
|
on result order, some tests fail. If you find a flaky test to ignore output
|
|
ordering you should change the tag "the result should be" to "the result
|
|
should be (ignoring element order for lists)".
|
|
|
|
- Behave can't escape character '|' and it throws a parse error. The query was
|
|
then changed and the result was returned with a different name.
|
|
|
|
`Comparability.feature` tests are failing because integers are compared to
|
|
strings what is not allowed in openCypher.
|
|
|
|
## QA engine issues:
|
|
|
|
Comparing tables with ordering doesn't always work, example:
|
|
|
|
```
|
|
ORDER BY x DESC
|
|
| x | y | | x | y |
|
|
| 3 | 2 | | 3 | 1 |
|
|
| 3 | 1 | | 3 | 2 |
|
|
| 1 | 4 | | 1 | 4 |
|
|
```
|
|
|
|
Side effect aren't tracked or verified, example:
|
|
|
|
```
|
|
| +properties | 1 |
|
|
| -properties | 1 |
|
|
```
|
|
|
|
This is because Memgraph currently doesn't give out the list of side effects
|
|
that happend on query execution.
|