memgraph/tests/qa
Matej Ferencevic 28590aea53 Fix variable expand between same symbol
Summary:
This diff fixes the variable expand operator to work correctly then the start
and destination nodes use the same symbol or when the destination symbol is an
existing symbol.

Previously, the variable expand operator produced paths that were both
completely wrong (they shouldn't have been produced) and nonexistent (they
didn't even exist in the storage). Invalid data was produced because of a
wrong equality check that was introduced in D1703.

This issue was reported externally and the supplied test case was:
```
CREATE (p1:Person {id: 1})-[:KNOWS]->(:Person {id: 2})-[:KNOWS]->(:Person {id: 3})-[:KNOWS]->(:Person {id: 4})-[:KNOWS]->(p1);
MATCH path = (pers:Person {id: 3})-[:KNOWS*2]->(pers) RETURN path;
```

Also, tests have been added so the behavior remains correct.

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2793
2020-07-01 00:17:25 +02:00
..
steps Refactor QA 2018-12-04 12:33:48 +01:00
tests Fix variable expand between same symbol 2020-07-01 00:17:25 +02:00
.gitignore Refactor QA 2018-12-04 12:33:48 +01:00
apollo_runs.yaml Remove distributed 2019-07-17 15:23:42 +02:00
continuous_integration Make memgraph with storage v2 the main binary 2019-12-06 10:51:16 +01:00
environment.py Refactor QA 2018-12-04 12:33:48 +01:00
qa.py Refactor QA 2018-12-04 12:33:48 +01:00
README.md Refactor QA 2018-12-04 12:33:48 +01:00
requirements.txt Removed unnecessary dependencies from QA. 2017-08-02 12:06:42 +02:00

Memgraph quality assurance

Python script used to run quality assurance tests against Memgraph. To run the script execute:

source ve3/bin/activate
./qa.py --help
./qa.py memgraph_V1

The script requires one positional parameter that specifies which test suite should be executed. All available test suites can be found in the tests/ directory.

openCypher TCK tests

The script uses Behave to run Cucumber tests.

Some gotchas exist when adding openCypher TCK tests to our QA engine:

  • In some tests example injection did not work. Behave stores the first row in Cucumber tables as headings and the example injection failed to work. To correct this behavior, one row was added to tables where injection was used.

  • Some tests don't have fully defined result ordering. Because the tests rely on result order, some tests fail. If you find a flaky test to ignore output ordering you should change the tag "the result should be" to "the result should be (ignoring element order for lists)".

  • Behave can't escape character '|' and it throws a parse error. The query was then changed and the result was returned with a different name.

Comparability.feature tests are failing because integers are compared to strings what is not allowed in openCypher.

QA engine issues:

Comparing tables with ordering doesn't always work, example:

ORDER BY x DESC
| x | y |    | x | y |
| 3 | 2 |    | 3 | 1 |
| 3 | 1 |    | 3 | 2 |
| 1 | 4 |    | 1 | 4 |

Side effect aren't tracked or verified, example:

| +properties | 1 |
| -properties | 1 |

This is because Memgraph currently doesn't give out the list of side effects that happend on query execution.