mirror of
https://github.com/LCTT/TranslateProject.git
synced 2024-12-26 21:30:55 +08:00
Xingyu.Wang
GitHub
commit
d4f0432e11
@ -1,513 +0,0 @@
|
||||
[#]: subject: "3 ways to test your API with Python"
|
||||
[#]: via: "https://opensource.com/article/21/9/unit-test-python"
|
||||
[#]: author: "Miguel Brito https://opensource.com/users/miguendes"
|
||||
[#]: collector: "lujun9972"
|
||||
[#]: translator: "Yufei-Yan"
|
||||
[#]: reviewer: " "
|
||||
[#]: publisher: " "
|
||||
[#]: url: " "
|
||||
|
||||
3 ways to test your API with Python
|
||||
======
|
||||
Unit testing can be daunting, but these Python modules will make your
|
||||
life much easier.
|
||||
![Puzzle pieces coming together to form a computer screen][1]
|
||||
|
||||
In this tutorial, you'll learn how to unit test code that performs HTTP requests. In other words, you'll see the art of API unit testing in Python.
|
||||
|
||||
Unit tests are meant to test a single unit of behavior. In testing, a well-known rule of thumb is to isolate code that reaches external dependencies.
|
||||
|
||||
For instance, when testing a code that performs HTTP requests, it's recommended to replace the real call with a fake call during test time. This way, you can unit test it without performing a real HTTP request every time you run the test.
|
||||
|
||||
The question is, _how can you isolate the code?_
|
||||
|
||||
Hopefully, that's what I'm going to answer in this post! I'll not only show you how to do it but also weigh the pros and cons of three different approaches.
|
||||
|
||||
Requirements:
|
||||
|
||||
* [Python 3.8][2]
|
||||
* pytest-mock
|
||||
* requests
|
||||
* flask
|
||||
* responses
|
||||
* VCR.py
|
||||
|
||||
|
||||
|
||||
### Demo app using a weather REST API
|
||||
|
||||
To put this problem in context, imagine that you're building a weather app. This app uses a third-party weather REST API to retrieve weather information for a particular city. One of the requirements is to generate a simple HTML page, like the image below:
|
||||
|
||||
![web page displaying London weather][3]
|
||||
|
||||
The weather in London, OpenWeatherMap. Image is the author's own.
|
||||
|
||||
To get the information about the weather, you must find it somewhere. Fortunately, [OpenWeatherMap][2] provides everything you need through its REST API service.
|
||||
|
||||
_Ok, that's cool, but how can I use it?_
|
||||
|
||||
You can get everything you need by sending a `GET` request to: `https://api.openweathermap.org/data/2.5/weather?q={city_name}&appid={api_key}&units=metric`. For this tutorial, I'll parameterize the city name and settle on the metric unit.
|
||||
|
||||
### Retrieving the data
|
||||
|
||||
To retrieve the weather data, use `requests`. You can create a function that receives a city name as a parameter and returns a JSON. The JSON will contain the temperature, weather description, sunset, sunrise time, and so on.
|
||||
|
||||
The example below illustrates such a function:
|
||||
|
||||
|
||||
```
|
||||
def find_weather_for(city: str) -> dict:
|
||||
"""Queries the weather API and returns the weather data for a particular city."""
|
||||
url = API.format(city_name=city, api_key=API_KEY)
|
||||
resp = requests.get(url)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
The URL is made up of two global variables:
|
||||
|
||||
|
||||
```
|
||||
BASE_URL = "<https://api.openweathermap.org/data/2.5/weather>"
|
||||
API = BASE_URL + "?q={city_name}&appid={api_key}&units=metric"
|
||||
```
|
||||
|
||||
The API returns a JSON in this format:
|
||||
|
||||
|
||||
```
|
||||
{
|
||||
"coord": {
|
||||
"lon": -0.13,
|
||||
"lat": 51.51
|
||||
},
|
||||
"weather": [
|
||||
{
|
||||
"id": 800,
|
||||
"main": "Clear",
|
||||
"description": "clear sky",
|
||||
"icon": "01d"
|
||||
}
|
||||
],
|
||||
"base": "stations",
|
||||
"main": {
|
||||
"temp": 16.53,
|
||||
"feels_like": 15.52,
|
||||
"temp_min": 15,
|
||||
"temp_max": 17.78,
|
||||
"pressure": 1023,
|
||||
"humidity": 72
|
||||
},
|
||||
"visibility": 10000,
|
||||
"wind": {
|
||||
"speed": 2.1,
|
||||
"deg": 40
|
||||
},
|
||||
"clouds": {
|
||||
"all": 0
|
||||
},
|
||||
"dt": 1600420164,
|
||||
"sys": {
|
||||
"type": 1,
|
||||
"id": 1414,
|
||||
"country": "GB",
|
||||
"sunrise": 1600407646,
|
||||
"sunset": 1600452509
|
||||
},
|
||||
"timezone": 3600,
|
||||
"id": 2643743,
|
||||
"name": "London",
|
||||
"cod": 200
|
||||
```
|
||||
|
||||
The data is returned as a Python dictionary when you call `resp.json()`. In order to encapsulate all the details, you can represent them as a `dataclass`. This class has a factory method that gets the dictionary and returns a `WeatherInfo` instance.
|
||||
|
||||
This is good because you keep the representation stable. For example, if the API changes the way it structures the JSON, you can change the logic in just one place, the `from_dict` method. Other parts of the code won't be affected. You can even get information from different sources and combine them in the `from_dict` method!
|
||||
|
||||
|
||||
```
|
||||
@dataclass
|
||||
class WeatherInfo:
|
||||
temp: float
|
||||
sunset: str
|
||||
sunrise: str
|
||||
temp_min: float
|
||||
temp_max: float
|
||||
desc: str
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "WeatherInfo":
|
||||
return cls(
|
||||
temp=data["main"]["temp"],
|
||||
temp_min=data["main"]["temp_min"],
|
||||
temp_max=data["main"]["temp_max"],
|
||||
desc=data["weather"][0]["main"],
|
||||
sunset=format_date(data["sys"]["sunset"]),
|
||||
sunrise=format_date(data["sys"]["sunrise"]),
|
||||
)
|
||||
```
|
||||
|
||||
Now, you'll create a function called `retrieve_weather`. You'll use this function to call the API and return a `WeatherInfo` so you can build your HTML page.
|
||||
|
||||
|
||||
```
|
||||
def retrieve_weather(city: str) -> WeatherInfo:
|
||||
"""Finds the weather for a city and returns a WeatherInfo instance."""
|
||||
data = find_weather_for(city)
|
||||
return WeatherInfo.from_dict(data)
|
||||
```
|
||||
|
||||
Good, you have the basic building blocks for our app. Before moving forward, unit test those functions.
|
||||
|
||||
### 1\. Testing the API using mocks
|
||||
|
||||
[According to Wikipedia][4], a mock object is an object that simulates the behavior of a real object by mimicking it. In Python, you can mock any object using the `unittest.mock` lib that is part of the standard library. To test the `retrieve_weather` function, you can then mock `requests.get` and return static data.
|
||||
|
||||
#### pytest-mock
|
||||
|
||||
For this tutorial, you'll use `pytest` as your testing framework of choice. The `pytest` library is very extensible through plugins. To accomplish our mocking goals, use `pytest-mock`. This plugin abstracts a bunch of setups from `unittest.mock` and makes your testing code very concise. If you are curious, I discuss more about it in [another blog post][5].
|
||||
|
||||
_Ok, enough talking, show me the code._
|
||||
|
||||
Here's a complete test case for the `retrieve_weather` function. This test uses two fixtures: One is the `mocker` fixture provided by the `pytest-mock` plugin. The other one is ours. It's just the static data you saved from a previous request.
|
||||
|
||||
|
||||
```
|
||||
@pytest.fixture()
|
||||
def fake_weather_info():
|
||||
"""Fixture that returns a static weather data."""
|
||||
with open("tests/resources/weather.json") as f:
|
||||
return json.load(f)
|
||||
|
||||
[/code] [code]
|
||||
|
||||
def test_retrieve_weather_using_mocks(mocker, fake_weather_info):
|
||||
"""Given a city name, test that a HTML report about the weather is generated
|
||||
correctly."""
|
||||
# Creates a fake requests response object
|
||||
fake_resp = mocker.Mock()
|
||||
# Mock the json method to return the static weather data
|
||||
fake_resp.json = mocker.Mock(return_value=fake_weather_info)
|
||||
# Mock the status code
|
||||
fake_resp.status_code = HTTPStatus.OK
|
||||
|
||||
mocker.patch("weather_app.requests.get", return_value=fake_resp)
|
||||
|
||||
weather_info = retrieve_weather(city="London")
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
If you run the test, you get the following output:
|
||||
|
||||
|
||||
```
|
||||
============================= test session starts ==============================
|
||||
...[omitted]...
|
||||
tests/test_weather_app.py::test_retrieve_weather_using_mocks PASSED [100%]
|
||||
============================== 1 passed in 0.20s ===============================
|
||||
Process finished with exit code 0
|
||||
```
|
||||
|
||||
Great, your tests pass! But... Life is not a bed of roses. This test has pros and cons. I'll take a look at them.
|
||||
|
||||
#### Pros
|
||||
|
||||
Well, one pro already discussed is that by mocking the API's return, you make your tests easier. Isolate the communication with the API and make the test predictable. It will always return what you want.
|
||||
|
||||
#### Cons
|
||||
|
||||
As for cons, the problem is, what if you don't want to use `requests` anymore and decide to go with the standard library's `urllib`. Every time you change the implementation of `find_weather_for`, you will have to adapt the test. A good test doesn't change when your implementation changes. So, by mocking, you end up coupling your test with the implementation.
|
||||
|
||||
Also, another downside is the amount of setup you have to do before calling the function—at least three lines of code.
|
||||
|
||||
|
||||
```
|
||||
...
|
||||
# Creates a fake requests response object
|
||||
fake_resp = mocker.Mock()
|
||||
# Mock the json method to return the static weather data
|
||||
fake_resp.json = mocker.Mock(return_value=fake_weather_info)
|
||||
# Mock the status code
|
||||
fake_resp.status_code = HTTPStatus.OK
|
||||
...
|
||||
```
|
||||
|
||||
_Can I do better?_
|
||||
|
||||
Yes, please, follow along. I'll see now how to improve it a bit.
|
||||
|
||||
### Using responses
|
||||
|
||||
Mocking `requests` using the `mocker` feature has the downside of having a long setup. A good way to avoid that is to use a library that intercepts `requests` calls and patches them. There is more than one lib for that, but the simplest to me is `responses`. Let's see how to use it to replace `mock`.
|
||||
|
||||
|
||||
```
|
||||
@responses.activate
|
||||
def test_retrieve_weather_using_responses(fake_weather_info):
|
||||
"""Given a city name, test that a HTML report about the weather is generated
|
||||
correctly."""
|
||||
api_uri = API.format(city_name="London", api_key=API_KEY)
|
||||
responses.add(responses.GET, api_uri, json=fake_weather_info, status=HTTPStatus.OK)
|
||||
|
||||
weather_info = retrieve_weather(city="London")
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
Again, this function makes use of our `fake_weather_info` fixture.
|
||||
|
||||
Next, run the test:
|
||||
|
||||
|
||||
```
|
||||
============================= test session starts ==============================
|
||||
...
|
||||
tests/test_weather_app.py::test_retrieve_weather_using_responses PASSED [100%]
|
||||
============================== 1 passed in 0.19s ===============================
|
||||
```
|
||||
|
||||
Excellent! This test pass too. But... It's still not that great.
|
||||
|
||||
#### Pros
|
||||
|
||||
The good thing about using libraries like `responses` is that you don't need to patch `requests` ourselves. You save some setup by delegating the abstraction to the library. However, in case you haven't noticed, there are problems.
|
||||
|
||||
#### Cons
|
||||
|
||||
Again, the problem is, much like `unittest.mock`, your test is coupled to the implementation. If you replace `requests`, your test breaks.
|
||||
|
||||
### 2\. Testing the API using an adapter
|
||||
|
||||
_If by using mocks I couple our tests, what can I do?_
|
||||
|
||||
Imagine the following scenario: Say that you can no longer use `requests`, and you'll have to replace it with `urllib` since it comes with Python. Not only that, you learned the lesson of not coupling test code with implementation, and you want to avoid that in the future. You want to replace `urllib` and not have to rewrite the tests.
|
||||
|
||||
It turns out you can abstract away the code that performs the `GET` request.
|
||||
|
||||
_Really? How?_
|
||||
|
||||
You can abstract it by using an adapter. The adapter is a design pattern used to encapsulate or wrap the interface of other classes and expose it as a new interface. This way, you can change the adapters without changing our code. For example, you can encapsulate the details about `requests` in our `find_weather_for` and expose it via a function that takes only the URL.
|
||||
|
||||
So, this:
|
||||
|
||||
|
||||
```
|
||||
def find_weather_for(city: str) -> dict:
|
||||
"""Queries the weather API and returns the weather data for a particular city."""
|
||||
url = API.format(city_name=city, api_key=API_KEY)
|
||||
resp = requests.get(url)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
Becomes this:
|
||||
|
||||
|
||||
```
|
||||
def find_weather_for(city: str) -> dict:
|
||||
"""Queries the weather API and returns the weather data for a particular city."""
|
||||
url = API.format(city_name=city, api_key=API_KEY)
|
||||
return adapter(url)
|
||||
```
|
||||
|
||||
And the adapter becomes this:
|
||||
|
||||
|
||||
```
|
||||
def requests_adapter(url: str) -> dict:
|
||||
resp = requests.get(url)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
Now it's time to refactor our `retrieve_weather` function:
|
||||
|
||||
|
||||
```
|
||||
def retrieve_weather(city: str) -> WeatherInfo:
|
||||
"""Finds the weather for a city and returns a WeatherInfo instance."""
|
||||
data = find_weather_for(city, adapter=requests_adapter)
|
||||
return WeatherInfo.from_dict(data)
|
||||
```
|
||||
|
||||
So, if you decide to change this implementation to one that uses `urllib`, just swap the adapters:
|
||||
|
||||
|
||||
```
|
||||
def urllib_adapter(url: str) -> dict:
|
||||
"""An adapter that encapsulates urllib.urlopen"""
|
||||
with urllib.request.urlopen(url) as response:
|
||||
resp = response.read()
|
||||
return json.loads(resp)
|
||||
|
||||
[/code] [code]
|
||||
|
||||
def retrieve_weather(city: str) -> WeatherInfo:
|
||||
"""Finds the weather for a city and returns a WeatherInfo instance."""
|
||||
data = find_weather_for(city, adapter=urllib_adapter)
|
||||
return WeatherInfo.from_dict(data)
|
||||
```
|
||||
|
||||
_Ok, how about the tests?_
|
||||
|
||||
To test r`etrieve_weather`, just create a fake adapter that is used during test time:
|
||||
|
||||
|
||||
```
|
||||
@responses.activate
|
||||
def test_retrieve_weather_using_adapter(
|
||||
fake_weather_info,
|
||||
):
|
||||
def fake_adapter(url: str):
|
||||
return fake_weather_info
|
||||
|
||||
weather_info = retrieve_weather(city="London", adapter=fake_adapter)
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
If you run the test you get:
|
||||
|
||||
|
||||
```
|
||||
============================= test session starts ==============================
|
||||
tests/test_weather_app.py::test_retrieve_weather_using_adapter PASSED [100%]
|
||||
============================== 1 passed in 0.22s ===============================
|
||||
```
|
||||
|
||||
#### Pros
|
||||
|
||||
The pro for this approach is that you successfully decoupled your test from the implementation. Use [dependency injection][6] to inject a fake adapter during test time. Also, you can swap the adapter at any time, including during runtime. You did all of this without changing the behavior.
|
||||
|
||||
#### Cons
|
||||
|
||||
The cons are that, since you're using a fake adapter for tests, if you introduce a bug in the adapter you employ in the implementation, your test won't catch it. For example, say that we pass a faulty parameter to `requests`, like this:
|
||||
|
||||
|
||||
```
|
||||
def requests_adapter(url: str) -> dict:
|
||||
resp = requests.get(url, headers=<some broken headers>)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
This adapter will fail in production, and the unit tests won't catch it. But truth to be told, you also have the same problem with the previous approach. That's why you always need to go beyond unit tests and also have integration tests. That being said, consider another option.
|
||||
|
||||
### 3\. Testing the API using VCR.py
|
||||
|
||||
Now it's finally the time to discuss our last option. I have only found about it quite recently, frankly. I've been using mocks for a long time and always had some problems with them. `VCR.py` is a library that simplifies a lot of the tests that make HTTP requests.
|
||||
|
||||
It works by recording the HTTP interaction the first time you run the test as a flat YAML file called a _cassette_. Both the request and the response are serialized. When you run the test for the second time, `VCR.py` will intercept the call and return a response for the request made.
|
||||
|
||||
Now see how to test `retrieve_weather` using `VCR.py below:`
|
||||
|
||||
|
||||
```
|
||||
@vcr.use_cassette()
|
||||
def test_retrieve_weather_using_vcr(fake_weather_info):
|
||||
weather_info = retrieve_weather(city="London")
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
_Wow, is that it? No setup? What is that `@vcr.use_cassette()`?_
|
||||
|
||||
Yes, that's it! There is no setup, just a `pytest` annotation to tell VCR to intercept the call and save the cassette file.
|
||||
|
||||
_What does the cassette file look like?_
|
||||
|
||||
Good question. There's a bunch of things in it. This is because VCR saves every detail of the interaction.
|
||||
|
||||
|
||||
```
|
||||
interactions:
|
||||
\- request:
|
||||
body: null
|
||||
headers:
|
||||
Accept:
|
||||
- '*/*'
|
||||
Accept-Encoding:
|
||||
- gzip, deflate
|
||||
Connection:
|
||||
- keep-alive
|
||||
User-Agent:
|
||||
- python-requests/2.24.0
|
||||
method: GET
|
||||
uri: [https://api.openweathermap.org/data/2.5/weather?q=London\&appid=\][7]<YOUR API KEY HERE>&units=metric
|
||||
response:
|
||||
body:
|
||||
string: '{"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":800,"main":"Clear","description":"clearsky","icon":"01d"}],"base":"stations","main":{"temp":16.53,"feels_like":15.52,"temp_min":15,"temp_max":17.78,"pressure":1023,"humidity":72},"visibility":10000,"wind":{"speed":2.1,"deg":40},"clouds":{"all":0},"dt":1600420164,"sys":{"type":1,"id":1414,"country":"GB","sunrise":1600407646,"sunset":1600452509},"timezone":3600,"id":2643743,"name":"London","cod":200}'
|
||||
headers:
|
||||
Access-Control-Allow-Credentials:
|
||||
- 'true'
|
||||
Access-Control-Allow-Methods:
|
||||
- GET, POST
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Length:
|
||||
- '454'
|
||||
Content-Type:
|
||||
- application/json; charset=utf-8
|
||||
Date:
|
||||
- Fri, 18 Sep 2020 10:53:25 GMT
|
||||
Server:
|
||||
- openresty
|
||||
X-Cache-Key:
|
||||
- /data/2.5/weather?q=london&units=metric
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
version: 1
|
||||
```
|
||||
|
||||
_That's a lot!_
|
||||
|
||||
Indeed! The good thing is that you don't need to care much about it. `VCR.py` takes care of that for you.
|
||||
|
||||
#### Pros
|
||||
|
||||
Now, for the pros, I can list at least five things:
|
||||
|
||||
* No setup code.
|
||||
* Tests remain isolated, so it's fast.
|
||||
* Tests are deterministic.
|
||||
* If you change the request, like by using incorrect headers, the test will fail.
|
||||
* It's not coupled to the implementation, so you can swap the adapters, and the test will pass. The only thing that matters is that you request is the same.
|
||||
|
||||
|
||||
|
||||
#### Cons
|
||||
|
||||
Again, despite the enormous benefits compared to mocking, there are still problems.
|
||||
|
||||
If the API provider changes the format of the data for some reason, the test will still pass. Fortunately, this is not very frequent, and API providers usually version their APIs before introducing such breaking changes. Also, unit tests are not meant to access the external API, so there isn't much to do here.
|
||||
|
||||
Another thing to consider is having end-to-end tests in place. These tests will call the server every time it runs. As the name says, it's a more broad test and slow. They cover a lot more ground than unit tests. In fact, not every project will need to have them. So, in my view, `VCR.py` is more than enough for most people's needs.
|
||||
|
||||
### Conclusion
|
||||
|
||||
This is it. I hope you've learned something useful today. Testing API client applications can be a bit daunting. Yet, when armed with the right tools and knowledge, you can tame the beast.
|
||||
|
||||
You can find the full app on [my GitHub][8].
|
||||
|
||||
* * *
|
||||
|
||||
_This article was originally published on the [author's personal blog][9] and has been adapted with permission._
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/21/9/unit-test-python
|
||||
|
||||
作者：[Miguel Brito][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://opensource.com/users/miguendes
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/puzzle_computer_solve_fix_tool.png?itok=U0pH1uwj (Puzzle pieces coming together to form a computer screen)
|
||||
[2]: https://miguendes.me/how-i-set-up-my-python-workspace
|
||||
[3]: https://opensource.com/sites/default/files/sbzkkiywh.jpeg
|
||||
[4]: https://en.wikipedia.org/wiki/Mock_object
|
||||
[5]: https://miguendes.me/7-pytest-plugins-you-must-definitely-use
|
||||
[6]: https://stackoverflow.com/questions/130794/what-is-dependency-injection
|
||||
[7]: https://api.openweathermap.org/data/2.5/weather?q=London\&appid=\
|
||||
[8]: https://github.com/miguendes/tutorials/tree/master/testing_http
|
||||
[9]: https://miguendes.me/3-ways-to-test-api-client-applications-in-python
|
||||
501
translated/tech/20210921 3 ways to test your API with Python.md
Normal file
501
translated/tech/20210921 3 ways to test your API with Python.md
Normal file
@ -0,0 +1,501 @@
|
||||
[#]: subject: "3 ways to test your API with Python"
|
||||
[#]: via: "https://opensource.com/article/21/9/unit-test-python"
|
||||
[#]: author: "Miguel Brito https://opensource.com/users/miguendes"
|
||||
[#]: collector: "lujun9972"
|
||||
[#]: translator: "Yufei-Yan"
|
||||
[#]: reviewer: " "
|
||||
[#]: publisher: " "
|
||||
[#]: url: " "
|
||||
|
||||
用 Python 测试 API 的 3 种方式
|
||||
=====
|
||||
|
||||
单元测试可能令人生畏，但是这些 Python 模块会使你的生活变得更容易。
|
||||
|
||||
![Puzzle pieces coming together to form a computer screen][1]
|
||||
|
||||
在这个教程中，你将学到如何对执行 HTTP 请求代码的进行单元测试。也就是说，你将看到用 Python 对 API 进行单元测试的艺术。
|
||||
|
||||
单元测试是指对单个行为的测试。在测试中，一个众所周知的经验法则就是隔离那些需要外部依赖的代码。
|
||||
|
||||
比如，当测试一段执行 HTTP 请求的代码时，建议在测试过程中，把真正的调用替换成一个假的的调用。这种情况下，每次运行测试的时候，就可以对它进行单元测试，而不需要执行一个真正的 HTTP 请求。
|
||||
|
||||
问题就是，_怎样才能隔离这些代码？_
|
||||
|
||||
这就是我希望在这篇博文中回答的问题！我不仅会向你展示如果去做，而且也会权衡不同方法之间的优点和缺点。
|
||||
|
||||
要求：
|
||||
|
||||
* [Python 3.8][2]
|
||||
* pytest-mock
|
||||
* requests
|
||||
* flask
|
||||
* responses
|
||||
* VCR.py
|
||||
|
||||
### 使用一个天气状况 REST API 的演示程序
|
||||
|
||||
为了更好的解决这个问题，假设你正在创建一个天气状况的 app。这个 app 使用第三方天气状况 REST API 来检索一个城市的天气信息。其中一个需求是生成一个简单的 HTML 页面，像下面这个图片：
|
||||
|
||||
![web page displaying London weather][3]
|
||||
|
||||
伦敦的天气，OpenWeatherMap。图片是作者自己制作的。
|
||||
|
||||
为了获得天气的信息，必须得去某个地方找。幸运的是，通过 [OpenWeatherMap][2] 的 REST API 服务，可以获得一切需要的信息。
|
||||
|
||||
_好的，很棒，但是我该怎么用呢？_
|
||||
|
||||
通过发送一个 `GET` 请求到：`https://api.openweathermap.org/data/2.5/weather?q={city_name}&appid={api_key}&units=metric`，就可以获得你所需要的所有东西。在这个教程中，我会把城市名字设置成一个参数，并确定公制单位。
|
||||
|
||||
### 检索数据
|
||||
|
||||
使用<ruby>`请求`<rt>requests</rt></ruby>来检索天气数据。可以创建一个接收城市名字作为参数的函数，然后返回一个 JSON。JSON 包含温度，天气状况的描述，日出和日落时间等数据。
|
||||
|
||||
下面的例子演示了这样一个函数：
|
||||
|
||||
|
||||
```
|
||||
def find_weather_for(city: str) -> dict:
|
||||
"""Queries the weather API and returns the weather data for a particular city."""
|
||||
url = API.format(city_name=city, api_key=API_KEY)
|
||||
resp = requests.get(url)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
这个 URL 是由两个全局变量构成：
|
||||
|
||||
```
|
||||
BASE_URL = "<https://api.openweathermap.org/data/2.5/weather>"
|
||||
API = BASE_URL + "?q={city_name}&appid={api_key}&units=metric"
|
||||
```
|
||||
|
||||
API 以这个格式返回了一个 JSON：
|
||||
|
||||
```
|
||||
{
|
||||
"coord": {
|
||||
"lon": -0.13,
|
||||
"lat": 51.51
|
||||
},
|
||||
"weather": [
|
||||
{
|
||||
"id": 800,
|
||||
"main": "Clear",
|
||||
"description": "clear sky",
|
||||
"icon": "01d"
|
||||
}
|
||||
],
|
||||
"base": "stations",
|
||||
"main": {
|
||||
"temp": 16.53,
|
||||
"feels_like": 15.52,
|
||||
"temp_min": 15,
|
||||
"temp_max": 17.78,
|
||||
"pressure": 1023,
|
||||
"humidity": 72
|
||||
},
|
||||
"visibility": 10000,
|
||||
"wind": {
|
||||
"speed": 2.1,
|
||||
"deg": 40
|
||||
},
|
||||
"clouds": {
|
||||
"all": 0
|
||||
},
|
||||
"dt": 1600420164,
|
||||
"sys": {
|
||||
"type": 1,
|
||||
"id": 1414,
|
||||
"country": "GB",
|
||||
"sunrise": 1600407646,
|
||||
"sunset": 1600452509
|
||||
},
|
||||
"timezone": 3600,
|
||||
"id": 2643743,
|
||||
"name": "London",
|
||||
"cod": 200
|
||||
```
|
||||
|
||||
当调用 `resp.json()` 的时候，数据是以 Python 字典的形式返回的。为了封装所有细节，可以用 `dataclass` 来代表。这个类有一个工厂方法，可以获得这个字典并且返回一个 `WeatherInfo` 实例。
|
||||
|
||||
这种办法很好，因为可以保持这种表示方法的稳定。比如，如果 API 改变了 JSON 的结构，就可以在同一个地方修改逻辑，在 `from_dict` 方法中。其他代码不会受影响。你也可以从不同的源获得信息，然后把他们都整合到 `from_dict` 方法中。
|
||||
|
||||
|
||||
```
|
||||
@dataclass
|
||||
class WeatherInfo:
|
||||
temp: float
|
||||
sunset: str
|
||||
sunrise: str
|
||||
temp_min: float
|
||||
temp_max: float
|
||||
desc: str
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "WeatherInfo":
|
||||
return cls(
|
||||
temp=data["main"]["temp"],
|
||||
temp_min=data["main"]["temp_min"],
|
||||
temp_max=data["main"]["temp_max"],
|
||||
desc=data["weather"][0]["main"],
|
||||
sunset=format_date(data["sys"]["sunset"]),
|
||||
sunrise=format_date(data["sys"]["sunrise"]),
|
||||
)
|
||||
```
|
||||
|
||||
现在来创建一个叫做 `retrieve_weather` 的函数。使用这个函数调用 API，然后返回一个 `WeatherInfo`，这样就可创建你自己的 HTML 页面。
|
||||
|
||||
```
|
||||
def retrieve_weather(city: str) -> WeatherInfo:
|
||||
"""Finds the weather for a city and returns a WeatherInfo instance."""
|
||||
data = find_weather_for(city)
|
||||
return WeatherInfo.from_dict(data)
|
||||
```
|
||||
|
||||
很好，我们的 app 现在有一些基础了。在继续之前，对这些函数进行单元测试。
|
||||
|
||||
### 1\. 使用 mock 测试 API
|
||||
|
||||
[根据维基百科][4]，<ruby>模拟对象<rt>mock object</rt></ruby>是通过模仿真实对象来模拟它行为的一个对象。在 Python 中，你可以使用 `unittest.mock` 库来<ruby>模拟<rt>mock</rt></ruby>任何对象，这个库是标准库中的一部分。为了测试 `retrieve_weather` 函数，可以模拟 `requests.get`，然后返回静态数据。
|
||||
|
||||
#### pytest-mock
|
||||
|
||||
在这个教程中，会使用 `pytest` 作为测试框架。通过插件，`pytest` 库是非常具有扩展性的。为了完成我们的模拟目标，要用 `pytest-mock`。这个插件抽象化了大量 `unittest.mock` 中的设置，也会让你的代码更简洁。如果你还好奇的话，我在[另一篇博文中][5]会有更多的讨论。
|
||||
|
||||
_好的，说的够多的了，现在看代码。_
|
||||
|
||||
下面是一个 `retrieve_weather` 函数的完整测试用例。这个测试使用了两个 fixture：一个是由 `pytest-mock` 插件提供的 `mocker` fixture, 还有一个是我们自己的。就是从之前请求中保存的静态数据。
|
||||
|
||||
|
||||
```
|
||||
@pytest.fixture()
|
||||
def fake_weather_info():
|
||||
"""Fixture that returns a static weather data."""
|
||||
with open("tests/resources/weather.json") as f:
|
||||
return json.load(f)
|
||||
|
||||
[/code] [code]
|
||||
|
||||
def test_retrieve_weather_using_mocks(mocker, fake_weather_info):
|
||||
"""Given a city name, test that a HTML report about the weather is generated
|
||||
correctly."""
|
||||
# Creates a fake requests response object
|
||||
fake_resp = mocker.Mock()
|
||||
# Mock the json method to return the static weather data
|
||||
fake_resp.json = mocker.Mock(return_value=fake_weather_info)
|
||||
# Mock the status code
|
||||
fake_resp.status_code = HTTPStatus.OK
|
||||
|
||||
mocker.patch("weather_app.requests.get", return_value=fake_resp)
|
||||
|
||||
weather_info = retrieve_weather(city="London")
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
如果运行这个测试，会获得下面的输出：
|
||||
|
||||
|
||||
```
|
||||
============================= test session starts ==============================
|
||||
...[omitted]...
|
||||
tests/test_weather_app.py::test_retrieve_weather_using_mocks PASSED [100%]
|
||||
============================== 1 passed in 0.20s ===============================
|
||||
Process finished with exit code 0
|
||||
```
|
||||
|
||||
很好，测试通过了！但是...生活并非一帆风顺。这个测试有优点，也有缺点。现在来看一下。
|
||||
|
||||
#### 优点
|
||||
|
||||
好的，有一个之前讨论过的优点就是，通过模拟 API 的返回值，测试变得简单了。将通信和 API 隔离，这样测试就可以预测了。这样总会返回你需要的东西。
|
||||
|
||||
#### 缺点
|
||||
|
||||
对于缺点，问题就是，如果不再想用 `requests` 了，并且决定回到标准库的 `urllib`，怎么办。每次改变 `find_weather_for` 的代码，都得去适配测试。好的测试是，当你修改代码实现的时候，测试时不需要改变的。所以，通过模拟，你最终把测试和实现耦合在了一起。
|
||||
|
||||
而且，另一个不好的方面是你需要在调用函数之前进行大量设置——至少是三行代码。
|
||||
|
||||
|
||||
```
|
||||
...
|
||||
# Creates a fake requests response object
|
||||
fake_resp = mocker.Mock()
|
||||
# Mock the json method to return the static weather data
|
||||
fake_resp.json = mocker.Mock(return_value=fake_weather_info)
|
||||
# Mock the status code
|
||||
fake_resp.status_code = HTTPStatus.OK
|
||||
...
|
||||
```
|
||||
|
||||
_我可以做的更好吗？_
|
||||
|
||||
是的，请继续看。我现在看看怎么改进一点。
|
||||
### 使用 responses
|
||||
|
||||
用 `mocker` 功能模拟 `requests` 有点问题，就是有很多设置。避免这个问题的一个好办法就是使用一个库，可以拦截 `requests` 调用并且给他们打<ruby>补丁<rt>patches</rt></ruby>。有不止一个库可以做这件事，但是对我来说最简单的是 `responses`。我们来看一下怎么用，并且替换 `mock`。
|
||||
|
||||
```
|
||||
@responses.activate
|
||||
def test_retrieve_weather_using_responses(fake_weather_info):
|
||||
"""Given a city name, test that a HTML report about the weather is generated
|
||||
correctly."""
|
||||
api_uri = API.format(city_name="London", api_key=API_KEY)
|
||||
responses.add(responses.GET, api_uri, json=fake_weather_info, status=HTTPStatus.OK)
|
||||
|
||||
weather_info = retrieve_weather(city="London")
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
这个函数再次使用了我们的 `fake_weather_info` fixture。
|
||||
|
||||
然后运行测试：
|
||||
|
||||
|
||||
```
|
||||
============================= test session starts ==============================
|
||||
...
|
||||
tests/test_weather_app.py::test_retrieve_weather_using_responses PASSED [100%]
|
||||
============================== 1 passed in 0.19s ===============================
|
||||
```
|
||||
|
||||
非常好！测试也通过了。但是...并不是那么棒。
|
||||
|
||||
#### 优点
|
||||
|
||||
使用诸如 `responses` 这样的库，好的方面就是不需要再给 `requests` <ruby>打补丁<rt>patch</rt></ruby>。通过将这层抽象交给库，可以减少一些设置。然而，如果你没注意到的话，还是有一些问题。
|
||||
|
||||
#### 缺点
|
||||
|
||||
和 `unittest.mock` 很像，测试和实现再一次耦合了。如果替换 `requests`, 测试就不能用了。
|
||||
|
||||
### 2\. 使用<ruby>适配器<rt>adapter</rt></ruby>测试 API
|
||||
|
||||
_如果用模拟让测试耦合了，我能做什么？_
|
||||
|
||||
设想下面的场景：假如说你不能再用 `requests` 了，而且必须要用 `urllib` 替换，因为这是 Python 自带的。不仅仅是这样，你了解了不要把测试代码和实现耦合，并且你想今后都避免这种情况。你想替换 `urllib`，也不想重写测试了。
|
||||
|
||||
事实证明，你可以抽象出执行 `GET` 请求的代码。
|
||||
|
||||
_真的吗？怎么做？_
|
||||
|
||||
可以使用<ruby>适配器<rt>adapter</rt></ruby>来抽象它。适配器是一种用来封装其他类的接口，并作为新接口暴露出来的一种设计模式。用这种方式，就可以修改适配器而不需要修改代码了。比如，在 `find_weather_for` 函数中，封装关于 `requests` 的所有细节，然后把这部分暴露给只接受 URL 的函数。
|
||||
|
||||
所以，这个：
|
||||
|
||||
```
|
||||
def find_weather_for(city: str) -> dict:
|
||||
"""Queries the weather API and returns the weather data for a particular city."""
|
||||
url = API.format(city_name=city, api_key=API_KEY)
|
||||
resp = requests.get(url)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
变成这样：
|
||||
|
||||
```
|
||||
def find_weather_for(city: str) -> dict:
|
||||
"""Queries the weather API and returns the weather data for a particular city."""
|
||||
url = API.format(city_name=city, api_key=API_KEY)
|
||||
return adapter(url)
|
||||
```
|
||||
|
||||
然后适配器变成这样：
|
||||
|
||||
```
|
||||
def requests_adapter(url: str) -> dict:
|
||||
resp = requests.get(url)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
现在到了重构 `retrieve_weather` 函数的时候：
|
||||
|
||||
|
||||
```
|
||||
def retrieve_weather(city: str) -> WeatherInfo:
|
||||
"""Finds the weather for a city and returns a WeatherInfo instance."""
|
||||
data = find_weather_for(city, adapter=requests_adapter)
|
||||
return WeatherInfo.from_dict(data)
|
||||
```
|
||||
|
||||
所以，如果你决定改为使用 `urllib` 的实现，只要换一下适配器：
|
||||
|
||||
|
||||
```
|
||||
def urllib_adapter(url: str) -> dict:
|
||||
"""An adapter that encapsulates urllib.urlopen"""
|
||||
with urllib.request.urlopen(url) as response:
|
||||
resp = response.read()
|
||||
return json.loads(resp)
|
||||
|
||||
[/code] [code]
|
||||
|
||||
def retrieve_weather(city: str) -> WeatherInfo:
|
||||
"""Finds the weather for a city and returns a WeatherInfo instance."""
|
||||
data = find_weather_for(city, adapter=urllib_adapter)
|
||||
return WeatherInfo.from_dict(data)
|
||||
```
|
||||
|
||||
_好的，那测试怎么做？_
|
||||
|
||||
为了测试 `retrieve_weather`, 只要创建一个在测试过程中使用的假的适配器：
|
||||
|
||||
```
|
||||
@responses.activate
|
||||
def test_retrieve_weather_using_adapter(
|
||||
fake_weather_info,
|
||||
):
|
||||
def fake_adapter(url: str):
|
||||
return fake_weather_info
|
||||
|
||||
weather_info = retrieve_weather(city="London", adapter=fake_adapter)
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
如果运行测试，会获得：
|
||||
|
||||
|
||||
```
|
||||
============================= test session starts ==============================
|
||||
tests/test_weather_app.py::test_retrieve_weather_using_adapter PASSED [100%]
|
||||
============================== 1 passed in 0.22s ===============================
|
||||
```
|
||||
|
||||
#### 优点
|
||||
|
||||
这个方法的优点是可以成功将测试和实现解耦。使用[<ruby>依赖注入<rt>dependency injection</rt></ruby>][6]在测试期间注入一个假的适配器。你也可以在任何时候更换适配器，包括在运行时。这些事情都不会改变任何行为。
|
||||
|
||||
#### 缺点
|
||||
|
||||
缺点就是，因为你在测试中用了假的适配器，如果在实现中往适配器中引入了一个 bug，测试的时候就不会发现。比如说，往 `requests` 传入了一个有问题的参数，像这样：
|
||||
|
||||
```
|
||||
def requests_adapter(url: str) -> dict:
|
||||
resp = requests.get(url, headers=<some broken headers>)
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
在生产环境中，适配器会有问题，而且单元测试没办法发现。但是事实是，之前的方法也会有同样的问题。这就是为什么不仅要单元测试，并且总是要集成测试。也就是说，要考虑另一个选项。
|
||||
|
||||
### 3\. 使用 VCR.py 测试 API
|
||||
|
||||
现在终于到了讨论我们最后一个选项了。诚实地说，我也是最近才发现这个。我用<ruby>模拟<rt>mock</rt></ruby>也很长时间了，而且总是有一些问题。`VCR.py` 是一个库，它可以简化很多 HTTP 请求的测试。
|
||||
|
||||
它的工作原理是将第一次运行测试的 HTTP 交互记录为一个 YAML 文件，叫做 _cassette_。请求和响应都会被序列化。当第二次运行测试的时候，`VCT.py` 将拦截对请求的调用，并且返回一个响应。
|
||||
|
||||
现在看一下下面如何使用 `VCR.py` 测试 `retrieve_weather`：
|
||||
|
||||
|
||||
```
|
||||
@vcr.use_cassette()
|
||||
def test_retrieve_weather_using_vcr(fake_weather_info):
|
||||
weather_info = retrieve_weather(city="London")
|
||||
assert weather_info == WeatherInfo.from_dict(fake_weather_info)
|
||||
```
|
||||
|
||||
_天呐，就这样？没有设置？`@vcr.use_cassette()` 是什么？_
|
||||
|
||||
是的，就这样！没有设置，只要一个 `pytest` 标注告诉 VCR 去拦截调用，然后保存 cassette 文件。
|
||||
|
||||
_cassette 文件是什么样？_
|
||||
|
||||
好问题。这个文件里有很多东西。这是因为 VCR 保存了交互中的所有细节。
|
||||
|
||||
|
||||
```
|
||||
interactions:
|
||||
\- request:
|
||||
body: null
|
||||
headers:
|
||||
Accept:
|
||||
- '*/*'
|
||||
Accept-Encoding:
|
||||
- gzip, deflate
|
||||
Connection:
|
||||
- keep-alive
|
||||
User-Agent:
|
||||
- python-requests/2.24.0
|
||||
method: GET
|
||||
uri: [https://api.openweathermap.org/data/2.5/weather?q=London\&appid=\][7]<YOUR API KEY HERE>&units=metric
|
||||
response:
|
||||
body:
|
||||
string: '{"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":800,"main":"Clear","description":"clearsky","icon":"01d"}],"base":"stations","main":{"temp":16.53,"feels_like":15.52,"temp_min":15,"temp_max":17.78,"pressure":1023,"humidity":72},"visibility":10000,"wind":{"speed":2.1,"deg":40},"clouds":{"all":0},"dt":1600420164,"sys":{"type":1,"id":1414,"country":"GB","sunrise":1600407646,"sunset":1600452509},"timezone":3600,"id":2643743,"name":"London","cod":200}'
|
||||
headers:
|
||||
Access-Control-Allow-Credentials:
|
||||
- 'true'
|
||||
Access-Control-Allow-Methods:
|
||||
- GET, POST
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Length:
|
||||
- '454'
|
||||
Content-Type:
|
||||
- application/json; charset=utf-8
|
||||
Date:
|
||||
- Fri, 18 Sep 2020 10:53:25 GMT
|
||||
Server:
|
||||
- openresty
|
||||
X-Cache-Key:
|
||||
- /data/2.5/weather?q=london&units=metric
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
version: 1
|
||||
```
|
||||
|
||||
_确实很多！_
|
||||
|
||||
真的！好的方面就是你不需要留意它。`VCR.py` 会为你安排好一切。
|
||||
|
||||
#### 优点
|
||||
|
||||
现在看一下优点，我可以至少列出五个：
|
||||
|
||||
* 没有设置代码。
|
||||
* 测试仍然是分离的，所以很快。
|
||||
* 测试是确定的。
|
||||
* 如果你改了请求，比如说用了错误的 header，测试会失败。
|
||||
* 没有与代码实现耦合，所以你可以换适配器，而且测试会通过。唯一有关系的东西就是请求必须是一样的。
|
||||
|
||||
|
||||
#### 缺点
|
||||
|
||||
再与模拟相比较，除了避免了错误，还是有一些问题。
|
||||
|
||||
如果 API 提供者出于某种原因修改了数据格式，测试仍然会通过。幸运的是，这种情况并不经常发生，而且在这种重大改变之前，API 提供者通常会给他们的 API 提供不同版本。
|
||||
|
||||
另一个需要考虑的事情是<ruby>就地<rt>in place</rt></ruby><ruby>端到端<rt>end-to-end</rt></ruby>测试。每次服务器运行的时候，这些测试都会调用。顾名思义，这是一个范围更广、更慢的测试。他们会比单元测试覆盖更多。事实上，并不是每个项目都需要使用它们。所以，就我看来，`VCR.py` 对于大多数人的需求来说都绰绰有余。
|
||||
|
||||
### 总结
|
||||
|
||||
就这么多了。我希望今天你了解了一些有用的东西。测试 API 客户端应用可能会有点吓人。然而，当武装了合适的工具和知识，你就可以驯服这个野兽。
|
||||
|
||||
在[我的 Github][8] 上可以找到完整的 app。
|
||||
|
||||
* * *
|
||||
|
||||
_这篇文章最早发表在[作者的个人博客][9]，并且已得到授权_
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/21/9/unit-test-python
|
||||
|
||||
作者：[Miguel Brito][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[https://github.com/Yufei-Yan](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://opensource.com/users/miguendes
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/puzzle_computer_solve_fix_tool.png?itok=U0pH1uwj (Puzzle pieces coming together to form a computer screen)
|
||||
[2]: https://miguendes.me/how-i-set-up-my-python-workspace
|
||||
[3]: https://opensource.com/sites/default/files/sbzkkiywh.jpeg
|
||||
[4]: https://en.wikipedia.org/wiki/Mock_object
|
||||
[5]: https://miguendes.me/7-pytest-plugins-you-must-definitely-use
|
||||
[6]: https://stackoverflow.com/questions/130794/what-is-dependency-injection
|
||||
[7]: https://api.openweathermap.org/data/2.5/weather?q=London\&appid=\
|
||||
[8]: https://github.com/miguendes/tutorials/tree/master/testing_http
|
||||
[9]: https://miguendes.me/3-ways-to-test-api-client-applications-in-python
|
||||
Loading…
Reference in New Issue
Block a user