[python]requests VS httpx VS aiohttp

前言

前段时间想着把一个python服务的接口逐渐改成异步的，其中用到requests的地方就要改成httpx或者aiohttp，有点好奇异步请求相较于同步请求有哪些提升，遂做了点小实验。
首先有个服务A提供接口，这个接口会停顿1秒，模拟数据库操作。服务B去请求服务A的这个接口，并把响应返回给客户端C。服务B提供4个接口，这4个接口分别用requests、httpx同步、httpx异步和aiohttp去请求服务A。
客户端使用wrk做请求测试。
实现服务A

服务A使用Go编写，用标准库即可完成

1. package main
3. import (
4.         "net/http"
5.         "time"
6. )
9. func main() {
10.         mux := http.NewServeMux()
11.         mux.HandleFunc("GET /a", func(w http.ResponseWriter, r *http.Request) {
12.                 time.Sleep(1 * time.Second)
13.                 w.Write([]byte("ok"))
14.         })
15.         if err := http.ListenAndServe("127.0.0.1:8000", mux); err != nil {
16.                 panic(err)
17.         }
18. }

复制代码

先用wrk直接请求试试，以此作为基准

1. $ wrk -t8 -c1000 -d30s http://127.0.0.1:8000/a
2. Running 30s test @ http://127.0.0.1:8000/a
3.   8 threads and 1000 connections
4.   Thread Stats   Avg      Stdev     Max   +/- Stdev
5.     Latency     1.00s     7.62ms   1.17s    97.34%
6.     Req/Sec   194.17    266.07     1.24k    86.82%
7.   29491 requests in 30.10s, 3.32MB read
8. Requests/sec:    979.85
9. Transfer/sec:    112.91KB

复制代码

服务B: FastAPI

先用FastAPI做服务B试试

1. from fastapi import FastAPI
2. import httpx
3. import aiohttp
4. import uvicorn
5. import requests
7. app = FastAPI()
8. url = "http://127.0.0.1:8000/a"
10. @app.get("/sync1")
11. def sync1():
12.     try:
13.         resp = requests.get(url, timeout=2)
14.         resp.raise_for_status()
15.     except Exception as e:
16.         print(f"sync request failed, {e}")
17.     else:
18.         return resp.text
19.    
20. @app.get("/sync2")
21. def sync2():
22.     try:
23.         with httpx.Client() as client:
24.             resp = client.get(url, timeout=2)
25.             resp.raise_for_status()
26.     except Exception as e:
27.         print(f"sync2 request failed, {e}")
28.     else:
29.         return resp.text
30.    
31. @app.get("/async1")
32. async def async1():
33.     try:
34.         async with httpx.AsyncClient(timeout=2) as client:
35.             resp = await client.get(url)
36.             resp.raise_for_status()
37.     except Exception as e:
38.         print(f"async1 request failed, {e}")
39.     else:
40.         return resp.text
41.    
42. @app.get("/async2")
43. async def async2():
44.     try:
45.         async with aiohttp.ClientSession() as session:
46.             async with session.get(url, timeout=2) as resp:
47.                 resp.raise_for_status()
48.                 content = await resp.text()
49.     except Exception as e:
50.         print(f"async2 request failed, {e}")
51.     else:
52.         return content
53.    
54. if __name__ == "__main__":
55.     uvicorn.run("server1:app", host="127.0.0.1", port=8001, workers=4, access_log=False)

复制代码

wrk请求结果。httpx不仅同步请求性能不如requests，没想到连异步请求性能也不如requests。而aiohttp以五倍多第二名的性能冠绝群雄。
APITotal requestQPStimeoutcomment/sync14640154.154480requests 同步请求/sync23631120.873570httpx 同步请求/async14313143.404254httpx 异步请求/async225379843.350aiohttp 异步请求异步比同步性能还差，着实有点费解，遂找大模型问了下，大模型回复说httpx默认配置参数不高，可以额外指定参数，还需要避免反复创建http client。似乎有点道理，但是同步性能不如开箱即用的requests，异步性能不如开箱即用的aiohttp，我为什么还要折腾httpx呢？
服务B: Flask

Flask 2.0 也支持异步接口，但是之前测试性能并不是很好，拉出来一并测试瞧瞧实力。
Flask 版本：3.1.1。因为gunicorn运行异步接口会报错，所以用的flask内置webserver。

1. from flask import Flask
2. import requests
3. import httpx
4. import logging
5. import aiohttp
7. app = Flask(__name__)
8. url = "http://127.0.0.1:8000/a"
10. @app.get("/sync1")
11. def sync1():
12.     try:
13.         resp = requests.get(url, timeout=2)
14.         resp.raise_for_status()
15.     except Exception as e:
16.         print("request failed")
17.         return "request failed"
18.     else:
19.         return resp.text
20.    
21. @app.get("/sync2")
22. def sync2():
23.     try:
24.         with httpx.Client() as client:
25.             resp = client.get(url, timeout=2)
26.             resp.raise_for_status()
27.     except Exception as e:
28.         print("request failed")
29.         return "request failed"
30.     else:
31.         return resp.text
33. @app.get("/async1")
34. async def async1():
35.     try:
36.         async with httpx.AsyncClient(timeout=2) as client:
37.             resp = await client.get(url, timeout=2)
38.             resp.raise_for_status()
39.     except Exception as e:
40.         print("request failed")
41.         return "request failed"
42.     else:
43.         return resp.text
44.    
45. @app.get("/async2")
46. async def async2():
47.     try:
48.         async with aiohttp.ClientSession() as session:
49.             async with session.get(url, timeout=2) as response:
50.                 response.raise_for_status()
51.                 resp = await response.text()
52.     except Exception as e:
53.         print("request failed")
54.         return "request failed"
55.     else:
56.         return resp
57.    
58. if __name__ == "__main__":
59.     werkzeug_logger = logging.getLogger("werkzeug")
60.     werkzeug_logger.disabled = True
61.     app.run(host="127.0.0.1", port=8001)

复制代码

测试结果。看来flask还是跟requests更搭，异步还不如同步。
APITotal requestQPStimeoutcomment/sync113279441.27248requests 同步请求/sync2232477.262323httpx 同步请求/async1233077.462330httpx 异步请求/async28277275.036887aiohttp 异步请求服务B: Sanic

再用Sanic测试一遍

1. from sanic import Sanic
2. from sanic.response import text
3. import requests
4. import httpx
5. import aiohttp
7. app = Sanic(__name__)
8. url = "http://127.0.0.1:8000/a"
10. @app.get("/sync1")
11. def sync1(request):
12.     try:
13.         resp = requests.get(url, timeout=2)
14.         resp.raise_for_status()
15.     except Exception as e:
16.         print(f"sync1 request failed, {e}")
17.     else:
18.         return text(resp.text)
20. @app.get("/sync2")
21. def sync2(request):
22.     try:
23.         with httpx.Client() as client:
24.             response = client.get(url, timeout=2)
25.             response.raise_for_status()
26.     except Exception as e:
27.         print(f"sync2 request failed, {e}")
28.     else:
29.         return text(response.text)
31. @app.get("/async1")
32. async def async1(request):
33.     try:
34.         async with httpx.AsyncClient() as client:
35.             response = await client.get(url, timeout=2)
36.             response.raise_for_status()
37.     except Exception as e:
38.         print(f"async1 request failed, {e}")
39.     else:
40.         return text(response.text)
42. @app.get("/async2")
43. async def async2(request):
44.     try:
45.         async with aiohttp.ClientSession() as session:
46.             async with session.get(url, timeout=2) as response:
47.                 response.raise_for_status()
48.                 content = await response.text()
49.     except Exception as e:
50.         print(f"async2 request failed, {e}")
51.     else:
52.         return text(content)
54. if __name__ == "__main__":
55.     app.run(host="127.0.0.1", port=8001, debug=False, access_log=False, workers=4)

复制代码

可能是我对Sanic了解不多，单就这个测试结果来看，Sanic根本不适合编写同步API。而且使用httpx异步请求的时候有大量报错，wrk结果显示 Non-2xx or 3xx responses: 1244
APITotal requestQPStimeoutcomment/sync1371.2335requests 同步请求/sync2160.5316httpx 同步请求/async15481182.095339httpx 异步请求/async228116934.670aiohttp 异步请求服务B: Go

最后再用Go实现下请求

1. func GetA(w http.ResponseWriter, r *http.Request) {
2.         resp, err := http.Get("http://127.0.0.1:8000/a")
3.         if err != nil {
4.                 log.Println(err)
5.         }
6.         defer resp.Body.Close()
7.         body, err := io.ReadAll(resp.Body)
8.         if err != nil {
9.                 log.Println(err)
10.         }
11.         w.Write(body)
12. }
14. func main() {
15.         mux := http.NewServeMux()
16.         mux.HandleFunc("GET /b", GetA)
17.         if err := http.ListenAndServe("127.0.0.1:8001", mux); err != nil {
18.                 panic(err)
19.         }
20. }

复制代码

测试结果，和直接请求服务A差别不大。

1. $ wrk -t8 -c1000 -d30s http://127.0.0.1:8001/b
2. Running 30s test @ http://127.0.0.1:8001/b
3.   8 threads and 1000 connections
4.   Thread Stats   Avg      Stdev     Max   +/- Stdev
5.     Latency     1.02s    85.69ms   1.66s    96.55%
6.     Req/Sec   210.49    175.05   740.00     63.65%
7.   29000 requests in 30.08s, 3.26MB read
8. Requests/sec:    963.97
9. Transfer/sec:    111.08KB

复制代码

小结

用python编写同步请求还是老老实实用requests，异步接口应该用aiohttp，httpx的性能只能说能用。

来源：程序园用户自行投稿发布，如果侵权，请联系站长删除
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！