0%

流量回放的实践--goreplay中间件开发

中间件可以帮助我们实现对数据的复杂操作,goreplay官方描述其支持多种语言的中间件开发,并且给了nodejs的官方库,这里我以python版本的api库为例进行中间件开发。

安装库

1
pip install gor

使用库

实现的功能,把监听serverA返回的responose和镜像serverB的response进行标识,并发送给后端服务,以便做数据对比以及记录跟踪。

middleware.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#! /usr/bin/env python3
# -*- coding: utf-8 -*-

import sys,requests,json,datetime
from gor.middleware import TornadoGor

def log(msg):
"""
Logging to STDERR as STDOUT and STDIN used for data transfer
@type msg: str or byte string
@param msg: Message to log to STDERR
"""
try:
msg = str(msg) + '\n'
except:
pass
sys.stderr.write(msg)
sys.stderr.flush()

def sendResponse(fromType,id,postData):
headers = {
'Content-Type': 'application/json'
}
data = {
'fromType': fromType,
'id': id,
'postData': postData
}
res = requests.post('http://127.0.0.1:5000/getResponse',data=json.dumps(data),headers=headers)
response = res.json()
logTime = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
log("[{}]: fromType={},id={},{}".format(logTime,fromType,id,response['msg']))

def on_request(proxy, msg, **kwargs):
sendResponse(1, msg.id, msg.http)
proxy.on('response', on_response, idx=msg.id, req=msg)

def on_response(proxy, msg, **kwargs):
sendResponse(2,msg.id,msg.http.split('\r\n\r\n')[1])
proxy.on('replay', on_replay, idx=kwargs['req'].id, req=kwargs['req'], resp=msg)

def on_replay(proxy, msg, **kwargs):
# proxy.set_http_header(msg.http, 'tttttttttt', '88888888888888')
sendResponse(3, msg.id, msg.http.split('\r\n\r\n')[1])

if __name__ == '__main__':
proxy = TornadoGor()
proxy.on('request', on_request)
proxy.run()

启用中间件

这里我是先在服务端的nginx里加了个配置,反向代理一个端口到serverA,并用goreplay监听这个端口,来实现的区分数据来源。

1
sudo ./gor  --input-raw :serverAPort  --output-http "http://serverB"  --input-raw-track-respons--output-http-track-response --prettify-http --middleware "python3 middleware.py"