⽀付宝红包
——双11的挑战与应对
蚂蚁⾦服-⽀付核算技术部
邱硕
???? ???????????
????
??
• ????
• ????
• ????
• ?????
• ????
????
•?????
•???????
•??????
•??????
•????
????? ??????
•????????
•???/???????
•??????
•???????
?????
????????
?????????
?????????
???????.?
?????????
?????????
?????????
????
??????????????????
???????
???????
?????????
?????????
????????
???????
????????�� ��� �
biz
data
share
data
�� cluster
server_1 server_2 server_n…
���� cluster
server_1 server_2 server_n…
�
��cluster
server_1 server_2 server_n…
DB_0 DB_1 DB_19…
�� cluster
server_1 server_2 server_n…
���� cluster
server_1 server_2 server_n…
�
� cluster
server_1 server_2 server_n…
DB_0 DB_1 DB_19…
spanner??????????
msgBroker …
�1 �
�
�1 �
�
read_db read_db
??
??
??
??
??(??)
??
???
????
?????
????
??????
??????
?????? ???????
??????——????
start transaction
lock budget record and get its value
(???????????);
subtract budget record by ∆
insert business orders...
commit
start transaction;
lock budget record and get its value;
<???????????>;
rollback;
start transaction;
insert business orders...
subtract budget record by ∆
when value is sufficient
commit
start transaction
insert business orders...
subtract budget record by ∆
when value is sufficient
rollback;
????
????
??
??
??
??
??
??
??????
??????
???????
50 tps —> 800 tps
??????——MySQL patch
MySQL
EXE:update
OK
EXE:commit
start transaction;
add business orders...
subtract budget record by ∆
when value is sufficient
commit
??
??
• DB?????????????
• ?????????
????????
?????
???????
800 tps —> 4000 tps
??????——????
• ?????????????????
• ????DB?????
DB� 2
DB� 1
bid:1
sbid:1
��: 10
sub_budget
bid:1
sbid:2
��: 10
sub_budget
DB� 10
bid:1
sbid:10
��: 10
sub_budget
…
����
��
�id:1��: 100
budget
??????
• ????
• ?????????
????
bid:1
sbid:1
��: 1
sub_budget
bid:1
sbid:2
��: 2
sub_budget
bid:1
sbid:10
��: 0
sub_budget
…
bid:1
sbid:3
��: 1
sub_budget
���� � �
��
�
��
bid:1
sbid:1
��: 4
sub_budget
bid:1
sbid:2
��: 0
sub_budget
bid:1
sbid:10
��: 0
sub_budget
bid:1
sbid:3
��: 0
sub_budget????
• ???????????
• ?????????
• ??????????
• ?????????
??????
• ??????????
?????
DB��1DB��2
DB��1
bid:1
sbid:1
��: 10
sub_budget
bid:1
sbid:2
��: 10
sub_budget
DB��10
bid:1
sbid:10
��: 10
sub_budget
…
bid:1; sbid:1; ��: 10
bid:1; sbid:2; ��: 10
…
bid:1; sbid:10; ��: 10
sub_budget_route
???????
��
��
��
?????
?????
• ??????????
• ??????????
????
• ???? ?????? ???? ??
• ???? ???? ?????? ??
??????
• ??????????
• ??????????
• ??????????
????
• ????????????????
• ????????????????
?????
DB��1DB��2
DB��1
bid:1
sbid:1
��: 10
sub_budget
bid:1
sbid:2
��: 10
sub_budget
DB��10
bid:1
sbid:10
��: 10
sub_budget
…
bid:1; sbid:1; ��: 10
bid:1; sbid:2; ��: 10
…
bid:1; sbid:10; ��: 10
sub_budget_route
???????
��
��
��
?????
?????
????????????50000 tps
???????4000 tps —> ≥10000~50000 tps
Budget
SubBudget_00
SubBudget_01
SubBudget_19
�
SubBudget_20
SubBudget_21
SubBudget_39
�
SubBudget_40
SubBudget_41
SubBudget_59
�
SubBudget_60
SubBudget_61
SubBudget_79
�
SubBudget_80
SubBudget_81
SubBudget_99
�
DB�
���
������
????1
������ ������ ������ ������
????2 ????3 ????4 ????5
????——???????
??50,000 tps
• ???????
• ??????
0
2
4
6
8
10
12
14
16
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
������%
������%
??????500,000+ tps
???? SOA?????????????
????? ?DB???????????
?DB?????-??
CAP??
• Consistency
• C1?????????????
• C2?????????????
• Availability
• ???????????????
??
• Partition-Tolerance
• ?????????
���
�
��
��
�����
�
�
��
CA
��1
���
���
��2 ��4 ��3
��
PA PA PA PA
C-(���
���)
A(����������)
P-(�������
)
??(??)
??
???
????
?????
????
??????
??????
?????? ???????
??????
??1 ??2 ??3 ??4
??5 ??6 ??7 ??8
??9 ??10 ??11 ??12
??13 ??14 ??15 ??16
??17 ??18 ??19 ??20
????
??????
??
???? ??????
???? ?????????
N1×N2×N3
• ??????????
• ????????
• ??????? ????
N1’+ N2’
????
????
??DB??
????
• ??????
• ?????????????????
• ???????????
• ????
• C2C??????????+?????
• ??????
• ??????????????????
• LRU????
• ??????????????????
pc00-03 pc04-07 pc16-19…
coupon_1
coupon_2
coupon_3
user_id_a
template_1
template_2
template_3
CZone
share ?????????
table sharding
????
????
?????
SQL??
• ??????
• ??????????????
• ????????????????
• SQL??
• O(n) —> O(1)
?????????
???????????
????????????
???????????
id=2
cur_amt=5
coupon
id=3
cur_amt=2
coupon
id=1
cur_amt=10
coupon
????16?
?????17?
???
????
??????
??????
????
????
??/???
??
??????
?????
????
??
??
????
???
?????
SQL????
DB patch
????????
SQL??
???? ????? ????
??
??
??
??
????
?? ????
????(???)?50 ->50,000 tps
????? 14 ?
????
• ????????
• ????????????
• ??????????????
????1
CDN?? CDN?? CDN??…
?????????
???
?????
????
??A
DB??
??A
DB??
??A
DB??
??A
DB??
??A
DB??
??B
DB??
mock
???
?????
???
?????
???
?????
???
?????
???
?????
CDN??
…
????N
???
?????
????
??A
DB??
??A
DB??
??A
DB??
??A
DB??
??A
DB??
??B
DB??
mock
???
?????
???
?????
???
?????
???
?????
???
?????
msg
msg
???
???
????????
• ?????
• ?????
• DB????????
• ??????
DB?? ????
• RPC????
• ????????
• ??????
????
????
• ????????
• ????
• ????
• ????
•????????
•???
•?????
•?????
•?????
•????
?????? ??????
•
•????????
•???????
•????????
??
•
0 ????
??
?? ????????
????
•??????
•????????
•??????
•??????
• DB????
•????
• …
•
•????????
•???
•?????
•?????
•?????
•????
?????? ??????
•
•????????
•???????
•????????
??
•
0 ????
??
?? ????????
????
•??????
•????????
•??????
•??????
• DB????
•????
• …
•
??
• ????
• ????
• ????
• ?????
• ????
• ??
• ????
• ????????
• ??
• ???????
• ??????
• ????
• ?????????
• ??
• ????????
0 5 10 15 20 25 30 35 40 45 50
1
2
3
4
5
6
7
8
9
10
0��� � 0���� �
0
1
2
1 2 3 4 5 6 7 8 9 10
�
��� ��� ��� �����
?????
????——????
??1
???? ???? ???? ???? ???? ???? ????????
???
???
????
???
???
???
???
???
??1
??1
??1
??2
??1
??3
??1
??4
??1
??5
??1
??6
??1
??7
??1
??8
??2
????
???
??2
??1
??1
????????
???
0
????
???
????
0
????
0
????
0
????
0
????
0
??1
??1
??1
??2
??1
??3
??1
??4
??1
??5
??1
??6
??1
??7
??1
??8
??2
????
???
??2
??1
????????
• ????
• ??????
• ????????tps
• ????
• ????????tps
• ??????????
?hit-count
• ??????
• ??????hit-count
??
????——??
???
??
??
??
????
????
???
??
????
????
????
100
80
??????
????
?????
?????
??
????
60
?????
• ????
• ?????????????
• ???????????????
?????
????
????——????
????
??????????????
????
??????????????
• ??????????
???????
• ??????????
• ????????
• ??????????
???11??• ???? • ????????????
• ????
• ?????????????
• ?????????????????
• ????
• ???????????????????
• ?????
• ?????(??????)???
• ????
• ????????????????
???
• ?? ???/DB???? ????????
• ???? ?????????? ??????? ???????
• ??? ?????????DB???
• ???? ??????? ???????
???? ??????
