怎么理解MySQL中半同步引起Master實例Crash

怎么理解MySQL中半同步引起Master實例Crash，很多新手對此不是很清楚，為了幫助大家解決這個難題，下面小編將為大家詳細講解，有這方面需求的人可以來學習下，希望你能有所收獲。

場景 : 
Crash發生時的數據庫版本: MySQL-5.7.12, 官方標注在5.7.17進行了fix;
開啟半同步的主從架構中, 從庫開啟半同步, 啟動/重啟slave線程導致Master實例Crash;

結論 :
mysql bug, 附上bug單鏈接: https://bugs.mysql.com/bug.php?id=79865

問題描述(摘抄):

Description: From 5.7，semi-sync add Ack_receiver thread for listening slave ack，which use select(). But select() can only listen socket fd between 1 and __FD_SET_SIZE(my os is 1024),  when socket fd is bigger than __FD_SET_SIZE, select() has no effect, and can never get  ack from slave,then semi-sync can't run normally.even more,select() use array store fds, when use FD_SET store fd which is bigger than __FD_SET_SIZE, array will overflow，so mysqld may crash。

主要問題就出在tcp連接的select方法, 通常, 操作系統通過宏FD_SET_SIZE來聲明一個進程中select能操作的文件描述符的最大數據, 然而通常情況下, 這個FD_SET_SIZE的值僅為1024;

實際上, 用epoll或者poll會比較少, select貌似是用的很少的;


問題復現:

準備一套MySQL-5.7.12的主從架構, 開啟半同步:


為了能盡量簡單的啟用大量的文件描述符, 這里利用MyISAM分區表的"特性";


這時候在主庫上連續執行select語句多次(>5);


這時候看一下主庫的文件描述符數量;


那么現在在開啟半同步的從庫上重啟一下slave, 同時tail一下主庫的日志;


在重啟線程幾秒鐘之后, 主庫就發生了Crash;

PS: 在測試的過程中, 多次執行了select語句, 然后確認主庫的半同步狀態也是ON的情況下迅速在從庫上重啟slave, 基本是必現的;
PPS: MyISAM表在open的時候會同時打開所有的分區文件, 所以能比較方便的模擬占用大量文件描述符的情景;
(MyISAM分區表: http://blog.itpub.net/29510932/viewspace-2134679/)
PPPPPPPS:  _(:з」∠)_


附上測試用的腳本與Crash的信息

1. CREATE TABLE `myisam_t` (
   

2.   `id` int(11) DEFAULT NULL
   

3. ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
   

4. /*!50100 PARTITION BY HASH (id)
   

5. PARTITIONS 2000 */

點擊(此處)折疊或打開

2017-04-28T22:10:00.731611+08:00 5092 [Note] Start binlog_dump to master_thread_id(5092) slave_server(13043), pos(, 4)

2017-04-28T22:10:01.648365+08:00 5092 [Note] Start semi-sync binlog_dump to slave (server_id: 13043), pos(, 4)

*** buffer overflow detected ***: /usr/sbin/mysqld terminated

 Backtrace: 

/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7fcdfc7981af]

/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fcdfc81dcf7]

/lib/x86_64-linux-gnu/libc.so.6(+0xf6f10)[0x7fcdfc81bf10]

/lib/x86_64-linux-gnu/libc.so.6(+0xf8c67)[0x7fcdfc81dc67]

/usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver17get_slave_socketsEP6fd_set+0x83)[0x7fcc73d4a493]

/usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver3runEv+0x603)[0x7fcc73d4aaf3]

/usr/lib/mysql/plugin/semisync_master.so(ack_receive_handler+0x19)[0x7fcc73d4aba9]

/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xe90784]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fcdfdf650a4]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fcdfc80d87d]

點擊(此處)折疊或打開

1. 14:10:01 UTC - mysqld got signal 6 ;
   

2. This could be because you hit a bug. It is also possible that this binary
   

3. or one of the libraries it was linked against is corrupt, improperly built,
   

4. or misconfigured. This error can also be caused by malfunctioning hardware.
   

5. Attempting to collect some information that could help diagnose the problem.
   

6. As this is a crash and something is definitely wrong, the information
   

7. collection process might fail.
   

   
   

8. key_buffer_size=8388608
   

9. read_buffer_size=131072
   

10. max_used_connections=5
    

11. max_threads=9999
    

12. thread_count=8
    

13. connection_count=2
    

14. It is possible that mysqld could use up to
    

15. key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21899362 K  bytes of memory
    

16. Hope that's ok; if not, decrease some variables in the equation.
    

    
    

17. Thread pointer: 0x0
    

18. Attempting backtrace. You can use the following information to find out
    

19. where mysqld died. If you see no messages after this, something went
    

20. terribly wrong...
    

21. stack_bottom = 0 thread_stack 0x40000
    

22. /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xe77fec]
    

23. /usr/sbin/mysqld(handle_fatal_signal+0x459)[0x7a7019]
    

24. /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fcdfdf6c8d0]
    

25. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fcdfc75a067]
    

26. /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fcdfc75b448]
    

27. /lib/x86_64-linux-gnu/libc.so.6(+0x731b4)[0x7fcdfc7981b4]
    

28. /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fcdfc81dcf7]
    

29. /lib/x86_64-linux-gnu/libc.so.6(+0xf6f10)[0x7fcdfc81bf10]
    

30. /lib/x86_64-linux-gnu/libc.so.6(+0xf8c67)[0x7fcdfc81dc67]
    

31. /usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver17get_slave_socketsEP6fd_set+0x83)[0x7fcc73d4a493]
    

32. /usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver3runEv+0x603)[0x7fcc73d4aaf3]
    

33. /usr/lib/mysql/plugin/semisync_master.so(ack_receive_handler+0x19)[0x7fcc73d4aba9]
    

34. /usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xe90784]
    

35. /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fcdfdf650a4]
    

36. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fcdfc80d87d]
    

37. The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
    

38. information that should help you find out what is causing the crash.

看完上述內容是否對您有幫助呢？如果還想對相關知識有進一步的了解或閱讀更多相關文章，請關注億速云行業資訊頻道，感謝您對億速云的支持。