PPC Daemon crash

Hello,

I had ppcoind crash a few times under a high load.

Crashdump:

Program terminated with signal 11, Segmentation fault. #0 std::string::size (this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/basic_string.h:629 629 { return _M_rep()->_M_length; } (gdb) bt full #0 std::string::size (this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/basic_string.h:629 No locals. #1 std::string::compare (this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/basic_string.h:2021 __size = <value optimized out> __r = 0 #2 operator< <char, std::char_traits<char>, std::allocator<char> > ( this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/basic_string.h:2317 No locals. #3 std::less<std::string>::operator() (this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/stl_function.h:230 No locals. #4 std::_Rb_tree<std::string, std::pair<std::string const, std::string>, std::_Select1st<std::pair<std::string const, std::string> >, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::_M_lower_bound ( this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/stl_tree.h:986 No locals. #5 std::_Rb_tree<std::string, std::pair<std::string const, std::string>, std::_Select1st<std::pair<std::string const, std::string> >, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::lower_bound ( this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/stl_tree.h:745 ---Type <return> to continue, or q <return> to quit--- No locals. #6 std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::lower_bound (this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/stl_map.h:701 No locals. #7 std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::operator[] (this=0x7f147dfb5200, __k=...) at /usr/include/c++/4.4/bits/stl_map.h:447 No locals. #8 0x00000000005126a0 in ReadHTTPHeader (stream=<value optimized out>, mapHeadersRet=<value optimized out>) at bitcoinrpc.cpp:2526 strHeader = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xbded78 "content-length"}} strValue = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x5b0e148 "64"}} str = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xcd7a38 "Content-Length: 64\r"}} nColon = 95478088 nLen = 0 ---Type <return> to continue, or q <return> to quit--- #9 0x00000000005129a8 in ReadHTTP (stream=..., mapHeadersRet=..., strMessageRet=...) at bitcoinrpc.cpp:2543 nStatus = 0 nLen = <value optimized out> #10 0x00007f1483806200 in thread_proxy () from /usr/lib/libboost_thread.so.1.42.0 No symbol table info available. #11 0x00007f148262b8ca in start_thread (arg=<value optimized out>) at pthread_create.c:300 __res = <value optimized out> pd = 0x7f147d7b5700 unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139725981308672, -3460540281538488748, 139726063624128, 0, 89438800, 3, 3589731760211482196, 3589567054196758100}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <value optimized out> freesize = <value optimized out> __PRETTY_FUNCTION__ = "start_thread" #12 0x00007f1481be6b6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 No locals. #13 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- No symbol table info available.

At first I thought it was a malformed rpc request but it’s not. It crashes in STL libraries. Maybe thread synchronization?

Running on debian squeeze 2.6.32-5-amd64

Any thoughts?

Was this on D7, or was this a personal instance? Also, we’re you running the v0.3.0 (official release) or was this built from one of the later community builds (v0.3.1 or v 0.3.2)?

You mentioned that request load may be a problem; what was the load like at the time of the crash?

@Mrbickle, have you seen anything like this on Ecoining?

I’m using v0.3.0 version with gbt modifications: https://github.com/TheSeven/ppcoin/commits/master

It happened on D7 and I was actually wrong with the load. Pool server load shouldn’t have impact on ppcoind load. RPC calls are timed by constant intervals.
It crashed maybe 4 times in the period of half year.