Am I the only one experiencing ppcoind crashes recently?

Thireus · December 10, 2014, 9:59am

Hi all,

I’ve been running ppcoind on OS X with Peerunity and on Debian. With a 20 days uptime for OS X and about 37 days for Debian.

Yesterday my OS X ppcoind that comes with Peerunity just froze, those are the last logs:

askfor block 8dc257ba0b4f05245f17 1418115832000000 askfor block 8dc257ba0b4f05245f17 1418115952000000 askfor block 8dc257ba0b4f05245f17 1418116072000000 askfor block 8dc257ba0b4f05245f17 1418116192000000 received block 8dc257ba0b4f05245f17 SetBestChain: new best=8dc257ba0b4f05245f17 height=147481 trust=3910403534937878 moneysupply=21925325.456857 ProcessBlock: ACCEPTED 2014-12-09 08:58:54 UTC Flushing wallet.dat Flushed wallet.dat 16ms askfor tx 9443620ba25a915edd0d 0 sending getdata: tx 9443620ba25a915edd0d askfor tx 9443620ba25a915edd0d 1418115599000000 askfor tx 9443620ba25a915edd0d 1418115719000000 askfor tx 9443620ba25a915edd0d 1418115839000000 askfor tx 9443620ba25a915edd0d 1418115959000000 askfor tx 9443620ba25a915edd0d 1418116079000000 askfor tx 9443620ba25a915edd0d 1418116199000000 addUnchecked(): size 0 CTxMemPool::accept() : accepted 9443620ba2 ResendWalletTransactions()

Today, same behavior, but this time on Debian:

getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 2014-12-10 07:37:23 UTC Flushing wallet.dat Flushed wallet.dat 2ms received getdata for: block f7d28f05f1e198c3a0b4 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1

For Debian, here is what the ppcoind says:

$ ps aux | grep ppcoind | grep -v grep user1 20551 2.2 1.8 479956 297288 ? SLl nov.03 1194:39 ./ppcoind

SLl:

S interruptible sleep (waiting for an event to complete) L has pages locked into memory (for real-time and custom IO) l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)

The process is still in this stage, how can I debug it?
Has anyone been having the same issue lately? (could be related to the Full Active Node drop)

Ben · December 10, 2014, 2:37pm

What version of Peerunity’s daemon (peerunityd) are you using?

Thireus · December 10, 2014, 2:54pm

On OS X, peerunity v0.1.1, directly downloaded from peercoin.net.

On debian, ppcoind v0.4.0ppc-4-g5ace24f-beta.

Ben · December 10, 2014, 4:06pm

I have not recently experienced issues with the daemon stalling. Usually, if there’s a failure, it will be fatal and you’ll see that in the logs.

Could you be experiencing intermitent connectivity issues?

Thireus · December 10, 2014, 4:10pm

[quote=“Ben, post:4, topic:3149”]I have not recently experienced issues with the daemon stalling. Usually, if there’s a failure, it will be fatal and you’ll see that in the logs.

Could you be experiencing intermitent connectivity issues?[/quote]

On my OS X it is possible, but definitely not on my Debian.
The ppcoind process on my Debian is still in its zombie state. What would you recommend to debug it, dump its memory? What is weird is that there’s nothing else in the log file…

Edit: Good news on Debian, it looks like a simple “./ppcoind getinfo” unlocked the process.. The process is still running…

2014-12-10 07:37:23 UTC Flushing wallet.dat Flushed wallet.dat 2ms received getdata for: block f7d28f05f1e198c3a0b4 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 askfor block 22b0d04226be17792710 0 sending getdata: block 22b0d04226be17792710 askfor block 22b0d04226be17792710 1418197496000000 received block 22b0d04226be17792710 SetBestChain: new best=22b0d04226be17792710 height=147633 trust=3918627810633874 moneysupply=21927337.806857 ProcessBlock: ACCEPTED received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 received getdata for: block 22b0d04226be17792710 getblocks -1 to 00000000000000000000 limit 500 2014-12-10 07:44:59 UTC Flushing wallet.dat Flushed wallet.dat 1ms askfor block 30e46cf56c317ace654e 0

Thireus · December 10, 2014, 4:41pm

Well, everything seems to look fine now on Debian. ???
The process is still running, SLl does not mean zombie state at all.

Closed.

Ben · December 10, 2014, 4:46pm

I’m going to look in my debug.log file to see if I notice this happening for me, too, but when I look at your log files it appears that you’re receiving multiple sets of data for the same block.

received getdata for: block 77c10ea151ed866ed8d1
received getdata for: block 77c10ea151ed866ed8d1
received getdata for: block 77c10ea151ed866ed8d1

Thireus · December 10, 2014, 4:49pm

[quote=“Ben, post:7, topic:3149”]I’m going to look in my debug.log file to see if I notice this happening for me, too, but when I look at your log files it appears that you’re receiving multiple sets of data for the same block.

received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1[/quote]

Yes, this has been always happening. I thought it was normal :/.

Ben · December 10, 2014, 5:04pm

It may be, I’m going to review my logs now to see if I see indications of the same behavior.

Thireus · December 10, 2014, 5:07pm

Thank you Ben!

Ben · December 10, 2014, 5:22pm

Can you review debug.log on your Debian machine and see if you can find these transactions? I’d like to see if there’s a discrepancy between what I see, and what you see.

ResendWalletTransactions()
askfor block 77c10ea151ed866ed8d1   0
sending getdata: block 77c10ea151ed866ed8d1
askfor block 77c10ea151ed866ed8d1   1418197042000000
askfor block 77c10ea151ed866ed8d1   1418197162000000
askfor block 77c10ea151ed866ed8d1   1418197282000000
askfor block 77c10ea151ed866ed8d1   1418197402000000
askfor block 77c10ea151ed866ed8d1   1418197522000000
askfor block 77c10ea151ed866ed8d1   1418197642000000
received block 77c10ea151ed866ed8d1
SetBestChain: new best=77c10ea151ed866ed8d1  height=147632  trust=3918563771336253  moneysupply=21927337.606857
ProcessBlock: ACCEPTED
2014-12-10 07:38:21 UTC Flushing wallet.dat
Flushed wallet.dat 28ms
getblocks -1 to 00000000000000000000 limit 500
ResendWalletTransactions()
askfor block 22b0d04226be17792710   0
askfor block 30e46cf56c317ace654e   0
sending getdata: block 22b0d04226be17792710
sending getdata: block 30e46cf56c317ace654e
askfor block 22b0d04226be17792710   1418197501000000
askfor block 30e46cf56c317ace654e   1418197501000000
askfor block 22b0d04226be17792710   1418197621000000
askfor block 30e46cf56c317ace654e   1418197621000000
askfor block 22b0d04226be17792710   1418197741000000
askfor block 30e46cf56c317ace654e   1418197741000000
askfor block 22b0d04226be17792710   1418197861000000
askfor block 30e46cf56c317ace654e   1418197861000000
askfor block 22b0d04226be17792710   1418197981000000
askfor block 22b0d04226be17792710   1418198101000000
askfor block 30e46cf56c317ace654e   1418197981000000
askfor block 30e46cf56c317ace654e   1418198101000000
received block 22b0d04226be17792710
SetBestChain: new best=22b0d04226be17792710  height=147633  trust=3918627810633874  moneysupply=21927337.806857
ProcessBlock: ACCEPTED

I noticed that on the machine where I pulled these logs from I’m running Peerunity v0.1.0 – you’d think that I, of all people, would be running a later version… too many machines and versions to keep up with, apparently

Thireus · December 10, 2014, 5:25pm

There you go:

askfor block 77c10ea151ed866ed8d1 0 sending getdata: block 77c10ea151ed866ed8d1 askfor block 77c10ea151ed866ed8d1 1418197041000000 received block 77c10ea151ed866ed8d1 SetBestChain: new best=77c10ea151ed866ed8d1 height=147632 trust=3918563771336253 moneysupply=21927337.606857 ProcessBlock: ACCEPTED received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 getblocks -1 to 00000000000000000000 limit 500 2014-12-10 07:37:23 UTC Flushing wallet.dat Flushed wallet.dat 2ms received getdata for: block f7d28f05f1e198c3a0b4 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 received getdata for: block 77c10ea151ed866ed8d1 askfor block 22b0d04226be17792710 0 sending getdata: block 22b0d04226be17792710 askfor block 22b0d04226be17792710 1418197496000000

Ben · December 10, 2014, 5:33pm

If it’s not an inconvenience, can you do the same on your OSX machine for comparison?

Thireus · December 10, 2014, 5:37pm

There:

askfor block 77c10ea151ed866ed8d1 0 sending getdata: block 77c10ea151ed866ed8d1 askfor block 77c10ea151ed866ed8d1 1418197033000000 askfor block 77c10ea151ed866ed8d1 1418197153000000 askfor block 77c10ea151ed866ed8d1 1418197273000000 askfor block 77c10ea151ed866ed8d1 1418197393000000 askfor block 77c10ea151ed866ed8d1 1418197513000000 askfor block 77c10ea151ed866ed8d1 1418197633000000 askfor block 77c10ea151ed866ed8d1 1418197753000000 received block 77c10ea151ed866ed8d1 SetBestChain: new best=77c10ea151ed866ed8d1 height=147632 trust=3918563771336253 moneysupply=21927337.606857 ProcessBlock: ACCEPTED 2014-12-10 07:38:04 UTC Flushing wallet.dat Flushed wallet.dat 19ms ResendWalletTransactions()

Ben · December 10, 2014, 5:59pm

Is the Debian machine running as a full node? The multiple occurances of received getdata for: block 77c10ea151ed866ed8d1 isn’t something that is replicated in either my log output or your OSX machine’s.

My OSX machine is not running as a full node.

Thireus · December 10, 2014, 6:01pm

[quote=“Ben, post:15, topic:3149”]Is the Debian machine running as a full node? The multiple occurances of received getdata for: block 77c10ea151ed866ed8d1 isn’t something that is replicated in either my log output or your OSX machine’s.

My OSX machine is not running as a full node.[/quote]

Yes my Debian is running as full node.
My OS X machine isn’t full node.