How to scale graphite up to X times and reduce cpu consumption twice?

So much customers who are working with Graphite could highlight two significant issues with Graphite:
1) It's coded with Python which pretty slow
2) It's bundled to single CPU core and need clustering for using all cores

I have hit first issue. I'm using Graphite with FastNetMon for awesome visual reports and I'm storing about 40 000 records per second.

So carbon daemon eat so much cpu:

top - 13:58:37 up 128 days, 19:32,  1 user,  load average: 3.88, 3.80, 3.76
Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.5 us,  8.8 sy,  0.0 ni, 66.2 id, 11.6 wa,  0.1 hi,  0.8 si,  0.0 st
KiB Mem:  32933416 total, 31923200 used,  1010216 free,   697700 buffers
KiB Swap:  3905532 total,   564568 used,  3340964 free. 26169476 cached Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAN  
20598 root      20   0 5343724 619232 543572 S 179.0  1.9  80770:18 fastnetmon            
22981 _graphi+  20   0  250300 128484   4756 S  21.3  0.4   0:06.81 carbon-cache          

So We could hit limit with about 200 000 records per second and it's not so fine.

I have found awesome product Go-Carbon. It's drop-in replacement for carbon daemon and it's implemented in Golang (very fast and reliable!).

So I have replaced my existing carbon with go-carbon this way. I'm using Debian 8 and all guide will be about it.

Install it:

wget https://github.com/lomik/go-carbon/releases/download/v0.6/go-carbon_0.6_amd64.deb
dpkg -i go-carbon_0.6_amd64.deb

Then we need to change path to original graphite data files (it's completely back compatible but please keep backup):

Then open file /etc/go-carbon/go-carbon.conf

And change following options:

schemas-file = "/etc/carbon/storage-schemas.conf"
data-dir = "/var/lib/graphite/whisper/"
user = "_graphite"
max-updates-per-second = 500 (because it's standard configuration for graphite: MAX_UPDATES_PER_SECOND = 500)

Then switch off python carbon and start go-carbon:

 /etc/init.d/carbon-cache stop
/etc/init.d/go-carbon start

So you still can use graphite web for reading data! And feed data with pickle of text protocol.  It's completely back compatible.

And it's lightning fast:

top - 13:57:44 up 128 days, 19:32,  1 user,  load average: 3.85, 3.78, 3.75
Tasks: 149 total,   1 running, 148 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.6 us, 12.5 sy,  0.0 ni, 62.0 id, 10.7 wa,  0.1 hi,  1.0 si,  0.0 st
KiB Mem:  32933416 total, 31888556 used,  1044860 free,   697680 buffers
KiB Swap:  3905532 total,   564568 used,  3340964 free. 26161116 cached Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20598 root      20   0 5343724 619232 543572 S 208.1  1.9  80768:40 fastnetmon  
21882 _graphi+  20   0  446480 102512   8028 S  10.7  0.3   6:59.83 go-carbon   

As you can see we got two times CPU usage reduction!  But you could use workers = X and max-cpu = X option for scaling go-carbon over all available cores!

We have awesome picture from author: