| by YoungTimes | No comments

C++多线程编程-分析每个函数的内存占用

最近遇到一个“内存占用不断增加”的问题,从代码上没有分析出任何异常,束手无策只有只有借助第三方工具Google Heap Profiler。Google Heap Profiler是Google开源性能内存检测工具GPerfTools系列工具中的一个,可以帮助我们检测代码中出现的内存泄露,帮助我们了解程序中每一行代码占用了多少内存。

1. Google Heap Profiler的安装

安装环境: Ubuntu 18.04

Google GPerfTools工具的Github: https://github.com/gperftools/gperftools

编译安装:

sudo apt-get install autoconf
bash autogen.sh
./configure 
make
make install

2.准备测试代码

编写如下测试代码:

#include <iostream>
#include <thread>

#include <iostream>
#include <unistd.h>

const size_t m_size = 1024 * 1024;

void memory_allocate_1() {
    char* mem_1 = new char[100 * m_size];

    // do something
    
    delete mem_1;
    mem_1 = nullptr;

    std::cout << "memory allocate 1 finished." << std::endl;
}

void memory_allocate_2() {
    char* mem_2 = new char[200 * m_size];

    // do something
    
    delete mem_2;
    mem_2 = nullptr;

    std::cout << "memory allocate 2 finished." << std::endl;
}

void memory_allocate_3() {
    char* mem_3 = new char[400 * m_size];

    // do something

    delete mem_3;
    mem_3 = nullptr;

    std::cout << "memory allocate 3 finished." << std::endl;
}

int main() {
    memory_allocate_1();

    std::thread t1(memory_allocate_2);
    std::thread t2(memory_allocate_3);
    t1.join();
    t2.join();

    return 0;
}

代码编译:

g++ -std=c++11 -g -pthread -o memory_analyze memory_analyze.cpp

3. 代码内存分析

官方Heap Profiler说明链接:

https://gperftools.github.io/gperftools/heapprofile.html

3.1 链接tcmalloc

Heap Profiler需要与tcmalloc配合使用,所以首先要在编译时链接tcmalloc。

g++ -std=c++11 -g -pthread -ltcmalloc -o memory_analyze memory_analyze.cpp

3.2 生成内存快照

3.2.1 侵入的方式

...

#include <gperftools/heap-profiler.h>

....

int main() {
    HeapProfilerStart("test");

    memory_allocate_1();

    std::thread t1(memory_allocate_2);
    std::thread t2(memory_allocate_3);
    t1.join();
    t2.join();

    HeapProfilerStop();

    return 0;
}

代码编译:

g++ memory_analyze.cpp -ltcmalloc -lprofiler -pthread -o memory_analyze

执行代码:

./memory_analyze 
 Starting tracking the heap
 Dumping heap profile to /tmp/heapprof.0001.heap (200 MB currently in use)
 Dumping heap profile to /tmp/heapprof.0002.heap (400 MB currently in use)
 Dumping heap profile to /tmp/heapprof.0003.heap (Exiting, 1 kB in use)

3.2.2 非侵入的方式

嵌入代码的方式比较灵活,可以针对单个代码片段进行测试。如果要针对整个程序进行分析,则更加简单,只需要链接tcmalloc,执行以下命令即可:

运行命令:$ HEAPPROFILE=/tmp/heapprof  [binary args]
$ env LD_PRELOAD=/usr/local/lib/libtcmalloc.so HEAPPROFILE=/tmp/heapprof  ./memory_analyze

程序输出:

Starting tracking the heap
Dumping heap profile to /tmp/heapprof.0001.heap (200 MB currently in use)
Dumping heap profile to /tmp/heapprof.0002.heap (400 MB currently in use)
Dumping heap profile to /tmp/heapprof.0003.heap (Exiting, 1 kB in use)

Heap Profiler两个配置参数:

HEAP_PROFILE_ALLOCATION_INTERVAL: 程序内存每增长这一数值之后就dump 一次内存,默认是1G(1073741824)

HEAP_PROFILE_INUSE_INTERVAL: 程序如果一次性分配内存超过这个数值dump 默认是100K

3.2.3 非侵入,也无需重新编译程序

env LD_PRELOAD="/usr/local/lib/libtcmalloc.so" ./memory_analyze

4. 内存快照分析

使用pprof分析堆(heap)内存使用,看看函数的内存使用分布。

$ pprof --text ./memory_analyze /tmp/heapprof.0001.heap
Using local file ./memory_analyze.
 Using local file /tmp/heapprof.0001.heap.
 Total: 200.0 MB
    200.0 100.0% 100.0%    200.0 100.0% memory_allocate_2
      0.0   0.0% 100.0%      0.0   0.0% __GI__IO_file_doallocate
      0.0   0.0% 100.0%      0.0   0.0% allocate_dtv
      0.0   0.0% 100.0%      0.0   0.0% std::thread::_S_make_state
      0.0   0.0% 100.0%      0.0   0.0% _IO_new_file_overflow
      0.0   0.0% 100.0%      0.0   0.0% _IO_new_file_xsputn
      0.0   0.0% 100.0%      0.0   0.0% __GI__IO_doallocbuf
      0.0   0.0% 100.0%      0.0   0.0% __GI__IO_fwrite
      0.0   0.0% 100.0%    200.0 100.0% __GI___clone
      0.0   0.0% 100.0%      0.0   0.0% __libc_start_main
      0.0   0.0% 100.0%      0.0   0.0% __pthread_create_2_1
      0.0   0.0% 100.0%      0.0   0.0% _start
      0.0   0.0% 100.0%      0.0   0.0% allocate_stack (inline)
      0.0   0.0% 100.0%      0.0   0.0% main
      0.0   0.0% 100.0%      0.0   0.0% memory_allocate_1
      0.0   0.0% 100.0%    200.0 100.0% start_thread
      0.0   0.0% 100.0%    200.0 100.0% std::__invoke
      0.0   0.0% 100.0%    200.0 100.0% std::__invoke_impl
      0.0   0.0% 100.0%      0.0   0.0% std::__ostream_insert
      0.0   0.0% 100.0%    200.0 100.0% std::error_code::default_error_condition
      0.0   0.0% 100.0%      0.0   0.0% std::operator<< 
      0.0   0.0% 100.0%    200.0 100.0% std::thread::_Invoker::_M_invoke
      0.0   0.0% 100.0%    200.0 100.0% std::thread::_Invoker::operator
      0.0   0.0% 100.0%      0.0   0.0% std::thread::_M_start_thread
      0.0   0.0% 100.0%    200.0 100.0% std::thread::_State_impl::_M_run
      0.0   0.0% 100.0%      0.0   0.0% std::thread::thread

还可以将内存结果可视化,可视化需要先安装graphviz和gv。

sudo apt-get install graphviz
sudo apt-get install gv
$ pprof --gv ./memory_analyze /tmp/heapprof.0002.heap
$ pprof --gv ./memory_analyze /tmp/heapprof.0002.heap

参考材料

http://pkxpp.github.io/2017/03/30/gperftools%E5%AE%89%E8%A3%85/

发表评论