BCC – Dynamic Tracing Tools for Linux Performance Monitoring
2017-11-19 00:00
597 查看
BPF CompilerCollection (BCC)这个工具集包含很多用来观测内核性能的工具,全部使用eBPF,并且提供了python的外部编程能力。本文前序文章: 张亦鸣:
eBPF 简史
本文转载自: https://www.tecmint.com/bcc-best-linux-performance-monitoring-tools/
关注Linuxer,订阅源源不断的一线技术文章
![](https://ss.csdn.net/p?https://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6byuuP1cF7kbteL0anJicn26b6BOP1u6v3HEIxMEgcq7dvOJKRF9AicGjQIIDKXbszXBu2MNSWLLMCagA/0?wx_fmt=gif)
BCC (BPF
Compiler Collection) is a powerful set of appropriate tools and example files for creating resourceful kernel tracing and manipulation programs. It utilizes extended BPF (Berkeley
Packet Filters), initially known as eBPF which
was one of the new features in Linux
3.15.
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8VCZIgVIaIYjwmcmicHiaZiakssUrNibjMDqjnnutCibX8VibOIUicxeicx1VSw/0?wx_fmt=png)
BCC/BPF – Dynamic Tracing Tools for Linux Performance Monitoring
Practically, most of the components used by BCC require Linux
4.1 or above, and its noteworthy features include:
Requires no 3rd party kernel module, since all the tools work based on BPF which is built into the kernel and BCC uses features added in Linux 4.x series.
Enables observation of software execution.
Comprises of several performance analysis tools with example files and man pages.
Best suited for advanced Linux users, BCC makes
it easy to write BPF programs
using kernel instrumentation in C,
and front-ends in Python and lua.
Additionally, it supports multiple tasks such as performance analysis, monitoring, network traffic control plus lots more.
Remember that BCC uses features
added in Linux kernel version 4.1 or
above, and as a requirement, the kernel should have been compiled with the flags set below:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8XgFR8c4rhmV6AJ3pF43zOY287Lv9NvbXnXgHS9YAF20TOn2HXsdCxw/0?wx_fmt=png)
To check your kernel flags, view the file /proc/config.gz or
run the commands as in the examples below:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8LHC8GoCxJRn69piaSVibKIDwacicXq7dNicsafAicXezx3OicVROl2mX53mg/0?wx_fmt=png)
After verifying kernel flags, it’s time to install BCC tools
in Linux systems.
Only the nightly packages are created for Ubuntu
16.04, but the installation instructions are very straightforward. No need of kernel upgrade or compile it from source.
$ echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
$ sudo apt-get update
$ sudo apt-get install bcc-tools
Begin by installing a 4.3+ Linux
kernel, from http://kernel.ubuntu.com/~kernel-ppa/mainline.
As an example, write a small shell script “bcc-install.sh”
with the content below.
Note: update PREFIX value
to the latest date, and also browse the files in the PREFIX
url provided to get the actual REL value,
substitute them in the shell script.
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8ic6RLkbibNe4V52uLibyibusKXZqDECwicTadPfFNlntLRpibLY7o0PicVnHA/0?wx_fmt=png)
Save the file and exit. Make it executable, then run it as shown:
Afterwards, reboot your system.
Next, run the commands below to install signed BCC packages:
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD
$ echo "deb https://repo.iovisor.org/apt trusty main" | sudo tee /etc/apt/sources.list.d/iovisor.list
$ sudo apt-get update
$ sudo apt-get install binutils bcc bcc-tools libbcc-examples python-bcc
All the BCC tools are installed
under
run them from the BCC Github repository
under
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8Qu4MlcCqt83oJXBUbibLB6ZDfPGiaKibR3vfBicGJ965fftBp4QoOE6DKw/0?wx_fmt=png)
We shall cover a few examples under – monitoring general Linux system performance and networking.
Let’s start by tracing all
This enable us tell us how various applications work by identifying their data files, config files and many more:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8lZXVaZeILKWcZbrUpJHyYRiaXYLBfwyoCdBdsMlqeia4mXhv9PQsaiaCg/0?wx_fmt=png)
In this example, it shows a summarized distribution of disk I/O latency using biolatecncy. After executing the command, wait for a few minutes and hit Ctrl-C to
end it and view the output.
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8VtibicvL3DslkfoQOiaPHMTfx6EOp7oNtejg6fhP1ibREJiamm8iamc0RXyw/0?wx_fmt=png)
In this section, we shall move to tracing new processes in execution using execsnoop tool.
Each time a process is forked by
it is shown in the output. However, not all processes are captured.
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8s1qBBTegWTNnuicC7Ot1zNxzCKa1WWhPiaA8SDfSYgBg4kicS2aZ5JqFg/0?wx_fmt=png)
Using ext4slower to trace
the ext4 file system common
operations that are slower than 10ms,
to help us identify independently slow disk I/O via the file system.
It only outputs those operations that exceed a threshold:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8p1XtV6MRzAia76gEGTw5ibhyJQ3z4Yib6vPPvBnXicWek3ibD7bSewxD6OQ/0?wx_fmt=png)
Next off, let’s dive into printing a line per disk I/O each second, with details such as process ID, sector, bytes, latency among others using biosnoop:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU80uBjX20bibNO6raPic8dV0jZGYF655RLsaRwaJy1DiaicSSsBKmTcVs27A/0?wx_fmt=png)
Thereafter, we proceed to using cachestat to
displays one line of summarized statistics from the system cache every second. This enables for system tuning operations by pointing out low cache hit ratio and high rate of misses:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8icOZBFZkxERZhwn3ESBaGTebqDsvxDAERNLjK764SwicM7LrJxMZsNcA/0?wx_fmt=png)
Monitoring TCP connections every second using tcpconnect.
Its output includes source and destination address, and port number. This tool is useful for tracing unexpected TCP connections, thereby helping us to identify inefficiencies in application configurations or an attacker.
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8K8XvibUtpoTJjBB7aQfvLc7g4AMExbyzVTa4r44AtZGgiaxzUWencsxQ/0?wx_fmt=png)
All the tools above can also be used with various options, to enable the help page for a given tool, make use of the
for example:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8Ca1hLXDHibrfBy9jbR6YaVLW24thgxRfxbianuwuxP6NymBNsXvfHXsg/0?wx_fmt=png)
To trace failed exec()s syscalls, employ the
as below:
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8tdDCciacv7icNRqdE3FE2s0WjDxjnGyDMruP43f6PjAWZic1CUnmzEQwg/0?wx_fmt=png)
The last example below demonstrates how to execute a custom trace operation. We are tracing a particular process using its PID.
First determine the process ID:
Later on, run the custom trace command.
In the command below:
a kernel function that is traced dynamically including its second argument as a string.
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_png/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8fONm3NiaYv3O1St216a6iciabVib7mQh93MUrUBcialQTbS0UZRfbnGHSgw/0?wx_fmt=png)
BCC is a powerful and easy-to-use
toolkit for various System administration tasks such as tracing system performance monitoring, tracing block device I/O, TCP functions, file system operations, syscalls, Node.js probes, plus lots more. Importantly, it ships in with several example files and
man pages for the tools to guide you, making it user friendly and reliable.
Last but not least, you can get back to us by sharing your thoughts about the subject, ask questions, make useful suggestions or any constructive feedback via the comment section below.
For more information and usage visit: https://iovisor.github.io/bcc/
![](https://ss.csdn.net/p?http://mmbiz.qpic.cn/mmbiz_gif/Ass1lsY6bytXAGsA0XemHR9YoIhR9DU8JpemeXKtpn6xdzfHdKkmPqAlJERagv1eNtkBjfkvXO0m0uXUPMR9ibg/0?wx_fmt=gif)
更多精彩内容,尽在阅读原文
eBPF 简史
本文转载自: https://www.tecmint.com/bcc-best-linux-performance-monitoring-tools/
关注Linuxer,订阅源源不断的一线技术文章
BCC (BPF
Compiler Collection) is a powerful set of appropriate tools and example files for creating resourceful kernel tracing and manipulation programs. It utilizes extended BPF (Berkeley
Packet Filters), initially known as eBPF which
was one of the new features in Linux
3.15.
BCC/BPF – Dynamic Tracing Tools for Linux Performance Monitoring
Practically, most of the components used by BCC require Linux
4.1 or above, and its noteworthy features include:
Requires no 3rd party kernel module, since all the tools work based on BPF which is built into the kernel and BCC uses features added in Linux 4.x series.
Enables observation of software execution.
Comprises of several performance analysis tools with example files and man pages.
Best suited for advanced Linux users, BCC makes
it easy to write BPF programs
using kernel instrumentation in C,
and front-ends in Python and lua.
Additionally, it supports multiple tasks such as performance analysis, monitoring, network traffic control plus lots more.
How To Install BCC in Linux Systems
Remember that BCC uses featuresadded in Linux kernel version 4.1 or
above, and as a requirement, the kernel should have been compiled with the flags set below:
To check your kernel flags, view the file /proc/config.gz or
run the commands as in the examples below:
After verifying kernel flags, it’s time to install BCC tools
in Linux systems.
On Ubuntu 16.04
Only the nightly packages are created for Ubuntu16.04, but the installation instructions are very straightforward. No need of kernel upgrade or compile it from source.
$ echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
$ sudo apt-get update
$ sudo apt-get install bcc-tools
On Ubuntu 14.04
Begin by installing a 4.3+ Linuxkernel, from http://kernel.ubuntu.com/~kernel-ppa/mainline.
As an example, write a small shell script “bcc-install.sh”
with the content below.
Note: update PREFIX value
to the latest date, and also browse the files in the PREFIX
url provided to get the actual REL value,
substitute them in the shell script.
Save the file and exit. Make it executable, then run it as shown:
$ chmod +x bcc-install.sh $ sh bcc-install.sh
Afterwards, reboot your system.
$ reboot
Next, run the commands below to install signed BCC packages:
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD
$ echo "deb https://repo.iovisor.org/apt trusty main" | sudo tee /etc/apt/sources.list.d/iovisor.list
$ sudo apt-get update
$ sudo apt-get install binutils bcc bcc-tools libbcc-examples python-bcc
How To Use BCC Tools in Linux Systems
All the BCC tools are installedunder
/usr/share/bcc/toolsdirectory. However, you can alternatively
run them from the BCC Github repository
under
/toolswhere they end with a
.pyextension.
We shall cover a few examples under – monitoring general Linux system performance and networking.
Trace open() syscalls
Let’s start by tracing all open()syscalls using opensnoop.
This enable us tell us how various applications work by identifying their data files, config files and many more:
Summarize Block Device I/O Latency
In this example, it shows a summarized distribution of disk I/O latency using biolatecncy. After executing the command, wait for a few minutes and hit Ctrl-C toend it and view the output.
Trace New Processes via exec() Syscalls
In this section, we shall move to tracing new processes in execution using execsnoop tool.Each time a process is forked by
fork()and
exec()syscalls,
it is shown in the output. However, not all processes are captured.
Trace Slow ext4 Operations
Using ext4slower to tracethe ext4 file system common
operations that are slower than 10ms,
to help us identify independently slow disk I/O via the file system.
It only outputs those operations that exceed a threshold:
Trace Block Device I/O with PID and Latency
Next off, let’s dive into printing a line per disk I/O each second, with details such as process ID, sector, bytes, latency among others using biosnoop:
Trace Page Cache hit/miss Ratio
Thereafter, we proceed to using cachestat todisplays one line of summarized statistics from the system cache every second. This enables for system tuning operations by pointing out low cache hit ratio and high rate of misses:
Trace TCP Active Connections
Monitoring TCP connections every second using tcpconnect.Its output includes source and destination address, and port number. This tool is useful for tracing unexpected TCP connections, thereby helping us to identify inefficiencies in application configurations or an attacker.
All the tools above can also be used with various options, to enable the help page for a given tool, make use of the
-hoption,
for example:
Trace Failed exec()s Syscalls
To trace failed exec()s syscalls, employ the -xoption with opensnoop
as below:
Trace Particular Process Functions
The last example below demonstrates how to execute a custom trace operation. We are tracing a particular process using its PID.First determine the process ID:
$ pidof firefox 15437
Later on, run the custom trace command.
In the command below:
-pspecifies the process ID,
do_sys_open()is
a kernel function that is traced dynamically including its second argument as a string.
Summary
BCC is a powerful and easy-to-usetoolkit for various System administration tasks such as tracing system performance monitoring, tracing block device I/O, TCP functions, file system operations, syscalls, Node.js probes, plus lots more. Importantly, it ships in with several example files and
man pages for the tools to guide you, making it user friendly and reliable.
Last but not least, you can get back to us by sharing your thoughts about the subject, ask questions, make useful suggestions or any constructive feedback via the comment section below.
For more information and usage visit: https://iovisor.github.io/bcc/
更多精彩内容,尽在阅读原文
相关文章推荐
- 6 Command Line Tools for Linux Performance Monitoring
- 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
- High-Performance Ray Tracing for Dynamic Scenes
- Linux - Sysstat [ All-in-One System Performance and Usage Activity Monitoring Tool For Linux]
- Linux Server Hacks, Volume Two: Tips & Tools for Connecting, Monitoring, and Troubleshooting
- Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
- Linux BPF/bcc for Oracle Tracing
- 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
- Top 25 Best Linux Performance Monitoring and Debugging Tools
- Knoppix Hacks: Tips and Tools for Using the Linux Live CD to Hack, Repair, and Enjoy Your PC
- Language-Directed Hardware Design for Network Performance Monitoring——Marple
- Linux System and Performance Monitoring
- Linux System and Performance Monitoring(Network篇)
- VmwareTools for linux的安装【转】
- Linux System and Performance Monitoring(CPU篇)
- Linux System and Performance Monitoring(CPU篇)
- Wireless tools for Linux 介绍 使用 交叉编译
- VMware tools for linux 安装注意事项及常见问题解决!
- VMware Tools for linux安装
- 20 Linux System Monitoring Tools Every SysAdmin Should Know