One of my systems was showing high system CPU utilization. While trying to diagnose this high system cpu usage, I discovered the profiler built into the kernel.
To enable Linux kernel profiling, you must set the profile parameter on the command line .
Boot the kernel with command line option
profile=2(or some other number instead of 2). This will cause a file
/proc/profileto be created. The number given after
profile=is the number of positions EIP is shifted right when profiling. So a large number gives a coarse profile. The counters are reset by writing to
/proc/profile. The utility
readprofilewill output statistics for you. It does not sort - you have to invoke
Here is my sorting command and some of the output, in case you were interested.
# /shared/readprofile | sort -nr | head -20 2420865 total 1.2731 1511144 default_idle 16979.1461 449875 shrink_active_list 503.7794 59360 schedule 22.6825 44295 shrink_inactive_list 24.4184 38420 list_del 417.6087 31976 generic__raw_read_trylock 1880.9412 27377 release_pages 92.8034 24292 __list_add 296.2439 18459 list_add 1845.9000
This allows me to look up each of these functions in the kernel source and make guesses as to why the system is spending so much time performing those tasks.
-  http://homepages.cwi.nl/~aeb/linux/profile.html