gdb - customize it the way <b>you</b> want

Making gdb do more (debugging) for less (re-typing)

gdb, out of the box, allows us to debug our code primitively.
Yet, gdb supports a large number of knobs and levers, all of which are quite documented.
But we, users, are too lazy to read documentation...
...and even more so, for software we've been using for years.
But gdb has acquired new features recently, and i decided to start exploring them when i said "SABENA!" (Such A Bloody Experience - Never Again!)
Note: even if you use front-end GUIs for gdb - this lecture will cover some features they do not provide.

Compiling C/C++ Programs For gdb Support

An executable program needs to have "debug symbols" in order to be properly debugged under gdb.
To compile a single-file program with debug info, use the '-g' flag:
```
gcc -g prog.c -o prog
```
The same manner works with g++ for C++ programs.
Use '-g' also when compiling separate source files of a multi-file program.
You don't need to compile all source files of a program with debug information...
... but then you'll only be able to step through the sources that were compiled with '-g'.

basic gdb commands - just to "clear the line"

Just to make everyone on the same line - we'll go over the basic gdb commands.
.. which basically allow you to run the program, stop it, and view its state.
which is the point of a debugger.

Running And Stopping

in the shell, 'gdb <executable> runs gdb on the desired program.
... 'gdb <executable> <PID>' attaches gdb to the already-running process with the given process id (PID).
Use the 'run' command to run the program. use 'run <parameters>' to give it parameters.
when the program runs, you can suspend it with ctrl-C, and get a gdb prompt.

Running And Stopping - cont.

In the gdb prompt:

'next' performs the next instruction (if a function call - performs the full function call).
'step' steps into the next command (for a function call - stops at the first instruction of the function).
'nexti' and 'stepi' do the same, at the machine-language level.
'cont' lets the program continue execution (until the next ctrl-C).

Examine execution stack and variables

'bt' shows you the execution stack (innermost function first).
'up' and 'down' move between stack frames, to allow examining their contents
...(but does NOT move the instruction pointer - i.e. 'next' will continue in the inner-most frame).
'frame <number>' moves to the given frame ('bt' shows frame numbers').
'frame' to see the current stack frame.
'print <expression>' evaluates an expression.
... In particular, use it to view contents of variables.
... Will look for local variables in the current stack frame.

Examine The Source Code

'list' shows a part of the "currently selected source file".
Whenever the debugger stops, the currently-executing function will become the "currently selected source file".
another 'list' will show the next lines from the source.
...'list <line number>' will show the source around the given line number in the current file.
'list <function>' will show the source of the given function (and make it the "current source").

Break-Point Basics

'info break' shows the list of currently defined program break-points.
'break <line>' will put a break-point in the given line of the current source file.
'break <function> will put a break-point at the first instruction of the given function.
...'break main' is a good way to allow you to debug step-by-step from the first instruction of the program...
...except for global variables initialization in C++ programs.
'delete <break-point number>' will delete the given breakpoint.
'delete' will delete all existing break-points.

Beautifying Output (make gdb closer to its GUIs)

We will now start seeing how to make gdb's output more comfortable.
This section will also be brief - there's nothing astonishing here...
Note: the goal is productivity, not impressing our non-Linux fellows...

the display command

To have a variable displayed every time you get the gdb prompt, 'display <variable>
This will stop working as soon as the variable moves out of scope.
Use 'display' to get a list of auto-displayed variables.
Use 'undisplay <number>' to stop the given auto-display.
Use 'undisplay' to stop all auto-displays.
Note: you can 'display <expression>', just like with print.

pretty printing

When printing variables - gdb prints them on a single long line.

Here is a convoluted struct printed by gdb:

(gdb) print v
$1 = {num1 = 4, num2 = 1999759, more = {val1 = 99 'c', val2 = 100 'd', arr1 = {
      0, 573440, 1146880, 1720320, 2293760, 2867200, 3440640, 4014080, 
      4587520, 5160960}}}

Lets make it nicer:

(gdb) set print pretty on 
(gdb) print v
$2 = {
  num1 = 4, 
  num2 = 1999759, 
  more = {
    val1 = 99 'c', 
    val2 = 100 'd', 
    arr1 = {0, 573440, 1146880, 1720320, 2293760, 2867200, 3440640, 4014080, 
      4587520, 5160960}
  }
}

Current Frame Information

To see the current stack frame's information:

(gdb) info frame 
Stack level 0, frame at 0xbfe59340:
 eip = 0x804834a in func_long_calc (c1.c:5); saved eip 0x80483bb
 called by frame at 0xbfe59350
 source language c.
 Arglist at 0xbfe59338, args: i=3
 Locals at 0xbfe59338, Previous frame's sp is 0xbfe59340
 Saved registers:
  ebp at 0xbfe59338, eip at 0xbfe5933c

To see the list of local variables in a frame:

(gdb) info locals 
j = 6
plastic = 1

print/<base>

It is useful to be able to print numbers in different bases (decimal, hex, octal, binary - see below):

(gdb) p i
$2 = 567
(gdb) p/x i
$3 = 0x237
(gdb) p/o i
$4 = 01067
(gdb) p/t i
$5 = 1000110111

examining memory with 'x'

Lets suppose we have a memory area and we want to view its contents.
We will use the 'x' command. to view 100 chars starting at 'arr':
```
x/80c arr
```
To view 20 4-byte integers in decimal format:
```
x/20dw arr
```
To view 10 long long (8-byte) integers in hex format:
```
x/10xg arr
```

Convenience Variables

gdb allows creating gdb-scoped variables "on the fly":
```
set $foo = 2 * i
```
To see the list of defined convenience variables:
```
 show convenience
```
To initialize a convenience variable only if it's not yet defined:
```
init-if-undefined $foo = 2 * i
```
Convenience variables are useful when writing gdb macros - we'll get to this later...

A Silly Feature - TUI

The gdb people envied all the gdb GUIs and decided to have their own.
To enter TUI mode, press ctrl-x followed by ctrl-a.
To leave TUI mode, repeat the same sequence.

50 Ways To Supply Program Parameters..

Run gdb with 'gdb -args <parameters>' to specify the default parameters for the program, from the shell (so later 'run' will use these parameters).
If you run the program with parameters once, using 'run' again will use the same parameters again.

The Annoying C-Preprocessor Macros

By default, gdb is not aware of C pre-processor macros, which is very annoying.
To solve this on Linux, compile the code like this:
```
gcc -gdwarf-2 -g3 prog.c -o prog
```

Inside gdb:

Breakpoint 1, main () at c_preprocessor_macros.c:14
14		int state = STATE_BAD;
(gdb) p STATE_BAD
$1 = 1
(gdb) info macro STATE_BAD
Defined at /home/choo/work/lin-club/gdb_-_customize_it/gdb_source_examples/c_preprocessor_macros.c:4
#define STATE_BAD 1

Breakpoint 2, main () at c_preprocessor_macros.c:19
19		int person2_qualities = FLAG_SMART | FLAG_TALL | FLAG_HEAVY;
(gdb) macro expand  FLAG_SMART | FLAG_TALL | FLAG_HEAVY
expands to: 2 | 4 | 8

Program Execution Control

gdb supplies a few more program execution control commands that are quite useful..
Let us see a few...

Finish And Until

'finish' lets the current function execute until it finishes.
'until <source line> makes gdb continue program execution until the given line is reached.
Note: 'until' can only be executed if the program already runs.

return

Sometimes we want a function to terminate prematurely during a debugging session.
For example:
1. A function computes a variable's value wrongly.
2. We found this, but still want to debug the rest of the program.
3. we can force the function to return immediately with the correct value:
```
return 59
```

Conditional Break-Points

Let us suppose we have a long loop, and we want to start debugging only at the 503rd iteration.

We do this using conditional break-points:

Breakpoint 1, func_long_calc (i=5000) at c1.c:5
5		int j = 5;
(gdb) list 8,10
8		for (plastic = 0; plastic < i; ++plastic) {
9			j += plastic;
10		}
(gdb) break 9 if plastic == 503
Breakpoint 4 at 0x804835a: file c1.c, line 9.
(gdb) cont
Continuing.

Breakpoint 4, func_long_calc (i=5000) at c1.c:9
9			j += plastic;
(gdb) print plastic 
$1 = 503

Other Break-Point Commands

'rbreak <regular expression>' sets break-points for all functions whose name matches the regular expression.
'disable <number>' disables (but does not delete) the given break-point.
'enable <number>' re-enables the given (previously disabled) break-point.
'tbreak' sets a temporary break-point - once it is hit, it is deleted.

Watch-Points

A watch-point makes the program stop when an expression is true, regardless of the function in which it happens.
A basic example: 'watch a == 1' makes the program stop when variable 'a' has the value... '1'.
Checking when my memory gets over-written with a zero byte:
```
watch *(char*)0x65476 == 0
```
Watch-points and other break-points are all seen using 'info break' or 'info watch'.
And use 'delete <watch-point number>' to delete a watch-point, just like with a break-point.

Hardware Vs. Software Watch-Points

By default, gdb attempts to use hardware-assisted break-points.
These get processed using the hardware, and have a lesser impact on program speed...
...relative to software watch-points.
There are a few other caveats to watch-points regarding multi-threaded programs - we'll see them later.

I Want To Do This - Again!

You can ask gdb to save a "checkpoint":
```
checkpoint
```
After stepping forward and finding you've gone further then you wanted, switch back to the checkpoint:
```
restart <checkpoint id>
```
To see the list of active check-points:
```
info checkpoints
```
What a checkpoint does: forks off a copy of the program, so gdb will be able to switch back to it later on.
To delete checkpoints:
```
delete checkpoint <checkpoint id>
```

Where Have My Source Files Gone?

The issue: gdb stores the full path to each source file, in the program's debug information.
The problem: when debugging on a different machine then where you compiled, gdb might not find the source code.
The solution: telling gdb where the source files are.
Use 'directory <directory>' to add directors in which gdb searches for source files.
Use 'show directories' to see the list of directories in which gdb searches for source files.
Use 'directory' with no parameters to reset this list.

gdb user-commands support

gdb allows creating macros in its own little language
As simple as blocks of often-repeated commands
Or more complicated, to view program structures
Useful especially when scanning core files (and being unable to invoke program helper functions)

Convenience Macros To Scan Data Structures

Suppose we have a generic list with nodes looking like this:

struct list_node {
        struct list_node *next;
        void *data;
};

And we have an instance of a list of 'struct access_data':

struct access_data {
        unsigned int clock_ticks;
        int count;
};

And a global variable pointing to the head of the list:
```
struct list_node *g_access_list_head;
```

Writing The 'access_data_first' Macro

We define the macro using the 'define' block:

define access_data_first
        set $ad_curr = g_access_list_head
        if $ad_curr == 0
                print "(null)"
        else
                print *(struct access_data*)$ad_curr->data
        end
end

We document the macro using the 'document' block:

document access_data_first
        prints the first node in the global access_data list, or "(null)"
end

Writing The 'access_data_next' Macro

We define the macro using the 'define' block:

define access_data_next
        if $ad_curr == 0
                print "(null list)"
        else
                if $ad_curr->next == 0
                        print "(null next)"
                else
                        set $ad_curr = $ad_curr->next
                        print *(struct access_data*)$ad_curr->data
                end
        end
end

Writing The 'access_data_next' Macro

Here is an example of using these macros:

Breakpoint 1, main () at c_list_gdb_macros.c:45
45		return 0;
(gdb) source c_list_gdb_macros.macros 
(gdb) access_data_first 
$1 = {clock_ticks = 0, count = 0}
(gdb) access_data_next 
$2 = {clock_ticks = 1, count = 10}
(gdb) access_data_next 
$3 = {clock_ticks = 2, count = 20}
(gdb) access_data_next 
$4 = {clock_ticks = 3, count = 30}
(gdb) access_data_first 
$5 = {clock_ticks = 0, count = 0}

Writing The 'access_data_at' Macro

And now the macro that prints the Nth element in the list:

define access_data_at
        if $argc == 1
                set $access_data_index = 0
                set $ad_curr = g_access_list_head
                while $access_data_index < $arg0
                        if $ad_curr == 0
                                loop_break
                        else
                                set $ad_curr = $ad_curr->next
                        end
                        set $access_data_index = $access_data_index + 1
                end
                if $ad_curr != 0
                        print *(struct access_data*)$ad_curr->data
                end
        else
                print "missing index"
        end
end

Command Hooks

gdb allows defining hooks, that will be executed before of after a command.
To define a pre-hook for the command 'echo':
```
define hook-echo
echo <<<---
end
```
To define a post-hook for the command 'echo':
```
define hookpost-echo
echo --->>>\n
end
```

Now, using the command 'echo':

(gdb) echo Hello World
<<<---Hello World--->>>
(gdb)

Program Execution run/stop Hooks

You can define a hook for whenever gdb stops or starts running the program.

For example, to make all breakpoints suspended in single-step mode, but enabled in running mode:

define hook-stop
disable breakpoints
end

define hook-run
enable breakpoints
end

define hook-continue
enable breakpoints
end

gdb and signals handling

gdb has special handling for signals.
By default, it catches them, and then forwards some of them to the program being debugged.
So if you debug programs that rely on signals - you should pay attention.

gdb - default signals handling

The 'info signals' command will show the current signals handling configuration:

(gdb) info signals 
Signal        Stop	Print	Pass to program	Description

SIGHUP        Yes	Yes	Yes		Hangup
SIGINT        Yes	Yes	No		Interrupt
SIGQUIT       Yes	Yes	Yes		Quit
...

The different options state that gdb does the following when the program receives the stated signal:
1. Stop: stop the program's execution and fall into gdb's prompt.
2. Print: print a message about the signal to the screen.
3. Pass: pass the signal to the program (if stopped - only after the user continues execution).

default signals handling - cont.

Use 'info signals SIGHUP' to see just the entry for SIGHUP...
...or 'info signals 1' (SIGHUP is signal number 1).
Numeric values work only up to 15. Above this - use symbolic names.
Note: some system calls use (now, or used in the past) signals.
Using them properly under gdb will require telling it not to stop the program when receiving them.

Changing Handling Of Signals

To change the way gdb handles a signal, use the 'handle' command:
'handle SIGHUP stop' will make gdb stop the program if it receives SIGHUP.
'handle SIGHUP nostop' will make gdb not stop the program.
'stop' implies 'print', and 'print' implies 'stop' - can't have 'stop' and 'noprint' for a signal.
Use print/noprint and pass/nopass in a similar manner for the other 2 options.

You may specify several options in a single command:

(gdb) handle SIGHUP stop pass
Signal        Stop	Print	Pass to program	Description
SIGHUP        No	No	Yes		Hangup

passing signals to programs

Use the 'signal' command to pass signals to your program.
If a signal is defined as 'stop' and 'pass', you can prevent it from being passed...
...by issuing the command 'signal 0' after gdb stopped, and before continuing the program...
...thought for some signals - you'll get the signal again and again.

Multi-Threading Features In gdb

'info threads' shows this about threads:

(gdb) info threads
  11 Thread 0xb35a8b90 (LWP 23534)  0xb7f3c410 in __kernel_vsyscall ()
  10 Thread 0xb3da9b90 (LWP 23533)  0xb7f3c410 in __kernel_vsyscall ()
  9 Thread 0xb45aab90 (LWP 23532)  0xb7f3c410 in __kernel_vsyscall ()
...
* 1 Thread 0xb7f3a940 (LWP 23522)  0xb7f3c410 in __kernel_vsyscall ()

First number is gdb's thread number.
Second number is pthread's thread ID.
Third number is matching kernel LWP (Light-Weight Process) ID
Fourth number is address of function currently-executing in the thread.
Thread marked with '*' is the current thread.

Viewing Thread Stacks

Using 'bt' will show the stack of the current thread.
Typing 'thread 8' will switch to thread 8.
Typing 'thread apply all bt' will show stack traces of all the threads.
When viewing local variables - we'll see those of the current frame in the current thread.

Controlling Thread Scheduling

One annoying problem with threads under gdb - typing 'next' allows all the threads to run until the current thread stops...
...or until some other random thread stops...
...(and we get switched to it automatically).
This makes single-stepping through a multi-threaded program quite annoying.
How do we get around this???

Disabling The Scheduler

The solution - gdb allows us to disable the threads scheduler:
```
set scheduler-locking off
```
Now, if we type 'step' or 'next' (or even 'cont'), only the current thread will run.
We may use 'set scheduler-locking on' to revert to normal scheduling mode.
We may use 'set scheduler-locking step' to disable the scheduling only when using 'step'...
- (Using 'next' will still allow thread scheduling in this mode).

Thread-Specific Break-Points

It is possible to define break-points that only work on a specific thread.
Use the 'break <location> thread <gdb thread number>'.

'info breakpoints' will show thread-specific break-points:

(gdb) info break
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0x0804853f in thread_func
                                       at c_threads_sleep.c:24 thread 6
	stop only in thread 6
	breakpoint already hit 2 times

Very useful if several threads use the same code, but we only want to debug one of them.

Naming Names - How?

Problem: there is no thread name support in Linux's pthreads...
...in a program with tens of threads, how do we find the thread we want to debug?
Using 'info threads' will give a pile of crap, since the current function may not tell us which thread it is.
Solution: using gdb used-commands, gdb's logging and an external (Perl) script.

Naming Names - Using macros+logging+Perl

We will write a macro that prints the output of 'apply thread all bt' to a file...

... and lets a script parse it to find which stack belongs to which name.

define my_thread_names
  set logging off
  set logging file /tmp/tmp_file_gdb_names
  set pagination off
  set logging overwrite on
  set logging redirect on
  set logging on
  thread apply all bt
  set logging off
  set logging redirect off
  set logging overwrite off
  set pagination on
  shell ./gdb_bt_to_name.pl /tmp/tmp_file_gdb_names
  shell rm -f /tmp/tmp_file_gdb_names
end

Thread, LWP and 'top' with 'H'

Another problem: how do we see the threads in the system (e.g. CPU usage of each thread)?
Solution: the Linux 'top' command allows viewing threads:
- just hit 'H' when inside 'top'...
- ...at which point each thread will be seen on a separate line...
- ...and then the 'PID' field will show the LWP ID (as seen in gdb's 'info thread')

Debugging Multiple-Processes (forked) Programs

Suppose that your program tries to fork()...
...and you run this under gdb.
What shall gdb do? remain with the parent?
If it did - how will you debug a forking program when your bug is with the child?
Not to mention - debugging a daemon?

follow-fork setting

Well, gdb will do what you tell it to, with:
```
set follow-fork-mode <parent|child>
```
By default - it follows the parent.
You should set this attribute before your program calls fork, of-course.
The process not being followed - will continue running normally.
Tough luck - cannot follow both...
...or can we? on Linux we can - see below..
...on other platforms you could follow parent, make child sleep (e.g. sleep(60), and attach to child with another gdb instance.

detach-on-fork setting

If we tell gdb not to detach from the un-followed process:
```
set detach-on-fork off
```
The non-followed process will be suspended.
And forks will be followed to any depth.
This allows you to step through these processes one at a time.

Controlling Forked Processes

Use 'info forks' to see the list of forked processes.
Use 'fork <fork-id>' to switch between forked processes.
Use 'detach fork <fork-id>' to detach a forked process (and let it run along).
Use 'delete fork <fork-id>' to kill a forked processes.

Debugging Shared Libraries

When we want to debug shared libraries, we have two main scenarios:
1. Shared libs linked with the binary - no problem.
2. Shared libs linked using dlopen().
For the 2nd type of linking - we cannot set a break-point before the library was loaded...

...but gdb is kind enough to "wait for the symbols" on our behalf:

(gdb) b honolulu
Function "honolulu" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (honolulu) pending.
(gdb) info break
Num     Type           Disp Enb Address    What
1       breakpoint     keep y     honolulu

As soon as the library will be loaded - gdb will notice this, find the address to break in, and enable the break-point.

Debugging Python C Modules

A specific case of debugging shared libraries is Python(/Perl/whatever) C modules.
These modules are linked as shared libraries, and loaded by Python(/Perl/whatever) when we import(/use) them.

To debug such Python modules:

gdb /usr/bin/python <script [script parameters]>

When we get into gdb, we will set (pending) break-points in our C functions, and then 'run'.
When our break-point is hit, we'll be placed into C code, and be able to debug our code.
Note: i didn't try using gdb on a Python together with a Python debugger - but do try this at home.

References

gdb manual - online HTML version - http://sourceware.org/gdb/current/onlinedocs/gdb_toc.html

Originally written by

guy keren