%include "lin-club.mgp" %default 0 bgrad 0 0 1 0 1 "white" %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page %nodefault %bgrad 0 0 1 0 1 "white" %center, size 7, font "typewriter", fore "blue", back "white", vgap 20 %left, prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 60 60 1 %cont Haifa Linux Club %center Strace And the Linux Application %size 6 (S)trace, Observe and Conquer %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Topics Applications and system calls strace - what is it? "interesting" system calls anatomy of a typical application strace - very useful flags strace use example - the hidden config file typical application launch procedures %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Topics (Cont.) strace use example - the networked client the story of daemons the trickiness of tracing scripts ltrace - tracing C library functions %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Applications And System Calls An application is one (or more) sets of executable files that \ together accomplish a task (or three). An application accesses (almost) all system resources (files, etc.) \ by invoking system calls. A system call invocation is usually performed by using a 'software \ interrupt' - thus making all system call invocations pass via a single kernel \ function. This implies that hooks can be easily added when system calls are \ invoked. One major hook allows one process to 'trace' system calls invoked \ by another process. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Applications And System Calls (Cont.) Because system calls are the only means to access resources like \ files, tracing them could tell us a lot about what an application does, when \ and how. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace - What Is It? strace is an application that allows us to run another application, \ and see all (or a selected set of) the system call invocations this \ application performs. We normally run it as: strace /bin/mkdir /tmp/bla and get output such as (full output trimmed). %size 2, font "typewriter", fore "black", back "white" execve("/bin/mkdir", ["/bin/mkdir", "/tmp/bla"], [/* 34 vars */]) = 0 uname({sys="Linux", node="simey", ...}) = 0 brk(0) = 0x804c384 open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/i586/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib/i586/mmx", 0xbfffeb20) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/i586/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib/i586", 0xbfffeb20) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib/mmx", 0xbfffeb20) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 %size 5, font "standard", fore "blue", back "white" (This is just small part of the output). %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page "Interesting" System Calls In order to be able to properly understand the output of strace, lets first \ examine a list of interesting system calls that repeat often. For full \ details about a system call - refer to its manual page. open - opens a given file for reading or for writing. returns a file \ descriptor that may be used to access the file. close - close a previously opened file. read, write - read from/write to a given file descriptor. execve - load and execute a given binary file under the current process. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page "Interesting" System Calls (Cont.) brk - increase the size of the program's heap (usually invoked by \ malloc, when it needs more virtual memory). stat, stat64 - get information about a file (does it exist? what is \ its length? access permissions?) umask - defines a set of flags that will be used to 'mask out' \ file permissions set for new files this process will create. mmap - load a given file and map its contents into part of the virtual \ memory of an application. reading this memory will actually read from the file. munmap - unmap a previously mmaped file. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page "Interesting" System Calls (Cont.) _exit - terminate the current process. fork - create a child process. returns the PID (process ID) of the \ child process. uname - check information about the operating system (kernel version, \ CPU architecture, operating system name, etc.) socket - create a socket, for network communications. connect - open a connection, using a given socket, to a given \ network address. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Anatomy Of A Typical Application With what we learned so far, lets see the anatomy of the complete invocation \ of the "mkdir /tmp/bla" command. First, the application file (/bin/mkdir) is executed: %size 2, font "typewriter", fore "black", back "white" execve("/bin/mkdir", ["/bin/mkdir", "/tmp/bla"], [/* 34 vars */]) = 0 %size 5, font "standard", fore "blue", back "white" Then operating system info is checked (uname), and the application's heap is set using brk. %size 2, font "typewriter", fore "black", back "white" uname({sys="Linux", node="simey", ...}) = 0 brk(0) = 0x804c384 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Anatomy Of A Typical Application (Cont.) Next, the dynamic linker performs loading of shared libraries. First, it \ checks if there's a file (/etc/ld.so.preload) telling whether some extra \ libraries should be loaded: %size 2, font "typewriter", fore "black", back "white" open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) %size 5, font "standard", fore "blue", back "white" and then goes to search for applications linked with the application in \ various directories. First, in directories defined by the LD_LIBRARY_PATH \ environment variable: %size 2, font "typewriter", fore "black", back "white" open("/opt/xmps/lib/i586/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib/i586/mmx", 0xbfffeb20) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/i586/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib/i586", 0xbfffeb20) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib/mmx", 0xbfffeb20) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/opt/xmps/lib", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Anatomy Of A Typical Application (Cont.) Then, the dynamic linker looks for those libraries in a list of directories \ defined in the file /etc/ld.so.cache: %size 2, font "typewriter", fore "black", back "white" open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=58807, ...}) = 0 old_mmap(NULL, 58807, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40013000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0hq\1\000"..., 1024) = 1024 fstat64(3, {st_mode=S_IFREG|0755, st_size=1343176, ...}) = 0 old_mmap(NULL, 1207648, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40022000 mprotect(0x40140000, 36192, PROT_NONE) = 0 old_mmap(0x40140000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x11e000) = 0x40140000 old_mmap(0x40145000, 15712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40145000 close(3) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40149000 munmap(0x40013000, 58807) = 0 brk(0) = 0x804c384 brk(0x804c3b4) = 0x804c3b4 brk(0x804d000) = 0x804d000 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Anatomy Of A Typical Application (Cont.) The next step is to find out some locale information. Locales are used to \ support different alphabets and character types, specific to various \ countries. The active locale name is stored in an environment variable, which \ is read via a normal function, not a system call. The system calls are used to \ read a file containing aliases for locale names. %size 2, font "typewriter", fore "black", back "white" open("/usr/share/locale/locale.alias", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=2601, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 read(3, "# Locale name alias data base.\n#"..., 4096) = 2601 brk(0x804e000) = 0x804e000 read(3, "", 4096) = 0 close(3) = 0 munmap(0x40013000, 4096) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Anatomy Of A Typical Application (Cont.) Now comes the (boring) part that reads various files that define properties \ of the active locale ("he_IL" in our case): %size 2, font "typewriter", fore "black", back "white" open("/usr/lib/locale/en_US/LC_IDENTIFICATION", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=370, ...}) = 0 old_mmap(NULL, 370, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40013000 close(3) = 0 open("/usr/lib/locale/en_US/LC_MEASUREMENT", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=28, ...}) = 0 old_mmap(NULL, 28, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40014000 close(3) = 0 open("/usr/lib/locale/en_US/LC_TELEPHONE", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=64, ...}) = 0 old_mmap(NULL, 64, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000 close(3) = 0 open("/usr/lib/locale/en_US/LC_ADDRESS", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=160, ...}) = 0 old_mmap(NULL, 160, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40016000 close(3) = 0 open("/usr/lib/locale/en_US/LC_NAME", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=82, ...}) = 0 old_mmap(NULL, 82, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40017000 close(3) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Anatomy Of A Typical Application (Cont.) %size 2, font "typewriter", fore "black", back "white" open("/usr/lib/locale/en_US/LC_PAPER", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=39, ...}) = 0 old_mmap(NULL, 39, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000 close(3) = 0 open("/usr/lib/locale/en_US/LC_MESSAGES", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 close(3) = 0 open("/usr/lib/locale/en_US/LC_MESSAGES/SYS_LC_MESSAGES", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=57, ...}) = 0 old_mmap(NULL, 57, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40019000 close(3) = 0 open("/usr/lib/locale/en_US/LC_MONETARY", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=291, ...}) = 0 old_mmap(NULL, 291, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4001a000 close(3) = 0 open("/usr/lib/locale/en_US/LC_COLLATE", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=21499, ...}) = 0 old_mmap(NULL, 21499, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4001b000 close(3) = 0 open("/usr/lib/locale/en_US/LC_TIME", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=2456, ...}) = 0 old_mmap(NULL, 2456, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40021000 close(3) = 0 open("/usr/lib/locale/en_US/LC_NUMERIC", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=59, ...}) = 0 old_mmap(NULL, 59, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4014a000 close(3) = 0 open("/usr/lib/locale/he_IL/LC_CTYPE", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=173764, ...}) = 0 old_mmap(NULL, 173764, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4014b000 close(3) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Anatomy Of A Typical Application (Cont.) We then set the permissions mask for the process: %size 2, font "typewriter", fore "black", back "white" umask(0) = 02 umask(02) = 0 %size 5, font "standard", fore "blue", back "white" Then perform the actual command - creation of the directory, using the \ 'mkdir' system call: %size 2, font "typewriter", fore "black", back "white" mkdir("/tmp/bla", 0777) = 0 %size 5, font "standard", fore "blue", back "white" And finally the process exits. %size 2, font "typewriter", fore "black", back "white" _exit(0) = ? %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace - Very Useful Flags '-o ' - name of a file into which all system call \ invocations shall be logged. '-p PID' - trace an already active process, with the given PID. When \ strace is killed (ctrl-C), the process is detached from, and continues running. '-e' - trace only system calls matching the given filter. '-eopen' - trace only invocations of the 'open' system call. '-f' - follow fork mode - if the traced process forks off another \ process, start also tracing the new process, along with the old process. \ This operation is recursive to further forks. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Hidden Config File Lets suppose that we want to configure an application with a modified config \ file, only that we don't know the file's name. A-ha! we can strace it, filtering for 'open' syscall invocations: %size 2, font "typewriter", fore "black", back "white" open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/i586/mmx/libldap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/i586/libldap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("/opt/xmps/lib/mmx/libldap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) [.. some messages snipped ..] open("/etc/pine.conf", O_RDONLY) = 3 open("/home/choo/.pinerc", O_RDONLY) = 3 open("/etc/pine.conf.fixed", O_RDONLY) = 3 [ .. more messages snipped ..] %size 5, font "standard", fore "blue", back "white" As you can see, 'pine' opens 3 suspicious files - /etc/pine.conf, \ /home/choo/.pinerc and /etc/pine.conf.fixed . We can probably assume that the \ order of opening them, is the order in which they override each other's \ contents. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Typical Application Launch Procedures Lets get back to how a typical application launches: The binary is executed (execve). Shared libraries defined in LD_PRELOAD or in /etc/ld.so.preload, are \ loaded and linked with the application. Shared libraries the application was linked with, are loaded into \ memory. Locale information is loaded. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Typical Application Launch Procedures (Cont.) For network apps - network information is thought out. Configuration files are often read now. The application performs its chores. The application exits (_exit), unless it's a daemon. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client When an application serves as a network client, often tracing it can be used \ to identify networking problems. Lets see how a trace of a networked client \ looks: %size 2, font "typewriter", fore "black", back "white" execve("/usr/bin/ftp", ["ftp", "actcom.co.il"], [/* 37 vars */]) = 0 uname({sys="Linux", node="simey", ...}) = 0 brk(0) = 0x8065b40 [.. snip ..] %size 5, font "standard", fore "blue", back "white" Now comes the interesting part - network address lookup. First, the \ /etc/nsswitch.conf is opened. Interesting? read the man page to know what \ it does. %size 2, font "typewriter", fore "black", back "white" open("/etc/nsswitch.conf", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=1744, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 read(3, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1744 read(3, "", 4096) = 0 close(3) = 0 munmap(0x40013000, 4096) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) The, libraries used for host names lookup are loaded: %size 2, font "typewriter", fore "black", back "white" open("/opt/xmps/lib/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=58807, ...}) = 0 old_mmap(NULL, 58807, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40013000 close(3) = 0 open("/lib/libnss_files.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\34"..., 1024) = 1024 fstat64(3, {st_mode=S_IFREG|0755, st_size=42897, ...}) = 0 old_mmap(NULL, 39460, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x401ae000 mprotect(0x401b7000, 2596, PROT_NONE) = 0 old_mmap(0x401b7000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x8000) = 0x401b7000 close(3) = 0 munmap(0x40013000, 58807) = 0 %size 5, font "standard", fore "blue", back "white" then the application looks up data in /etc/services. 'man services' for \ more info: %size 2, font "typewriter", fore "black", back "white" open("/etc/services", O_RDONLY) = 3 fcntl64(3, F_GETFD) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 fstat64(3, {st_mode=S_IFREG|0644, st_size=19891, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 read(3, "# /etc/services:\n# $Id: services"..., 4096) = 4096 close(3) = 0 munmap(0x40013000, 4096) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) Next the terminal (attached to standard input, whose file descriptor is \ normally 0) is messed up with: %size 2, font "typewriter", fore "black", back "white" ioctl(0, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0 ioctl(0, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0 readlink("/proc/self/fd/0", "/dev/pts/6", 511) = 10 %size 5, font "standard", fore "blue", back "white" And now, the 'utmp' file is accessed - perhaps to find our login name? \ ('man utmp' for more info): %size 2, font "typewriter", fore "black", back "white" access("/var/run/utmpx", F_OK) = -1 ENOENT (No such file or directory) open("/var/run/utmp", O_RDWR) = -1 EACCES (Permission denied) open("/var/run/utmp", O_RDONLY) = 3 fcntl64(3, F_GETFD) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 _llseek(3, 0, [0], SEEK_SET) = 0 alarm(0) = 0 rt_sigaction(SIGALRM, {0x4018a51c, [], 0x4000000}, {SIG_DFL}, 8) = 0 alarm(1) = 0 fcntl64(3, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) = 0 read(3, "\10\0\0\0\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(3, "\2\0\0\0\0\0\0\0~\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 [.. snip ..] read(3, "\7\0\0\0\23\5\0\0pts/6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 fcntl64(3, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0 rt_sigaction(SIGALRM, {SIG_DFL}, NULL, 8) = 0 alarm(0) = 1 close(3) = 0 %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) Trying to connect to 'nscd' (possibly name server daemon?) via a unix-domain \ socket: %size 2, font "typewriter", fore "black", back "white" socket(PF_UNIX, SOCK_STREAM, 0) = 3 connect(3, {sin_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory) close(3) = 0 %size 5, font "standard", fore "blue", back "white" And then reading the user list file - /etc/passwd: %size 2, font "typewriter", fore "black", back "white" open("/etc/passwd", O_RDONLY) = 3 fcntl64(3, F_GETFD) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 fstat64(3, {st_mode=S_IFREG|0644, st_size=1296, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 read(3, "root:x:0:0:root:/root:/bin/tcsh\n"..., 4096) = 1296 close(3) = 0 munmap(0x40013000, 4096) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) We skip some unrelated mambo-jambo, and then '/etc/resolv.conf' is read: %size 2, font "typewriter", fore "black", back "white" open("/etc/resolv.conf", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=70, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 read(3, "search actcom.co.il\nnameserver "..., 4096) = 70 read(3, "", 4096) = 0 close(3) = 0 munmap(0x40013000, 4096) = 0 %size 5, font "standard", fore "blue", back "white" Another failed attempt to connect to 'nscd': %size 2, font "typewriter", fore "black", back "white" socket(PF_UNIX, SOCK_STREAM, 0) = 3 connect(3, {sin_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory) close(3) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) We now open the '/etc/host.conf' file - to know how to perform host name lookup: %size 2, font "typewriter", fore "black", back "white" open("/etc/host.conf", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=17, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 read(3, "order hosts,bind\n", 4096) = 17 read(3, "", 4096) = 0 close(3) = 0 munmap(0x40013000, 4096) = 0 %size 5, font "standard", fore "blue", back "white" We were told to consult '/etc/hosts' first (see the 'read' above) - lets do \ that: %size 2, font "typewriter", fore "black", back "white" open("/etc/hosts", O_RDONLY) = 3 fcntl64(3, F_GETFD) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 fstat64(3, {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 read(3, "127.0.0.1\t\tlocalhost\n204.141.72."..., 4096) = 230 read(3, "", 4096) = 0 close(3) = 0 munmap(0x40013000, 4096) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) Then we open the NIS+ library (used to lookup various types of info, \ including host name lookup - if we had a NIS+ server, that is): %size 2, font "typewriter", fore "black", back "white" open("/opt/xmps/lib/libnss_nisplus.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=58807, ...}) = 0 old_mmap(NULL, 58807, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40013000 close(3) = 0 open("/lib/libnss_nisplus.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\33\0\000"..., 1024) = 1024 fstat64(3, {st_mode=S_IFREG|0755, st_size=44383, ...}) = 0 brk(0x8068000) = 0x8068000 old_mmap(NULL, 40172, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x401b8000 mprotect(0x401c1000, 3308, PROT_NONE) = 0 old_mmap(0x401c1000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x8000) = 0x401c1000 close(3) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) We'll skip the loading of the name server library, and the NIS \ library, and the DNS library, and the resolver library... Lets see the conversation with the DNS server. we know it's the DNS server, \ because of the address (192.114.47.4): %size 2, font "typewriter", fore "black", back "white" socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 connect(3, {sin_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.114.47.4")}}, 28) = 0 send(3, "\333=\1\0\0\1\0\0\0\0\0\0\6actcom\2co\2il\0\0\34\0\1", 30, 0) = 30 gettimeofday({1032887191, 463122}, NULL) = 0 poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 5000) = 1 recvfrom(3, "\333=\205\200\0\1\0\0\0\1\0\0\6actcom\2co\2il\0\0\34\0"..., 1024, 0, {sin_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.114.47.4")}}, [16]) = 88 close(3) = 0 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 connect(3, {sin_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.114.47.4")}}, 28) = 0 send(3, "\333>\1\0\0\1\0\0\0\0\0\0\6actcom\2co\2il\6actcom"..., 43, 0) = 43 gettimeofday({1032887191, 491101}, NULL) = 0 poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 5000) = 1 recvfrom(3, "\333>\205\203\0\1\0\0\0\1\0\0\6actcom\2co\2il\6actcom"..., 1024, 0, {sin_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.114.47.4")}}, [16]) = 101 close(3) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) We now get connected to the FTP server itself: %size 2, font "typewriter", fore "black", back "white" socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 connect(3, {sin_family=AF_INET, sin_port=htons(21), sin_addr=inet_addr("192.114.47.1")}}, 16) = 0 getsockname(3, {sin_family=AF_INET, sin_port=htons(32875), sin_addr=inet_addr("192.117.103.230")}}, [16]) = 0 setsockopt(3, SOL_IP, IP_TOS, [16], 4) = 0 fcntl64(3, F_GETFL) = 0x2 (flags O_RDWR) fstat64(3, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40013000 _llseek(3, 0, 0xbfffe540, SEEK_CUR) = -1 ESPIPE (Illegal seek) fcntl64(3, F_GETFL) = 0x2 (flags O_RDWR) brk(0x8069000) = 0x8069000 fstat64(3, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40014000 _llseek(3, 0, 0xbfffe540, SEEK_CUR) = -1 ESPIPE (Illegal seek) fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 6), ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000 write(1, "Connected to actcom.co.il (192.1"..., 42) = 42 %size 5, font "standard", fore "blue", back "white" The last 'write' is to STDOUT (normally file descriptor 1). %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) Some reading of data from the FTP server (logo): %size 2, font "typewriter", fore "black", back "white" rt_sigaction(SIGINT, {0x804e960, [INT], SA_RESTART|0x4000000}, {0x80535a0, [INT], SA_RESTART|0x4000000}, 8) = 0 read(3, "220 actcom.co.il FTP server read"..., 4096) = 36 write(1, "220 actcom.co.il FTP server read"..., 35) = 35 rt_sigaction(SIGINT, {0x80535a0, [INT], SA_RESTART|0x4000000}, {0x804e960, [INT], SA_RESTART|0x4000000}, 8) = 0 setsockopt(3, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page strace Use Example - The Networked Client (Cont.) Now the user+password dialog (read from user, send to server): %size 2, font "typewriter", fore "black", back "white" write(1, "Name (actcom.co.il:choo): ", 26) = 26 read(0, "choo\n", 1024) = 5 rt_sigaction(SIGINT, {0x804e960, [INT], SA_RESTART|0x4000000}, {0x80535a0, [INT], SA_RESTART|0x4000000}, 8) = 0 write(3, "USER choo\r\n", 11) = 11 rt_sigaction(SIGINT, {0x804e960, [INT], SA_RESTART|0x4000000}, {0x804e960, [INT], SA_RESTART|0x4000000}, 8) = 0 read(3, "331 Password required for choo.\r"..., 4096) = 33 write(1, "331 Password required for choo.\n", 32) = 32 rt_sigaction(SIGINT, {0x804e960, [INT], SA_RESTART|0x4000000}, {0x804e960, [INT], SA_RESTART|0x4000000}, 8) = 0 rt_sigaction(SIGINT, {0x80535a0, [INT], SA_RESTART|0x4000000}, {0x804e960, [INT], SA_RESTART|0x4000000}, 8) = 0 open("/dev/tty", O_RDWR|O_CREAT|O_TRUNC, 0666) = 4 ioctl(4, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0 ioctl(4, SNDCTL_TMR_CONTINUE, {B38400 opost -isig icanon -echo ...}) = 0 fstat64(4, {st_mode=S_IFCHR|0666, st_rdev=makedev(5, 0), ...}) = 0 ioctl(4, SNDCTL_TMR_TIMEBASE, {B38400 opost -isig icanon -echo ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000 write(4, "Password:", 9) = 9 read(4, "mypasswd\n", 4096) = 9 write(4, "\n", 1) = 1 ioctl(4, SNDCTL_TMR_CONTINUE, {B38400 opost isig icanon echo ...}) = 0 close(4) = 0 munmap(0x40017000, 4096) = 0 rt_sigaction(SIGINT, {0x804e960, [INT], SA_RESTART|0x4000000}, {0x80535a0, [INT], SA_RESTART|0x4000000}, 8) = 0 write(3, "PASS mypasswd\r\n", 15) = 15 %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page The Story Of Daemons Daemons usually run in the background. Thus, they should be traced \ using 'strace -p' or 'strace -f '. Daemons often fork off child processes, making the strace output \ from all of them go to a single log file, unless we use 'strace -ff'. Daemons are passive - to test them we use 'strace', wait till they \ are ready, and then connect a client to them and look at the strace log... %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page The Trickiness Of Tracing Scripts When we launch a script, the system launches the script interpreter \ and gives it the script file name as a command-line argument. Thus, we don't see the script's execution as we do a C program's \ execution. The script performs many of its operations via external programs, \ including invocation of external scripts - so we need 'strace -f' to trace it. Sometimes, we trace a regular system utility, and the trace makes no \ sense. It is likely that this 'system utility' is actually a script - check \ it with 'less' to find if this is the case. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page The Trickiness Of Tracing Scripts (Cont.) Sometimes, a binary application has a script front-end. For example - \ Netscape, Mozilla. Read the script source, find the name of the actual \ binary, and trace that binary instead of the script. %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page ltrace - Tracing C Library Functions ltrace replaces the C library with a version that allows tracking \ any C library function invocation. ltrace could be used where strace sees nothing - e.g. when a program \ makes no system calls for a long period - while still performing C library \ functions. Of-course, if you have the complete source - you might as well use \ a debugger directly. %size 5, font "standard", fore "blue", back "white" %prefix " " %image "/usr/src/linux/Documentation/logo.gif" 0 20 20 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%