• modern computers → 100s of programs
• programs can interfere with each other
• complex!

Consider the following UNIX/Linux command:

$ ps

The output looks like:

    PID TTY          TIME CMD
2384593 pts/15   00:00:00 bash
2384610 pts/15   00:00:00 ps

This shows that two processes are running, for the current user: - bash: the shell/terminal that is running (where you are typing yoru commands) - ps: the command we just ran to see the lis of processes

We can expand this by providing some flags to ps, e.g.,

$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0 170060 14172 ?        Ss   Oct17   2:48 /sbin/init
root           2  0.0  0.0      0     0 ?        S    Oct17   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   Oct17   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   Oct17   0:00 [rcu_par_gp]
root           6  0.0  0.0      0     0 ?        I<   Oct17   0:00 [kworker/0:0H-kblockd]
root           9  0.0  0.0      0     0 ?        I<   Oct17   0:00 [mm_percpu_wq]
root          10  0.0  0.0      0     0 ?        S    Oct17   0:05 [ksoftirqd/0]
root          11  0.1  0.0      0     0 ?        I    Oct17  13:03 [rcu_sched]
root          12  0.0  0.0      0     0 ?        S    Oct17   0:01 [migration/0]
root          13  0.0  0.0      0     0 ?        S    Oct17   0:00 [idle_inject/0]
root          15  0.0  0.0      0     0 ?        S    Oct17   0:00 [cpuhp/0]
root          16  0.0  0.0      0     0 ?        S    Oct17   0:00 [cpuhp/1]
root          17  0.0  0.0      0     0 ?        S    Oct17   0:00 [idle_inject/1]
root          18  0.0  0.0      0     0 ?        S    Oct17   0:03 [migration/1]
root          19  0.0  0.0      0     0 ?        S    Oct17   0:01 [ksoftirqd/1]
root          21  0.0  0.0      0     0 ?        I<   Oct17   0:00 [kworker/1:0H-kblockd]
root          23  0.0  0.0      0     0 ?        S    Oct17   0:00 [cpuhp/2]
root          24  0.0  0.0      0     0 ?        S    Oct17   0:00 [idle_inject/2]
root          25  0.0  0.0      0     0 ?        S    Oct17   0:04 [migration/2]
root          26  0.0  0.0      0     0 ?        S    Oct17   0:01 [ksoftirqd/2]
root          28  0.0  0.0      0     0 ?        I<   Oct17   0:00 [kworker/2:0H-kblockd]
root          29  0.0  0.0      0     0 ?        S    Oct17   0:00 [cpuhp/3]
root          30  0.0  0.0      0     0 ?        S    Oct17   0:00 [idle_inject/3]
root          31  0.0  0.0      0     0 ?        S    Oct17   0:04 [migration/3]
root          32  0.0  0.0      0     0 ?        S    Oct17   0:01 [ksoftirqd/3]
root          34  0.0  0.0      0     0 ?        I<   Oct17   0:00 [kworker/3:0H]
root          35  0.0  0.0      0     0 ?        S    Oct17   0:00 [cpuhp/4]
root          36  0.0  0.0      0     0 ?        S    Oct17   0:00 [idle_inject/4]
root          37  0.0  0.0      0     0 ?        S    Oct17   0:04 [migration/4]
root          38  0.0  0.0      0     0 ?        S    Oct17   0:01 [ksoftirqd/4]
root          40  0.0  0.0      0     0 ?        I<   Oct17   0:00 [kworker/4:0H-kblockd]
root          41  0.0  0.0      0     0 ?        S    Oct17   0:00 [cpuhp/5]
...

Note: we see the list of processes for all users. The above is not a complete list, as it would have been really long.

We can filter out results too…

[sibin@ubuntu-vlab01 ~] ps aux | grep sibin
root     2384373  0.0  0.0  14984  9936 ?        Ss   17:14   0:00 sshd: sibin [priv]
sibin    2384387  0.0  0.0  18744  9960 ?        Ss   17:14   0:00 /lib/systemd/systemd --user
sibin    2384388  0.0  0.0 172308  6144 ?        S    17:14   0:00 (sd-pam)
sibin    2384397  0.0  0.0 277760 14352 ?        Ssl  17:14   0:00 /usr/bin/pulseaudio --daemonize=no --log-target=journal
sibin    2384399  0.0  0.0 508784 24620 ?        SNsl 17:14   0:00 /usr/libexec/tracker-miner-fs
sibin    2384407  0.0  0.0   8248  5228 ?        Ss   17:14   0:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
sibin    2384426  0.0  0.0 237112  7560 ?        Ssl  17:14   0:00 /usr/libexec/gvfsd
sibin    2384431  0.0  0.0 312808  6600 ?        Sl   17:14   0:00 /usr/libexec/gvfsd-fuse /run/user/1003/gvfs -f -o big_writes
sibin    2384439  0.0  0.0 312076  9944 ?        Ssl  17:14   0:00 /usr/libexec/gvfs-udisks2-volume-monitor
sibin    2384450  0.0  0.0 314124  9044 ?        Ssl  17:14   0:00 /usr/libexec/gvfs-afc-volume-monitor
sibin    2384455  0.0  0.0 235512  7096 ?        Ssl  17:14   0:00 /usr/libexec/gvfs-gphoto2-volume-monitor
sibin    2384460  0.0  0.0 233276  6140 ?        Ssl  17:14   0:00 /usr/libexec/gvfs-goa-volume-monitor
sibin    2384464  0.0  0.0 550560 34816 ?        Sl   17:14   0:00 /usr/libexec/goa-daemon
sibin    2384471  0.0  0.0 312152  9144 ?        Sl   17:14   0:00 /usr/libexec/goa-identity-service
sibin    2384477  0.0  0.0 233100  6288 ?        Ssl  17:14   0:00 /usr/libexec/gvfs-mtp-volume-monitor
sibin    2384592  0.0  0.0  15136  6736 ?        S    17:14   0:00 sshd: sibin@pts/15
sibin    2384593  0.0  0.0   9420  6088 pts/15   Ss   17:14   0:00 -bash
sibin    2385179  0.0  0.0   9760  3996 pts/15   R+   17:19   0:00 ps aux
sibin    2385180  0.0  0.0   6432  2620 pts/15   S+   17:19   0:00 grep --color sibin

The above command (ps aux | grep sibin) is sending the output of one command, ps aux to the input of another command, grep sibin (using a “pipe”). Hence, we are only listing those lines from the output of ps aux that has a string that matches the word, sibin.

Note: neither command/process (ps/grep) needs to be aware of each other. hey both carry out their normal functionality. It if the beauty and elgance of the UNIX design that allows this chaining of processes to work as well as it does. Technically most UNIX commands/processes can be chained together to get very advance functionality from very simple components.

8.2 process

• process ID pid
• memory acessible only to process
  • code, global data, stack, heap]
• integer descriptors that identify resources
  • [file system, networking, communication]
• current working directory pwd
• owner sibin
• parent → creator, responsible for it
• arguments passed from command line
• environment variables → system config information

``

8.3 fork()

Create a new, child, process → from the state of the calling process

Defined in <unistd.h>.

8.3.1 after fork()

8.3.2 process creation/termination/coordination API

Let’s take a look at all the function available in C for dealing with processes.

8.3.2.1 process vs program

• program → compiled version of code
• process → program executing in memory

8.3.3 pdt_t

• process identifier defined in <sys/types.h>
• same as a (signed) int

8.3.4 let’s revisit fork()

A very simple interface, for a very complex mechanism!

pid_t fork(void);

The new (“child”) process get a copy of the resources as shown before:

Note: some important nuances: - the child starting executing from the fork() instruction onwards! - all previous instructions don’t matter for the child - the parent, on the other hand, will also continue to execute, in parallel with the child.

/* CSC 2410 Code Sample 
 * intro to fork
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>

int main()
{
    printf( "BEFORE calling fork()" ) ;

    pid_t pid = fork() ;

    printf( "AFTER calling fork()" ) ;

    printf( "\n" ) ;
    return 0 ;
}

The output from the above code:

BEFORE calling fork()AFTER calling fork()

BEFORE calling fork()AFTER calling fork()

Hold on a second! We said that the child starts executing from the fork() instruction and not before. So, why is the BEFORE calling fork() being printed twice? We should, ideally, see the following behavior: - one instance of BEFORE calling fork() → parent only - two instances of AFTER calling fork() → parent and child

We are seeing a side-effect/vagary of the implementation of printf() and really any function that outputs to a standard destination e.g., stdio (standard input/output), stderr (standard error), etc. These output “locations” (rather called, “streams) are buffered, i.e., instead of immediately writing to the screen (in the case of stdio), the string/data are written to temporary inernal buffer that is then periodically cleansed (i.e., written out to the screen). The buffers are cleared out in the following situations: - when the buffer gets full (different systems have varied sizes, e.g., 64k) - when the buffer is explicitly flushed, e.g., using stdout - when a newline character (\n) is printed

So, in the above code example, while the first printf() is written only once (by the parent), the printf("\n") statement is present in both, parent and child. Remember that when a fork happens, the child gets a copy of all resources of the parent and that include the output buffer – which, at that point, includes the BEFORE calling fork() text. After the fork(), there are two processes with unflushed output buffers, the parent and the child. Hence, the \n causes both buffers to be flushed thus resulting in the above output.

There are two ways to fix this problem and get the desired behavior: 1. add \n to the end of Every printf() 2. force a flush after every printf() or perhaps just before every fork()

The call to flush a stream is:

int fflush( FILE* stream ) ; 

defined in <stdio.h>.

So, we let’s look at the above code again, this time with fflush (since the \n method is trivial and is left as an exercise):

/* CSC 2410 Code Sample 
 * intro to fork
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>

int main()
{
    printf( "BEFORE calling fork()" ) ;
    fflush(stdout) ; // to flush the standard output

    pid_t pid = fork() ;

    printf( "AFTER calling fork()" ) ;

    printf( "\n" ) ;
    return 0 ;
}

The output is:

BEFORE calling fork()AFTER calling fork()

AFTER calling fork()

See man fflush for more information.

8.3.5 fork() return values

Now, let’s look back at the interface for:

pid_t fork(void);

We see that is returns a pid_t. fork() returns two distinct values, depending on which process the call is returning to:

8.3.6 fork Questions

1. How many processes (not including the initial one) are created in the following?

   #include <unistd.h>
   
   int
   main(void)
   {
       fork();
       fork();
   
       return 0;
   }

2. What are the possible outputs for:

   #include <unistd.h>
   #include <stdio.h>
   #include <sys/wait.h>
   
   int
   main(void)
   {
       pid_t parent = getpid();
       pid_t child;
   
       child = fork();
       if (getpid() == parent) {
           printf("p");
           wait(NULL); /* why is `NULL` OK here? */
           printf("w");
       } else {
           printf("c");
       }
   
       return 0;
   }

8.4 wait()

Slides

Parents are responsible for managing their children!

so, they…wait()!

pid_t wait(int *wstatus)

• wait for state changes in a child
• called by a parent
• parents can wait for,
• all children
• specific child
• a subset of children

defined in <sys/wait.h>

A child changes “state” if one of following happens: - child terminated/exited - child was stopped by a signal * - child was resumed by a signal

[* will discuss signals later]

8.4.0.1 wait() return value

/* CSC 2410 Code Sample 
 * intro to fork() and wait()
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main()
{
        pid_t child_pid ;
        pid_t parent ;

        printf( "------------------------------\n" ) ;
        printf( ":::BEFORE::: \
                 getpid() = %d \
                 parent = %d \
                 child = %d\n",
             getpid(), child_pid, parent ) ;
        printf( "------------------------------\n" ) ;

        // create 5 children
        for( unsigned int i = 0 ; i < 5 ; ++i )
        {
                child_pid = fork() ;

                // make sure the children don't create more children!
                if(!child_pid)
                        break ;
        }

        // wait for the kids
        int wait_status ;
        pid_t wait_result = wait( &wait_status ) ;

        printf( ":::AFTER::: " ) ;
        child_pid ? printf( "---PARENT!--- " ) : printf( "---CHILD!--- " ) ;
        printf( "getpid() = %d \
                 parent = %d \
                 child = %d\n",
             getpid(), child_pid, parent ) ;

        printf( "\n" ) ;
        return 0 ;
}

Note: the output can vary! Depends on which process gets to writing its buffer first.

------------------------------
:::BEFORE:::         getpid() = 2492442          parent = 1612788000         child = 374366320
------------------------------
:::AFTER::: ---CHILD!--- getpid() = 2492443          parent = 0          child = 374366320

:::AFTER::: ---CHILD!--- getpid() = 2492444          parent = 0          child = 374366320

:::AFTER::: ---CHILD!--- getpid() = 2492445          parent = 0          child = 374366320

:::AFTER::: ---PARENT!--- getpid() = 2492442         parent = 2492447        child = 374366320
:::AFTER::: ---CHILD!--- getpid() = 2492446          parent = 0          child = 374366320


:::AFTER::: ---CHILD!--- getpid() = 2492447          parent = 0          child = 374366320

OR

------------------------------
:::BEFORE:::         getpid() = 2492454          parent = -2016386784        child = 780062336
------------------------------
:::AFTER::: ---CHILD!--- getpid() = 2492455          parent = 0          child = 780062336

:::AFTER::: ---CHILD!--- getpid() = 2492456          parent = 0          child = 780062336

:::AFTER::: ---CHILD!--- getpid() = 2492457          parent = 0          child = 780062336

:::AFTER::: ---CHILD!--- getpid() = 2492458          parent = 0          child = 780062336
:::AFTER::: ---PARENT!--- getpid() = 2492454         parent = 2492459        child = 780062336


:::AFTER::: ---CHILD!--- getpid() = 2492459          parent = 0          child = 780062336

OR many other combinations. We are seeing the inherent nondeterminism in parallel execution.

8.4.1 process vs thread

A minor detour. One may ver well be confused about the difference between a “process” and a “thread”. The main difference is that a process, owns resources while a thread is just a unit of execution. This table should help clarify the difference:

* for creation: i.e., the creation of a process is a system call, i.e., we need to ask the operating system to help us since the OS is what allocates resources to newly formed procsses. A new thread, on the other hand, doesn’t require us to invoke a system call and is often created using user-space libraries (*e.g., look up pthreads).

8.5 exit()

Slides

exit is the function that used to terminate the current process.

void exit(int status);

as defined in <stdlib.h. The input argument is the exit status:

And note that exit() does not explicitly return any value. All the information is passed via the status field.

The following two pieces of code are identical:

#include <stdlib.h>

int main()
{
    exit(EXIT_SUCCESS) ;
}

#include <stdlib.h>

int main()
{
    return EXIT_SUCCESS ;
}

8.5.1 Interpreting exit() status

Despite wait’s status return value being a simple int, the bits of that integer mean very specific things. See the man page for wait for details, but we can use a set of functions (really “macros”) to interpret the status value: - WIFEXITED(status) will return 0 if the process didn’t exit (e.g. if it faulted instead), and non-0 otherwise - WEXITSTATUS(status) will get the intuitive integer value that was passed to exit (or returned from main), but assumes that this value is only 8 bits large, max (thus has a maximum value of 256)

both defined in <sys/wait.h>.

8.5.2 Relationship between wait() and exit()

The relationship between these two calls can be confusing but they are closelfy related.

1. first fork() creates a new child:

2. if the parent is waiting for the child, then the return value of wait() is the pid of that child.

3. when the child calls exit(status) then that status is written into the status variable of the wait(&status) call.

Note: the interface for wait():

pid_t wait(int *wstatus)

the input is a pointer to an integer. Hence, the value passed from exit() is written into that integer and is accessible after the wait() call returns.

4. this goes for any exit() call from the child.

/* CSC 2410 Code Sample 
 * intro to fork(), wait() and exit()
 * Fall 2023
 * (c) Sibin Mohan
 */


#include <unistd.h>     // fork(), getpid()
#include <sys/wait.h>   // wait()
#include <stdlib.h>     // exit()
#include <stdio.h>


#define NUM_CHILDREN 5

int main()
{
    pid_t pid, child_pid ;

    // create multiple children 
    for( unsigned int i = 0 ; i < NUM_CHILDREN ; ++i )
    {
        pid = fork() ;
        if( !pid )
        {
            // we are inside the child
            printf( "---CHILD %d--- exiting with %d\n", getpid(), i+1 ) ;

            // both of these terminate immediately AND return the same value
            // identical, really!
            if( i % 2 )
                exit( i+1 ) ;
            else
                return i+1 ;
        }
    }

    /* Inside the parent, wait until the wait() call returns -1 -> no more children left
     * take the return value from wait(), put it in "child_pid" and compare that to "-1"
     */
    int status ;
    while( ( child_pid = wait( &status ) ) != -1 )
    {
        // all children are done!
        if( WIFEXITED(status) )
        {
            // check that the process didn't terminate with any errors
            // note that the output of WIFEEXITED is non-zero for normal exit

            printf( "Inside :::PARENT::: where child %d exited with status: %d\n", 
                    child_pid, 
                    (char) WEXITSTATUS(status) ) ;
        }
    }

    printf( "\n" ) ;
    return 0 ;
    // exit(EXIT_SUCCESS) ;
    // return EXIT_SUCCESS ;
}

Note: the output of this is alo non-deterministic as we can see with multiple runs of it:

---CHILD 2566089--- exiting with 1
---CHILD 2566090--- exiting with 2
---CHILD 2566091--- exiting with 3
---CHILD 2566092--- exiting with 4
Inside :::PARENT::: where child 2566089 exited with status: 1
---CHILD 2566093--- exiting with 5
Inside :::PARENT::: where child 2566090 exited with status: 2
Inside :::PARENT::: where child 2566091 exited with status: 3
Inside :::PARENT::: where child 2566092 exited with status: 4
Inside :::PARENT::: where child 2566093 exited with status: 5

---CHILD 2568612--- exiting with 1
---CHILD 2568613--- exiting with 2
---CHILD 2568614--- exiting with 3
---CHILD 2568615--- exiting with 4
Inside :::PARENT::: where child 2568612 exited with status: 1
---CHILD 2568616--- exiting with 5
Inside :::PARENT::: where child 2568613 exited with status: 2
Inside :::PARENT::: where child 2568614 exited with status: 3
Inside :::PARENT::: where child 2568615 exited with status: 4
Inside :::PARENT::: where child 2568616 exited with status: 5

This non-determinism is a product of the isolation that is provided by processes. The OS switches back and forth between processes frequently (up to thousands of time per second!) so that if one goes into an infinite loop, others will still make progress. But this also means that the OS can choose to run any of the processes that are trying to execute at any point in time! We cannot predict the order of execution, completion, or wait notification. This non-deterministic execution is called concurrency. You’ll want to keep this in mind as you continue to learn the process APIs, and when we talk about IPC, later.

8.5.3 life after exit()?

Remember that when main() returns, it is the same as exit() because main() calls exit().

Investigating main return → exit via gdb. You can see this by diving into any program with gdb -tui, breakpointing before the return (e.g. b 5), and single-stepping through the program. You’ll want to layout asm to drop into “assembly mode”, and single step through the assembly and if you want to step through it instruction at a time use stepi or si. You can see that it ends up calling __GI_exit. __GI_* functions are glibc internal functions, so we see that libc is actually calling main, and when it returns, it is then going through its logic for exit.

C provides you with additional control of what happens when you exit from a program. For instance, you may need to clean up resources, e.g., release some memory, close some files or network connections, write some debug information to a file, etc. Hence, the following functions can be are called once exit() is invoked: |function| defined in | |——–|——–| | on_exit| <stdlib.h>| | atexit| <stdlib.h>| | _exit| <unistd.h>| ||

Let’s look at these in more details.

8.5.3.1 on_exit()

typdef void (*function)( int , void * ) ;
int on_exit( function my_func, void *arg ) ;

• register a user-defined function pointer
• my_func
• to be called while exiting from main()
typdef void (*function)( int , void * ) ;
• function receives → exit status & argument
• we can pass data/args to that function

8.5.3.2 atexit()

typedef void (*function)(void) ;
int atexit(function my_func);

Program output:

array = 3 2341 2344 8632 747645 

• register a user-defined function pointer
• my_func
• to be called while exiting from main()
typedef void (*function)(void) ;
• much simpler function/interface → no args!
• no data/args passed to that function

Note: some nuances: 1. neither atexit() nor on_exit() immediately terminate the process. 2. you can register multiple functions using either one; these are called in reverse order of registrations 3. atexit() is standard c, while on_exit() may not be!

8.5.3.3 _exit()

void _exit(int status);

• immediate termination of process!
• no return value
• output buffers are not flushed
• does not call the funcs registered by others

/* CSC 2410 Code Sample 
 * intro to fork(), wait() and exit()
 * Fall 2023
 * (c) Sibin Mohan
 */


#include <unistd.h>     // fork(), getpid()
#include <sys/wait.h>   // wait()
#include <stdlib.h>     // exit()
#include <stdio.h>


#define NUM_CHILDREN 1

// function signature to match on_exit()
void cleanup( int status, void* args )
{
    free( args ) ;
    printf( "AFTER Exit(): Doing some cleanup. Freeing memory %hhx status = %d\n", args, status ) ;
}

void simple_cleanup()
{
    printf( "Goodbye cruel world!\n" ) ;
}

int main()
{
    pid_t pid, child_pid ;

    int* some_memory = (int*)malloc(1024) ;
    // register the function and also tell it what to cleanup
    on_exit( cleanup, some_memory ) ;

    // register the simpler atexit function
    atexit(simple_cleanup) ;
    

    // create multiple children 
    for( unsigned int i = 0 ; i < NUM_CHILDREN ; ++i )
    {
        pid = fork() ;
        if( !pid )
        {
            // we are inside the child
            printf( "---CHILD %d--- exiting with %d\n", getpid(), i+1 ) ;

            // both of these terminate immediately AND return the same value
            // identical, really!
            if( i % 2 )
                exit( i+1 ) ;
            else
                return i+1 ;
        }
    }

    /* Inside the parent, wait until the wait() call returns -1 -> no more children left
     * take the return value from wait(), put it in "child_pid" and compare that to "-1"
     */
    int status ;
    while( ( child_pid = wait( &status ) ) != -1 )
    {
        // all children are done!
        if( WIFEXITED(status) )
        {
            // check that the process didn't terminate with any errors
            // note that the output of WIFEEXITED is non-zero for normal exit

            printf( "Inside :::PARENT::: where child %d exited with status: %d\n", 
                    child_pid, 
                    (char) WEXITSTATUS(status) ) ;
        }
    }

    printf( "\n" ) ;
    return 0 ;
    // exit(EXIT_SUCCESS) ;
    // return EXIT_SUCCESS ;
}

The output:

---CHILD 2608965--- exiting with 1
Goodbye cruel world!
AFTER Exit(): Doing some cleanup. Freeing memory a0 status = 1
Inside :::PARENT::: where child 2608965 exited with status: 1

Goodbye cruel world!
AFTER Exit(): Doing some cleanup. Freeing memory a0 status = 0

But, what happens if we change the following lines?

    // register the simpler atexit function
    atexit(simple_cleanup) ;

    int* some_memory = (int*)malloc(1024) ;
    // register the function and also tell it what to cleanup
    on_exit( cleanup, some_memory ) ;

The output now looks like (what’s the difference?):

---CHILD 2609577--- exiting with 1
AFTER Exit(): Doing some cleanup. Freeing memory a0 status = 1
Goodbye cruel world!
Inside :::PARENT::: where child 2609577 exited with status: 1

AFTER Exit(): Doing some cleanup. Freeing memory a0 status = 0
Goodbye cruel world!

Finally, we change the return (i+1) to _exit(i+1) as follows:

            if( i % 2 )
                exit( i+1 ) ;
            else
                // return i+1 ;
                _exit(i+1) ;

What does the output look like now?

---CHILD 2610300--- exiting with 1
Inside :::PARENT::: where child 2610300 exited with status: 1

AFTER Exit(): Doing some cleanup. Freeing memory a0 status = 0
Goodbye cruel world!

8.6 Command Line Arguments

I think that we likely have a decent intuition about what the command-line arguments are`:

$ ls /bin /sbin

The ls program takes two arguments, /bin and /sbin. How does ls access those arguments?

Lets look at a chain of programs that exec each other. The first program (that you see here) is called inline_exec_tmp, and the programs 03/args?.c are subsequently executed.

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

char *prog = "./03/args1.bin";

int
main(int argc, char *argv[])
{
    char *args[] = {prog, "hello", "world", NULL};

    if (argc != 1) return EXIT_FAILURE;

    printf("First program, arg 1: %s\n", argv[0]);
    fflush(stdout);

    /* lets execute args1 with some arguments! */
    if (execvp(prog, args)) {
        perror("exec");
        return EXIT_FAILURE;
    }

    return 0;
}

Program output:

array = 3 2341 2344 8632 747645 

args1.c is

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <unistd.h>

char *prog = "./03/args2.bin";

int
main(int argc, char *argv[])
{
    int i;
    char **args;        /* an array of strings */

    printf("Inside %s\n", argv[0]);

    /* lets just pass the arguments on through to args2! */
    args = calloc(argc + 1, sizeof(char *));
    assert(args);

    args[0] = prog;
    for (i = 1; i < argc; i++) {
        args[i] = argv[i];
    }
    args[i] = NULL; /* the arguments need to be `NULL`-terminated */

    if (execvp(prog, args)) {
        perror("exec");

        return EXIT_FAILURE;
    }

    return 0;
}

…and args2.c is

#include <stdio.h>

int
main(int argc, char *argv[])
{
    int i;

    printf("Inside %s\n", argv[0]);

    for (i = 0; i < argc; i++) {
        printf("arg %d: %s\n", i, argv[i]);
    }

    return 0;
}

So we see the following.

• It is convention that the first argument to a program is always the program itself. The shell will always ensure that this is the case (NB: the shell execs your programs).
• The rest of the arguments are passed as separate entries in the array of arguments.
• The v variants of exec require the NULL termination of the argument array, something that is easy to mess up!

Parsing through the command-line arguments can be a little annoying, and getopt can help.

8.7 Environment Variables

Environment variables are UNIX’s means of providing configuration information to any process that might want it. They are a key-value store¹⁰ that maps an environment variable to its value (both are strings).

Environment variables are used to make configuration information accessible to programs. They are used instead of command-line arguments when:

• You don’t want the user worrying about passing the variable to a program. For example, you don’t want the user to have to pass their username along to every program as an argument.
• You don’t know which programs are going to use the configuration data, but any of them could. You don’t want to pass a bunch of command-line variables into each program, and expect them to pass them along to each child process.

Example common environment variables include:

• PATH - a :-separated list of file system paths to use to look for programs when attempt to execute a program.
• HOME - the current user’s home directory (e.g. /home/gparmer).
• USER - the username (e.g. gparmer).
• TEMP - a directory that you can use to store temporary files.

Many programs setup and use their own environment variables. Note that environment variables are pretty pervasively used – simple libraries exist to access them from python, node.js, rust, java, etc…

You can easily access environment variables from the command line:

$ echo $HOME
/home/gparmer
$ export BESTPUP=penny
$ echo $BESTPUP
penny

Any program executed from that shell, will be able to access the “BESTPUP” environment variable. The env command dumps out all current environment variables.

8.7.1 Environment Variable APIs

So how do we access the environment variable key-value store in C? The core functions for working with environment variables include:

• getenv - Get an environment variable’s value.
• setenv - Set one of environment variable’s value (used by the shell to set up children’s variables).
• clearenv - Reset the entire environment.
• environ array - This is the array of environment variables you’ll see in the man pages. You don’t want to access/modify this directly, but you can imagine it is used to back all of the previous calls.

#include <stdlib.h>
#include <stdio.h>
#include <assert.h>

int
main(int argc, char *argv[])
{
    char *u = getenv("USER");
    char *h = getenv("HOME");

    assert(u && h);

    printf("I am %s, and I live in %s\n", u, h);

    return 0;
}

Program output:

array = 3 2341 2344 8632 747645 

You can see all of the environmental variables available by default with:

$ env
SHELL=/bin/bash
DESKTOP_SESSION=ubuntu
EDITOR=emacs -nw
PWD=/home/gparmer/repos/gwu-cs-sysprog/22/lectures
LOGNAME=gparmer
HOME=/home/gparmer
USERNAME=gparmer
USER=gparmer
PATH=/home/gparmer/.local/bin::/home/gparmer/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/racket/bin/
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
...

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <assert.h>

int
main(int argc, char *argv[])
{
    char *u = getenv("USER");

    assert(u);
    printf("user: %s\n", u);
    fflush(stdout);

    if (setenv("USER", "penny", 1)) {
        perror("attempting setenv");
        exit(EXIT_FAILURE);
    }

    if (fork() == 0) {
        char *u = getenv("USER");
        char *args[] = { "./03/envtest.bin", "USER", NULL };

        /* environment is inherited across *both* `fork` and `exec`! */
        printf("user (forked child): %s\n", u);
        fflush(stdout);
        if (execvp("./03/envtest.bin", args)) {
            perror("exec");

            return EXIT_FAILURE;
        }
    }

    return 0;
}

Program output:

array = 3 2341 2344 8632 747645 

03/envtest.c is

#include <stdlib.h>
#include <stdio.h>
#include <assert.h>

int
main(int argc, char *argv[])
{
    char *e = "NOARG";
    char *v = "NOVAL";

    if (argc == 2) {
        e = argv[1];
        v = getenv(e);
        if (!v) {
            v = "";
        }
    }

    printf("Environment variable %s -> %s\n", e, v);

    return 0;
}

A common use of environment variables is the “home” directory in your shell. How is this implemented?

$ cd ~

The ~ means “my home directory”. To understand what directory is a user’s home directory, you can getenv(HOME)!

8.7.2 An Aside: Creating Processes with posix_spawn

fork and exec are not the only functions to execute a program. posix_spawn also enables the creation of a new process that execute a given program. posix_spawn performs three high-level actions:

1. fork to create a new process,
2. a set of “file actions” that update and modify the files/descriptors for the new process, that are specified in an array argument to posix_spawn, and
3. exec to execute a program and pass arguments/environmental variables.

It is strictly more limited in what it can do than fork and exec, but this is often a feature not a bug. fork is really hard to use well, and can be quite confusing to use. It is considered by some to be a flawed API. Thus the focus of posix_spawn on specific executing a new program can be quite useful to simply programs.

8.8 Representations of Processes

We’ve seen how to create and manage processes and how to execute programs. Amazingly, modern systems have pretty spectacular facilities for introspection into executing processes. Introspection facilities generally let you dive into something as it runs. The most immediate example of this is gdb or any debugger that let you dive into an implementation. Looking at pause.c:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

const int global_readonly = 0;
int global = 0;

int
main(void)
{
    int stack_allocated;
    int *heap = malloc(sizeof(int));

    printf("pid %d\nglobal (RO):\t%p\nglobal:      \t%p\nstack:      \t%p\nheap:      \t%p\nfunction:\t%p\n",
           getpid(), &global_readonly, &global, &stack_allocated, heap, main);

    pause();

    return 0;
}

Program output:

pid 9310
global (RO):    0x55be6d40b008
global:         0x55be6d40d014
stack:          0x7ffc7abafc7c
heap:           0x55be6da162a0
function:       0x55be6d40a1c9

Lets take the process identifier, and dive into the process! The “proc filesystem” in Linux is a part of the file system devoted to representing processes. There is a subdirectly in it for each process in the system. Lets check out process 9310.

$ cd /proc/9310/
$ ls
arch_status  cgroup      coredump_filter  exe      io         maps       mountstats  oom_adj        patch_state  sched      smaps         statm    timers
attr         clear_refs  cpuset           fd       limits     mem        net         oom_score      personality  schedstat  smaps_rollup  status   timerslack_ns
autogroup    cmdline     cwd              fdinfo   loginuid   mountinfo  ns          oom_score_adj  projid_map   sessionid  stack         syscall  uid_map
auxv         comm        environ          gid_map  map_files  mounts     numa_maps   pagemap        root         setgroups  stat          task     wchan
$ cat maps
55be6d409000-55be6d40a000 r--p 00000000 08:02 1315893                    /home/ycombinator/repos/gwu-cs-sysprog/22/lectures/03/pause.bin
55be6d40a000-55be6d40b000 r-xp 00001000 08:02 1315893                    /home/ycombinator/repos/gwu-cs-sysprog/22/lectures/03/pause.bin
55be6d40b000-55be6d40c000 r--p 00002000 08:02 1315893                    /home/ycombinator/repos/gwu-cs-sysprog/22/lectures/03/pause.bin
55be6d40c000-55be6d40d000 r--p 00002000 08:02 1315893                    /home/ycombinator/repos/gwu-cs-sysprog/22/lectures/03/pause.bin
55be6d40d000-55be6d40e000 rw-p 00003000 08:02 1315893                    /home/ycombinator/repos/gwu-cs-sysprog/22/lectures/03/pause.bin
55be6da16000-55be6da37000 rw-p 00000000 00:00 0                          [heap]
7ff4a127f000-7ff4a12a4000 r--p 00000000 08:02 11797912                   /lib/x86_64-linux-gnu/libc-2.31.so
7ff4a12a4000-7ff4a141c000 r-xp 00025000 08:02 11797912                   /lib/x86_64-linux-gnu/libc-2.31.so
7ff4a141c000-7ff4a1466000 r--p 0019d000 08:02 11797912                   /lib/x86_64-linux-gnu/libc-2.31.so
7ff4a1466000-7ff4a1467000 ---p 001e7000 08:02 11797912                   /lib/x86_64-linux-gnu/libc-2.31.so
7ff4a1467000-7ff4a146a000 r--p 001e7000 08:02 11797912                   /lib/x86_64-linux-gnu/libc-2.31.so
7ff4a146a000-7ff4a146d000 rw-p 001ea000 08:02 11797912                   /lib/x86_64-linux-gnu/libc-2.31.so
7ff4a146d000-7ff4a1473000 rw-p 00000000 00:00 0
7ff4a1495000-7ff4a1496000 r--p 00000000 08:02 11797896                   /lib/x86_64-linux-gnu/ld-2.31.so
7ff4a1496000-7ff4a14b9000 r-xp 00001000 08:02 11797896                   /lib/x86_64-linux-gnu/ld-2.31.so
7ff4a14b9000-7ff4a14c1000 r--p 00024000 08:02 11797896                   /lib/x86_64-linux-gnu/ld-2.31.so
7ff4a14c2000-7ff4a14c3000 r--p 0002c000 08:02 11797896                   /lib/x86_64-linux-gnu/ld-2.31.so
7ff4a14c3000-7ff4a14c4000 rw-p 0002d000 08:02 11797896                   /lib/x86_64-linux-gnu/ld-2.31.so
7ff4a14c4000-7ff4a14c5000 rw-p 00000000 00:00 0
7ffc7ab90000-7ffc7abb1000 rw-p 00000000 00:00 0                          [stack]
7ffc7abe1000-7ffc7abe4000 r--p 00000000 00:00 0                          [vvar]
7ffc7abe4000-7ffc7abe5000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

There’s a lot going on here. The most important parts of the ranges of addresses on the left that tell us where we can find the pause.bin program’s code, read-only global data, global read-writable data, heap, and stack! We also see that the number of maps in a very simple process is surprisingly large. Let me manually focus in on a few parts of this.

...
55be6d40a000-55be6d40b000 r-xp ... /03/pause.bin <-- &main      (0x55be6d40a1c9)
55be6d40b000-55be6d40c000 r--p ... /03/pause.bin <-- &global_ro (0x55be6d40b008)
...
55be6d40d000-55be6d40e000 rw-p ... /03/pause.bin <-- &global    (0x55be6d40d014)
...
55be6da16000-55be6da37000 rw-p ... [heap]        <-- heap       (0x55be6da162a0)
...
7ff4a14c4000-7ff4a14c5000 rw-p ...
7ffc7ab90000-7ffc7abb1000 rw-p ... [stack]       <-- stack      (0x7ffc7abafc7c)
...

We can see that each of the variables we’re accessing in the program are at addresses that correspond to the ranges of memory in the maps. Even more, the /proc/9310/mem file contains the actual memory for the process! This is really amazing: we can watch the memory, even as it changes, as a process actually executes. This is how debuggers work!

As you can see, processes are data too!

8.9 Process Exercises

8.9.1 Exercise 1: fibonacci with fork

Implement forkonacci! This will solve the nth fibonacci number, using fork for the recursion, and a combination of exit to “return” and wait to retrieve the returned value, instead of function calls. The normal recursive implementation:

unsigned int
fibonacci(unsigned int n)
{
    if (n == 0 || n == 1) return n;

    return fibonacci(n - 1) + fibonacci(n - 2);
}

This will work decently for n <= 12, but not for n > 12. Why¹¹?

8.9.2 Exercise 2: Shell Support for Environment Variables

How do you think that the shell-based support for environment variables (export BESTPUP=penny) is implemented? Specifically, if we do the following…

$ export BESTPUP=penny
$ ./03/envtest.bin BESTPUP  // this just prints the environment variable
Environment variable BESTPUP -> penny

…which process is using which APIs? Put another way, how is export implemented?

8.9.3 Exercise 3: Process Detective

The /proc filesystem is a treasure trove of information. You can dive into the process with pid N’s information in /proc/N/. You’ll only have access to the informationx for your processes. So how do you find the pid of your processes? You can find the pid of all of the processes that belong to you using ps aux | grep gparmer | awk '{print $2}'¹², but replacing your username for gparmer.

Choose one of those, and go into its directory in /proc. It it doesn’t work, you might have chosen a process that has terminated since you find its ID. Try again.

What do you think the following files contain:

• cmdline
• environ
• status

Some of the files are binary, which means you might need to use xxd (a hexdecimal binary view program) to look at them. In xxd, pay attention to the right column here.

8.10 Executing Other Processes

Slides

fork() only makes clones of itself. So, we’re limited to the functions defined in that program. A shell can execute other programs! Recall the difference between a program and process: - program → compiled version of code - process → program executing in memory

So, we want to create a new process and then execute a new program. This is done via the exec() family of calls:

     int execl(const char *path, const char *arg0, ..., /*, (char *)0, */);
     int execle(const char *path, const char *arg0, ..., /* (char *)0 char *const envp[] */);
     int execlp(const char *file, const char *arg0, ..., /*, (char *)0, */);
     int execv(const char *path, char *const argv[]);
     int execvp(const char *file, char *const argv[]);
     int execvP(const char *file, const char *search_path, char *const argv[]);

They are defined in <unistd.h>.

These calls will do the following: - Stop executing in the current process. - Reclaim all memory within the current process. - Load the target program into the process. - Start executing in the target program (i.e. starting normally, resulting in main execution).

Note: the main insight is that the same process continues execution but it now executes a new program.

8.10.1 Sequence of operations

1. initial process executing [parent]

2. fork() a new process [child]:

3. replace old code [child]

4. with new code [child]

5. with new code [child]

6. reclaim resources [child]

7. execute new program [child | from main()]

A few handy side-effects:

• The execution of the new program inherits the process identifier (pid_t) and the parent/child relationships of the process.
• Comparably, the descriptors are inherited into the new program’s execution.
• The environment variables (see section below) pass to the new program’s execution.
• Many other process properties are comparably inherited. With the exception of the process memory, you can assume, by default, that process properties are inherited across an exec.

Good side effects | Shared Resources

Note: only memory not shared | Resources not Shared

Detour: Variadic functions.

Sometimes, we don’t know how many arguments are needed for a function, e.g., printf(). Also, command line functions, e.g., ls vs ls -l. In this instance, the shell is calling fork() and one of the exec() functions to launch ls which starts from…its own main().

So, how does ls (or any other program) know what arguments are passed to it, and more importantly, how many?

The actual signature of main() is: C DNE int main( int argc, char* argv[] ) - argc tells us now many arguments have been passed and - char* argv[] is the actual set of arguments, i.e. an array of strings!

Note: the first argument is always the name of the program! Hence, we always have at least one argument.

/* CSC 2410 Code Sample 
 * exec() family of system calls | arguments to main()
 * Fall 2023
 * (c) Sibin Mohan
 */


#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <errno.h>

int main( int argc, char* argv[] ) 
{
    if( argc )
    {
        // we get a positive number of arguments
        // note that the first argument is always the name of the program
        // so we ALWAYS have AT LEAST ONE argument

        printf( "Command Line Args received:\t" ) ;
        for( unsigned int i = 0 ; i < argc ; ++i )
        {
            printf( "%s ", argv[i] ) ;
        }
    } 

    printf( "\n" ) ;
    return 0 ;
}

8.11 exec_() family

Multiple ways to launch a new program: - execl - execlp - execle - execv - execvp

The naming scheme is quite annoying and hard to remember, but the man page has a decent summary. The trailing characters correspond to specific operations that differ in how the command-line arguments are passed to the main:

1. execl() and execlp(): pass the argmuments directly to the exec() call:

 int execl( const char *path, const char *arg, ..., (char*)0 ) ;

The program gets the argument via the argc and argv method described earlier.

Note: the last argument has to be (char*)0, i.e., a NULL. This is so that the progam can figure out when the list of arguments is done.

2. execv() and execvp(): pass argmuments in null terminated array, argv[]

int execv( const char *path, char *const argv[] ) ;

The caller actually creates an array of strings and passes the arguments using that.

Note: no NULL termination in the function call.

3. execle() and execvpe(): environment variables of caller are passed

int execle( const char *pathname, const char *arg, .../*, (char *) NULL, char *const envp[] */ ) ;
int execvpe(const char *file, char *const argv[], char *const envp[]) ;

(l means pass the arguments to this exec call to the program, while v means pass the arguments as an array of the arguments into the exec call), how the program’s path is specified (by default, an “absolute path” starting with / must be used, but in a v variant, the binary is looked up using comparable logic to your shell), and how environment variables are passed to the program. For now, we’ll simply use execvp, and cover the rest in subsequent sections.

8.11.1 execve()

All of the above are layers on top of execve()

#include "unistd.h"

int execve(const char *pathname, char *const argv[],
char *const envp[]);

8.11.2 fork() and exec()

Consider the following piece of code:

/* CSC 2410 Code Sample 
 * exec() 
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <unistd.h>     // fork(), getpid()
#include <sys/types.h>  // pid_t
#include <sys/wait.h>   // wait()
#include <stdlib.h>     // exit()

// int main() // not actual signature
int main( int argc, char* argv[] )
{
    char* program = "/bin/ls" ;
    char* arg1 = "-al" ;
    char* arg2 = "/home" ;

    printf( "BEFORE EXEC!\n" ) ;

    int ret = execl( program, "banana", arg1, arg2, NULL ) ;

    printf( "AFTER EXEC!\n" ) ;

    printf( "\n" ) ;
    return 0 ;
}

Note: the printf( "AFTER EXEC!\n" ) ; and further code will never execute as the code for the current process is completely replaced by the code for the program called using execl(), i.e., /bin/ls.

So, to get the behavior that we want, i.e., for some post-processing/messages, etc., we must first fork() and new child process and then run execl() in the child process!

Updating the previous code:

/* CSC 2410 Code Sample 
 * exec() 
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <unistd.h>     // fork(), getpid()
#include <sys/types.h>  // pid_t
#include <sys/wait.h>   // wait()
#include <stdlib.h>     // exit()

// int main() // not actual signature
int main( int argc, char* argv[] )
{
    char* program = "/bin/ls" ;
    char* arg1 = "-al" ;
    char* arg2 = "/home" ;

    printf( "BEFORE EXEC!\n" ) ;

    pid_t child = fork() ;

    if( !child)
    {
        // in child process
        int ret = execl( program, "banana", arg1, arg2, NULL ) ;

        // ideally never comes here!
        // as the child process' code has now been replaced
        perror("what happened!") ;
    }

    int status ;
    pid_t pid = wait(&status) ;

    if( WIFEXITED(status) )
    {
            // the child exited normally
            printf( "--PARENT--: Child %d exited with status %d\n",
                    pid, WEXITSTATUS(status) ) ;
    }

    printf( "AFTER EXEC!\n" ) ;

    printf( "\n" ) ;
    return 0 ;
}

8.11.3 Launch Any Child Processes

If we have another program that we’ve written, e.g., child1.c:

/* CSC 2410 Code Sample 
 * exec() child 1
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <unistd.h>

int main(int argc, char* argv[])
{
    /* Does nothing. Just prints the arguments passed to this program */
    printf( "INSIDE CHILD 1: argc = %d, argv[0] = %s, argv[1] = %s, argv[2] = %s\n", 
                                            argc, argv[0], argv[1], argv[2] ) ;
 
    return 0 ;
}

Once it is compiled and linked and ready as an executable, say, child1, we can do:

/* CSC 2410 Code Sample 
 * exec() 
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <unistd.h>     // fork(), getpid()
#include <sys/types.h>  // pid_t
#include <sys/wait.h>   // wait()
#include <stdlib.h>     // exit()

// int main() // not actual signature
int main( int argc, char* argv[] )
{
    char* program = "./child1" ;
    char* args = "hello" ;
    char* args2 = "world" ;

    pid_t child = fork() ;

    if( !child)
    {
        // in child process
        int ret = execl( program, "banana", args, args2, NULL ) ;

        // ideally never comes here!
        perror("what happened!") ;
    }

    int status ;
    pid_t pid = wait(&status) ;

    if( WIFEXITED(status) )
    {
            // the child exited normally
            printf( "--PARENT--: Child %d exited with status %d\n",
                    pid, WEXITSTATUS(status) ) ;
    }

    printf( "\n" ) ;
    return 0 ;
}

9 Process Descriptors

Slides

We’re going to start discussing how processes can manipulate the resources it has access to.

9.0.1 Key Mechanisms

for a process to use effectively

• current working directory, pwd
• ability to manipulate descriptors
  • “file descriptors”
• controlling “exceptional” control flow, via signals

9.0.2 current working directory

• each process has a “working directory”
• base for any relative pathrs
• all file system paths are one of,
  • absolute paths → they begin with “/”
  • relative paths

Relative paths are quite frequently used when we interact with the shell. Every time you type cd blah/, you’re saying “please change the current working directory to blah/” which is a directory in the current working directory.

A simple API to interact with the current working directory: |function| operation | |——–|——–| | getcwd| gets the current process’ working dir | | chdir| enables process to change dir | ||

both defined in <unistd.h>.

A quick listing of the directory structure of Linux:

Let’s look at a simple code example:

#include <unistd.h>
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    char *wd = getcwd(NULL, 0);

    assert(wd);
    printf("Current directory: %s\n", wd);
    free(wd);

    if (chdir("..") == -1) {
        perror("chdir");
        abort();
    }
    wd = getcwd(NULL, 0);
    printf("New current dir: %s\n", wd);
    free(wd);

    printf( "\n" ) ;
    return 0;
}

Note: the command cd is actually not a program, and is instead a shell-internal function. Try using the which program (used to find the location of a known program):

$ which ls
/bin/ls
$ which pwd
/bin/pwd
$ which cd
$

9.0.3 Process Descriptors

• each process has a set of descriptors
• associated with a system resource
  • integer → passed to a family of APIs

Most processes have at least three: |number | descriptor | What it is | |——–|——–|——–| |0 | STDIN_FILENO | standard input | |1 | STDOUT_FILENO | standard output | |2 | STDERR_FILENO | standard error perror()| ||

defined in unistd.h.

9.0.3.1 Standard input, output and error

• infinite sequence of bytes or “channels”
• can terminate, if they run out of data
  • e.g., reaches end of file
  • user hits ctrl-d

When we type at the shell, we’re providing a channel of data that is sent to the standard input of the active process. When a process prints, it sends a sequence of characters to the standard output, and because programs can loop, that stream can be infinite!

STDIN_FILENO = 0 is the standard input, or the main way the process gets input from the system. As such, the resource is often the terminal if the user directly provides input, or sometimes the output of a previous stage in a command-line pipeline. STDOUT_FILENO = 1 is the standard output, or the main way the process sends its output to the system. When we call printf, it will, by default, output using this descriptor. STDERR_FILENO = 2 is the standard error, or the main way that the process outputs error messages. perror (and other error-centric APIs) output to the standard error.

Each of these descriptors is associated with a potentially infinite sequence of bytes, or channel¹³. When we type at the shell, we’re providing a channel of data that is sent to the standard input of the active process. When a process prints, it sends a sequence of characters to the standard output, and because programs can loop, that stream can be infinite! Channels, however, can terminate if they run out of data. This could happen, for example, if a channel reading through a file reaches the end of the file, or if the user hits cntl-d on their terminal to signify “I don’t have any more data”.

9.0.4 File Descriptor Operations

Now these are file descriptors so we treat them as files!

File descriptors are analogous to pointers. The descriptors effectively point to channel resources.

Some core operations on files: |operation| interface | |———|———–| |pull bytes from channel into buffer | read | |send bytes from channel into buffer | write |duplicate a file descriptor | dup/dup2/dup3 | |deallocate a file descriptor | close | ||

Note:: - printf() is a write to standard out - close doesn’t necessarily remove channel - analogous to removing a pointer

9.0.4.1 dup() vs dup2() vs dup3()

• dup() returns smallest unused number as fd
• dup2() same as dup() but uses given fd
• if given fd exists, then it is closed, atomically
• dup3() same as dup2() but takes flags as input

Look at the man dup page for more information.

Let’s see some of these calls in action:

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <assert.h>
#include <stdlib.h>

int
main(void)
{
    char *hw = "hello world\n";
    char output[256];
    int fd;
    ssize_t amnt; /* signed size */

    amnt = write(STDOUT_FILENO, hw, strlen(hw));
    if (amnt == 0) { /* maybe STDOUT writes to a file with no disk space! */
        /* this is *not* an error, so errno not set! */
        printf("Cannot write more data to channel\n");
        exit(EXIT_FAILURE);
    } else if (amnt > 0) {
        /* normally, the return value tells us how much was written */
        assert(amnt == (ssize_t)strlen(hw));
    } else { /* amnt == -1 */
        perror("Error writing to stdout");
        exit(EXIT_FAILURE);
    }

    amnt = write(STDERR_FILENO, hw, strlen(hw));
    assert(amnt >= 0);

    fd = dup(STDOUT_FILENO);
    assert(fd >= 0);

    /* We can write formatted data out to stdout manually! */
    snprintf(output, 255, "in: %d, out: %d, err: %d, new: %d\n",
             STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO, fd);
    output[255] = '\0';
    amnt = write(fd, output, strlen(output));
    /* new file descriptors are supposed to use the lowest unused descriptor! */

    /* make a descriptor available */
    close(STDIN_FILENO); /* STDIN is no longer really the input! */
    fd = dup(STDOUT_FILENO);
    printf("New descriptor @ %d\n", fd);

    return 0;
}

You can run this, and redirect the standard error to a file to see that writing to standard error is a different operation than writing to standard output. For example: $ prog 2> errors.txt will redirect file descriptor 2 (stderr) to the file.

Lets focus in a little bit on read and write. First, it is notable that the buffer they take as an argument (along with its length) is simply an array of bytes. It can be a string, or it could be the bytes that are part of an encoded video. Put another way, by default, channels are just sequences of bytes. It is up to our program to interpret those bytes properly.

Second, we need to understand that the return value for read/write has four main, interesting conditions:

#include <unistd.h>
#include <stddef.h>
#include <string.h>
#include <errno.h>

int
main(void)
{
    ssize_t amnt;
    char *hi = "more complicated than you'd think...";
    ssize_t hi_sz = strlen(hi);

    amnt = write(STDOUT_FILENO, hi, hi_sz);

    /* Can often mean that we are not able to write to the resource */
    if (amnt == 0) {
        /*
         * Keep trying to write, or give up.
         * Common return value for `read` when a file has no more data, or a pipe is closed.
         */
    } else if (amnt > 0 && amnt < hi_sz) {
        /*
         * Didn't write everythign we wanted, better call write again sending
         * data starting at `&hi[amnt]`, of length `hi_sz - amnt`.
         */
    } else if (amnt == hi_sz) {
        /*
         * Wrote out everything! Wooo!
         */
    } else { /* amnt == -1 */
        /* Could be a genuine error, but not always... */
        if (errno == EPIPE || errno == EAGAIN || errno == EINTR || errno == EWOULDBLOCK) {
            /* conditions we should probably handle properly */
        } else {
            /* error in the channel! */
        }
    }

    return 0;
}

It is common to have a convention on how channel data is structured. UNIX pipeline encourage channels to be plain text, so that each program can read from their standard input, do some processing that can involved filtering out data or transforming it, and send the result to the standard out. That standard output is sent to the standard input of the next process in the pipeline. An example in which we print out each unique program that is executing on the system:

$ ps aux | tr -s ' ' | cut -d ' ' -f 11 | sort | uniq

Each of the programs in the pipeline is not configured to print out each unique process, and we are able to compose them together in a pipeline to accomplish the goal.

9.1 Pipes

Slides

Processes have resources

Note that descriptors 0-2 are automatically set up: - STDIN_FILENO - STDOUT_FILENO - STDERR_FILENO

But how do we create resources? But first off, what are resources? There are many different resources in a UNIX system, but three of the main ones:

1. files and other file-system objects (e.g. directories),
2. sockets that are used to communicate over the network, and
3. communication facilities like pipes that are used to send data and coordinate between processes.

Each of these has very different APIs for creating the resources. We’ll discuss files later, and will now focus on pipes.

We’ve seen pipes before:

$ ps aux | tr -s ' ' | cut -d ' ' -f 11 | sort | uniq

(look up each of those commands).

Not what we’re talking about…well, not exactly!

9.1.1 unix ‘pipes’

(short for “pipelines”)

A finite channel → a sequence of bytes:

A pipe is accessed using two (file) descriptors¹⁴,

This can be represented as follows:

The sequence of bytes written to the pipe will be correspondingly read out in a FIFO manner.

9.1.2 pipes api

int pipe( int pipefd[2] ) ;

defined in <unistd.h>.

The pipe and pipe2 functions create a pipe resource, and return the two file descriptors that reference the readable and writable ends of the pipe.

Return values

We use a familiar interface to send/receive data from a pipe, viz., read() and write().

Let’s look at a simple example of how to use a pipe.

/* CSC 2410 Code Sample 
 * intro to pipes
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <assert.h>
#include <sys/param.h> /* MIN */

#define BUFFER_SIZE 16

int main()
{
    char from[BUFFER_SIZE] = {'\0'} ;
    char to[BUFFER_SIZE] = {'\0'} ;

    int pipe_fds[2] ; // the two FDs for reading/writing

    memset( from, 'x', BUFFER_SIZE-1 ) ;
    memset (to, '-', BUFFER_SIZE-1 ) ;

    if( pipe(pipe_fds) )
    {
        // non zero is an error!
        perror( "Pipe creation failed!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    printf( "BEFORE\n\t from: %s\n\t to: %s\n", from, to ) ;

    ssize_t write_return = write( pipe_fds[1], &from, BUFFER_SIZE ) ;
    assert( write_return == BUFFER_SIZE ) ; // check how many bytes were written

    ssize_t read_return = read( pipe_fds[0], &to, BUFFER_SIZE ) ;

    printf( "AFTER\n\t from: %s\n\t to: %s\n", from, to ) ;


    printf( "\n" ) ;
    return 0 ;
}

Output, as expected:

BEFORE
         from: xxxxxxxxxxxxxxx
         to: ---------------
AFTER
         from: xxxxxxxxxxxxxxx
         to: xxxxxxxxxxxxxxx

Now, let’s change things a bit. Let’s make the from buffer really LARGE!

/* CSC 2410 Code Sample 
 * intro to pipes
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <assert.h>
#include <sys/param.h> /* MIN */

#define BUFFER_SIZE 16
#define LARGE_BUFFER_SIZE 1<<18


int main()
{
    char from[LARGE_BUFFER_SIZE] = {'\0'} ;
    char to[BUFFER_SIZE] = {'\0'} ;

    int pipe_fds[2] ; // the two FDs for reading/writing

    memset( from, 'x', LARGE_BUFFER_SIZE-1 ) ;
    memset (to, '-', BUFFER_SIZE-1 ) ;

    if( pipe(pipe_fds) )
    {
        // non zero is an error!
        perror( "Pipe creation failed!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    ssize_t write_return = write( pipe_fds[1], &from, LARGE_BUFFER_SIZE ) ;
    printf( "Here!\n" ) ;
    assert( write_return == LARGE_BUFFER_SIZE ) ; // check how many bytes were written

    ssize_t read_return = read( pipe_fds[0], &to, BUFFER_SIZE ) ;

    printf( "AFTER\n\t to: %s\n", from, to ) ;


    printf( "\n" ) ;
    return 0 ;
}

Output is empty and the program doesn’t terminate. Why? Two reasons:

1. the size of a pipe is limited (usually 64k on moder linux)
2. write() blocks until a read() clears some space in the pipe buffer.

You can get/set the size of the pipe buffer. See man 7 pipe for more details.

so, what do we need to do? We need to clear the pipe buffer by using read.

Why does this not work?

/* CSC 2410 Code Sample 
 * intro to pipes
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <assert.h>
#include <sys/param.h> /* MIN */

#define BUFFER_SIZE 16
#define LARGE_BUFFER_SIZE 1<<18


int main()
{
    char from[LARGE_BUFFER_SIZE] = {'\0'} ;
    char to[LARGE_BUFFER_SIZE] = {'\0'} ;

    int pipe_fds[2] ; // the two FDs for reading/writing

    memset( from, 'x', LARGE_BUFFER_SIZE-1 ) ;
    memset (to, '-', LARGE_BUFFER_SIZE-1 ) ;

    if( pipe(pipe_fds) )
    {
        // non zero is an error!
        perror( "Pipe creation failed!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    ssize_t write_return = write( pipe_fds[1], &from, LARGE_BUFFER_SIZE ) ;
    printf( "Here!\n" ) ;
    assert( write_return == LARGE_BUFFER_SIZE ) ; // check how many bytes were written

    ssize_t read_return = read( pipe_fds[0], &to, LARGE_BUFFER_SIZE ) ;

    printf( "\n" ) ;
    return 0 ;
}

Because the write() is still blocked! We haven’t cleared the pipe buffer. The write() call has not returned and so the read() call cannot run!

Note: this is not parallel execution!

So, let’s fix it. By writing and reading inside a loop, using smaller chunks of read/write each time.

/* CSC 2410 Code Sample 
 * intro to pipes
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <assert.h>
#include <sys/param.h> /* MIN */

#define BUFFER_SIZE 16
#define LARGE_BUFFER_SIZE 1<<18

#define WRITE_CHUNK 1<<8


int main()
{
    char from[LARGE_BUFFER_SIZE] = {'\0'} ;
    char to[LARGE_BUFFER_SIZE] = {'\0'} ;

    int pipe_fds[2] ; // the two FDs for reading/writing

    memset( from, 'x', LARGE_BUFFER_SIZE-1 ) ;
    memset (to, '-', LARGE_BUFFER_SIZE-1 ) ;

    int buffer_size = sizeof(from) ;

    if( pipe(pipe_fds) )
    {
        // non zero is an error!
        perror( "Pipe creation failed!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    size_t written = 0 ;
    while( buffer_size )
    {
        ssize_t write_return, read_return ;
        size_t write_amount = MIN( buffer_size, WRITE_CHUNK ) ;

        write_return = write( pipe_fds[1], &from[written], write_amount ) ;
        if( write_return < 0 )
        {
            perror( "Error writing to pipe!" ) ;
            exit(EXIT_FAILURE) ;
        }

        read_return = read( pipe_fds[0], &to[written], write_return ) ;
        assert( read_return == write_return ) ;

        // what's going on here?
        buffer_size -= write_return ;
        written += write_return ;
    }

    assert( memcmp( from, to, sizeof(from) ) == 0 ) ;
    printf( "from and to are IDENTICAL!\n" ) ;

    printf( "\n" ) ;
    return 0 ;
}

Now, let’s make this more interesting (and actually useful)! Let’s send data between processes, i.e., using fork()!

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <assert.h>

/* Large array containing 2^20 characters */
char from[1 << 20];
char to[1 << 20];

int main(void)
{
    int pipe_fds[2]; /* see `man 3 pipe`: `[0]` = read end, `[1]` = write end */
    pid_t pid;
    size_t buf_sz = sizeof(from);

    if (pipe(pipe_fds) == -1) {
        perror("pipe creation");
        exit(EXIT_FAILURE);
    }

    /* descriptors copied into each process during `fork`! */
    pid = fork();

    if (pid < 0) {
        perror("fork error");
        exit(EXIT_FAILURE);
    } else if (pid == 0) { /* child */
        ssize_t ret_w;

        close(pipe_fds[0]); /* we aren't reading! */
        ret_w = write(pipe_fds[1], from, buf_sz);
        if (ret_w < 0) {
            perror("write to pipe");
            exit(EXIT_FAILURE);
        }
        assert((size_t)ret_w == buf_sz);
        printf("Child sent whole message!\n");
    } else { /* parent */
        ssize_t ret_r;
        ssize_t rest = buf_sz, offset = 0;

        close(pipe_fds[1]); /* we aren't writing! */
        while (rest > 0) {
            ret_r = read(pipe_fds[0], &to[offset], rest);
            if (ret_r < 0) {
                perror("read from pipe");
                exit(EXIT_FAILURE);
            }
            rest   -= ret_r;
            offset += ret_r;
        }

        printf("Parent got the message!\n");
    }

    return 0;
}

Program output:

array = 3 2341 2344 8632 747645 

The concurrency of the system enables separate processes to be active at the same time, thus for the write and read to be transferring data through the pipe at the same time. This simplifies our code as we don’t need to worry about sending chunks of our data.

Note that we’re closeing the end of the pipe that we aren’t using in the corresponding processes. Though the file descriptors are identical in each process following fork, each process does have a separate set of those descriptors. Thus closing in one, doesn’t impact the other.

Remember, processes provide isolation!

9.1.3 The Shell

We can start to understand part of how to a shell might be implemented now!

Setting up pipes. Lets start with the more obvious: for each | in a command, the shell will create a new pipe. It is a little less obvious to understand how the standard output for one process is hooked up through a pipe to the standard input of the next process. To do this, the shell does the following procedure:

1. Create a pipe.
2. fork the processes (a fork for each process in a pipeline).
3. In the upstream process close(STDOUT_FILENO), and dup2 the writable file descriptor in the pipe into STDOUT_FILENO.
4. In the downstream process close(STDIN_FILENO), and dup2 the readable file descriptor in the pipe into STDIN_FILENO.

Due to this careful usage of close to get rid of the old standard in/out, and dup or dup2 to methodically replace it with the pipe, we can see how the shell sets up the processes in a pipeline!

Lets go over an example of setting up the file descriptors for a child process. This does not set up the pipe-based communication between two children, so is not sufficient for a shell; but it is well on the way. Pipes contain arbitrary streams of bytes, not just characters. This example will

1. setup the input and output of two files to communicate over a pipe, and
2. send and receive binary data between processes.

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/wait.h>

void perror_exit(char *s)
{
    perror(s);
    exit(EXIT_FAILURE);
}

int main(void)
{
    int fds[2];
    pid_t pid;

    /* make the pipe before we fork, so we can acccess it in each process */
    if (pipe(fds) == -1) perror_exit("Opening pipe");

    pid = fork();
    if (pid == -1) perror_exit("Forking process");

    if (pid == 0) {       /* child */
        /* Same as above, but for standard output */
        close(STDOUT_FILENO);
        if (dup2(fds[1], STDOUT_FILENO) == -1) perror_exit("child dup stdout");

        close(fds[0]);
        close(fds[1]);

        printf("%d %c %x", 42, '+', 42);
        fflush(stdout); /* make sure that we output to the stdout */

        exit(EXIT_SUCCESS);
    } else {              /* parent */
        int a, c;
        char b;

        /* close standard in... */
        close(STDIN_FILENO);
        /* ...and replace it with the input side of the pipe */
        if (dup2(fds[0], STDIN_FILENO) == -1) perror_exit("parent dup stdin");
        /*
         * if we don't close the pipes, the child will
         * always wait for additional input
         */
        close(fds[0]);
        close(fds[1]);

        scanf("%d %c %x", &a, &b, &c);

        printf("%d %c %x", a, b, c);
        if (wait(NULL) == -1) perror_exit("parent's wait");
    }

    return 0;
}

Program output:

42 + 2a

Closing pipes. reading from a pipe will return that there is no more data on the pipe (i.e. return 0) only if all write-ends of the pipe are closed.

This makes sense because we think of a pipe as a potentially infinite stream of bytes, thus the only way the system can know that there are no more bytes to be read, is if the write end of the pipe cannot receive more data, i.e. if it is closed.

This seems simple, in principle, but when implementing a shell, you use dup to make multiple copies of the write file descriptor. In this case, the shell must be very careful to close its own copies because if any write end of a pipe is open, the reader will not realize when there is no more data left. If you implement a shell, and it seems like commands are hanging, and not exiting, this is likely why.

Question. If we wanted to have a parent, shell process setup two child connected by |, what would the final picture (in the image of the pictures above) look like?

9.2 Signals

Slides

We’re used to sequential execution in our processes. Each instruction executes after the previous in the “instruction stream”. However, systems also require exceptional execution patterns. What happens when you access memory that doesn’t exist (e.g. *(int *)NULL); when you divide by 0; when a user types cntl-c; when a process terminates; or when you make a call to access a descriptor which somehow is not accessible outside of the call¹⁵?

Consider the following code where we try to access (dereference) address NULL.

/* CSC 2410 Code Sample 
 * intro to signals
 * Fall 2023
 * (c) Sibin Mohan
 */

int main()
{
    // some standard errors
    int* a = (int*)NULL ;
    *a = 10 ;

    return 0 ;
}

What happens with the following code?

/* CSC 2410 Code Sample 
 * intro to signals
 * Fall 2023
 * (c) Sibin Mohan
 */

int main()
{
    // some standard errors
    int div = 100/0 ;

    return 0 ;
}

Programs crash! Unless you plan to “handle” it!

9.2.1 Enter ‘signals’

UNIX’s signals provide asynchronous execution in a process: - provide asynchronous execution - when a signal activates, - a “signal handler” function is activated - regardless of what was executing!

Signals can be used for, - dealing with exceptions, e.g., - invalid access → *(int *)NULL - divide by 0 - tracking time → ualarm - parent/child coordination → SIGTERM - …

Let’s look at a basic setup for using signals:

#include <signal.h> /* sigaction and SIG* */
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>

// this is the interface for the signal fault handler!
void sig_fault_handler(int signal_number, siginfo_t *info, void *context)
{
    /* printf has problems here; see "The Dark Side of Signals" */
    printf("My downfall is the forbidden fruit at address %p.\n", info->si_addr);

    /* Question: what happens if we comment this out? */
    exit(EXIT_FAILURE);

    return;
}

int main(void)
{
    sigset_t masked;
    struct sigaction siginfo;
    int ret;

    sigemptyset(&masked);
    sigaddset(&masked, SIGSEGV);
    siginfo = (struct sigaction) {
        .sa_sigaction = sig_fault_handler,
        .sa_mask      = masked,
        .sa_flags     = SA_RESTART | SA_SIGINFO /* we'll see this later */
    };

    if (sigaction(SIGSEGV, &siginfo, NULL) == -1) {
        perror("sigaction error");
        exit(EXIT_FAILURE);
    }

    printf("Lets live dangerously\n");
    ret = *(int *)NULL;
    printf("I live!\n");

    return 0;
}

Program output:

42 + 2a

We can actually execute in the signal handler when we access invalid memory! We can write code to execute in response to a segmentation fault. This is how Java prints out a backtrace.

Let’s explore the concepts in the above code and the signals interface in general.

9.2.2 Signals Interface

9.2.2.1 sigset_t

• a bitmap → one bit per signal
• used to program the signal mask
• filled with 0s or 1s

value	meaning
	signals can nest
	signals cannot nest

Essentially….sigset_t tells us…when a signal executes, - can others occur? - if so, which ones?

9.2.2.2 sigset_t functions

Defined in <signal.h> to manage the sigset_t data structure. |name|function| |—-|——–| |sigemptyset| initialize/empty the signal set
i.e., “unmask” all| |sigfullset| initializefill the signal set
i.e., “mask” all|
|sigaddset|add specific signal to set
i.e., “mask” specific one| |sigdelset|remove specific signal from set
i.e., “mask” specific one| |sigismember|checks whether given signal is
part of the set|

9.2.2.3 sig_fault_handler

• the signal "handler"
• function that is called asynchronously
  • when corresponding event happens
• user defined

In the example of SIGSEGV above, here, the signal handler is called whenever we access invalid memory (e.g. segmentation fault).

9.2.2.4 sigaction

The function that sets up signal handler for a specific signal:

#include "signal.h"

int sigaction( int signum, 
               const struct sigaction *restrict act, 
               struct sigaction *restrict oldact ) ;

9.2.2.5 struct sigaction

struct sigaction {
            void     (*sa_handler)(int);
            void     (*sa_sigaction)(int, siginfo_t *, void *);
            sigset_t   sa_mask;
            int        sa_flags;
            void     (*sa_restorer)(void);
} ;

9.2.2.6 Sequence of Actions

…for using signals

9.2.3 Notable Signals

There are signals for quite a few events. A list of all of the signals is listed in man 7 signal and glibc has decent documentation. Notable signals include: |signal| description | |——–|——–| |SIGCHILD | child process terminated | |SIGINT | user typed ctrl-c | |SIGSTOP/SIGCONT | stop/continue child execution | |SIGTSTP | user typed ctrl-z | |SIGTPIPE | write to a pipe with no reader | |SIGSEGV | invalid memory access [segmentation fault] | |SIGTERM/SIGKILL | kill a process; SIGTERM can be caught, SIGKILL not | |SIGHUP | kill terminal that created shell | |SIGALRM | notification that time has passed | |SIGUSR1/SIGUSR2 | user-defined signal handlers | ||

Note: SIGKILL/SIGTOP - cannot be “caught” - to deal with unresponsive processes - SIGCONT → continue process after SIGSTOP

Each signal has a default behavior that triggers if you do not define a handler. These are: * ignore * terminate process * stop process from executing * continue execution

9.2.4 Minor Detour | sa_sigaction

We have seen (and used) this struct:

struct sigaction {
    void     (*sa_handler)(int);
    void     (*sa_sigaction)(int, siginfo_t *, void *);
    sigset_t   sa_mask;
    int        sa_flags;
    void     (*sa_restorer)(void);
};

Program output:

42 + 2a

Why two function pointers? - (*sa_handler) - (*sa_sigaction)

One takes more information than the other

void     (*sa_handler)(int);`
void     (*sa_sigaction)(int, siginfo_t *, void *);

Program output:

42 + 2a

choice of which → depends on a flag that is set.

9.2.4.1 Flags?

Provide flags to sa_flags member:

man 2 sigaction for more details.

9.2.5 Signals | Further Examples

9.2.5.1 Tracking Time with Signals

Now let’s use another signal, SIGALRM. Use the following code example:

/* CSC 2410 Code Sample 
 * intro to signals
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <signal.h>
#include <unistd.h>
#include <string.h>

// why volatile?
volatile int timers = 0 ;

// the signal hanlder
void my_timer_handler( int signal_number, siginfo_t* info,
                                            void* context )
{
    // Handler for SIGALRM

    printf( "Inside Alarm Handler!\n" ) ;

    ++timers ;
    return ;
}

// the function pointer for the signal handler type
typedef void(*signal_handler_function)( int, siginfo_t*, void* ) ;

// We can use a function to set up the signal, 
// hide all the sigset, sigaction stuff using this
// EXPLAIN this function
void setup_signal( int signal_number, 
                   signal_handler_function func,
                   int sa_flags )
{
    sigset_t masked ; // bitmask
    sigemptyset( &masked ) ; // clear the mask
    sigaddset( &masked, signal_number ) ; // set only bit for SIGSEGV
    struct sigaction siginfo = (struct sigaction){
        .sa_sigaction = func,
        .sa_mask = &masked,
        .sa_flags = sa_flags 
    } ; 

    if( sigaction( signal_number, &siginfo, NULL ) == -1 )
    {
        perror( "sigaction failed!" ) ;
        exit(EXIT_FAILURE) ;
    }
}

int main()
{
    int t = timers ;
    setup_signal( SIGALRM, my_timer_handler, (SA_RESTART | SA_SIGINFO) ) ;

    pid_t pid ;

    if( (pid = fork()) == 0 )
    {
        // Child process
        pause() ;
        exit( EXIT_SUCCESS ) ;
    }

    // We did setup_signal BEFORE fork(). Parent/child both get signal info!
    
    ualarm( 1000, 1000 ) ; // 1000 us --> 1 ms
    // alarm(1) ; // same as previous ualarm? SHOW BOTH!

    while (t < 10)
    {
        if( timers > t )
        {
            printf( "Count: %d\n", t ) ;
            t = timers ;
        }
    }

    printf( "\n" ) ;
    return 0 ;
}

*Question**: Track and explain the control flow through this program.

SIGKILL and SIGSTOP are unique in that they cannot be disabled, and handlers for them cannot be defined. They enable non-optional control of a child by the parent.

9.2.5.2 Parent/Child Coordination with Signals

Another example of coordination between parent and child processes. We can use signals to get a notification that a child has exited! Additionally, we can send the SIGTERM signal to terminate a process (this is used to implement the kill command line program – see man 1 kill).

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <unistd.h> /* kill, pause */
#include <assert.h>
#include <sys/wait.h>

void sig_handler(int signal_number, siginfo_t *info, void *context)
{
    switch(signal_number) {
    case SIGCHLD: {
        /* see documentation on `siginfo_t` in `man sigaction` */
        printf("%d: Child process %d has exited.\n", getpid(), info->si_pid);
        fflush(stdout);
        break;
    }
    case SIGTERM: {
        printf("%d: We've been asked to terminate. Exit!\n", getpid());
        fflush(stdout);
        exit(EXIT_SUCCESS);
        break;
    }}

    return;
}

void setup_signal(int signo, void (*fn)(int , siginfo_t *, void *))
{
    sigset_t masked;
    struct sigaction siginfo;
    int ret;

    sigemptyset(&masked);
    sigaddset(&masked, signo);
    siginfo = (struct sigaction) {
        .sa_sigaction = fn,
        .sa_mask      = masked,
        .sa_flags     = SA_RESTART | SA_SIGINFO
    };

    if (sigaction(signo, &siginfo, NULL) == -1) {
        perror("sigaction error");
        exit(EXIT_FAILURE);
    }
}

int main(void)
{
    pid_t pid;
    int status;

    setup_signal(SIGCHLD, sig_handler);
    setup_signal(SIGTERM, sig_handler);

    /*
     * The signal infromation is inherited across a fork,
     * and is set the same for the parent and the child.
     */
    pid = fork();
    if (pid == -1) {
        perror("fork");
        exit(EXIT_FAILURE);
    }

    if (pid == 0) {
        pause(); /* stop execution, wake upon signal */

        exit(EXIT_SUCCESS);
    }

    printf("%d: Parent asking child (%d) to terminate\n", getpid(), pid);
    kill(pid, SIGTERM); /* send the child the TERM signal */

    /* Wait for the sigchild notification of child termination! */
    pause();

    /* this should return immediately because waited for sigchld! */
    assert(pid == wait(&status));
    assert(WIFEXITED(status));

    return 0;
}

Program output:

42 + 2a

Note: You want to run this a few times on your system to see the output. The auto-execution scripts of the lectures might cause wonky effects here due to concurrency.

We now see a couple of new features:

• The SIGCHLD signal is activated in the parent when a child process exits.
• We can use the kill function to send a signal to a another process owned by the same user (e.g. gparmer).
• The pause call says to stop execution (to pause) until a signal is triggered.

A couple of additional important functions:

• raise will trigger a signal in the current process (it is effectively a kill(getpid(), ...)).
• ualarm will set a recurring SIGALRM signal. This can be quite useful if your program needs to keep track of time in some way.

9.2.6 The Dark Side of Signals

Signals are dangerous mechanisms in some situations. It can be difficult to use them properly, and avoid bugs. Signals complication data-structures as only functionality that is re-eentrant should be used in signal handlers, and they complicate the logic around all system calls as they can interrupt slow system calls.

Two main problems: 1. problems with “slow” system calls 2. only “reentrant” data structures

9.2.6.1 “Slow” System Calls

• many library calls can block
  • e.g., wait or read
• what if → signal sent when blocked?
• default → sys call will return immediately
• wait → returns even if child didn’t exit
• read → returns despite not reading data!

So how do you tell the difference between the blocking function returning properly, or returning because it was interrupted by a signal? The answer is, of course, in the man pages – look at the return/errno values: - function will return -1 - errno will be set to EINTR

Given this, we see the problem with this design: now the programmer must add logic for every single system call that can block to check this condition. Yikes¹⁶.

Luckily, UNIX provides a means to disable the interruption of blocking calls by setting the SA_RESTART flag to the sa_flags field of the sigaction struct passed to the sigaction call.

Note: that the code above already sets this as I consider it a default requirement if you’re setting up signal handlers.

The use of SA_RESTART can have interesting side effects, especially for slow system calls! Lets see the explicit interaction with between the slow call wait, and the signal handler:

/* CSC 2410 Code Sample 
 * intro to signals
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <signal.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>

void my_timer_handler( int signal_number, siginfo_t* info,
                                            void* context )
{
    // Handler for SIGALRM

    printf( "Inside Alarm Handler!\n" ) ;

    return ;
}

// the function pointer for the signal handler type
typedef void(*signal_handler_function)( int, siginfo_t*, void* ) ;

// We can use a function to set up the signal, 
// hide all the sigset, sigaction stuff using this
void setup_signal( int signal_number, 
                   signal_handler_function func,
                   int sa_flags )
{
    sigset_t masked ; // bitmask
    sigemptyset( &masked ) ; // clear the mask
    sigaddset( &masked, signal_number ) ; // set only bit for given signal
    struct sigaction siginfo = (struct sigaction){
        .sa_sigaction = func,
        .sa_mask = &masked,
        .sa_flags = sa_flags 
    } ; 

    if( sigaction( signal_number, &siginfo, NULL ) == -1 )
    {
        perror( "sigaction failed!" ) ;
        exit(EXIT_FAILURE) ;
    }
}

int main()
{

    // comment out one or the other of this to see the different behaviors
    setup_signal( SIGALRM, my_timer_handler, (SA_SIGINFO) ) ;
    // setup_signal( SIGALRM, my_timer_handler, (SA_RESTART | SA_SIGINFO) ) ;

    pid_t pid ;

    if( (pid = fork()) == 0 )
    {
        // Child process
        pause() ; // wait for a signal
        exit( EXIT_SUCCESS ) ;
    }

    alarm(1);

    while (1)
    {
        pid_t ret = wait(NULL) ;

        if( ret == -1 )
        {
            if( errno == EINTR )
            {
                // Child didn't exit properly

                printf( "System call interrupted by Signal\n" ) ;
                kill( pid, SIGTERM ) ; // end the child process

                // return -1 ;
            }
            else if( errno == ECHILD )
            {
                // this code may NEVER execute!
                printf( "Child exited cleanly\n" ) ;
                return 0 ;
            }
        }
    }

    printf( "\n" ) ;
    return 0 ;
}

Program output:

42 + 2a

Comment out one of the other of the setup_signal function calls in main() to see very different behaviors.

9.2.6.2 Re-entrant Computations

Signal handlers execute by interrupting the currently executing instructions, regardless what computations they are performing. Because we don’t really know anything about what was executing when a signal handler started, we have to an issue. What if an action in the signal handler in some way conflicts with the action it interrupted?

Consider printf, - copies data into a buffer - calls write → send buffer to standard output - a signal raised between the two? - signal calls printf!

The data written by the earlier printf() will be overwritten/discarded by the later one!

Any function that has these issues is called non-reentrant. Yikes.

Consider the following example that overwrites errno with (potentially) disastrous effects!

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <unistd.h>
#include <string.h>
#include <assert.h>

void sig_handler(int signal_number, siginfo_t *info, void *context)
{
    /*
     * Reset `errno`! In a real program, this might instead be a call that causes
     * an error setting `errno` to whatever the error is for *that call*.
     */
    errno = 0;

    return;
}

void setup_signal(int signo)
{
    sigset_t masked;
    struct sigaction siginfo;
    int ret;

    sigemptyset(&masked);
    sigaddset(&masked, signo);
    siginfo = (struct sigaction) {
        .sa_sigaction = sig_handler,
        .sa_mask      = masked,
        .sa_flags     = SA_RESTART | SA_SIGINFO
    };

    if (sigaction(signo, &siginfo, NULL) == -1) {
        perror("sigaction error");
        exit(EXIT_FAILURE);
    }
}

int main(void)
{
    setup_signal(SIGUSR1);

    assert(read(400, "not going to work", 10) == -1);
    raise(SIGUSR1);
    printf("errno should be \"Bad file descriptor\", but has value \"%s\"\n", strerror(errno));

    return 0;
}

Program output:

42 + 2a

The set of functions you can call in a signal handler (i.e. that are re-entrant) are listed in the manual page: man 7 signal-safety.

Notably these do not include the likes of - printf/snprintf - malloc - exit (though _exit is fine) - functions that set errno! - …

It is hard to do much in a program of any complexity without snprintf (called by printf), malloc, or use errno. A very common modern way to handle this situation is to create a pipe into which the signal handlers write a notification (e.g. the signal number), while the main flow execution reads from the pipe. This enables the main execution in our programs to handle these notifications. However, this is only really possible and feasible when we get to poll later, and can block waiting for any of a number of file descriptors, including this pipe.

Note that in most cases, you won’t see a bug due to using non-re-entrant functions in your signal handlers. These bugs are heavily non-deterministic, and are dependent on exactly what instructions were interrupted when the signal activated. This may feel like a good thing: buggy behavior is very rare! But reality is opposite: rare, non-deterministic bugs become very very hard to debug and test. Worse, these bugs that pass your testing are more likely to happen in customer’s systems that might have different concurrency patterns. So it is nearly impossible to debug/test for these bugs, and they are more likely to cause problems in your customer’s environment. Again, yikes.

10 Files and File Handling

Slides

The filesystem is one of the core abstractions on UNIX systems and indeed, on modern OSes. Each file is identified by a path through directories.

Remember the UNIX philosophy: everything is a file!

This is a strange statement as it raises the question “what wouldn’t normally be a file?” Some examples:

Note:

• We know of processes as executable instances of programs so certainly they cannot be files, right? We’ve seen that they are represented by files in /proc/*! These files are used to provide “task monitor” type functionality (showing running processes) and gdb.
• Devices are the physical parts of our computers like screens, keyboards, USB devices, hard-drives, and networks. They are all represented by files in /dev/*! You can actually cat a file directly to your disk¹⁷!
• Want random values? /dev/random.
• Want the complete absence of anything? /dev/null¹⁸.
• Want to update your power settings? There are files in /sys/* for that.

Remember the high-level directory structure of modern Linux:

10.1 Basic File Access

When everything is a file, it can all be manipulated and accessed using the exactly same functions and APIs as the normal files in the system, e.g., open, close, read, write, etc.

This means that all of the shell programs we’ve seen can be used not just to operate on files, but also on processes or devices!

Here are the basic APIs for file operations:

1. open( path, flags, ... )
   • open a file → identified by `path
   • return → file descriptor → to access file

“flags” must be one of

• O_RDONLY → only read from the file
• O_WRONLY → only write from the file
• O_CREAT → create the file if it doesn’t exist
• can use bitwise OR: O_RDONLY | O_CREAT

When using O_CREAT → pass the third arg, “mode”:

open("my_file_name.txt", O_RDWR | O_CREAT, 0700)

Whenever you see “flags”, you should think of them as a set of bits, and each of the options as a single bit. The above example will create the file, my_file_name.txt if it doesn’t exist already and open it for reading and writing.

Note that when you pass in O_CREAT, you should pass in the third argument, the mode (for now, just always pass in 0700!).

2. read, write
   • generic functions for
     • getting data from,
     • sending data to,
   • “descriptors” → file descriptors in this case.

We’ve seen them before when using pipes!

3. close
   • to “close” any descriptor
   • akin to a free for descriptors
4. stat
   • get information about file pointed by path
   • into the info structure
   • int stat(path, struct stat *info)
   • fstat → same, but uses fd

    struct stat { 
    dev_t    st_dev;    /* device inode resides on */
    ino_t    st_ino;    /* inode's number */
    mode_t   st_mode;   /* inode protection mode */
    nlink_t  st_nlink;  /* number of hard links to the file */
    uid_t    st_uid;    /* user-id of owner */
    gid_t    st_gid;    /* group-id of owner */
    dev_t    st_rdev;   /* device type, for special file inode */
    struct timespec st_atimespec;  /* time of last access */
    struct timespec st_mtimespec;  /* time of last data modification */
    struct timespec st_ctimespec;  /* time of last file status change */
    off_t    st_size;   /* file size, in bytes */
    quad_t   st_blocks; /* blocks allocated for file */
    u_long   st_blksize;/* optimal file sys I/O ops blocksize */
    u_long   st_flags;  /* user defined flags for file */
    u_long   st_gen;    /* file generation number */
};

The structure is documented in the man page, but it includes, for example, the file size. It is defined in <sys/stat.h>.

5. unlink()
   • try and remove a file
   • e.g., called by the rm program

It is really important that quite a few interesting functions operate on descriptors, which might reference pipes, files, the terminal, or other resources. This means that processes can operate on descriptors and not care what resources back them. This enables shells to implement pipelines and the redirection of output to files, all without the processes knowing! It enables forked processes to inherit these resources from its parent. In this way, descriptors and the functions to operate on them are a polymorphic foundation (in the sense of object-oriented polymorphism) to accessing the system.

Now, let’s look at an example of basic file operations:

/* CSC 2410 Code Sample 
 * intro to reading and writing on files
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <assert.h>
#include <sys/stat.h>

#define TWEET_LEN 280

int main()
{
    char tweet[TWEET_LEN] ;
    int fd = open( "./daffodils.txt", O_RDONLY ) ;
    if( fd == -1 )
    {
        perror( "File daffodils.txt failed to open!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    // what is the size of the file?
    struct stat file_info ;
    int ret = fstat( fd, &file_info ) ;
    assert( ret >=0 ) ;
    printf( "Number of characters in file: %ld\n", file_info.st_size ) ;
    
    // Read TWEET_LEN number of characters from file
    ret = read( fd, tweet, TWEET_LEN ) ;
    assert( ret == TWEET_LEN ) ;
    write( STDOUT_FILENO, tweet, TWEET_LEN  ) ;

    ret = read( fd, tweet, TWEET_LEN ) ;
    assert( ret == TWEET_LEN ) ;
    write( STDOUT_FILENO, tweet, TWEET_LEN  ) ;

    close(fd) ; // close the file.

    return 0 ;
}

10.2 Moving Around in Files

We see something a little strange with files as compared to pipes.

Files vs Pipes:

For files, subsequent reads and writes must progress through the contents of the file, but we must also to be able to go back and read previously read data.

Lets understand what happens when we read or write from a file – we’ll focus only on read to start, but both behave similarly.

First, each fd¹⁹ tracks an “offset” - offset determines where we read/write in file

Lets go through an example of a file that contains the alphabet:

For freshly opened files → offset = 0

A read(fd, buf, 10) that returns 10 (saying that 10 bytes or characters a through j was successfully read) advances offset by 10.



Thus, an additional read will start reading from the file at k. The offset, then, enables subsequent reads to iterate through a file’s contents.

write() uses the offset identically.

There are many cases in which we might want to modify the offset. For example, databases want to jump around a file to find exactly the interesting records, playing audio and video requires us to skip around a file finding the appropriate frames. So, we use:

off_t lseek(int fd, off_t update, int whence)

“whence”?

how to update the offset

eof → end of file

Let’s update our previous example to use lseek() to reset to the start of the file.

/* CSC 2410 Code Sample 
 * intro to reading and writing on files
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <assert.h>
#include <sys/stat.h>

#define TWEET_LEN 280

int main()
{
    char tweet[TWEET_LEN] ;
    int fd = open( "./daffodils.txt", O_RDONLY ) ;
    if( fd == -1 )
    {
        perror( "File daffodils.txt failed to open!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    // what is the size of the file?
    struct stat file_info ;
    int ret = fstat( fd, &file_info ) ;
    assert( ret >=0 ) ;
    printf( "Number of characters in file: %ld\n", file_info.st_size ) ;
    
    // Read TWEET_LEN number of characters from file
    ret = read( fd, tweet, TWEET_LEN ) ;
    assert( ret == TWEET_LEN ) ;
    write( STDOUT_FILENO, tweet, TWEET_LEN  ) ;

    ret = read( fd, tweet, TWEET_LEN ) ;
    assert( ret == TWEET_LEN ) ;
    write( STDOUT_FILENO, tweet, TWEET_LEN  ) ;

    // Reset to the start of the file
    ret = lseek( fd, 0, SEEK_SET ) ;
    if( ret == -1 )
    {
        perror( "lseek failed!" ) ;
        exit( EXIT_FAILURE ) ;
    } 

    printf( "\n\n ------ AFTER LSEEK ---------\n\n" ) ;
    // Read TWEEDT_LEN number of characters from file
    ret = read( fd, tweet, TWEET_LEN ) ;
    assert( ret == TWEET_LEN ) ;
    write( STDOUT_FILENO, tweet, TWEET_LEN  ) ;

    close(fd) ; // close the file.

    return 0 ;
}

10.3 Other Ways to Access Files

So far,

• read & write are fine interfaces for files
• require many interactions with multiple calls

There are two additional methods to access files:

• memory mapping
• streams

10.3.1 Memory Mapping Files | mmap()

UNIX also provides a way to directly “map” a file into a process, enabling to appear directly in memory, thus be accessible using normal memory accesses.

This is cool because your changes in memory update the file directly!

void *mmap(addr, len, prot, flags, fd, offset)

defined in <sys/mman.h>.

10.3.1.1 mmap arguments:

return value is the address at which the file is mapped.

Overview:

Some nuances:

1. The flags argument indicates the type of mapped object:

So, if you want you changes to be reflected in the actual file, then make sure you set the flag as: MAP_SHARED.

Look at man mmap for more details on the prot and flags options.

2. Normally, addr, the address in memory at which you want to map, is NULL, which means “please map it whereever you want”. The return value is the address at which the file is mapped, or NULL if there was an error (see the man page and errno values).

3. stat is quite useful to find the file’s size if you want to map it all in.

4. prot enables you to choose some properties for the mapping, e.g., PROT_READ, and PROT_WRITE that ask the system to map the memory in readable, or writable modes respectively (you can choose both with PROT_READ | PROT_WRITE). Note: the type of accesses here must match request accesses previously requested in open – for example, if we passed O_RDONLY in to open, mmap will return an error if we ask for PROT_WRITE.

10.3.1.2 munmap()

• unmap a previously mapped file
• deletes “mapping” for specified address range

int munmap(void *addr, size_t len);

Consider the following code:

/* CSC 2410 Code Sample 
 * accessing files using mmap()
 * Fall 2023
 * (c) Sibin Mohan
 */

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <string.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <assert.h>
#include <sys/types.h>
#include <sys/stat.h>

#define TWEET_LEN 280
#define NUM_TWEETS 2

int main()
{
    int fd = open( "./daffodils.txt", O_RDONLY ) ;
    // int fd = open( "./daffodils.txt", O_RDWR ) ;
    if( fd == -1 )
    {
        perror( "File daffodils.txt failed to open!" ) ;
        exit(EXIT_FAILURE) ;
    }

    // char* addr = mmap( NULL, TWEET_LEN, ( PROT_READ | PROT_WRITE ), MAP_SHARED, fd, 0 ) ;
    char* addr = mmap( NULL, TWEET_LEN, ( PROT_READ | PROT_WRITE ), MAP_PRIVATE, fd, 0 ) ;
    if( addr == NULL )
    {
        perror( "nmmap failed!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    // write out a tweet length 
    write( STDOUT_FILENO, addr, TWEET_LEN ) ;

    // change the case of "by" to "BY"
    char* by = strstr( addr, "by" ) ;
    assert(by) ;
    by[0] = 'B' ;
    by[1] = 'Y' ;

    // printf( "%c %c\n", by[0], by[1] ) ;

    // write out a tweet length 
    write( STDOUT_FILENO, addr, TWEET_LEN ) ;

    munmap( addr, TWEET_LEN ) ;
    close(fd) ;

    printf( "\n" ) ;
    return 0 ;
}

Note:

1. will the above code work?
2. will the file reflect the changes?
3. what happens if the following line is uncommented (and the other, corresponding line is commented out?)

char* addr = mmap( NULL, TWEET_LEN, ( PROT_READ | PROT_WRITE ), MAP_SHARED, fd, 0 ) ;

4. how will you fix the (inevitable) problem that occurs?

10.3.1.3 mmap and memory allocation

• can use mmap to allocate memory!
• fd = 0
• prot = PROT_READ | PROT_WRITE
• flags = MAP_ANONYMOUS | MAP_PRIVATE
• mmap returns → newly allocated memory
• malloc() actually calls mmap!

10.3.2 Stream-Based I/O

There is actually a third way to access files beyond “raw” read/write and mmap – streams! Streams are an abstraction on top of the “raw” descriptor accesses using read and write.

Streams:

• a sequence of characters
• includes functions to
  • put characters in one end
  • take characters out on one end

Streams are identified by the type FILE *, and all the APIs for operating on streams take FILEs instead of file descriptors.

We’ve seen streams before…a lot!

• stdin, stdout, stderr
• FILE* names
• STDOUT_FILENO, STDIN_FILENO and STDERR_FILENO
• printf() writes to a stream! Hence, it is buffered I/O

Why use Streams?

From the GNU manual: “Streams provide a higher-level interface, layered on top of the primitive file descriptor facilities. The stream interface treats all kinds of files pretty much alike…The main advantage of using the stream interface is that the set of functions for performing actual input and output operations (as opposed to control operations) on streams is much richer and more powerful than the corresponding facilities for file descriptors. The file descriptor interface provides only simple functions for transferring blocks of characters, but the stream interface also provides powerful formatted input and output functions (printf and scanf) as well as functions for character- and line-oriented input and output.”

There is also the issue of portability: “(raw) file descriptors are not as portable as streams. You can expect any system running ISO C to support streams, but non-GNU systems may not support file descriptors at all, or may only implement a subset of the GNU functions that operate on file descriptor”

file streams → opened/read/written/closed as streams

10.3.2.1 File Stream Interface

Some notes:

1. access rights (while opening a file) that we’re asking for (reading or writing to the file), are specified in a more intuitive mode string, e.g., "r" to read, "r+" for read/write, etc.
2. streams provide the feof function that tells us if we are at the end of a file.

Now, let’s look at our previous example of printing the daffodils.txt file in tweet lengths, this time using streams:

#include <stdio.h>
#include <assert.h>

int main(void)
{
    // why do we need the +1?
    char tweet[TWEET_LEN+1] ;

    FILE* f = fopen( "./daffodils.txt", "r" ) ;
    if( f == NULL )
    {
        perror( "File open failed!" ) ;
        exit( EXIT_FAILURE ) ;
    }

    // read TWEET_LEN characters from the file
    int ret = fread( tweet, TWEET_LEN, 1, f ) ;
    tweet[TWEET_LEN] = '\0' ;

    // write it to stdout
    fprintf( stdout, "%s", tweet ) ;
    fflush(stdout) ;

    fclose(f);

    printf( "\n" ) ;
    return 0;
}

Program output:

42 + 2a

10.3.2.2 Streams as buffered I/O

So it doesn’t seem like we’re getting much of use out of these streams compared to raw descriptors. Streams are an optimization over read/write as they buffer data.

Buffering essentially means that your fwrites actually write data into a “buffer” in memory, and only when either you write out a \n or when the buffer gets large enough does a write get called to send the data to the system.

Why do this?

By buffering data in memory rather than making a write for each fwrite, we’re saving any overheads associated with each write (to be discussed in a couple of weeks).

However, this means that just because we called printf, doesn’t mean it will output immediately!

Lets see this how this works in practice:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    printf("hello world");
    _exit(EXIT_SUCCESS); /* this exits immediately without calling `atexit` functions */

    return 0;
}

Program output:

42 + 2a

Well that’s not good!

The buffered data is copied into each process, then it is output when an \n is encountered.

#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(void)
{
    printf("hello ");
    write(STDOUT_FILENO, "world ", 6);
    printf("\n"); /* remember, streams output on \n */

    return 0;
}

Program output:

42 + 2a

Both printf and the write are, technically, to the standard output, but printf is to a stream. The buffered printf output is not written out immediately, thus ordering can get messed up.

Yikes.

Last example of the complexities of streams. What happens if you get a segfault!? Runtime errors can mess up expected behaviors.

#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(void)
{
    int a;

    printf("hello ");
    a = *(int *)NULL;

    return 0;
}

Program output:

42 + 2a

Even though we fault after the printf, we don’t see the printf’s output!

We now know why: the printf wrote into a buffer, and didn’t yet write to the system! Thus the segmentation fault, that terminates the process, happens before the buffer is actually written!

10.3.2.3 Flushing Stream Buffers

It is imperative that streams give us some means to force the buffer to be output! Thus, streams provide a means of flushing the stream’s buffers, and sending them out to the system (e.g. using write).

• fflush - flush out the buffer associated with a specific stream.

Fixing the previous examples:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    printf("hello ");
    fflush(stdout); // "flush" out the buffer to standard output
    write(STDOUT_FILENO, "world ", 6);
    printf("\n"); /* remember, streams output on \n */

    printf("hello world");
    fflush(stdout);
    _exit(EXIT_SUCCESS); /* this exits immediately without calling `atexit` functions */

    return 0;
}

Program output:

42 + 2a

#include <stdio.h>
#include <unistd.h>

int main(void)
{
    printf("hello world");
    fflush(stdout); // "flush" out the buffer to standard output
    fork();
    printf("\n"); /* remember, streams output on \n */

    return 0;
}

Program output:

42 + 2a

10.3.2.4 Other Useful Stream Functions

A few other functions for streams that can be useful:

• fileno: get the file descriptor number associated with a stream. This can be useful if you need to use an API that takes a descriptor, rather than a stream * FILE.
• getline: read out a line (delimited by a \n) from a stream. Since reading input, line at a time, is a pretty common thing, this is a useful function.

#include <stdio.h>
#include <assert.h>
#include <stdlib.h>
#include <string.h>

int main(void)
{
    FILE *f = fopen("./05/prufrock.txt", "r");
    size_t s = 0;
    char *line = NULL;
    int i;

    assert(f);
    printf("The new genre: abridged, tweet-worthy poetry...\n\n");
    for (i = 0; getline(&line, &s, f) != -1; i++) {
        if (i % 15 == 0) {
            fwrite(line, 1, strlen(line), stdout);
            /* same as printf("%s", line); */
        }
    }
    if (!feof(f)) {
        perror("getline");
        exit(EXIT_FAILURE);
    }
    free(line);
    fclose(f);

    return 0;
}

Program output:

42 + 2a

10.4 Accessing Directories

Files aren’t the only thing we care to access in the system – what about directories! We want to be able to create directories, delete them, and be able to read their contents. These core operations are what underlies the implementation of the ls program, or, similarly, what happens when you double-click on a directory in a graphical file explorer!

• opendir, closedir - Open (and close) a directory, and return a directory stream, a DIR *, to it, which is backed by a descriptor. Yes, another descriptor!

• readdir - This takes a DIR * to a directory, and returns a structure that includes the “current” file or directory in the directory. This includes the name of the returned object in d_name, and type (in the d_type field) which is one of:

  • DT_BLK - This is a block device.
  • DT_CHR - This is a character device.
  • DT_DIR - This is a directory.
  • DT_FIFO - This is a named pipe (FIFO).
  • DT_LNK - This is a symbolic link.
  • DT_REG - This is a regular file.
  • DT_SOCK - This is a UNIX domain socket.
  • DT_UNKNOWN - The file type could not be determined.

  The two main ones we care about are directories and files. Subsequent calls to readdir will return the “next” file or directory.

• scandir & glob - Two functions that enable you to get directory contents based on some simple search patterns and logic. scandir lets you pass in a directory stream, and a couple of functions that are used to sort directory contents output, and filter it (enable you to say that you don’t want to look at some directory contents). glob, on the other hand, lets you search for “wildcards” designated by * to select all files/direcotries that match the string.

#include <sys/types.h>
#include <sys/stat.h>
#include <dirent.h>
#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

long file_size(char *dir, char *file);

int main(void)
{
    DIR *d = opendir("./05/");
    struct dirent *entry;

    assert(d);

    errno = 0;
    while ((entry = readdir(d)) != NULL) {
        char *type;

        if (entry->d_type == DT_DIR)      type = "Directory";
        else if (entry->d_type == DT_REG) type = "File";
        else                              type = "Unknown";

        if (entry->d_type == DT_DIR || entry->d_type != DT_REG) {
            printf("%10s: %s\n", type, entry->d_name);
        } else { /* entry->d_type == DT_REG */
            printf("%10s: %s (size: %ld)\n", type, entry->d_name, file_size("05", entry->d_name));
        }
    }
    if (errno != 0) {
        perror("Reading directory");
        exit(EXIT_FAILURE);
    }
    closedir(d);

    return 0;
}

long file_size(char *dir, char *file)
{
    struct stat finfo;
    char buf[512];
    int ret;

    memset(buf, 0, 512); /* zero out the buffer to add '\0's */
    snprintf(buf, 512, "./%s/%s", dir, file);

    ret = stat(buf, &finfo);
    assert(ret == 0);

    return finfo.st_size;
}

Program output:

42 + 2a

To make changes in the first system hierarchy, we need an additional set of functions.

• mkdir(path, mode) - This function is used in, for example, the mkdir program. Don’t confuse the program you use in the shell, with the function you can call from C.
• rmdir(path) - Deletes a directory that is empty. This means that you have to unlink and rmdir the directory’s contents first.
• rename(from, to) - Change the name of file or directory from from to to. You can imagine this is used by the mv command line program.

#include <sys/stat.h>
#include <sys/types.h>
#include <assert.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>

int main(void)
{
    int ret;
    int fd;

    ret = mkdir("05/newdir", 0700);
    assert(ret == 0);
    fd = open("05/newdir/newfile", O_RDWR | O_CREAT, 0700);
    assert(fd >= 0);
    ret = write(fd, "new contents", 13);
    assert(ret == 13);
    ret = rename("05/newdir", "05/newerdir");
    assert(ret == 0);
    ret = unlink("05/newerdir/newfile");
    assert(ret == 0);
    ret = rmdir("05/newerdir");
    assert(ret == 0);

    printf("If there were no errors, we\n\t"
        "1. created a directory, \n\t"
        "2. a file in the directory, \n\t"
        "3. change the directory name,\n\t"
        "4. removed the file, and\n\t"
        "5. removed the directory\n");

    return 0;
}

Program output:

42 + 2a

10.5 Exercises

Write a tree clone, with disk usage information. Though it isn’t installed, by default, tree outputs the filesystem at a specific directory:

$ tree .
.
├── 01_lecture.md
├── 02_exercises.md
├── prufrock.txt
└── test_directory
    └── crickets.txt

10.5.1 Task 1: Markdown Tree

We’ll simplify the output to print out markdown lists with either D or F for directories or files. If we run the program in the 05 directory:

$ ./t
- 01_lecture.md (20759)
- 02_exercises.md (1683)
- prufrock.txt (6044)
- test_directory
    - crickets.txt (26)

Some starter code that focuses on printing out info for a single directory.

Task: Change this code to print out the hierarchy of files and directories, rather than just the contents of a single directory.

#include <sys/types.h>
#include <sys/stat.h>
#include <dirent.h>
#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* helper functions */
void indent(int n);
int ignoredir(char *dir);
long file_size(char *dir, char *file);

/*
 * Return the size of the directory (sum of all files inside).
 * The easiest implementation here is recursive where this is called
 * for each directory.
 */
size_t
print_dir(char *dir, int depth)
{
    DIR *d = opendir(dir);
    struct dirent *entry;

    assert(d);
    errno = 0;

    /* Go through each dir/file in this directory! */
    while ((entry = readdir(d)) != NULL) {
        /* we'll ignore . and .. */
        if (ignoredir(entry->d_name)) continue;

        /* print out the proper indentation */
        indent(depth);
        if (entry->d_type == DT_DIR) {
            printf("- D %s\n", entry->d_name);
        } else if (entry->d_type == DT_REG) {
            printf("- F %s (%ld)\n", entry->d_name, file_size(dir, entry->d_name));
        }
        /* we'll ignore everything that isn't a file or dir */
    }
    if (errno != 0) {
        perror("Reading directory");
        exit(EXIT_FAILURE);
    }
    closedir(d);

    return 0;
}

int
main(void)
{
    print_dir("./", 0);

    return 0;
}

/* Indent `n` levels */
void
indent(int n)
{
    for (int i = 0; i < n; i++) printf("    ");
}

/* Should we ignore this directory? */
int
ignoredir(char *dir)
{
    return strcmp(dir, ".") == 0 || strcmp(dir, "..") == 0;
}

long
file_size(char *dir, char *file)
{
    struct stat finfo;
    char buf[512];
    int ret;

    memset(buf, 0, 512); /* zero out the buffer to add '\0's */
    snprintf(buf, 512, "%s/%s", dir, file);

    ret = stat(buf, &finfo);
    assert(ret == 0);

    return finfo.st_size;
}

Program output:

42 + 2a

10.5.2 Task 2: File and Directory Sizes

Task: Add output for the size of files, and the size of each directory – the sum of the size of all contained directories and files.

Again, if we run the program in the 05 directory:

$ ./t
- 01_lecture.md (20759)
- 02_exercises.md (1683)
- prufrock.txt (6044)
- test_directory
    - crickets.txt (26)
    (26)
(28512)

The test_directory size is the (26), while the ./ directory has size (28512). Your specific sizes might differ.

11 I/O: Inter-Process Communication

We’ve seen that processes often coordinate through pipes, signals, and wait. This is necessary when coordinating between parents and children in shells (both on the terminal, and GUIs). These are useful when children have a common parent (thus can share pipes), and where the parent knows that the children want to communicate before they are execed. Modern systems require a lot of flexibility than this. Generally, we want a process to be able to decide who to coordinate with as they execute given their own goals. For example, the shell – when it executes a child – doesn’t know if it wants to have a GUI, thus needs to talk to the window server.

We require more dynamic, more flexible forms of coordination between processes. Generally, we call this coordination Inter-Process Communication or IPC. When do we generally want more flexible forms of IPC than pipes, signals, and wait?

1. If the parent doesn’t know what other processes with which a child wants IPC.
2. If the parent didn’t create the other process with which a child wants IPC.
3. The process with which we want IPC belongs to a different user.
4. The process with which we want IPC has access to special resources in the system (e.g. graphics cards, other hardware, or filesystem data).

Even applications that are implemented as many processes often require IPC between those processes.

11.1 IPC Example: Modern Browsers

Browsers all require a lot of IPC. Each tab is a separate process, and there is a single browser management process that manages the GUI and other browser resources. This is motivated by security concerns: a tab that fails or is compromised by an attacker cannot access the data of another tab! In the early (first 15-20 years) of browsers, this wasn’t the case, and you had to be careful about opening your banking webpage next to webpages that were suspect. The browser management process communicates with each of the child tab processes to send them user input (mouse clicks, keyboard input), and receive a bitmap to display on screen. Lets see this structure:

$ ps aux | grep firefox
... 8029 ... /usr/lib/firefox/firefox -new-window
... 8151 ... /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser ... 8029 ...
...

I have an embarrassing number of tabs open, so I’ve taken the liberty of manually filtering this output. We can see that the process with pid 8029 is the parent, browser management process, and the process 8151 is a child managing a tab (note, you can see processes arranges as a process tree using ps axjf). There are many other children processes managing tabs.

Lets try and figure out how they communicate with each other. First, lets look at the file descriptors of the tab’s processes (the child).

$ ls -l /proc/8151/fd/
0 -> /dev/null
1 -> socket:[100774]
...
11 -> /memfd:mozilla-ipc
4 -> socket:[107464]
5 -> socket:[107467]
7 -> pipe:[110438]

We see that there is no standard input (/dev/null is “nothing”), and that there are a few means of IPC:

• pipe - We know these!
• memfd - A Linux-specific way of sharing a range of memory between the processes. Stores to the memory can be seen by all sharing processes.
• socket - Another pipe-line channel in which bytes written into the socket can be read out on the other side. A key difference is that socket connections between processes can be created without requiring the inheritance of descriptors via fork.

Lets look at the browser management process, and see what’s going on. Notably, we want to see how the tab’s processes communicate with it. Thus, lets look at the descriptors for the parent:

$ ls -l /proc/8029/fd/ | awk '{print $9 $10 $11}'
110 -> /home/ycombinator/.mozilla/.../https+++mail.google.com...data.sqlite
171 -> /home/ycombinator/.mozilla/.../https+++www.youtube.com....data.sqlite
123 -> /home/ycombinator/.mozilla/.../formhistory.sqlite
130 -> /home/ycombinator/.mozilla/.../bookmarks.sqlite
58 -> /home/ycombinator/.mozilla/.../cookies.sqlite
3 -> /dev/dri/renderD128
43 -> /dev/dri/card0
...
100 -> /memfd:mozilla-ipc
60 -> socket:[107464]
64 -> socket:[107467]
82 -> pipe:[110438]

This is heavily filtered. The first, highlighted section shows that the browser management process uses a database (sqlite) to access a webpage’s data (see gmail and youtube), store form histories, bookmarks, and cookies. It also uses the dri device is part of the Direct Rendering Infrastructure which enables it to communicate with the X Window Server, and display what we see on screen. Finally, we see it uses the same means of IPC as the client, and if we carefully look at the [ids] of the sockets and pipes, we can see that they match those in the child tab process! We can see that the tab has multiple IPC channels open with the parent, and can access them with file descriptors.

11.2 Goals of IPC Mechanisms

There are multiple IPC mechanisms in the system, and they all represent different trade-offs. They might be good for some things, and bad at others.

• Transparent IPC. Do applications need to be changed at all to perform IPC? If not, they are transparently leveraging IPC. Shell-driven pipes have a huge benefit that they enable transparent IPC!
• Named IPC. If we want multiple process to communicate that can’t rely on a comment parent to coordinate that communication, they need a means to find an “name” the IPC mechanism. Named IPC is in conflict with transparent IPC as we likely need to use a specific API to access the IPC mechanism.
• Channel-based IPC. Often we want to send messages between processes such that a sent message, when received is removed from the channel – once read, it won’t be read again. This enables processes to receive some request, do some processing on it, then move on to the next request.
• Multi-client communication. We often want to create a process that can provide services to multiple other “client” processes. Clients request service, and the “server” process receives these requests, and provides replies.

Lets assess pipes in this taxonomy:

• ✓ Transparent IPC. - Pipes are pervasive for a reason! They enable composition of programs via pipelines despite the programs not even knowing they are in the pipeline!
• ✗ Named IPC. - Na. The parent has to set it all up – lots of dup, close, and pipe.
• ✓ Channel-based IPC. - Data written to a pipe is removed by the reader.
• ✗ Multi-client communication. - Pipes are really there to pass a stream of data from one process to the other.

11.3 Files & Shared Memory

If the goal is to send data from one process to another, one option is found in the filesystem (FS): can’t we just use files to share? Toward this, we saw in the section on FS I/O that we can open a file, and read and write (or fread/fwrite) from it from multiple processes. This will certainly get data from one process to the other. However, it has a number of shortcomings:

1. If you want to pass a potentially infinite stream of data between processes (think pipes), then we’d end up with an infinite file. That translates to your disk running out of space very quickly, with little benefit. Take-away: files are for a finite amount of data, not for an infinite stream of data.
2. Given this, processes must assume that the file has a specific format, and they have indication of if the other process has accessed/read data in the file. An example of a specific format: maybe the file is treated as a single string of a constant length.
3. There isn’t any indication about when other processes have modified the file, thus the processes might overwrite each other’s data²⁰.

To emphasize these problems, lets try and implement a channel in a file to send data between processes. We just want to send a simple string repetitively from one process to the other.

#include <stdio.h>
#include <assert.h>
#include <string.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <fcntl.h>

int
main(void)
{
    int ret;

    ret = creat("string_data", 0777);
    assert(ret != -1);

    if (fork() == 0) {
        char *msg[] = {"penny for pawsident", "america for good doggies"};
        int fd = open("string_data", O_WRONLY);
        assert(fd != -1);

        /* send the first message! */
        ret = write(fd, msg[0], strlen(msg[0]) + 1);
        assert(ret == (int)strlen(msg[0]) + 1);
        /* send the second message! */
        ret = lseek(fd, 0, SEEK_SET);
        ret = write(fd, msg[1], strlen(msg[1]) + 1);
        assert(ret == (int)strlen(msg[1]) + 1);

        close(fd);
        exit(EXIT_SUCCESS);
    } else {
        char data[32];
        int fd = open("string_data", O_RDONLY);
        assert(fd != -1);

        memset(data, 0, 32);
        ret = read(fd, data, 32);
        assert(ret != -1);
        printf("msg1: %s\n", data);

        ret = lseek(fd, 0, SEEK_SET);
        assert(ret != -1);
        ret = read(fd, data, 32);
        assert(ret != -1);
        printf("msg2: %s\n", data);
    }
    wait(NULL);
    unlink("string_data");

    return 0;
}

Program output:

42 + 2a

You can see that there are some problems here. If we run it many times, we can see that sometimes we don’t see any messages, sometimes we only see the first, sometimes we only see the second, and other times combinations of all three options. Thus, they are not useful for channel-based communication between multiple processes.

On the other side, files have a very useful properti that we’d like to use in a good solution to IPC: they have a location in the FS that the communicating processes can both use to find the file, thus avoiding the need for a shared parent.

• ✗ Transparent IPC - We have to open the file explicitly.
• ✓ Named IPC. - Each file has a path in the filesystem.
• ✗ Channel-based IPC. - When we read out of a file, the data remains, making it hard to know what we’ve read, and what is yet to be written. A stream of messages placed into a file also will expend your disk!
• ✗ Multi-client communication. - It isn’t clear how multiple clients can convey separate requests, and can receive separate replies from a server.

An aside: you can use mmap to map a file into the address space, and if you map it MAP_SHARED (instead of MAP_PRIVATE), then the memory will be shared between processes. When one process does a store to the memory, the other process will see that store immediately! We can use this to pass data between processes, but we still have many of the problems above. How do we avoid conflicting modifications to the memory, get notifications that modifications have been made, and make sure that the data is formatted in an organized (finite) manner?

11.4 Named Pipes

The core problem with files is that they aren’t channels that remove existing data when it is “consumed” (read). But they have the significant up-side that they have a “name” in the filesystem that multiple otherwise independent processes can use to access the communication medium.

Named pipes or FIFOs are like pipes in that one process can write into the FIFO, and another process can read from it. Thus, unlike files, they have the desirable property of channels in which data read from the channel is consumed. However, like files (and unlike pipes) they have a “name” in the filesystem – FIFOs appear in the filesystem along-side files. The stat function will let you know that a file is a FIFO if the st_mode is S_IFIFO. Since these pipes appear in the filesystem, they are called named pipes. Two processes that wish to communicate need only both know where in the filesystem they agree to find the named pipe.

Lets see an example of using named pipes:

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <assert.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

void
proc1(void)
{
    int fd, ret;

    fd = open("office_hours", O_WRONLY);
    assert(fd != -1);

    ret = write(fd, "halp!", 5);
    assert(ret == 5);
    close(fd);
}

void
proc2(void)
{
    char msg[6];
    int fd, ret;

    memset(msg, 0, 6);
    fd = open("office_hours", O_RDONLY);
    assert(fd != -1);

    ret = read(fd, msg, 5);
    assert(ret == 5);
    close(fd);

    printf("What I hear at office hours is \"%s\".", msg);
    unlink("office_hours");
}

int
main(void)
{
    int ret;

    /* This is how we create a FIFO. A FIFO version of creat. */
    ret = mkfifo("office_hours", 0777);
    assert(ret == 0);

    if (fork() == 0) {
        proc1();
        exit(EXIT_SUCCESS);
    } else {
        proc2();
        wait(NULL);
    }

    return 0;
}

Program output:

42 + 2a

There is one very awkward named pipe behavior. Processes attempting to open a named pipe will block (the open will not return) until processes have opened it separately as both readable and writable. This is awkward because how would a process know when another process might want to communicate with it? If a process gets IPC requests from many others (think the browser manager), then it doesn’t want to block awaiting a new communication; it wants to service the requests of other processes.

Regardless, we do see named pipes as a cool option: named pipes enable us to use the filesystem to identify the pipe used for IPC. This enables communicating processes without shared parents to leverage IPC. This enables pipes to live up to the UNIX motto: everything is a file.

11.4.1 Challenges with Named Pipes for Multi-Client Communication

Lets check out an example that demonstrates how using named pipes for communication between a single process and multiple clients has a number of challenges.

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <assert.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*
 * Receive requests, and send them immediately as a response.
 * You can imagine that interesting computation could be done
 * to formulate a response.
 */
void
instructor(void)
{
    int req, resp, ret;
    char msg[32];

    req = open("requests", O_RDONLY);
    assert(req != -1);
    resp = open("responses", O_WRONLY);
    assert(resp != -1);

    while (1) {
        ret = read(req, msg, 32);
        if (ret == 0) break;
        assert(ret != -1);
        ret = write(resp, msg, ret);
        if (ret == 0) break;
        assert(ret != -1);
    }

    close(req);
    close(resp);
}

/*
 * Make a "request" with our pid, and get a response,
 * hopefully also our pid.
 */
void
student(void)
{
    int req, resp, ret;
    char msg[32];

    req = open("requests", O_WRONLY);
    assert(req != -1);
    resp = open("responses", O_RDONLY);
    assert(resp != -1);

    ret = snprintf(msg, 32, "%d", getpid());
    ret = write(req, msg, ret);
    assert(ret != -1);
    ret = read(resp, msg, 32);
    assert(ret != -1);

    printf("%d: %s\n", getpid(), msg);

    close(req);
    close(resp);
}

void
close_fifos(void)
{
    unlink("requests");
    unlink("responses");
}

int
main(void)
{
    int ret, i;
    pid_t pids[3];

    /* clients write to this, server reads */
    ret = mkfifo("requests", 0777);
    assert(ret == 0);
    /* server sends replies to this, clients read */
    ret = mkfifo("responses", 0777);
    assert(ret == 0);

    /* create 1 instructor that is lecturing */
    if ((pids[0] = fork()) == 0) {
        instructor();
        return 0;
    }
    /* Create 2 students "listening" to the lecture */
    for (i = 0; i < 2; i++) {
        if ((pids[i + 1] = fork()) == 0) {
            student();
            return 0;
        }
    }

    atexit(close_fifos);
    sleep(1);
    for (i = 0; i < 3; i++) kill(pids[i], SIGTERM);
    while (wait(NULL) != -1);

    return 0;
}

Program output:

42 + 2a

If executed many times, you see the expected result:

167907: 167907
167908: 167908

…but also strange results:

167941: 167940167941

If this is executed many times, we see a few properties of named pipes.

1. There isn’t really a way to predict which student sees which data – this is determined by the concurrency of the system.
2. Sometimes a student is “starved” of data completely.
3. The instructor doesn’t have any control over which student should receive which data.

Named pipes summary. These solve an important problem: how can we have multiple processes find a “pipe” to use for communication even if they don’t have a parent to coordinate that communication? They use a filesystem path/name to identify the pipe. However, they are not suitable for a single processes (a server) to communicate with multiple clients as they don’t enable the communication for each client to be separated in the server.

• ✗ Transparent IPC. - we have to use the mkfifo API explicitly.
• ✓ Named IPC. - the named pipe is represented as a file in the filesystem.
• ✓ Channel-based IPC. - read data is removed from the pipe.
• ✗ Multi-client communication. - a server cannot tell the difference between clients, nor can they send responses to specific clients.

11.5 UNIX Domain Sockets

Sockets are the mechanism provided by UNIX to communicate over the network! However, they can also be used to communicate between processes on your system through UNIX domain sockets.

A few key concepts for domain sockets:

1. They are presented in the filesystem as a file, similar to named pipes. This enables them to be accessed by multiple communicating processes that don’t necessarily share a parent to coordinate that communication.
2. Each new client that attempts to connect to a server will be represented as a separate descriptor in the server, thus enabling the server to separate its communication with on, from the other. The goal is to create a descriptor for each pair of communicating processes.
3. Each descriptor to a domain socket is bi-directional – it can be read and writen to, thus making communication back and forth quite a bit easier.

11.5.1 Domain sockets for Multi-Client Communication

Lets look at an example were we want a server to receive a client’s requests as strings, and to reply with those same strings. This isn’t useful, per-say, but demonstrates how this communication can happen. Notably, we want to enable the server to communicate with different clients!

• A server receives IPC requests from clients. Note that this is similar to how a server on the internet serves webpages to multiple clients (and, indeed, the code is similar!).
• The server’s functions include socket, bind, and listen. socket creates a domain socket file descriptor
• The server creates a separate descriptor for each client using accept.
• The client’s functions include socket and connect.

Most of these functions are complex and have tons of options. Most of them have been distilled into the following functions in 06/domain_sockets.h:

• int domain_socket_server_create(const char *file_name) - Create a descriptor to the “server” end of the IPC channel identified by file_name in the filesystem, similar to the named pipes before.
• int domain_socket_client_create(const char *file_name) - Create a descriptor to the “client” end of the IPC channel identified by a file name. One constraint is that the server must create the domain socket first, or else this call (the connect) will fail.

The server’s descriptor is not meant to be used for direct communication (i.e. should not be used for read, and write). Instead, it is used to create new descriptors, one per client! With a descriptor per-client, we have the fortunate ability to communicate explicitly with each client without the same problem of messages getting messed up in named pipes.

11.5.1.1 Setting up Domain Sockets

Two functions that both take an argument which is the domain socket name/path in the file system, and return a descriptor to the socket. For the most part, you can just use these functions in your code directly by using 06/domain_sockets.h.

#include "06/domain_sockets.h"

#include <errno.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <sys/wait.h>

void
unlink_domain_socket(int status, void *filename)
{
    unlink(filename);
    free(filename);
}

#define MAX_BUF_SZ 128

void
server(int num_clients, char *filename)
{
    char buf[MAX_BUF_SZ];
    int new_client, amnt, i, socket_desc;

    socket_desc = domain_socket_server_create(filename);
    if (socket_desc < 0) exit(EXIT_FAILURE); /* should do proper cleanup */
    on_exit(unlink_domain_socket, strdup(filename));

    /*
     * Service `num_clients` number of clients, one at a time.F
     * For many servers, this might be an infinite loop.
     */
    for (i = 0; i < num_clients; i++) {
        /*
         * We use this new descriptor to communicate with *this* client.
         * This is the key function that enables us to create per-client
         * descriptors. It only returns when a client is ready to communicate.
         */
        if ((new_client = accept(socket_desc, NULL, NULL)) == -1) exit(EXIT_FAILURE);
        printf("Server: New client connected with new file descriptor %d.\n", new_client);
        fflush(stdout);

        amnt = read(new_client, buf, MAX_BUF_SZ - 1);
        if (amnt == -1) exit(EXIT_FAILURE);
        buf[amnt] = '\0'; /* ensure null termination of the string */
        printf("Server received message (sz %d): \"%s\". Replying!\n", amnt, buf);
        fflush(stdout);

        /* send the client a reply */
        if (write(new_client, buf, amnt) < 0) exit(EXIT_FAILURE);
        /* Done with communication with this client */
        close(new_client);
    }
    close(socket_desc);

    exit(EXIT_SUCCESS);
}

void
client(char *filename)
{
    char msg[MAX_BUF_SZ];
    int  amnt = 0, socket_desc;

    socket_desc = domain_socket_client_create(filename);
    if (socket_desc < 0) exit(EXIT_FAILURE);
    printf("1. Client %d connected to server.\n", getpid());
    fflush(stdout);

    snprintf(msg, MAX_BUF_SZ - 1, "Citizen %d: Penny for Pawsident!", getpid());
    amnt = write(socket_desc, msg, strlen(msg) + 1);
    if (amnt < 0) exit(EXIT_FAILURE);
    printf("2. Client %d request sent message to server.\n", getpid());
    fflush(stdout);

    if (read(socket_desc, msg, amnt) < 0) exit(EXIT_FAILURE);
    msg[amnt] = '\0';
    printf("3. Client %d reply received from server: %s\n", getpid(), msg);
    fflush(stdout);

    close(socket_desc);

    exit(EXIT_SUCCESS);
}

int
main(void)
{
    char *channel_name = "pennys_channel";
    int nclients = 2;
    int i;

    if (fork() == 0) server(nclients, channel_name);
    /* wait for the server to create the domain socket */
    sleep(1);
    for (i = 0; i < nclients; i++) {
        if (fork() == 0) client(channel_name);
    }
    /* wait for all of the children */
    while (wait(NULL) != -1);

    return 0;
}

Program output:

42 + 2a

The server’s call to accept is the key difference of domain sockets from named pipes. It enables us to have per-client descriptors we can use to separately communicate (via read/write) to each client.

• ✗ Transparent IPC. - We have to explicitly use the socket APIs.
• ✓ Named IPC. - Uses a file name to name the domain socket.
• ✓ Channel-based IPC. - When data is read, it is removed from the socket.
• ✓ Multi-client communication. - The server separates each client’s communication into separate descriptors.

Aside: Sockets are the main way to communicate over the network (i.e. to chat with the Internet). The APIs you’d use to create network sockets are the same, it simply requires setting up the socket and the binding of the socket in a network-specific manner.

11.6 IPC Exercises

Find the programs:

• 06/domain_socket_server.c - a sample server.
• 06/domain_socket_client.c - a sample client.

Both require the 06/domain_sockets.h header file. You must run the server first to create the domain socket “file”. If you run the server, and it complains that “server domain socket creation”, then you might need to rm the domain socket file on the command line first. It already exists from a previous run of the server, so the server cannot create it again!

Your job is to try to figure what in the hey these do! Answer the following questions:

• Describe what the server does to handle each client request.
• How many clients can the server handle at a time?
• Is the number of clients limited by the server waiting on client/child processes? Recall when talking about background tasks in a shell that the wait is key to the system’s concurrency/behavior.
• Describe what the client does. What data does it send to the server?
• Describe the behavior when you use the client with the server.

The programs can be compiled directly with gcc (e.g. gcc domain_socket_server.c -o server; gcc domain_socket_client.c -o client). Use these programs on the command line to send data from the client to server.

• How can you use the client and server programs to send data between them?
• How can you make multiple clients connect to the server?

12 Reinforcing Ideas: Assorted Exercises and Event Notification

In the following, if you aren’t positive of the answer, please run the program! Note that we’re omitting error checking in these programs to keep them terse. Remember that in your programs, you must check and react to all errors. This will require you to use the man pages to look up the required #includes. If the output doesn’t match a high-level intuition, how would you modify the program to match the intuition? What are all of the potential output of the following programs? Why?

12.0.1 fork and Stream Behavior

fork();
fork();
printf("z");

printf("z");
fork();
fork();

printf("z");
fork();
write(STDOUT_FILENO, ".", 1);

12.0.2 Wait Behavior

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/wait.h>

int
main(void)
{
    pid_t child;
    int i;
    int wait_param = 0;     /* or WNOHANG */
    int output_w_write = 0; /* or 1 */

    for (i = 0; i < 2; i++) {
        child = fork();
        if (child == 0) {
            sleep(1);
            if (output_w_write) write(STDOUT_FILENO, ".\n", 2);
            else                printf(".\n");
            exit(EXIT_SUCCESS);
        }
        waitpid(child, NULL, wait_param);
        write(STDOUT_FILENO, "Post-fork\n", 10);
    }
    /* ...are we done here? */

    return 0;
}

• What if change wait_param to equal WNOHANG?
• What if we change output_w_write to 1?
• Why do we need to reap children? Why would this actually matter?

12.0.3 read Behavior

#include <stdio.h>
#include <unistd.h>
#include <sys/wait.h>
#include <string.h>

int
main(void)
{
    pid_t child;
    int fds[2];
    char *msg = "What type of doggo is Penny?\n";

    pipe(fds);

    if ((child = fork()) == 0) {
        /* recall: `cat` reads its stdin, and outputs it to stdout */
        char *args[] = {"cat", NULL};

        close(STDIN_FILENO);
        dup2(fds[0], STDIN_FILENO);
        execvp(args[0], args);
    }
    write(fds[1], msg, strlen(msg));
    printf("100%% good girl.");
    wait(NULL);

    return 0;
}

• There are multiple bugs here. Find them and squish ’em.

12.0.4 Signals

What is the following doing?

#include <execinfo.h>
#include <stdlib.h>
#include <stdio.h>
#include <signal.h> /* sigaction and SIG* */

void
bt(void)
{
    void *bt[128];
    char **symbs;
    int nfns, i;

    nfns = backtrace(bt, 128);
    symbs = backtrace_symbols(bt, nfns);
    for (i = 0; i < nfns; i++) {
        printf("%s\n", symbs[i]);
    }
    free(symbs);
}

void
bar(int *val)
{
    *val = 42;
    bt();
}

void
foo(int *val)
{
    bar(val);
}

void
sig_fault_handler(int signal_number, siginfo_t *info, void *context)
{
    printf("Fault triggered at address %p.\n", info->si_addr);
    bt();
    exit(EXIT_FAILURE);

    return;
}

int
main(void)
{
    sigset_t masked;
    struct sigaction siginfo;
    int ret;
    int val;

    sigemptyset(&masked);
    sigaddset(&masked, SIGSEGV);
    siginfo = (struct sigaction) {
        .sa_sigaction = sig_fault_handler,
        .sa_mask      = masked,
        .sa_flags     = SA_RESTART | SA_SIGINFO /* we'll see this later */
    };

    if (sigaction(SIGSEGV, &siginfo, NULL) == -1) {
        perror("sigaction error");
        exit(EXIT_FAILURE);
    }

    foo(&val);
    printf("---\nMoving on...\n---\n");
    fflush(stdout);
    foo(NULL);

    return 0;
}

• What are the various aspects of the output? What do the numbers mean? What do the strings mean?
• Explain what is happening before the “Moving on…”.
• Explain what is happening after the “Moving on…”.
• There is a lot of information there (lines being output) that aren’t useful to the programmer. Make the output more useful by printing only the parts of the output that a programmer would find useful.

12.0.5 Communication with Multiple Clients

Communicating with multiple clients is hard. Domain sockets are complicated, but there are challenges around blocking on reads. What if one client is very “slow”, and we block waiting for them?

#include "06/domain_sockets.h"

#include <errno.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <sys/wait.h>
#include <assert.h>

void
panic(char *msg)
{
    perror(msg);
    exit(EXIT_FAILURE);
}

void
client(char *filename, int slowdown)
{
    int i, socket_desc;
    char b;

    socket_desc = domain_socket_client_create(filename);
    if (socket_desc < 0) {
        perror("domain socket client create");
        exit(EXIT_FAILURE);
    }

    /* delay after creating connection, but before communicating */
    sleep(slowdown);
    if (write(socket_desc, ".", 1) == -1) panic("client write");
    if (read(socket_desc, &b, 1) == -1)   panic("client read");
    printf("c: %c\n", b);

    close(socket_desc);
    exit(EXIT_SUCCESS);
}

void
client_slow(char *filename)
{
    client(filename, 3);
}

void
client_fast(char *filename)
{
    client(filename, 1);
}

int
main(void)
{
    char *ds = "domain_socket";
    int socket_desc, i;

    socket_desc = domain_socket_server_create(ds);
    if (socket_desc < 0) {
        /* remove the previous domain socket file if it exists */
        unlink(ds);
        socket_desc = domain_socket_server_create(ds);
        if (socket_desc < 0) panic("server domain socket creation");
    }

    /* TODO: change this order. What changes? */
    if (fork() == 0) client_slow(ds);
    if (fork() == 0) client_fast(ds);

    /* handle two clients, one after the other */
    for (i = 0; i < 2; i++) {
        int ret, new_client, i;
        char b;

        new_client = accept(socket_desc, NULL, NULL);
        if (new_client == -1) panic("server accept");

        /* read from, then write to the client! */
        if (read(new_client, &b, 1) == -1)   panic("server read");
        if (write(new_client, "*", 1) == -1) panic("server write");
        close(new_client);
    }

    close(socket_desc);
    /* reap all children */
    while (wait(NULL) != -1) ;

    return 0;
}

• See the TODO in the main. When you change the order, what do you see?
• Is there generally a problem when we use both fast and slow clients?

12.1 Revisiting IPC with Multiple Clients

12.1.1 System Services

The systemctl command enables you to understand many of the services on the system (of which there are many: systemctl --list-units | wc -l yields 244 on my system).

$ systemctl --list-units
...
  cups.service           loaded active     running   CUPS Scheduler
...
  gdm.service            loaded active     running   GNOME Display Manager
...
  ssh.service            loaded active     running   OpenBSD Secure Shell server
...
  NetworkManager.service loaded active     running   Network Manager
...
  openvpn.service        loaded active     exited    OpenVPN service
...

I’ve pulled out a few selections that are easier to relate to:

• CUPS which is the service used to receive print jobs.
• GDM which manages your graphical logins on Ubuntu.
• OpenSSH which manages your remote logins through ssh.
• NetworkManager that you interact with when you select which wifi hotspot to use.
• OpenVPN which handles your VPN logins to the school network. You can see that the service is currently “exited” as I’m not running VPN now.

Each of these services communicates with many clients using domain sockets.

12.1.2 Understanding Descriptor Events with poll

Each of these services is a process that any client send requests to. We’ve seen that domain sockets can help us to talk to many different clients as each is represented with a separate file descriptor. How does the service process know which of the file descriptors has information available on it? Imagine the following case:

1. a client, A, connects to a service.
2. a client, B, connects to the same service.
3. B immediately sends a request, while A goes into a sleep(100) (or, more realistically, simply does some expensive computation).

If the server issues a read on the descriptor for A, it will block for 100 seconds! Worse, it won’t service B despite it making a request immediately. Why are we waiting for a slow client when there is a fast client with data already available? Yikes.

We’d really like a facility that can tell us which descriptor’s have data and are ready to be read from, and which are ready to be written to! Luckily, UNIX comes to the rescue with its event notification APIs. These are APIs that let us understand when a file descriptor has an event (i.e. a client writes to it) and is now readable or writable. These include three functions: poll, select, and (the more modern) epoll. Lets look at how to use poll!

12.1.3 Event Loops and poll

Lets look at some pseudocode for using poll.

fdinfo[NUM_FDS] = {
    # initialized to all fds of interest, listening for read and write events
}
while True:
    poll(fdinfo, NUM_FDS, -1) # -1 = no timeout
    for fdi in fdinfo:
        # check and respond to each of the possible events
        if fdi.revents & (POLLHUP | POLLERR):
            # process closed fds
        if fdi.revents & POLLIN:
            # read off of, or accept on the file desciptor
        if fdi.revents & POLLOUT:
            # write to the file desciptor

This yields what is called an event loop – we loop, each time processing a single event. You see event loops in most GUIs (where each event is a key/mouse press), and in web programming where javascript callbacks are executed by the browser’s event loop. Importantly, we only process descriptors that have events, thus can avoid blocking on descriptors that don’t have available data! This solves our previous problem: a server won’t block awaiting communication with a client that is delayed, or never writes to a channel, instead only reading/writeing to descriptors that are ready.

12.1.4 poll API

• int poll(struct pollfd *fds, nfds_t nfds, int timeout) - We pass in an array of struct pollfds of length nfds, with each entry corresponding to a single file descriptor we want to get information about. The timeout is in milliseconds, and enables poll to return after that amount of time returns even if none of the file descriptors has an event. A negative timeout is interpreted as “infinite”, while 0 means that poll will return immediately with any current events.

Lets check out the struct pollfd:

struct pollfd {
    int   fd;         /* file descriptor */
    short events;     /* requested events */
    short revents;    /* returned events */
};

When we make the poll call, we populate the fd and events fields with the file descriptors we’re interested in, and which events we’re interested in retrieving. events is a “bitmap” which means each bit in the value denotes a different type of event, and we bitwise or event types together when writing them into events. These event types include:

• POLLIN - is there data available to be read? If the file descriptor is the domain socket that we use for accept, then this POLLIN means that a new client request is ready to accept.
• POLLOUT - is there data available to be writen?

We can almost always set events = POLLIN | POLLOUT as we wait to wait for both.

When poll returns, we determine which events happened by looking at the contents of the revents field. In addition to POLLIN and POLLOUT which tell us if data is ready to be read or written, we have the following:

• POLLHUP - The other side of the pipe closed its descriptor! Subsequent reads to the descriptor will return 0. We can likely close our descriptor.
• POLLERR - Again, there was some sort of a problem with the descriptor, and we should likely close it, terminating communication.

12.1.5 Example poll Code

Lets put this all together. For simplicity, in this example, we’ll assume that we can always write to a descriptor without blocking. This isn’t generally true if you’re writing large amounts of data, and in “real code” you’d also want to handle POLLOUT appropriately.

#include "06/domain_sockets.h"

#include <errno.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <sys/wait.h>
#include <assert.h>
#include <poll.h>

void
panic(char *msg)
{
    perror(msg);
    exit(EXIT_FAILURE);
}

void
client(char *filename)
{
    int i, socket_desc;

    sleep(1); /* await the domain socket creation by the server */
    socket_desc = domain_socket_client_create(filename);
    if (socket_desc < 0) {
        perror("domain socket client create");
        exit(EXIT_FAILURE);
    }

    for (i = 0; i < 5; i++) {
        char b;
        if (write(socket_desc, ".", 1) == -1) panic("client write");
        if (read(socket_desc, &b, 1) == -1)   panic("client read");
        printf("c %d: %c\n", getpid(), b);
    }

    close(socket_desc);
    exit(EXIT_SUCCESS);
}

/* we can track max 16 fds */
#define MAX_FDS 16

void
server(char *filename)
{
    int socket_desc, num_fds = 0;
    struct pollfd poll_fds[MAX_FDS];

    /*** Initialize the domain socket and its pollfd ***/
    socket_desc = domain_socket_server_create(filename);
    if (socket_desc < 0) {
        unlink(filename); /* remove the previous domain socket file if it exists */
        socket_desc = domain_socket_server_create(filename);
        if (socket_desc < 0) panic("server domain socket creation");
    }

    /* Initialize all pollfd structs to 0 */
    memset(poll_fds, 0, sizeof(struct pollfd) * MAX_FDS);
    poll_fds[0] = (struct pollfd) {
        .fd     = socket_desc,
        .events = POLLIN,
    };
    num_fds++;

    /*** The event loop ***/
    while (1) {
        int ret, new_client, i;

        /*** Poll; if we don't get a client for a second, exit ***/
        ret = poll(poll_fds, num_fds, 1000);
        if (ret == -1) panic("poll error");
        /*
         * If we timeout, break out of the loop.
         * This isn't what you'd normally do as servers stick around!
         */
        if (ret == 0) break;

        /*** Accept file descriptor has a new client connecting! ***/
        if (poll_fds[0].revents & POLLIN) {
            if ((new_client = accept(socket_desc, NULL, NULL)) == -1) panic("server accept");
            /* add a new file descriptor! */
            poll_fds[num_fds] = (struct pollfd) {
                .fd = new_client,
                .events = POLLIN
            };
            num_fds++;
            poll_fds[0].revents = 0;
            printf("server: created client connection %d\n", new_client);
        }
        /*** Communicate with clients! ***/
        for (i = 1; i < num_fds; i++) {
            if (poll_fds[i].revents & (POLLHUP | POLLERR)) {
                printf("server: closing client connection %d\n", poll_fds[i].fd);
                poll_fds[i].revents = 0;
                close(poll_fds[i].fd);
                /* replace the fd to fill the gap */
                poll_fds[i] = poll_fds[num_fds - 1];
                num_fds--;
                /* make sure to check the fd we used to fill the gap */
                i--;

                continue;
            }
            if (poll_fds[i].revents & POLLIN) {
                char b;

                poll_fds[i].revents = 0;
                /* our server is simply sending a '*' for each input character */
                if (read(poll_fds[i].fd, &b, 1) == -1) panic("server read");
                if (write(poll_fds[i].fd, "*", 1) == -1) panic("server write");
            }
        }
    }
    close(socket_desc);

    exit(EXIT_SUCCESS);
}

int
main(void)
{
    char *ds = "domain_socket";

    if (fork() == 0) client(ds);
    if (fork() == 0) client(ds);
    server(ds);

    return 0;
}

13 Libraries

Up till now, we’ve been focusing on UNIX programming APIs, and how to interact with various aspects of the system. Part of this discussion has been how various programs can be orchestrated to cooperate. Pipelines enable larger functionalities to be composed out of multiple programs in which the output of one goes to the input of the next. System servers provide services to clients that use IPC to harness their functionality. This enables servers that control system resources (like wifi) to let many clients have limited and controlled access to those resources.

The next few chapters dive under the hood of processes. We’ll investigate

1. how programs are represented and how to understand their format,
2. how programs are organized and can share code, and
3. how they interact with the system.

How do we think about our system’s programs? We’re used to writing our own programs, and like to think of the code we write being relatively self-contained. In contrast, quite a bit of system programming is about providing functionality that can be used by any program on the system. What if you wanted to implement the world’s best key-value store, and wanted to enable anyone to use your implementation!?

We think of these shared functionalities as falling into one of two categories:

1. Libraries that represent bodies of code that can be pull into our applications. You’re already quite used to these! libc is the core library that provides all of the functions that we use when we include many of the headers in the class, for example, #include <stdlib.h>, #include <stdio.h>, or #include <string.h>.
2. Services which track different aspects of the system, and provide information to our programs. They tend to be programs that are always running on your system, and your programs can communicate with them to harness their functionality. These include services that receive print requests, that service ssh sessions, that provide your vpn connection, and that display your graphical user interface, and many other functions. We’ve already seen many of the IPC and communication foundations that make these tick.

This chapter, we’ll focus on libraries as a foundation for sharing software in systems.

13.1 Libraries - Goals and Overview

Before we start, consider:

• When you #include a ton of files, do you think that all of the related functionality is compiled into your program? That would mean that every program has the functionality compiled into it.
• If so, how much memory and disk space do you think that will take?
• Do you think there are other options? What might they be?

Visually, this is what it would look like to compile with “libraries” that are defined as normal *.o files.

The goals of libraries are to:

• Enable programs to leverage shared implementations of specific functionality (e.g. the contents of string.h).
• Provide a uniform set of programming APIs to programs.
• Attempt to save memory by making all programs in the system as small as possible by not taking memory for all of the library code in each program.
• Enable the system to upgrade both libraries and programs (for example, if a security compromise is found).

Libraries are collections of functionality that can be used by many different programs. They are code that expand the functionality of programs. This differs from services which are separate programs. Libraries have two core components that mirror what we understand about C:

1. header files that share the types of the library API functions and data, and
2. the code to implement those APIs.

Lets set up some visual nomenclature.

There are two main ways library code is integrated into programs:

• Static libraries. These are added into your program when you compile it.
• Shared or dynamic libraries. These are integrated into your program, dynamically, when you run the program.

We’ll discuss each of these in subsequent sections

13.2 Libraries - Header Files

We can see most of the header files in /usr/include, within which you’ll find some familiar files:

$ ls /usr/include/std*
/usr/include/stdc-predef.h  /usr/include/stdint.h  /usr/include/stdio_ext.h  /usr/include/stdio.h  /usr/include/stdlib.h

Yes, this means that you have the source code for much of the standard library at your fingertips! But how does gcc know to find all of these files when we use #include <...>? The following gcc incantation gives us a good idea:

$ gcc -xc -E -v -
...
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/9/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

These are all of the “include paths” that are searched, by default, when we do an #include <>.

Aside: In Makefiles, we often want to add to tell the compiler to look for header files in our own files Thus, we update the include paths by adding multiple -I<dir> options to the gcc command line. We can see how this works:

$ gcc -I. -xc -E -v -
...
#include <...> search starts here:
.
/usr/lib/gcc/x86_64-linux-gnu/9/include
...

The include paths are searched in order, so you can see that when we explicitly tell gcc to include a directory -I. here, we’re asking to to check in the current directory (.) first.

13.3 Linking Undefined Symbols to References Across Objects

Before we proceed to figure out how multiple *.o objects are combined into an executable, lets learn what it means for one object to depend on another for some functionality. To understand this, we have to understand how object files (*.o) and binaries think about symbols and linking them together.

When your program uses a library’s API, the library’s code is linked into your program. Linking is the act of taking any symbols that are referenced, but not defined in your object (*.o) files, and the definition of those symbols in the objects that provide them. A symbol is a functions or global variable in your program. Each symbol has a representation in your object files so that they can be referenced across objects. We have to understanding the linking operation to dive into how libraries can be added into, thus accessible from your program. Lets peek into how our objects think about the world using a number of programs that can introspect on the objects including nm, objdump, and readelf.

As an example, lets look at your ptrie implementation. We know that each of the tests (in tests/0?-*.c) depends on your ptrie implementation in ptrie.c. What does that look like?

$ nm tests/01-add.o
                 U _GLOBAL_OFFSET_TABLE_
000000000000022a T main
                 U printf
                 U ptrie_add
                 U ptrie_allocate
                 U ptrie_free
                 U __stack_chk_fail
0000000000000000 t sunit_execute
0000000000000000 d test
0000000000000184 T test_add
0000000000000144 T test_alloc

The left column is the address of the symbol, and the character in the middle column gives us some information about the symbol:

• T - this symbol is part of the code of the *.o file, and is visible outside of the object (i.e. it isn’t static). Capital letters mean that the symbol is visible outside the object which simply means it can be linked with another object.
• t - this symbol is part of the code, but is not visible outside of the object (it is static).
• D - a global variable that is visible.
• d - a global variable that is not visible.
• U - an undefined symbol that must be provided by another object file.

For other symbol types, see man nm.

We can see what we’d expect: the test defines its own functions for (e.g. main, test_add), but requires the ptrie_* functions and printf to be provided by another object file. Now lets check out the ptrie objects.

$nm ptrie.o
...
                 U calloc
                 U free
...
                 U printf
00000000000005ae T ptrie_add
00000000000003fc t ptrie_add_internal
0000000000000016 T ptrie_allocate
0000000000000606 T ptrie_autocomplete
00000000000000f8 T ptrie_free
...
                 U putchar
                 U puts
                 U strcmp
                 U strdup
                 U strlen

We can see that the ptrie.o object depends on other objects for all of the functions we’re using from stdio.h, stdlib.h, and string.h, provides all of the functions that are part of the public API (e.g. ptrie_add), and some other symbols (e.g. ptrie_add_internal) that cannot be linked to other object files.

After we link the ptrie into the test (with gcc ... tests/01_add.o ptrie.o -o tests/01_add.test),

$ nm tests/01_add.test | grep alloc
                 U calloc@@GLIBC_2.2.5
0000000000001566 T ptrie_allocate
00000000000013ad T test_alloc

Now we can see that there are no longer any undefined references (U) to ptrie_allocate, so the test object has now found that symbol within the ptrie.o object. Additionally, we can see that some symbols (e.g. calloc here) are still undefined and have some mysterious @@... information associated with them. You might guess that this somehow tells us that the function should be provided by the standard C library (glibc on Ubuntu).

Lets see another example visually using the libexample/* files:

Where are we? OK, lets summarize so far. Objects can have undefined symbol references that get linked to the symbols when combined with the objects in which they are defined. How is this linking implemented?

Question: What does a file format mean? We talk about the format of webpages being html, images, being png, documents being docx or pdf. What do these formats mean? What is the format of object and executable binaries?

13.3.1 ELF Object Format

First, we have to understand something that is a little amazing: all of our objects and binaries have a defined file format called the Executable and Linkable Format (ELF)²¹.

We can confirm that these files are ELF with

$ xxd ptrie.o | head -n 1
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............

xxd dumps out a file in hexadecimal format, and head just lets us filter out only the first line (i.e. at the head). The first characters in most formats often denote the type of file, in this case an ELF file.

This reveals a cool truth: programs are data, plain and simple. They happen to be data of a very specific format, ELF. nm, readelf, and objdump are all programs that simply understand how to dive into the ELF format, parse, and display the information.

As programs are simply data encapsulated in the ELF format, we can start to understand what it means to link two objects together.

An in-depth explanation follows. Now we get to an interesting part of the ELF objects:

$ readelf -r tests/01_add.o | grep ptrie
000000000151  001500000004 R_X86_64_PLT32    0000000000000000 ptrie_allocate - 4
000000000179  001600000004 R_X86_64_PLT32    0000000000000000 ptrie_free - 4
000000000191  001500000004 R_X86_64_PLT32    0000000000000000 ptrie_allocate - 4
0000000001dd  001800000004 R_X86_64_PLT32    0000000000000000 ptrie_add - 4
00000000021f  001600000004 R_X86_64_PLT32    0000000000000000 ptrie_free - 4

The -r flag here outputs the “relocation” entries of the elf object.

This output says that there are number of references to the ptrie_* functions in the code, and enumerates each of them. You can imagine that these are simply all of the places in the code that these functions are called. The first column gives us the offset into the ELF object where the reference to the function is made – in this case where it is called. For example, at the offset 0x151²² into the ELF object, we have a call to ptrie_allocate. This is important because the tests/01_add.o does not know where the ptrie_allocate function is yet. It will only know that when it links with the ptrie implementation!

Lets check out what this looks like in the ELF object’s code. For this, we’ll use objdump’s ability to dump out the “assembly code” for the object file. It also will try and print out the C code that corresponds to the assembly, which is pretty cool.

$ objdump -S tests/01_add.o
...
sunit_ret_t
test_alloc(void)
{
 144:   f3 0f 1e fa             endbr64
 148:   55                      push   %rbp
 149:   48 89 e5                mov    %rsp,%rbp
 14c:   48 83 ec 10             sub    $0x10,%rsp
    struct ptrie *pt;

    pt = ptrie_allocate();
 150:   e8 00 00 00 00          callq  155 <test_alloc+0x11>
 155:   48 89 45 f0             mov    %rax,-0x10(%rbp)
...

I’ve printed out the addresses in the object around offset 0x151 because the first reference to ptrie_allocate is made there (see readelf output above). We can see an instruction to call test_alloc. However, you can see that the address that the binary has for that function is 00 00 00 00, or NULL! This is what it means for a function to be undefined (recall: the nm output for the symbol is U).

Now we can understand what a linker’s job is:

What does a Linker do? A linker takes all of the undefined references to symbols from the relocation records, and rewrites the binary to populate those references with the address of that symbol (here updating 00 00 00 00 to the actual address of test_alloc).

In our Makefiles, we always use gcc both for compiling *.c files into objects, but also linking objects together into a binary.

13.3.2 Linking Summary

Lets back up and collect our thoughts.

• Our programs are collections of a number of ELF object files.
• We’ve seen that some symbols in an object file are undefined if they are functions or variables that are not defined in a .c file.
• Though their types are defined in header files, their implementation must be found somewhere.
• Each reference to an undefined symbol has a relation entry that enables the linker to update the reference once it knows the ELF object that provides it.
• When another ELF exports the symbol that is undefined elsewhere, they can be linked to turn the function calls of undefined functions into function calls, and references to global variables be actual pointers to memory.

This is the foundation for creating large programs out of small pieces, and to enable some of those pieces to be shared between programs.

We’ll also see that libraries enable an opportunity to save memory. For example,

• static libraries enable only the object files that are needed by a program to be compiled into it, and
• dynamic libraries enable the code for a library to be shared between all processes that execute it in the system.

13.4 Static Libraries

A static library is simply a collection of ELF object (*.o) files created with gcc -c. They are collected together into a static library file that has the name lib*.a. You can think of these files as a .zip or .tar file containing the ELF objects.

Static libraries of the form lib*.a and are created with the ar (archive) program from a collection of objects. An example from the ptrie library (expanded from its Makefile) that creates libptrie.a:

$ ar -crs libptrie.a *.o

So a static library is just a collection of ELF objects created by our compilation process. If we are writing a program that wants to use a library, first you make sure to include its header file(s). Then, when compiling, we have to tell the compiler which library we want to link with by using a few compiler flags:

$ gcc -o tests/01_add.test 01_add.o -L. -lptrie

The last two flags are the relevant ones:

• -L. says “look for static libraries in the current directory (i.e. .)” – other directories can be included, and a few default directories are used, and
• -lptrie says “please link me with the ptrie library” which should be found in one of the given directories and is found in a file called libptrie.a.

Note that the linker already searches a few directories for libraries (i.e. default -L paths):

$ gcc -print-search-dirs | grep libraries | sed 's/libraries: =//' | tr -s ":" '\n'
/usr/lib/gcc/x86_64-linux-gnu/9/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/9/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/../lib/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/9/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib/
/lib/x86_64-linux-gnu/9/
/lib/x86_64-linux-gnu/
/lib/../lib/
/usr/lib/x86_64-linux-gnu/9/
/usr/lib/x86_64-linux-gnu/
/usr/lib/../lib/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/lib/
/usr/lib/gcc/x86_64-linux-gnu/9/../../../
/lib/
/usr/lib/

As many of these paths are in directories that any user can access, this is how the functionality of these libraries can be accessed by any program wishing to use them. As we compile our ptrie library as a static library, you’ve already seen one example of these in use.

13.4.1 Saving Memory with Static Libraries

Static libraries do provide some facilities for trying to shrink the amount of memory required for library code. If this was all done naively, then all of the object files in a static library could get loaded into a program with which it is linked. This means that for $N$ programs that link with a library whose objects take $X$ bytes, we’ll have $N \times X$ bytes devoted to storing programs on disk, and running the programs in memory (if they all run at the same time). Some static libraries can be quite large.

Instead, static libraries are smarter. If a static library contains multiple .o files, only those object files that define symbols that are undefined in the program being linked with, are compiled into the program. This means that when designing a static library, you often want to break it into multiple .o files along the lines of different functionalities that separately used by different programs²³.

Some projects take this quite far. For example musl libc is a libc replacement, and it separates almost every single function into a separate object file (i.e. in a separate .c file!) so that only the exact functions that are called are linked into the program.

13.5 Shared/Dynamic Libraries

Shared or dynamic libraries (for brevity, I’ll call them only “dynamic libraries”, but both terms are commonly used) are linked into a program at runtime when the program starts executing as part of exec.

Recall that even executable binaries might still have undefined references to symbols. For example, see calloc in the example below:

$ nm tests/01_add.test | grep calloc
                 U calloc@@GLIBC_2.2.5

Though we can execute the program tests/01_add.test, it has references to functions that don’t exist in the program! How can we possibly execute a program that has undefined functions; won’t the calls to calloc here be NULL pointer dereferences?

To understand how dynamic linking works, lets look at the output of a program that tells us about dynamic library dependencies, ldd.

$ ldd tests/01_add.test
        linux-vdso.so.1 (0x00007ffff3734000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb502adf000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb502cfa000)

ldd also simply parses through the ELF program, and determines which dynamic libraries are required by the program. For now, we’ll ignore the linux-vdso, but the other two entries are interesting. We can see that the C standard library, libc is being provided by /lib/x86_64-linux-gnu/libc.so.6. These are both dynamic libraries, as are most *.so.?* and *.so files. If we check out that object we see that it provides calloc.

$ objdump -T /lib/x86_64-linux-gnu/libc.so.6  | grep calloc
000000000009ec90 g    DF .text  0000000000000395  GLIBC_2.2.5 __libc_calloc
000000000009ec90  w   DF .text  0000000000000395  GLIBC_2.2.5 calloc

So this library provides the symbols we require (i.e. the calloc function)!

13.5.1 Compiling and Executing with Dynamic Libraries

If we want our program to use a dynamic library, we have to compile it quite similarly to when we wanted to use static libraries:

$ gcc -o dyn_prog *.o -L. -lptrie

So everything is somewhat normal here; we’re saying “look for the library in this directory”, and compile me with the ptrie library. To create the dynamic library:

$ gcc -Wall -Wextra -fpic -I. -c -o ptrie.o ptrie.c
$ gcc -shared -o libptrie.so ptrie.o

The first line is the normal way to create an object file from C, but includes a new flag, -fpic. This tells the compiler to generate “Position Independent Code”, or PIC, which is code that can, seemingly magically, be executed when the code is loaded into any address! The dynamic library cannot assume which addresses it will be loaded into as many dynamic libraries might be loaded into an executable, thus the PIC requirement.

The second line creates the dynamic (shared) library. By convention, all dynamic libraries end with .so which you can think of as “shared object”. This is the line that is dynamic-library equivalent to the ar command for static libraries.

Now we have a binary that has been told were the library is; lets execute it!

$ ./dyn_prog
./dyn_prog: error while loading shared libraries: libptrie.so: cannot open shared object file: No such file or directory

Whaaaaaa? If we dive in a little bit, we can see:

$ nm ./dyn_prog | grep ptrie_alloc
                 U ptrie_alloc

But I thought that we wanted all symbols to be defined when we create a binary? Why is the library’s ptrie_alloc not linked into the program? We can see that not all symbols are defined in a program when we are using dynamic libraries as they are linked when we try and run the program!

We now can see the main practical difference between static and dynamic libraries:

• Static libraries are compiled into the program when you compile with the -lptrie directive.
• Dynamic libraries not not compiled into the program, and are instead linked into the program when it is executed (i.e. at the time of exec).

We can start to see why the program wont execute when we look at its dynamic library dependencies:

$ ldd ./dyn_prog
        linux-vdso.so.1 (0x00007ffdf3ca0000)
        libptrie.so => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff9921e8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff9923f8000)

It isn’t finding libptrie.so!

To execute the program, we have to properly set an environment variable that will tell the program, when it is executed, where to look for the dynamic library:

$ LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ./dyn_prog
...success...

This is some strange shell syntax. LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH essentially says “The environment variable called LD_LIBRARY_PATH should be updated to prepend the diretory ./ onto the front of it. This is conceptually similar to something like lib_path = "./:" + lib_path in a language that supports string concatenation. When ./dyn_prog is executed, the updated LD_LIBRARY_PATH environment variable is visible in that program using the normal getenv. So there is some part of the initialization of ./dyn_prog that looks at this environment variable to figure out where to look for the dynamic library.

To confirm why we can now execute the program, we can again look at the library dependencies:

$ LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ldd prog_dynamic
        linux-vdso.so.1 (0x00007ffc939f8000)
        libptrie.so => ./libptrie.so (0x00007f70d316f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f70d2f66000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f70d3236000)

If we expand the LD_LIBRARY_PATH to include the current directory, the program can find the library.

If we want to see the default paths that are searched for dynamic libraries:

$ ldconfig -v 2>/dev/null | grep -v ^$'\t'
/usr/local/lib:
/lib/x86_64-linux-gnu:
/lib32:
/libx32:
/lib:

In fact, ldconfig -v will also print out all of the potential default dynamic libraries we can link with.

13.5.2 exec with Dynamic Linking

How does the libc.so library get linked into our program? Diving in a little bit further, we see that ld is our program’s “interpreter”:

$ readelf --program-headers tests/01_add.test
Elf file type is DYN (Shared object file)
Entry point 0x1180
There are 13 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x00000000000002d8 0x00000000000002d8  R      0x8
  INTERP         0x0000000000000318 0x0000000000000318 0x0000000000000318
                 0x000000000000001c 0x000000000000001c  R      0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000000008e8 0x00000000000008e8  R      0x1000
  LOAD           0x0000000000001000 0x0000000000001000 0x0000000000001000
                 0x0000000000000e55 0x0000000000000e55  R E    0x1000
...

A lot is going on here, but we see the reference to ld-linux-x86-64.so.2 that we saw previously in the ldd output. We can now see that the library (that I’ll call simply ld) is a “program interpreter” (see “Requesting program interpreter: /lib64/ld-linux-x86-64.so.2”) for our program²⁴. This is the core of understanding how our program can execute while having undefined references to functions provided by another library.

When we call exec, we believe that the program we execute takes over the current process and starts executing. This is not true! Instead, if a program interpreter is defined for our program (as we see: it is), the interpreter program’s memory is loaded along side our program, but then the interpreter is executed instead of our program! It is given access to our program’s ELF object, and ld’s job is to finish linking and loading our program. We have to add ./ to the LD_LIBRARY_PATH to make the program execute because ld reads that environment variable when it is trying to link and load in your program!

13.5.3 Diving into Dynamic Linking at exec

But wait. Why does exec end up loading ld, which then loads our program? Why load ld to link our program instead of just running our program? Because ld also loads and links all of the dynamic libraries (.sos) that our program depends on!!!

We can confirm that ld is loaded into our program by executing it in gdb, blocking it on breakpoint, and outputting its memory maps (cat /proc/262648/maps on my system):

555555554000-555555555000 r--p 00000000 08:02 527420                     /home/gparmer/repos/gwu-cs-sysprog/22/hw_solns/02/tests/01_add.test
555555555000-555555556000 r-xp 00001000 08:02 527420                     /home/gparmer/repos/gwu-cs-sysprog/22/hw_solns/02/tests/01_add.test
555555556000-555555557000 r--p 00002000 08:02 527420                     /home/gparmer/repos/gwu-cs-sysprog/22/hw_solns/02/tests/01_add.test
555555557000-555555558000 r--p 00002000 08:02 527420                     /home/gparmer/repos/gwu-cs-sysprog/22/hw_solns/02/tests/01_add.test
555555558000-555555559000 rw-p 00003000 08:02 527420                     /home/gparmer/repos/gwu-cs-sysprog/22/hw_solns/02/tests/01_add.test
7ffff7dbe000-7ffff7de3000 r--p 00000000 08:02 2235639                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7ffff7de3000-7ffff7f5b000 r-xp 00025000 08:02 2235639                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7ffff7f5b000-7ffff7fa5000 r--p 0019d000 08:02 2235639                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7ffff7fa5000-7ffff7fa6000 ---p 001e7000 08:02 2235639                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7ffff7fa6000-7ffff7fa9000 r--p 001e7000 08:02 2235639                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7ffff7fa9000-7ffff7fac000 rw-p 001ea000 08:02 2235639                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7ffff7fac000-7ffff7fb2000 rw-p 00000000 00:00 0
7ffff7fc9000-7ffff7fcd000 r--p 00000000 00:00 0                          [vvar]
7ffff7fcd000-7ffff7fcf000 r-xp 00000000 00:00 0                          [vdso]
7ffff7fcf000-7ffff7fd0000 r--p 00000000 08:02 2235423                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7ffff7fd0000-7ffff7ff3000 r-xp 00001000 08:02 2235423                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7ffff7ff3000-7ffff7ffb000 r--p 00024000 08:02 2235423                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7ffff7ffc000-7ffff7ffd000 r--p 0002c000 08:02 2235423                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7ffff7ffd000-7ffff7ffe000 rw-p 0002d000 08:02 2235423                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

Recall that this dumps all of the memory segments of the process.

There are two important observations:

• /usr/lib/x86_64-linux-gnu/libc-2.31.so is present in the process so somehow got linked and loaded into the process.
• /usr/lib/x86_64-linux-gnu/ld-2.31.so is also present in the process; indeed it was the first code to execute in the process!

13.5.4 Dynamic Loading Summary

Lets summarize the algorithm for executing a dynamically loaded program.

1. exec is called to execute a new program.

2. The program’s ELF file is parsed, and if a “program interpreter” is set, the interpreter is instead executed.

3. In most cases, the interpreter is ld, and it is loaded into the process along with the target program.

4. Execution is started at ld, and it:

   1. Loads all of the dynamic libraries required by the program, looking at the LD_LIBRARY_PATH environment variable to help look for them.
   2. Links the program such that all of its references to library symbols (i.e. calloc above) are set to point to library functions.
   3. Executes the initialization procedures (and eventually main) of our program.

This is all quite complex. Why would we ever want all of this complexity over the simplicity of static libraries?

13.5.5 Saving Memory with Dynamic Libraries

First, lets look at how much memory a library takes up.

$ ls -lh /usr/lib/x86_64-linux-gnu/libc-2.31.so
-rwxr-xr-x 1 root root 2.0M Dec 16  2020 /usr/lib/x86_64-linux-gnu/libc-2.31.so

So libc takes around 2MiB. Other libraries take quite a bit more:

• Graphics card drivers (e.g. libradeonsi_dri.so is 23 MiB)
• Databases clients (e.g. libmysqlclient.so is 7.2 MiB)
• Languages (e.g. firefox’s javascript engine, libmozjs.so is 11 MiB, and webkit’s libjavascriptcoregtk.so is 26 MiB)
• …

If $N$ programs must load in multiple libraries (totaling, say, $X$ MiB), then $N \times X$ MiB is consumed. For context, my system is currently running 269 programs (ps aux | wc -l), so this memory can add up!

Dynamic libraries enable us to do a lot better. The contents memory for the library is the same regardless the process it is present in, and an Operating System has a neat trick: it can make the same memory appear in multiple processes if it is identical and cannot be modified²⁵. As such, dynamic libraries typically only require memory for each library once in the system (only $X$), as opposed for each process.

13.6 Static vs. Dynamic Library Memory Usage Summary

When a program is linked with a static library, the result is relatively small as only the objects in the library that are necessary are linked into the program. When executed as a process, the memory consumed will only include the necessary objects of the library. In contrast, a program that uses a dynamic library links in the entire library at runtime into a process. Thus, for a single process, a static library approach is almost always going to be more memory-efficient.

However, in a system with many executing processes, the per-library memory, rather than per-process memory, will result in significantly less memory consumption. Requiring only a single copy of a larger dynamic library shared between all processes, uses less memory than multiple copies of smaller compilations of the library in each program.

14 Organizing Software with Dynamic Libraries: Exercises and Programming

Lets dive a little deeper into how software is structured on our system, and the trade-offs that different libraries make in the system.

14.1 Exercise: Understanding Library Memory Consumption

Lets investigate the libexample from the figures in last week’s lectures. See the 08/libexample/ directory for examples of using normal linking (make naive), static library linking (make static), and dynamic library linking (make dynamic). The source includes:

• prog and prog2 that are programs that use libraries.
• example.c is a single file that creates the first library which generates libexample.a and libexample.so.
• example2.c and example2b.c are the files that create the second library which generates libexample2.a and libexample2.so.

Each of the library objects uses around 750KB of memory. To help you set the LD_LIBRARY_PATH to execute the dynamic libraries, you can run the dynamic library examples using

$ ./run_dyn.sh ./prog_dynamic

1. Use nm and ldd to understand which libraries and object files within those libraries each of the programs require.
2. Use ls -lh to see the size of each of the resulting executable programs. Explain the differences in sizes.

14.2 Library Trade-offs

• Question: What types of systems might want to always use only static libraries?
• Question: What types of systems might want to always use only dynamic libraries?
• Question: OSX uses dynamically linked libraries for a few common and frequently used libraries (like libc), and static libraries for the rest. Why?

14.3 Programming Dynamic Libraries