According to Wikipedia, Concurrency is: the ability of different parts or units of a program, algorithm, or problem to be executed out-of order or in partial order, without affecting the final outcome. It’s not primarily about executing things in parallel or simultaneously. However, parallel or simultaneous execution presents concurrent challenges. Concurrency primarily arises when working with multiprocessing systems, whether time-shared or parallel. Traditionally, this has been a tricky area for programmers. In essence, using threads in your programs was a call for problems, and the traditional advice was always to avoid them as much as possible.

For years, multithreading development was avoided like the plague because it was hard to debug (there were no good tools at that time) and because the problems arising in your programs were often hard to reproduce. Then there was a time when everyone just added threads to run single tasks without synchronization or shared resources. They would simply start a thread to run a function, influenced by the idea that multithreaded programs were more efficient. Well, as we will see throughout this course, many of those programs didn’t add any performance improvement; in fact, it was pretty much the opposite.

Nowadays, with the advent of new programming languages, this has slightly changed because the language or the framework already takes care of many of the most common problems, or, alternatively, it restricts what you can do and, therefore, many of the issues with concurrent programming just disappear. However, whenever you need to build a concurrent application that is not trivial or you need to do something that falls outside the nominal use case covered by your tool, you can run into troubles very easily if you don’t understand the fundamentals.

So, we’ll deep dive into the related concepts so you get solid foundations to make your concurrent code solid rock. I anticipate that we’ll have to explore other parts of the system, sometimes hardware-related, to fully understand how all this works. Let’s get started with the basics.

Thread and Processes

In common operating systems, the way to run different parts of a program in pieces or out of order is either creating threads or creating processes. Deep inside the operating systems, there’s not much difference between the former or the latter (they are all some kind of task down there), but at the programmer level, they’re pretty different—as different as the way to program each one.

The API for working with processes has been around since the very beginning of multitasking operating systems and is very simple and stable. On the other hand, threads have had a rough time before they became, let’s say, stable. In the early days, there were different libraries you could use with different APIs, each with their own issues, such as LinuxThreads, Native POSIX Threads, GNU Portable Threads, or POSIX Threads.

From a programmer’s point of view, the main difference between a thread and a process is that threads belong to processes and share the process addressing space, while each process has its own addressing space. What does this mean for the programmer? Roughly speaking, any global data in your process is accessible to all the threads you create within that process. But any global data in a process isn’t accessible by other processes, even when one of those processes has been created from another and they share quite some data (like file descriptors, for example).

Thread and Processes (Continued)

Reality is a bit more complicated than that. When a process creates a child process (and that is the only way to create a new process, at least in UNIX systems), the child process starts as a perfect copy of the parent process. This means it will have read access to all data in the parent process. This is true until any of them overwrites that data. Up to that point, parent and child are sharing the exact same physical memory page. When any of them writes to the page, the kernel will fire the so-called COW (Copy-on-Write) process, which means that the process writing will get a brand-new physical memory page allocated, initialized with the same data contained by the shared page. This process effectively makes each process have its own private page. From that point on, the data seen by the parent and the child are no longer the same (they live in separated memory pages or addressing spaces, if you prefer).

Let’s see how this works with an example and also introduce the process management API in an indirect way.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include <sys/types.h>  // Needed for wait
#include <sys/wait.h>

int global_var = 1;

int main ()
{
  pid_t child;

  if ((child = fork ()) < 0) {   // Create a new process
    perror ("fork:");
    exit (EXIT_FAILURE);
  }
  if (child == 0) // Child process
    {
      global_var ++;
      printf ("CHILD: GLobal variable : %d\n", global_var);
      return 0; // The child process finish
    }
  else // This is the father
    {
      int status;
      wait (&status);
      printf ("FATHER: GLobal variable : %d\n", global_var);
    }

  return 0;
}

The program above creates a new process using the system call fork. The fork system call creates a new process that is an exact copy of the original process invoking it. When the system call is completed, there are two processes in the system, almost identical, each continuing its execution just after fork. The way for each of those processes to know who is who is the value returned by fork. If it is 0, that is the child process, the newly created process. Otherwise, fork returns the process identifier (the so-called PID).

Why is it like this? Well, the process management system is designed as a hierarchy (which makes perfect sense). Each process in the system has a single parent, except for process 1, which has no parent and is also the original parent of all processes. Therefore, it makes sense that the one receiving the PID is the parent, because a parent can have multiple child processes. As a child can only have one single parent, it can get that information from one single system call very easily. Actually, that system call is getppid.

Traditionally, process 1 was init for many years. In recent systems using systemd, process 1 is actually systemd.

Back to our program, in the child process (the one for which fork appears to return 0), a global variable is increased and then the child process ends. The parent (the one for which fork returns the pid of the child process) will just wait (using the wait system call) for the child process to finish and then will print the global variable too.

There is a waitpid system call that allows a parent to wait for the execution of a specific child, specifying its PID. wait on the other hand will wait until any child finishes. In this example, we just have one child process, so any of them will work fine.

If you compile and run the program, the result is:

$ make process
cc     process.c   -o process
$ ./process
CHILD: GLobal variable : 2
FATHER: GLobal variable : 1

When the child process is created, all the addressing space of the parent is copied (actually, the page table is copied, and the actual memory pages are shared). That’s why the initial value of the global variable is 1. At this point, whenever any of the processes—parent or child—writes to the global variable, the kernel will allocate a new physical page for that address and update the page table of the process. From that point on, the virtual address assigned to the global variable will be assigned to a different physical page for each of the processes. Effectively, parent and child have separate addressing spaces, even when all data that is not changed or read-only will be shared. Think about it as an optimization.

MEMORY MANAGEMENT PRIMER

In case the last paragraph sounds a bit confusing, let’s quickly explain how the memory management system in the kernel works. The Linux kernel uses paging to manage system memory. Paging just organizes the memory in blocks of a fixed size. For Linux, nowadays, it’s 4KB. This size is always a power of 2, which is very convenient. 4KB is 0x1000 in hexadecimal. So, the first page will go from 0x0000 to 0x0fff. The second page will go from 0x1000 to 0x1fff, and the fifth page will go from 0x4000 to 0x4fff. As you can see with this organization, we can just get the page number by isolating the higher bits in the address. The lower bits are the offset inside the page that allows us to refer to any byte in that memory area.

However, there’s a problem if, for example, two users run the same program. A program is designed to be loaded at a specific address (this is not fully true anymore, but it helps to understand how all this works). The first time we load the program, everything is fine. But if another user loads it again, the same memory addresses will be overwritten, and the first program will end up seeing the data of the second one.

To avoid this, the kernel creates a virtual space addressing using a page table. Every time a process accesses a memory address, that address is decomposed into a page number and an offset (as described above). The page number is used to access a table of physical memory pages. This way, the same virtual address can point to different physical pages, and the other way around, different virtual addresses can point to the same physical page (think, for example, of processes sharing code but loaded in different addresses).

So, when a child process is created, its page table is initialized with the values from its parent. They have the exact same virtual and physical memory map. Those pages are shared by both processes. When any of them tries to write, the Copy-On-Write process is fired, and the kernel allocates a new physical page to that process’ virtual address page, effectively assigning different physical pages to the same virtual address for each process.

As you can see, a lot of things happen even when we’ve just run a simple system call. Even when all this happens super fast, you have to consider if it makes sense in your application. Sometimes programmers just launch a process or a thread to do a small task that will likely be done in the main program with much less overhead if you carefully manage your I/O waits. Enough about fork.

Threads

Let’s see what happens when we use a thread:

#include <stdio.h>
#include <stdlib.h>

#include <pthread.h>

int global_var = 1;

void *func (void *p) {
  global_var ++;
  printf ("THREAD: Global variable: %d\n", global_var);
}

int main ()
{
  pthread_t tid;
  void      *r;

  if (pthread_create (&tid, NULL, func, NULL) < 0) exit (EXIT_FAILURE);
  pthread_join (tid, &r);
  printf ("MAIN: Global variable: %d\n", global_var);
  return 0;
}

As we can see, the program is quite different, so are the APIs. Actually, despite how different this may look, both fork and pthread_create rely on the same system calls at the lower level. In any case, we can see how the program creates a thread using pthread_create and instructs it to start execution on function func. Then it just waits for the thread to finish using pthread_join instead of wait. The result of this program is:

$ gcc -o thread thread.c -lpthread
$ ./thread
THREAD: Global variable: 2
MAIN: Global variable: 2

Which is what we’re expecting, as all threads share the same addressing space.

TLS. How to Make Threads Have Their Own Storage

We’ve just seen how the global variables in the program are automatically shared by all threads, no matter what. But what can we do if we need each thread to have its own version of a global variable? One of the more well-known cases is the errno variable.

In case you don’t know, errno is a global variable that is set to an error code each time a standard C Library function fails. This works great on single-threaded applications, but when we have multiple threads, the variable can be easily overwritten by one thread before the other one reads and processes the error value returned by its last function call. In cases like that, we’ll appreciate having our own copy of the global variable per thread.

To support these cases, the concept of Thread Local Storage (TLS) was introduced, providing each thread with a local storage where they can keep data that is not accessible to other threads, as for example, with the errno example. For the thread, the variables in this area may look like global variables; however, they can’t be seen from other threads.

There are a couple of ways of doing this depending on what you want. For single variables, the best way is to just annotate the variable with __Thread_local. This is the C11 standard way of putting a variable in the TLS. Before that, GCC was supporting the __thread decorator for the same purpose. You can change the previous program like this:

__thread int global_var = 1;

or like this

_Thread_local int global_var = 1;

And compile and run it again. You’ll see as each thread gets its own copy of the variable.

However, if you want to allocate more memory than a bunch of variables or you need to do that in a more dynamic way, there are a few POSIX standard functions that can help us. Let’s take a look at the following example:

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

pthread_key_t key;

void *worker(void *arg)
{
    int *value = malloc(sizeof(int));
    *value = *(int *)arg;

    pthread_setspecific(key, value);

    printf("%d\n", *(int *)pthread_getspecific(key));

    free(value);

    return NULL;
}

int main(void)
{
    pthread_key_create(&key, NULL);

    pthread_t t1, t2;
    int a = 10;
    int b = 20;

    pthread_create(&t1, NULL, worker, &a);
    pthread_create(&t2, NULL, worker, &b);

    pthread_join(t1, NULL);
    pthread_join(t2, NULL);

    pthread_key_delete(key);
}

The pthread_setspecific and pthread_getspecific functions allow us to associate some value (typically a pointer) to a key that has to be created using pthread_key_create. All threads will use the same key, but the setspecific and getspecific functions will store the values associated with that key for that specific thread. In the example, each thread allocates memory for a single integer, but it could allocate memory for a complex structure or a big array. This way, the same function access different data when executed by different threads.

But how does this work? For processes, we’ve seen how each process has its own page table, and the COW process duplicates physical pages on demand, effectively separating the addressing spaces of the two processes. But now we have one single process and one single page table.

TLS Internal Details

The TLS is implemented as a memory block that goes together with the thread stack. Yes, every thread needs its own stack; otherwise, calling functions from a thread will really be a mess. Actually, that will make the program crash most of the time.

Actually, the system needs some extra information for effectively managing threads, as its ID, status, etc. So, whenever a thread is created, quite a few things happen:

• The thread stack is allocated. As mentioned, without a stack, things will hardly work.
• The TLS itself is allocated. This is another memory block usually placed beside the stack after some guard space.
• Then it comes the TCB or Thread Control Block that contains housekeeping information for managing the thread. Among other things, it contains the position of the TLS for that thread.
• Finally, it also has to allocate memory for the pthread descriptor with the details pthread requires itself.

So, with all this information, the memory map for a just-created thread (leaving the common elements as .text aside), will look like this:

Higher Memory Addresses
+-----------
| Thread Stack
~
|
+--------------
| Page Guard
+-----------------
| TLS
| TCB
| pthread data
+--------------
Lower Memory Addresses

The exact layout of the memory depends on the implementation, but this is usually the case. A big chunk of memory big enough to hold the required data is allocated using mmap, and the different data structures are fit there. Note the guard page between the stack and the TLS. That will detect a stack overflow, producing a segmentation fault in case our thread stack grows off limits.

There is one last detail we need to know regarding TLS. How does the program access the TLS values, placed at different locations for each thread, when all threads run exactly the same code?

Yes, that is a problem, so the access to variables on the TLS is special. Let’s do a small modification to our last test program and make func modify a regular global variable and a TLS global variable:

int               global_var = 1;
_Thread_local int global_var1 = 1;


void *func (void *p) {
  global_var ++;
  global_var1 ++;
  printf ("THREAD: Global variable: %d %d\n", global_var, global_var1);
}

Let’s recompile and take a look to the asm:

$ make tls
$ objdump -d tls
$ objdump -d tls | grep -A 25 ":"
0000000000001169 :
    1169:   55                      push   %rbp
    116a:   48 89 e5                mov    %rsp,%rbp
    116d:   48 83 ec 10             sub    $0x10,%rsp
    1171:   48 89 7d f8             mov    %rdi,-0x8(%rbp)
    1175:   8b 05 b5 2e 00 00       mov    0x2eb5(%rip),%eax        # 4030 
    117b:   83 c0 01                add    $0x1,%eax
    117e:   89 05 ac 2e 00 00       mov    %eax,0x2eac(%rip)        # 4030 
    1184:   64 8b 04 25 fc ff ff    mov    %fs:0xfffffffffffffffc,%eax
    118b:   ff
    118c:   83 c0 01                add    $0x1,%eax
    118f:   64 89 04 25 fc ff ff    mov    %eax,%fs:0xfffffffffffffffc
    1196:   ff
    1197:   64 8b 14 25 fc ff ff    mov    %fs:0xfffffffffffffffc,%edx
    119e:   ff
    119f:   8b 05 8b 2e 00 00       mov    0x2e8b(%rip),%eax        # 4030 
    11a5:   89 c6                   mov    %eax,%esi
    11a7:   48 8d 05 5a 0e 00 00    lea    0xe5a(%rip),%rax        # 2008 <_IO_stdin_used+0x8>
    11ae:   48 89 c7                mov    %rax,%rdi
    11b1:   b8 00 00 00 00          mov    $0x0,%eax
    11b6:   e8 75 fe ff ff          call   1030 
    11bb:   90                      nop
    11bc:   c9                      leave
    11bd:   c3                      ret

00000000000011be 
:

We can see clearly the access to global_var using a RIP-relative addressing, typical on x86_64 PIE binaries. But then, we found these lines:

    1184:   64 8b 04 25 fc ff ff    mov    %fs:0xfffffffffffffffc,%eax
    118b:   ff
    118c:   83 c0 01                add    $0x1,%eax
    118f:   64 89 04 25 fc ff ff    mov    %eax,%fs:0xfffffffffffffffc
    

This is the increment of global_var1. Taking into account that 0xfffffffffffffffc is -4 and this is how the program accesses the TLS. Whenever a thread gets executed, as part of the context switching, the segment register FS is set to point to the TLS (actually to the TCB). In this case, the TLS contains a single int, so the -4 value. You can change the variable to be a long and see what happens in the ASM.

Deep Dive into Thread Control Block

If you are not interested in these details, you can safely skip this section, but I have to say this is pretty cool ;). To get a better image of all these structures in memory, let’s modify our small example to dump some data so we can make sense out of it.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <pthread.h>

long               global_var = 1;
_Thread_local long global_var1 = 1;

#define SIZE 10*8 // 10 words should be enough

void *func (void *args) {
  unsigned char *buffer = (unsigned char*)(&global_var1);
  char           buffer1[16];
  unsigned long  *p;

  // Change this to thew value that doesn't fire the canary
  // May be different in yout boc
  memset (buffer1, 0x42, 24);

  // Let's put some meaningful value in our TLS variable for easy
  // identify it. Amd proint some data
  global_var1 = 0x1122334455667788;
  printf ("THREAD %lx\n", pthread_self());
  
  puts ("Dumping Stack");
  for (int i = 0; i < SIZE; i+=8) {
    p = (unsigned long*)(buffer1 + i);
    printf ("%p + %02d -> %p\n", buffer1, i, *p);
  }

  puts ("Dumping TLS");
  for (int i = 0; i < SIZE; i+=8) {
    p = (unsigned long*)(buffer  + i);
    printf ("%p + %02x -> %p\n", buffer + i, i, *p);
  }

}

int main ()
{
  pthread_t tid;
  void      *r;
  if (pthread_create (&tid, NULL, func, NULL) < 0) exit (EXIT_FAILURE);
  printf ("MAIN: tid: %p\n", tid);
  pthread_join (tid, &r);
  printf ("MAIN: Global variable: %d\n", global_var);
  return 0;
}

A few comments regarding the new thread function. First, I added a local buffer and a memset to force a buffer overflow. This is because we want to take a look at canaries. You’ll see why in a second. Note that in your box, you may need to write less or more bytes into the buffer to force the stack canary to fire up. Just start at 16 and increase until the program fails. That will allow you to easily identify the canary in the stack. Then, we dump the TLS using the pointer to the one and only variable we stored there. Let’s compile the program with the -fstack-protector flag and take a look at the output.

$ gcc -o tls1 tls1.c -fstack-protector
tls1.c: In function ‘func’:
tls1.c:19:3: warning: ‘memset’ writing 24 bytes into a region of size 16 overflows the destination [-Wstringop-overflow=]
   19 |   memset (buffer1, 0x42, 24);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~
tls1.c:14:18: note: destination object ‘buffer1’ of size 16
   14 |   char           buffer1[16];
      |                  ^~~~~~~
$ ./tls1
MAIN: tid: 0x7fdc8a21c6c0
THREAD 7fdc8a21c6c0
Dumping Stack
0x7fdc8a21beb0 + 00 -> 0x4242424242424242
0x7fdc8a21beb0 + 08 -> 0x4242424242424242
0x7fdc8a21beb0 + 16 -> 0x4242424242424242
0x7fdc8a21beb0 + 24 -> 0x9ee0cecb5a79d600
0x7fdc8a21beb0 + 32 -> 0xffffffffffffff40
0x7fdc8a21beb0 + 40 -> 0x7fdc8a2b2b7b
0x7fdc8a21beb0 + 48 -> (nil)
0x7fdc8a21beb0 + 56 -> 0x7fdc8a21c6c0
0x7fdc8a21beb0 + 64 -> 0x80
0x7fdc8a21beb0 + 72 -> 0xf25d33579c048a17
Dumping TLS
0x7fdc8a21c6b8 + 00 -> 0x1122334455667788
0x7fdc8a21c6c0 + 08 -> 0x7fdc8a21c6c0
0x7fdc8a21c6c8 + 10 -> 0x563d58ce32b0
0x7fdc8a21c6d0 + 18 -> 0x7fdc8a21c6c0
0x7fdc8a21c6d8 + 20 -> 0x1
0x7fdc8a21c6e0 + 28 -> (nil)
0x7fdc8a21c6e8 + 30 -> 0x9ee0cecb5a79d600
0x7fdc8a21c6f0 + 38 -> 0xbaf406d166543142
0x7fdc8a21c6f8 + 40 -> (nil)
0x7fdc8a21c700 + 48 -> (nil)
MAIN: Global variable: 1

Ignore the compilation warning, as that is intentional, and let’s focus on the program dump. In the stack, we can see our overflowed buffer with the 0x42 value up to offset 0x24. If we write one more byte, we will be overwriting the canary. So, for this thread, the canary is 0x9ee0cecb5a79d600. Note that the canary is randomly generated in each execution, so you will get a completely different value in your machine.

Now let’s look at the TLS dump. First, note that the pthread_t pointer is actually pointing to 0x7fdc8a21c6c0. That is our TCB, that follows just after the TLS. The fancy value we set into our TLS variable is easily identified, and we can see how it just sits behind the TCB. The TCB contains a pointer to itself, then it comes some other stuff, including the DTV or Dynamic Thread Vector pointer used for managing TLS for dynamic libraries. We won’t go into that, but now you know it exists. Then it follows the self pointer that once again points to the thread descriptor that, for this case, is again the same block. There are a few words containing some flags, and then we find the canary value.

If you have looked into canaries at some point, you have seen code like this:

11b1:   48 89 7d b8             mov    %rdi,-0x48(%rbp)
11b5:   64 48 8b 04 25 28 00    mov    %fs:0x28,%rax
11bc:   00 00
11be:   48 89 45 f8             mov    %rax,-0x8(%rbp)

Which is exactly what we are looking at. The FS register points to the Thread Control Block. Note that in our dump, TLS is at offset 0 and the TCB is actually at offset 8, so in our dump, we have to look at offset 0x30 to find the canary. As you can see, each thread has its own canary value. Single-threaded applications also have a TCB containing the canary and similar details.

One last thing to finish this deep dive. Whenever we add data on the TLS, the compiler adds two sections to the final binary: .tdata and .tbss. They are kind of analogous to the regular .data and .bss. The former contains the initialization values for initialized global variables, while the second contains the size of the non-initialized values that the application will need.

For a threaded application, the .tdata and .tbss are used as a template. Whenever a thread is created, a TLS with a size enough to fit the data defined by those two sections is allocated. Then, for the initialized values, the .tdata content is copied into the TLS, at the beginning, and the rest of the TLS will be used to hold uninitialized values (something like int array[1024]).

If we take a quick look at our binary (it doesn’t have a .tbss, but you can create one easily):

$ readelf -S tls1 | grep -A1 tdata
  [20] .tdata            PROGBITS         0000000000003dc8  00002dc8
       0000000000000008  0000000000000000 WAT       0     0     8

If we dump the referred content in the binary we get:

$ xxd -s $((0x00002dc8)) -l 7 -e -g 8t tls1
00002dc8:   00000000000001                   .......

Which is the value used to initialize our TLS variable. You can change it to something else and take again a look at the resulting binary.

Well, this is it for the TLS deep dive. Let’s get back to business.

How to share data with processes?

You may be wondering if it is possible to get the same result we have got using threads with processes instead. The answer is yes, but in order to achieve that, we need to use the so-called IPC API (InterProcess Communication). All multitasking operating systems have more than one way to communicate processes.

In UNIX systems, there are many different ways to do it. We can use pipes or named pipes, Unix sockets, socketpairs,…. But the more generic API, and also the one that matches best the threads API is the so-called System V IPC. System V IPC interface defines three main objects to enable this inter-process communication: Shared Memory, Semaphores, and Messages.

Right now, for the simple example that we are working on, we can easily use shared memory to allow two or more processes to share a piece of memory… In other words, to make our global variable, global among processes.

Using this System V IPC element, our original program will change like this:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include <sys/ipc.h>
#include <sys/shm.h>

#include <sys/types.h>  // Needed for wait
#include <sys/wait.h>

int main ()
{
  pid_t  child;
  key_t  key;
  int    shared_id;
  int    *global_var;

  // Create a unique key for the shared memory block
  key = ftok (".", 'd');
  
  if ((shared_id= shmget (key, sizeof (int), IPC_CREAT | 0666)) < 0) {
    perror ("shmget:");
    exit (EXIT_FAILURE);
  }

  // Map shared memory in the father's process address space
  global_var  = (int*) shmat (shared_id, NULL, 0);

  *global_var = 1; // Let's initialise the global var to 1
  
  // Create a new process
  if ((child = fork ()) < 0) {
    perror ("fork:");
    exit (EXIT_FAILURE);
  }
  if (child == 0) // Child process
    {
      *global_var = *global_var + 1;
      printf ("CHILD: GLobal variable : %d\n", *global_var);
      shmdt ((void*)global_var); // Deattach from shared memory
      return 0; // The child process finish
    }
  else // This is the father
    {
      int status;
      wait (&status);
      sleep (1);
      printf ("FATHER: GLobal variable : %d\n", *global_var);
      shmdt ((void*)global_var);
    }

  return 0;
}

When we run this program, now the result is the expected one:

$ ./shared_memory
CHILD: GLobal variable : 2
FATHER: GLobal variable : 2

Some comments on the previous code:

• The ftok function allows us to create an unique identifier that we can easily locate from other processes because it is associated to an entity in the filesystem.
• In the general case, when the child process is a different program, it will also need to call ftok and shmget in order to find out what is the right shared memory identifier or somehow manage to get the shared memory id returned by the process that created the memory.
• Also, in the general case, we have to call shmat on each process to attach the shared memory block into the process addressing space. In general, each process may have a completely different address to access the shared memory, however, that can be forced using the second parameter and appropriate flags. In our example, as we are not calling exec, the shared memory is also mapped in the child process, and we can just access it. If we call execve to run a different program, which is the normal case, we will have to get the shared memory id and attach it as described in this bullet and the previous one.
• Finally, the shared memory block shall be destroyed calling shmdt.

The POSIX Way

System V IPC is a classic, still works, and it’s maybe the one that may work whenever you have to deal with an old system. For modern systems, there’s a POSIX-based interface, in a sense a bit more straightforward, and in the last instance, this is how the System V IPC shared memory is implemented. The overall concepts are exactly the same.

The POSIX version of our previous program is this:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include <sys/mman.h>
#include <sys/stat.h>        /* For mode constants */
#include <fcntl.h>           /* For O_* constants */

#include <sys/types.h>  // Needed for wait
#include <sys/wait.h>

#define SM_FILENAME "/sm"
#define SM_SIZE sizeof(int)

int main ()
{
  pid_t  child;
  key_t  key;
  int    shared_id;
  int    *global_var;

  if ((shared_id = shm_open (SM_FILENAME, O_CREAT | O_RDWR, 0666)) < 0) {
    perror ("shm_open1:");
    exit (EXIT_FAILURE);
  }
  ftruncate (shared_id, SM_SIZE);
  global_var = (int*) mmap (NULL, SM_SIZE, 0666, MAP_SHARED,
                shared_id, 0);
  if (global_var == MAP_FAILED) {
    perror ("mmap:");
    exit (EXIT_FAILURE);
  }

  *global_var = 1;
  printf ("Creating Process...\n");
  // Create a new process
  if ((child = fork ()) < 0) {
    perror ("fork:");
    exit (EXIT_FAILURE);
  }
  if (child == 0) // Child process
    {
      *global_var = *global_var + 1;
      printf ("CHILD: GLobal variable : %d\n", *global_var);

      close (shared_id);
      munmap (global_var, SM_SIZE);
      shm_unlink (SM_FILENAME);
      return 0; // The child process finish
    }
  else // This is the father
    {
      int status;
      wait (&status);
      sleep (1);
      printf ("FATHER: GLobal variable : %d\n", *global_var);
      
      close (shared_id);
      munmap (global_var, SM_SIZE);
      shm_unlink (SM_FILENAME);
    }

  return 0;
}

Some comments on this code:

• shm_open actually creates a file at /dev/shm. This has two implications.
• The first one is that the first parameter to shm_open, has to be in the form of /some_name… Any other name will produce an invalid argument error.
• The second is that the size of the file is the size of the shared memory. So if the file is just created (using the O_CREAT flag), a call to ftrunc is needed to add size to the file.

The POSIX interface makes more sense in the overall UNIX philosophy in the sense that it makes more explicit the use of files and memory mapping (actually using the standard mmap for that with file backing). I personally love the System V one… because it was just the first I learned.

A deep dive into POSIX shared memory

Let’s look a little bit deeper, not much this time, just a little bit. There is a structure in the Linux kernel named the page cache which caches disk blocks into physical memory pages. This process happens whenever we access a file on the disk. Even when our application in user space seems to allocate a buffer and read some random size data into it, at the kernel level, a whole page is allocated and filled with as many data from the disk as possible. That page stays in the cache, associated to that file, and whoever tries to access that file again, will get access to that page immediately. In other words, the disk is only hit once.

In our previous shared memory example, what we did was to create a file and use mmap to create a so-called VMA. A Virtual Memory Area is a structure that defines a virtual memory block. It will mark a block of virtual addresses with some metadata and also associate them to the page table. However, mmap just creates that structure and no real physical memory is allocated yet.

When the process tries to read (or write) that memory for the first time, depending on the virtual address used, the kernel will load the relevant block of the file and store it in a physical memory page that now belongs to the page cache. The Process Page Table is updated and now it points to that physical page. Any process reading its own mapping of that file-backed memory block will be linked to the same physical memory page (no new read from disk).

This is roughly what we described when a child process read some data in a memory block already mapped by the parent. However, when any of the process wants to write into that memory, there is no Copy-On-Write process because the memory block was marked as MAP_SHARED. So all the involved processes will keep seeing the same cache page, and the changes any one of them does. What happens in this case is that the page is marked as dirty and eventually should be written back into the disk.

Note that it is possible to create memory blocks not backed up by files using MAP_ANONYMOUS, but in that case, the only way to get another process to be able to map the same memory is forking the process. The use of the file is actually a way to let other processes be able to find the same physical pages.

Conclusion

In this introductory paper, we have gone through the basics on how to create threads and processes and also explored the differences between them. We also introduced the two main APIs for inter-process communication on UNIX systems and got ready for the next round!