I wrote a pretty simple program to get started. Something like this:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include <sys/types.h>  // Needed for wait
#include <sys/wait.h>


int global_var = 1;

int main ()
{
  pid_t child;

  // Create a new process
  if ((child = fork ()) < 0) {
    perror ("fork:");
    exit (EXIT_FAILURE);
  }
  if (child == 0) // Child process
    {
      global_var ++;
      printf ("CHILD: GLobal variable : %d\n", global_var);
      return 0; // The child process finish
    }
  else // This is the father
    {
      int status;
      wait (&status);
      sleep (1);
      printf ("FATHER: GLobal variable : %d\n", global_var);
    }

  return 0;
}

This is actually part of the introduction to my Concurrent Programming Course that I recommend to read. Anyhow, the program is pretty simple and it’s the classical example where the COW process is fired.

The child process starts with a page table pointing to shared pages with the parent process. Let’s focus on the .data segment/section where the global_variable lives. The first thing the child process does is write into that variable. That should cause a page fault that fires the Copy-On-Write process that, just does that… copy on write. A new page is allocated, the content of the old page is copied into the new page, the page table gets updated and the write happens in a new memory page. Everything looks great.

So I thought. Wouldn’t it be nice to be able to visualize this process?

Accessing Page Tables

In principle, page tables cannot be accessible from user space. But there are a few things that can be done without writing a kernel module to do that. The most direct one is the access to the /proc/pid/pagemap, so let’s try to use it.

This file contains a 64bit value for each virtual page in the system. It’s a huge file and it’s binary so you cannot just cat it. I tried and I had to cancel the process. The content of each of these entries is specified in the kernel documentation page linked above, but let’s reproduce it here for our own convenience.

• Bits 0-54 page frame number (PFN) if present
• Bits 0-4 swap type if swapped
• Bits 5-54 swap offset if swapped
• Bit 55 pte is soft-dirty (see Soft-Dirty PTEs)
• Bit 56 page exclusively mapped (since 4.2)
• Bit 57 pte is uffd-wp write-protected (since 5.13) (see Userfaultfd)
• Bit 58 pte is a guard region (since 6.15) (see madvise (2) man page)
• Bits 59-60 zero
• Bit 61 page is file-page or shared-anon (since 3.5)
• Bit 62 page swapped
• Bit 63 page present

For our experiment, we’re only interested in the PFN (Page frame number) that, effectively, is the physical address associated to that page and the bit 63 that indicates if the page is present or not. We’ll come to this bit a bit later :).

So I wrote the following function in my program to be able to access the process page table:

int print_physical_mem (char *title,uint8_t *p) {
  const size_t pagesize = sysconf(_SC_PAGESIZE);
  int          fd;
  uint64_t     vaddr, vpn, entry, pfn, phys;
  off_t        offset;
  
  if ((fd = open ("/proc/self/pagemap", O_RDONLY)) < 0) {
    perror ("open");
    return 1;
  }

  vaddr  = (uint64_t)p;
  vpn    = vaddr / pagesize;
  offset = vpn * sizeof(uint64_t);

  if (lseek (fd, offset, SEEK_SET) == (off_t)-1) {
    perror("lseek");
    return 1;
  }
  if (read (fd, &entry, sizeof(entry)) != sizeof(entry)) {
    perror("read");
    return 1;
  }
  close(fd);
  
  printf ("--[%d:%s]----------------------------------\n",
      getpid(), title);
  printf ("Virtual Address   : %p\n", p); 
  printf ("Raw pagemap entry : 0x%016lx\n", entry);
  printf ("Present           : %d\n", (int)((entry >> 63) & 1));

  pfn = entry & ((1ULL << 55) - 1);
  printf("PFN               : 0x%lx\n", pfn);

  if (pfn != 0 && (int)((entry >> 63) & 1) == 1) {
    printf("Physical addr     : 0x%016lx\n",
       pfn * pagesize + (vaddr % pagesize));
  } else {
    printf("Physical addr     : Not assigned\n");
  }
  
  printf ("------------------------------------\n");
  return 0;
}

The function is pretty straightforward. Takes the virtual address we pass as parameter, calculates the virtual page number (VPN – just dividing by the page size) and then it calculates the offset of that page in the file, that is just the page number times the size of each entry, that is 64 bits. Then we just print the information for that entry as per the description in the kernel documentation (see above).

Let’s try

The mystery arises

Now that we can peek into the page table, let’s modify the main program to show this information. This is how it looked like:

int main ()
{
  printf ("Virtual Address: %p\n", &global_var);
  print_physical_mem ("FATHER", (uint8_t*)&global_var);
  // Create a new process
  if ((child = fork ()) < 0) {
    perror ("fork:");
    exit (EXIT_FAILURE);
  }
  if (child == 0) // Child process
    {
      print_physical_mem ("CHILD - BEFORE WRITE", (uint8_t*)&global_var);
      global_var ++;
      print_physical_mem ("CHILD - AFTER WRITE", (uint8_t*)&global_var);      
      printf ("CHILD: GLobal variable : %d\n", global_var);
      printf ("============================\n");
      return 0; // The child process finish
    }
  else // This is the father
    {
      sleep (5);
      printf ("FATHER: GLobal variable : %d\n", global_var);
      print_physical_mem ("FATHER", (uint8_t*)&global_var);
      printf ("..................................\n");
    }

  return 0;
}

I first call the function at the very beginning to show the initial page table entry (the one from the parent process). Then, in the child process, I print the entry again before and after modifying the variable. In the parent process, I wait 5 seconds to ensure the child has modified the global variable before printing again the page table.

What you would expect is that after the write in the global variable by the child process, a new physical page was assigned to it and the parent process will keep the original physical page. This is how the Copy-On-Write process works, right? Let’s see what do we get.

$ sudo ./process1
Virtual Address: 0x55b10616d070
--[1550034:FATHER]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000000ad83fa
Present           : 1
PFN               : 0xad83fa
Physical addr     : 0x0000000ad83fa070
------------------------------------
--[1550035:CHILD - BEFORE WRITE]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000000ad83fa
Present           : 1
PFN               : 0xad83fa
Physical addr     : 0x0000000ad83fa070
------------------------------------
--[1550035:CHILD - AFTER WRITE]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000000ad83fa
Present           : 1
PFN               : 0xad83fa
Physical addr     : 0x0000000ad83fa070
------------------------------------
CHILD: GLobal variable : 2
============================
FATHER: GLobal variable : 1
--[1550034:FATHER]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000001a97e21
Present           : 1
PFN               : 0x1a97e21
Physical addr     : 0x0000001a97e21070
------------------------------------
..................................

Of course, if you run this in your machine, you’ll get completely different numbers. Actually, you’ll get different numbers each time you run the program. But let’s look at each of these specific blocks one by one.

--[1550034:FATHER]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000000ad83fa
Present           : 1
PFN               : 0xad83fa
Physical addr     : 0x0000000ad83fa070
------------------------------------

This is the initial page entry for the global_var virtual address located at 0x55b10616d070. This virtual address is at physical memory page 0xad83fa. So far, so good. Then we get the child print outs:

--[1550035:CHILD - BEFORE WRITE]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000000ad83fa
Present           : 1
PFN               : 0xad83fa
Physical addr     : 0x0000000ad83fa070
------------------------------------
--[1550035:CHILD - AFTER WRITE]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000000ad83fa
Present           : 1
PFN               : 0xad83fa
Physical addr     : 0x0000000ad83fa070
------------------------------------
CHILD: GLobal variable : 2

This is weird! The global variable is updated correctly, but the child process’s physical address isn’t updated. It still points to 0xad83fa. But the stranger thing comes after:

FATHER: GLobal variable : 1
--[1550034:FATHER]----------------------------------
Virtual Address   : 0x55b10616d070
Raw pagemap entry : 0x8180000001a97e21
Present           : 1
PFN               : 0x1a97e21
Physical addr     : 0x0000001a97e21070
------------------------------------

The parent process is the one that got the new physical page, even when the child process was the one causing the write page fault that fires the Copy-On-Write.

Further Investigation

That result didn’t make much sense; it looks like if, for whatever reason, the kernel was assigning a new page to the parent process even when the child process was the one writing to the page. That doesn’t make much sense but could happen. After all, I don’t really know how the kernel works at that level of details. So I modify once again my example and instead of using a global variable, I allocate my own data page to be sure I have control on who writes to that page.

The program now looks like this:

int main ()
{
  pid_t child;

  // Memory allocated
  int *p = mmap(NULL,
        4096,
        PROT_READ | PROT_WRITE,
        MAP_PRIVATE | MAP_ANONYMOUS,
        -1,
        0);

  print_physical_mem ("FATHER", (uint8_t*)p);
  *p = 42;
  print_physical_mem ("FATHER", (uint8_t*)p);
  // Create a new process
  if ((child = fork ()) < 0) {
    perror ("fork:");
    exit (EXIT_FAILURE);
  }
  if (child == 0) // Child process
    {
      print_physical_mem ("CHILD - BEFORE WRITE", (uint8_t*)p);
      *p = 43;
      printf ("CHILD: GLobal variable : %d\n", *p);
      print_physical_mem ("CHILD - AFTER WRITE", (uint8_t*)p);
      printf ("============================\n");
      return 0; // The child process finish
    }
  else // This is the father
    {
      sleep (5);
      printf ("FATHER: GLobal variable : %d\n", *p);
      print_physical_mem ("FATHER", (uint8_t*)p);
      printf ("..................................\n");
    }

  return 0;
}

Basically, the same program, but this time, the variable to modify is allocated using mmap, ensuring it’s in its own page. The output of this program is the following:

$ sudo ./process2
--[1554732:FATHER]----------------------------------
Virtual Address   : 0x7fef11d71000
Raw pagemap entry : 0x0080000000000000
Present           : 0
PFN               : 0x0
Physical addr     : Not assigned
------------------------------------
--[1554732:FATHER]----------------------------------
Virtual Address   : 0x7fef11d71000
Raw pagemap entry : 0x81800000014ed847
Present           : 1
PFN               : 0x14ed847
Physical addr     : 0x00000014ed847000
------------------------------------
--[1554733:CHILD - BEFORE WRITE]----------------------------------
Virtual Address   : 0x7fef11d71000
Raw pagemap entry : 0x80800000014ed847
Present           : 1
PFN               : 0x14ed847
Physical addr     : 0x00000014ed847000
------------------------------------
CHILD: GLobal variable : 43
--[1554733:CHILD - AFTER WRITE]----------------------------------
Virtual Address   : 0x7fef11d71000
Raw pagemap entry : 0x8180000001720ba6
Present           : 1
PFN               : 0x1720ba6
Physical addr     : 0x0000001720ba6000
------------------------------------
============================
FATHER: GLobal variable : 42
--[1554732:FATHER]----------------------------------
Virtual Address   : 0x7fef11d71000
Raw pagemap entry : 0x81800000014ed847
Present           : 1
PFN               : 0x14ed847
Physical addr     : 0x00000014ed847000
------------------------------------
..................................

The first thing to note is that, just after the mmap, there’s no physical memory assigned to the virtual address yet. Once we access that address (we wrote value 42 in that memory address), a physical page is assigned to it, in this case 0x14ed847. Now everything works as expected. The child process gets a new physical page after writing to the virtual address and the parent process keeps the original page.

So what happened with our initial example?

Unveiling the Mystery

When you hear what happened, you’ll immediately understand, but it took me a while. Actually, I had to go to bed, and the very next day, with a fresh mind, I see the problem very quickly. The issue is lazy binding.

Dynamic binaries in Linux usually do lazy binding. That means that external symbols are resolved only when needed, in other words, when the function is first called in the program. The symbol resolution implies an update of the GOT table of the program. Let’s take a look where that table is:

$ readelf -l process1

Elf file type is DYN (Position-Independent Executable file)
Entry point 0x1100
There are 14 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x0000000000000310 0x0000000000000310  R      0x8
  INTERP         0x0000000000000394 0x0000000000000394 0x0000000000000394
                 0x000000000000001c 0x000000000000001c  R      0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000890 0x0000000000000890  R      0x1000
  LOAD           0x0000000000001000 0x0000000000001000 0x0000000000001000
                 0x0000000000000571 0x0000000000000571  R E    0x1000
  LOAD           0x0000000000002000 0x0000000000002000 0x0000000000002000
                 0x000000000000032c 0x000000000000032c  R      0x1000
  LOAD           0x0000000000002dd0 0x0000000000003dd0 0x0000000000003dd0
                 0x00000000000002a4 0x00000000000002a8  RW     0x1000
  (....)
  
 Section to Segment mapping:
  Segment Sections...
   00
   01     .interp
   02     .note.gnu.property .note.gnu.build-id .interp .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt
   03     .init .plt .plt.got .text .fini
   04     .rodata .eh_frame_hdr .eh_frame .note.ABI-tag
   05     .init_array .fini_array .dynamic .got .got.plt .data .bss
   (...)

As you can see, the .data as well as the .got and .got.plt sections are all at segment 05, all in the same page. So what was happening was that the call to sleep to ensure that the child process had modified the global variable was firing another write (the GOT update) in the same page containing the global variable. In other words, when we call sleep in the parent process, we’re forcing a page fault in the parent and therefore it is the one getting the new physical page.

You can easily check this just by calling sleep once at the very beginning of the program. That way, the symbol is already resolved before creating the child process and won’t be any write later.

SUMMARY

There you go, a mystery case fired by an innocent sleep in our program to force some execution order. In this case, this wasn’t a concurrency-related problem. Actually, there was no problem at all, just something to understand, but it may be an example of the consequences of innocent function calls in a program that may look like innocuous but, unless you know what happens under the hood, will look like bad magic seen from outside. Hope you enjoyed the reading, it was a lot of fun for me to write the programs and this write-up :)