System calls (syscalls) are special requests made to an operating system by a program. The point of these is to bridge the privilege gap between user and kernel mode in a controlled way, without exposing too much powerful behaviour.

Trap instructions

To execute a syscall, programs must execute special trap instructions, which jump into the kernel and change the privilege mode to kernel mode. When done, it calls a return-from-trap instruction which returns into the caller and reduces the privilege mode.

On x86, the processor must push the program counter, flags, and a few other registers (including registers that hold important segment information, like where the code/data/stack are in memory) onto the kernel stack, which is per process. This manages process context and ensures the program continues executing correctly.

The kernel sets up a trap table at boot time for the CPU to store in hardware (via a privileged instruction). This trap table holds the addresses of trap handlers (much like interrupt handlers) for interrupts or when a program makes a syscall. Then, when a program makes a syscall request, it requests a particular syscall number instead of a specific handler (for security reasons).

UNIX-based systems

On UNIX-based systems (including Linux), there are ~453 total syscalls. See Linux syscalls for a non-exhaustive list.

Additionally:

  • strace <PROGRAM> — a command-line utility that can trace all system calls a process makes.
  • If a syscall fails, the error code is saved in errno. This should be saved to a local variable if other functions are later used that might set it.