A broad class of drivers run in the operating system kernel, called kernel-mode drivers (KMD).
All code running in kernel mode shares a single virtual address space. KMDs aren’t isolated from other drivers or the OS or from user mode. If a KMD mistakenly writes to the wrong virtual address, it could compromise data belonging to the OS or another driver. And if a KMD crashes, the entire OS crashes.
Virtualisation of memory adds some problems.