disclaimer: some may find some of this content opinionated, but it stems from arbitrary external authorities making choices on users' behalf and not allowing that to be changed. Various manifestations of that occur on the x86 and in existing kernels
> Someone: "Virtual memory is free"
> Me: "yeah, no"
See something odd? I practically cannot use the last ~50Gi of my physical memory. Virtual memory, which doesn't exist physically, runs out prior to the actual physical resource. "Why?" Exactly
We can all guess who holds the most virtual allocations:
Name : msedgewebview2
Id : 28620
VM(GB) : 3614.21
WS(GB) : 0.31
Commit(MB) : 324
Name : msedge
Id : 15308
VM(GB) : 3612.44
WS(GB) : 0.29
Commit(MB) : 218
Name : msedgewebview2
Id : 68292
VM(GB) : 3612.35
WS(GB) : 0.11
Commit(MB) : 105
Name : msedge
Id : 64504
VM(GB) : 3612.34
WS(GB) : 0.52
Commit(MB) : 539
Name : msedgewebview2
Id : 81696
VM(GB) : 3612.18
WS(GB) : 0.01
Commit(MB) : 2
Why are we even seeing numbers like this? It's allowed but arguably trains people to be lazy. This should never have been allowed. There is one number that I care about, which is how much of my physical memory remains usable. That is not even represented straight because the literal number in the leading screenshot is not truthful. The remaining physical memory there is not, in fact, how much I can allocate. These lies and ambiguity is my problem with virtual memory, since that's the mechanism that is enabling this in the first place. What I want is a set of absolute values. I want to know exactly how much memory I have left to use now. I want to know that this address x is really the absolute address in memory and not some translation.
Commit and physical allocations (WS is physical):
Name Id Commit(GB) WS(GB)
---- -- ---------- ------
vmmemWSL 45456 42.59 31.9
msedge 22332 3.99 0.4
WindowsTerminal 25696 3.13 0.45
Surprise, surprise, there's a browser. And the last time I remember, a terminal emulator was for forwarding text between a user and a shell via a pty. In a tty, there wouldn't even be a terminal "emulator" and you'd talk directly to the shell.
MemTotal: 131903100 kB
MemFree: 93283484 kB
MemAvailable: 81792668 kB
Buffers: 632 kB
Cached: 6299256 kB
Active: 1629512 kB
Inactive: 30473308 kB
Active(anon): 6860 kB
Inactive(anon): 27818836 kB
Active(file): 1622652 kB
Inactive(file): 2654472 kB
Unevictable: 2456 kB
Mlocked: 2456 kB
Dirty: 192 kB
AnonPages: 25805596 kB
Mapped: 605452 kB
Shmem: 2020524 kB
KReclaimable: 212984 kB
Slab: 520648 kB
SReclaimable: 212984 kB
SUnreclaim: 307664 kB
KernelStack: 25504 kB
PageTables: 89324 kB
CommitLimit: 131903100 kB
Committed_AS: 87014536 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 54024 kB
Percpu: 12416 kB
Hugepagesize: 2048 kB
DirectMap4k: 185344 kB
DirectMap2M: 41756672 kB
DirectMap1G: 93323264 kB
The Linux perspective. Most memory is not available but idle. Why can't I use the memory that nobody else is using right now? People got trained to abuse memory because the platform allowed it. The web is the best example of this behavior out there.
There are two popular ways people chose to work with virtual memory:
Under NT, without a page file, virtual memory, which is a made-up construct, you run out of system memory before you even hit the physical limit. If you have plenty of memory, using a page file is rather counterintuitive. Even if nothing uses that page file, the fact that this has to be done in the first place is the source of ambiguity and confusion. They all say that it's "free," but it isn't. First of all, you have to look at a completely different number than your remaining physical memory. Second of all, page tables, TLB, and everything else that has to be done to support this nonsense.
On Linux, the kernel happily hands out memory it doesn't actually have. It trusts any program allowed to allocate that it will surely not use all of that memory. If that memory were money, something like this would be something like a Ponzi scheme, which is a felony. Programmers got used to creating gigantic memory pools and relying on them just working, which is what causes problems like what you can see above. And this is the case only because the substrate allowed it. I once disabled overcommit on Linux and had something that already allocated some arbitrarily large amount of memory and the system immediately deadlocked. As far as I could tell, it was a deadlock since I/O was disconnected right away (everything froze and the keyboard LED went down so something killed at least USB power) and the ssh server didn't respond. On the next boot, nothing of browser sort was starting. These people were never told "no" by the platform, so they abused it. It goes beyond browsers, too, but browsers are always at the top of the list when they're running.
So, it's two non-ideal ways to deal with something that shouldn't have existed.
And no, you don't need it for "isolation" or "protection." Those can be solved differently.
At the end of the day, I want visibility and control over my machine. Crosstalk's Memview shows the entire address range, which includes I/O, of the machine being run. You can observe and modify values in any way you see fit. I want that but on my entire system.
Since I am building a new platform, I chose not to subscribe to this. The early DANGLING, the one that ran on the AMD64, identity-mapped virtual addresses to the physical ones (technically, its bootloader did that and not the C code itself, but the point remains). I'd rather have avoided that completely, but since to enter long mode, you are forced to enable paging, it was the only way to avoid virtual addressing logic within DANGLING.
MISC doesn't have any concept of virtual memory. The role of protection and isolation is the kernel's responsibility, which, in my case, is DANGLING. DANGLING maintains a record of who asked for memory and why (allocations must include an explanation in source code, which is treated like a required comment) and gates access. Because it knows who asked, it knows to not let anyone else seek/peek/poke that memory. When someone calls the kernel to, say, allocate at address x but x is owned by something else, then DANGLING refuses. The same effectively applies to peeking. DANGLING is a MISC program like any other, so layering seek/peek/poke via it doesn't cost much. There are currently no rings and no multi-user mode, so all askers are the same entity and are effectively not gated, but once that's implemented, it will also filter access based on that, at which point the cost will slightly increase. And even then, there will not be much separation between a user and the machine. MISC is about responsibility, ownership, non-ambiguity, visibility, and control. You bought your hardware and want to use it, so that should be possible instead of platforms setting arbitrary limits on what you can and cannot do with your machine. "Security" also doesn't get an infinite pass. That's the same argument as "think of the children" or "but terrorists" and then everyone suffers.