Protection and Security
Protection is the mechanism that decides whether an access is allowed; security is the policy about what should be allowed and the assurance the mechanism cannot be bypassed. Confusing the two is why systems with correct permissions still get compromised.
The Subject, the Object, and the Monitor
Protection and security in operating system design starts from one observation: every access-control question has the same shape. A subject — a process acting for a user — attempts an operation on an object — a file, a socket, a memory region — and a reference monitor consults a rule to allow or deny it.
For that to mean anything the monitor must satisfy three properties. It must be non-bypassable: every access goes through it, with no side door. It must be tamper-proof: a subject cannot rewrite the rules that constrain it. And it must be verifiable: small and simple enough to be checked, which is why the trusted computing base is kept deliberately small.
Hardware supplies the non-bypassable part. The MMU checks permissions on every memory access, and privilege levels prevent user code from reprogramming it. Software alone cannot enforce this, because the enforcing code would run at the privilege it is trying to constrain.
OS protection and security name two different things — some courses write it os security and protection, or protection & security in operating system, but the split is the same. Protection is that mechanism. Security is the wider question — the policy about what should be permitted, plus authentication of who a subject is, plus confidence the mechanism has no holes. A machine with flawless permission checks and a shared password is perfectly protected and entirely insecure.
- Subject, object, and a rule checked by a reference monitor
- Non-bypassable, tamper-proof, verifiable
- The MMU and privilege levels make it non-bypassable
- Correct mechanism plus bad policy is still a breach
Access Control Models
The conceptual model is the access matrix: subjects as rows, objects as columns, permitted operations in the cells. Nobody stores it that way — it is enormous and almost entirely empty — so real systems slice it.
Storing it by column gives an access control list: each object carries the list of who may do what to it. This makes 'who can read this file' instant and 'what can this user reach' a full scan. Windows NTFS and POSIX ACLs work this way.
Storing it by row gives capabilities: each subject holds unforgeable tokens granting specific rights. This makes the subject's authority easy to enumerate and delegate, and revocation hard — you must find every copy of the token. Unix file descriptors are capabilities: once open, the descriptor grants access even if permissions change afterwards.
Classic Unix permissions are a compressed ACL — three classes (owner, group, other) times three bits (read, write, execute) fits in nine bits, which was a real consideration in 1970 and is still efficient. On a directory the bits mean something different worth remembering: execute permits traversal, and write permits creating and deleting entries, which is why write on a directory lets you remove a file you cannot even read.
| Access control list | Capability | |
|---|---|---|
| Stored with | The object | The subject |
| Answers quickly | Who may access this file? | What may this subject reach? |
| Answers slowly | What can this user reach? | Who can reach this object? |
| Revocation | Easy — edit the list | Hard — find every copy |
| Delegation | Awkward | Natural — pass the token |
| Example | POSIX permissions, NTFS ACLs | Unix file descriptors |
- The access matrix is the model; nobody stores it whole
- ACLs slice by object; capabilities slice by subject
- File descriptors are capabilities — access outlives permission changes
- Directory write allows deleting files you cannot read
Terms, operations, and practical uses
The model
- SubjectA process acting on behalf of a user.
- ObjectA file, socket, or memory region being accessed.
- Reference monitorMust be non-bypassable, tamper-proof, and verifiable.
- Protection vs securityMechanism versus policy plus assurance.
Access control
- Access matrixSubjects by objects; too large and sparse to store.
- ACLSliced by object — who may touch this file.
- CapabilitySliced by subject — an unforgeable token of authority.
- File descriptorA capability: access outlives a permission change.
Privilege
- Least privilegeBounds the damage of the bug you did not prevent.
- setuidRuns as the file owner; the classic escalation target.
- CapabilitiesCAP_NET_BIND_SERVICE instead of all of root.
- seccompRestricts a process to the system calls it actually needs.
Three subjects, one protected file
# A reference monitor. Every access is a subject, an object, and a rule --
# and the rule is checked before the operation, not after.
OWNER, GROUP, OTHER = 6, 4, 0 # mode 0640: rw- r-- ---
READ, WRITE = 4, 2
shadow = {"owner": "root", "group": "shadow", "mode": (OWNER, GROUP, OTHER)}
subjects = {
"root": {"uid": 0, "groups": ["root"]},
"alice": {"uid": 1000, "groups": ["shadow"]}, # group can read only
"www-data": {"uid": 33, "groups": ["www-data"]}, # neither owner nor group
}
def permitted(who, obj, want):
s = subjects[who]
if s["uid"] == 0:
return True, "" # root bypasses the check
bits = obj["mode"][0] if who == obj["owner"] else \
obj["mode"][1] if obj["group"] in s["groups"] else obj["mode"][2]
if bits & want:
return True, ""
return False, "no write" if want == WRITE else "no read"
results = []
for who, want in (("root", WRITE), ("alice", WRITE), ("www-data", READ)):
ok, why = permitted(who, shadow, want)
results.append(f"{who}:" + ("allow" if ok else f"deny({why})"))
# least privilege held: the web server could not read the password file
breached = permitted("www-data", shadow, READ)[0]
print(" ".join(results) + " | least privilege "
+ ("breached" if breached else "held"))#include <algorithm>
#include <iostream>
#include <map>
#include <string>
#include <vector>
using namespace std;
// A reference monitor. Every access is a subject, an object, and a rule --
// and the rule is checked before the operation, not after.
const int READ = 4, WRITE = 2;
struct Subject {
int uid;
vector<string> groups;
};
struct Object {
string owner, group;
int mode[3];
};
map<string, Subject> subjects = {
{
"root", {0, {"root"}}
},
{
"alice", {1000, {"shadow"}}
}, // group can read only
{
"www-data", {33, {"www-data"}}
}, // neither owner nor group
};
Object shadowFile {
"root", "shadow", {6, 4, 0}
}; // mode 0640
pair<bool, string> permitted(const string& who, const Object& obj, int want) {
Subject s = subjects[who];
if (s.uid == 0) return {true, ""}; // root bypasses the check
int bits;
if (who == obj.owner) bits = obj.mode[0];
else if (find(s.groups.begin(), s.groups.end(), obj.group) != s.groups.end())
bits = obj.mode[1];
else bits = obj.mode[2];
if (bits & want) return {true, ""};
return {false, want == WRITE ? "no write" : "no read"};
}
int main() {
vector<pair<string, int>> checks = {{"root", WRITE}, {"alice", WRITE}, {"www-data", READ}};
string out;
for (auto& c : checks) {
auto r = permitted(c.first, shadowFile, c.second);
out += c.first + ":" + (r.first ? "allow" : "deny(" + r.second + ")") + " ";
}
bool breached = permitted("www-data", shadowFile, READ).first;
cout << out << "| least privilege " << (breached ? "breached" : "held") << "\n";
}import java.util.List;
import java.util.Map;
class Main {
// A reference monitor. Every access is a subject, an object, and a rule --
// and the rule is checked before the operation, not after.
static final int READ = 4, WRITE = 2;
record Subject(int uid, List<String> groups) {
}
record Obj(String owner, String group, int[] mode) {
}
static Map<String, Subject> subjects = Map.of(
"root", new Subject(0, List.of("root")),
"alice", new Subject(1000, List.of("shadow")), // read only
"www-data", new Subject(33, List.of("www-data"))); // neither
static Obj shadow = new Obj("root", "shadow", new int[]{6, 4, 0}); // 0640
static String[] permitted(String who, Obj obj, int want) {
Subject s = subjects.get(who);
if (s.uid() == 0) return new String[]{"true", ""}; // root bypasses
int bits;
if (who.equals(obj.owner())) bits = obj.mode()[0];
else if (s.groups().contains(obj.group())) bits = obj.mode()[1];
else bits = obj.mode()[2];
if ((bits & want) != 0) return new String[]{"true", ""};
return new String[]{"false", want == WRITE ? "no write" : "no read"};
}
public static void main(String[] args) {
String[] who = {"root", "alice", "www-data"};
int[] want = {WRITE, WRITE, READ};
StringBuilder out = new StringBuilder();
for (int i = 0; i < who.length; i++) {
String[] r = permitted(who[i], shadow, want[i]);
out.append(who[i]).append(":")
.append(r[0].equals("true") ? "allow" : "deny(" + r[1] + ")").append(" ");
}
boolean breached = permitted("www-data", shadow, READ)[0].equals("true");
System.out.println(out + "| least privilege " + (breached ? "breached" : "held"));
}
}/etc/shadow, mode 0640, owner rootroot:allow alice:deny(no write) www-data:deny(no read) | least privilege heldRun the example step by step
Least Privilege, Authentication, and setuid
Most treatments of operating system protection and security converge on one principle. Least privilege says every component should hold the minimum authority needed for its job, and no more. Its value is not in preventing the breach but in bounding it: a compromised web server running as www-data cannot read /etc/shadow, so one bug becomes an incident rather than a catastrophe.
It shows up everywhere in practice — dropping root after binding a privileged port, running each service as its own user, containers with restricted capabilities, and seccomp filters that allow only the handful of system calls a process actually needs. File system security in OS terms is the same rule applied to files: mode bits and ACLs decide who reaches what.
Authentication establishes who a subject is, by something known (a password), held (a hardware key), or measured (a fingerprint). Passwords must be stored as salted hashes from a deliberately slow function — bcrypt, scrypt, Argon2 — because a fast hash lets an attacker with the file try billions of guesses per second. Multi-factor works because compromising two independent kinds is much harder than one.
setuid is where least privilege gets interesting. A setuid-root binary runs with root's authority regardless of who launched it, which is how an ordinary user changes their own password in a root-owned file. It is also the classic privilege-escalation route: any bug in a setuid binary is a bug with root's authority behind it, which is why their number is kept small and why modern systems prefer fine-grained capabilities — granting CAP_NET_BIND_SERVICE rather than all of root.
- Least privilege bounds the damage of an inevitable bug
- Store passwords with a deliberately slow salted hash
- setuid is necessary and is the classic escalation target
- Fine-grained capabilities beat all-or-nothing root
Isolation and Modern Defences
Process isolation via virtual memory is the OS's strongest guarantee, and attacks have historically worked by subverting it from inside a process. A buffer overflow writes past an array into the return address on the stack, redirecting execution to attacker-controlled code — the mechanism behind decades of remote exploits.
The layered defences all attack a different precondition. DEP/NX marks the stack and heap non-executable, so injected data cannot be run as code. Attackers answered with return-oriented programming, chaining existing code fragments instead of injecting new ones. ASLR randomises the load addresses of the stack, heap, and libraries so those fragments cannot be located reliably. Stack canaries place a known value before the return address and check it before returning, so a linear overflow is detected.
None is complete, and that is the point — defence in depth means an attacker must defeat all of them at once. Memory-safe languages remove the whole class instead, which is why Rust and Go are displacing C in new systems code.
Isolation now extends past the process. Containers use namespaces and cgroups to give a process its own view of the filesystem, network, and PIDs while sharing one kernel — a kernel bug therefore breaks out. Virtual machines isolate at the hardware layer with a much smaller shared surface. And Spectre and Meltdown showed the boundary can leak through timing side channels in speculative execution, without violating any permission check at all — a reminder that a correct mechanism can still be observable.
| Defence | Removes | Defeated by |
|---|---|---|
| Stack canary | Silent overwrite of the return address | Overwriting a pointer instead; leaking the canary |
| DEP / NX | Executing injected data | Return-oriented programming |
| ASLR | Knowing where code lives | An information leak revealing addresses |
| Memory-safe language | The whole bug class | (nothing in this class) |
- DEP stops injected code; ROP reuses existing code instead
- ASLR hides the addresses ROP needs; canaries catch linear overwrites
- Containers share a kernel; VMs share far less
- Side channels leak across a boundary nothing technically violated