Evi Nemeth pioneered the discipline of UNIX system administration. She taught and mentored computer science students at the University of Colorado Boulder, was visiting faculty member at Dartmouth College and UC San Diego, and helped bring Internet technology to the developing world through her work with the Internet Society and the United Nations.

Garth Snyder has worked at NeXT and Sun and holds a BS in Engineering from Swarthmore College and an MD and an MBA from the University of Rochester.

Trent R. Hein (@trenthein) is a serial entrepreneur who is passionate about practical cybersecurity and automation. Outside of technology, he loves hiking, skiing, fly fishing, camping, bluegrass, dogs, and the Oxford comma. Trent holds a BS in Computer Science from the University of Colorado.

Ben Whaley is the founder of WhaleTech, an independent consultancy. He was honored by Amazon as one of the first AWS Community Heroes. He obtained a B.S. in Computer Science from the University of Colorado at Boulder.

Dan Mackin’s (@dan_mackin) long-standing passion for technology inspired him to get a BS in Electrical and Computer Engineering from the University of Colorado at Boulder. He applies Linux and other open source technologies not only in his day job, but also to automation, monitoring, and weather metrics collection projects at home. Dan loves spending time with his wife and dog, skiing, movies, sailing, and backcountry touring.

Tribute to Evi xl

Preface xlii

Foreword xliv

Acknowledgments xlvi

Section One: Basic Administration 1

Chapter 1: Where to Start 3

Essential duties of a system administrator 4

Suggested background 7

Linux distributions 8

Example systems used in this book 9

Notation and typographical conventions 12

Units 13

Man pages and other on-line documentation 14

Other authoritative documentation 16

Other sources of information 18

Ways to find and install software 19

Where to host 25

Specialization and adjacent disciplines 26

Recommended reading28

Chapter 2: Booting and System Management Daemons 30

Boot process overview 30

System firmware 32

Boot loaders 35

GRUB: the GRand Unified Boot loader 35

The FreeBSD boot process 39

System management daemons .41

systemd in detail 44

FreeBSD init and startup scripts 57

Reboot and shutdown procedures 59

Stratagems for a nonbooting system 60

Chapter 3: Access Control and Rootly Powers 65

Standard UNIX access control 66

Management of the root account69

Extensions to the standard access control model 79

Modern access control 83

Recommended reading89

Chapter 4: Process Control 90

Components of a process 90

The life cycle of a process 93

ps: monitor processes 98

Interactive monitoring with top101

nice and renice: influence scheduling priority102

The /proc filesystem 104

strace and truss: trace signals and system calls 105

Runaway processes 107

Periodic processes109

Chapter 5: The Filesystem 120

Pathnames 122

Filesystem mounting and unmounting 122

Organization of the file tree125

File types 126

File attributes132

Access control lists 140

Chapter 6: Software Installation and Management 153

Operating system installation 154

Managing packages 162

Linux package management systems 164

High-level Linux package management systems 166

FreeBSD software management175

Software localization and configuration 178

Recommended reading 181

Chapter 7: Scripting and the Shell 182

Scripting philosophy 183

Shell basics 189

sh scripting 198

Regular expressions 209

Python programming 215

Ruby programming 223

Library and environment management for Python and Ruby 229

Revision control with Git 235

Recommended reading 241

Chapter 8: User Management 243

Account mechanics 244

The /etc/passwd file 245

The Linux /etc/shadow file250

FreeBSD's /etc/master.passwd and /[...] files 252

The /etc/group file 254

Manual steps for adding users 255

Scripts for adding users: useradd, adduser, and newusers 260

Safe removal of a user’s account and files264

User login lockout265

Risk reduction with PAM 266

Centralized account management 266

Chapter 9: Cloud Computing 270

The cloud in context 271

Cloud platform choices 273

Cloud service fundamentals 276

Clouds: VPS quick start by platform283

Cost control 291

Recommended Reading 293

Chapter 10: Logging 294

Log locations296

The systemd journal 299

Syslog 302

Kernel and boot-time logging 318

Management and rotation of log files 319

Management of logs at scale 321

Logging policies 323

Chapter 11: Drivers and the Kernel 325

Kernel chores for system administrators 326

Kernel version numbering 327

Devices and their drivers 328

Linux kernel configuration339

FreeBSD kernel configuration 344

Loadable kernel modules 346

Booting 348

Booting alternate kernels in the cloud 355

Kernel errors356

Recommended reading 359

Chapter 12: Printing 360

CUPS printing 361

CUPS server administration 365

Troubleshooting tips 369

Recommended reading 371

Section Two: Networking 373

Chapter 13: TCP/IP Networking 375

TCP/IP and its relationship to the Internet 375

Networking basics 378

Packet addressing384

IP addresses: the gory details 387

Routing 398

IPv4 ARP and IPv6 neighbor discovery 401

DHCP: the Dynamic Host Configuration Protocol402

Security issues 406

Basic network configuration 410

Linux networking417

FreeBSD networking 425

Network troubleshooting 428

Network monitoring 437

Firewalls and NAT 440

Cloud networking448

Recommended reading 457

Chapter 14: Physical Networking 459

Ethernet: the Swiss Army knife of networking460

Wireless: Ethernet for nomads 469

SDN: software-defined networking 473

Network testing and debugging474

Building wiring 475

Network design issues476

Management issues 478

Recommended vendors 479

Recommended reading 480

Chapter 15: IP Routing 481

Packet forwarding: a closer look482

Routing daemons and routing protocols 485

Protocols on parade 488

Routing protocol multicast coordination490

Routing strategy selection criteria 490

Routing daemons492

Cisco routers494

Recommended reading 496

Chapter 16: DNS: The Domain Name System 498

DNS architecture 499

DNS for lookups 500

The DNS namespace 502

How DNS works 503

The DNS database512

The BIND software 525

Split DNS and the view statement 541

BIND configuration examples 543

Zone file updating547

DNS security issues 551

BIND debugging 568

Recommended reading 576

Chapter 17: Single Sign-On 578

Core SSO elements 579

LDAP: “lightweight” directory services 580

Using directory services for login 586

Alternative approaches594

Recommended reading 595

Chapter 18: Electronic Mail 596

Mail system architecture 597

Anatomy of a mail message600

The SMTP protocol 603

Spam and malware 605

Message privacy and encryption 607

Mail aliases 608

Email configuration 612

sendmail 613

Exim 640

Postfix 658

Recommended reading 672

Chapter 19: Web Hosting 674

HTTP: the Hypertext Transfer Protocol 674

Web software basics 682

Web hosting in the cloud 694

Apache httpd696

NGINX 704

HAProxy 710

Recommended reading 714

Section Three: Storage 715

Chapter 20: Storage 717

I just want to add a disk! 718

Storage hardware 721

Storage hardware interfaces 730

Attachment and low-level management of drives 733

The software side of storage: peeling the onion 739

Disk partitioning 742

Logical volume management 747

RAID: redundant arrays of inexpensive disks 753

Filesystems 762

Traditional filesystems: UFS, ext4, and XFS 763

Next-generation filesystems: ZFS and Btrfs 772

ZFS: all your storage problems solved 773

Btrfs: “ZFS lite” for Linux 783

Data backup strategy 788

Recommended reading 790

Chapter 21: The Network File System 791

Meet network file services 791

The NFS approach794

Server-side NFS 801

Client-side NFS 807

Identity mapping for NFS version 4 810

nfsstat: dump NFS statistics 811

Dedicated NFS file servers 812

Automatic mounting 812

Recommended reading 818

Chapter 22: SMB 819

Samba: SMB server for UNIX 820

Installing and configuring Samba 821

Mounting SMB file shares 825

Browsing SMB file shares 826

Ensuring Samba security 826

Debugging Samba827

Recommended reading 829

Section Four: Operations 831

Chapter 23: Configuration Management 833

Configuration management in a nutshell834

Dangers of configuration management 834

Elements of configuration management 835

Popular CM systems compared 841

Introduction to Ansible 852

Introduction to Salt 871

Ansible and Salt compared 893

Best practices895

Recommended reading 899

Chapter 24: Virtualization 900

Virtual vernacular901

Virtualization with Linux 905

FreeBSD bhyve 910

VMware910

VirtualBox 911

Packer 911

Vagrant 913

Recommended reading 914

Chapter 25: Containers 915

Background and core concepts 916

Docker: the open source container engine 919

Containers in practice937

Container clustering and management 942

Recommended reading 948

Chapter 26: Continuous Integration and Delivery 949

CI/CD essentials 951

Pipelines 955

Jenkins: the open source automation server 961

CI/CD in practice964

Containers and CI/CD978

Recommended reading 980

Chapter 27: Security 981

Elements of security 983

How security is compromised 983

Basic security measures 987

Passwords and user accounts 992

Security power tools 996

Cryptography primer1005

SSH, the Secure SHell1016

Firewalls 1027

Virtual private networks (VPNs) 1030

Certifications and standards 1031

Sources of security information 1034

When your site has been attacked 1037

Recommended reading 1038

Chapter 28: Monitoring 1040

An overview of monitoring 1041

The monitoring culture 1044

The...