Linux System Programming Techniques & Concepts

For Developers - Build Linux Programmable Libraries, Makefiles, Memory Management, Compilation & Linking, C Programming

What you'll learn
Learn Advance Programming Concepts in C/C++
Develop and Integrate C/C++ Libraries
Automate Build Process using Makefile
Use of Function Pointers as Callbacks
Static and Dynamic Libraries and Linking
Compilation Process of a GCC Compiler
Organize the code into Header and Source files
Understanding Memory allocations, Heap Memory Management, Memory Leaks
Stack Memory, Stack overflow, Stack Corruption, Stack Registers
Paging Concepts and Page Tables
Concept of TLVs
Requirements
Basic knowledge of Programming in C/C++ is required
Zeal and enthusiasm to learn and sharpen the skills
Good to have basic OS knowledge but not mandatory
Description
There are already innumerable courses/tutorials on the internet/Udemy which teach almost every aspect of C programming language. All those courses sound the same, talk the same and deliver more or less the same knowledge. How is this course different from others? The intention of this course is to make you ready for System programming Technical interviews, interview level - from Beginner to Intermediate. This course is for (future) Developers, not for testers or System Administrators.I choose to create this course to fill the gap between novice/beginner and intermediate/Advanced Programmers. This course assumes that you are at least above average in programming (in any programming language, but preferable in C/C++) - know memory allocations, all kinds of loops, function calls, pointers etc.  In this course, I won't teach C/C++ (there are already many courses on it online) but would teach programming techniques and low-level details regarding how C program works behind the scenes - All topics are very important from an interview point of view. My Target is to grow this course into Linux/C Bible.Yet, I always feel, you need to outsmart your fellow colleagues in this era of stiff competition, and therefore, I tried this attempt to present you the wisdom and knowledge which is of utmost importance for a programmer. I have seen though students could write good C/C++ programs, yet they lack the clarity on how one should write better organized, Manageable, extensible, and programmable codes in the form of libraries. For example, Students are too good at competitive programming, yet do not know how to write a simple Makefile, the reason being, Academics do not teach and students do not care to learn. In this course, I will cover the topics related to creating Linux System Libraries (release 1) with Advanced language-agnostic Programming Concepts (release 2) which can be well applied if you happen to become a programmer in other languages tomorrow. The Concepts like MultiThreading, Thread-Synchronization, Socket Programming, IPCs, etc demand a separate course on each topic altogether. Currently, I have a course on IPC, pls check it out. These topics are out of scope for this course. The Course shall be delivered into two Releases :      Release 1 Building and Managing a Library:  This covers the basics regarding how one should create and organize his code as a Library.     Release 2 Memory Management Concepts:  This covers Advance concepts on Linux Memory Management specifically. Please check the Table of Contents for more info.Why you should *NOT* do this course?1. Please Excuse this course if you are the ultimate beginner in C programming !!2. There is no point of doing this course if you don't like hitting the keyboard, and are lazy enough to watch lecture VDOs only3. If you want everything cooked and served on your plate. Course Pre-requisite :Above Beginners Level in C/C++ programmingGood to have basic OS knowledgeZeal to excel and CodeNo Third-Party librariesWhatever logic you implement, you need to implement it from scratch. Like all my other courses, this course does not suggest taking the help of any third-party library to get the jobs done. The use of external libraries completely defeats the purpose of the course. Warning: This course has auto system-generated subtitles which may not be perfect. Please disable subtitles at your convenience.Curriculum :**********************************************Release 1 Building and Managing a Library**********************************************Section 1 : Introduction to Libraries What is LibraryRelationship between Library and ApplicationEx : Doubly linked list as a LibraryQuick Compilation StepsSummarySection 2 : Header FilesRelationship between Source and header filesText Substitution MethodText Substitution Method - Example and DemoSection 3 : Preprocessing DirectivesProblem of Duplicate inclusion of Hdr filesPre-processing DirectivesSolution to Duplicate inclusion of Hdr filesSection 4 : Correct way of Using Structures and FunctionsStructures - Define and Use thumb RuleFunctions - Declare and Use thumb ruleThe problem of Recursive DependencyA solution to Recursive DependencySection 5 : Creation of Static and Dynamic LibrariesResuming with Doubly Linked List LibraryQuick Creation of Static and Dynamic LibrariesLinking with Static LibraryLinking with Dynamic LibrarySection 7: Understanding four stages of Compilation Four stages of C/C++ CompilationHow Dynamic Library works ?Dynamic Linking : Linking with Dynamic LibraryComparison - Static Vs Dynamic LinkingSection 8 : Building using a Makefile What are Makefiles and why do we need itFunctions of MakefileMakefile Dependency treeSteps of Writing a MakefileAssignment on MakefileSection 9 : Run-time Programmable libraries What are Programmable Libraries?Steps to Program the librariesRegistering of the callbacks with Librarieskey_match callbackcomparison_fn callbackDelegation of Application-specific operations to LibrariesSection 10 : Writing Iterators using MacrosWhat are Iterative Macros ?Why we need Iterative Macros ?How to Write Iterative Macros - For Trees and Linked ListsExercisesSection 11 : Glue Based Libraries and Data structuresWhat are the Glue Based Libraries?Introducing Glthreads - A Glued LinkedListGlthreads Vs Traditional Linked ListStructure field offsetGLThread Operations Code WalkGLThread Benefits  Section 12 : Bit Level ProgrammingLogical OperatorsImplementing BIT manipulating C macrosUsing Enums as BitsBit Pattern MatchingBitMaps************************************************Release 2 Memory Management Concepts************************************************Section 13 : Memory Layout of Linux ProcessVirtual Memory Basics  Memory Layout of Linux Process  Example: Memory Layout of Linux Process  Exercise on size command  Section 14 : Stack Memory ManagementStack Memory Basics and Contents  Stack-Overflow and Prevention  Stack Memory Corruption  Common Cpu RegistersProcedure Call Mechanism - Step by StepPurpose of Base Pointer register (ebp)  Procedure Return Mechanism - Step by StepLab session  Section 15 : Heap Memory ManagementIntroduction and GoalsHow Malloc Works Top of Heap Memory region - break pointerHeap Memory Mgmt Sys Calls - brk and sbrkMeta and Data BlocksHow free() worksBlock SplittingBlock MergingMemory Illness - Problem of FragmentationSection 16 : Concept of PagingIntroduction to PagingByte Addressable Memory32 bit and 64 bit Machine ArchitectureAddress Bus and Data bus         Physical Vs Virtual AddressPhysical Memory Frames        Virtual Address CompositionPage TablePaging In ActionShared Physical MemorySection 17 : Multilevel PagingSection 18 : Demand PagingSection 19 : Memory Management for Multi-threaded ProcessThe intention of this course is to make you ready for System programming Technical interviews from beginners to upto 8-9 yrs of experience.Q. What are the frequently asked questions by interviewers in a technical round when someone writes C/C++/System Programming language on their resume?Answer : If i am interviewer, what questions i would ask depends on his no of years of experience in C.1–3 yrs of experience — I would have asked:Double pointersdesign a Macro to return the size of the structureTwo Dimensional Arrays, passing and returning arrays from a fnDifferent stages of C program compilationhow fork() worksWhat are various ways to debug memory corruptions.various IPCsHeap and Stack memory-based Question4–6 yrs of experience - I would have asked:How memory is allocated by the OSInternal and external fragmentation, what can be done to avoid itSystem calls, strace()Trade-of of one IPC over othervarious ways to communicate with kernel and comparisonData (De)Serialization in CRPC in Ccallbacks advanced applicationtypedef Vs #defineGeneric programming in C using macrosThread SynchronizationHeap and Stack memory-based Question7+ yrs of experience - I would have askedHave you designed any system module to solve any problemDesign thread library 0 what functionalities would you incorporate in and how?What are Dos and Dont’s for writing a robust and flexible libraryHow to write generic code in CVarious ways to implement timers in C, and comparison of approachesHow do Interrupts work ?IPCs and comparisonHow would you convert a C code to C++ and vice versaHow to write a tool to detect memory leaks Or garbage collectionDesign your own memory allocation tool. Why would you write your own memory allocation scheme?When to go for Multi-process design over Multi-threaded design and vice versaHow ValGrind tool worksIn production code, would you favor recursive but simple logic, Or Nonrecursive but complex logic, and why?If you analyze the pattern,Candidate with 1–3 yrs of experience, I would choose to ask more of a direct and straightforward Questions.Candidate with 4–6 yrs of experience, I would choose to ask more advanced technical C Question plus some comparison of approaches based QuestionsCandidate with 7+ yrs of experience, I would choose to ask more of a design and Analysis based Question.Note that, since the Question pertains to C and System Programming, so I have not mentioned Data structures and Algorithms.

Overview

Section 1: Introduction

Lecture 1 Introduction

Lecture 2 Join Telegram Group

Lecture 3 Setting up Linux Development Environment

Lecture 4 Begin - Doubly Linked List as a Library

Lecture 5 Quick Compilation Steps

Lecture 6 Summary

Section 2: Understanding Header Files

Lecture 7 What are Header Files and their Purpose ?

Lecture 8 Relationship between Source and Header Files

Lecture 9 Text Substitution

Lecture 10 Text Substitution Example

Lecture 11 Text Substitution Demonstration

Section 3: Preprocessing Directives

Lecture 12 Problem Of Duplicate Inclusion of Header Files

Lecture 13 What are Pre-processing Directives ?

Lecture 14 Solution to Duplicate Inclusion of Hdr files using Preprocessing Directives

Section 4: Correct Way of Using Structures and Functions

Lecture 15 Structure Definitions and Use

Lecture 16 Function Declaration and Use

Lecture 17 Recursive Dependency

Lecture 18 Solution to Recursive Dependency

Lecture 19 Summary

Section 5: Quick Creation of Static and Dynamic Libraries

Lecture 20 Resuming with Doubly Linked List Library

Lecture 21 Static and Dynamic Libraries - Quick Creation

Lecture 22 Linking with Static Library

Lecture 23 Linking with Dynamic Library

Lecture 24 Summary

Section 6: Four Stages of Compilation Process

Lecture 25 4 Stages of Compilation Process

Lecture 26 Stage 1 of 4 - Preprocessing stage

Lecture 27 Stage 2 of 4 - Compilation Stage

Lecture 28 Stage 3 of 4 - Assembler Stage

Lecture 29 Stage 4 of 4 - Linking Stage

Section 7: Build Project using Makefiles

Lecture 30 Introducing Makefile

Lecture 31 Makefile Analogy - Dependency Tree

Lecture 32 Makefile Assignment - part1

Lecture 33 Makefile Assignment - part2

Lecture 34 Final Makefile

Section 8: Programmable Libraries - Generics

Lecture 35 Introduction

Lecture 36 Revisiting DLL

Lecture 37 Problem Statement

Lecture 38 Solution - Responsibility Delegation

Lecture 39 Using Programmable DLL Library - Code Walk

Lecture 40 Search Callback Summary

Lecture 41 Comparison Callback

Lecture 42 Comparison Callback Demo

Lecture 43 Summary

Section 9: Iterative Macros

Lecture 44 Getting Started

Lecture 45 For loop Semantics

Lecture 46 Iterative Macros For LinkedList

Lecture 47 Iterative Macros for Binary Trees

Lecture 48 Tree Iterators Prerequisites

Lecture 49 Tree Iterators Implementation

Lecture 50 Summary

Section 10: Glue Based Data structures

Lecture 51 Introduction

Lecture 52 Glthreads Definition

Lecture 53 Difference - Traditional Vs Glue Based DLL

Lecture 54 Structure Field Offset

Lecture 55 GLNode Insertion

Lecture 56 GLThread Iteration

Lecture 57 GLNode Removal

Lecture 58 Code Walk - Demo

Lecture 59 Traditional List Problems

Lecture 60 GLThread Benefits

Section 11: Opaque Pointers

Lecture 61 Introduction

Lecture 62 Typical Library Design

Lecture 63 Problem Statement

Lecture 64 Solution Strategy

Lecture 65 Conclusion

Section 12: Bit Programming

Lecture 66 Introduction to Bit Programming

Lecture 67 XOR Operator

Lecture 68 Using Bits as Flags

Lecture 69 Using Enums as Bits

Lecture 70 Bit Pattern Matching

Lecture 71 Bit Pattern Matching Steps

Lecture 72 BitMaps

Lecture 73 BitMap APIs and Implementation

Section 13: Machine Endianness

Lecture 74 Machine Endianness

Lecture 75 Program to find Machine Endianness

Section 14: TLV Based Communication

Lecture 76 TLVs Introduction

Lecture 77 Why we need TLVs ?

Lecture 78 Why we need TLVs - Another scenario

Lecture 79 Understanding TLVs

Lecture 80 TLV Addressing the problem of Heterogeneity

Lecture 81 TLV Addressing the problem of Software Upgrade

Lecture 82 Data Structure - STREAMS

Lecture 83 TLV De-Serialization using STREAMS

Section 15: Working with Linux Posix Timers

Lecture 84 Agenda and Prerequisites

Lecture 85 Timer Relevance

Lecture 86 Timer Types

Lecture 87 Posix APIs for Timers

Lecture 88 Timer Design

Lecture 89 Timer Creation Steps

Lecture 90 Timer Implementation and Demo

Section 16: Memory Layout of Linux Process

Lecture 91 Agenda

Lecture 92 Virtual Memory Memory Basics

Lecture 93 Memory Layout of Linux Process

Lecture 94 Example : Memory Layout of Linux Process

Lecture 95 Exercise on size command

Section 17: Stack Memory Management

Lecture 96 Stack Memory Basics

Lecture 97 Stack Memory Contents

Lecture 98 Stack-Overflow and Prevention

Lecture 99 Stack Memory Corruption

Lecture 100 Procedure Call and Return - Getting Started

Lecture 101 Common Cpu Registers

Lecture 102 Procedure Call Mechanism

Lecture 103 Purpose of Base Pointer register (ebp)

Lecture 104 Formalizing Procedure Call Algorithm

Lecture 105 Procedure Return - Goals

Lecture 106 Procedure Return Explained - Step by Step

Lecture 107 Formalizing Procedure Return Algorithm

Section 18: Heap Memory Management

Lecture 108 Heap Memory Management - Goals and Introduction

Lecture 109 Malloc() - Quick Revision

Lecture 110 Break pointer

Lecture 111 brk and sbrk system calls

Lecture 112 Malloc Version 1.0

Lecture 113 Problem Statement

Lecture 114 Heap Memory Management Requirement

Lecture 115 MetaBlock and DataBlock

Lecture 116 Allocations and Deallocations

Lecture 117 Block Splitting

Lecture 118 Block Merging

Lecture 119 Problem of Fragmentation

Section 19: Concepts of Paging

Lecture 120 Introduction

Lecture 121 Byte Addressable Memory

Lecture 122 What is 32 bit Or 64 bit system ?

Lecture 123 Bus System

Lecture 124 CPU Generates Virtual Addresses

Lecture 125 Data bus and Address Bus

Lecture 126 Mapping of Virtual Address to Physical Address

Lecture 127 Physical Pages and Frames

Lecture 128 Page Swapping

Lecture 129 Virtual Memory Pages

Lecture 130 1:1 Mapping between Physical and Virtual Page

Lecture 131 Virtual Address Composition

Lecture 132 Page Table

Lecture 133 Paging in Action

Lecture 134 Multiple Process Scenario

Lecture 135 Resolve External Fragmentation

Lecture 136 Page Allocation to a Process - Part1

Lecture 137 Page Allocation to a Process - Part2

Lecture 138 Shared Physical Pages

Lecture 139 Page tables Problems

Lecture 140 Page Table Problem 1 - Large Page Table Size Matters

Lecture 141 Page Table Problem 2 - Need for Contiguous Main Memory

Lecture 142 Page Table Problem 3 - Page Table Hollowness

Section 20: Multi Level Paging

Lecture 143 Introduction

Lecture 144 Multi Level Paging In Action - Part1

Lecture 145 Multi Level Paging In Action - Part2

Section 21: Paging on Demand

Lecture 146 Problem Statement

Lecture 147 Demand Paging Steps

Lecture 148 Effective Access Time

Section 22: Memory Management for Multi-threaded Processes

Lecture 149 Introduction

Lecture 150 Virtual Memory Management

Lecture 151 Page Table Management

Lecture 152 Thread Termination

Section 23: Bonus

Lecture 153 Bonus Lecture - Time to get some Candies !!

Students who have basic knowledge of C/C++ programming,Warning : Not for Absolute beginners in Programming,Who want to take their programming Skills from Beginner to Professional Level,Final year students looking to crack the Interview Questions on OS, C programming