Commit 5a878a39 authored by Niklas Rieken's avatar Niklas Rieken
Browse files

added first few lines for linear programming

parent d8731e7e
\documentclass[12pt, a4paper]{article}
\usepackage[top=1cm,bottom=2cm,left=2.5cm, right=2.5cm]{geometry}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{mathtools}
\usepackage{array}
\usepackage{graphicx}
\usepackage{float}
\usepackage{tikz}
\usetikzlibrary{calc}
\usepackage{listings}
\usepackage{enumitem}
\usepackage{wrapfig}
%\usepackage{bbold}
\usepackage{bbm}
\usepackage[justification=centering]{caption}
\usetikzlibrary{arrows, automata, graphs, shapes, petri, decorations.pathmorphing}
\parindent = 0pt
\frenchspacing
\let\emptyset\varnothing
\let\epsilon\varepsilon
\let\rho\varrho
\let\theta\vartheta
% define environments
\theoremstyle{definition}
\newtheorem{definition}{Definition}
\newtheorem*{remark*}{Remark}
\theoremstyle{plain}
\newtheorem{lemma}[definition]{Lemma}
\newtheorem{theorem}[definition]{Theorem}
\renewcommand{\labelenumi}{(\roman{enumi})}
\renewcommand{\labelenumi}{(\roman{enumi})}
\author{Niklas Rieken}
\title{Linear Programs}
\begin{document}
\maketitle
A \emph{linear program} (\emph{LP}) consists of \emph{variables} $x = (x_1, \ldots, x_n)$, an \emph{objective function} $z(x) = c^Tx$ with $c \in \mathbb{Q}^n$ that can either be maximized or minimized, and \emph{constraints} $a^T x \sim b$ with $a \in \mathbb{Q}^n, b \in \mathbb{Q}, \sim \in \{\leq, =, \geq\}$. An LP is in \emph{canonical form} if it is written as
\begin{align*}
\text{maximize } & c_1 x_1 + c_2 x_2 + \ldots + c_n x_n\\
\text{subject to } & a_{11} x_1 + a_{12} x_2 + \ldots + a_{1n} x_n &&\leq b_1\\
& a_{21} x_1 + a_{22} x_2 + \ldots + a_{2n} x_n &&\leq b_2\\
& \ldots & \\
& a_{m1} x_1 + a_{m2} x_2 + \ldots + a_{mn} x_n &&\leq b_m\\
& x_1, x_2, \ldots, x_n &&\geq 0
\end{align*}
or shorter
\begin{align*}
\text{maximize } & \sum_{j=1}^n c_j x_j\\
\text{subject to } & \sum_{j=1}^n a_{ij} x_j \leq b_i & i \in \{1, \ldots, m\}\\
& x_j \geq 0 & j \in \{1, \ldots, n\}
\end{align*}
or even shorter
\begin{align*}
\text{maximize } & c^T x\\
\text{subject to } & Ax \leq b\\
& x \geq 0.
\end{align*}
We call the $x_j$ ($j \in \{1, \ldots, n\}$) \emph{decision variables}, $c^T$ the \emph{objective function coefficients}, $A$ the \emph{coefficient matrix}, and $b$ the \emph{right hand side} of the LP. Moreover, we write $a_i^T$ for the coefficients of the $i$-th constraint. Note that every LP has an ''equivalent`` LP in canonical form:
\begin{itemize}
\item $\text{minimze } c^T x = \text{maximize } -c^T x$,
\item $a_i^T x \geq b_i$ iff $-a_i^T x \leq -b_i$,
\item $a_i^T x = b_i$ iff $a_i^T x \leq b_i$ and $a_i^T x \geq b_i$,
\item $x_j \leq 0$ iff $-x_j \geq 0$,
\item $x_j$ unbounded: Replace $x_j = x_j^+ - x_j^-$ with $x_j^+, x_j^- \geq 0$.
\end{itemize}
Every constraint (from an LP in canonical form) defines a \emph{halfspace} in $\mathbb{R}^n$ and its separating hyperplane has $a_i^T$ as its normal vector.
\begin{figure}[H]
\centering
\begin{tikzpicture}
\draw[-] (0, 0) -- (3, -2);
\draw[->,red,thick] (2, -1.333) -- (2.3077, -.8715);
\draw[white,fill=gray!20] (0, 0) -- (-.5, 0) -- (-.5, -2.5) -- (3, -2.5) -- (3, -2);
\node (feas) at (.7, -1.7) {\scriptsize{''feasible``}};
\node (infeas) at (2.3, -.3) {\scriptsize{''infeasible``}};
\end{tikzpicture}
\end{figure}
We call the set of points that satisfy all constraints \emph{feasible region} $P \coloneqq \{x \in \mathbb{R}^n \mid Ax \leq b, x \geq 0\}$. $P$ is a \emph{polyhedron}, i.e. the intersection of finitely many halfspaces. If $P$ is bounded, it is called a \emph{polytope}.
\begin{figure}[H]
\centering
\begin{tikzpicture}
\begin{scope}
\path[fill=gray!20] (0, 0) coordinate(p1) -- ++(35:2.5) coordinate(p2) -- ++(-45:2.5) coordinate(p3) -- ++(-120:3.5) coordinate(p4) -- ++(150:3) coordinate(p5);
\node at (barycentric cs:p1=1,p2=1,p3=1,p4=1,p5=1) {$P$};
\foreach \X [count=\Y] in {2,...,6} {
\ifnum\X=6
\path (p\Y) -- (p1) coordinate[pos=-0.2](a\Y) coordinate[pos=1.2](a1) coordinate[pos=0.5](m1);
\draw (a\Y) -- (a1);
\draw[->,red,thick] (m1) -- ($(m1)!1.2cm!90:(p1)$) node[pos=1.2]{$a_{\Y}^T$};
\else
\path (p\Y) -- (p\X) coordinate[pos=-0.2](a\Y) coordinate[pos=1.2](a\X) coordinate[pos=0.5](m\X);
\draw (a\Y) -- (a\X);
\draw[->,red,thick] (m\X) -- ($(m\X)!1.2cm!90:(p\X)$) node[pos=1.25]{$a_{\Y}^T$};
\fi
}
\node (bounded) at (1.5, -5) {\scriptsize{''bounded``}};
\end{scope}
\begin{scope}[shift={(6, 1)}]
\path[fill=gray!20] (1, -1) coordinate(q1) -- (3, -4) coordinate(q2) -- (5, -4) coordinate(q3) -- (6, -1) coordinate(q4);
\node at (barycentric cs:q1=1,q2=1,q3=1,q4=1) {$P$};
\foreach \X [count=\Y] in {2, 3, 4} {
\path (q\Y) -- (q\X) coordinate[pos=-.2](b\Y) coordinate[pos=1.2](b\X) coordinate[pos=.5](n\X);
\draw (b\Y) -- (b\X);
\draw[->,red,thick] (n\X) -- ($(n\X)!1.2cm!-90:(q\X)$) node[pos=1.25]{$a_{\Y}^T$};
}
\node (unbounded) at (4, -6) {\scriptsize{''unbounded``}};
\end{scope}
\end{tikzpicture}
\end{figure}
If the constraints contradict themselves (for instance, $x + y \leq 1$, $x \geq 2$, and $x, y \geq 0$), then $P = \emptyset$, a corresponsing LP is called \emph{infeasible}. Constraints might also be \emph{redundant}, i.e. a constraint can be omitted without increasing the polyhedron (for example, $x + y \geq 0$ and $x, y \geq 0$).
\begin{figure}[H]
\centering
\begin{tikzpicture}
\begin{scope}
\path (0, 0) coordinate(p1) -- (-3, -2) coordinate(p2) -- (1, -3) coordinate(p3);
\path (p1) -- (p2) coordinate[pos=-.2](a1) coordinate[pos=1.2](a2) coordinate[pos=.3](m2);
\draw (a1) -- (a2);
\draw[->,red,thick] (m2) -- ($(m2)!.8cm!90:(p2)$);
\path (p2) -- (p3) coordinate[pos=-.2](a2) coordinate[pos=1.2](a3) coordinate[pos=.5](m3);
\draw (a2) -- (a3);
\draw[->,red,thick] (m3) -- ($(m3)!.8cm!90:(p3)$);
\path (p3) -- (p1) coordinate[pos=-.2](a3) coordinate[pos=1.2](a1) coordinate[pos=.4](m1);
\draw (a3) -- (a1);
\draw[->,red,thick] (m1) -- ($(m1)!.8cm!90:(p1)$);
\node (infeasible) at (-.5, -3.5) {\scriptsize{''infeasible``}};
\end{scope}
\begin{scope}[shift={(4.5, 0)}]
\path[fill=gray!20] (4, 1) coordinate(q1) -- (0, 0) coordinate(q2) -- (0, -2) coordinate(q3) -- (4, -3) coordinate(q4);
\node at (barycentric cs:q1=1,q2=1,q3=1,q4=1) {$P$};
\foreach \X [count=\Y] in {2, 3, 4} {
\path (q\Y) -- (q\X) coordinate[pos=-.3](b\Y) coordinate[pos=1.3](b\X) coordinate[pos=.5](n\X);
\draw (b\Y) -- (b\X);
\draw[->,red,thick] (n\X) -- ($(n\X)!.8cm!-90:(q\X)$);
}
\draw (-.5, .5) -- (-.5, -2.5);
\draw[->,red,thick] (-.5, -1.2) -- (-1.2, -1.2);
\node (redundant) at (1.5, -3.5) {\scriptsize{''redundant``}};
\end{scope}
\end{tikzpicture}
\end{figure}
Polyhedra are \emph{convex}, i.e. they satisfy Jensen's inequality: For all $x, y \in P$ and all $\theta \in [0, 1]$, $(1-\theta)x + \theta y \in P$.
\begin{proof}
Let $x, y \in P$ and $\theta \in [0, 1]$, then
$$
(\underbrace{1-\theta}_{\geq 0})\underbrace{x}_{\geq 0} + \underbrace{\theta}_{\geq 0} \underbrace{y}_{\geq 0} \geq 0
$$
and moreover,
\begin{align*}
A((1-\theta)x + \theta y) &= (1-\theta)\underbrace{Ax}_{\leq b} + \theta \underbrace{Ay}_{\leq b}\\
&\leq (1-\theta)b + \theta b\\
&= b.\qedhere
\end{align*}
\end{proof}
The \emph{dimension} $\dim P$ is the dimension of the smallest affine subspace containg $P$. If $P \subseteq \mathbb{R}^n$ with $\dim P = n$, then $P$ is called \emph{full dimensional}.
For a given $x \in P$, a constraint $a_i^T x \leq b_i$ is called \emph{active} (or \emph{binding}) if $a_i^T x = b_i$.
A \emph{face} with respect to $H \subseteq \{1, \ldots, m\}$ is
$$
F \coloneqq \{x \in P \mid a_i^T x \leq b_i \text{ active in } x, i \in H\}.
$$
\begin{figure}
\centering
\begin{tikzpicture}
%TODO scopes
\end{tikzpicture}
\end{figure}
Every face itself is again a polyhedron. A point $x \in P$ is called \emph{feasible solution}, $x^\ast \in P$ such that $c^T x^\ast \geq c^T x$ for all $x \in P$ is called \emph{optimal solution} (provided that the objective function should be maximized), $c^T x^\ast$ is called \emph{optimum}.
\begin{theorem}
For every LP exactly one of the following hold:
\begin{enumerate}
\item The LP is infeasible, i.e. $P = \emptyset$.
\item The optimum is unbounded, i.e. for all $M > 0$, there exists $x \in P$ with $c^T x \geq M$.
\item There exists a finite optimal solution.
\end{enumerate}
If the last case is true, then the optimal sollution is assumed at a vertex of $P$.
\end{theorem}
\begin{figure}
\centering
\begin{tikzpicture}
%TODO scopes
\end{tikzpicture}
\end{figure}
\begin{lemma}
An $n$-dimensional polyhedron given by $m$ constraints has at most $\binom{m}{n}$ vertices, i.e. finitely many.
\end{lemma}
Thus, of all (usually infinitely many) feasible points, only finitely many are relevant.
Idea for the Simplex algorithm: Move from vertex to vertex such that the objective value only increases (decreases, respectively, if objective is to minimize).
\begin{figure}
\centering
\begin{tikzpicture}
%TODO
\end{tikzpicture}
\end{figure}
We introduce \emph{slack variables} to fill gaps between constraints and corresponding right hand side.
$$
y \coloneqq b - Ax, \quad y \geq 0.
$$
This yields to \emph{standard form} for LPs:
\begin{align*}
\text{minimize } & c^T x & \text{minimize } & c^T x\\
\text{subject to } & Ax + y = b & \text{subject to } & Ax = b\\
& x, y \geq 0 & & x \geq 0
\end{align*}
where the LP on the right is obtained by setting $x_{n+i} = y_i$ for $i \in \{1, \ldots, m\}$ and we extend $c$ and $A$ in the obvious way.
The transformation from canonical form to standard form preserves dimension and vertices of the polyhedron.
\begin{figure}
\centering
\begin{tikzpicture}
%TODO scopes
\end{tikzpicture}
\end{figure}
Note that the slack variable $y_i = 0$ iff the $i$-th constraint is active. % complementary slackness?
We can also interpret $x$ as slack varaiable, just note that $x_j = 0$ iff $x_j \geq 0$ active.
Let $A_1, A_2, \ldots, A_n, A_{n+1}, \ldots, A_{n+m}$ be the columns of $A$ (the latter $m$ columns are unit vectors for slack varaiables). For $J \subseteq (1, \ldots, n+m)$, let $A_J$ denote the matrix consisting of columns $A_j$ with $j \in J$, e.g. for
$$
A = \left(
\begin{matrix}
3 & 7 & 0 & -1 & 1 & 0\\
-1 & -1 & -2 & 2 & 0 & 1
\end{matrix}
\right),
\quad
J = (5, 2)
\implies
A_J = \left(
\begin{matrix}
1 & 7\\
0 & -1
\end{matrix}
\right).
$$
A \emph{basis} $B = (B_1, \ldots, B_m) \subseteq (1, \ldots, n-m)$ is a subset of $m$ column indices such that the corresponding columns are linearly independent. $N = (1, \ldots, n+m) \setminus B$ is called \emph{non basis}. Variables $x_j$ with $j \in B$ are called \emph{basic variables}, and \emph{non-basic variables} if $i \in N$.
A vector $x \in \mathbb{Q}^{n+m}$ is a \emph{basic solution} to $Ax = b$, $x \geq 0$ if there is a basis $B$ such that
\begin{itemize}
\item $A_B x_B = b$ (\emph{uniqueness}),
\item $x_N = 0$ (\emph{at boundary}, \emph{vertex}).
\end{itemize}
If additionally $x_B \geq 0$ holds $x$ is called \emph{feasible basic solution}.
\begin{theorem}
Every feasible basic solution corresponds to exactly one vertex of $P$.
\end{theorem}
Basic solution are also called \emph{extreme point solutions}.
We need $\dim P$ constraints to describe a vertex. The non-basic variables correspond to the active constraints.
\begin{figure}[h]
\centering
\begin{tikzpicture}
%TODO
\end{tikzpicture}
\end{figure}
Note that the basis in one vertex is not necessarily unique. We call those vertices \emph{degnerate}.
\begin{figure}[h]
\centering
\begin{tikzpicture}
\end{tikzpicture}
\end{figure}
Given a standard form LP with $Ax = b$, $x \geq 0$ and basis $B$:
\begin{align*}
& A_B x_B + A_N x_N &&= b\\
\iff & x_B &&= \underbrace{A_B^{-1} b}_{\eqqcolon \bar{b}} - \underbrace{A_B^{-1} A_N}_{\eqqcolon \bar{A}_N} \underbrace{x_N}_{= 0}.
\end{align*}
Using the objective function $z(x) = c^T x$:
\begin{align*}
z(x) &= c^T x\\
&= c_B^T x_B + c_N^T x_N\\
&= c_B^T(A_B^{-1} b - A_B^{-1} A_N x_N) + c_N^T x_N\\
&= c_B^T A_B^{-1} - c_B^T A_B^{-1} A_N x_N + c_N^T x_N\\
&= \underbrace{c_B^T A_B^{-1} b}_{\eqqcolon \bar{z}} + (\underbrace{c_N^T - c_B^T A_B^{-1} A_N}_{\eqqcolon \bar{c}_N^T~\emph{''reduced cost``}}) \underbrace{x_N}_{= 0}.
\end{align*}
Optimality condition: Basis $B$ (and the corresponding vertex) optimal if reduced cost $\bar{c}_N^T \leq 0$. Intuitively, no non-basic variable can be increased without decreasing the value of $z$.
Otherwise we can find a non-basic variable that can ''improve`` the objective function. This means we deactivate a constraint (increasing its slack) and move to another vertex:
\begin{itemize}
\item Initially: Vertex given by $B, N$.
\item If there is a non-basic variable $x_s$, $s \in N$, with $\bar{c}_s > 0$, it is beneficial to increase $x_s$ (currently $x_s = 0$).
\item Since $x_B = A_B^{-1} b - A_B^{-1} A_N x_N$, the values of basic-variables decrease if $A_B^{-1} A_s > 0$.
\item The maximum value for $x_s$ is determined by ''the first`` basic variable which becomes $0$.
\item If this never happens, i.e. $A_B^{-1} A_s \leq 0$, then $x_s$ can be arbitrarily increased; in this case the objective value is unbounded.
\end{itemize}
The first basis: If $b \geq 0$, all slack variables are a feasible basis, i.e. $B = (n+1, \ldots, n+m)$, and hence, $A_B = A_B^{-1} = \mathbbm{1}_m$ and $\bar{A}_N = A_N$, $\bar{b} = b$. The first basic solution is then $x_{n+i} = b_i$ for $i \in \{1, \ldots, m\}$ and $x_1 = \ldots = x_n = 0$ (i.e. the origin). Since $c_B^T = 0$ (all slack), we have $\bar{c}_N^T = C_N^T$ and $\bar{z} = 0$. For calculation by hand, we can store all coefficients in a \emph{dictionary} (or \emph{tableau}):
\begin{table}[h]
\centering
\begin{tabular}{c|ccc|}
\cline{2-4}
\ & $\bar{c}_N^T$ & $0$ & $\bar{z}$\\
\cline{2-4}
$x_B$ & $\bar{A}_N$ & $\mathbbm{1}_m$ & $\bar{b}$\\
\cline{2-4}
\ & $x_N$ & $x_B$ & \
\end{tabular}
\end{table}
\end{document}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment