Appendix A   Select Matrix Algebra

 
Here are some select matrix algebra related to the dissertation. For detailed results, please refer to Appendix M (Searle et al., 1992), Chapters 15 and 16 (Harville, 1997) and Chapters 7 and 8 (Schott, 1997). Some unconventional results not found in the literature but used in the dissertation are given in Lemmas.

 
A.1   Special Matrices and Operators

A.1.1   Matrix Element-Wise Notations

These compact notations are from Appendix M.3 (Searle et al., 1992). It is especially helpful in algebraic simplifications when typical elements are easily specified, but naming each matrix is not needed. Under most circumstances of incomplete data, the direct element-wise operation is usually desired, and these notations just serve that purpose very well.

Familiar notation for a matrix A of order p × q is A = {a_ij} or (a_ij), where a_ij is the element that is in the ith row and jth column of A for i = 1, ..., p and j = 1, ..., q. We abbreviate this to

A = {_m a_ij} _, = {_m a_ij}_i,j = {_m a_ij},

using m to indicate that the elements inside the braces are being arrayed as a matrix; and sufficient detail of subscripts follows the braces as is necessary, depending on context. For diagonal matrix, indicate with letter d as follows,

A =

= {d ai} = {d ai}.

For row and column vectors, use r and c respectively. Let u = (u₁, u₂, ..., u_p), then

u = {_r u_i} = {_r u_i},   and   u' = {_c u_i} = {_c u_i}.

Extension to partitioned matrices is straightforward, for example,

And it can also used in a nested manner, for example,

y_i = {_c y_ij},   and   y = {_c y_i} = {_c {_c y_ij} }.

 
A.1.2   The Direct Product

For matrices A = (a_ij)_m×n and B_r×s, the direct product (also called the Kronecker product) of A and B is a matrix of order mr × ns defined as

In particular, I_m I_n = I_mn. For more than two matrices,

A B C = A (B C) = (A B) C,

Some useful properties of direct products follow.

1. For a being a scalar: a A = A a = aA.

2. For x and y being vectors: x' y = y x' =yx'.

3. In transposing, (A B)'= A' B'.

4. For A and B being nonsingular square matrices, (A B)^-1 = A^-1 B^-1.

5. Given conformability of regular matrix multiplication,

(A B)(X Y) = (AX) (BY).

(A.1)

6. For partitioned matrices, [ A₁   A₂ ] B = [ A₁ B   A₂ B ].

7. Rank and trace, r_{A B} = r_A r_B and tr(A B) = tr(A) tr(B).

8. For A and B being square matrices, |A_m×m B_n×n| = |A| ⁿ |B| ^m.

 
A.1.3   The Vec Operator

The matrix operator vec(X) creates a column vector from the columns of matrix A by locating them one under the other. For a matrix X = {_r x_j} of order p×q, define vec(X) = {_c x_j} of order pq×1. For example,

A very useful property of operator vec is with matrix trace,

tr(AB) = (vec A')' vec B.

(A.2)

Relationship with Direct Product

For any matrix A_n×m,

vec A = (Im A) vec Im.

(A.3)

In general, the following relationship is very useful,

vec(ABC) = (C' A) vec(B).

(A.4)

Apply it to the inverse of a nonsingular matrix A^-1 = A^-1AA^-1, then

vec(A-1) = (A'-1 A-1) vec(A) = (A' A)-1 vec(A).

(A.5)

The following useful results can be verified easily.

Lemma A.1   For vectors a_n×1 and b_m×1,

a b = vec(ba') = (In b)a.

(A.6)

Proof.   Directly comparing elements in both sides gives the first part; and the second part is due to equation (A.1) as a b = (I_n b)(a 1) = (I_n b) a.

Lemma A.2   For vectors a and b,

vec(a' b) = a b.

(A.7)

Proof.   This is due to the property (2) of direct product and the prior Lemma.

Special Matrix P_m,k,n

This matrix is used in Chapter 4, which moves vec in and out of a matrix. Define the constant matrix P_m,k,n as

(A.8)

and then

(A.9)

This matrix is very useful in the context of vec due to the following properties.

Lemma A.3   For matrices A_m×n and B_k×n with same number of columns,

(A.10)

This property can be verified directly using column vectors of A and B. Due to (A.3), the following equation is more general. For matrices A_m×r and B_k×r with same number of columns, and any matrix C_n×s,

(A.11)

Or for matrices A_r×m and B_r×k with same number of rows and any matrix C_s×n,

(A.12)

 
A.2   Matrix Differentiation

Some classical useful results regarding to vector and matrix differentiation are briefly listed in this section. For detailed results, please refer to Appendix M.7 (Searle et al., 1992), Chapter 15 (Harville, 1997) and Chapter 8 (Schott, 1997). Some results not found in the literature but used in the dissertation are given in Lemmas.

A.2.1   Scalars and Vectors

Differentiation with Scalars

Let λ be a scalar and vector x_q×1. Define

If scalar y is a function of matrix X, and matrix A = (a_ij) is a function y, define

For a being unrelated to x,

(a'x) = (x'a) = a,     and     (a'x) = (x'a) = a'.

Differentiation with Vectors

Suppose vector y_p×1 is a differentiable function of vector x_q×1. Define

(A.13)

a matrix of order q×p. Its transpose is a matrix of order p×q as

In particular, . For matrices A and B being not functions of x,

(Ax) = A     and     (x'B) = B.

(A.14)

The Chain Rule

For vector x,y,z, the chain rule is,

(A.15)

Matrix With Respect To All Elements

For vector a_n×1 not function of matrix X_n×m,

= Im a     and     = a Im.

(A.16)

For general matrix X_n×m,

(A.17)

A.2.2   Products and Quadratic Forms

Products

Suppose p×1 vectors u and v are differentiable functions of vector x,

(A.18)

If A and B are matrices and t is a scalar, it is easy to verify

(A.19)

Lemma A.4   Suppose matrix A and vector b are differentiable functions of vector t_p×1, then

(A.20)

Proof.   Since and by (A.19), easy to have the first equation. The second simply re-arranges the first and reflects the fact

Direct Product

For vectors x_n×1 and y_m×1,

(A.21)

Quadratic Forms

(x'Ax) = Ax + A'x	for asymmetric A,
(x'Ax) = 2Ax	for symmetric A.

A.2.3   Inverses

For scalar t and nonsingular matrix X, XX^-1 = I gives

(A.22)

By equation (A.4), it is a special case of the following where t is a vector,

vec(X-1) = - (X'-1 X-1) vec(X).

(A.23)

In particular by equation (A.17),

= - X'-1 X-1.

(A.24)

Assume that A and B are constant matrices and matrix X is a function of vector t,

vec(AX-1B) = - ((X-1B)' (AX-1)) vec(X).

(A.25)

This is due to equations (A.4) and (A.23). In particular by equation (A.17),

= - (X^-1B)' (AX^-1).

For vector a we have the following parallel results,

vec(X-1a)	= - ((X-1a)' X-1) vec(X).	(A.26)
	= - (a' X'-1) X-1.

A.2.4   Determinants and Traces

Determinants

Suppose the elements of square matrix A are not functionally related and denote the cofactor of a_ij in |A| by |A_ij|, we have

= |Aij|	for asymmetric A,
= (2 - δij) |Aij|	for symmetric A,

where δ_ij is the Kronecker delta, δ_ij = 0 for i ≠ j and δ_ij =1 for i=j.

Suppose elements of matrix A are functions of scalar t, then

For vector t, since tr(AB) = (vec A')' vec B, it is easy to verify

and

(A.27)

Traces

Suppose matrix A and matrix X are not functionally related. Then

A.2.5   The Second Derivative

Suppose f (x) is a differentiable scalar function of vector x_q×1. The second derivative is a symmetric matrix

(A.28)

Lemma A.5   Assume vector is a differentiable function of vector θ_p×1, and l() is a scalar differentiable function of , then

(A.29)

Proof.   This is due to the chain rule (A.15) and the Lemma A.4 with setting θ = t, A= and b=. Finally equation (A.4) is applied to vec().