The Representation of $\mathrm{SU}(2)$

Let $\mathcal{H}_j$ be the space of polynomial functions on $\mathbb{C}^2$ that are homogemeous of degree $2j$. An element in $\mathcal{H}_j$  is a polynomial in complex variables $x$ and $y$ that is a linear combination of polynomials $x^py^q$ where $p+q=2j$. $\mathcal{H}_j$ has dimension $2j+1$ since it has a basis given by
$$x^{2j},x^{2j-1}y,x^{2j-2}y^2,\cdots,y^{2j}.$$
For any $g\in\mathrm{SU}(2)$, let $U_j(g)$ be the linear transformation of $\mathcal{H}_j$ given by
$$U_j(g)f(v)=f(g^{-1}v)$$
for $f\in\mathcal{H}_j$ and $v\in\mathbb{C}^2$. Then $U_j$ is a representation: $U_j(I)$ is the identity. For any $g,h\in\mathrm{SU}(2)$,
\begin{align*}
U_j(g)U_j(h)f(v)&=U_(h)f(g^{-1}v)\\
&=f(h^{-1}g^{-1}v)\\
&=f((gh)^{-1}v)\\
&=U_j(gh)f(v)
\end{align*}
for $f\in\mathcal{H}_j$ and $v\in\mathbb{C}^2$.

Physicists call $U_j$ spin-$j$ representation. Since $2j+1$ has to be a positive integer, we have spin-0 representation, spin-$\frac{1}{2}$ representation, spin-1 representation, etc. It is interesting to see the correspondence between spin-$j$ representation and particles. The only known spin-0 particle is Higgs-boson, the so-called God particle, which is responsible for giving masses to bosons. The Higgs-boson appears to have been discovered recently by the LHC (Large Hadron Collider) at CERN. Spin-$\frac{1}{2}$ particles are fermions which include all quarks and leptons. Spin-1 particles are gauge bosons (force-carrying particles) such as photons, W bosons, Z bosons, gluons. Curiously there are currently no spin-$\frac{3}{2}$ particles predicted in particle physics. The hypothetical gravitons are believed to be spin-$2$ particles.

Proposition. The spin-0 representation of $\mathrm{SU}(2)$ is equivalent to the trivial representation in which every element of the group acts on $\mathbb{C}$ as the identity.

Proposition. The spin-$\frac{1}{2}$ representation of $\mathrm{SU}(2)$ is equivalent to the fundamental representation in which every element $g\in\mathrm{SU}(2)$ acts on $\mathbb{C}^2$ by matrix multiplication.

Note that the $U_j$ are irreducible and that they are all of the irreducible representations.

$\mathbb{R}^3$ can be identified with the set of $2\times 2$ Hermitian matricies of the form
\begin{align*}
X&=\begin{pmatrix}
z & x-iy\\
x+iy & -z
\end{pmatrix}\\
&=z\begin{pmatrix}
1 & 0\\
0 & -1
\end{pmatrix}+x\begin{pmatrix}
0 & 1\\
1 & 0
\end{pmatrix}+y\begin{pmatrix}
0 & -i\\
i & 0
\end{pmatrix}\\
&=x\sigma_1+y\sigma_2+z\sigma_3,
\end{align*}
where $\sigma_1,\sigma_2,\sigma_3$ are called the Pauli spin matrices in physics. Define an inner product $\langle\ ,\ \rangle$ on the Hermitian matrices by
$$\langle X, Y\rangle=\frac{1}{2}\mathrm{tr}(XY).$$
In particular,
$$|X|^2=\frac{1}{2}\mathrm{tr}(X^2)=-\det X.$$
With this inner product, the identification is an isometry. $\mathrm{SU}(2)$ acts on $\mathbb{R}^3$ via the representation
$$\rho:\mathrm{SU}(2)\longrightarrow\mathrm{GL}(3,\mathbb{R})$$
defined by
$$\rho(U)X=UXU^{-1}$$
for $U\in\mathrm{SU}(2)$ and $X\in\mathbb{R}^3$. It turns out that $X\longmapsto UXU^{-1}$ is an orientation preserving isometry of $\mathbb{R}^3$, so
$$\rho:\mathrm{SU}(2)\longrightarrow\mathrm{SO}(3).$$
Since both $U$ and $-U$ result the same isometry, the representation $\rho:\mathrm{SU}(2)\longrightarrow\mathrm{SO}(3)$ is a $2:1$ map. Since $\mathrm{SU}(2)=S^3$ is simply-connected, $\rho$ is a universal convering map and that we have
$$\mathrm{SU}(2)/\mathbb{Z}_2=\mathrm{SO}(3).$$
The quotient group $\mathrm{SU}(2)/\mathbb{Z}_2$ is denoted by $\mathrm{PSU}(2)$ and called the projective special unitary group.

The double cover i.e. $2:1$ cover of $\mathrm{SO}(n)$ is called the spin group and is denoted by $\mathrm{Spin}(n)$. For $n>2$, $\mathrm{Spin}(n)$ is simply-connected so it is the universal cover of $\mathrm{SO}(n)$. Some examples of spin groups are
\begin{align*}
\mathrm{Spin}(1)&=\mathrm{O}(1)=\mathbb{Z}_2=\{\pm I\}\\
\mathrm{Spin}(2)&=\mathrm{U}(1)=\mathrm{SO}(2)\\
\mathrm{Spin}(3)&=\mathrm{SU}(2)\\
\mathrm{Spin}(4)&=\mathrm{SU}(2)\times\mathrm{SU}(2)
\end{align*}
Note that $\mathrm{SO}(3)\subset\mathrm{GL}(3,\mathbb{R})\subset\mathrm{GL}(3,\mathbb{C})$, so for any $g\in\mathrm{SU}(2)$, $\rho(g):\mathbb{C}^3\longrightarrow\mathbb{C}^3$. Hence, $\rho$ is in fact equivalent to the spin-1 representation of $\mathrm{SU}(2)$.

In quantum mechanics, unitary representation is particularly important. Let $\mathcal{H}$ be the Hilbert space of states derived from a quantum mechanical system. Let $\rho$ be a representation of a Lie group $G$ on $\mathcal{H}$ i.e. $\rho:G\longrightarrow\mathrm{GL}(\mathcal{H})$. $\rho$ is called a unitary representation if
$$\langle\rho(g)\psi,\rho(g)\phi\rangle=\langle\psi,\phi\rangle$$
for all $g\in G$ and $\psi,\phi\in\mathcal{H}$. Intuitively each $\rho(g)$ may be understood as a rotation. For example, say $G=\mathrm{SO}(3)$. First rotating the particle by some amount $h\in\mathrm{SO}(3)$ and then rotating it by some amount $g\in\mathrm{SO}(3)$ should have the same effect as rotating it by the amount $gh\in\mathrm{SO}(3)$. That is $\rho(g)\rho(h)=\rho(gh)$. This tells why we need a representation in quantum mechanics. In quantum mechanics, the inner product $\langle\ ,\ \rangle$ measures probability. For instance if a particle is in the state $\psi$, then $\langle\psi,\phi\rangle$ is the probability of finding the particle in the state $\phi$. Rotating a particle amounts to a change of coordinates and the state of a particle should not depend on a change of coordinates. Hence, in quantum mechanics we require representation to be unitary.

Lastly I would like to mention the difference between bosons and fermions in terms of representation. Let $f\in\mathcal{H}_j$, the spin-$j$ representation space. Then
$$U_j(-I)f(v)=f(-v)=(-1)^{2j}f(v)$$
since $f$ is a homogeneous polynomial of degree $2j$. This implies that
$$U_j(-I)=(-1)^{2j}.$$
Hence, $U_j$ maps both $I$ and $-I$ to the identity if $j$ is an integer, while it does not if $j$ is a half-integer.

References:

[1] John Baez, Javier P. Muniain, Gauge Fields, Knots and Gravity, World Scientific 1994

[2] Brian C. Hall, Lie Groups, Lie Algebras, and Representations, An Elementary Introduction, Springer-Verlag 2003

Irreducible Representations of $\mathrm{U}(1)$

A representation $\rho$ of a group $G$ on a vector space $V$ always has the subspace $\{0\}$ and $V$ itself as invariant subspaces. Here a subspace $W\subset V$ is invariant means that $\rho(g)W\subset W$ for every $g\in G$. if $\rho$ has no other invariant subspaces, we say that it is irreducible.

Theorem. If $G$ is compact, every representation of $G$ is equivalent to a direct sum of irreducible representations.

This theorem is important for physicists since most Lie groups that are important in physics are compact. The theorem also says that if $G$ is compact, irreducible representations are the building blocks of other representations.

Example. For each $n\in\mathbb{Z}$, define $\rho_n:\mathrm{U}(1)\longrightarrow\mathrm{GL}(1,\mathbb{C})$ by
$$\rho_n(e^{i\theta})v=e^{i n\theta}v.$$
Then each $\rho_n$ is irreducible since $\mathbb{C}$ has no nontrivial vector subspaces. What is really important about this example is that any complex 1-dimensional representation is equivalent to $\rho_n$ for some $n\in\mathbb{Z}$. (Prove this!)

Schur’s Lemma. Let $\rho: G\longrightarrow\mathrm{GL}(V)$ be an irreducible complex representation and let $\phi: V\longrightarrow V$ an interwining map of $V$ with itself (i.e. $\phi(\rho(g)v)=\rho(g)(\phi(v))$ for all $g\in G$, $v\in V$). Then $\phi=\lambda I$ for some $\lambda\in\mathbb{C}$.

Suppose the group $G$ is abelian and $g\in G$. Then
\begin{align*}
\rho(g)(\rho(g’)v)&=\rho(gg’)v\\
&=\rho(g’g)v\\
&=\rho(g’)(\rho(g)v)
\end{align*}
for all $g’\in G$ , $v\in V$. Since $\rho(g)$ is an interwining map of $V$ with itself, $\rho(g)$ is a scalar multiple of $I$ by Schur’s Lemma. So every subspace of $V$ is invariant and hence $\rho$ is 1-dimensional. This means that any irreducible representation of $\mathrm{U}(1)$ is equivalent to one the $\rho_n$. Since $\mathrm{U}(1)$ is compact, any finite dim representation of $\mathrm{U}(1)$ is given as a direct sum of the $\rho_n$.

In quantum mechanics the electric charge of a particle is assumed to be a (integer) multiple of a certain unit charge $q$ i.e. charge is quantized. (This was indeed the case as confirmed by experiments.) In terms of representation, this means that a particle with charge $nq$ transforms according to $\rho_n$ of $\mathrm{U}(1)$. If we move a particle of charge $nq$ around a loop $\gamma$ in spacetime, its wave function is multiplied by a certain phase
$$e^{-\frac{i}{\hbar}nq\oint_\gamma A}\in\mathrm{U}(1)$$
where $A$ is the vector potential or more generally a connection as
$$\rho_n\left(e^{-\frac{i}{\hbar}q\oint_\gamma A}\right)v=e^{-\frac{i}{\hbar}nq\oint_\gamma A}v.$$

Proposition. The tensor product $\rho_n\otimes\rho_m$ is equivalent to $\rho_{n+m}$.

I will leave the proof of this proposition as an exercise. This proposition has an interesting physical implication. If we have two particles corresponding to two different representations of a group, a bound state corresponds to the tensor product of the two representations. The lectric charge of such a bound state is the sum of the charges of the constituents.

References:

[1] John Baez, Javier P. Muniain, Gauge Fields, Knots and Gravity, World Scientific 1994

[2] Brian C. Hall, Lie Groups, Lie Algebras, and Representations, An Elementary Introduction, Springer-Verlag 2003

Lie Group Actions and Lie Group Representations

$\mathrm{SO}(3)$ acts on $\mathbb{R}^3$ meaning that each element of $\mathrm{SO}(3)$ defines a linear transformation (rotation) of $\mathbb{R}^3$. So we can say that $\mathrm{SO}(3)$ describes the rotational symmetry of $\mathbb{R}^3$.

Definition. A group $G$ is said to act on a vector space $V$ if there exists a map $\rho:G\longrightarrow\mathrm{GL}(V)$ such that
$$\rho(gh)v=\rho(g)\rho(h)v,\ \forall v\in V.$$
We also say that $\rho$ is a representation of $G$ on $V$.

Lie groups are closely related to the fundamental forces of our universe, electromagnetism, weak force, strong force, and gravity. Different Lie groups give different equations called Yang-Mills equations, which describe different forces. Those Lie groups involved with fundamental forces are called symmetry groups (usually by mathematicians) or gauge groups (usually by physicists, in mathematics gauge group means
the group of gauge transformations.). The Lie group $\mathrm{U}(1)=\mathrm{SO}(2)$ is the gauge group for electromagnetism. Yang-Mills equations with $\mathrm{U}(1)$ are simply Maxwell’s equations. (They are linear equations.) $\mathrm{SU}(3)$ is the gauge group for strong force and $\mathrm{SU}(2)\times\mathrm{U}(1)$ is the gauge group for electroweak force. The groups such as $\mathrm{SU}(2)\times\mathrm{U}(1)$ are called direct products. Recall the following propositon from abstract algebra:

Proposition. Given groups $G$ and $H$, the cartesian product $G\times H$ becomes a group with product
$$(g,h)\cdot (g’,h’)=(gg’,hh’),$$
identity $e=(e_G,e_H)$, and inverse $(g,h)^{-1}=(g^{-1},h^{-1})$.

The group $G\times H$ is called the direct product or direct sum of $G$ and $H$. (We use the notation $G\oplus H$ for direct sum.) $G\times H$ is abelian if and only if both $G$ and $H$ are abelian. If $G$ and $H$ are Lie groups, so is $G\times H$.

The gauge group of the entire standard model (physical theory that attempts to unify electromagnetism, strong and weak force) is $\mathrm{SU}(3)\times\mathrm{SU}(2)\times\mathrm{U}(1)$. Physicists have been looking for a nicer group which has $\mathrm{SU}(3)\times\mathrm{SU}(2)\times\mathrm{U}(1)$ as its subgroup. The simplest choice is $\mathrm{SU}(5)$. The $\mathrm{SU}(5)$ model was proposed by Sheldon Glashow and it predicts the decay of protons. However there has been no sign of proton decay after many years of experiments and most physicists do not believe protons would decay. The standard model does not include gravity. Physicists do not know which Lie group is involved with gravity. Possible candidates are $\mathrm{SO}^+(3,1)$ or $\mathrm{SL}(2,\mathbb{C})$. So far all the attempts to come up with a gauge theory of gravity have failed.

There are many examples that hint us that symmetry plays a crucial role in particle theory. Representation is closely related to symmetry and there is a wonderful connection between the charge of a particle and the representation of gauge group, namely the charge of a particle amounts to a choice of a representation for the gauge group. This will be discussed in a separate post. Let us study a bit more about representations.

Equivalent Representations

Two representations
$$\rho: G\longrightarrow\mathrm{GL}(V),\ \rho’: G\longrightarrow\mathrm{GL}(V)$$
are equivalent if there exists an isomorphism $T:V\longrightarrow V$ with
$$\rho(g)T=T\rho'(g),\ \forall g\in G.$$

Getting New Representation from Old Ones

Direct Sum

Let $G$ be a (Lie) group and $\rho: G\longrightarrow\mathrm{GL}(V)$, $\rho’:G\longrightarrow\mathrm{GL}(V’)$ representations. The direct sum $\rho\oplus\rho’:G\longrightarrow\mathrm{GL}(V\oplus V’)$ of $\rho$ and $\rho’$ is a representation defined by $$(\rho\oplus\rho’)(v,v’)=(\rho(g)v,\rho'(g)v’),\ \forall v\in V,v’\in V’.$$

Tensor Product

Let $V$ and $V’$ be finite dimensional vector spaces. Pick a basis $\{e_i\}$ for $V$ and a basis $\{e’_j\}$ for $V’$. Then the tensor product $V\otimes V’$ is the vector space whose basis is given by all expressions of the form $e_i\otimes e’_j$. If $v=v^ie_i\in V$ and $v’=v’^je’_j\in V’$, then
$$v\otimes v’=v^iv’^je_i\otimes e’_j.$$
So clearly we obtain,
$$\dim V\otimes V’=(\dim V)(\dim V’).$$
While this definition of tensor product is easy to understand, it does depend on the choice of bases. There is a general definition of tensor product with universal mapping property.

Theorem. Let $V$, $W$ be finite vector spaces over a field $K$. Then there exists a finite dimensional space $T$ over $K$ and a bilinear map
\begin{align*}
V\times W&\longrightarrow T\\
(v,w)&\longmapsto v\otimes w
\end{align*}
satisfying the following properties

T1. If $U$ is a vector space over $K$ and $g:V\times W\longrightarrow U$ is a bilinear map, then there exists uniquely a linear map $g_\ast:T\longrightarrow U$ such that for every $(v,w)\in V\times W$,
$$g(v,w)=g_\ast (v\otimes w).$$
In other words, the following diagram commutes.
$$\begin{array}{ccc}
& & T\\
& \nearrow &\downarrow\\
V\times W & \longrightarrow & U
\end{array}$$

T2. If $\{v_1,\cdots,v_n\}$ is a basis of $V$ and $\{w_1,\cdots,w_n\}$ is a basis of $W$, then $v_i\otimes w_j$, $i=1,\cdots,n$, $j=1,\cdots,m$ form a basis of $T$

For the proof of this theorem, see for instance [3]. The vector space $T$ is denoted by $V\otimes W$ and called the tensor product of $V$ and $W$.

Let $\rho:G\longrightarrow\mathrm{GL}(V)$ and $\rho’: G\longrightarrow\mathrm{GL}(V’)$. The tensor product $\rho\otimes\rho’: G\longrightarrow\mathrm{GL}(V\otimes V’)$ is a representation defined by
$$(\rho\otimes\rho’)(g)(v\otimes v’)=\rho(g)v\otimes\rho'(g)v’.$$

Subrepresentations

Let $\rho$ be a representation of a group $G$ on the vector space $V$. Suppose that $V’$ is an invariant space $V$ i.e. $\rho(g)(V’)\subset V’$ for all $g\in G$. Define a representation $\rho’: G\longrightarrow\mathrm{GL}(V’)$ by
$$\rho'(g)v=\rho(g)v,\ \forall v\in V’.$$
$\rho’$ is called a subrepresentation of $\rho$.

References:

[1] John Baez, Javier P. Muniain, Gauge Fields, Knots and Gravity, World Scientific 1994

[2] Brian C. Hall, Lie Groups, Lie Algebras, and Representations, An Elementary Introduction, Springer-Verlag 2003

[3] Serge Lang, Linear Algebra, 2nd Edition, Addison-Wesley Publishing Co. 1972

More Examples of Lie Groups: 3-Sphere as a Lie group, The 3-Dimensional Heisenberg Group

3-Sphere $S^3$ as a Lie Group

Consider 4 elements $1,i,j,k$ satisfying the following relation
\begin{align*}
1^2&=1,\ i^2=j^2=k^2=-1,\\
1i&=i1=i,\ 1j=j1=j,\ 1k=k1=k,\\
ij&=-ji=k,\ jk=-kj=i,\ ki=-ik=j.
\end{align*}
Let $\mathbb{H}$ be the algebra spanned by $1,i,j,k$ over $\mathbb{R}$
$$\mathbb{H}=\{a1+bi+cj+dk:a,b,c,d\in\mathbb{R}\}.$$
Then $\mathbb{H}\cong\mathbb{R}^4$ as a vector space over $\mathbb{R}$. Define $||q||$ of $q=a1+bi+cj+dk\in\mathbb{H}$ by
$$||q||^2:=q\bar q=a^2+b^2+c^2+d^2,$$
where $\bar q=a1-bi-cj-dk$.

Set $S^3=\{q\in\mathbb{H}: ||q||=1\}$. Then $S^3$ is a unit sphere in $\mathbb{R}^4$. $S^3$ is closed under the multiplication of $\mathbb{H}$. So, $S^3$ is a group. In fact, it is a Lie group.

The 3-Dimensional Heisenberg Groups

Set
$$G=\left\{(x,y,z):=\begin{pmatrix}
1 & y & z\\
0 & 1 & x\\
0 & 0 & 1
\end{pmatrix}: x,y,z\in\mathbb{R}\right\}.$$
Define a multiplication in $G$ by
\begin{align*}
(x,y,z)\cdot (a,b,c)&=\begin{pmatrix}
1 & y & z\\
0 & 1 & x\\
0 & 0 & 1
\end{pmatrix}\begin{pmatrix}
1 & b & c\\
0 & 1 & a\\
0 & 0 & 1
\end{pmatrix}\\
&=\begin{pmatrix}
1 & y+b & z+ya+c\\
0 & 1 & x+a\\
0 & 0 & 1
\end{pmatrix}\\
&=(x+a,y+b,z+ya+c)\in G.
\end{align*}
The identity element is $(0,0,0)$ and that $(x,y,z)^{-1}=(-x,-y,xy-z)$. $G$ is a Lie subgroup of $\mathrm{GL}(3,\mathbb{R})$.

Let $\gamma:(-\epsilon,\epsilon)\longrightarrow G$ be a regular curve in $G$ such that $\gamma(0)=(0,0,0)=\begin{pmatrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 &1 \end{pmatrix}$.  Then                                        \begin{align*}\dot\gamma(t)&=\frac{\partial\gamma}{\partial x}\frac{dx}{dt}+\frac{\partial\gamma}{\partial y}\frac{dy}{dt}+\frac{\partial\gamma}{\partial z}\frac{dz}{dt}\\
&=\begin{pmatrix}0 & 0 & 0\\0 & 0 & 1\\0 & 0&0\end{pmatrix}\frac{dx}{dt}+\begin{pmatrix}0&1&0\\0&0&0\\0&0&0\end{pmatrix}\frac{dy}{dt}+\begin{pmatrix}0&0&1\\0&0&0\\0&0&0\end{pmatrix}\frac{dz}{dt}.\end{align*}
Let
$$\mathfrak{e}_1:=\begin{pmatrix}0&0&0\\0&0&1\\0&0&0\end{pmatrix},\mathfrak{e}_2:=\begin{pmatrix}0 & 1 & 0\\ 0 & 0 & 0\\0 & 0 & 0\end{pmatrix},\mathfrak{e}_3:=\begin{pmatrix}0 & 0 & 1\\0 & 0 & 0\\0 & 0 & 0\end{pmatrix}.$$
Then $\mathfrak{e}_1,\mathfrak{e}_2,\mathfrak{e}_3$ generate the Lie algebra $\mathfrak{g}$ of $G$. The Lie algebra $\mathfrak{g}$ is called Heisenberg algebra. Note the commutation relation
$$[\mathfrak{e}_1,\mathfrak{e}_3]=[\mathfrak{e}_2,\mathfrak{e}_3]=0,\ [\mathfrak{e}_1,\mathfrak{e}_2]=-\mathfrak{e}_3$$
which resembles the commutation relation $[\hat p,\hat x]=-i\hbar$ in quantum mechanics.

Lie Brackets (for $n\times n$ Matrices)

Definition. Given any two $n\times n$ matrices $A$ and $B$ define
$$[A,B]:=AB-BA.$$
$[\ ,\ ]$ is a bilinear operator on the Lie algebra $\mathfrak{gl}(n)$ of the general linear group $\mathrm{GL}(n)$ and is called Lie bracket. Note that with $[\ ,\ ]$, $\mathfrak{gl}(n)$ becomes an algebra over $\mathbb{R}$ or $\mathbb{C}$. So Lie algebra is actually an algebra. Lie bracket plays an important role in physics, especially in quantum mechanics. Physicists call it commutator.

Proposition. For any $n\times n$ matrices $A$, $B$ and $C$, the following identity holds
$$[A,[B,C]]+[B,[C,A]]+[C,[A,B]]=0.$$
This identity is called the Lie identity or Jacobi identity.

Lie algebra can be studied from purely algebraic point of view without knowing its relationship with Lie group. The algebraists’ definition of Lie algebra is:

Definition. A Lie algebra $\mathfrak{g}$ is an algebra over $\mathbb{R}$ or $\mathbb{C}$ with a bilinear vector product $[\ ,\ ]:\mathfrak{g}\times\mathfrak{g}\longrightarrow\mathfrak{g}$ satisfying the Jacobi identity.

In case you are interested, there are a couple of good books on algebraic approach of Lie algebra. They are

Hans Samelson, Notes on Lie Algebra, 2nd Edition, Springer 1990

J. E. Humphreys, Introduction to Lie Algebras and Representation Theory, Springer 1973

Proposition. $\mathfrak{so}(n)$ is a Lie algebra with $[\ ,\ ]$.

Proof. In here we have shown that $\mathfrak{so}(n)$ is the set of all $n\times n$ skew-symmetric matrices. It suffices to show that $\mathfrak{so}(n)$ is closed under $[\ ,\ ]$.

For any $A,B\in\mathfrak{so}(n)$,
\begin{align*}
{}^t[A,B]&={}^t(AB-BA)\\
&={}^t(AB)-{}^t(BA)\\
&={}^tB{}^tA-{}^tA{}^tB\\
&=BA-AB\\
&=-[A,B].
\end{align*}
Hence $[A,B]\in\mathfrak{so}(n)$.