Definition. Given any two $n\times n$ matrices $A$ and $B$ define
$$[A,B]:=AB-BA.$$
$[\ ,\ ]$ is a bilinear operator on the Lie algebra $\mathfrak{gl}(n)$ of the general linear group $\mathrm{GL}(n)$ and is called Lie bracket. Note that with $[\ ,\ ]$, $\mathfrak{gl}(n)$ becomes an algebra over $\mathbb{R}$ or $\mathbb{C}$. So Lie algebra is actually an algebra. Lie bracket plays an important role in physics, especially in quantum mechanics. Physicists call it commutator.

Proposition. For any $n\times n$ matrices $A$, $B$ and $C$, the following identity holds
$$[A,[B,C]]+[B,[C,A]]+[C,[A,B]]=0.$$
This identity is called the Lie identity or Jacobi identity.

Lie algebra can be studied from purely algebraic point of view without knowing its relationship with Lie group. The algebraists’ definition of Lie algebra is:

Definition. A Lie algebra $\mathfrak{g}$ is an algebra over $\mathbb{R}$ or $\mathbb{C}$ with a bilinear vector product $[\ ,\ ]:\mathfrak{g}\times\mathfrak{g}\longrightarrow\mathfrak{g}$ satisfying the Jacobi identity.

In case you are interested, there are a couple of good books on algebraic approach of Lie algebra. They are

Hans Samelson, Notes on Lie Algebra, 2nd Edition, Springer 1990

J. E. Humphreys, Introduction to Lie Algebras and Representation Theory, Springer 1973

Proposition. $\mathfrak{so}(n)$ is a Lie algebra with $[\ ,\ ]$.

Proof. In here we have shown that $\mathfrak{so}(n)$ is the set of all $n\times n$ skew-symmetric matrices. It suffices to show that $\mathfrak{so}(n)$ is closed under $[\ ,\ ]$.

For any $A,B\in\mathfrak{so}(n)$,
\begin{align*}
{}^t[A,B]&={}^t(AB-BA)\\
&={}^t(AB)-{}^t(BA)\\
&={}^tB{}^tA-{}^tA{}^tB\\
&=BA-AB\\
&=-[A,B].
\end{align*}
Hence $[A,B]\in\mathfrak{so}(n)$.