It can be easily shown that
$${\rm SO}(2)=\left\{\left(\begin{array}{cc}
\cos\theta & -\sin\theta\\
\sin\theta & \cos\theta
\end{array}
\right): \theta\in[0,2\pi)\right\}\cong{\rm S}^1=\{e^{i\theta}:
\theta\in[0,2\pi)\}.$$Let $\gamma(t)=\left(\begin{array}{cc}
\cos\theta(t) & -\sin\theta(t)\\
\sin\theta(t) & \cos\theta(t)
\end{array}
\right)\in\mathrm{SO}(2)$ with $\theta(0)=0$ and $\dot\theta(0)\ne 0$. Then $\gamma(t)$ be a differentiable (regular) curve in ${\rm SO}(2)$ such that
$\gamma(0)=I$. Thus
$$\dot{\gamma}(0)=\left(\begin{array}{cc}
0 & -1\\
1 & 0
\end{array}\right)\left(\frac{d\theta}{dt}\right)_{t=0}$$
is a tangent vector to $\mathrm{SO}(2)$ at the identity $I$. Hence, the tangent space of ${\rm SO}(2)$ at $I$ is a line i.e. ${\rm SO}(2)$ is a one-dimensional Lie group. (We already know that ${\rm SO}(2)$ is a one-dimensional Lie group since it is identified with the unit circle ${\rm S}^1$.)

Remark. $\dot\gamma(0)=\left(\begin{array}{cc}
0 & -1\\
1 & 0
\end{array}\right)$ is a skew-symmetric matrix, i.e., $\dot\gamma(0)+{}^t\dot\gamma(0)=0$.

Let $\gamma: (-\epsilon,\epsilon)\buildrel{\rm
diff}\over\longrightarrow{\rm O}(n)$ such that $\gamma(0)=I$. Then $\dot{\gamma}(0)$ is a tangent vector to ${\rm O}(n)$ at $I$. Since $\gamma(t)\in{\rm O}(n)$, $$\gamma(t)\cdot{}^t\gamma(t)=I$$ for each $t\in(-\epsilon,\epsilon)$. Thus,
$$\dot{\gamma}(0)\cdot{}^t\gamma(0)+\gamma(0)\cdot\dot{{}^t\gamma}(0)=0.$$ Since ${}^t\gamma(0)=\gamma(0)=I$, $$\dot{\gamma}(0)+\dot{{}^t\gamma}(0)=\dot{\gamma}(0)+{}^t\dot{\gamma}(0)=0.$$ Hence, we see that any tangent vector to ${\rm O}(n)$ at $I$ is represented as a skew-symmetric $n\times n$ matrix. Conversely, we want to show that every skew-symmetric $n\times n$ matrix is a tangent vector to ${\rm O}(n)$ at $I$.

Suppose that $A$ is a $n\times n$ skew-symmetric matrix. As discussed here,
$$e^{At}=I+At+\frac{(At)^2}{2!}+\cdots+\frac{(At)^n}{n!}+\cdots=I+At+\frac{A^2}
{2!}t^2+\cdots+\frac{A^n}{n!}t^n+\cdots$$
is an $n\times n$ matrix.

If $AB=BA$, then by Cauchy’s Theorem,
$$\left(\sum_{k=0}^\infty\frac{A^k}{k!}\right)\left(\sum_{l=0}^\infty\frac{B^l}
{l!}\right)=\sum_{m=0}^\infty\sum_{p=0}^m\frac{A^{m-p}B^p}{(m-p)!p!}=\sum_{m=0}^\infty\frac{(A+B)^m}{m!}.$$ This implies that $e^Ae^B=e^{A+B}$ if $AB=BA$. In particular, $e^{A}e^{-A}=e^0=I$ so that $e^A$ is non-singular. If $A$ is skew-symmetric, then ${}^t(e^{At})=e^{{}^tAt}=e^{-At}$ and so $e^{At}\cdot{}^t(e^{At})=I$, i.e., $e^{At}\in{\rm O}(n)$. Now, $\displaystyle\frac{de^{At}}{dt}=Ae^{At}$ and $\dot{e^{At}}(0)=A$, i.e., the skew-symmetric matrix $A$ is a tangent vector to ${\rm O}(n)$ at $I$.

Proposition. The tangent space of ${\rm O}(n)$ or ${\rm SO}(n)$ at $I$ is the set of all $n\times n$ skew-symmetric matrices. Denote by ${\mathfrak o}(n)$ (${\mathfrak s\mathfrak o}(n)$) the tangent space of ${\rm O}(n)$ (${\rm SO}(n)$, respectively) at $I$. Note that $\dim{\mathfrak o}(n)=\displaystyle\frac{1}{2}n(n-1)$. This can be easily shown.

Definition. The tangent space ${\mathfrak o}(n)$ (${\mathfrak s\mathfrak o}(n)$) to the Lie group ${\rm O}(n)$ (${\rm SO}(n)$, respectively) at $I$ is called the Lie algebra of ${\rm O}(n)$ (${\rm SO}(n)$, respectively).

Definition. A group $(G,\cdot,{}^{-1},e)$ is a Lie group if $G$ is also a differentiable manifold and the binary operation $\cdot: G\times G\longrightarrow G$ and the unary operation (inverse) ${}^{-1}: G\longrightarrow G$ are smooth maps.

A subgroup of a Lie group is not necessarily a Lie subgroup.

Theorem. [C. Chevalley] Every closed subgroup of a Lie group is a Lie subgroup.

Examples of Lie Groups.

1. Let $M(m,n)=\{m\times n-\mbox{matrices over}\ \mathbb{R}\}\cong\mathbb{R}^{mn}$. Let $A=(a_{ij})\in M(m,n)$. Define an identification map\begin{align*}M(m,n)&\longrightarrow\mathbb{R}^{mn}\\(a_{ij})&\longmapsto(a_{11},\cdots,a_{1n};\cdots;a_{m1},\cdots,a_{mn}).\end{align*} We can naturally define topology on $M(m,n)$ by the identification map. $M(m,n)$ is covered by a single chart and the identification map is the coordinate map.
2. The General Linear Group ${\rm GL}(n)$: Let $\mathrm{GL}(n)=\{\mbox{non-singular}\ n\times n-\mbox{matrices}\}$. Define a map\begin{align*}\mathrm{GL}(n)&\longrightarrow\mathbb{R}\\A&\longmapsto\det A.\end{align*} This map is onto and continuous since $\det A$ is a polynomial function of entries $a_{ij}$ of $A$. $\mathrm{GL}(n)=\det^{-1}(\mathbb{R}-\{0\})$is an open subset of $\mathbb{R}^{n^2}$, so that it is a submanifold of $\mathbb{R}^{n^2}$. This group is called the general linear group. The set of all $n\times n$ non-singular real (complex) matrices is denoted by $\mathrm{GL}(n;\mathbb{R})$ ($\mathrm{GL}(n;\mathbb{C})$, resp.). More generally, the set $n\times n$ non-singular matrices whose entries are the elements of a field $F$ is denoted by $\mathrm{GL}(n;F)$ or $\mathrm{GL}(V)$ where $V$ is the vector space isomorphic to $F^n$. Note that $\mathrm{GL}(V)$ is also the set of all linear isomorphisms of $V$.
3. The Orthogonal Group $\mathrm{O}(n)$: The orthogonal group $\mathrm{O}(n)$ is defined to be the set $$\mathrm{O}(n)=\{n\times n-\mbox{orthogonal matrices}\},$$ i.e., $$A\in\mathrm{O}(n)\Longleftrightarrow A\cdot{}^tA=I,$$ where ${}^tA$ is the transpose of $A$ and $I$ is the $n\times n$ identity matrix.
4. The Special Orthogonal Group $\mathrm{SO}(n)$: The special orthogonal group is defined to be the following subgroup of $\mathrm{O}(n)$: $$\mathrm{SO}(n)=\{A\in\mathrm{O}(n): \det A=1\}.$$
5. The Special Linear Group $\mathrm{SL}(n)$: The special linear group is defined to be the following subgroup of $\mathrm{GL}(n)$ $$\mathrm{SL}(n)=\{A\in\mathrm{GL}(n): \det A=1\}.$$
6. The Unitary Group $\mathrm{U}(n)$: The unitary group $\mathrm{U}(n)$ is the set of all $n\times n$-unitary matrices, i.e. $$\mathrm{U}(n)=\{U\in\mathrm{GL}(n;\mathbb{C}): UU^\ast=I\},$$ where $U^\ast={}^t\bar U$. Physicists often write $U^\ast$ as $U^\dagger$. $\mathrm{U}(n)$ is a Lie subgroup of $\mathrm{GL}(n;\mathbb{C})$.
7. The Special Unitary Group $\mathrm{SU}(n)$: The special unitary group $\mathrm{SU}(n)$ is a Lie subgroup of $\mathrm{U}(n)$ and $\mathrm{SL}(2;\mathbb{C})$ $$\mathrm{SU}(n)=\{U\in\mathrm{SL}(2;\mathbb{C}):UU^\ast=I\}.$$

Proposition. For any $n\times n$ real or complex matrix $X$,
$$e^X:=\sum_{m=0}^\infty\frac{X^m}{m!}$$ converges and is a continuous function.

Proof. For the proof of the proposition click here.

Definition. Let $G$ be a matrix Lie group. The Lie algebra of $G$, denoted by $\mathfrak{g}$, is the set of all matrices $X$ such that $e^{tX}\in G$ for all $t\in\mathbb{R}$.

Definition. A function $A:\mathbb{R}\longrightarrow\mathrm{GL}(n;\mathbb{C})$ is called a one-parameter subgroup of $\mathrm{GL}(n;\mathbb{C})$ if

1. $A$ is continuous;
2. $A(0)=I$;
3. $A(t+s)=A(t)A(s)$ for all $t,s\in\mathbb{R}$.

Theorem. If $A$ is a one-parameter subgroup of $\mathrm{GL}(n;\mathbb{C})$, then there exists uniquely an $n\times n$-complex matrix $X$ such that $A(t)=e^{tX}$ for all $t\in\mathbb{R}$.

In differential geometry, the Lie algebra $\mathfrak{g}$ is defined to be the tangent space $T_eG$ to $G$ at the identity $e$. The two definitions coincide if $G$ is $\mathrm{GL}(n;\mathbb{C})$ or its Lie subgroup. If $X\in\mathfrak{g}$ then by definiton $e^{tX}\in G$ for all $t\in\mathbb{R}$. The one-parameter subgroup  $\{e^{tX}:t\in\mathbb{R}\}$ of $G$ can be regarded as a differentiable curve $\gamma:\mathbb{R}\longrightarrow G$ such that $\gamma(0)=e$ where $e$ is the $n\times n$ identity matrix $I$. Thus $\dot\gamma(0)=X$ is the tangent vector to $G$ at the identity $e$, i.e. $X\in T_eG$. Conversely, $X\in T_eG$. Let $\{\phi_t:G\longrightarrow G\}_{t\in\mathbb{R}}$ be the flow generated by $X$, i.e.
$$\frac{d}{dt}\phi_t(p)=X_{\phi_t(p)}.$$ Then $\phi_t$ is smooth, $\phi_0=e$, and $\phi_t\circ \phi_s=\phi_{t+s}$. That is, $\{\phi_t:G\longrightarrow G\}_{t\in\mathbb{R}}$ is a one-parameter subgroup of $\mathrm{GL}(n;\mathbb{C})$. Hence by the above Theorem, there exists uniquely an $n\times n$-complex matrix $Y$ such that $A(t)=e^{tY}$. Since $\dot A(0)=Y$, $Y=X$ i.e. $A(t)=e^{tX}\in G\leq\mathrm{GL}(n;\mathbb C)$. Therefore $X\in\mathfrak{g}$.

Physicists’ convention: In the physics literature, the exponential map $\exp:\mathfrak{g}\longrightarrow G$ is usually given by $X\longmapsto e^{iX}$ instead of $X\longmapsto e^X$. The reason for that comes from quantum mechanics and it will be discussed later.

References:

[1] Andrew Baker, Matrix Groups, An Introduction to Lie Group Theory, Springer 2001

[2] Brian C. Hall, Lie Groups, Lie Algebras, and Representations: An Elementary Introduction, Springer-Verlag 2004