In the standard Hermitian quantum mechanics, the space of states of a quantum mechanical system forms a complex Hilbert space in general. But often the space of states becomes a real vector space. When the space of states is a real vector space, an alternative quantum mechanics may be considered.
Let $\mathbb{V}$ be the complex vector space of states of a quantum mechanical system. Define a map $J: \mathbb{V}\longrightarrow \mathbb{V}$ by
$$J\psi=\bar\psi$$
for every $\psi\in \mathbb{V}$. Then $J$ is an anti-linear involution called a real structure on $\mathbb{V}$. A real structure on a complex vector space is a way to decompose the complex vector space $\mathbb{V}$ into the direct sum of two real vector spaces. Any $\psi\in \mathbb{V}$ may be written as
$$\psi=\psi^++\psi^-,$$
\begin{align*}
\psi^+&:=\frac{1}{2}(\psi+J\psi)=\frac{1}{2}(\psi+\bar\psi)=\psi_{\mathrm{re}},\\
\psi^-&:=\frac{1}{2}(\psi-J\psi)=\frac{1}{2}(\psi-\bar\psi)=i\psi_{\mathrm{im}}.
\end{align*}
$J$ satisfies the properties:
\begin{align*}
J\psi^+&=\psi^+,\\
J\psi^-&=-\psi^-.
\end{align*}
Hence, one obtains a direct sum of vector spaces
$$\mathbb{V}=\mathbb{V}^+\oplus \mathbb{V}^-,$$
where
\begin{align*}
\mathbb{V}^+&=\{\psi\in \mathbb{V}: J\psi=\psi\},\\
\mathbb{V}^-&=\{\psi\in \mathbb{V}: J\psi=-\psi\}.
\end{align*}
Both $\mathbb{V}^+$ and $\mathbb{V}^-$ are real vector spaces. $\mathbb{V}^+$ is isomorphic to $\mathbb{V}^-$ via the isomorphism $\psi\longmapsto i\psi$. So,
$$\dim_{\mathbb{R}}\mathbb{V}^+=\dim_{\mathbb{R}}\mathbb{V}^-=\dim_{\mathbb{C}}\mathbb{V}$$
if $\dim_{\mathbb{C}}\mathbb{V}<\infty$. Let $\mathbb{V}^+=\mathbb{V}_{\mathbb{R}}$. Then $\mathbb{V}^-$ can be written as $i\mathbb{V}_{\mathbb{R}}$, so $\mathbb{V}$ may be viewed as the complexification of $\mathbb{V}_{\mathbb{R}}$
$$\mathbb{V}_{\mathbb{R}}^\mathbb{C}=\mathbb{V}_{\mathbb{R}}\otimes_{\mathbb{R}}\mathbb{C}.$$
The real structure $J$ may be used to define an inner product $\langle\ ,\ \rangle$ on $\mathbb{V}$: For any $\varphi,\psi\in \mathbb{V}$,
\begin{align*}
\langle\varphi,\psi\rangle&=\int (J\varphi)\psi dV\\
&=\int\bar\varphi\psi dV.
\end{align*}
This inner product is of course the standard Hermitian product known as Dirac braket $\langle\ |\ \rangle$ to physicists. Therefore, the real structure gives rise to the Hilbert space structure on $\mathbb{V}$ as in the standard Hermitian quantum mechanics.
In studying a quantum mechanical system, it is not necessary to use complex-valued state functions but one may instead use real-valued state functions. For example, let us consider the Hamiltonian for quantum harmonic oscillator
$$\hat H=-\frac{\hbar^2}{2m}\frac{\partial}{\partial x^2}+\frac{m}{2}\omega^2\hat x^2.$$
The stationary Schrödinger equation $\hat H\psi=E\psi$ determines the eigenstates
$$\psi_n(\xi)=2^{-\frac{n}{2}}\pi^{-\frac{1}{4}}(n!)^{-\frac{1}{2}}e^{-\frac{\xi^2}{2}}H_n(\xi),$$
where $\xi=\sqrt{\frac{m\omega}{\hbar}}x$, and eigenvalues (energies)
$$E_n=\left(n+\frac{1}{2}\right)\hbar\omega$$
for $n=0,1,2,\cdots$. $H_n(x)$ are the Hermite polynomials and they satisfy the parity relation
$$H_n(x)=(-1)^nH_n(-x).$$
The eigenstates $\psi_n(\xi)$ forms a real Hilbert space with the usual Hermitian product (Dirac braket).
This time, let $\mathbb{V}$ be the real vector space of states of a quantum mechanical system. Let $\varphi:\mathbb{R}^3\longrightarrow\mathbb{R}$. $\varphi$ is said to be even under parity if $\varphi(-\mathbf{r})=\varphi(\mathbf{r})$. $\varphi$ is said to be odd under parity if $\varphi(-\mathbf{r})=-\varphi(\mathbf{r})$. Any real-valued function defined on $\mathbb{R}^3$ may be written as the sum of an even function and an odd function. Any $\varphi(\mathbf{r})\in \mathbb{V}$ can be written as
$$\varphi(\mathbf{r})=\frac{1}{2}[\varphi(\mathbf{r})+\varphi(-\mathbf{r})]+\frac{1}{2}[\varphi(\mathbf{r})-\varphi(-\mathbf{r})].$$
$\frac{1}{2}[\varphi(\mathbf{r})+\varphi(-\mathbf{r})]$ is an even function and $\frac{1}{2}[\varphi(\mathbf{r})-\varphi(-\mathbf{r})]$ is an odd function. Define a map $J: \mathbb{V}\longrightarrow \mathbb{V}$ by
$$J\varphi(\mathbf{r})=\varphi(-\mathbf{r})$$
for every $\varphi(\mathbf{r})\in \mathbb{V}$ i.e. $J$ is the parity. Then $J$ is a linear involution. Any $\varphi(\mathbf{r})\in \mathbb{V}$ may be written as
$$\varphi(\mathbf{r})=\varphi^+(\mathbf{r})+\varphi^-(\mathbf{r}),$$
where
\begin{align*}
\varphi^+(\mathbf{r})&=\frac{1}{2}[\varphi(\mathbf{r})+J\varphi(\mathbf{r})]=\frac{1}{2}[\varphi(\mathbf{r})+\varphi(-\mathbf{r})],\\
\varphi^-(\mathbf{r})&=\frac{1}{2}[\varphi(\mathbf{r})-J\varphi(\mathbf{r})]=\frac{1}{2}[\varphi(\mathbf{r})-\varphi(-\mathbf{r})].
\end{align*}
We have
\begin{align*}
J\varphi^+(\mathbf{r})&=\varphi^+(\mathbf{r}),\\
J\varphi^-(\mathbf{r})&=-\varphi^-(\mathbf{r}).
\end{align*}
The map $J$ is called the fundamental symmetry (see [1] for example). One gets a direct sum of vector spaces
$$\mathbb{V}=\mathbb{V}^+\oplus \mathbb{V}^-,$$
where
\begin{align*}
\mathbb{V}^+&=\{\varphi\in \mathbb{V}: J\varphi=\varphi\},\\
\mathbb{V}^-&=\{\varphi\in \mathbb{V}: J\varphi=-\varphi\}.
\end{align*}
If $\varphi(\mathbf{r})$ is even, then $\nabla\cdot\varphi(\mathbf{r})$ is odd and if $\varphi(\mathbf{r})$ is odd, then $\nabla\cdot\varphi(\mathbf{r})$ is even. If we say two functions $\varphi_1(\mathbf{r})$ and $\varphi_2(\mathbf{r})$ are identical if $\varphi_1-\varphi_2$ is constant, then $\mathbb{V}^+$ is isomorphic to $\mathbb{V}^-$ via the linear map $\nabla\cdot\varphi(\mathbf{r})$.
The fundamental symmetry may be used to define an inner product $\langle\ ,\ \rangle$ on $\mathbb{V}$ as follows: For any $\varphi(\mathbf{r}),\psi(\mathbf{r})\in \mathbb{V}$,
\begin{align*}
\langle\varphi(\mathbf{r}),\psi(\mathbf{r})\rangle&=\int (J\varphi(\mathbf{r}))\psi(\mathbf{r})dV\\
&=\int\varphi(-\mathbf{r})\psi(\mathbf{r})dV.
\end{align*}
For any $\varphi^+(\mathbf{r})\in \mathbb{V}^+$, $\varphi^-(\mathbf{r})\in \mathbb{V}^-$, we have
\begin{align*}
\langle\varphi^+(\mathbf{r}),\varphi^+(\mathbf{r}\rangle&=\int[\varphi^+(\mathbf{r})]^2dV\geq 0,\\
\langle\varphi^-(\mathbf{r}),\varphi^-(\mathbf{r})\rangle&=-\int[\varphi^-(\mathbf{r})]^2dV\leq 0
\end{align*}
We may assume that each $\varphi(\mathbf{r})$ is separable i.e. $\varphi(\mathbf{r})=X(x)Y(y)Z(z)$. ($\varphi(\mathbf{r})$ is a solution of the stationary Schrödinger equation $\hat H\varphi=E\varphi$ and this is a condition that can be imposed when we solve the equation.) Furthermore, we may assume that each of $X(x)$, $Y(y)$ and $Z(z)$ is either even or odd. Since $\varphi(-\mathbf{r})=X(-x)Y(-y)Z(-z)$, if $\varphi(-\mathbf{r})$ is odd, then at least one of $X(x)$, $Y(y)$, $Z(z)$ must be odd. For any $\varphi^+(\mathbf{r})\in \mathbb{V}^+$, $\varphi^-(\mathbf{r})\in \mathbb{V}^-$, $\varphi^+(\mathbf{r})\varphi^-(\mathbf{r})$ is odd. Let us write $\varphi^+(\mathbf{r})\varphi^-(\mathbf{r})=X(x)Y(y)Z(z)$ and say $X(x)$ is odd. Then
\begin{align*}
\langle\varphi^+(\mathbf{r}),\varphi^-(\mathbf{r}\rangle&=\int \varphi^+(\mathbf{r})\varphi^-(\mathbf{r})dV\\
&=\int_{-\infty}^\infty X(x)dx\int_{-\infty}^\infty Y(y)dy\int_{-\infty}^\infty Z(z)dz\\
&=0.
\end{align*}
Hence, $\varphi^+(\mathbf{r})$ and $\varphi^-(\mathbf{r})$ are orthogonal for all $\varphi^+(\mathbf{r})\in \mathbb{V}^+$, $\varphi^-(\mathbf{r})\in \mathbb{V}^-$, and so the direct sum $\mathbb{V}^+\oplus \mathbb{V}^-$ can be replaced by the orthogonal sum $\mathbb{V}^+\dotplus \mathbb{V}^-$. Note that $\langle\ ,\ \rangle$ is indefinte, so the fundamental symmetry gives rise to a Krein space structure rather than a Hilbert space structure on $\mathbb{V}$. Here, we are using a particular fundamental symmetry which is the parity, however our discussion may be generalised in terms of an arbitrary fundamental symmetry $J$. Although it may seem redundant, physically the inner product needs to be redefined as
\begin{align*}
\langle\varphi(\mathbf{r}),\psi(\mathbf{r})\rangle&=\int\overline{J\varphi(\mathbf{r})}\psi(\mathbf{r})dV\\
&=\int\bar\varphi(\mathbf{r})\psi(\mathbf{r})dV
\end{align*}
for any $\varphi(\mathbf{r}),\psi(\mathbf{r})\in \mathbb{V}$. The reason is that in quantum mechanics $\langle\ ,\ \rangle$ often interacts with operators that may be complex.
In the above example of quantum harmonic oscillator, let $\psi_n^+(\xi)=\psi_{2n}(\xi)$ and $\psi^-_{2n+1}(\xi)$ for $n=0,1,2,\cdots$. Then we have $\langle\psi^+_m,\psi^+_n\rangle=\delta_{mn}$, $\langle\psi^-_m,\psi^-_n\rangle=-\delta_{mn}$, and $\langle\psi^+_m,\psi^-_n\rangle=0$. Let $\mathbb{V}^+$ and $\mathbb{V}^-$ be spaces spanned by the orthogonal bases $\mathcal{B}^+=\{\psi^+_n(\xi): n=0,1,2,\cdots\}$ and $\mathcal{B}^-=\{\psi^-_n(\xi): n=0,1,2,\cdots\}$, respectively. Then $\mathbb{V}=\mathbb{V}^+\dotplus \mathbb{V}^-$ is a Krein space.
When the state functions of a quantum mechanical system are real-valued, the state functions may form a real Hilbert space or a Krein space depending on the choice between the inner products we discussed above. Hence, one may expect to have a quantum theory alternative, in which case a Krein space arises, to the standard Hermitian quantum mechanics. This alternative quantum theory is PT-Symmetric Quantum Mechanics which was first introduced by Carl M. Bender. PT-symmetric quantum mechanics has been promoted as a new quantum theory which admits a certain type of complex non-Hermitian Hamiltonians, the so-called PT-symmetric Hamiltonians. I recently showed that PT-symmetric quantum mechanics is indeed a Hermitian quantum mechanics and that those complex non-Hermitian (they are actually Hermitian) Hamiltonians are not really physical. (They would result the violation of unitarity.) After all, once disregarding those unphysical complex Hamiltonians, this so-called PT-symmetric quantum mechanics is not that much different at all from the standard Hermitian quantum mechanics. I will write more about this some other time. In the meantime, interesting readers may be referred to my recent papers [2], [3] on the issue.
Update 1: To clarify, here I meant a PT-symmetric Hamiltonian being Hermitian by it being self-adjoint with respect to the indefinite Hermitian inner product above. That’s why I said “PT-symmetric quantum mechanics is a Hermitian quantum mechanics.”
Update 2: My stance on PT-symmetric quantum mechanics has changed as I have a much better understanding of what’s going on. I considered complex PT-symmetric Hamiltonians were unphysical due to physicists’ insistance on using definite inner product because they want to interpret $|\psi|^2$ as probability like the case in ordinary quantum mechanics. But if we can relax about the meaning of $|\psi|^2$ as not necessarily probability but something that needs to be preserved under time evolution, the issue of unitarity violation can be resolved and complex PT-symmetric Hamiltonians become physical. It is possible that the interpretation of $|\psi|^2$ being probability is valid only in the ordinary quantum mechanics and if there is a new formulation of quantum mechanics, perhaps it may require a new set of rules and interpretations.
References:
[1] János Bognár, Indefinite Inner Product Spaces, Springer-Verlag 1974
[2] Sungwook Lee, $PT$-symmetric quantum mechanics is a Hermitian quantum mechanics, arXiv:1312.7738 [quant-ph]
[3] Sungwook Lee, On Finite $J$-Hermitian Quantum Mechanics, arXiv:1401.5149 [quant-ph]
