General spinor rotations
If we write every spinor in equation \(\eqref{eq:spinor_transformation_matrix_expression}\) in its 2-component form, we can write the rotation of the spinor basis as: \[\begin{equation} \label{eq:Pauli_spinor_rotations_2_component} \require{physics} \begin{pmatrix} % \phi'_{1 \alpha} & \phi'_{2 \alpha} & \cdots & \phi'_{M \alpha} \\ \phi'_{1 \beta} & \phi'_{2 \beta} & \cdots & \phi'_{M \beta} \end{pmatrix} % = % \begin{pmatrix} % \phi_{1 \alpha} & \phi_{2 \alpha} & \cdots & \phi_{M \alpha} \\ \phi_{1 \beta} & \phi_{2 \beta} & \cdots & \phi_{M \beta} \end{pmatrix} % \vb{U} % \thinspace , \end{equation}\] or written for the spatial orbitals separately: \[\begin{align} & \phi'_{P \alpha} = \sum_Q^M \phi_{Q \alpha} U_{QP} \\ & \phi'_{P \beta} = \sum_Q^M \phi_{Q \beta} U_{QP} % \thinspace . \end{align}\] Writing the spinor rotations like this, the scalar product of two spinors remains invariant, as we have: \[\begin{align} \braket*{\phi'_P}{\phi'_Q} % &= \braket*{ \phi'_{P \alpha} }{ \phi'_{Q \alpha} } % + \braket{ \phi'_{P \beta} }{ \phi'_{Q \beta} } \\ &= \sum_{RS}^M U^*_{RP} U_{SQ} \Big( % \braket*{ \phi_{P \alpha} }{ \phi_{Q \alpha} } % + \braket*{ \phi_{P \beta} }{ \phi_{Q \beta} } % \Big) \\ &= \delta_{PQ} % \thinspace . \end{align}\]
A first question arises immediately: may we mix the \(\alpha\)- and \(\beta\)-components? Let’s explore this question using a small example spinor basis: \[\begin{align} & \phi_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix} % & \phi_2 = \begin{pmatrix} -i \\ i \end{pmatrix} % \thinspace . \end{align}\] A mixing of the \(\alpha\)- and \(\beta\)-components would mean that we could write the transformations of the separate spatial orbitals as the following rearrangement: \[\begin{equation} \begin{pmatrix} % \phi'_{1 \alpha} & \cdots & \phi'_{M \alpha} & \phi'_{1 \beta} & \cdots & \phi'_{M \beta} \end{pmatrix} % = % \begin{pmatrix} % \phi_{1 \alpha} & \cdots & \phi_{M \alpha} & \phi_{1 \beta} & \cdots & \phi_{M \beta} \end{pmatrix} % \vb{U} % \thinspace , \end{equation}\] where the rotation matrix \(\vb{U}\) is now of dimension \((2M \times 2M)\), carrying the notion of a possible mixing between the \(\alpha\)- and \(\beta\)-components. In our example, we could use the unitary transformation matrix: \[\begin{equation} \vb{U} = % \begin{pmatrix} 0 & 0 & -i & 0 \\ 0 & i & 0 & 0 \\ 0 & 0 & 0 & -1 \\ 1 & 0 & 0 & 0 \end{pmatrix} \thinspace , \end{equation}\] which would yield the following spinors after doing the unitary transformation: \[\begin{align} & \phi'_1 = \begin{pmatrix} i \\ -i \end{pmatrix} % & \phi'_2 = \begin{pmatrix} 1 \\ -1 \end{pmatrix} % \thinspace . \end{align}\] This shows that there exist \((2M \times 2M)\) unitary transformations (i.e. `mixing’ transformations) that do not yield a new orthonormal set, so we therefore conclude that we can not mix the spatial orbitals used for the \(\alpha\)-component with those used for the \(\beta\)-component in this general framework.
Can we do separate transformations for the \(\alpha\)-components and the \(\beta\)-components? We would then write the rotations of the underlying spatial orbitals as: \[\begin{align} & \phi'_{P \alpha} = \sum_Q^M \phi_{Q \alpha} U^{\alpha}_{QP} \\ & \phi'_{P \beta} = \sum_Q^M \phi_{Q \beta} U^{\beta}_{QP} % \thinspace , \end{align}\] in which both \(\vb{U}^{\alpha}\) and \(\vb{U}^{\beta}\) now are unitary matrices. Does this transformation yield spinors that are again orthonormal? We can work it out: \[\begin{align} \braket*{\phi'_P}{\phi'_Q} % &= \braket*{ \phi'_{P \alpha} }{ \phi'_{Q \alpha} } % + \braket*{ \phi'_{P \beta} }{ \phi'_{Q \beta} } \\ &= \sum_{RS}^M % U^{\alpha *}_{RP} U^{\alpha}_{SQ} % \braket*{ \phi_{P \alpha} }{ \phi_{Q \alpha} } % + \sum_{RS}^M % U^{\beta *}_{RP} U^{\beta}_{SQ} % \braket*{ \phi_{P \beta} }{ \phi_{Q \beta} } % \thinspace , \end{align}\] after which we are stuck: it is not because the original spinors \(\set{\phi_P}\) are orthonormal that the underlying spatial orbitals \(\set{\phi_{P \alpha}}\) and \(\set{\phi_{P \beta}}\) are orthonormal on their own. We therefore conclude that we may not separately transform the \(\alpha\)- and \(\beta\)-components in this kind of general framework.
As a conclusion, in the general framework of 2-component Pauli spinors, we may only rotate the spinors according to equation \(\eqref{eq:Pauli_spinor_rotations_2_component}\). Any misconceptions about these general spinor rotations seems to be due to an unrightful application of concepts that can only be used in the unrestricted or restricted formalism.
The generator or general spinor rotations thus is still: \[\begin{equation} \hat{\kappa} = % \sum_{P>Q}^M % \prescript{\text{R}}{}{ \kappa_{PQ} } % \hat{E}^-_{PQ} % + i \qty( % \sum_{P}^M % \prescript{\text{I}}{}{ \kappa_{PP} } % \hat{N}_P % + \sum_{P>Q}^M % \prescript{\text{I}}{}{ \kappa_{PQ} } % \hat{E}^+_{PQ} % ) % \tag{\ref{eq:spinor_rotation_generator}} \thinspace , \end{equation}\] and for the special case of rotations of real spinors, we still have: \[\begin{equation} \prescript{\text{R}}{}{\hat{\kappa}} = % \sum_{P>Q}^M % \prescript{\text{R}}{}{\kappa_{PQ}} % \hat{E}^-_{PQ} % \tag{\ref{eq:spinor_rotation_generator_real}} \thinspace . \end{equation}\]