Let \(y= \coth(x)\):
\[ y = \frac{e^x + e^{-x}}{e^x - e^{-x}}\]
In order to find the formula for the inverse of \(\coth\) we need to make \(x\) the subject of the above formula.
\[\begin{gathered} ye^x - ye^{-x} = e^x + e^{-x} \\ ye^x - e^{x} = e^{-x} + ye^{-x} \end{gathered}\]
If we multiply both sides by \(e^x\):
\[\begin{gathered} y(e^x)^2 - (e^{x})^2 = 1 + y\\ (e^x)^2(y-1) = 1 + y \end{gathered}\]
Now we can make \(x\) the subject:
\[\begin{gathered} (e^x)^2 = \frac{1 + y}{y-1} \\ x = \ln \left( \sqrt{ \left( \frac{1 + y}{y-1} \right) } \right) \end{gathered}\]
We can simplify this:
\[x = \ln \left( \left( \frac{1 + y}{y-1} \right) ^{(1/2)} \right) = \frac{1}{2} \ln \left( \frac{y + 1}{y-1} \right) \]
Since \(\ln(a)\) is not defined when \(a\) is negative, then \((y+1)/(y-1)\) cannot be negative as well. Therefore, \(\operatorname{arcoth}(y)\) is only defined when \(|y|>1\).