We consider a discrete-time Markov chain $\boldsymbol{ \Phi}$ on a general state-space ${ \sf X}$, whose transition probabilities are parameterized by a real-valued vector $\boldsymbol{ \theta}$. Under the assumption that $\boldsymbol{ \Phi}$ is geometrically ergodic with corresponding stationary distribution $\pi(\boldsymbol{ \theta})$, we are interested in estimating the gradient $\nabla \alpha(\boldsymbol{ \theta})$ of the steady-state expectation $$\alpha(\boldsymbol{ \theta}) = \pi( \boldsymbol{ \theta}) f.$$ To this end, we first give sufficient conditions for the differentiability of $\alpha(\boldsymbol{ \theta})$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.
Categories: math.PR