The surprising answer is with Franklin Roosevelt. The addition of the Morning Worship Service was begun by FDR in 1933 and then repeated in 1937 and 1941, fixing the practice in inaugural tradition. FDR was also the first to feature an invocation by a religious leader (and a benediction). FDR also said a prayer at one inauguration.
My resident citizen mentions Lincoln's Second Inaugural Address:
The Almighty has his own purposes. "Woe unto the world because of offenses! for it must needs be that offenses come; but woe to that man by whom the offense cometh." If we shall suppose that American slavery is one of those offenses which, in the providence of God, must needs come, but which, having continued through his appointed time, he now wills to remove, and that he gives to both North and South this terrible war, as the woe due to those by whom the offense came, shall we discern therein any departure from those divine attributes which the believers in a living God always ascribe to him? Fondly do we hope--fervently do we pray--that this mighty scourge of war may speedily pass away. Yet, if God wills that it continue until all the wealth piled by the bondsman's two hundred and fifty years of unrequited toil shall be sunk, and until every drop of blood drawn by the lash shall be paid by another drawn with the sword, as was said three thousand years ago, so still it must be said, "The judgments of the Lord are true and righteous altogether."
With malice toward none; with charity for all; with firmness in the right, as God gives us to see the right, let us strive on to finish the work we are in; to bind up the nation's wounds; to care for him who shall have borne the battle, and for his widow, and his orphan--to do all which may achieve and cherish a just and lasting peace among ourselves, and with all nations.