Good catch, actually there were five. J Adams, his son JQ Adams, Martin Van Buren, Andrew Johnson, Trump. Wilson and Nixon weren't around to see their successor sworn in for other reasons.
https://www.whitehousehistory.org/notably-absent-presidential-no-shows-on-inauguration-day#:~:text=Five outgoing presidents – John Adams,not participate for other reasons.