The occupier has the right to refuse you entry to the premises.
CH25650 explains what to do in such a
case.
If you are on an inspection of premises and the occupier
decides not to continue with the visit then you should withdraw
immediately. You should offer another time to complete your work or
remove the records presented for inspection, see
CH23300, and continue inspection of them
on HMRC premises.
If the visit needs to be cut short due to external factors
on either side, withdraw politely after making arrangements to
continue your work on another day.