In deciding whether I would have been an employee of my client, will you only look at the contract I have signed, or the contract between the client and an agency?

HMRC will take account of all relevant contracts in order to discover whether the relationship between a worker and a client would have been one of employment, if there had been no intermediary. This would include any contracts between the client and an agency, and between the agency and the worker's intermediary.