It seems to me that there must be some smallest unit of matter capable of doing computation, some sort of computational 'atom', in the ancient Greek sense of the word. This would impose an upper bound on the amount of number crunching you could possibly do with a finite amount of mass.
For instance, I would argue that a system of N atoms can not simulate a system of N+1 atoms in real time with perfect fidelity. This limit holds no matter how clever you are, no matter how efficiently you implement your simulator. If you were able to, then your computer would be able to simulate itself, and still have 1 atom worth of power left over. The simulated machine could in term simulate itself, and itself have one atom of left over. This spirals down to infinity and you'd have an infinite number of atoms left over, each of which could be harnessed to do some small amount of work.
Therefore - if we are being simulated by a computer built in a world whose physics match our own, and we are being simulated in real time, then the computer simulating us must be at least as big as our entire universe.