Even if I took every word of this at face value, I feel like people would notice the CPU usage pretty quickly, from the heat coming off the chip if nothing else. Local inference is about as subtle as a coinminer.
assuming you need to be fast that’s true, but if you don’t care how much time you take then you can spread the processing out… this is exactly how MoE models are much easier to run locally, but you could go much further
Even if I took every word of this at face value, I feel like people would notice the CPU usage pretty quickly, from the heat coming off the chip if nothing else. Local inference is about as subtle as a coinminer.
assuming you need to be fast that’s true, but if you don’t care how much time you take then you can spread the processing out… this is exactly how MoE models are much easier to run locally, but you could go much further