A BitTorrent Swarm for Large Language Models

Petals lets people run enormous language models by distributing their layers across a network of volunteer computers.

“Run large language models at home, BitTorrent-style.”

Instead of downloading and running the entire model, participating servers host different groups of transformer layers. Your request travels through a temporary chain of those servers, with each machine processing its portion before passing the intermediate result onward.

It is a really interesting way to make models such as Llama, Mixtral, Falcon, and BLOOM available to people without several hundred gigabytes of GPU memory. The published performance reaches up to six tokens per second for Llama 2 70B and four tokens per second for Falcon 180B. That is usable for experimentation and basic chat, although latency, peer availability, and network quality will always make public swarms less predictable than local or centralized infrastructure.

The security implications concern me more.

Petals does not simply send every peer a plain-text copy of the prompt. Servers process intermediate tensors for the model layers they host. But the project plainly warns that public peers may be able to recover information about inputs and outputs, alter the results, or return incorrect data. It specifically advises people not to process sensitive information through the public swarm.

The system is designed so other users cannot execute arbitrary code on a participating server, but that does not make the inference itself private or trustworthy. A malicious node still sits directly inside the flow of computation.

The idea of sharing enough collective hardware to make huge models accessible is fascinating. I would just need much stronger privacy, verification, and trust mechanisms before sending anything meaningful through computers operated by strangers.

Explore Petals, read the distributed inference paper, and review the project’s security and privacy guidance.