Subtitle: The biggest factor wasn't a producer config.
I set acks=all and replication.factor=3 on a Kafka cluster last week. Then I watched one scenario crawl at 0.42 MB/s with a p99 latency of 72 seconds while another, on the same cluster with the same durability guarantees, pushed 70.2 MB/s at 81 ms p99.
I expected the producer settings everyone talks about (batch.size, linger.ms) to explain most of that gap. They didn't. The biggest factor was a broker config I almost didn't test.