Repository navigation
Replacing rand_chacha with chacha20 #934
Description
@tarcieri has recently implemented ChaCha*Rng via the chacha20 crate. docs are here. Breaking out into a new issue this isn't really the topic of #872...
This brings the question: should we prefer this over the current (@kazcw's) implementation?
First things first, the rand_chacha crate has 44 reverse dependencies, is clearly a rand family crate and is recommended in our book. There is no plan to retire this crate.
Second, lets look at a few stats.
Unsafe
Here are the results of running cargo geiger on the current rand_core (after #931):
$ cargo geiger
<snip>
Metric output format: x/y
x = unsafe code used by the build
y = total unsafe code found in the crate
Symbols:
:) = No `unsafe` usage found, declares #![forbid(unsafe_code)]
? = No `unsafe` usage found, missing #![forbid(unsafe_code)]
! = `unsafe` usage found
Functions Expressions Impls Traits Methods Dependency
0/0 0/0 0/0 0/0 0/0 ? rand_chacha 0.2.2
2/2 557/626 0/0 0/0 14/22 ! ├── ppv-lite86 0.2.6
0/0 22/22 0/0 0/0 0/0 ! └── rand_core 0.5.1
2/2 579/648 0/0 0/0 14/22
And on chacha20:
$ cargo geiger --no-default-features --features rng
<snip>
Functions Expressions Impls Traits Methods Dependency
7/14 199/427 0/0 0/0 1/2 ! chacha20 0.3.1
0/0 22/22 0/0 0/0 0/0 ! ├── rand_core 0.5.1
2/4 50/150 1/1 0/0 3/3 ! │ └── getrandom 0.1.14
0/0 0/0 0/0 0/0 0/0 ? │ ├── cfg-if 0.1.10
2/2 73/95 0/0 0/0 5/11 ! │ └── libc 0.2.66
0/0 0/0 0/0 0/0 0/0 ? └── stream-cipher 0.3.2
0/0 0/0 0/0 0/0 0/0 ? ├── blobby 0.1.2
0/1 0/231 0/0 0/0 0/0 ? │ └── byteorder 1.3.2
0/1 0/223 0/18 0/8 0/5 ? └── generic-array 0.12.3
0/0 0/51 0/0 0/0 0/0 ? └── typenum 1.11.2
11/22 344/1199 1/19 0/8 9/21
Something's wrong here: stream-chiper is only a dev-dependency and rand_core is depended on in exactly the same way as in rand_chacha. So only the first line is relevant.
Lines of code
Tokei output for rand_chacha:
$ tokei rand/rand_chacha/ cryptocorrosion/utils-simd/ppv-lite86/
-------------------------------------------------------------------------------
Language Files Lines Code Comments Blanks
-------------------------------------------------------------------------------
Markdown 2 70 70 0 0
Rust 9 4285 3864 164 257
TOML 2 48 42 0 6
-------------------------------------------------------------------------------
Total 13 4403 3976 164 263
-------------------------------------------------------------------------------
(83% of this is from ppv-lite86).
For chacha20:
$ tokei
-------------------------------------------------------------------------------
Language Files Lines Code Comments Blanks
-------------------------------------------------------------------------------
Markdown 2 170 170 0 0
Rust 12 1617 1163 203 251
TOML 1 47 40 0 7
-------------------------------------------------------------------------------
Total 15 1834 1373 203 258
-------------------------------------------------------------------------------
There are no dependencies we need, so that's it. A nice improvement.
Benchmarks
(64-bit Haswell)
Here's rand_chacha:
$ cargo bench --bench generators chacha
Compiling ...
running 16 tests
test gen_bytes_chacha12 ... bench: 356,133 ns/iter (+/- 5,261) = 2875 MB/s
test gen_bytes_chacha20 ... bench: 539,102 ns/iter (+/- 8,237) = 1899 MB/s
test gen_bytes_chacha8 ... bench: 263,023 ns/iter (+/- 15,029) = 3893 MB/s
test gen_u32_chacha12 ... bench: 1,689 ns/iter (+/- 168) = 2368 MB/s
test gen_u32_chacha20 ... bench: 2,475 ns/iter (+/- 60) = 1616 MB/s
test gen_u32_chacha8 ... bench: 1,300 ns/iter (+/- 56) = 3076 MB/s
test gen_u64_chacha12 ... bench: 4,058 ns/iter (+/- 324) = 1971 MB/s
test gen_u64_chacha20 ... bench: 4,472 ns/iter (+/- 647) = 1788 MB/s
test gen_u64_chacha8 ... bench: 3,252 ns/iter (+/- 421) = 2460 MB/s
test init_chacha ... bench: 30 ns/iter (+/- 7)
and here's chacha20:
running 16 tests
test gen_bytes_chacha12 ... bench: 1,184,161 ns/iter (+/- 75,680) = 864 MB/s
test gen_bytes_chacha20 ... bench: 1,894,244 ns/iter (+/- 16,309) = 540 MB/s
test gen_bytes_chacha8 ... bench: 824,644 ns/iter (+/- 52,944) = 1241 MB/s
test gen_u32_chacha12 ... bench: 4,607 ns/iter (+/- 97) = 868 MB/s
test gen_u32_chacha20 ... bench: 7,351 ns/iter (+/- 121) = 544 MB/s
test gen_u32_chacha8 ... bench: 3,257 ns/iter (+/- 407) = 1228 MB/s
test gen_u64_chacha12 ... bench: 9,396 ns/iter (+/- 1,104) = 851 MB/s
test gen_u64_chacha20 ... bench: 14,856 ns/iter (+/- 750) = 538 MB/s
test gen_u64_chacha8 ... bench: 6,431 ns/iter (+/- 625) = 1243 MB/s
test init_chacha ... bench: 14 ns/iter (+/- 0)
Hmm, looks like chacha20 needs some help:
$ export RUSTFLAGS="-C target-cpu=native"
$ cargo bench --bench generators chacha
...
test gen_bytes_chacha12 ... bench: 500,699 ns/iter (+/- 28,157) = 2045 MB/s
test gen_bytes_chacha20 ... bench: 789,107 ns/iter (+/- 52,585) = 1297 MB/s
test gen_bytes_chacha8 ... bench: 357,024 ns/iter (+/- 23,501) = 2868 MB/s
test gen_u32_chacha12 ... bench: 2,139 ns/iter (+/- 56) = 1870 MB/s
test gen_u32_chacha20 ... bench: 3,260 ns/iter (+/- 116) = 1226 MB/s
test gen_u32_chacha8 ... bench: 1,571 ns/iter (+/- 32) = 2546 MB/s
test gen_u64_chacha12 ... bench: 4,472 ns/iter (+/- 55) = 1788 MB/s
test gen_u64_chacha20 ... bench: 6,714 ns/iter (+/- 421) = 1191 MB/s
test gen_u64_chacha8 ... bench: 3,338 ns/iter (+/- 46) = 2396 MB/s
Closer, but still behind rand_chacha (which gets negligible boost from target-cpu=native thanks to auto-detection).
Running chacha20's built-in benchmarks, I get around 6-6.2 cycles/byte without target-cpu=native, and 2.5-2.7 with; this is significantly short of the 1.4 cycles/byte @tarcieri claims so something gives here (perhaps just CPU-specific optimisations).
Of course, there's more to this than a few stats, and number-of-unsafe-usages is not a particularly useful comparison (since it says nothing about the size of the unsafe blocks). This is all I have time for right now. Thanks to all authors (also significantly @newpavlov).
Activity
tarcieri commentedon Jan 17, 2020
You'll need
-Ctarget-feature=+avx2to enable the AVX2 backend.tarcieri commentedon Jan 17, 2020
Regarding
unsafe, most of those arecore::archinvocations of CPU intrinsics, which are presently all unsafe but aside from unaligned loads and stores otherwise have safe semantics (and ideally, in future versions of Rust, could be safe).Probably the most questionable use of unsafe is here:
https://github.com/RustCrypto/stream-ciphers/blob/master/chacha20/src/rng.rs#L74
It'd be good to evaluate various safe ways of replacing that. Really it's just a same-size type conversion which ideally could be done with a type cast or using
[u8; BUFFER_SIZE]in the first place, which isn't possible given the current trait bounds (since BUFFER_SIZE is 64 or 128-bytes, and many traits are only impl'd for arrays up to 32 elements)dhardy commentedon Jan 17, 2020
I get the same results with
RUSTFLAGS="-C target-cpu=native -C target-feature=+avx2".tarcieri commentedon Jan 17, 2020
I just benchmarked the
chacha20crate w\ AVX2 againstc2-chachaand got proportionally similar numbers.These are on a 2.3 GHz Intel Core i9, benchmarked with
-Ctarget-cpu=native(which I guess activates+avx2)chacha20cratec2-chachaSo ~1.3 cpb for the
chacha20crate, and ~0.9 cpb for thec2-chachacrate (i.e.c2-chachais ~45% faster)It looks like
ppv-lite86is capable of using AVX2 as well.chacha20: Add c2-chacha benchmark for comparison
str4d commentedon Aug 9, 2021
I've made some significant improvements to
chacha20's performance in RustCrypto/stream-ciphers#261. I believe the remaining gap between it andc2-chachawould be addressed by RustCrypto/stream-ciphers#262.dhardy commentedon Sep 24, 2022
This issue is now 32 months old! Is it worth revisiting?
cipheris now a required dependency.If we don't do this, possibly we should instead support
zeroizeforrand_chacha.tarcieri commentedon Sep 24, 2022
@dhardy the v0.9 release made
ciphera hard dependency. It also removed the RNG implementations with the goal of being able to generically use any stream cipher as an RNG, but that support never shipped, so removing the RNG implementations was probably premature.I'd be happy to add the RNG types back and make the
cipherdependency optional if there's interest in usingchacha20in lieu ofrand_chacha.dhardy commentedon Sep 24, 2022
The reason that never happened so far is because (1) there is not (to me) a clear motivation for switching and (2) this crate hasn't seen a lot of maintenance activity recently. I'm not aware of any problem with
rand_chacha, though perhaps it should supportzeroize.A motivation against is the larger number of crates in the dependency tree; something many would not consider an issue but a few people have brought it up in the past.
9 for
chacha20vs 7 for the whole ofrand(with default features).tarcieri commentedon Sep 24, 2022
Yeah, we could make the
cipherdependency optional again like it was in v0.8 ofchacha20newpavlov commentedon Sep 24, 2022
cipherbecame a required dependency because this issue has not moved anywhere. It allowed to simplify code a bit, but it's relatively easy to make it optional again. We can do it, if the number of dependencies is the only roadblock.The main motivation is to unify code bases, so improvements in
chacha20would be immediately shared with therandecosystem and potentialrandcontributors would help us as well. As a minor improvement it would also reduce size of dependency tree and resulting binaries for projects which use bothThreadRngandchacha20.dhardy commentedon Sep 25, 2022
We should ensure @kazcw has the chance to comment here too.
Since the libraries don't have a huge scope and are basically done already, is this actually motivation?
Is this common?
My POV is still: there is little real motivation either way. But if enough other people feel strongly about this, it will likely persuade me.
tarcieri commentedon Sep 25, 2022
@dhardy I wouldn’t consider any of these libraries “done” per se.
AFAIK none of them have e.g. ARM NEON support and only provide SIMD backends on x86.
Once
std::simdis stable (or even before) it would probably be good to have an implementation of ChaCha which leverages it to provide portable SIMD to other platforms as well (which could potentially also provide NEON support as well)tarcieri commentedon Oct 3, 2023
@nstilt1 I'm not sure that's relevant to this issue, which is about retiring
rand_chachain favor ofchacha20nstilt1 commentedon Oct 21, 2023
I saw this issue when I was looking for issues regarding the lack of support for
zeroizeinrand_chacha. Then, I was looking into the open issues ofppv-lite86to see if I could handle some issues and possibly pack them into my open pull request. There are several significant open issues in that library, including:I was going to attempt the refactor process with help from AutoGPT, but then I realized that
chacha20already supports NEON andzeroize... so... I'm thinking of switching tochacha20in my personal project and adding theCryptoRngimpl for it in my fork ofrandbecause I'm not quite a macro magician :/34 remaining items
hpenne commentedon Jun 11, 2025
@tarcieri Does this mean that a new chacha20 release is necessary before it is practical to attempt to replace rand_chacha with chacha20, possibly because chacha20 is not yet compatible with rand_core v0.9.0?
tarcieri commentedon Jun 11, 2025
The prereleases are
rand_corev0.9 compatible, but we don't have a stable release out yet: https://docs.rs/chacha20/0.10.0-rc.0/chacha20/struct.ChaCha8Rng.htmlHopefully some time in the next couple of months.
hpenne commentedon Jun 13, 2025
I forked and did an exploratory replacement of rand_chacha with chacha20. It seemed easier that expected. chacha20 is already adapted for this through its support for rand_core, and few changes were necessary. All existing tests pass, but I have so far not investigated thoroughly if things like nonces etc. are properly set up now that the implementation has been replaced. I did have a initial glance at it, and that seemed to indicate that when StdRng::from_seed is called, then the nonce is all zeros and that this was the case when rand_chacha was used as well. Is that the case, that the nonce was not initialized when calling from_seed? It was a bit unexpected, so my initial thought is that I may have missed something.
@tarcieri However, I ran into a problem in rayon-monte-carlo.rs because the set_stream method seems unavailable for ChaCha8Rng. It's in the crate, and get_stream is available. Could the problem be that StreamId is not exported from chacha20?
@dhardy You wrote on Reddit that you were looking for a chacha20 integration that "satisfies the same test vectors". Did you mean that chacha20 should satisfy the same test vectors used by rand_chacha? Test vectors for chacha kind of belong in chacha20 if that is where the implementation is. It seems that there are decent test in there.
tarcieri commentedon Jun 13, 2025
@hpenne the method is here: https://docs.rs/chacha20/0.10.0-rc.0/chacha20/struct.ChaCha8Rng.html#method.set_stream
Though the
StreamIdtrait isn't re-exported (perhaps it could be), you can pass any of the types listed in the docs, and the method will callIntofor you.dhardy commentedon Jun 14, 2025
@hpenne
rngs/std.rscurrently only has one (very short) test vector. I'd like to see this expanded (possibly copying therand_chachatests or published ChaCha test vectors). There are two purposes for this:randthat we are using the advertised CSPRNG, and that this works as expected for various input keys/nonces/countersStdRngafter the next breaking release, it is required to list these in the changelog; vectors assure that such changes will not be overlooked.)hpenne commentedon Jun 14, 2025
Of course. I've done similar things myself and should have known better. My bad. For some reason the method did not show up in my IDE, but it is available. The actual issue was that the code in rayon-monte-carlo.rs passed a u64, not the u128 that chacha20 supports, so a simple conversion to u128 fixes it.
hpenne commentedon Jun 14, 2025
@dhardy That makes sense and it seems logical to base this on tests from rand_chacha. I see that it contains tests for five IETF test vectors for this Chacha variant (32 bit counter). These can be moved to std.rs to verify reproducibility. I might consider removing any extra testing not directly related to verification of these test vectors.
hpenne commentedon Jun 15, 2025
@dhardy I submitted a PR, but I made the stupid mistake of not testing locally with the MSRV version first. There are a number of build failures that are probably due to that. I suspect that perhaps the technique I used to avoid multiple versions of rand_core requires a newer compiler. I will look into it.
dhardy commentedon Jun 15, 2025
@hpenne yes, the MSRV will need to be increased. Once you have a working build, copy the
Cargo.lockto the.msrv.dhardy commentedon Jun 16, 2025
There's talk in #1642 about how to add the dependency on
chacha20given that in turn depends onrand_core. This may provide some motivation not to do this (but instead to implement the RNG traits from within this repo).Additional motivation for this is that it wouldn't lock us out of merging
rand_coreback intorandin the future.newpavlov commentedon Jun 16, 2025
I thought that MSRV bump was a non-starter for
randin the near future?It would've been better if
randv0.9 has migrated to the new edition (i.e. it had MSRV 1.85+) and we could relay on the MSRV-aware resolver, but compared to the current MSRV it would be a really steep bump.In other words, I think we should first decide when we plan to migrate to the new edition and only then consider migration to
chacha20.dhardy commentedon Jun 17, 2025
@newpavlov I agree that 1.85 (20th Feb 2025) is a steep MSRV bump. It looks more reasonable to release this as
randv0.10 while keepingrand_corev0.9 as-is. That way there isn't too large a compatibility gap betweenrandv0.9 and v0.10 (the largest part of this being theDistributiontrait).But that's too off-topic for this issue, so I'll open a new one.
chacha20: bring back `rand_core` support (#333)
chacha20: bring back `rand_core` support (#333)