Start moving to Uint8Array in new APIs?

[benjamingr]

There was a suggestion by @jasnell to use Uint8Arrays in new APIs over Buffers as well as a weigh-in by @sindresorhus saying it is easier to author cross-platform APIs when using Uint8Arrays.

Here is a context #41553 (comment)

That is, the ask here is that Node.js should prefer Uint8Arrays over Buffers in new APIs.

What does everyone think? Should we stick to Buffer (which is a subclass of Uint8Array as a reminder) or prefer Uint8Arrays over buffers when possible in new APIs?

cc @nodejs/buffer @nodejs/streams

[ronag]

That will cause soo much confusion. I believe there are several methods/props that Buffer override and which act differently than Uint8Array.

[ronag]

In particular .slice work differently. Not sure if there are others.

[ronag]

I'm fine with using Uint8Array for Web apis which define the type. But for node api's I think we should stick with Buffer.

[mcollina]

It really depends on the subsystems we are targeting. It's impossible to make a generic call.

[paulmillr]

I have a big problem with Buffers.

1. buffer.slice() is mutable copy, uint8array.slice() is immutable. This is very bad. Recently got hit with a bug report. I was checking for the input to be instanceof Uint8Array and did array.slice(), then operated on the copy. But that doesn't work with buffers! The buffers were mutated even after copy. Why are they instance of Uint8Arrays if the behavior is different? Regression in 1.5.0 paulmillr/noble-ed25519#45
2. They are not supported in browsers and will never be. Adding an additional in-browser shim for buffers is bad.
3. They expose private information to a global variable. Imagine you're developing some secure software. You reason about zeroing data etc. Buffer.from hex creates 8Kb buffers instead of 32 bytes #41467

// Somewhere in your code
const privateBuf = Buffer.from(privateKey, 'hex');

// Rogue package can access
Buffer.from('1').buffer
// Which will of course show the contents of `privateBuf`
// No need in complex memory dumps!

This happens because there is 8KB shared buffer reused for all Buffer.from calls! There is zero need in making this as subtle as it is right now. Buffer.allocUnsafe seems like a good name for "using a part of global shared buffer", Buffer.from is not. Search GitHub code for the snippet and tell me how many people know about the "feature".

[benjamingr]

buffer.slice() is mutable copy, uint8array.slice() is immutable. This is very bad. Recently got hit with a bug report. I was checking for the input to be instanceof Uint8Array and did array.slice(), then operated on the copy. But that doesn't work with buffers! The buffers were mutated even after copy. Why are they instance of Uint8Arrays if the behavior is different? Regression in 1.5.0 paulmillr/noble-ed25519#45

FWIW: I agree having an API that behaves like a subclass but "lies" about keeping the same API structure is super-confusing.

[benjamingr]

What about the following suggestion:

• Old modules that use Buffer will use Buffer (for example readable streams that build heavily on them).
• New modules and APIs will prefer Uint8Arrays whenever possible for example the new HTTP API (if that's fetch that's easy).
• Every API that takes a Buffer will accept a Uint8Array (that might already be the case).
• Every API that returns a Buffer will continue returning a Buffer to not break the ecosystem (stuff like .slice).

Less likely to reach consensus but I'd still like that:

• "Soft deprecate" buffer.slice and recommend .subarray() much more strongly.

[jasnell]

We will never be able to get rid of Buffer. The generic rules of thumb I have in mind are:

1. unless an API specifically calls for the unique features/differences of Buffer, it should return Uint8Array instead. ( Also keep in mind that it's trivial to create a Buffer from a Uint8Array. e.g. Buffer.from(u8.buffer).
   )

2. Any API that accepts a Buffer should also accept Uint8Array (that is, Buffer should never be required as the only option) and docs should reflect that Uint8Array is accepted.

Examples where Buffer may be needed:

1. The API needs to work with hex or base64 encoded data.
2. The API needs Buffer's idea of slice
3. The code path is particularly performance sensitive and Buffer's pooling/uninitialized memory is needed

[addaleax]

"Soft deprecate" buffer.slice and recommend .subarray() much more strongly.

Yes please :)

1. Also keep in mind that it's trivial to create a Buffer from a Uint8Array. e.g. Buffer.from(u8.buffer).

This is a great example to show that it’s not actually trivial, because Buffer.from(u8.buffer) works 95 % of the time, and will do the absolute wrong thing the other 5 % of the time. And unfortunately, this has become a fairly common bug to encounter. 😕 Maybe we should add something like Buffer.fromView(abv: ArrayBufferView) as a shorthand for Buffer.from(abv.buffer, abv.byteOffset, abv.byteLength)?

Any API that accepts a Buffer should also accept Uint8Array (that is, Buffer should never be required as the only option) and docs should reflect that Uint8Array is accepted.

That’s also the current state of things – even Buffer.prototype’s own methods work when called on Uint8Arrays.

[benjamingr]

This is a great example to show that it’s not actually trivial, because Buffer.from(u8.buffer) works 95 % of the time, and will do the absolute wrong thing the other 5 % of the time.

I ran into that as well (even though I knew about the bug before) so I'd like to echo it's a problem.

That’s also the current state of things – even Buffer.prototype’s own methods work when called on Uint8Arrays.

That's neat I didn't know that.