amazing. can lean on multipart-post quite a bit:

headers = {}
uri = URI("https://example.com/my_api")
data = URI("https://i.etsystatic.com/39938216/r/il/11dead/4579446101/il_fullxfull.4579446101_1irk.jpg").read
form_data = {
  file: ::Multipart::Post::UploadIO.new(StringIO.new(data), "image/jpg", "image.jpg")
}
multipart_post = Net::HTTP::Post::Multipart.new(uri.path, form_data, headers)
headers["content-type"] = multipart_post["content-type"]
headers["content-length"] = multipart_post["content-length"]
body = multipart_post.body_stream.read

task = Async do
  internet = Async::HTTP::Internet.instance
  internet.post(uri.to_s, headers, body)
end
response = task.wait

does that seem about right and optimal?

should body always be an array?

Async::HTTP::Internet is more permissive on what it accepts as the body:

In essence, it's okay to provide an Array of String chunks. Async::Client is less permissive and expects an instance of the a body class.

However, there are alternatives which can help you solve different problems, like streaming from disk, etc. You can pass an instance of those too, e.g. https://github.com/socketry/async-http/blob/main/lib/async/http/body/slowloris.rb is a body wrapper that simulates a slow upload/download.

it's okay to provide an Array of String chunks

and i'm assuming it's okay to provide a string (to Async::HTTP::Internet).

do you see any advantages to doing what Net::HTTP::Post::Multipart is doing a bit more manually and providing string chunks? honestly uploading files is not a big part of what our falcon server is doing so i'm not too concerned about making it maximally optimal. but i would appreciate more clarity on what details affect concurrency and what areas i should be paying more attention to (like, is there a better time to call task.wait to handle the response?).

understandable if there are too many opaque/unspoken details, though. and thanks for all the guidance.

IMHO, if I was building a hypothetical Protocol::HTTP::Multipart, I'd provide wrappers for loading files from disk, and from memory, and then a container to put them all together. I'd probably avoid trying to chunk the entire thing instead breaking it into "multipart header", "file chunk1", "file chunk2", ...", "multipart header", "memory chunk1", "memory chunk2", "end of multipart" etc.

It's definitely optimal to chunk IO according to host page size and/or network MTU. It's also better to only load chunks from disk as they are being sent, as this would avoid using a lot of memory if you had to load the entire thing. However, if it's already in memory, there is very little point to write it out to disk first or try and behave identically to an IO.

In terms of chunking, Async::HTTP assumes you will break things into optimal chunks - i.e. if you provide an array of 1 chunk that's 1000MiB, it will try to write that all in one go. What that means, is that if you have HTTP/1.1 chunked encoding, it will try to write a 1000MiB chunk. HTTP/1.0 and 2 have different flow control mechanisms, so the data would be buffered and broken down into smaller pieces. So I wouldn't think about it too much and instead aim for page-aligned chunks. I tried to be a little helpful here by exposing this:

https://github.com/socketry/io-stream/blob/7adeff467e8d038a260b712e4d7c372fc2ff8fa3/lib/io/stream/generic.rb#L15-L19

I would recommend you use those constants for "streaming IO" chunk size, and in addition, I recommend you can experiment changing those values to see the impact on throughput. Certainly I've done that and it's very interesting to do a sweep from 1 byte -> 1MiB in small increments.

like, is there a better time to call task.wait to handle the response?

Basically, you should avoid waiting on things.

If your goal is to stream from one source to another, in theory you can do that but I'm not sure how effectively you can implement it with the multipart-post gem.

Basically, you should chain the data from the incoming file to the place you are trying to post it.

In simple cases you can literally write this:

::Protocol::HTTP::Middleware.build do |request|
  Async::HTTP::Internet.instance.post(..., request.body)

This will ensure that as soon as the client start sending the file, you will start streaming to the post request, i.e. the minimum TTFB possible. You have to be a little more mindful about error handling though. In theory we could do this with the hypothetical "Protocol::HTTP::Multipart" that doesn't exist yet... it would be pretty neat.