Thank you for sharing this insightful post. I am currently exploring Spring Boot and Quarkus, particularly in the context of streaming uploads.
In your article, you introduce the "uploadToS3" method for streaming files to S3. While this approach is technically sound, I initially interpreted it as a solution for streaming file uploads directly from the client to S3. Upon closer reading, I realized that the current implementation first uploads the file in its entirety to the Quarkus server, where it is stored on the filesystem (with the default configuration), and then streams it from disk to S3.
This method is certainly an improvement over keeping the entire file in memory. However, for optimal resource efficiency, it might be beneficial to stream the file directly from the client to the S3 bucket as the data is received.
For the benefit of future readers, a solution that enables true streaming from the client to S3 could be very valuable. I have experimented with such an approach, though I am unsure if it fully aligns with idiomatic Quarkus practices. If you are interested, I would be happy to write a short blog post about it for you to reference.
Hi Michael! Yeah, this is a broad and deep topic. I have a real "streaming" post (upload in chunks) sitting in the publication pipeline. Might combine the two at some point.
I'd love to read about your approach and am happy to link it here!
My approach is simply taking an InputStream on a PUT endpoint. This seems like a reasonable approach to me for the use case of uploading files only. If you need a MultipartForm with POST you would have to parse the the upload, of course. Since the latter is not my use case at the moment, I did not follow up on this.
Looking again at my approach it seems like making a blog post out of this is a little bit exaggerated. However, this was what I was looking for and I had trouble finding (probably my fault in not reading the great Quarkus documentation properly).
Many thanks for the link! I am curious for the article that you will publish on this example. This is a lot more than what I wanted and it looks very interesting, to upload a file in chunks.
Thank you for sharing this insightful post. I am currently exploring Spring Boot and Quarkus, particularly in the context of streaming uploads.
In your article, you introduce the "uploadToS3" method for streaming files to S3. While this approach is technically sound, I initially interpreted it as a solution for streaming file uploads directly from the client to S3. Upon closer reading, I realized that the current implementation first uploads the file in its entirety to the Quarkus server, where it is stored on the filesystem (with the default configuration), and then streams it from disk to S3.
This method is certainly an improvement over keeping the entire file in memory. However, for optimal resource efficiency, it might be beneficial to stream the file directly from the client to the S3 bucket as the data is received.
For the benefit of future readers, a solution that enables true streaming from the client to S3 could be very valuable. I have experimented with such an approach, though I am unsure if it fully aligns with idiomatic Quarkus practices. If you are interested, I would be happy to write a short blog post about it for you to reference.
Hi Michael! Yeah, this is a broad and deep topic. I have a real "streaming" post (upload in chunks) sitting in the publication pipeline. Might combine the two at some point.
I'd love to read about your approach and am happy to link it here!
My approach is simply taking an InputStream on a PUT endpoint. This seems like a reasonable approach to me for the use case of uploading files only. If you need a MultipartForm with POST you would have to parse the the upload, of course. Since the latter is not my use case at the moment, I did not follow up on this.
Looking again at my approach it seems like making a blog post out of this is a little bit exaggerated. However, this was what I was looking for and I had trouble finding (probably my fault in not reading the great Quarkus documentation properly).
If you want to take a peek at a chucked approach. That is here https://github.com/myfear/ejq_substack_articles/tree/main/file-upload-progress
The blog is scheduled but I don't remeber for when right now. Thanks for sharing!
Many thanks for the link! I am curious for the article that you will publish on this example. This is a lot more than what I wanted and it looks very interesting, to upload a file in chunks.