uploadSplitItems

This function uploads a file in chunks to an Amazon S3 bucket using the S3Client.

Particularly useful for handling streams of unknown size, since it automatically splits the flow into separate files, allowing for seamless processing and storage.

When the split size exceeds 5MB, the function automatically utilizes S3's multipart file upload to prevent retaining large chunks of data in-memory. If it is smaller, the function uses the put object operation for a quicker and more efficient processing.

val s3Client:  S3Client = ...
val bucket = "my-bucket"
val usersFlow = flowOf<User> { ... }

s3Client
    .uploadSplitItems(
        bucket = bucket,
        upstream = usersFlow,
        splitEach = TimeWindow(1000, 1.seconds),
        key = { part -> "folder/file-part-$part" }
    ) { user ->
        // convert the user into an array of bytes
    }
    .collect { response ->
        println("Uploaded part: ${response.key}")
    }

Parameters