0%

Speed up Data Migration using JPA with Channels and CountDownLatch

March 15, 2025

Kotlin

Springboot

Bear in mind not every annotation for joining table is lazy-loading by default (eager-loading harms the data migration speed significantly).

We have studied how to create rate-limited concurrent tasks in golang here. With kotlin we can immitate the same concept directly!

    suspend fun createDefaultTags() = coroutineScope {
        val batchSize = 20
        val total = teamRepository.count().toDouble()
        val totalPages = Math.ceil(total/batchSize).toInt()
        println("BatchSize: $batchSize, Pages: $totalPages")

        val latch = CountDownLatch(totalPages)
        val channel = kotlinx.coroutines.channels.Channel<Unit>(5)

        for (page in 0..<totalPages) {
            launch(Dispatchers.IO) {
                try {
                    channel.send(Unit)
                    batchInsertTags(page, batchSize, totalPages)
                } catch (e: Exception) {
                    throw e
                } finally {
                    channel.receive()
                    latch.countDown()
                }
            }
        }
        println("Waiting for everything to finish ...")
        latch.await()
        println("Finished!")
    }

In otherwords,

  • CountDownLatch plays the role as sync.WaitGroup and
  • Channel<Unit> plays the role as chan struct{}.