Here I am complaining about Concurrency, because every other place on the internet shows this as a silver bullet, without telling the downsides and possible fixes you will need to do.
So, to make this simple, I’m gonna go through a list of problems that you will have as soon as you use concurrency, either with multithreading or horizontal scaling.
I’m planning to update this post every time I encounter a new problem.
Conflicting Scheduled Jobs
I’m planning on writing an article just about this, but if your code is like this, you can expect problems to come:
@Scheduled(every = "1m")
fun sync() {
val items = repository.findAll()
items.forEach { item ->
otherService.process(item)
item.delete()
}
}If you scale your project horizontally, adding more instances to your cluster, you may be lucky for the first few days. However, soon you will see this job will run at the same time on all your instances. So… you will need to fix this, maybe with a distributed lock or through a smarter select query.
I recommend this talk made by Rafael Ponte:
https://www.youtube.com/watch?v=ghpljMg8Ecc&t=106s
Silent Assassin In-Memory Cache
If you’re caching things in your project and distribute horizontally, each machine has its own in-memory cache. Soon, every instance you have will cache the same items. Then you can maybe kill all machines at the same time with an OutOfMemory.
Here, a solution is moving this cache to another safe place and maintaining your applications stateless. Redis is a nice solution for distributed cache.
Race Conditions
This can happen either with horizontal scaling or a simple monolith. If your app has state, you’re in a sad situation because multiple consumers can try to use the same resource at the same time.
To solve this, you can try optimistic/pessimistic locking or think of a smarter way to get the items you want to use without conflicting with other consumers.
Websockets Sessions
If you have multiple consumers who need to be in the same websocket session (like a chat), guess what happens if you distribute your cluster.. Yeah! If each instance has their own websocket session, you’re in trouble. Some users starts missing messages and things get weird.
You can:
- change your session implementation to be distributed
- try to build a code that makes every group of people to be in the same server (and session)
- give up on websockets and do polling