Repository navigation
Conversation
github-actions
Bot
added
the
t-tooling
a738bbb to
97286df
Compare
Mantisus
changed the title
ConcurrencySystem to share one concurrency budget between crawlersConcurrencySystem to share one concurrency budget between crawlers
028fd2c to
b6e4fad
Compare
| self._concurrency_system = concurrency_system | ||
| self._autoscaled_pool = autoscaled_pool = AutoscaledPool( | ||
| concurrency_system=concurrency_system, | ||
| consumer=ConcurrencyConsumer(id=str(self._id)), |
There was a problem hiding this comment.
Note: self._id is shared by crawlers that share state, and a user-set id can collide with an auto-assigned one. Two such crawlers get equal (frozen) consumers, and both pools number tasks from "0", which breaks the "process-unique" contract for a custom per-consumer system. The bundled ConcurrencySystem ignores the consumer, so nothing breaks today. Maybe build one consumer per crawler in __init__ from its own class counter (e.g. f'{self.__class__.__name__}-{n}') and reuse it across runs?
✍️ Drafted by Claude Code
There was a problem hiding this comment.
Good point.
I'd prefer that we switch to using strings for IDs and use them not only for state. We already have an issue for that: #1699
But this will work as a temporary solution.
There was a problem hiding this comment.
yes, we could do it, maybe could you prepare PR for that at first, then rebase this one?
| current concurrency 3, desired 5, overloaded: cpu 58% of the time (limit 40%) | ||
| ``` | ||
|
|
||
| To read the concurrency values in code, use `crawler.concurrency_system`. |
There was a problem hiding this comment.
Maybe we should mention the logging changes as well?
There was a problem hiding this comment.
Do you think anything should be added to the “## Autoscaling logs moved to DEBUG” section?
There was a problem hiding this comment.
Logger names changed as well, right (_autoscaling -> autoscaling)? So probably those please.
b9f1b02 to
b6e4fad
Compare
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
39a2094 to
9294fec
Compare
This branch has not been deployed
4 participants
Description
Load monitoring and concurrency scaling move from
AutoscaledPoolinto a newConcurrencySystem. Pass one system to several crawlers through the newconcurrency_systemoption to cap their combined concurrency. The autoscaling classes are now public incrawlee.autoscaling.Autoscaling logs move to DEBUG, and
max_tasks_per_minutenow counts the tasks started over the last 60 seconds. Breaking changes are in the v2 upgrade guide.Issues
✍️ Drafted by Claude Code