Documentation
README
AsyncOpenAI Concurrency: the Hidden httpx Pool Cap
Problem
Async batch scorers typically gate concurrency with asyncio.Semaphore(N).
Raising N above ~100 silently does nothing: the OpenAI SDK's default httpx
transport caps the connection pool at max_connections=100, so excess tasks
queue inside httpx instead of reaching the provider. The semaphore looks like
the throttle but is not the binding constraint — there is no error, just a
throughput ceiling.
Context / Trigger Conditions
This is the opening of the README. Read the full README on GitHub.