Prevent overwhelming APIs with too many concurrent requests:
func fetchAll(urls []string, maxConcurrent int) []Result {
sem := make(chan struct{}, maxConcurrent) // semaphore
results := make(chan Result, len(urls))
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
sem <- struct{}{} // acquire slot
defer func() { <-sem }() // release slot
results <- fetch(u)
}(url)
}
go func() {
wg.Wait()
close(results)
}()
var out []Result
for r := range results {
out = append(out, r)
}
return out
}
How it works: The buffered channel sem acts as a counting semaphore. Only maxConcurrent goroutines can hold a slot at once—others block on sem <- struct{}{} until a slot is released.
Reuse N workers instead of spawning per-task:
func workerPool(urls []string, workers int) []Result {
jobs := make(chan string, len(urls))
results := make(chan Result, len(urls))
// Start fixed workers
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for url := range jobs {
results <- fetch(url)
}
}()
}
// Send jobs
for _, url := range urls {
jobs <- url
}
close(jobs)
// Wait and close results
go func() {
wg.Wait()
close(results)
}()
var out []Result
for r := range results {
out = append(out, r)
}
return out
}
When to use: When you want predictable resource usage. Workers stay alive and pull from a job queue.
First error cancels all in-flight requests:
import "[golang.org/x/sync/errgroup](<http://golang.org/x/sync/errgroup>)"
func fetchAllOrFail(ctx context.Context, urls []string) ([]Result, error) {
g, ctx := errgroup.WithContext(ctx)
results := make([]Result, len(urls))
for i, url := range urls {
i, url := i, url // capture loop variables
g.Go(func() error {
res, err := fetchWithContext(ctx, url)
if err != nil {
return err // cancels ctx → other goroutines exit
}
results[i] = res
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
Key behavior: When one goroutine returns an error, the context is canceled. Other goroutines should check ctx.Done() to exit early.
Control requests per second:
import "[golang.org/x/time/rate](<http://golang.org/x/time/rate>)"
func fetchRateLimited(urls []string, rps int) []Result {
limiter := rate.NewLimiter(rate.Limit(rps), rps) // rps tokens/sec, burst = rps
results := make(chan Result, len(urls))
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
limiter.Wait(context.Background()) // blocks until token available
results <- fetch(u)
}(url)
}
go func() { wg.Wait(); close(results) }()
var out []Result
for r := range results { out = append(out, r) }
return out
}
Use case: Respect API rate limits (e.g., "max 100 requests per second").