Golang Split data into batches to concurrent/parallel update in DB
Given a large slice with data that needs to be updated in the sql-like DB.
Update in one SQL query can be long and lead to problems.
How to split this data into batches to concurrent/parallel update with small queries? For example, `UPDATE tablename SET uid=12345 WHERE url IN (...)`
At least a small example ... I already broke my head(
```
type Task struct {
URL string
UID string
}
type Pool struct {
numWorkers int
tasks chan Task
done chan bool
}
func New(numWorkers int, buffer int) *Pool {
return &Pool{
numWorkers: numWorkers,
tasks: make(chan Task, buffer),
done: make(chan bool),
}
}
func (p *Pool) Run() {
wg := &sync.WaitGroup{}
for i := 0; i < p.numWorkers; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
for {
select {
case task := <-p.tasks:
// execute
case <-p.done:
close(p.tasks)
return
}
}
}(i)
}
wg.Wait()
p.done <- true
}
func (p *Pool) AddTask(urls []string, uid string) {
for _, url := range urls {
p.tasks <- Task{URL: url, UID: uid}
}
}
```
