Where and why should you use iterators in Go?
Iterators are a language feature that allows you to traverse elements of a collection or data stream without having to load everything into memory at once. Go introduced them last year, and it sparked quite a reaction: many developers didn’t like the change, didn’t understand why iterators were needed or where to use them. The most common complaint I heard was that iterators made the language more complicated — while simplicity has always been a cornerstone of Go’s philosophy.
Since their release, I, like many others, haven’t actually used iterators in real-world work. All my tasks were easily handled with the usual tools, and I never ran into a problem that could only be solved with iterators.
In this article, I decided to dive in and figure out where iterators could actually be useful in practice. I read a bunch of articles, checked out developers’ feedback, asked around on Reddit — and in the end, I put together a list of cases where iterators can genuinely come in handy. I’ve already used a few of them in my own projects, and I’ve got to admit — it turned out to be pretty convenient.
All usage scenarios can be logically grouped into a few major categories.
Lazy data reading
When we talk about using iterators to work with large datasets, we’re primarily talking about lazy reading. For example, when reading from a file or a database with millions of rows, we don’t want to load everything into memory at once — we want to read line by line and process things sequentially.
The most common real-world case for lazy reading is is reading from a database. So let’s take a look at how iterators can be used in this context — and why they might actually make things more convenient.
Line-by-line reading from the database with additional processing
Like it or not, in relational databases we often end up storing data in JSON or other formats — which is why most modern databases even have dedicated types for this.
Let’s say we’re working with Postgres and we’ve decided to store data in CBOR — a binary format similar to JSON, but more efficient in terms of size and speed. We want to read data from the database and unmarshal each CBOR entry directly into a Go struct.
The classic way to read data sequentially from a database — without loading all rows into memory — is by using the Next() method from the sql.Rows type in the standard "database/sql" package:
rows, err = db.Query("select data from users_serialized")
if err != nil {
panic(err)
}
defer rows.Close()
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return
}
var user User
if err := cbor.Unmarshal(raw, &user); err != nil {
return
}
fmt.Printf("User: %+v\n", user)
}We could move this code into a repository and reuse it wherever needed. But if at some point we need to do extra processing on the User struct — like if a new field is added in JSON format that we also want to unmarshal — we’d either have to load all users into memory and filter them afterward, or write a new method with the added logic.
An alternative approach is to use iterators for this instead:
func UsersFromRows(rows *sql.Rows) iter.Seq[User] {
return func(yield func(User) bool) {
defer rows.Close()
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return
}
var user User
if err := cbor.Unmarshal(raw, &user); err != nil {
return
}
if !yield(user) {
return
}
}
}
}
...
rows, err := db.Query("select data from users_serialized")
if err != nil {
panic(err)
}
for user := range UsersFromRows(rows) {
fmt.Printf("User: %+v\n", user)
}In this case, we can reuse the iterable method UsersFromRows() and process each user however we need — without loading all users into memory.
The code above looks convenient, but it’s missing one important detail: error handling. Iterators can’t resume after interruption and don’t have a built-in way to propagate errors outward. So, we’ll still need to complicate the logic a bit to support that. The most straightforward way to do it is to return the error alongside the value:
type User struct {
ID int
Name string
Err error
}
func UsersFromRows(rows *sql.Rows) iter.Seq[User] {
return func(yield func(User) bool) {
defer rows.Close()
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
yield(User{Err: err})
return
}
var user User
if err := cbor.Unmarshal(raw, &user); err != nil {
yield(User{Err: err})
return
}
if !yield(user) {
return
}
}
}
}
...
rows, err := db.Query("select data from users_serialized")
if err != nil {
panic(err)
}
for user := range UsersFromRows(rows) {
if user.Err != nil {
// ...
}
fmt.Printf("User: %+v\n", user)
}Working with paginated APIs
Another common use case is working with APIs that use pagination. In this case, we’re not using iterators to save memory — the API already returns a fixed number of items per page. Instead, iterators serve as a tool to encapsulate the logic of fetching and traversing pages.
A great example of a paginated API was shared by a Reddit user in the comments under my post asking whether developers actually use iterators:
I have wrapped the various AWS paginated APIs with great results.
Let’s take a look at a real-world example of using iterators to handle API pagination in AWS.
Reading files page by page from AWS S3 with pagination
type Object struct {
Value types.Object
Err error
}
func S3Objects(ctx context.Context, client *s3.Client, bucket string) iter.Seq[Object] {
return func(yield func(Object) bool) {
var token *string
for {
resp, err := client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
Bucket: &bucket,
ContinuationToken: token,
})
if err != nil {
yield(Object{Err: err})
return
}
for _, obj := range resp.Contents {
if !yield(Object{Value: obj}) {
return
}
}
if resp.IsTruncated != nil && !*resp.IsTruncated {
break
}
token = resp.NextContinuationToken
}
}
}
func main() {
ctx := context.Background()
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
panic(err)
}
client := s3.NewFromConfig(cfg)
bucket := "some-bucket-name"
for obj := range S3Objects(ctx, client, bucket) {
fmt.Printf("Object: %s (%d bytes)\n", *obj.Value.Key, *obj.Value.Size)
}
}And honestly, I find this really convenient. Right now, I’m working on a project with tons of dependencies — both internal services and external ones — and about half of them use token-based pagination in their APIs. When used properly and in a well-structured project, iterators can make working with these kinds of APIs a lot easier.
Data pipelines
Building pipelines is another powerful use case for iterators that I’ve discovered for myself.
type User struct {
ID int
Name string
Err error
}
func UsersFromRows(rows *sql.Rows) iter.Seq[User] {
return func(yield func(User) bool) {
defer rows.Close()
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
yield(User{Err: err})
return
}
var user User
if err := cbor.Unmarshal(raw, &user); err != nil {
yield(User{Err: err})
return
}
if !yield(user) {
return
}
}
}
}
...
rows, err := db.Query("select data from users_serialized")
if err != nil {
panic(err)
}
for user := range UsersFromRows(rows) {
if user.Err != nil {
// ...
}
fmt.Printf("User: %+v\n", user)
}Now imagine that this loop is being used in a dozen different places across the project, and in some of them, additional filtering is needed — for example, in some cases we only want to return active users:
for user := range UsersFromRows(rows) {
if user.Err != nil {
// ...
}
if !user.IsActive {
continue
}
fmt.Printf("User: %+v\n", user)
}And in some cases, we might want to limit the selection to just 10 users:
count := 0
for user := range UsersFromRows(rows) {
if user.Err != nil {
// ...
}
if !user.IsActive {
continue
}
count++
if count >= 10 {
break
}
fmt.Printf("User: %+v\n", user)
}Over time, these kinds of filters tend to grow in number, and eventually we end up with dozens of loops throughout the codebase — some parts of the logic are duplicated, others differ slightly, and there’s no clean way to extract and reuse them.
This is where pipelines come in — a chain of iterators, where the stream of data passes through a series of functions step by step.
For example, let’s say we want to return only active users, extract just their names, and limit the result to 10 entries. To filter users based on a specific condition, we can write a custom iterator:
func Filter[T any](seq iter.Seq[T], pred func(T) bool) iter.Seq[T] {
return func(yield func(T) bool) {
seq(func(item T) bool {
if pred(item) {
return yield(item)
}
return true
})
}
}To map a user to their name, we use a Map iterator:
func Map[T any, R any](seq iter.Seq[T], transform func(T) R) iter.Seq[R] {
return func(yield func(R) bool) {
seq(func(item T) bool {
return yield(transform(item))
})
}
}And we’ll also write an additional iterator to limit the number of users returned:
func Take[T any](seq iter.Seq[T], n int) iter.Seq[T] {
return func(yield func(T) bool) {
count := 0
seq(func(item T) bool {
if count >= n {
return false
}
count++
return yield(item)
})
}
}And the usage will look like this:
...
activeNames := Take(
Map(
Filter(UsersFromDB(db), func(u User) bool {
return u.Active
}),
func(u User) string {
return u.Name
},
),
10,
)
for name := range activeNames {
fmt.Println("active user:", name)
}
Here are the benefits:
– You can combine iterators for filtering or transforming data in any order, reducing code duplication
– Better testability: you just need to cover the iterator logic with tests, and then you can test the entire pipeline against expected results
Conclusion
Iterators are a powerful mechanism — and honestly, pretty simple once you start working with them and get used to the concept. While writing this article, I tried out a bunch of iterator use cases, and the ones I highlighted above are definitely going into my toolbox.
I plan to keep updating this article as I come across more interesting use cases, so feel free to subscribe :)
I also run an English-language Telegram channel with a Go content digest:
https://t.me/digest_golang (en)
And a personal Telegram channel where I post notes, tools, and share my experience: