Whether you're polling message queues, batching records to a datastore, or concurrently uploading a file in pieces to S3, interacting with external services is often complex. When this complexity mingles with your application logic your code becomes messy, hard to test, and fragile to change.
What if you could cleanup the mess with the use of Elixir streams? By choosing to describe these resources as `Enumerable` and `Collectable`, we can encapsulate this complexity and allow the purpose of the code to shine through.
Everyone is familiar with the classic streaming example:
File.stream!(""source.txt"") # source
&String.capitalize/1) # logic
File.stream!(""destination.txt"")) # destination
The power of this way of writing code has very little to do with files, and far more to do with the way in which the `Enumerable` and `Collectable` protocols separate the concerns at work. There is tremendous untapped potential here for declaratively integrating external services with program logic simply and succinctly.
By implementing Enumerable for SQS, and Collectable for a task cluster, a stream could be just as easily be:
SQS.stream!(""task-queue"") # source
|> Stream.map(&build_task/1)# logic
Tasks.cluster(""task-cluster"")) # destination
In my talk I'll cover concrete, practical ways you can encapsulate external resources in this way, yielding tremendous benefits with respect to clarity, versatility, and testing. I'll also go over some of the challenges presented by this approach, to include difficulties faced when making such streams proper OTP citizens.
Future changes to the standard library are likely to make great strides toward overcoming these difficulties, but they're already surmountable. My talk will show you how I've done it, and the decisions that you'll likely encounter along the way.