A client of mine would like to ingest device generated event data into our platform. I’d like to be able to use Extend for normalizing the data I receive into an output that can be processed by my post task execution scripts. Since this device generated data has a fairly high throughput (up to 10 req/s), I want to batch the normalization process so I can use 1 webtask execution to normalize an array of events.
My ideal implementation would be for my end user to not worry about iterating over the array and just focus on writing a task that implements a normalization function for a single event. My ingestion service would queue events, execute the webtask once with an array of events after a certain time or event count, and receive an array of outputs as a response.
My initial thinking is that this would be a problem best solved by a custom compiler, however I’m struggling with how best to execute the webtask script in the compiler. Should I be using an eval?