CodePilot Automation: A Piece of Cake

CodePilot—the brain child of Twin Engine Labs co-founder and CEO Keith Hanson—is a “productized,” cloud-operated software-as-a-service (SaaS)1 and an enrichment (read: upgrade) of Huginn2, a powerful open-source tool for automating and integrating technology services. The service provided by CodePilot (also known internally and on the site as Automaton) is sometimes referred to as middleware3 by the tech community. The power of automation brings a sleek and simplistic approach to the complexity of the business and technology integration processes. By utilizing agents in CodePilot to build configurations of integrations, the world of automation comes to life in exciting and useful ways!

The scenarios in Automaton are designed by automation analysts, affectionately referred to by one client as the “CodePilot Magicians.” This service brings a much-needed edge to the world of middleware— pre-configured or custom solutions! So, how do the analysts work their “magic”? The answer can be found in the dynamism of agent configuration and integration. At first glance, understanding the powerful automation tools of Automaton may appear, to the uninitiated, like ancient runes from some faraway planet, but once the basic flow is understood everything becomes much more accessible to CodePilot's partners (e.g. co-pilots).

Bored by techie details? Scroll to the end for a high-level overview of a CodePilot scenario!

In order to better grasp scenario configurations one can approach Automaton like a cookbook chockfull of measurements and actions that a novice baker would likely balk at when considering certain techniques for the first time—Automaton is an oven! Take a look at this article from epicurious.com in order to better relate to the analogy (chefs are exempt). The resources available to automation analysts at CodePilot ensure quick delivery of solutions, and many scenarios only require brief formatting before being put to work on their customer's behalf. Therefore, the following process has already been configured, tested, debugged, and prepared. It’s basically like the pre-made birthday cakes in the grocery store. The only thing missing is the name written in icing and the corresponding number of candles.

Getting Started: Naming Conventions & Structure

Yes, there are a plethora of abbreviated words, jargon, and technical language that accompany this niche, but learning some of this language and understanding the processes will create a smoother, sleeker experience. The most appropriate place to begin our journey is to understand how agents are named and why certain naming schemes are used and when.

Figure 1 CodePilot Scenario

Typically, the first agent in a scenario will be assigned the word “START,” followed by a colon. All agents usually include the function of the agent in the name. It’s also considered helpful to include the type of agent in the name, such as Post, JavaScript, etc. The final asset to consider is whether or not the resulting events should be linked together, and if the answer is yes, then the agent’s payload will be “chained” to the payloads of at least one other agent.

In order to identify whether or not the events are being chained between agents, two inte-gers chained by two other integers will precede the name of the agent. That all may appear pretty confusing, but it is actually simple: 00_01 START: Get Contacts from CRM. The numbers preceding the name, an indication of chained events, serve to allow formatting later on which improves efficiency and access.

Post Agents: Going Shopping

In order to get things “cooking” the first step is to shop for ingredients. The typical scenario will begin with either a webhook4 or an API request5. In short, these tools allow information to pass between two points: a messenger sending a request and a receiver accepting a request. When these messages are sent via an API request an agent must be configured to the specifications of the receiving party (i.e. third party services). The party that responds to a request could be likened to Amazon. You navigate to Amazon’s website, select products which are placed in your shopping cart and check out, Amazon ships the merchandise you purchased, and your local postal worker delivers the product to your door.

In Automaton, the agent most frequently utilized in the beginning of a scenario (and often in the midst of one as well) is the Post Agent (see Figure 2). To begin configuring a Post Agent it is essential to read the API documentation and parameters (options) set forth by the recipient of the request. The parameters will include a URL, the method (POST, GET, PUT, PATCH, and DELETE), the payload (i.e. body, results), and any required headers or authorization requirements6.

Figure 2 Post Agent

The aforementioned methods serve various functions, much like spices and seasonings: POST sends a request to submit data, whereas GET sends requests for data to be sent back to the sender (simplified, but accurate). Basically, each method is an exchange of information between web servers. As seen in Figure 3, these type of requests can be significantly powerful due to the extent of information that can be accessed via a simple request. This data, in our analogy of baking, can be viewed as the raw ingredients—milk, butter, eggs: (keys and value pairs/data in arrays) that will be mixed together to create the “cake” that will satisfy any sweet tooth.

Figure 3 JSON Parsed Payload

Split Agents

Next, the ingredients must be taken out of the shopping bag and organized according to how they will be used. The payloads received from API requests will contain JSON7 objects—key and value pairs—and arrays, such as “items” in Figure 3, inside of an array denoted by the use of brackets: “items”: [ … ]. Having access to this wealth of data is excellent, but the properties (the key and value pairs) must be separated into individual arrays outside of the main array in order to access one of the most powerful tools available in Automaton—liquid interpolation (more on that further down).

So, since we need to separate the data into individual results we will be using a Split Agent. As the name denotes, this agent will split the primary array of the initial payload into individual arrays. In the Split agent, paths are assigned for what is being separated from the original pay-load. When splitting arrays, a precise path will return the individual data. For example, the orig-inal payload with “items” would be split (separated) by the following path to the array:

“array_path”: “.json_result.items”

The resulting events generated by the Split Agent will provide us with the separated data needed. However, for the sake of accessibility, this data will require formatting, and this—like mixing the ingredients into a uniform batter—will require that we assign interpolated values of each key and value pair.

Event Formatting & Liquid Interpolation

The Event Formatting Agent, a flexible and powerful accessory in our process, assigns new key and value pairs that correspond to paths in the split data. For instance, instead of being required to type "data": ".00_02.item.data" later in the scenario, one interpolated value can be assigned, and later this will allow it to be referred to as “{{.data}}”. This creates dynamic access to information within a payload. The keys on the left are given arbitrary names, meaning that they can be given any qualifier, such as apple, dog, first_name, etc.

Figure 4 Formatted Events

In the example below the formatting used is referred to as liquid interpolation, a process created by the e-commerce site, Shopify (denoted by two sets of curly braces surrounded by quotation marks “{{liquid.interpolation}}” ). Having dynamic access to data promotes a simpler approach to performing tasks in a workflow. Another helpful quality of liquid interpolation is the ability to filter the data being accessed by using the pipe ( | ) key, located above the enter key on most keyboards. Examples of filters and the various uses of liquid interpolation can be found on Shopify’s site8.

JavaScript Agents: Specialty Equipment

Sometimes a baker or chef has to pull out the heavy duty tools, such as an industrial food processor. That’s one way to think about a JavaScript9 agent. There are many, many agents available in Automaton for use in scenarios, but occasionally limitations are encountered. Instead of spending hours and hours attempting to work around the limits of pre-configured agents, analysts will sometimes resort to manually coding (hard-coding) a JavaScript (JS) agent. This allows an analyst to perform a specified task within the scenario that would otherwise not be possible.

Figure 4 Java Script Agent

For example, if the only results needed from Email Hunter were those with a score of 90 or above (the simulation at the end will go into greater detail concerning Email Hunter), then a JavaScript agent could be configured to only allow such results through from the previous payload. This precision magnifies the power of CodePilot’s automation capabilities.

Serving up Cake: CodePilot Lead Enrichment Scenario

Finally, we have assembled the tools and skills to bake a mouth-watering and (data)rich cake. Now that everything is properly aligned, the almost limitless possibilities of automation are at your fingertips. The following (partial) scenario is based on a real-life solution, and it will review the important details that were previously mentioned (If you scrolled through the detail-rich information, links are available in the endnotes for further reading).

Figure 5 Lead Enrichment Scenario

First, a Post Agent was used to retrieve leads from a CRM (Base in this example). Once the leads were received the results were split into individual items via a Split Agent. Then, an Event Formatting Agent was utilized in order to simplify the path to the data being used later in the scenario. Next, the formatted lead/contact information was sent, via another Post Agent, to Email Hunter10, a service that discovers emails based on first name, last name, and the domain of the person in question. Once the results were received from Email Hunter, a JavaScript Agent was configured to filter those results on the basis of the confidence score.

Afterward, the JavaScript Agent payload was formatted for use in the subsequent Post Agent that used a PUT method to update the results in the CRM. Notice that the Split Agent, the second Event Formatting Agent, and the final Post Agent are numerically ordered. This reveals at a glance that each of these agents’ events are chained in the scenario, meaning that each payload is attached and crafted into one distinct payload. Chained events allow for the use of liquid interpolation, along with formatting the events.

We now have a fully functioning contact enrichment scenario. This process can be emulated using any list of contacts or leads and applied to your current resources with only a few simple configurations. Now, after getting a taste of the potential, it’s possible to understand how relevant the services of CodePilot are to the modern marketplace. Imagine what other possibilities exist for incorporating these tools with your business? With the team at CodePilot, the sky is the limit!

Visit codepilot.io to learn more about automating your business and/or life!

1Software as a Service: http://searchcloudcomputing.techtarget.com/definition/Software-as-a-Service
2About Huginn: https://github.com/cantino/huginn/wiki
3Middleware: http://www.webopedia.com/TERM/M/middleware.html
4Webhooks: https://sendgrid.com/blog/whats-webhook/
5REST API: http://www.restapitutorial.com/lessons/restquicktips.html
6API Doc. Example: https://developers.getbase.com/docs/rest/articles/introduction
7Information on JSON: http://www.json.org/
8Shopify’s Liquid Interpolation Doc.: https://help.shopify.com/themes/liquid/basics
9JavaScript Tutorial: http://www.w3schools.com/js/
10About Email Hunter: https://emailhunter.co/about

James David Wade

About James David Wade

James is an automation analyst for Twin Engine Labs' CodePilot.io. He also happens to be our chief documentarian and an excellent editor who enjoys helping our customers better understand what we do.