Configure and Move Tailored Datasets with Pipelines

Pipelines are the center of data movement and payload orchestration in the Intelligence Hub. Use Pipelines to curate, apply logic, and optimize datasets for specific applications using a simple but flexible graphical user interface. Build out stages in a pipeline to model, filter, buffer, or transform data for optimized delivery to consuming applications and device nodes.

Download Software

Request Demo

Oversee data movement between systems with Pipelines.

Data Motion

Orchestrate the movement of data on time intervals or logic-based events. Delay the delivery of data or buffer data based on time or size. Maintain the state of data that has been moved.

Data Processing

Sequentially transform data with pre-built processing stages, custom expressions, or even third-party JavaScript libraries. Process complex event streams of data structures with modeling, validation, and end-to-end observability. Tailor modeled data to meet the needs of multiple consuming applications and services. Set pipelines as “callable” to expose them to other pipelines or the REST Data Server.

Data I/O

Compose data extraction routines from simple input reads to sophisticated parameter-based queries, multi-stage lookups, cross references, payload appending, and more. Publish data to destination systems and dynamically drive schema, topics, keys, and identifiers. Define success/failure criteria of an integration and enforce these criteria with custom error handling.

PRODUCT DEMO

See Pipelines in Action

Use this guided demo to step through the process of building and configuring a Pipeline in HighByte Intelligence Hub. Learn how to navigate the graphical user interface to create pipelines that can stream and transform industrial data.

Enable complex data processing with the Intelligence Hub for a variety of use cases.

Curate Payloads

Most target systems have limitations on how they can process inbound data. They may be incapable of parsing or filtering structured data. They may consume multiple records in batches or files. Use Pipelines to dynamically break up objects and arrays, discard unnecessary elements to facilitate easy consumption, and buffer data on time or record count. Publish as JSON payloads or as CSV and Parquet files—or compress those files prior to delivery. This enables target systems to efficiently consume industrial data regardless of how source systems produce and transmit it.

Simplify Data Pipelines

The transportation of contextualized data is a challenge for many industrial organizations. Solve this pain point with Pipelines. Use Pipelines to read data from an input or instance source and then publish that data to one or many target connections. Pipelines can be triggered by events or intervals while running the execution of each stage in the pipeline. Pipeline Replay and Debug Mode allow you to observe data transformations from stage to stage, discover and react to errors, and monitor pipeline and stage statistics for delays. Internal subscriptions make building event-driven pipelines across a distributed, enterprise deployment seamless.

Pipeline AI Agent

As more integrations and data pipelines are deployed in an organization, there is more configuration to understand, manage, and debug. The new Pipeline AI Agent is an in-app agent that connects to your language model service of choice and allows you to conversationally summarize, create, and edit pipeline configuration. Once the agent proposes changes, the UI displays exactly what was added or modified, giving you full visibility into how your configuration would be affected. You can debug and test the changes, then accept or reject them and continue iterating. The Pipeline AI Agent makes sophisticated pipeline configuration more accessible across teams and skill levels, with human discretion in the loop at every step.

Engineer Sequential Stages and Conditional Actions

Some data sets must be sourced from multiple systems, where the data from one system is being used in the subsequent read. Based on the values sourced, the pipeline data may go down one or multiple conditional paths with unique structuring, transformation, filtering, and writing to target systems. Pipelines can employ metadata to facilitate dynamic reads as well as curate the presentation of data to the needs of the systems consuming it. Additionally, users can organize a sequence of pipeline stages into a loop, simplifying repetitive and iterative data processing.

Model Data in Pipelines

Industrial data is not only produced as primitive tags or points, but also as complex structures. Instead of decomposing these structures and explicitly mapping them into model instances, Pipelines can validate and re-shape incoming data structures to adhere to model definitions. This is ideal for enforcing data quality or modeling many data sources that produce similar data structures with subtle differences.

Stage Type	Stage Name
Common	Breakup, Filter, Flatten, Model, Model Validation, Size Buffer, Timed Buffer
Control	Delay, Error, For Each, On Change, Return, Subpipeline, Switch, While
Format	CSV, GZip, JSON, Parquet, XML, Zip
I/O	Merge Read, Read, Smart Query, Write, Write New
Parse	CSV, JSON, Parquet, XML
Transform	JavaScript, JSONata
Trigger	API, Callable, Cron, Event, Flow, Namespace, Polled

PIPELINES FEATURES AND FUNCTIONALITY

Use HighByte Intelligence Hub to complete the following tasks:

Trigger pipeline execution based off of event, time interval, schedule, or API call

Read inputs and instances or query the Namespace to build the pipeline value payload

Model, filter, buffer, transform, format, and compress data for the unique data consumption needs of target systems

Use the on-change stage to enable event-based delivery and report-by-exception

Persist variables according to the length of the manufacturing event with state management capabilities

Track individual event executions at a granular level, including execution time and success/failure

Use the switch stage to apply switch statement logic within a Pipeline; in the event of a failed write, easily define how to handle and remediate the error with conditional logic

Employ metadata and logic in sequenced stages to dynamically shape the presentation and delivery of data

Compose custom transformation stages with JavaScript expressions to satisfy advanced use cases

Manage and monitor Pipeline data processing stages, status, performance, and individual event executions in real time

Observe state and errors at-a-glance for each Pipeline stage

Monitor performance with high-level metrics tabulating completions, queues, errors, and more

Ready to try HighByte Intelligence Hub?

Download the software to get hands-on access to all the features and functionality within HighByte Intelligence Hub and start testing in your unique environment.

Download Software

Configure and Move Tailored Datasets with Pipelines

Oversee data movement between systems with Pipelines.

Data Motion

Data Processing

Data I/O

See Pipelines in Action

Enable complex data processing with the Intelligence Hub for a variety of use cases.

Curate Payloads

Simplify Data Pipelines

Pipeline AI Agent

Engineer Sequential Stages and Conditional Actions

Model Data in Pipelines

Manipulate data with these Pipeline Stages.

Stage Type

Stage Name

PIPELINES FEATURES AND FUNCTIONALITY

Use HighByte Intelligence Hub to complete the following tasks:

Ready to try HighByte Intelligence Hub?

Intelligence Hub

Resources

Company

Customers