We are introducing a completely new way of building your pipelines on Instill VDP to shift our tool to focus on better supporting developers in the AI and data domain. The goal of VDP is to help you dynamically build versatile pipelines for complex and unique use cases that require minimal maintenance with high data observability.

Recipe as Code

The new pipeline builder UI allows you to edit YAML recipes next to the pipeline preview for improved debugging and data flow visualization. This means you can edit recipe.yml files in your own editors like VSCode and simply import them into VDP, streamlining the recipe sharing process. Additionally, the editor offers smart hints and syntax highlighting, making it easier to build and modify recipes efficiently.

Watch this tutorial on how to build recipes in the new pipeline editor:

Quick tips:

• Use Control+O or Cmd+O shortcut to add components easily

• Use Control+K or Cmd+K shortcut to search for things

• Use Control+S or Cmd+S shortcut to save your recipe manually

• Use Control+Enter or Cmd+Enter shortcut to run the pipeline

• Click the Getting Started button at the bottom-left corner to view the guide in the pipeline preview panel.

High Observability

The pipeline preview offers a visual representation of data flow between components, striking a balance between code-level control and visual management. This feature is particularly beneficial for data engineers working on complex pipelines, as it provides real-time streaming feedback to simplify debugging. The preview also enhances component presentation, making it easier to build, test, and maintain sophisticated data pipelines efficiently.

You can try exploring the recipe of the structured-text-summarizer pipeline to transform a paragraph of unstructured text into structured JSON output in the new pipeline editor.

Learn more about this feature by joining our Discord where we showcase these new features and you can ask questions in the Weekly Office Hours.