DATASET¶
Purpose: The Dataset module is the core schema management layer of the ETL platform. It allows users to define, organize, and manage MongoDB dataset structures — acting as the foundation for all data pipelines, jobs, and connections in the system. Before any data can flow through the platform, a dataset schema must exist to describe its shape and rules.
Datasets Page (Main View)¶
This is the Datasets listing page — the central hub where all dataset schemas are created, searched, and managed. It is part of a broader ETL (Extract, Transform, Load) management platform backed by MongoDB.
- Title:
Datasets - Subtitle: "Manage your MongoDB dataset schemas"
- Top-right CTA:
+ Create Datasetbutton — the primary entry point for adding a new dataset
Toolbar¶
| Element | Description |
|---|---|
| Search bar | Quickly locate a dataset by name, ID, or the creator's identity |
| Items dropdown | Displays the total count of datasets (e.g., 12 Items) with pagination or view control |
| Sort dropdown | Orders the list by Creation Date — useful for finding the latest schemas |
| Filter icon | Opens additional filtering options for narrowing down results |
| Refresh button | Manually reloads the dataset list to reflect the latest changes |
Empty State¶
When no datasets have been created yet, the page displays a friendly prompt to get started:
🗂️ No datasets yet "Get started by creating your first dataset schema" [ Create Your First Dataset ] (teal CTA button)
Create New Dataset Modal¶
This modal is the schema builder interface — it lets users define the name, visibility, and field-level structure of a new dataset before it is saved to the system. Every field configured here directly determines how data is stored, validated, and queried in MongoDB.
1. Dataset Identity¶
| Field | Description |
|---|---|
| Dataset Name (required) | A text input for naming the dataset — limited to 30 characters (tracked via a live 0/30 counter). The name should be descriptive and unique within the organization. |
| Visibility Level (required) | A dropdown that controls access scope. Defaults to Organization, meaning all members can view or use this dataset. |
2. Schema Fields¶
This section is the heart of the modal — where you define the structure of your dataset by configuring each field (column) it will contain.
Column Definitions¶
| Column | Purpose |
|---|---|
| Field Name (required) | The internal key/identifier used in the database (e.g., customer_id). Should follow naming conventions — no spaces. |
| Label | A friendly, human-readable name shown in the UI. Defaults to Auto-generated if left blank. |
| Data Type (required) | Defines the kind of data the field holds (e.g., String, Integer, Boolean, Date). Must be selected from a dropdown. |
| Default Value | An optional value automatically applied when no data is provided for this field during ingestion. |
| Unique | When checked, enforces that no two records can have the same value in this field — useful for IDs or emails. |
| Allow Null | When checked, permits this field to be left empty. Uncheck to make the field mandatory at the data level. |
| Allow Index | When checked, MongoDB will index this field, significantly speeding up search and filter operations on it. |
| Max Length | Caps the number of characters or size of data allowed in this field — useful for enforcing data quality. |
| Actions | A 🗑️ delete icon to remove a field from the schema before saving. |
Input Mode Toggle¶
A Manual / File Upload toggle offers two ways to define fields: - Manual — Add and configure fields one by one using the form rows - File Upload — Upload a structured file (e.g., CSV) and let the system auto-detect and populate the schema fields
Status Bar¶
"1 field(s) defined • You can switch to Manual mode to add more fields"
Indicates the current number of configured fields and guides the user on how to add more.
3. Footer Actions¶
| Button | Behavior |
|---|---|
| Cancel | Closes the modal and discards all unsaved changes |
| Create Dataset (primary, teal) | Submits and saves the dataset schema — remains disabled until all required fields are properly filled |
⚠️ "Required fields must be filled before saving." — A validation reminder ensuring incomplete schemas cannot be accidentally saved.