Tasks

Flexible Access for Different Users

Plexus accommodates different types of users by providing multiple ways to work with tasks:

• Command Line Users: Technical team members can use Plexus's command-line tools directly, working with YAML configurations and advanced features
• Worker Daemon Users: Organizations can run Plexus worker daemons that plug into the task dispatch system, allowing tasks to be managed through the dashboard
• Dashboard Users: Team members can initiate and monitor tasks entirely through the web interface, without needing technical expertise

Common Infrastructure for All

While team members might use different tools based on their roles and expertise, everyone operates on the same underlying infrastructure:

• Shared Terminology: Whether using the dashboard or command line, everyone uses the same concepts of Items, Sources, Scorecards, and Evaluations
• Unified Business Process: Data scientists working with data sources, ML engineers fine-tuning models, and account representatives coordinating with clients all connect through the same task system
• Seamless Collaboration: Technical consultants can set up advanced configurations that non-technical team members can then operate through the dashboard

Connecting Your Organization

The task system serves as the coordination layer that brings together team members with different skills and responsibilities. Whether someone is creating specialized ML models, managing client relationships, or analyzing problem domains, the task system ensures everyone can work together effectively while using the tools that best suit their needs.

Task Distribution

When you initiate an operation—whether through the dashboard or command line—here's what happens:

• The operation is converted into a task with a unique identifier
• Plexus examines the task requirements (like whether it needs GPU access or specific data access)
• The task is routed to an appropriate worker computer that can handle those requirements
• The worker processes the task and sends back regular progress updates

Worker Computers

Worker computers are machines running the Plexus worker daemon. They can be:

• Local Machines: Computers in your office running the worker daemon
• Cloud Servers: Remote machines set up specifically for processing tasks
• Specialized Hardware: Machines with GPUs or other specific capabilities

Each worker advertises its capabilities (like having a GPU or access to certain data), allowing Plexus to route tasks to the most appropriate worker.

Task Communication

Tasks maintain constant communication between the dashboard and workers:

• Status Updates: Workers regularly report their progress
• Resource Usage: Information about CPU, memory, and GPU usage
• Error Reporting: Detailed information if something goes wrong
• Results Storage: Completed task results are stored for later access

Real-Time Progress Tracking

As tasks run, you can monitor their progress in real time. The dashboard shows:

• Progress bars showing overall completion and current stage
• Time estimates based on current processing speed
• Detailed status messages explaining the current operation
• Performance metrics like items processed per second

1. Creation

Tasks are created when you initiate operations through the dashboard. Each task is assigned a unique ID and configured based on your requirements.

2. Queuing

Tasks are intelligently queued and distributed to specialized worker nodes. Our system matches tasks with workers that have the right capabilities (like GPU access for intensive operations).

3. Execution

Worker nodes process tasks and provide continuous updates. You can monitor progress in real-time through the dashboard, with detailed status information at each stage.

4. Completion

When processing finishes, tasks are marked as complete and their results are stored securely. You can access these results through the dashboard for review and analysis.