Real-Time Monitoring

In today’s fast-paced AI environments, waiting to discover issues isn’t an option. Picept’s real-time monitoring gives you immediate visibility into your AI system’s performance, letting you catch and address issues before they impact your users.

Alert Configuration

Alerts in Picept are configured as part of your evaluation payload. Here’s how to set them up:

"alert": {
    "enabled": True,
    "options": {
        "email": ["team@company.com"],
        "webhook": "https://your-alert-endpoint.com",
        "on_type_status": [{"sentiment analysis": False}]  # Trigger alert when sentiment analysis fails
    }
}

Let’s break down each component:

Alert Options

enabled: Master switch to turn alerts on/off
options: Configure how and when you want to be notified
- email: List of email addresses to receive notifications
- webhook: URL where alert data should be sent
- on_type_status: Specify which evaluator failures should trigger alerts

Here’s a complete example showing alerts in context:

payload = {
    "evaluation_name": "customer-feedback",
    
    "alert": {
        "enabled": True,
        "options": {
            "email": ["team@company.com"],
            "webhook": "https://your-alert-endpoint.com",
            "on_type_status": [{"sentiment analysis": False}]
        }
    },
    
    "dataset": {
        'file_name': "data-x27",
        "id": "S3:xxx1234",
        "response": 'The restaurant service was disgraceful...'
    },
    
    "evaluators": {
        "content safety": {
            "response": "response",
            "explanation": True,
            "judge_model": "gpt-4o-mini[openai]",
            "criteria": {
                "topic_detection": {
                    "enabled": True,
                    "expected_topics": ['mortgage loans']
                }
            }
        },
        "sentiment analysis": {
            "input_text": "response",
            "explanation": True,
            "judge_model": "gpt-4o[openai]",
            "passing_criteria": "Positive"
        }
    }
}

When an evaluation fails the specified criteria (in this case, negative sentiment), Picept will:

Send an email to the specified addresses
POST the alert data to your webhook endpoint
Include detailed information about why the evaluation failed