Uptime Monitoring

Aexy's uptime monitoring module provides endpoint health checking for HTTP, TCP, and WebSocket services. Monitors run on configurable intervals and automatically create support tickets when services go down.

Features#

• Multi-Protocol Checks: HTTP, TCP, and WebSocket endpoint monitoring
• Configurable Intervals: 1, 5, 15, 30, or 60 minute check intervals
• Automatic Incident Management: Incidents created after consecutive failures
• Ticket Integration: Automatically creates and closes support tickets
• Notifications: Slack, email, and webhook alerts
• SSL Monitoring: Track SSL certificate expiration (HTTP checks)
• Post-Mortem Notes: Root cause and resolution documentation

Check Types#

HTTP Check#

Monitors HTTP/HTTPS endpoints with configurable settings.

Collected Metrics:

• Response time (ms)
• Status code
• SSL certificate expiry (days remaining)
• Error messages

TCP Check#

Monitors TCP port availability.

Collected Metrics:

• Connection time (ms)
• Connection success/failure
• Error type (timeout, connection_refused)

WebSocket Check#

Monitors WebSocket endpoints with optional message validation.

Collected Metrics:

• Connection time (ms)
• Handshake success
• Response validation result
• Error messages

Incident Management#

How It Works#

1. Consecutive Failures: Monitor tracks consecutive failed checks
2. Threshold Reached: When failures >= threshold (default: 3), incident is created
3. Ticket Created: Support ticket automatically created with incident details
4. Notifications Sent: Slack/email/webhook alerts dispatched
5. Recovery Detected: When check succeeds, incident is resolved
6. Ticket Closed: Linked ticket automatically closed with resolution notes

Failure Flow#

Check fails
    ↓
Increment consecutive_failures
    ↓
consecutive_failures >= threshold?
    ├─ No → Schedule next check
    └─ Yes → Create incident
                ↓
            Create support ticket
                ↓
            Send notifications

Recovery Flow#

Check succeeds
    ↓
Has ongoing incident?
    ├─ No → Reset counters
    └─ Yes → Resolve incident
                ↓
            Close linked ticket
                ↓
            Send recovery notifications (if enabled)

Incident States#

Monitor Status#

Notifications#

Supported Channels#

Notification Events#

• Incident Created: When threshold failures reached
• Incident Acknowledged: When team acknowledges incident
• Incident Resolved: When service recovers (if notify_on_recovery enabled)

Webhook Payload#

{
  "event": "incident.created",
  "monitor": {
    "id": "uuid",
    "name": "API Server",
    "check_type": "http",
    "url": "https://api.example.com/health"
  },
  "incident": {
    "id": "uuid",
    "status": "ongoing",
    "started_at": "2024-01-15T10:30:00Z",
    "error_message": "Connection timeout after 30s"
  },
  "workspace_id": "uuid"
}

Ticket Integration#

Automatic Ticket Creation#

When an incident is created, a support ticket is automatically generated:

• Title: [UPTIME] {monitor_name} is down
• Severity: high
• Priority: urgent
• Description: Monitor details, error message, timestamp
• Team: Assigned to configured team (if set)

Automatic Ticket Closure#

When service recovers:

1. Resolution comment added to ticket:

   Service has recovered.
   
   Duration: 15 minutes
   Started: 2024-01-15 10:30:00 UTC
   Resolved: 2024-01-15 10:45:00 UTC
   Total checks during incident: 15
   Failed checks: 15
   

2. Ticket status set to closed
3. Recovery notifications sent (if enabled)

API Endpoints#

Monitors#

GET    /workspaces/{id}/uptime/monitors                    # List monitors
POST   /workspaces/{id}/uptime/monitors                    # Create monitor
GET    /workspaces/{id}/uptime/monitors/{monitor_id}       # Get monitor
PATCH  /workspaces/{id}/uptime/monitors/{monitor_id}       # Update monitor
DELETE /workspaces/{id}/uptime/monitors/{monitor_id}       # Delete monitor
POST   /workspaces/{id}/uptime/monitors/{monitor_id}/pause # Pause monitoring
POST   /workspaces/{id}/uptime/monitors/{monitor_id}/resume # Resume monitoring
POST   /workspaces/{id}/uptime/monitors/{monitor_id}/test  # Run immediate test
GET    /workspaces/{id}/uptime/monitors/{monitor_id}/checks # Get check history
GET    /workspaces/{id}/uptime/monitors/{monitor_id}/stats # Get monitor stats

Incidents#

GET    /workspaces/{id}/uptime/incidents                   # List incidents
GET    /workspaces/{id}/uptime/incidents/{incident_id}     # Get incident
PATCH  /workspaces/{id}/uptime/incidents/{incident_id}     # Update incident
POST   /workspaces/{id}/uptime/incidents/{incident_id}/acknowledge # Acknowledge
POST   /workspaces/{id}/uptime/incidents/{incident_id}/resolve     # Resolve

Statistics#

GET    /workspaces/{id}/uptime/stats                       # Workspace stats

Database Tables#

uptime_monitors#

Stores monitor configurations.

uptime_checks#

Stores individual check results (time-series data).

uptime_incidents#

Stores incident records linked to tickets.

Temporal Schedules#

Defined in backend/src/aexy/temporal/schedules.py:

{
    "id": "uptime-process-due-checks",
    "activity": "process_due_checks",
    "input_module": "aexy.temporal.activities.uptime",
    "interval": timedelta(seconds=60),
    "queue": TaskQueue.OPERATIONS,
},
{
    "id": "uptime-cleanup-old-checks",
    "activity": "cleanup_old_checks",
    "input_module": "aexy.temporal.activities.uptime",
    "interval": timedelta(hours=24),
    "queue": TaskQueue.OPERATIONS,
},

Activity Descriptions#

Inspect runs and history in the Temporal UI at http://localhost:8080.

Frontend Pages#

Configuration#

Environment Variables#

No additional environment variables required. Uses existing:

• REDIS_URL - For caching and LLM rate limiting
• TEMPORAL_ADDRESS - For the Temporal workflow engine that runs checks
• DATABASE_URL - For PostgreSQL storage

Check Intervals#

Default Settings#

File Structure#

frontend/src/
├── app/
│   ├── (app)/uptime/
│   │   ├── page.tsx                    # Dashboard
│   │   ├── monitors/
│   │   │   ├── page.tsx                # Monitors list
│   │   │   └── [monitorId]/page.tsx    # Monitor detail
│   │   ├── incidents/
│   │   │   ├── page.tsx                # Incidents list
│   │   │   └── [incidentId]/page.tsx   # Incident detail
│   │   └── history/page.tsx            # Check history
│   └── products/uptime/page.tsx        # Product landing page
└── lib/
    └── uptime-api.ts                   # Uptime API client

backend/src/aexy/
├── models/uptime.py                    # SQLAlchemy models
├── schemas/uptime.py                   # Pydantic schemas
├── services/
│   ├── uptime_service.py               # Business logic
│   └── uptime_checker.py               # Check executors
├── api/uptime.py                       # REST endpoints
└── temporal/activities/uptime.py       # Temporal activities (process_due_checks, execute_check, cleanup_old_checks)

Troubleshooting#

Monitor Not Running Checks#

1. Verify monitor is active (is_active = true)
2. Check the Temporal worker is running (uptime checks run as Temporal scheduled activities):

   docker compose logs temporal-worker
   

3. Open the Temporal UI at http://localhost:8080 and verify the uptime schedule is active and recent runs are succeeding.

Checks Timing Out#

1. Increase timeout_seconds on the monitor
2. Check if endpoint is accessible from container network
3. Verify any firewall rules allow outbound connections

Ticket Not Created#

1. Verify consecutive_failures_threshold has been reached
2. Check ticket service is accessible
3. Review Temporal worker logs for errors

Notifications Not Sending#

1. Verify notification channel is configured on monitor
2. Check Slack integration is connected (for Slack alerts)
3. Verify webhook URL is reachable
4. Review Temporal worker logs

SSL Errors#

1. If certificate is valid but failing, try setting verify_ssl = false
2. Check certificate chain is complete
3. Verify system trust store includes required root CAs

WebSocket Connection Failures#

1. Verify WS URL uses correct protocol (ws:// or wss://)
2. Check if server requires specific subprotocols
3. Test connection manually with wscat:

   wscat -c wss://your-endpoint.com/ws
   

Access Control#

Uptime monitoring is enabled for:

• Engineering Bundle: Full access
• Full Access Bundle: Full access

Permission required: can_view_uptime

Configure access in Settings > Access for your workspace.