Technical Test Design

Comprehensive framework for evaluating technical candidates through practical implementation challenges that mirror real-world business scenarios and production requirements.

Evaluation Criteria Framework

Beyond Traditional Coding Tests

Modern AI engineering requires assessment beyond algorithm and data structure knowledge:

Business Context Understanding

• Ability to translate business requirements into technical solutions
• User experience design for non-technical stakeholders
• Integration complexity management

Multi-Tool Orchestration

• Coordinating APIs, databases, and external services
• Error handling across system boundaries
• Graceful degradation patterns

Production Readiness

• Security best practices implementation
• Configuration management
• Deployment and monitoring considerations

Example: "La Boulangère Augmentée" Test

Scenario Design: Build AI agent for bakery management

• Business Owner: Madeleine Crouton (non-technical user)
• Real Data: 4 interconnected Notion databases
• Practical Tools: Stock, sales, email, weather, production planning

Evaluation Dimensions:

1. Architecture Quality

   • Framework selection and justification
   • Separation of concerns
   • Error handling and edge cases
   • Code organization and modularity

2. Integration Complexity

   • notion-api-integration with multiple databases
   • Email system integration with confirmation workflows
   • External API integration (weather, etc.)
   • Multi-step business logic coordination

3. User Experience

   • Conversational interface design
   • Human-in-the-loop approval workflows
   • Error messages and guidance
   • Response time and feedback

4. Production Patterns

   • Environment configuration management
   • Security practices (API key protection)
   • Logging and observability
   • Deployment readiness

Implementation Assessment

Required Scenarios

Tests typically include mandatory scenarios that candidates must complete:

Scenario 1: Data Analysis

• "Combien de croissants j'ai vendu la semaine dernière ?"
• Tests: Database querying, data aggregation, presentation

Scenario 2: Business Logic

• "Il me reste combien de beurre ?"
• Tests: Stock management, threshold detection, reorder logic

Scenario 3: External Communication

• "Prépare-moi un mail pour commander 50kg de farine T65"
• Tests: Multi-step workflows, confirmation patterns, external integration

Advanced Evaluation Criteria

Framework Expertise

• Deep understanding vs surface-level implementation
• Appropriate pattern selection for use case
• Performance and scalability considerations

Error Handling Maturity

# Basic (insufficient):
result = api.call()

# Advanced (evaluated positively):
try:
    result = await api.call_with_retry()
except APIRateLimitError:
    await asyncio.sleep(60)
    result = await api.call()
except APIConnectionError as e:
    return fallback_response(f"Service temporarily unavailable: {e}")

Security Awareness

• API key management and rotation
• Input validation and sanitization
• RBAC and permission patterns
• Audit logging for sensitive operations

Multi-Framework Comparison Strategy

Advanced Assessment: Implementing same solution across multiple frameworks

• langgraph-agent-patterns vs OpenAI Agents SDK vs CrewAI
• Framework-appropriate patterns and idioms
• Performance and maintainability trade-offs
• Team expertise and learning curve considerations

Comparative Evaluation Matrix

Framework	Architecture	Tool Integration	State Management	Learning Curve
LangGraph	Graph-based	Native tool calling	Complex state graphs	Steep
OpenAI Agents	Function-based	Built-in functions	Session-based	Moderate
CrewAI	Multi-agent	Role-based tools	Agent coordination	Moderate

Beyond Code: Holistic Assessment

Communication and Documentation

• README quality and completeness
• Code comments and inline documentation
• Architecture decision records
• User guides and setup instructions

Business Acumen

• Understanding of domain-specific challenges
• Prioritization of features based on business impact
• Scalability and maintenance considerations
• Cost optimization awareness

Collaboration Indicators

• Git workflow and commit quality
• Code review readiness
• Testing strategy and implementation
• CI/CD integration patterns

Red Flags in Submissions

Technical Debt Indicators

• Hard-coded credentials or configuration
• Missing error handling
• Monolithic architecture without separation
• No testing or validation

Inexperience Markers

• Copy-paste implementation without understanding
• Inappropriate framework choice for requirements
• Over-engineering simple scenarios
• Under-engineering complex integrations

Post-Test Discussion Framework

Technical Deep Dive

• Architecture decision rationale
• Trade-offs and alternative approaches
• Scaling considerations and bottlenecks
• Integration challenge solutions

Business Context

• User experience design decisions
• Feature prioritization rationale
• Production deployment strategies
• Maintenance and evolution plans

See also

• ai-sisters - Company context implementing this framework
• langgraph-agent-patterns - Technical patterns being evaluated
• notion-api-integration - Integration complexity assessment