DELMIA AI Quality Assurance Engineer

 Perform quality assessment of DS DELMIA applications, validating both AI-driven and rule-based features against DS quality standards and real customer usage.

• Define and execute test strategies for AI/ML capabilities, covering:
– Functional correctness
– Model behavior consistency
– Data dependency and sensitivity
– Edge cases and failure modes

• Model Behavioral Testing: Design test suites to evaluate model performance (Precision, Recall, F1-score) specifically within applications designed for Digital Manufacturing

• Agentic Evaluation :
Metric-Based Validation: Define and implement unit tests for LLM outputs using metrics such as Faithfulness, Answer Relevancy, Contextual Precision, and Hallucination scores.

• Tool-Calling Accuracy: Rigorously test the agent’s ability to select and execute the correct "tools" or APIs within the application environment based on user intent

• Regression Testing for LLMs: Use "Golden Datasets" to ensure that updating the underlying model does not degrade the agent's reasoning or domain-specific knowledge

• Distinguish between model limitations, data issues, and software defects, and communicate findings clearly to software and data engineering teams.
• Leverage customer usage data, telemetry, and feedback to continuously refine AI test coverage and improve quality effectiveness.

• Statistical Evaluation: Statistical way to present Test results for AI features, multiple-pass evaluations to find a confidence interval.

Cross-Functional Collaboration

• Work closely with AI/ML engineers, data stewards, and product teams to understand model intent, assumptions, and limitations.
• Contribute to defining AI quality metrics, acceptance criteria, and certification guidelines aligned with DS standards.