Theme

The ongoing digital engineering transformation, combined with AI, can help us build better, more complex systems faster and at lower cost. This demands a paradigm shift from document-based, siloed practices to using authoritative sources of truth for data and decision management across the system lifecycle. Realizing this vision faces significant challenges, including limited understanding of digital engineering ROI and the concerning limitations and failure modes of current AI—especially LLMs—which often lack context and domain expertise.

Key Points

  1. The systems engineering community is often blamed for project failures, schedule overruns, and cost blowouts due to complexity, as seen in notable cases like the F-35 program.
  2. Digital engineering transformation is viewed as a remedy, moving from document-centric processes to authoritative sources of truth that manage data and decisions across the lifecycle.
  3. A NAVSEA-sponsored study indicated digital engineering could cut design cycles for large-scale Navy assets by 50% and make outcomes more predictable.
  4. Benefits are unevenly distributed, with potential time savings up to 90% in system-level modeling, 50% in information exchange, and 34% in review meetings.
  5. AI can speed digital engineering adoption, but experiments with LLMs reveal failure modes: premature requirement definition, wildly inaccurate numerical estimates, and a tendency to overspecify solutions.
  6. In a requirements engineering study, an LLM identified only 35% of actual issues (true positives) and missed 65% (Type II errors), largely due to weak contextual linkages and limited domain judgment.
  7. Despite the unreliability of off-the-shelf LLMs, specialized, agent-based AI systems that embed organizational processes and constraints show promise.
  8. AI can enable a bottom-up systems engineering approach, rapidly generating solutions from existing commercial off-the-shelf parts.
  9. For mission engineering—an NP-hard challenge—AI can be highly effective by combining reinforcement learning with high-fidelity digital mission models.
  10. Key integration challenges include efficient scaling, shortages of skilled personnel, and representing interdisciplinary knowledge with contextual awareness.

Highlights

  • "Why don't we stop doing systems engineering on these mutually exclusive documents and analytical methods, and instead start using authoritative sources of truth for data and decision management across the system's life cycle." -- [Insert Speaker Name]

Topics

  1. The Promise and Challenge of Digital Engineering
  2. The systems engineering community struggles with complexity, leading to delays and cost overruns. Digital engineering is proposed as a solution, shifting from siloed, document-based work to an integrated platform using an authoritative source of truth. However, organizations poorly understand ROI and how to adapt practices post-transformation.
  3. Quantifying the Impact of Digital Engineering Transformation A NAVSEA-sponsored study on naval asset sustainment programs found that implementing digital engineering could reduce design cycles by 50% and increase predictability. Gains are uneven across activities, with system-level modeling and analysis seeing reductions up to 90%, information exchange by 50%, and review meetings by 34%.
    • The study examined the workflow of a Navy systems engineering team handling design changes for assets like aircraft carriers.
    • Analysis showed digital engineering could halve design cycle times for large-scale Navy asset sustainment.
    • The largest gains occur in system-level modeling and analysis.
  4. The Perils and Limitations of AI in Systems Engineering AI can accelerate digital engineering adoption, but off-the-shelf models show significant failure modes. In one study, an AI prematurely defined requirements, gave extremely inaccurate numerical estimates, and overspecified solutions. In another, an LLM acting as a requirements assistant had a 65% Type II error rate, missing most issues due to lack of contextual awareness and domain judgment.
    • AI-generated systems engineering artifacts were rated 99% similar to human-expert outputs by NLP algorithms.
    • Critical failure modes include premature requirements definition, grossly inaccurate numerical estimates, and overspecification.
    • The LLM’s true positive rate for identifying requirement issues was only 35%, missing 65% of real problems (Type II errors).
    • The primary weakness is failing to establish contextual linkages and exercise domain-specific judgment—an absence of situational awareness.
  5. Alternative Applications of AI in Engineering Beyond current LLMs, AI supports new engineering paradigms. One is a bottom-up design process where AI rapidly composes solutions from libraries of commercial off-the-shelf parts, then tests them in high-fidelity digital environments. Another is mission engineering, where reinforcement learning combined with digital mission models can tackle NP-hard problems like tactical deployment and portfolio planning.
    • AI can enable bottom-up design, assembling systems from available commercial parts.
    • Mission engineering is NP-hard; reinforcement learning paired with digital models can be highly effective.
    • Applications include mission planning, tactical execution, and potentially portfolio planning and asset acquisition.

Suggestions

  • As systems engineers, we must be rigorous about verification and validation.
  • Do not adopt off-the-shelf LLMs without careful consideration; if used, they must be designed and constrained thoughtfully.

AI Suggestions

  • The core of this lesson is applying “AI-Enhanced Digital Engineering.” Start by evaluating a current manual process and identifying opportunities for digital integration, then run a small-scale pilot to grasp AI-Enhanced Digital Engineering.
  • Core content of “AI-Enhanced Digital Engineering”: Combine an authoritative source of truth (digital engineering) with AI capabilities to accelerate and improve systems design, analysis, and mission planning—while critically managing AI’s limitations.

Extracurricular Resources:

AI Digital Engineering Transformation
Verified by ExactMetrics