Experimental evaluation exercise ( 3 models x 3 drugs x 2 context conditions)

Sheet 2.

Please provide your answers to these 3 questions according to the format set out:

Question 1

1. Summary With Medical Context

· Provide a brief summary of the model’s answers using medical context

2. Summary Without Context

· Provide a brief summary of the model’s answers without medical context

3. Comparison Score Table

Include a table comparing model performance with vs. without medical context across four dimensions: (Rate: 0 - 5)

· Accuracy

· Completeness

· Clarity

· Consistency

4. Findings

· Highlight 2 to 3 key errors found in the model outputs.

· Describe how context improved accuracy, clarity, or reliability.

5. Ethical Reflection

· Discuss safety risks of incorrect or incomplete drug information.

· Explain potential hallucination risks and their real-world consequences.

Upload your analysis file.

Question 2.

Create a NABC-format presentation

The slides must follow the NABC Innovation framework as discussed in the lecture:

N — Need

· What user problem are you solving?

· Who needs a safer drug-information tool? (target user)

A — Approach

· Your proposed solution: “A local AI-powered drug information assistant enhanced with medication context.”

· How it works? How it works differently with context.

B — Benefits

· Benefits to end users: e.g.( accuracy, safety, clarity, offline reliability)

· Evidence from your evaluation: improvement with context

C — Competition

· Existing alternatives (e.g. websites, other tools)

· What makes your NABC solution better?

Your 5 slides (Title + NABC) must clearly follow N → A → B → C and use data from your evaluation to justify the idea.

Upload File

Question 3

Create a 6 slide SWOT Presentation (title + SWOT) that summarizes your findings from the capstone project evaluating local LLMs with and without RxNorm drug-fact context.

Include:

1. Key Findings Slide

· Summarize the  most significant evaluation results: accuracy gains, clarity improvements, reduction in hallucinations, or errors that persisted.

2. SWOT Analysis Slide(s)

· Present a clear SWOT analysis for the hypothetical product:“A local AI-powered drug-information assistant enhanced with RxNorm context.”

Upload File