Professor Franklin
Accessibility settings

Preferences are saved in this browser when storage is available.

Chapter 2 · Reading 8 of 9

Distinguishing evasion, poisoning and model manipulation

Security+ SY0-801 · Objectives 2.6

Yesterday the assistant gave counselors the correct scholarship deadline. Today it confidently supplies last year’s date. The team says the model has not changed. That statement can be accurate even while the system is giving different answers.

Locate the change in the lifecycle

Evasion changes inputs at inference time in a way that produces misleading decisions. Training-data poisoning influences what a system learns during training. Model manipulation can also involve unauthorized changes to weights, adapters, evaluation thresholds or the deployed model version. A retrieval system introduces another surface. Replacing a document can change answers while model weights remain identical. Separate those mechanisms because the evidence needed to diagnose them lives in different systems.

Version the model, configuration, retrieval index and evaluation set independently. Record the provenance and approval of each release. Compare performance on a held-out baseline and recent realistic examples. Restricted write permissions, signed releases and independent review help protect the pipeline. Evaluation data also needs protection. A test set that changes alongside a suspect model can hide a regression.

Investigate a sudden performance drop

Suppose an assistant gives incorrect retention advice only for recent records. Establish the earliest confirmed bad result and whether a service, data feed or prompt template changed around that time. Compare outputs from approved versions using synthetic cases that do not expose personal information. Check whether errors reproduce for several roles or only where document permissions differ. An ordinary data refresh or distribution shift remains a plausible cause.

If a release is implicated, preserve hashes and audit history, pause consequential actions and revert through the approved deployment path. Verify that earlier results return and retest unaffected workflows. Record residual uncertainty about intent. A change in behavior shows that an investigation is warranted. Attribution requires independent evidence of who changed which artifact under what authority.

NIST adversarial machine learning taxonomy

Replaying one question

Set aside real student records and use a synthetic appeal. Keep the approved model, question and evaluation conditions fixed. Inspect the documents selected for the answer. If an outdated handbook displaced the current edition, the librarian responsible for the collection can fix provenance and publication dates. Retraining the model would leave the bad source available. If the correct documents were returned, examine the release manifest and other configuration changes before deciding what to roll back.

A useful incident note names the smallest change that explains the result and the evidence that could disprove it. It also states what the assistant was allowed to do with its answer. An incorrect summary shown to a reviewer and an incorrect instruction executed by a connector carry very different consequences.

Model lineage comparison

Inspect the three records, then use the evidence to answer the question.

1. Baseline

Approved model digest M8 and retriever index R3.

2. Current deployment

Production model digest M8 and retriever index R4.

3. Evaluation set

Errors cluster around documents introduced in R4.

Apply the reading

What comparison would distinguish a retrieval change from a model release?

Compare your reasoning

Hold the approved model and evaluation set fixed while comparing independently versioned retrieval sources.