What I learned building SentinelOps
Isolation Forest scores a seven-number vector. The useful part was keeping the model out of the API, and refusing to call a distance from a synthetic baseline a root cause.
SentinelOps was my final-year project while I was pursuing a master’s in Information Technology at MIVA Open University. SentinelOps is an incident platform I built as three processes: a Next.js console, a NestJS API, and a FastAPI service that scores one feature vector with Isolation Forest. The useful lesson was not “add a model.” It was where the model is allowed to sit, and what the score is allowed to mean.
The model does not read the database
Nest owns MongoDB. On a prediction it loads the service, recent metrics, anomalies, and the dependency graph, then builds seven numbers in a fixed order: error rate, latency, CPU, memory, an existing anomaly score, dependency risk, and a count of error logs. That vector is the contract. Python never opens Mongo.
If I change how error rate is derived, I change it in Nest. The model still receives the same seven fields. Retraining is the same kind of split. train.py fits Isolation Forest and writes model.pkl. FastAPI loads that file on startup. Nest does not change. A new baseline is a new file and a restart of the Python process, not a deploy of the API.
The baseline is synthetic
The forest is fit on 800 synthetic healthy samples, not on production incidents. Error rate sits near 1 percent, latency near 120 milliseconds, CPU near 35 percent, memory near 45 percent, and log errors around a Poisson mean of 2. The score is how far a live vector sits from that envelope.
scikit-learn’s decision_function is higher when a point looks normal, so the service flips the sign and squashes it into a 0–1 anomaly score. predict() returns −1 for an outlier. The observation is treated as anomalous when that label is −1, or when the scaled score crosses the configured threshold.
I would not call that a root cause. It is a distance from a healthy envelope I invented. If live traffic is healthier, or noisier, than those normals, the score moves for the wrong reason. The next real step is to refit on healthy production windows. Adding another algorithm does not fix a baseline you have not earned.
Comparison models stay off the request
One-Class SVM and local outlier factor are useful, and they are easy to leave on the hot path “just to see.” I kept them on an offline comparison: FastAPI exposes GET /eval/compare, and Nest exposes that as GET /api/v1/ai/evaluation. Same seven features. Production scoring is Isolation Forest only. An evaluation that is allowed to be slow is a different product from a predict call an operator is waiting on.
The candidates are hypotheses
After the score, the Python service ranks explanations. A metric becomes a candidate only when it crosses a fixed line:
- error rate at 0.05
- latency at 400 milliseconds
- CPU at 80 percent
- memory at 85 percent
- error logs at 10 in the window
A dependency becomes a candidate from its health, error rate, latency, and share of traffic. Each candidate is written as a ranked hypothesis, not causal proof. I wanted that sentence in the payload, not only in a comment. An operator who reads “root cause” will treat a threshold as a finding.
When the model is down, the API still answers
If FastAPI is unreachable and the fallback flag is on, Nest returns a rule-based explanation instead of failing the request, and the response says which mode it used. A missing model should weaken the explanation. It should not take the console down.
Research measurement is a different pipe
I also needed usability numbers for the study: whether an operator finished the task, how long it took, a few Web Vitals, and a System Usability Scale score. Those events do not go into logs, metrics, or OpenTelemetry. The browser posts an allowlisted event. The user id comes from the JWT, not from the client. The rows expire after 90 days. The summary endpoint returns aggregates to admin and devops roles, not the raw events. Mixing that into operational telemetry would have made both the incident data and the study data worse.
What I would repeat
Keep the model in a process that can be restarted without the API. Make the feature vector a versioned contract. Keep bake-offs off the request. Label a hypothesis as a hypothesis. And do not build the story of the system on a score whose baseline you have not earned yet.
The three repos: sentinel-ops-api, sentinel-ops-ai, and sentinel-ops-fe.