--- title: "Do LLMs beat classic models at spotting network attacks?" canonical: https://papersays.com/p/do-llms-beat-classic-models-at-catching-network-attacks date: 2026-10-11 modified: 2026-10-12 hub: "AI & Machine Learning" doi: "10.1007/s10462-025-11432-2" license: "CC BY 4.0" --- # Do LLMs Beat Classic Models at Catching Network Attacks? > Small language models learned to flag network attacks and scored above 95 percent. But a plain classic method, XGBoost, still won, faster and far cheaper. Researchers at the University of Zaragoza tested whether small language models can spot attacks in smart-home network traffic better than classic machine learning. Fine-tuned language models scored above 95 percent on the F1 measure and beat every deep learning model, but XGBoost, a classic tree-based method, still came out on top at 0.9696 while using about 4% of the processor. The language models were also about 10,000 times slower. The evidence is limited: one dataset, small models, and a single run per model. ## How did the researchers test language models on network attacks? The authors used a public dataset, CIC IoT 2023, recorded from 105 real smart devices such as cameras, sensors and smart speakers. They kept four kinds of traffic: ordinary household traffic and three attack types, namely flooding attacks (DDoS), the Mirai botnet, which hijacks smart gadgets, and scanning, where an attacker probes a network for weak spots. Each connection was summarised as a single record, called a flow, with facts such as its duration, bytes sent each way and how it ended. Timestamps, IP addresses and ports were removed, so no model could simply memorise where attacks came from. Nine models then faced the same data: three classic machine learning methods (including XGBoost), three deep learning networks and three small language models (GPT-2, GPT-Neo-125M and LLaMA-3.2-1B). The language models read each flow as a line of text and were mostly fine-tuned, meaning a general model gets brief extra training on one job, on a single graphics card with 6 GB of memory. Every model trained on 10,000 or 50,000 flows per class and was tested on unseen flows. ## Which model was best at detecting attacks? XGBoost had the best scores overall. Its F1-score, a measure that rewards catching attacks while avoiding false alarms, reached 0.9696 with the larger training set, and it used about 4% of the processor during testing. The language models did well too. Their scores stayed above 95 percent, and the best one, GPT-Neo after fine-tuning, reached 0.9618 on the four-way task. That beat every deep learning model; two of them, GRU and LeNet-5, fell below 0.85 on the same task, mostly by confusing ordinary traffic with scanning. The big difference was speed. The fastest language model setup handled at most 517 flows per second, against 1,581,027 for XGBoost, making the language models about 10,000 times slower. Their mistakes were also slightly more frequent: in the attack-or-not test, XGBoost let 3.3% of attack traffic pass as normal, against 3.8% for the best language model. ## Should security tools use language models for intrusion detection? Intrusion detection software often runs close to the devices it protects, such as a small box on a home or office network, where memory and processing power are limited. In this study the classic models needed under a megabyte of disk space and no graphics card, while the language models needed gigabytes of storage and a graphics card. A fair reading is that, for sorting large volumes of structured traffic, the classic method remains the one to beat: faster, cheaper to run and slightly more accurate. The language models still earned a place in the conversation, since they came close on accuracy and held steady even with the smaller training set. The authors suggest a combined design for future work, in which a fast classic model sorts traffic and a language model is brought in for the hard or ambiguous cases, possibly to describe an alert in plain language. ## What are the limits of this intrusion detection study? Several limits shape how far the results reach: - **Small models only.** The largest language model had about 1.3 billion parameters; larger models such as GPT-4 or LLaMA-2-7B were not tested and may do better. - **Single runs.** Each model was trained and scored once, without repeated runs or cross-validation, so the small gaps between models carry no statistical test. - **One dataset, known attacks.** All results come from one dataset of known attack types, so new kinds of attack or other networks could change the picture. - **Opaque decisions.** The explanation method used (SHAP) only worked on the tree models, so how the language models reached their decisions is unknown. The study also does not show that language models are useless for security, since they came within about a point of the winner. It does not show that a language model can explain its alerts in plain language; that was left for future work. ## Takeaways - **SAYS** Small language models scored above ninety-five percent at spotting attacks, but XGBoost still won at almost ninety-seven. - **SAYS** The language models were about ten thousand times slower than the classic methods. - **SAYS** This was one dataset with known attacks and small models, so bigger models and new kinds of attack are still untested. - **EXPLAINS** Removing timestamps, IP addresses and ports keeps models from memorising where attacks come from instead of learning how they behave. - **INTERPRETS** A fair reading is that a classic model should stay the default for high-volume traffic, with language models reserved for harder cases. ## Ask the paper ### Are language models better than XGBoost at detecting network attacks? Not in this study: XGBoost scored slightly higher, and ran about 10,000 times faster. ### Why did the researchers remove IP addresses from the data? So the models had to learn how attacks behave instead of memorising where they came from. ### Which traffic was hardest to classify? Scanning, which was often confused with ordinary household traffic by both the best language model and XGBoost. ## The paper - Title: Evaluating large language models effectiveness for flow-based intrusion detection: a comparative study with ML and DL baselines - Authors: Lorena Mehavilla, María Rodríguez, José García, Álvaro Alesanco - Venue: Artificial Intelligence Review, 9 January 2026 - DOI: [10.1007/s10462-025-11432-2](https://doi.org/10.1007/s10462-025-11432-2) - Licence: CC BY 4.0 [Read this page on the site](https://papersays.com/p/do-llms-beat-classic-models-at-catching-network-attacks)